In my last post, I described how you can build OpenTelemetry Collector container images that are secure by default. In this post, we’ll take it a step further by building and pushing that image to Azure Container Registry and hosting it in Azure Container Apps.
There is a Github repository if you just want to skip to the content.
Components
To host this container, we’ll be using a public Container Apps Environment that will give us a public URL. I’ll cover hosting this inside an existing VNET for private access in a later post.
We’ll be using Azure Container Registry for hosting the image so we can keep everything inside Azure, and inside of a single Resource Group.
Step 1: Create the registry and extract credentials
The first step is to create the Azure Container Registry, and then make a call to get the credentials.
Note: Due to naming collisions of type names in Pulumi, I find it’s good practice to add using directives that redirect namespaces and make the code more readable. In this example, I’ve added using ACR = Pulumi.AzureNative.ContainerRegistry;
var registry = new ACR.Registry("securecollector", new()
{
AdminUserEnabled = true,
ResourceGroupName = resourceGroup.Name,
Sku = new ACR.Inputs.SkuArgs
{
Name = ACR.SkuName.Basic,
},
});
var credentials = Output
.Tuple(resourceGroup.Name, registry.Name)
.Apply(items =>
ACR.ListRegistryCredentials.InvokeAsync(new ACR.ListRegistryCredentialsArgs
{
ResourceGroupName = items.Item1,
RegistryName = items.Item2
}));
var adminUsername = credentials.Apply(credentials => Output.CreateSecret(credentials.Username));
var adminPassword = credentials.Apply(credentials => Output.CreateSecret(credentials.Passwords[0].Value));
Unfortunately, although we can use Entra ID identities here (Service Principals), there are some downsides which is why I prefer the approach of long-lived credentials. There are 2 alternative approach that I’ll outline here, but I won’t be covering in detail.
- System Managed Identity
You can set the Container App to use a System Managed Identity, so that the App creates it’s own Identity when it’s created, this is normally my preferred approach for Azure resources. However, for Container apps, it’s brings in a race condition as your container can’t start until you’ve added the required permissions to the identity, but you don’t have the Identity until it’s been created. - User Managed Identity
You can set the Container App to use a User Managed Identity, this means you create the identity ahead of time. This seems great, until you understand that creating users requires a higher level of permission than assigning roles. Therefore your CI/CD pipeline will now need access to create Service Principals which not ideal in a secure environment.
My hope is that Azure decide to add the ability to use an identity from the Container Apps Environment identity for pulling images. That way you’ll have the identity at the point you create the environment. This is similar to the Execution Role vs Task Role separation that AWS provides for Fargate tasks.
You’ll also notice here that I’m using the syntax Output.CreateSecret() for the username and password. This is important to ensure that these credentials don’t make it into the state files.
Step 2: Build and Push the image
Next up is building and pushing the image. This is done using the Pulumi.Docker package.
var image = new Pulumi.Docker.Image("collector-image", new()
{
ImageName = Output.Format($"{registry.LoginServer}/securecollector:latest"),
Build = new Pulumi.Docker.Inputs.DockerBuildArgs
{
Context = "../../docker-collector/",
},
Registry = new Pulumi.Docker.Inputs.RegistryArgs
{
Server = registry.LoginServer,
Username = adminUsername!,
Password = adminPassword!,
},
});
Be sure that your context doesn’t share a path tree with your infrastructure. I normally have a directory structure such that the infra and docker directories are at the same level. This ensure that you don’t end up rebuilding the container each time you change infrastructure code.
The image name needs to be in the format of <registry-url>/<imagename>:<version> and we’re doing that here using the Output.Format() method as we don’t know the url of the registry until it’s been created.
The dockerfile is taken from the previous post, so it’s using the building, and all you need to do is make sure that your OpenTelemetry Collector Config file is in that directory.
FROM ghcr.io/martinjt/ocb-config-builder:latest as build
COPY config.yaml /config/config.yaml
RUN /builder/build-collector.sh /config/config.yaml
FROM cgr.dev/chainguard/static:latest
COPY --from=build /app/otelcol-custom /
COPY config.yaml /
EXPOSE 4317/tcp 4318/tcp 13133/tcp
CMD ["/otelcol-custom", "--config=/config.yaml"]
Note: You MUST ensure that you have the Healthcheck extension configured in your collector config so that a healthcheck can be setup in container apps to the separate port.
Step 3: Create the Container Apps Environment
This is the simple part, and the only change from the default here is that I’m ensuring that we don’t create logs and push them to Log Analytics. This is debatable, and potentially the only time I would consider sending logs to Log Analytics.
var containerAppEnvironment = new ManagedEnvironment("collector-env", new ManagedEnvironmentArgs
{
ResourceGroupName = resourceGroup.Name,
AppLogsConfiguration = new AppLogsConfigurationArgs
{
Destination = ""
},
});
Step 4: Create the Collector app
This is the main part, and requires a fairly large setup. We’ll split this into 2 parts.
Generic configuration:
var collectorApp = new ContainerApp("collector", new ContainerAppArgs
{
EnvironmentId = containerAppEnvironment.Id,
ResourceGroupName = resourceGroup.Name,
ContainerAppName = "collector",
Configuration = new ConfigurationArgs
{
Ingress = new IngressArgs
{
External = true,
TargetPort = 4318
},
Secrets = {
new SecretArgs
{
Name = "honeycomb-api-key",
Value = config.RequireSecret("honeycomb-api-key")
},
new SecretArgs
{
Name = "registry-pwd",
Value = adminPassword!
}
},
Registries =
{
new RegistryCredentialsArgs
{
Server = registry.LoginServer,
Username = adminUsername!,
PasswordSecretRef = registryPasswordSecret.Name,
}
},
},
In this part, we setup the ingress to make sure it’s externally facing, tell the app the credentials for the registry, and setup the secrets that are available. This is also where we link the secret for the registry password to the App.
The next part is where we setup the container itself.
Template = new TemplateArgs
{
Containers = {
new ContainerArgs
{
Name = "collector",
Image = image.ImageName,
Env = {
new EnvironmentVarArgs {
SecretRef = honeycombApiKeySecret.Name,
Name = "HONEYCOMB_API_KEY"
}
},
Probes = {
new ContainerAppProbeArgs {
HttpGet = new ContainerAppProbeHttpGetArgs {
Path = "/",
Port = 13133,
},
Type = Type.Readiness
},
new ContainerAppProbeArgs {
HttpGet = new ContainerAppProbeHttpGetArgs {
Path = "/",
Port = 13133,
},
Type = Type.Liveness
}
}
}
},
}
The important parts here are the fact that we’re using the name of the image from the Pulumi.Docker object we created, we’re adding an environment variable from the pulumi secret. Finally we’re setting up Readiness and Liveness checks so that they use a different port than the ingest port.
That’s it, you can have an output now using collectorApp.LatestRevisionFqdn that will give you the URL to you new collector.
Conclusion
Deploying using Container apps is fairly simple, but still has some holes. In a future post, we’ll cover how to secure our new public collector.
Have fun!
MartinDotNet 
Leave a comment