Creating an AKS cluster with WebApplication Routing using Pulumi

I’ve recently starting looking at K8s and Observability, which required me to start thinking about how to create a Cluster. Life is too short to build your own, so naturally I turned to the Azure Kubernetes Service. Further to that, I needed to be able to iterate on it, and know how it’s configured, which meant using some kind of IaC. I’m a massive fan of Pulumi as it means I can build everything with C# and only learn a few SDK concepts of Pulumi to get things working.

In this post, I’ll go through how to get a basic AKS Cluster up and also how to add WebApplication Routing so that you can get an automatic single ingress which is routed using domains and paths set as annotations.

Step 1 – State storage

Pulumi requires a location for storing state. State is where Pulumi stores details of all the resources that it’s created, or knows about. This is what’s used to know when to deploy or change resources. For this you have 3 main options.

1. Local Storage on your file system
2. Azure Blob Storage
3. Pulumi Cloud

By far the easiest option is Pulumi Cloud, and for production, that’s what I’d recommend. For playing around, Local Storage on your file system is fine, but removes the ability to share your solution, and also use CI/CD. So in this post I’ll cover Azure Blob Storage.

Create a new Storage Account in Azure. Where this lives doesn’t matter, it can be in any resource group, it can also technically be in another subscription. For simplicity, I’d recommend

• A Separate Storage Account to any other data
• Keep it in the same subscription as the resources it manages
• Have a separate Resource group

The important part is permissions. All users who will be manipulating the stack need “Storage Blob Data Contributor” permissions for the full container. This means that you can create Pulumi Stacks.

Step 2 – Login to Azure Storage with Pulumi

Now that we have the datastore created, we need to setup our local machine to use that datastore to store the state. We do this by “logging in” from the Pulumi CLI. If you don’t already have the Pulumi CLI installed, you can get details on how to install if from the the Pulumi docs.

Before we can Login to the storage container, we also need to login to Azure using the Azure CLI, so you’ll need to install that and perform the login steps

For logging into an Azure state storage backend we use a command that looks like this

pulumi login azblob://<container-name>?storage_account=<storage account name>

The container-name here can be anything you’d like. My preference is to keep is contained to what “system” it’s part of. That’s because if you want to share state between multiple different applications that each have their own Pulumi infrastructure, they must be in the same container. The storage-account-name should be the one you created in step one.

Note: You may need to put single quotes around the azblob part in some shells (particularly zsh)

Step 3 – Build the infrastructure code

Now we’ve done the boring setup stuff, we can create our Pulumi infrastructure code. We do this using the Pulumi CLI. It has the ability to create a C# solution from scratch, and provides you with prompts to get you started.

pulumi new azure-csharp

From here, you’ll be asked for

• Project Name – This must be unique within the state container you created in Step 2
• Description – This doesn’t really matter, it’s useful in some state backends like Pulumi Cloud
• Stack Name – This is normally mapped to something like the environment it’s being deployed to like Dev/UAT/Staging/Prod)
• Azure Location – This the region you’ll be applying the infrastructure to. You can override this later if you want to deploy multiple regions etc.

This will create a C# project for you that contains all the right package references. However, it also adds some code to create a Resource Group and a Storage Account. We don’t need those, so we’ll remove them and replace them with our AKS cluster code.

Remove everything from the Program.cs file inside the RunAsync delegate and replace it with this:

    var config = new Config();
    var dnsResourceGroup = config.Require("dnsResourceGroup");
    var dnsZoneName = config.Require("dnsZoneName");
    var dnsZone = GetZone.Invoke(new GetZoneInvokeArgs{
        ResourceGroupName = dnsResourceGroup,
        ZoneName = dnsZoneName
    });

    var resourceGroup = new ResourceGroup("pulumi-blog-aks");

    var cluster = new ManagedCluster("pulumi-blog-aks", new ManagedClusterArgs
    {
        Sku = new ManagedClusterSKUArgs
        {
            Name = ManagedClusterSKUName.Base,
            Tier = ManagedClusterSKUTier.Free
        },
        DnsPrefix = "pulumi-blog-aks",
        ResourceGroupName = resourceGroup.Name,
        NodeResourceGroup = resourceGroup.Name.Apply(rg => $"{rg}-nodes"),
        Identity = new ManagedClusterIdentityArgs
        {
            Type = ResourceIdentityType.SystemAssigned
        },
        EnableRBAC = true,
        KubernetesVersion = "1.27.1",
        IngressProfile = new ManagedClusterIngressProfileArgs
        {
            WebAppRouting = new ManagedClusterIngressProfileWebAppRoutingArgs
            {
                Enabled = true,
                DnsZoneResourceId = dnsZone.Apply(z => z.Id)
            }
        },
        AgentPoolProfiles = new[]
        {
            new ManagedClusterAgentPoolProfileArgs
            {
                Name = "agentpool",
                Count = 1,
                Mode = AgentPoolMode.System,
                OsType = OSType.Linux,
                Type = AgentPoolType.VirtualMachineScaleSets,
                VmSize = VirtualMachineSizeTypes.Standard_A2_v2.ToString(),
                MaxPods = 110
            }
        }
    });

    const string DnsZoneContributorRoleDefinitionId = "/providers/Microsoft.Authorization/roleDefinitions/befefa01-2a29-4197-83a8-272ff33ce314";

    var roleAssignment = new RoleAssignment("cluster-dns-contributor", new()
    {
        PrincipalId = cluster.IngressProfile.Apply(ip => ip?.WebAppRouting!.Identity.ObjectId!),
        PrincipalType = PrincipalType.ServicePrincipal,
        RoleDefinitionId = DnsZoneContributorRoleDefinitionId,
        Scope = dnsZone.Apply(z => z.Id)
    });

    return new Dictionary<string, object?>
    {
        ["clusterName"] = cluster.Name,
        ["clusterResourceGroup"] = resourceGroup.Name
    };

There are a few important parts here that we’ll walk through.

The first is pulling in our config to get the DNS Zone information.

    var config = new Config();
    var dnsResourceGroup = config.Require("dnsResourceGroup");
    var dnsZoneName = config.Require("dnsZoneName");
    var dnsZone = GetZone.Invoke(new GetZoneInvokeArgs{
        ResourceGroupName = dnsResourceGroup,
        ZoneName = dnsZoneName
    });

This will get 2 config settings, and use those settings to identify the DNS Zone in Azure. This is the DNS Zone that AKS will use to add our Kubernetes ingresses to make them available for on a domain name rather than an IP address.

Next is the ResourceGroup and the Cluster itself. For this, we’re going to create a cluster with a “Standard A2 Version 2” class of node. All clusters need to have a Node Pool created that is marked as “System” at a minimum.

    var cluster = new ManagedCluster("pulumi-blog-aks", new ManagedClusterArgs
    {
...
    }

A couple of other important things to note here:

• DnsPrefix is for the management interface, you need to give it a name, but it doesn’t really matter
• IngressProfile is a new setting, and means you need to ensure you have the correct namespace imported (Pulumi.AzureNative.ContainerService.V20230502Preview)
• Identity must be set, otherwise there is no identity created, and therefore you can’t allow the cluster to managed DNS records.

The last resource allows the Cluster’s identity to manage DNS records.

    const string DnsZoneContributorRoleDefinitionId = "/providers/Microsoft.Authorization/roleDefinitions/befefa01-2a29-4197-83a8-272ff33ce314";

    var roleAssignment = new RoleAssignment("cluster-dns-contributor", new()
    {
        PrincipalId = cluster.IngressProfile.Apply(ip => ip?.WebAppRouting!.Identity.ObjectId!),
        PrincipalType = PrincipalType.ServicePrincipal,
        RoleDefinitionId = DnsZoneContributorRoleDefinitionId,
        Scope = dnsZone.Apply(z => z.Id)
    });

You’ll notice here that we have a constant with a “magic string”. This is because role definitions in Azure are strings with guids, and not easy to get hold of. Just paste it in and get on with your life 😀

There’s a bit of “magic” in the PrincipalId but it’s really just getting the managed identity from the cluster, and specifically the one used by the WebApplication Routing component.

Finally, we need to output the information about the cluster so that we can use that to login.

    return new Dictionary<string, object?>
    {
        ["clusterName"] = cluster.Name,
        ["clusterResourceGroup"] = resourceGroup.Name
    };

Step 4 – Bringing up the infrastructure

Now we have our infrastructure code, we need to get Pulumi to “Bring it up” or deploy it. This is done using the pulumi up command. However, first, we need to add some settings as our code requires it.

pulumi config set dnsResourceGroup <resource-group>
pulumi config set dnsZoneName <dns-zone-name>

Note: if you didn’t follow the start of this post to create a new project using the Pulumi CLI, you’ll also need to set a config for azure-native:location that points to the region you’re deploying in.

Now we can run pulumi up which will show us that we need to create 4 resources and ask us to confirm we want to do that

Once this is done you should have a cluster.

Step 5 – Testing the cluster

Now we have the cluster, it would be useful to be able to know how to workout if it “actually” did what we wanted. This is where our return parameters come in.

Pulumi has a great feature that allows us to take the outputs and set them as shell variables. You will, however, need to set your Passphrase as an environment variable first.

export PULUMI_CONFIG_PASSPHRASE=<your-passphrase>
eval $(pulumi stack output --shell)

This command will take the outputs from our stack (the cluster name and it’s resource group) and add them as environment variables. Which means we can pass them to an Azure command that will get us the credentials for our Cluster known as a “Kubeconfig”. This does mean that you’ll also need to have the Kubectl command installed locally, you can follow the steps from their docs

az aks get-credentials -n $clusterName -g $clusterResourceGroup --overwrite-existing

We can then run the kubectl command to see if we have some pods running. Specifically, we’ll look to see if we have pods running for the WebApplication Routing system.

kubectl get pods -n app-routing-system

You should see 3 pods, 2 are the nginx proxies, and the final one being the pod that will update the external DNS

Conclusion

Deploying AKS is a lot easier than I thought, there’s a lot of nuances in the advanced use cases and I’ll cover those in later posts.

You can find the repository on my Github if you want to just clone it and run it.