Creating an AKS cluster with an Application Gateway and Pulumi

I’m a huge fan of Pulumi and I’ve had the need to create an AKS cluster recently so thought I’d give it a go with Pulumi. It turns out that adding an Application Gateway isn’t the easiest thing to do.

What is Application Gateway?

Azure Application Gateway is the Layer 7 (HTTP level essentially) load balancer that allows you to route traffic from a single endpoint to multiple backends using the path. It’s similar to what AWS calls the Application Load Balancer (or ALB). Ultimately, it’s a single public endpoint, and allows your services to sit in a private area. In relation to AKS (Azure Kubernetes Service) it can have a direct link into the pods to make them public and also provide additional services like WAF (Web Application Firewall) and SSL Termination.

Getting Started

The first thing you need to do is login to Pulumi. The easiest approach if you don’t have a Pulumi Cloud account is to use the local file state, which is similar to how Terraform would do it.

pulumi login file://pulumi-state

This will create a directory in your current path that will store the state of your stacks. Make sure to ignore this if you add the solution to git.

Next, you’ll need to login to the Azure Subscription you want to deploy the cluster to. This requires the Azure CLI to be installed, and for you to have access to an Azure subscription you can deploy to.

az login

Then we can create a initial template app from the Pulumi command line.

pulumi new azure-csharp

We should now have a C# project with the correct Pulumi libraries installed.

Adding an AKS Cluster

When I’m creating isolated segments of my infrastructure in Pulumi, I like to encapsulate them in a ComponentResource, this is nice as it provides a logical boundary around all the bits that are need for that segment.

public class AKSCluster : ComponentResource
{
    public AKSCluster(string name, AKSClusterArgs? args, ComponentResourceOptions? options = null)
        : base("aks-demo:aks:cluster", name, args, options)
    {
    }
}
public class AKSClusterArgs : Pulumi.ResourceArgs
{
}

We can then add in the bits we need. The Prerequistes for an Azure ManagedCluster resource are a Resource Group, a ServicePrincipal and an SSH Key.

        // Create an Azure Resource Group
        var resourceGroup = new ResourceGroup(name);

        // Create an AD service principal
        var adApp = new Application(name, new ApplicationArgs
        {
            DisplayName = name
        });
        var adSp = new ServicePrincipal("aksSp", new ServicePrincipalArgs
        {
            ApplicationId = adApp.ApplicationId
        });
        var adSpPassword = new ServicePrincipalPassword("aksSpPassword", new ServicePrincipalPasswordArgs
        {
            ServicePrincipalId = adSp.Id,
            EndDate = "2099-01-01T00:00:00Z"
        });

        // Generate an SSH key
        var sshKey = new PrivateKey("ssh-key", new PrivateKeyArgs
        {
            Algorithm = "RSA",
            RsaBits = 4096
        });

None of the bits here are of interest, just that they exist and have names.

Now we have the prerequisites we can create the cluster.

        var cluster = new ManagedCluster(name, new ManagedClusterArgs
        {
            ResourceGroupName = resourceGroup.Name,
            AgentPoolProfiles =
            {
                new ManagedClusterAgentPoolProfileArgs
                {
                    Count = 3,
                    MaxPods = 110,
                    Mode = AgentPoolMode.System,
                    Name = "agentpool",
                    OsType = OSType.Linux,
                    Type = AgentPoolType.VirtualMachineScaleSets,
                    VmSize = "Standard_DS2_v2",
                }
            },

            DnsPrefix = "AzureNativeprovider",
            EnableRBAC = true,
            KubernetesVersion = "1.26.3",
            LinuxProfile = new ContainerServiceLinuxProfileArgs
            {
                AdminUsername = "testuser",
                Ssh = new ContainerServiceSshConfigurationArgs
                {
                    PublicKeys =
                    {
                        new ContainerServiceSshPublicKeyArgs
                        {
                            KeyData = sshKey.PublicKeyOpenssh,
                        }
                    }
                }
            },
            NodeResourceGroup = $"{name}-nodes",
            ServicePrincipalProfile = new ManagedClusterServicePrincipalProfileArgs
            {
                ClientId = adApp.ApplicationId,
                Secret = adSpPassword.Value
            }
        });

Again, there’s nothing special here, it’s things that need to exist. You should pay attention to the AgentProfiles part, particularly the Count of the number of worker nodes and the VmSize as those will be the cost drivers.

I’d suggest that you also want to add an output for the resource group and the name of the cluster so you can access them:

    public AKSCluster() { ...     
       // aks cluster setup code

        this.ClusterName = cluster.Name;

        this.ClusterResourceGroup = resourceGroup.Name;

    }

    [Output("clusterName")]

    public Output<string> ClusterName { get; set; }

    [Output("clusterResourceGroup")]

    public Output<string> ClusterResourceGroup { get; set; }

Now you can add this to the Stack in your program.cs

using System.Collections.Generic;

return await Pulumi.Deployment.RunAsync(() =>
{
    var cluster = new AKSCluster("aks-otel-demo", new AKSClusterArgs());

    // Export the primary key of the Storage Account
    return new Dictionary<string, object?>
    {
        ["clusterName"] = cluster.ClusterName,
        ["clusterResourceGroup"] = cluster.ClusterResourceGroup
    };
});

Now we should be able to run pulumi to bring up the cluster.

pulumi up

Once this has been created, you can use the outputs to get the kubectl context

az aks get-credentials -g <resourceGroup> -n <clusterName>

Adding the Application Gateway

There are 2 ways to add the Application Gateway to the cluster. The easy way, and the production way.

In large production systems, you’ll want to manage the Gateway in some very specific ways such as custom VNETs, peering, WAF rulesets, etc. To do this, you will need to create a separate Application Gateway resource that I won’t cover here. Once you have the gateway you’ll need the Id and you can then add that to the cluster.

The key part from a pulumi perspective (and Bicep, or anything based on ARM Templates and the rest APIs) is the property called AddonProfiles. Inparticular, a key called `IngressApplicationGateway”.

The easy way… this is to have the AKS cluster build the Gateway for you and add a subnet for you inside the managed cluster.

            AddonProfiles = {
                ["IngressApplicationGateway"] = new ManagedClusterAddonProfileArgs {
                    Enabled = true,
                    Config = {
                        ["subnetCIDR"] = "10.225.0.0/16"
                    }
                }
            },

As this is part of the Cluster, you don’t get the ability to configure it very much, which is a downside. For small applications, that will be perfectly fine I would think. For me, I’m mainly using this to demo applications, so that isn’t really a concern for me.

If you already have an ApplicationGateway, you can configure the access and the specific Ingress controller using this.

            AddonProfiles = {
                ["IngressApplicationGateway"] = new ManagedClusterAddonProfileArgs {
                    Enabled = true,
                    Config = {
                        ["applicationGatewayId"] = <myId>
                    }
                }
            },

Getting the Application Gateway IP

Unfortunately, you don’t have access to the generated Application Gateway IP, so there’s a few things that you need to get that. Fortunately, we are able to get the Id of the gateway, meaning that we can use that to get access to all the things we need.

First, we need to install the Azure.Core package so we can get parse the Id to get all the required details in a way that’s safer than assuming names and resource groups. Then we can do the following (rather complicated code)

        GatewayIp = cluster.AddonProfiles.Apply<string?>(a => {
            var appGatewayResourceId = new Azure.Core.ResourceIdentifier(
                a!["IngressApplicationGateway"]!
                    .Config!["effectiveApplicationGatewayId"]);

            var appGatewayDetails = GetApplicationGateway.Invoke(new GetApplicationGatewayInvokeArgs {
                ApplicationGatewayName = appGatewayResourceId.Name,
                ResourceGroupName = appGatewayResourceId.ResourceGroupName!
            });
            return appGatewayDetails.Apply<string?>(a => {
                var publicIpId = a?.FrontendIPConfigurations.First()?
                    .PublicIPAddress?.Id;
                if (publicIpId == null)
                    return "";

                var publicIpResourceId = new ResourceIdentifier(publicIpId);
                var publicIp = GetPublicIPAddress.Invoke(new GetPublicIPAddressInvokeArgs {
                    PublicIpAddressName = publicIpResourceId.Name,
                    ResourceGroupName = publicIpResourceId.ResourceGroupName!
                });
                return publicIp.Apply(a => a.IpAddress);
            });
        });

Ultimately, this is:

Getting the ApplicationGatewayId that Cluster created
Parsing that with Azure.Core.ResourceIdentifier
Getting the full details of the ApplicationGateway from Azure
Getting the Id of the Public IP address associated with the Gateway
Getting the full details of the Public IP Address from Azure
Returning the actual IP address associated with that resource.

You can now use this IP, or just put it as an output.

Conclusion

Setting up AKS as IaC is actually pretty easily, but Application Gateway integration is not. I hope this gets you a little further in your journey!

In the next post, I’ll couple this with deploying a helm chart and coupling that to an ingress.