Helm 3 - CI/CD with Azure DevOps using Azure Container Registry (ACR) and Azure Kubernetes Service (AKS)Jan 30, 2021 | Utkarsh Shigihalli | 6 min read Helm Kubernetes Azure DevOps Azure Container Registry Azure Kubernetes Service AKS ACR
I have been bit late to Kubernetes world, but ever since I have started using it, I have been part of a great teams building awesome applications and using Helm package manager. With Helm, you package your Kubernetes application as charts, which are then stored in Helm chart repo. Helm also has a templating engine allowing you to set values in your charts dynamically allowing you to manage your applications more easily. Azure Container Registry (ACR) currently supports publishing Helm 3 charts to ACR and it is currently in preview.
In this post we will see how we can publish a sample Helm chart to ACR and also deploy the application to Azure Kubernetes Service (AKS) by consuming the published chart from ACR. We will also use ACR’s repository scoped tokens - a preview feature which offer great benefits.
We will cover following things in this post.
- Setting up the ACR to use repository scoped tokens
- Create scope-maps
- Create a token
- Create AKS instance on Azure
For CI and CD, I am going to use Azure DevOps YAML multi-stage pipelines in this post and our pipeline will be divided as below.
2) CI Stage
- Get the latest chart from GitHub
- Publish to ACR
3) CD Stage
- Pull the latest version of chart from ACR
- Deploy the pulled to AKS
- View the deployed service in Azure DevOps
You can authenticate with ACR in multiple ways.
Using a service principal - You can create separate service principals. This gives access to whole ACR and all the repositories inside that ACR instance. More info here
Using managed identity - Using user-assigned or system-assigned managed identity for easily consuming charts on an Azure VM. More info here
Using image pull secret - For your Kubernetes (K8s) cluster (including unmanaged AKS instance), you can also define a image pull secret, which then lets K8s cluster do pull images and run the application. More info on how you can do this with AKS is here
Using a repository scoped token - This feature of ACR is currently in preview and only available on Premium container registry service tier. This allows you to define scope maps and repository specific tokens for your ACR.
Setting up the ACR to use repository scoped tokens
I am going to use Tokens to authenticate ourselves with ACR. The biggest advantage is that, as a registry owner, you can have a single instance of ACR across your organisation and provide access to certain teams only selected repositories. There are other interesting scenarios highlighted in the documentation, mainly…
- Allow IoT devices with individual tokens to pull an image from a repository
- Provide an external organisation with permissions to a specific repository
- Limit repository access to different user groups in your organisation. For example, provide write and read access to developers who build images that target specific repositories, and read access to teams that deploy from those repositories.
If you have an existing ACR instance (which in my case I do), the first step is to upgrade your ACR instance to use
Premium and click
Creating scope maps
Once, you are on Premium tier, you will be able to to create a tokens. But before we can create tokens, let us define two scope maps. Scope maps when associated with tokens sets permissions on what can be done on ACR.
- chartpull - a scope limiting read permission to
helmdemo/vote-apprepo. I am using
- chartpush - a scope limiting write permission inside
helmdemo/vote-apprepo. I am using
The idea here is that, development team pushes the charts to ACR and some other team is consuming the charts/application. But you can extend this idea to let users external to your organisation to pull images only from specific repositories.
So to create scope map, go to
Scope maps under
Repository permissions sections and then click
+ Add. In my case, I am limiting these scopes only to
I follow the same process to create another scope named
chartpull scope with
- Notice that in
chartpushscope we also allow
content/readpermission. This is because, you need
content/readscope along with
content/writescope if you are going to push charts to ACR (for the repo defined in the scope).
- You can also use Azure CLI to do the same. Use
az acr tokencommand. Refer the documentation for more information.
Once you defined scopes, you are ready to create tokens. Go to
Repository permissions section and click
+ Add. Then name the token (e.g.
helmdemopull) and select the scope map you created above. In my case I am creating two tokens, one for pull and another one for push.
You can repeat the process to create another token. Refreshing the screen you will see something like this below.
You will notice that these tokens (
helmdemopush) do not yet have passwords generated. Lets create passwords. Click on the token, click the icon under
Actions column for either
password2. You will also have an option to set an expiration date for the password if you planning to allow token to be valid only for specific days.
2. CI Stage
Get the latest chart from GitHub
To get the latest source from the GitHub repo, in our YAML pipeline we add a
resources section and point to the repo on GitHub. We will also need to pass the
endpoint parameter with the name of the GitHub service connection. More info about how to do this is here and here.
resources: repositories: - repository: helmrepo type: github name: utkarshpoc/azure-vote-helm-chart endpoint: github
Publish chart to ACR
Before we write steps to publish to ACR, we need to ensure we store the tokens we generated in the pipeline as variables. Here I am adding tokens as variables using the variables UI of the pipeline and marking passwords as secret.
You can also define these in Azure Pipelines Library if you intend to use these tokens in multiple pipelines.
I also have few static and non-secret variables in YAML itself so that they are source controlled using
variables as below.
variables: acr.name: acrdemoutkarsh acr.repo.name: helmdemo/vote-app
The steps involved for publishing our Helm chart are simple.
- Installing Helm 3 on the agent
- Login to the ACR using Helm
- Save and push the chart
1. Installing Helm 3 on the agent
You can do this using OOB
HelmInstaller task. The YAML is as below. Here I am using
helmVersion, but you can stick specific to a version (3.x and above).
- task: HelmInstaller@0 displayName: install helm inputs: helmVersion: 'latest' installKubectl: false
2. Login to the ACR using Helm
We can use simple
script step to login to the registry.
- script: | helm registry login $(acr.name).azurecr.io --username $(acr.push.username) --password $(acr.push.password) displayName: login to acr using helm
But if you run this, at the time of writing you will get an error as below
Error: this feature has been marked as experimental and is not enabled by default. Please set HELM_EXPERIMENTAL_OCI=1 in your environment to use this feature
As the error states, this is because, publishing Helm charts which follow Open Container Initiative (OCI) standard is experimental and you are required to set
1. We can do this by updating our
variables section in the YAML. The updated yaml looks as below.
variables: acr.name: acrdemoutkarsh acr.repo.name: helmdemo/vote-app HELM_EXPERIMENTAL_OCI: 1
3. Save and push the chart to ACR
Next step is to save this chart locally and create an alias for the chart with our ACR registry URL. So our YAML looks as below.
- script: | helm chart save $(build.sourcesdirectory)/src/azure-vote-helm-chart/ $(acr.name).azurecr.io/$(acr.repo.name):latest displayName: save the chart and set the alias
And finally, we push the Helm chart to ACR using
helm chart push command.
- script: | helm chart push $(acr.name).azurecr.io/$(acr.repo.name):latest displayName: push the chart to acr
So our CI stage is now complete.
2. CD Stage
Pull the latest version of chart from ACR
The first step is to pull the latest version of the chart (one with tag
latest) to our agent machine and extract it in a folder. We can do that in pipeline as below. Notice I have added a new stage name
cd and its dependent of stage
- stage: cd displayName: CD dependsOn: ci jobs: - deployment: helm_publish_aks displayName: deploy to aks environment: name: PROD resourceName: helmdemo resourceType: Kubernetes strategy: runOnce: deploy: steps: - task: HelmInstaller@0 displayName: install helm inputs: helmVersion: 'latest' installKubectl: false - script: | echo "$(acr.pull.password)" | helm registry login $(acr.name).azurecr.io --username $(acr.pull.username) --password-stdin displayName: login to acr using helm - bash: | helm chart pull $(acr.name).azurecr.io/$(acr.repo.name):latest displayName: get helm chart on agent - bash: | helm chart export $(acr.name).azurecr.io/$(acr.repo.name):latest --destination $(build.stagingdirectory) displayName: export the chart to folder
Because each job in Azure DevOps run in a separate agent, I have to ensure agent has Helm tool, so install Helm tool again in the first step. Also, note that I am using token with scope-map permission set only to pull charts.
Deploy the chart to AKS
Now that the latest chart is pulled from our ACR and is available on agent, only step remaining is to deploy to AKS. We can do that using
- task: HelmDeploy@0 displayName: deploy chart to aks inputs: connectionType: 'Azure Resource Manager' azureSubscription: '$(azure.service.connection)' azureResourceGroup: 'demos' kubernetesCluster: 'aksdemoutkarsh' namespace: 'helmdemo' command: 'upgrade' chartType: 'FilePath' chartPath: '$(build.stagingdirectory)/azure-vote/' releaseName: 'helmdemo' arguments: '--create-namespace --install'
View the deployed service in Azure DevOps
If you have added your Kubernetes cluster as a resource in Azure DevOps environment, you can now view deployed services directly from Azure DevOps.
You can also view deployed services in YAML and also see any ports exposed.
Browsing the exposed port, you will see application.
That is it! If you have read this far, thank you 🙏🏼. As we saw, with the support of tokens and OCI artifacts, ACR is one of the best in class container registry. Also, with deep integration with AKS (features like browsing services, logs, yaml), Azure DevOps brings in lots of productivity for your teams. Hope you found this post informative. If so, please share and tweet!