Perfecting Continuous Delivery of NuGet packages for Azure Artifacts

Jun 18, 2019 |

Twitter Facebook Linkedin Email WhatsApp

In order to release software often and consistently, it is essential that software dependencies are managed using a good package management solution. Managing dependencies, if not planned well, can over a period of time become extremely difficult to maintain - especially due to difficulty in managing versions, testing of the packages and nested dependencies. With the increased focus from organizations to break monolithic applications into micro-services, teams have started to break their hard dependencies into manageable packages (NuGet, npm and others.)

Azure Artifacts is Microsoft’s solution to package management. Originally available as a separate extension on Visual Studio Marketplace, it is now pre-installed in Azure DevOps Services and Azure DevOps Server 2019, TFS 2018, and 2017.

This post was originally published on Microsoft TechNet UK - TechNetUK

Why this post

Setting up Continuous Integration and Delivery (CI/CD) for NuGet packages is covered previously in my blog post and also in my recent book on Azure DevOps. There is also enough information about this in Microsoft docs.

However, I have seen most clients I have worked with struggle mainly after setting up CI/CD for their NuGet package. Without proper planning, managing and continuous delivery of NuGet packages become cumbersome.

Organizations are specifically interested to know,

What branching strategy to use
What will be developer workflow and how will developers add fixes to already released NuGet packages
What will be their versioning strategy for NuGet packages in continuous integration scenarios
How will the typical build pipeline for NuGet package looks like
Within the organization how to ensure teams only consume stable NuGet packages and unstable packages are made available to few selected users for testing purposes.

This post tries to answer these questions and specifically shows how Azure Artifacts becomes an answer to all the above questions.

Refresher on setting up Azure Artifacts

If you have never used Azure Artifacts, here is a quick refresher on getting started.

Creating a feed

Setting up artifacts is easy. Head over to Artifacts service and click on + New feed. You will be prompted with the screen below.

Ensure you also select Use packages from public sources through this feed option. This will allow Azure Artifacts to cache your package dependencies in Azure Artifacts so that they are available even if your upstream source npmjs.com or nuget.org is offline.

Feed settings

Once you create the feed, on the top right corner, you will see Feed Settings menu and clicking that will take you to a page where you can configure additional settings.

Inside Feed settings screen you can configure retention policies for your packages, add additional upstream sources etc, views (we will cover more about views soon).

Branching strategy

For NuGet packages, at least for our organization, we decided to use GitHubFlow branching strategy. It is simple to understand and idea is that you will always have a stable master branch and all development is carried out in feature branches and merged back to master.

Picture credit: https://guides.github.com/introduction/flow/

For effective working of GitHubFlow branching strategy, teams have to agree on following rules.

master branch is always in a state that it could be released.
Pull requests should not be merged to master until they are ready to go out.

This GitHubFlow branching strategy allows us to release stable NuGet packages from master with the assumption that we will always have a stable NuGet package. Any under development NuGet packages from feature\* branches will still be published to Artifacts but as you will see later in this article - it is only made available for other interested developers so that they consume, test and provide feedback.

Developer workflow

Typical developer workflow in our teams delivering NuGet packages is as below.

Developer pulls the latest changes from the master branch on to his local machine.
The developer then creates a feature branch from master. e.g feature\my-awesome-feature
Makes the changes to the code and commits the changes in feature branch.
As soon as the commit is made and synced with Azure DevOps, Azure DevOps CI pipeline triggers.
CI build creates the NuGet package, versions it (tag NuGet package alpha)
Pushes the package to Artifacts.
The developer now consumes the latest alpha NuGet package from Artifacts and tests it locally.
Once the developer is satisfied, he is ready to make the pull request to merge the changes from feature branch to stable master branch.
Makes PR (pull request) to master, allowing others to validate and provide review comments if any.
Now CI build triggers for the master branch.
CI build compiles the code and validates changes.
Once the build is successful, the new NuGet package from master (without alpha tag) is pushed to the Artifacts feed.

Versioning Strategy

Packages are immutable - this means once you publish a particular version of a package to a feed, that version number is permanently reserved. You cannot upload a newer revision package with that same version number, or delete it and upload a new package at the same version

Since NuGet packages are immutable, how you version your NuGet package becomes a very key thing to consider.

Microsoft recommends that NuGet versions should ideally convey 3 pieces of information.

the nature of the change,
the risk of the change,
and the quality of the package.

By using semantic versioning we can convey both nature (1) and risk of change (2). We use something known as a tag for conveying the quality of change (3).

It will be our practice that our NuGet packages follow versions in <major>.<minor>.<patch>-<tag> format.

major version when you make incompatible API changes,
minor version when you add functionality in a backwards-compatible manner, and
patch version when you make backwards-compatible bug fixes.
tag will help us specify the quality of our changes, i.e if it is coming from feature branch our NuGet package will have the tag alpha - Example: 1.0.0-alpha001 and if it is coming from master we do not apply any tag.

Following semantic versioning notation requires developers to upfront decide what kind of changes they will be making (major, minor or backward compatible change) to NuGet packages. In a CI scenario, we would like to automate this versioning. We will soon see how we do it in our pipeline.

Git versioning

We use GitVersion tool to automatically follow our versioning strategy described above. The tool relies on a simple configuration file committed to the same repository as our NuGet code. This tool automatically determines the semantic version based on the commit history on the repository.

Configuration file You create configuration (gitversion.yml) file using gitversion init command. Our configuration is as below.

mode: Mainline
next-version: 1.0.0
branches:
  feature:
    tag: alpha
  master:
    tag: ''
ignore:
  sha: []

As you can probably guess from the configuration file above, we are using Mainline versioning mode. We then specify that our initial version for this repository is 1.0.0. Using mainline mode will also ensure that patch version is incremented every merge to master.

Next, using the branches section, we specify that for feature branches we tag versions with alpha string and for master branch we don’t apply any additional tag (so that the packages from the master will just have versions like 1.0.1 for example).

For more information on installing and using GitVersion tool from command-line refer the documentation

Manually updating versions

Although using GitVersion tool allows us to automatically version our build and NuGet packages, developers need to override the version as they will decide whether to update major or minor version based on the change they are making. They can do so using the following approach and GitVersion respects these.

Using commit messages - For example
- Adding +semver: breaking or +semver: major will cause the major version to be increased.
- Adding +semver: feature or +semver: minor as a commit message will cause the minor version to be increased.
- Similarly using +semver: patch updates the patch version.
Branch name
- If you create a branch with the version number in the branch name such as release-1.2.0 or feature/1.0.1 then GitVersion will take the version number from the branch name as a source.
Using git tags
- By tagging a commit, GitVersion will use that tag for the version of that commit, then increment the next commit automatically based on the increment rules for that branch

Marketplace Extension

This tool is available as an Azure DevOps pipeline extension. By installing this extension into your organization you will be able to use this tool in your Azure pipelines.

Link to the extension: https://marketplace.visualstudio.com/items?itemName=gittools.gitversion

Build pipeline

Now that we know how our branching looks, what our versioning strategy will be, its easy to create the build pipeline to do that in Azure DevOps.

One build pipeline for all branches

As you saw in the developer workflow above, we need to build and publish a NuGet package for both master and feature branches. Azure DevOps allows us to use the one build definition to build for both the branches. We do that by going to Triggers hub in the edit mode of the pipeline and enable continuous integration. We also add Branch filters to include only master and feature/* branches. This will automatically trigger our build every time there is a commit in any of those branches.

Steps in our build pipeline

CI pipeline for our .NET Core NuGet package looks as below. The YAML content of the full pipeline is as below.

pool:
  name: LabAgents

variables:
  BuildConfiguration: 'release'

steps:
- task: gittools.gitversion.gitversion-task.GitVersion@4
  displayName: GitVersion
  inputs:
    preferBundledVersion: false

- task: DotNetCoreInstaller@0
  displayName: 'Use .NET Core sdk 2.2.104'
  inputs:
    version: 2.2.104

- task: DotNetCoreCLI@2
  displayName: 'dotnet restore from feed'
  inputs:
    command: restore
    projects: '$(Parameters.projects)'
    vstsFeed: '168c2416-9cd1-4e62-e89vb-5665da67a44c'
    includeNuGetOrg: false
    versioningScheme: byBuildNumber

- task: DotNetCoreCLI@2
  displayName: 'dotnet build'
  inputs:
    projects: '$(Parameters.projects)'
    arguments: '--configuration $(BuildConfiguration) /p:Version=$(GitVersion.NuGetVersion)'
    versioningScheme: byBuildNumber

- task: DotNetCoreCLI@2
  displayName: 'dotnet pack'
  inputs:
    command: pack
    packagesToPack: '$(Parameters.projects)'
    nobuild: true
    versioningScheme: byEnvVar
    versionEnvVar: GitVersion.NuGetVersion

- task: DotNetCoreCLI@2
  displayName: 'dotnet nuget push'
  inputs:
    command: push
    publishVstsFeed: '168c2416-9cd1-4e62-e89vb-5665da67a44c'
    versioningScheme: byBuildNumber

Notice, we have added GitVersion task (a first task in the YAML) to automatically version our build number (which uses gitversion.yml) and again in dotnet pack task (a fifth task in the YAML) to automatically version our NuGet packages using provided variable GitVersion.NuGetVersion.

Once you run the build, the versions will automatically be determined and applied.

Notice, commits from the feature branch has the tag alpha and builds on master branch does not have any tag as specified in the gitversion.yml file.

Ensure teams consume only stable NuGet packages

Another big challenge with traditional package management solutions is, as soon as you publish, the NuGet package is made available to everyone immediately. This requires consumers to understand semantic versioning and based on the version number figure out if something is a breaking change and then decide whether to use the package. This also slows down the developers of the NuGet package as they have to now make sure packages are thoroughly tested before releasing to the public. Azure DevOps helps you to solve it with a concept called as Views.

Views in Azure Artifacts

From the Microsoft docs,

Views allow to share package-versions that have been tested, validated, or deployed but hold back packages still under development and packages that didn’t meet a quality bar.

By default, Azure Artifacts provide 3 views: @local, @prerelease, and @release. The @local view is a special view and cannot be renamed.

Granular permission management for views

You can apply fine-grained control on who has access to which views under Feed settings.

In our organization, @local is available only to developers and architects. @prerelease view is accessible for Testers/early adopters. Finally, packages in @release view are available to all the users of the organization.

For more on Security and Permission management refer documentation

Delivering and promoting packages in views

The workflow of delivering packages using views is as below.

Hence, in our organization, we publish both alpha packages (packages from feature branches) and non-alpha packages (packages from master branch) are always published to @local view first. That is,

On successful completion of CI (from feature and master branches), the package is published to @local feed. Note, packages from feature branches are tagged with special tag alpha conveying the quality of the package.
When a package is ready for early adopters, developers select that package and its dependency graph and promote it to the @prerelease view.
When the package is deemed of sufficient quality to be released, we promote that package into the @release view.

Conclusion

As you saw in this post, there are a lot of things to consider for faster delivery of NuGet packages in continuous integration scenarios. As you saw Azure Artifacts has lots of great features like upstream sources and views etc. to help meet your needs. With the right branching and versioning strategy, you can set up continuous delivery with minimum human intervention so that you can deliver your features faster.

I hope this post helped you answer a few of the other things to consider while you plan to deliver your NuGet packages.