As your team starts to deploy resources to Kubernetes regularly, it becomes necessary for you as a cluster administrator to maintain good standards and consistency of the Kubernetes resources. Be it, ensuring all the resources have set of labels, or ensuring you only pull images from your enterprise container registry. Gatekeeper is a well known policy enforcement tool using Open Policy Agent (OPA) - which is a opensource, Cloud Native Computing Foundation (CNCF) project.

But did you know you can validate policies on your Kubernetes manifests before you deploy them on to the cluster? In this post, we will see how we can govern our deployments using Conftest and OPA policy agent.

However, Gatekeeper is installed on the cluster and thus ensures no policy is broken at deployment time. This means that any validation of policies happen only when you are trying to deploy resources to cluster. While this ensures that no resource violates the policy, you would like to know about these policies much earlier in your CI/CD pipeline. Doing policy validations much to the left of your deployment pipeline ensures your deployment going smooth when necessary.

This is where Conftest helps. Conftest relies on OPA and policies are written using Rego - thus the policies you write for Gatekeeper will be compatible with Conftest. But more importantly with Conftest, you can validate your local manifests against OPA policies locally and ensure your resources are compliant before you deploy them.

Installation

Installation is really easy if you are on Mac - For other platforms refer to the documentation

brew install conftest

Folder structure

By default, Conftest expects you to maintain your policies under policy folder at the same location as your Kubernetes resources. If you prefer a different path, you will want to pass it using CLI or set environment variable CONFTEST_POLICY.

📂 src
    📂 k8s
        📄 deployment.yml
        📄 service.yml
        📁 policy
            📄 replica.rego
            📄 labels.rego
    📂 app
        📄 main.ts
        📄 package.json

Writing Policies

As mentioned previously, policies are written in Rego. I struggled to write policies initially and constantly went back to documentation. However, once you write couple of policies, you will get a hang of it. Take a look at the simple policy to check every deployment has at least 2 replicas.

package main

deny_replicas[msg] {
    input.kind == "Deployment"                          # check if it is a Deployment
    input.spec.replicas < 2                             # And the replicas are < 2
    msg := "Deployments must have 2 or more replicas"   # show the error message and fail the test
}

input is the complete yaml document from our deployment yaml (see below) and we we are checking if kind is equal to Deployment. If its deployment, we move to next line in the constraint and check if spec.replicas is less that 2.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mynodeapi-dep
  labels:
    app: dep-k8s-nodejs-api
spec:
  replicas: 1
  selector:
    ...

You can write other policies similar to the one above to validate various aspects of Kubernetes resources. Let us see few examples.

This policy validates our resources have the required labels and fail if any labels from required_deployment_labels object are not found.

package main

import data.kubernetes

name = input.metadata.name

required_deployment_labels {
	input.metadata.labels["app.kubernetes.io/name"]
	input.metadata.labels["app.kubernetes.io/instance"]
	input.metadata.labels["app.kubernetes.io/version"]
	input.metadata.labels["app.kubernetes.io/component"]
	input.metadata.labels["app.kubernetes.io/part-of"]
	input.metadata.labels["app.kubernetes.io/managed-by"]
}

violation[msg] {
	input.kind == "Deployment"
	not required_deployment_labels
	msg = "Must include Kubernetes recommended labels: https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/#labels"
}

Reference container images only from our enterprise Azure Container Registry

package main

deny[msg] {
    input.kind == "Deployment"
    some i
    image := input.spec.template.spec.containers[i].image
    not startswith(image, "myacr.azurecr.io") # validate images start with endpoint for our container registry
    msg := sprintf("image '%v' comes from untrusted registry", [image])
}

As you can see rules can be very powerful.

Testing

The command to test resources using Conftest is conftest test <PATH>. Since I would like to test all resources under k8s folder, I pass folder path as below.

conftest test ./k8s

Running this you will see the output as below (ignore other errors as I have other policies). The test failed because we have set deny rule if spect.replicas < 2 and in our case our deployment yaml has replicas: 1 (see spec section in the deployment yaml above).

Conftest failing due to policy violation

Using Conftest in GitHub Actions

Making Conftest work in your Continuous Integration (CI) process is simple. For demo purposes, I am using GitHub Actions in my repo here. If you run the tests, you will see the action fails with errors - see the output

My action workflow looks like below.

name: build

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/[email protected]

      - name: install conftest
        run: |
          wget https://github.com/open-policy-agent/conftest/releases/download/v0.30.0/conftest_0.30.0_Linux_x86_64.tar.gz
          tar xzf conftest_0.30.0_Linux_x86_64.tar.gz
          sudo mv conftest /usr/local/bin
          rm -rf conftest_0.30.0_Linux_x86_64.tar.gz

      - name: run conftest
        run: |
          conftest test $/k8s

Conclusion

As you can see, Conftest lets you validate and govern your Kubernetes resources efficiently and can easily be integrated with your CI workflows. This lets your team standardise the common practices, go through PR review process before eventually deploying to the cluster. Once deployed to cluster, you can use Gatekeeper to validate as well to full proof your workloads.


About author
Utkarsh Shigihalli
Utkarsh Shigihalli
Utkarsh is passionate about software development and has experience in the areas of Azure, Azure DevOps, C# and TypeScript. Over the years he has worked as an architect, independent consultant and manager in many countries including India, United States, Netherlands and United Kingdom. He is a Microsoft MVP and has developed numerous extensions for Visual Studio, Visual Studio Code and Azure DevOps.
We Are
  • onlyutkarsh
    Utkarsh Shigihalli
    Microsoft MVP, Technologist & DevOps Coach


  • arora_tarun
    Tarun Arora
    Microsoft MVP, Author & DevOps Coach at Avanade

Do you like our posts? Subscribe to our newsletter!
Our Book