How we implemented a release/promotion workflow with a single approval, using Kosli

Overview

A feature we often get asked about at Kosli is whether we can help support a release/promotion workflow: a workflow that deploys a known set of Artifacts from one runtime environment (eg beta/staging) into another runtime environment (eg production), typically in parallel. The simple answer is we can help, and in this blog we show the release workflow in the Kosli cyber-dojo demo project (an open sourced application for practising TDD from your browser).

Attest evidence in CI pipelines

cyber-dojo has 10 microservices, each housed in its own git repository (in github and gitlab). Each repository has a main CI pipeline that deploys into cyber-dojo’s beta environment. These deployments are continuous, and automated; there are no manual steps and no manual approvals:

• We create a Kosli Flow to represent each microservice. For example
• We use the Kosli CLI to attest the required SDLC evidence, in each CI pipeline. For example
• We use the Kosli CLI as an SDLC Gate, in each pipeline, to control entry to the beta Environment, based on meeting its requirements. For example

For example, here is the saver-ci Kosli Flow, showing the evidence gathered for the cyberdojo/saver repository’s workflow run for commit 737a681776ad1d2bfbf601f32f757d845f738528:

Automatically record runtime environments

The same Kosli CLI that makes attestations in the CI pipelines can also act as a “black box” flight-recorder, giving you a real-time, immutable log of all changes to your runtime environments. This is set up in the kosli-environment-reporter repository and records:

• cyber-dojo’s aws-beta cluster, where we are deploying from
• cyber-dojo’s aws-prod cluster, where we are deploying to (screenshot below, showing the 10 running services)

Kosli Environment recording connects the evidence, attested from the CI pipelines, for all running Artifacts, based on the Artifact digests. If any Artifact does not have binary provenance, the Kosli Environment will be non-compliant. For Artifacts that have binary provenance (they all do), Kosli knows the repository and commit it was built from (eg 737a681 for saver), and whether it meets the requirements of its individual SDLC. We will rely on this shortly to create a “composite” SDLC Control Gate for the release workflow.

Find the release Artifacts

With the attestations from the pipelines and the runtime environment recording we have all the information we need, and once again, we can use the Kosli CLI (or API) to query this information. In particular, we can use it to give us the diff between the Artifacts running in cyber-dojo’s two Environments. For example, here is the command to find the diff between what is currently running in cyber-dojo’s aws-beta and aws-prod runtime environments:

$ kosli diff snapshots aws-beta aws-prod ... --output=json

The JSON this command returns for each Artifact looks like this:

{
        "name": "244531986313.dkr.ecr.eu-central-1.amazonaws.com/saver:6e191a0",
        "fingerprint": "b3237b0e615e7041c23433faeee0bacd6ec893e89ae8899536433e4d27a5b6ef",
        "flow": "saver-ci",
        "commit_url": "https://github.com/cyber-dojo/saver/commit/6e191a0a86cf3d264955c4910bc3b9df518c4bcd",
        ...
}

Where:

• “name” is the name of the docker image
• “fingerprint” is the sha256 digest of the docker image
• “flow” is the name of the Kosli Flow with an artifact-attestation, for a docker image with a matching fingerprint
• “commit_url” tells us which commit triggered the pipeline run, which performed this artifact-attestation

Importantly, the diff JSON for each Artifact is split across three categories, and means we know if the Artifact is running:

1. in aws-beta but not in aws-prod
2. in both aws-beta and aws-prod
3. in aws-prod but not in aws-beta

For example, in the graphic below:

• newer versions of saver and differ are in 1)
• older versions of saver and differ are in 3)
• a version of creator, nginx, web, and dashboard is in 2).

Check for current deployments still in progress

If there is a deployment, to either aws-beta or aws-prod, currently in progress we will abandon the release. To detect this we:

• Combine the JSON information in categories 1 and 2 to tell us everything running in aws-beta.
• If there is more than one Artifact with the same Flow name in this combined list, a deployment is still in progress, for that Flow, in aws-beta.
• Combine the JSON information in categories 2 and 3 to tell us everything running in aws-prod.
• If there is more than one Artifact with the same Flow name in this combined list, a deployment is still in progress, for that Flow, in aws-prod.

For example, in the graphic below, there are two versions of differ, one in 1) and one in 2)

Combine the incoming/outgoing JSON information

We now augment the JSON for the Artifacts running in aws-beta but not in aws-prod (category 1), with information on which specific Artifact they will be replacing in aws-prod (category 3), based on a matching Flow name. For example, suppose the JSON for the Artifact from the saver-ci Flow in aws-beta is as above, and the JSON for the Artifact from the saver-ci Flow in aws-prod is as follows:

{
        "name": "244531986313.dkr.ecr.eu-central-1.amazonaws.com/saver:65827d7",
        "fingerprint": "cd8e30cf2027bce856d2b65c827afcbc0370f6e1a56c80514d4af2719df5c12f",
        "flow": "saver-ci",
        "commit_url": "https://github.com/cyber-dojo/saver/commit/65827d7aa09d9d40eca4210dfeef4d162cf32c1f",
        ...
}

We prefix the JSON from the saver-ci Artifact in aws-beta (only) with “incoming”, and the JSON from the saver-ci Artifact in aws-prod (only) with “outgoing”. We’ll also extract the repo-names, the commit-shas, and form a deployment-diff-url. The resulting JSON for each Artifact looks like this:

{
        "incoming_name": "244531986313.dkr.ecr.eu-central-1.amazonaws.com/saver:6e191a0",
        "incoming_fingerprint": "b3237b0e615e7041c23433faeee0bacd6ec893e89ae8899536433e4d27a5b6ef",
        "incoming_flow": "saver-ci",
        "incoming_commit_url": "https://github.com/cyber-dojo/saver/commit/6e191a0a86cf3d264955c4910bc3b9df518c4bcd",
        "incoming_repo_name": "saver",
        "incoming_commit_sha": "6e191a0a86cf3d264955c4910bc3b9df518c4bcd",
        ...
        "outgoing_name": "244531986313.dkr.ecr.eu-central-1.amazonaws.com/saver:65827d7",
        "outgoing_fingerprint": "cd8e30cf2027bce856d2b65c827afcbc0370f6e1a56c80514d4af2719df5c12f",
        "outgoing_flow": "saver-ci",
        "outgoing_commit_url": "https://github.com/cyber-dojo/saver/commit/65827d7aa09d9d40eca4210dfeef4d162cf32c1f",
        "outgoing_repo_name": "saver",
        "outgoing_commit_sha": "65827d7aa09d9d40eca4210dfeef4d162cf32c1f",
        ...
        "deployment_diff_url": "https://github.com/saver/compare/65827d7aa09d9d40eca4210dfeef4d162cf32c1f...6e191a0a86cf3d264955c4910bc3b9df518c4bcd"
}

Enable parallel workflow jobs for each release Artifact

We now have a JSON list, with each entry in the list containing the information for one Artifact to release/promote.

[
  {
     "incoming_name": "244531986313.dkr.ecr.eu-central-1.amazonaws.com/saver:6e191a0",
     "incoming_fingerprint": "b3237b0e615e7041c23433faeee0bacd6ec893e89ae8899536433e4d27a5b6ef",
     ...
  },
  {
     "incoming_name": "244531986313.dkr.ecr.eu-central-1.amazonaws.com/nginx:fa32058",
     "incoming_fingerprint": "0fd1eae4a2ab75d4d08106f86af3945a9e95b60693a4b9e4e44b59cc5887fdd1",
  }
]

This is exactly the format we need in Github Actions to run parallel jobs, one for each Artifact, using strategy:matrix:include yaml.

We start with a job whose sole purpose is to output this JSON information ready to be consumed by strategy:matrix:include:

env:
  KOSLI_API_TOKEN: ${{ secrets.KOSLI_API_TOKEN }}
  KOSLI_ORG:       ${{ vars.KOSLI_ORG }}               # cyber-dojo
  ...

jobs:
  ...
  find-promotions:
    needs: ...
    runs-on: ubuntu-latest
    outputs:
      promotions: "${{ steps.vars.outputs.promotions }}"
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 1

      - name: Setup Kosli CLI
        uses: kosli-dev/setup-cli-action@v2
        with:
          version: ${{ vars.KOSLI_CLI_VERSION }}

      - name: Generate JSON for each Artifact promotion for use in the following job's strategy:matrix:include
        id: vars
        run:
          echo "promotions=$(make promotions | jq --raw-output --compact-output .)" >> ${GITHUB_OUTPUT}

Any subsequent job can use this job’s output to run on all promotion Artifacts in parallel. In the steps we access the JSON information for the individual Artifact using the syntax ${{ matrix.KEY }}. For example:

jobs:
  ...
  some-job:
    if: ${{ needs.find-promotions.outputs.promotions != '[]' }}
    needs: [find-promotions]
    runs-on: ubuntu-latest
    strategy:
      matrix:
        include: ${{ fromJSON(needs.find-promotions.outputs.promotions) }}
    steps:
      - name: Example
        run:
          echo ${{ matrix.incoming_fingerprint }}

Add the SDLC control gate

With all the required information in the JSON matrix include, this step is straight-forward. There is a Kosli CLI command to query whether an Artifact with a given fingerprint, meets the compliance Policies currently attached to a given Environment. For example, does it have binary provenance, a pull-request, a security-scan, etc. We use the command twice; once to check the Artifact is compliant in the Environment it is currently running in (aws-beta), and again to check the Artifact is compliant for the Environment where it will be deployed to (aws-prod).

jobs:
  ...
  sdlc-control-gate:
    if: ${{ needs.find-promotions.outputs.promotions != '[]' }}
    needs: [find-promotions]
    runs-on: ubuntu-latest
    strategy:
      matrix:
        include: ${{ fromJSON(needs.find-promotions.outputs.promotions) }}
    steps:
      - name: Setup Kosli CLI
        uses: kosli-dev/setup-cli-action@v2
        with:
          version: ${{ vars.KOSLI_CLI_VERSION }}

      - uses: actions/checkout@v4
        with:
          fetch-depth: 1

      - name: Assert Artifact is compliant in aws-beta
        run:
          kosli assert artifact
            --fingerprint="${{ matrix.incoming_fingerprint }}"
            --environment="${KOSLI_AWS_BETA}"

      - name: Assert Artifact is compliant for aws-prod
        run:
          kosli assert artifact
            --fingerprint="${{ matrix.incoming_fingerprint }}"
            --environment="${KOSLI_AWS_PROD}"

      - name: Attest promotion to Kosli
        run: |
          echo "${PROMOTION}" > /tmp/promotion.json
          kosli attest custom --type=promotion \
            --annotate=deployment_diff_URL="${{ matrix.deployment_diff_url }}" \
            --attestation-data=/tmp/promotion.json \
            --fingerprint="${{ matrix.incoming_fingerprint }}" \
            --name="${{ matrix.incoming_repo_name }}.promotion"          

View the proposed release deployments

In the final step of the job above, we attest the JSON information for each Artifact into a Kosli Flow dedicated to recording the release evidence.

Having all the release evidence together in a single dedicated Kosli Flow makes it easy for the human-in-the-loop to make a final, go/no-go approval decision. For example, below we can see the evidence for the proposed parallel release of five Artifacts. In the one shown expanded (web) you can see the URL for web’s deployment-diff - the commit-level diff between the web Artifact running in aws-beta and the web Artifact running in aws-prod.

Approve the release deployments?

We make the final approval in the Github workflow, it looks like this

jobs:
  ...
  approve:
    if: ${{ needs.find-promotions.outputs.promotions != '[]' }}
    needs: [find-promotions, sdlc-control-gate]
    runs-on: ubuntu-latest
    environment:
      name: production
      url:  https://cyber-dojo.org
    ...

If approved, deploy to production

If the approval is made, we can do the actual deployments. Once again in Github Actions we can use the matrix:include to do all the deployments in parallel.
Each cyber-dojo repo has its own deployment/ directory which contains its terraform files. The actual deployment lives in its own reusable workflow, and needs, as its input, the git repo and commit-sha the Artifact was built from, so it can checkout this deployment/ directory and then run its terraform commands.

jobs:
  ...
  deploy-to-prod:
    if: ${{ needs.find-promotions.outputs.promotions != '[]' }}
    needs: [setup, find-promotions, approve]
    strategy:
      fail-fast: false
      matrix:
        include: ${{ fromJSON(needs.find-promotions.outputs.promotions) }}
    permissions:
      id-token: write
      contents: write
    uses: fivexl/gh-workflow-tf-plan-apply/.github/workflows/base.yml@v0.0.23
    with:
      checkout_repository:  cyber-dojo/${{ matrix.incoming_repo_name }}
      checkout_commit_sha:  ${{ matrix.incoming_commit_sha }}
      checkout_fetch_depth: 0
      ...

Summary

A release/promotion workflow deploys a known set of Artifacts from one runtime environment (eg beta/staging) into another runtime environment (eg production), typically in parallel. With Kosli you can solve the hard parts:

• You know the exact identity of the incoming and outgoing Artifacts.
• You know whether there is a deployment currently in progress.
• You can add an SDLC Control gate which passes only if all Artifacts have passed their SDLC requirements.
• You can attest the proposed release into its own Kosli Flow for easy inspection of each Artifact’s deployment-diff.

The last two steps are simpler:

• In the CI workflow, approve (or not) the release deployments.
• If approved, deploy to production.