Using Kosli attest in Github Actions Workflows - Some Do's and Don'ts

The heart of Kosli’s functionality lies in its attest command. Think of it as a digital notary for your CI process. Every time you complete a significant step in your pipeline (e.g., a security scan, a build, a deployment, etc) you use kosli attest to create an immutable record of that event.

However, integrating Kosli into your existing CI workflow isn’t always straightforward. You might find yourself grappling with questions like:

• How do I ensure Kosli records evidence even if a step fails?
• What’s the best way to structure my CI configuration to include Kosli attestations?
• How can I make Kosli attestations work seamlessly with my existing tools and processes?

In this article, we’ll explore these questions and more. We’ll start with a basic example of using Kosli with Snyk for image scanning, then dive into common pitfalls and best practices. Whether you’re just starting with Kosli or looking to optimize your existing setup, you’ll find practical, actionable advice to make your CI pipeline more robust and transparent.

Let’s begin by looking at a simple, albeit flawed, approach to integrating Kosli with Snyk:

A straw man example to start with

  snyk-container-scan:
    runs-on: ubuntu-latest
    ...
    steps:
      ...
      - name: Setup Snyk
        uses: snyk/actions/setup@master

      - name: Setup Kosli CLI
        uses: kosli-dev/setup-cli-action@v2
        with:
          version:
            ${{ vars.KOSLI_CLI_VERSION }}

      - name: Run Snyk, attest the evidence to Kosli
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        run: |
          snyk container test \
              ...
          kosli attest snyk \
              ...

There are several problems with this simplistic approach. 

• You have to remember to set up the Kosli CLI before the action. 
• You need to add | to the run: and also those pesky \ line-continuation characters for any command spanning more than one line. 
• If the snyk container test call finds a vulnerability and returns a non-zero $? exit code then the call to kosli attest snyk will simply not happen.

Don’t bracket the call with set +e/-e

One pattern we’ve seen is to bracket the call with set +e/-e bash commands, save the $? exit-code, call kosli attest, and then exit with the saved exit-code. Here’s what that looks like:

      - name: Run Snyk, attest the evidence to Kosli
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        run: |
          set +e
          snyk container test \
              ...
          STATUS=$?
          set -e
          kosli attest snyk \
              ...
          exit ${STATUS}

A variation uses the bash || operator.

      - name: Run Snyk, attest the evidence to Kosli
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        run: |
          STATUS=0
          snyk container test \
              ... || STATUS=$?
          kosli attest snyk \
              ...
          exit ${STATUS}

You can get this approach to work, but it’s not the most elegant or maintainable solution. It clutters your CI configuration and can make it harder for team members to understand the flow of your pipeline. We recommend a different approach which puts the kosli attest into its own step.

Don’t use continue-on-error: true

Another pattern we’ve seen to ensure the kosli attest always runs, is to put it into its own step, and add a continue-on-error: to the step performing the action. Here’s what it looks like:

  snyk-container-scan:
    runs-on: ubuntu-latest
    ...
    steps:
      ...
      - name: Setup Snyk
        uses: snyk/actions/setup@master
        
      - name: Run Snyk
        continue-on-error: true
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        run:
          snyk container test 
              ...

      - name: Setup Kosli CLI
        uses: kosli-dev/setup-cli-action@v2
        with:
          version:
            ${{ vars.KOSLI_CLI_VERSION }}

      - name: Attest the evidence to Kosli
        run:
          kosli attest snyk 
              ...

We don’t recommend this approach because if the action (in this case a snyk scan) fails, the job will nevertheless pass. This means you can’t rerun the failing CI jobs - you have to rerun the whole CI workflow.

So while the continue-on-error approach might seem tempting in its simplicity, it introduces new problems while solving others. It’s a classic case of a solution that treats the symptom rather than the underlying issue.

But don’t worry - there’s a better way. In the next section, we’ll explore a more robust and flexible approach that allows you to run Kosli attestations reliably without compromising the integrity of your CI pipeline. This solution strikes a balance between ensuring Kosli always gets its data and maintaining clear, actionable feedback about the success or failure of your CI jobs.

Do use if: success() || failure()

This is the best way we’ve found to ensure the kosli attest command always runs, and the workflow job still fails when it should. Use it on the kosli attest step and also on the step to install the Kosli CLI. With this pattern you usually don’t have to alter your existing step at all, you simply need to add two single-command steps after it. Note: we don’t recommend using if: always() since that can prevent jobs from being cancelled.

  snyk-container-scan:
    runs-on: ubuntu-latest
    ...
    steps:
      ...
      - name: Setup Snyk
        ...
        
      - name: Run Snyk
        ...
        
      - name: Setup Kosli CLI
        if: success() || failure()
        uses: kosli-dev/setup-cli-action@v2
        with:
          version:
            ${{ vars.KOSLI_CLI_VERSION }}

      - name: Attest the evidence to Kosli
        if: success() || failure()
        run:
          kosli attest snyk
            ...    

Do extend the if: as appropriate

In practice, customers CI workflows often have multiple on: conditions, and they’d prefer to only kosli attest for some of them. For example, when the CI workflow is running on the main/master branch, but not on a PR branch. This can be handled by extending the if: as follows:

  snyk-container-scan:
    runs-on: ubuntu-latest
    ...
    steps:
      ...
      - name: Setup Snyk
        ...

      - name: Run Snyk
        ...

      - name: Setup Kosli CLI
        if: ${{ github.ref == 'refs/heads/main' && (success() || failure()) }}
        uses: kosli-dev/setup-cli-action@v2
        with:
          version:
            ${{ vars.KOSLI_CLI_VERSION }}

      - name: Attest the evidence to Kosli
        if: ${{ github.ref == 'refs/heads/main' && (success() || failure()) }}
        run:
          kosli attest snyk
              ...

Do set the KOSLI_DRY_RUN Environment Variable

When the KOSLI_DRY_RUN environment variable is set to false, the kosli attest command sends its payload to Kosli, but when set to true, it instead prints out the payload it would have sent to Kosli.

For example:

############### THIS IS A DRY-RUN  ###############
this is the payload that would be sent in real run:
Content-Disposition: form-data; name="data_json"
{
    "artifact_fingerprint": "462c87efeb60423bc9a36ffa44c561d4749c3a859343...",
    "git_commit_info": {
        ...
    },
    "attestation_name": "snyk-scan",
    "origin_url": "https://github.com/kosli-dev/server/actions/...",
    "user_data": {},
    "snyk_results": {
        "schema_version": 1,
        "tool": {
            "name": "Snyk Container",
            "version": "1.1293.1"
        },
        "results": [
            {
                "high_count": 0,
                "medium_count": 0,
                "low_count": 0
            },
            {
                "high_count": 0,
                "medium_count": 0,
                "low_count": 0
            }
        ]
    }
}

We recommend creating as a GitHub Org-level Action-variable called KOSLI_DRY_RUN set to false and using it to set a global Workflow environment variable:

env:
  KOSLI_DRY_RUN: ${{ vars.KOSLI_DRY_RUN }}           # false
  ...

You can then dry-run all the kosli attest commands…

• in one Workflow by setting KOSLI_DRY_RUN to true in its yml file.
• in all Workflows in an GitHub Org by setting the Action-variable to true

Do use a step id: in a generic attestation

When you’re using the kosli attest snyk command, the compliance value is calculated for you, from the results of the snyk test. However, there is one kosli attest command that does not calculate the compliance value for you - kosli attest generic. In this case you provide the compliance value (of true or false) to the command. We recommend giving the step performing the action its own id: and then setting the KOSLI_COMPLIANT environment variable from its outcome. For example:

  lint:
    runs-on: ubuntu-latest
    steps:
      ...
      - name: Run lint on source
        id: lint
        run:
          make lint

      - name: Setup Kosli CLI
        ...

      - name: Attest evidence to Kosli Trail
        if: ${{ github.ref == 'refs/heads/main' && (success() || failure()) }}
        run: |
          KOSLI_COMPLIANT=$([ "${{ steps.lint.outcome }}" == 'success' ] && echo true || echo false)
          kosli attest generic \
            --name=runner.lint          

Summary

Using these patterns has the following advantages:

• Jobs that should fail do fail
• Existing steps in a job remain unchanged - no tracking down missing | or \
• Choose what conditions (eg branches) to run the kosli attest commands
• A simple one-command kosli attest step
• The kosli attest runs, even if the action they are attesting fails
• Run kosli attest commands in dry-run mode when necessary

You can browse a live GitHub Actions Workflow using this pattern in this open-source git repository