How to record events in your CI pipelines with Kosli Flows

In an ideal world CI pipelines would never fail and deployments would be easy to navigate. The reality is that the journey from commit to production can fail in subtle ways that can be hard to understand. And this problem is multiplied by the number of pipelines in your system.

Simple questions like “which of our 32 pipelines last deployed?”, “which pipelines don’t have Snyk scanning?” and “what should be running in production?” shouldn’t be hard, but getting to the answers usually requires a lot of digging across tools. 

Kosli Flows makes all of this much easier to understand. Flows provide a way to record, connect, and search all of the dynamic events in your pipelines. It makes accessing all the processes in your CI pipelines as easy as a git log.

This article will introduce you to three key concepts in Kosli: Flows, Flow Attestations, and Evidence Vault.

In the diagram below you can visualize all three. A Flow is where pipeline events like tests, security scans and deployments happen. These events are recorded using Flow Attestations, and these attestations are secured in Kosli’s tamper-evident Evidence Vault.

What does it mean to have a Flow in Kosli?

A CI pipeline represents a value stream for one or many artifacts, and it can vary in many ways:

• How it is triggered: git events, cron schedule, other webhooks
• What it builds: binaries, docker images, sources, infrastructure
• What qualification steps it performs: build, test, deployment, security scans, etc
• Deployment strategies: none, continuous, releases, etc

To keep track of what’s happening in production, we need to start with the end in mind. For any given artifact found in an environment, we want to know its entire lifecycle. In Kosli, this journey is stored in a Flow.

A Flow is a place to record data about different versions of an artifact over time.  

The Flow Name provides the common identity of multiple versions of an artifact. By that I mean, if you build a particular artifact (i.e.mycompany/notification-service:2d7ecd or libtraintracks-1.3.8364.so) your flow name could be notification-service or libtraintracks.  You can think of it as a sticky name for versions of an artifact that goes through a pipeline.

What is a Flow Attestation?

Every time your pipeline builds a new artifact you can record a specific Flow Attestation about this fact. It includes:

• Git commit info
• Artifact sha
• Build url
• Any other metadata you wish to store

With this you have a record of every built artifact that you can look up later. This concept has a name: Binary Provenance.

In a Flow you can also record other Software Development Lifecycle (SDLC) and pipeline events such as pull requests, test results, and security scans as Flow Attestations. You can create approvals, record deployment expectations, and view the whole timeline for any artifact.

And, if you record your environments with Kosli, we can also detect where and when it has been running! This gives you a provable chain of custody across your whole delivery process.

How does Flow Attestation work?

In Kosli, our data is a trusted source of documentation during compliance and security audits for your software delivery process. To reach these standards our records must be provable and tamper evident. And we need to store facts connected to things in the outside world, which we can’t control.

How can we achieve this? We use a technique called content addressable storage. This is a fancy way to say that the identity we connect our attestations and evidence to is a cryptographic fingerprint of a bunch of bytes - in this case the SHA1 git commit or SHA256 digest of the artifact. So, if you find any artifact anywhere in your systems, you can calculate its digest and find out its full history.

You can make Flow Attestations in two ways:

1. Against a git commit:

• Sometimes a process step (i.e. a pull request, or linting) occurs before an artifact is built
• Sometimes a commit can produce multiple artifacts
• This can apply to all Flows, or narrowed to a particular Flow

2. Against an artifact sha:

• For example security scanning, pending deployments, test results, etc.

When you want to record a Flow Attestation - a Snyk scan on an artifact for example - you can do it in your pipeline like this:

Evidence Vault: attaching evidence to an attestation

Sometimes you need to store more than just a record of what has happened. For example, if you have software audits, you need to verify a lot of information - pull requests, test reports, security scans, deployment logs, etc. This information is typically stored across your tools in unstructured ways which are often unsecured and unmanaged. It’s hard to collect and piece together all of the  evidence and that can cause long and painful audits.

As an example, let’s take a look at some typical audit conversations:

• Auditor: for this given change/deployment, can you show me the code review, security scan, deployment approval and test report?
• • Bad answer: sure, let me look up the commit for this - and see what pipeline run might have been triggered. Unfortunately this change is beyond our pipeline retention period, so we don’t have the log. I can probably find the security scan in our SAST tool, and the pull request I can search for.  Will that do? <repeat 50 times>
  • Worse answer: sure, we have the Change Advisory Board meeting record, where Bob signed off that we did things correctly.
  • Great answer: here’s the attestations which record that these steps have been completed and controlled for, and here’s the corroborating evidence as cryptographically verifiable artifacts. 

That’s why Kosli has the Evidence Vault. To attach any evidence to a Flow Attestation you can simply supply file paths as part of the command:

This allows you not only to say that your pipeline performed unit testing, but to actually upload the test result files. This frees you from having to find a secure place to store these files, and makes it trivial to download an Evidence Package from the Evidence Vault for any artifact.  

Flow Attestations and Evidence are append-only and immutable

An important aspect of Kosli is not only what we store, but how we store it. Every Flow Attestation and evidence is append-only - meaning you can’t change or delete any information. So, even if you upload the same evidence multiple times an additional entry is always recorded - all previous Flow Attestations remain unchanged.

This is by design. Kosli was created to provide a fully connected database of tamper-evident records that are easy to search, and easy to export, and make audits straightforward and painless.

How to use the data in Flows?

So now you know about Flows, Flow Attestations, and the Evidence Vault. How can you use them?

All this information is available via the web, but also via the command line. If you want to know if a commit is in production, or what its compliance status is, you can simply make a query to ask Kosli what it knows about the commit sha:

And now you have all the data at hand, you can create powerful automation in your DevOps such as:

• Circuit breakers in deployment pipelines
• Track lead times and deployment frequencies
• Request approvals from within pipelines

And this information can also be output as json, and accessed via our api.

Connecting the runtime environments

Remember earlier when we talked about Content Addressable Storage? Well, we use the same concept for storing the state of your environment. Every environment snapshot we record contains the SHA256 digests of the running artifacts, allowing you to track software through its complete lifecycle.

Integrations coming soon - Slack, Backstage, and more

We have more exciting features heading your way. You can already see all your DevOps history in Kosli through the browser and at the command line with Kosli’s free tier.

But soon all of this information will be available in Slack, Backstage, and many other locations. See below for a sneak peak of what that will look like.

If you want to register your interest as a potential beta tester, please let us know in our Slack community.