How to record an audit trail for any DevOps process with Kosli Trails

In this article I’m going to introduce Kosli Trails. This is a new feature that allows you to record an audit trail for any DevOps process. It’s already in production and being used to record Terraform pipelines, CI processes, server access, feature toggles, and more. 

How it all started - change management for DevOps 

Like most software startups, in Kosli’s early stage, we focused on solving a narrow problem: we wanted to solve change management for regulated software teams by recording the facts in DevOps pipelines. 

We thought that if we could find a way to record the build provenance, testing, security scans and approvals automatically in the pipeline in a secure, offsite, tamper evident way, then we believed that we could help regulated software teams break free of slow change management processes and painful audits. 

And it worked! Kosli customers typically see 10-100x increases in deployment frequency for standard changes, and can pass audits without any manual evidence gathering processes. A simple download to CSV from Kosli and they’re done. 

But, over time, we realized that the only way to really prove only compliant changes are running requires recording the runtime environments. So, we built an environment recorder that works with all kinds of runtimes: from on-premise servers, through to containerized workloads and serverless functions. And because we’re a bunch of version control geeks, we designed it to look and feel like version control, with log, diff and view commands.  

It is a contrary bet to change management. Rather than record a change and hope that it’s reality, we record a snapshot of the actual runtime and tie workloads back to verified sources via provable cryptographic fingerprints. 

What this means is that when it comes time to investigate your systems (for audit, incident response, or developer feedback) you can base your understanding on facts rather than tickets.

So, for instance, if an auditor asks “show me every change to the authentication system over the last 12 months” not only do you have the records, but also the immutable evidence that every change met your standards.

There’s more to change than unit tests and pull requests

Over time, our customers got even more ambitious. They started to wonder if they could use Kosli to automate record keeping for other critical technical changes. We started seeing our customers create “fake” software pipelines to record Terraform pipelines, server access logs, account provisioning, and feature flag changes. 

Being customer obsessed, we jumped on the opportunity to support these use cases without having to pretend they were software pipelines. The outcome was Kosli Audit Trails, a feature to natively record attestations for any process, via the command line or API.  

Plan to throw one away

“In most projects, the first system built is barely usable….Hence plan to throw one away; you will, anyhow.”

Fred Brooks, The Mythical Man-Month

An interesting thing happens when you put beta features in the hands of customers: you get real feedback. And when we released the early version of this feature to our customers, we received two clear signals:

1. There is a lot of commonality between Flows and Audit Trails, and
2. There are really valid use cases for combining these ideas.

So that gave us pause to reflect: should we continue down this path, or rather go back to the whiteboard to see if there is a cohesive feature waiting to emerge? It turns out that there was…

Trails for software pipelines

After a lot of discussion, whiteboarding, customer feedback, and good old fashioned hard work, we found a way forward.

The high level view introduces a concept called a “Trail” where you can define your expectations for a workflow in a yaml file regardless of whether it is for CI pipelines, user journeys, IaC pull requests, or any other workflow you’d like to record.

Here’s an example of a simple trail that records a CI pipeline trail, which builds two artifacts and has some basic attestations:

And here’s how that looks in Kosli:

What’s great about this is that now it is possible to record multiple artifacts in a single workflow.  And because you can now base this on any identifier, you can identify the workflow by jira issue, artifact sha256, commit SHA1, or any other identifier you choose.  

Right now, Trails are currently recording:

• CI pipeline run [example]
• Terraform pull request workflows [example]
• Server access logs
• Nightly cron jobs [example]
• Employee offboarding

The technical stuff… 👇

So how does this change what’s happening under the hood? What we’ve done is insert a “Trail” entity in the domain model that can record a trail of attestations.  

Here is the glossary:

• Flow: represents a business process or value stream in the user’s domain. The process may or may not include Software Development Lifecycle (SDLC) steps. The boundaries of the process are defined by the user. 
• Trail: is a single execution of a process. A trail consists of a set of named steps which are executed in a process. Each trail has a user-defined trail-id which is unique within the Flow. This user-defined ID has to follow a user-defined schema that works for the nature of their process. Examples include: artifact sha256, git commit sha1, merge request identifiers, jira tickets, etc.
• Slot: A step is a named “slot” or “key” in a trail for evidence. Attestations are reported against those named steps. Trails inherit the Flow steps by default, but the steps can be overwritten for any individual Trail.
• Attestation: is a typed JSON payload that a user reports to Kosli. Attestations can be connected with a Trail or to a specific Artifact.
• Artifact: are software packages identified by their SHA256 fingerprints. Artifacts are reported/created as part of a Trail. These are a type of attestation
• Attachment: Optional files to store with an attestation

Try it and let us know what you think 🤗

Trails are available now for all plans (including our free tier). We’d love for you to try and to tell us what you think. You can ask questions and provide feedback in our community Slack channel.