4 min read
If you deliver software in a regulated industry you have to be able to show that you are following a defined process. And that means being able to produce a record of what’s going on in your DevOps workflows.
When we have conversations about DevOps compliance with regulated software teams this topic frequently comes up. And what these teams require is best described by Carl Nygard as a Compliance System of Record (CSoR).
But, how should you design such a record? What properties should it have? What is the best way of collecting and storing the necessary information?
Whether you build your own, recruit an existing internal system, or choose a DevOps Change Management system like Kosli, we believe there are some important implementation choices you can make to ensure success.
In this article we’ll set out the 5 essential properties needed for a DevOps Compliance System of Record.
1. Content addressable identities
All decisions should be based on secure and provable identities defined by the contents of those systems. This is the foundation of Artifact Binary Provenance and there’s lots of ways to do it. In Kosli, our approach is to use the SHA256 sum of any binaries that we care to track.
This provides the assurance that what we qualify is what we deploy, and we can easily track software regardless of labels and metadata.
2. Secure, Append-only/Immutable data
It should not be possible to modify any data. From a compliance perspective we have to be able to trust that the record of decisions, and the data they were made on, haven’t been changed.
That isn’t to say we shouldn’t be able to update information. But, if we do, it should be done in a fully traceable and non-destructive manner. In other words, the data should be append-only.
This is actually quite hard to implement in most databases due to their fundamental design around CRUD transactions. There are ways to do this at the application level, but this requires additional guarantees that data cannot be updated “outside” of the application.
One way we have solved this problem is by using git as the data repository with no-force-push remotes to ensure no rewrites/rebasing. Alternatively, Kosli offers no write “access” to the backing database for customers. Either way, using append-only data structures and merkle-trees is a big step towards ensuring immutability.
3. Pipeline-friendly
A CSoR is essential for audit purposes, but we should never forget the most frequent user: the DevOps pipeline.
Now, what makes a tool easy to use from a pipeline? 🤔
- Small and efficient - no one wants a slow pipeline
- Trivial to store and query data
- Context aware - it should be easy to control via CI environment variables
- Circuit breaker - it should be easy to move into “pass-through” mode in emergencies
In Kosli, we have gone through several iterations of client tooling. The very first was based on curl commands directly to our APIs. We then created a python client that was a big improvement in terms of usability. Our latest client is written in Golang which makes it very fast to use in pipelines with distributions for all the major platforms and architectures.
4. Service-friendly
An easily overlooked use case for DevOps compliance is self-notification events. It should be simple for an application to make attestations or queries on its own state.
Why would you need this?
Well, many compliance and security events occur outside of a DevOps pipeline. Monitoring systems might want to notify deployments, platform provisioning tools might need to create new assets to track, and sensitive services might want to record audit logs.
The easiest way to implement this is to expose a REST-style API to your Compliance System of Record.
5. Easy to navigate
The final consideration is also the most important - humans. How do we make it easy to reason about high levels of change in modern distributed architectures?
- What’s in production?
- What are the recent changes to staging?
- Who approved this deployment?
- When was this artifact in production?
Designing a user interface which meets the needs of developers, compliance officers, security personnel, and auditors isn’t easy, but it is extremely valuable.
Whether you build your own solution, or adopt a tool like Kosli, these 5 elements will ensure a CSoR that will satisfy the demands of any regulator or regulatory standard.
