How to record a business process with Kosli’s Audit Trail

Have you ever needed to provide proof that a critical business process actually took place? It’s a painful process involving all kinds of paperwork, but it’s the reality for many organizations working in highly regulated industries. 

For these companies, records need to be kept for actions like the provisioning of user accounts and access to sensitive records. It’s necessary, but it’s manual and time-consuming work. In response to requests from our customers to develop automation for these tasks in Kosli, we are pleased to announce our Audit Trail feature. 

How to prove essential business processes are being followed

Technology orgs often have tools and automated workflows that perform extremely sensitive actions. These workflows can be security sensitive (such as escalating user privileges) or put business operations at risk (such as performing a database migration).

Despite the critical nature of these processes, for most companies there is no natural place to record them. This creates problems for incident response, compliance and audit, and plain old “whats-going-on?” questions in mission critical systems. 

This is why we’ve had many requests from our customers to record non-DevOps business processes in Kosli. Some of the uses cases are very interesting, such as:

• Provisioning and revoking user accounts and access
• Recording evidence of fraud detection processes (for purposes of financial crime investigation)
• Recording facts regarding scheduled IT risk management such as disaster recovery
• Recording feature toggle state changes
• Compliance case worker access to sensitive customer information
• Transaction records for cold storage transfer of cryptocurrencies

Rather than push the square peg of business workflows into our round hole of DevOps flows, we decided to support these use cases with a solution specifically designed to support the types of use cases mentioned above. 

We’ve made it easy to integrate from scripts with our CLI, or with your applications using our APIs.  Let’s take a look at how it works using a concrete example.

A Concrete Example - Provable Production Access Logs

This may sound a bit abstract, so let me show you a concrete example of how we use the Audit Trail capabilities in real life: to keep persistent and provable records of production access logs.

How it works is pretty simple - whenever a member of our team has shell access to a runtime environment, we log the session and send the logs and the user’s identity to a Kosli Audit Trail automatically.

How does this work?

1. For debugging or investigations, authorized developers can gain shell access to a container via SSO.  We have used cloud automation to capture three key files for every session and write them to S3:
2. 1. The user-identity details via Cloudtrail events
   2. The service-identity details via Cloudtrail events
   3. The shell session logs (the actual stdin/stdout captured by SSM)
3. When the user-identity file in 1(a) is written, we have cloud automation that attests this into a Kosli workflow (created from our public terraform module). The file is attached as evidence.
4. When the service-identity file in 1(b) is written, the same cloud automation attests this into the Kosli workflow. The file is attached as evidence.
5. When the user session finishes, the session logs write the file in 1(c). The same automation in the previous two steps attests this into the Kosli workflow. The file is attached as evidence.
6. Now in Kosli we have permanent records for every shell session.  

This provides essential visibility and transparency across the team for any manual investigative actions undertaken on our systems, and who did them. We can extend this basic workflow to provide additional attestations and evidence to record reviews and approvals, or subscribe to notifications for new Audit Trails.  

In the next section we’ll go into more detail. We’ll be showing the production access Audit Trail for https://cyber-dojo.org  The URL links provided are public (no login or credit-card required) so you can easily follow along.

How do Audit Trails work?

Four key concepts form the basis of Kosli’s Audit Trail:

• Audit trail: This is the identity for the critical business process. In our example for logging sessions to the cyber-dojo environments, there are two Audit Trails, one called `beta-server-access` for logging access to cyber-dojo’s AWS beta environment, and one called `prod-server-access` for logging access to cyber-dojo’s AWS prod environment

• Workflows: Individual instances of the process to be recorded. In our example, there are (currently) six workflows for cyber-dojo’s prod-server-access Audit Trail. Each workflow is given an identity, for example `ecs-execute-command-0199bb27be8f354b7`

Attestations: Immutable records for process events. In our example, we store three attestations `user-identity`, `service-identity` and `command-logs`

• Evidence: Files to be recorded as optional immutable attachments to an attestation. The evidence vault works with the Audit Trail feature so you can upload any supporting files that you need to store for safekeeping. 
• • Click the blue [Download evidence package] button
  • Unzip the downloaded archive
  • In the archive dir you will see three sub-dirs, one for each attestation
  • The log file in user-identity/ shows the user was jon@kosli.com
  • The log file in service-identity/ shows the service was runner

Open the log file in command-logs/ to see the shell session:

This is a genuine log. You can see that Jon wanted to know if the container had a read-only file-system. This is because cyber-dojo containers originally ran in a Kubernetes cluster and did run in a read-only file-system. They were migrated and now run in AWS instead, but the read-only file-system is still a todo. 

Log sensitive data and keep it secret 

The example above will log raw session details into a permanent record. Shell sessions could contain sensitive data such as outputting environment variables and secrets. In a production setting we would recommend encrypting these logs before sending them to Kosli to ensure this sensitive data remains secret.

When to store data in Kosli Audit Trails?

You might want to consider Kosli’s Audit Trail when:

• You need append-only, immutable records for any business process
• You need off site records (double-entry bookkeeping). For example, if you have an internal system-of-record with sensitive data, writing a “copy record” can help you prove it hasn’t been tampered with.
• You want to create a unified view of disparate IT events 

We are discovering a multitude of uses for Audit Trails. For example, one very interesting use case we have seen at one of our customers is to provide a “double-entry bookkeeping” for financial compliance approvals.   

Get in touch if you want to try Audit Trails in beta

This feature is currently in beta. If you have an interesting use case and would like access please reach out. We are in the process of gathering feedback and improving support for Audit Trails before rolling it out to all our users.

We have documentation for how the feature works available here. If you’d like an Audit Trail for shell access as shown in the example above for your own AWS account, you can take a look at the documentation and source code and the terraform module.