How to Detect Unauthorized Changes in Production with Kosli

Let’s not beat around the bush: change management is a prehistoric discipline desperately in need of fresh thinking. Its “best practices” are frankly terrible. Nobody honestly thinks manually filling out change tickets, waiting for CAB meetings and external approvals does anything to meaningly reduce risk. Change management is slow, inconsistent, doesn’t scale, and is prone to error.

Of course, lots of teams know this and more and more of them are delivering change via automated approvals in their DevOps pipelines. These pipelines automatically document every release and provide an opportunity to automate change decisions in a scalable and repeatable way. This provides more consistency, accuracy, and most importantly the combined benefits of speed with compliance you can actually trust.

Teams who deliver changes automatically to production in this way significantly outperform teams that rely on manual approvals. They have faster delivery and fewer failures.

But regardless of the method of delivering approved changes, there is a significant underlying risk that many organizations can’t detect: unauthorized changes in production.

What is an unauthorized change to production?

An unauthorized change is any change to production that didn’t go through the required risk management process.

Examples range from the benign (a developer deployed a test container to Kubernetes to validate a bug fix) to the malicious (a breach led to a crypto miner running on production infrastructure).

While unauthorized changes can be harmless, there are many cases where unauthorized changes can cause serious harm.

Why are unauthorized changes harmful?

If you work in a regulated industry, managing your IT risk is an integral and essential part of your authority to operate. In fact, as some banks have found out, failure to put in place risk controls mechanisms to detect this can lead to fines up to $85MUSD.

As well as being expensive, undetected changes make incidents hard to resolve. Imagine being the engineer on pager duty who is tasked with triaging alerts on a large distributed system.

Furthermore, undetected unauthorized changes expose a significant audit risk. The last thing you want your auditor to find is a running workload without proper documentation.

For me, what is really interesting is the parallels between the Swedbank and the Knight Capital incidents. In both cases there was an incomplete understanding of how changes were being applied to production systems due to insufficient observability and traceability - and this prolonged and amplified the impact of the incidents.

It leads to interesting questions: how many similar changes have been made that didn’t cause an outage? And how could you prove that you’ve never run an unauthorized workload?

Ask yourself - do you know exactly what’s running in your production environments right now?

How to detect unauthorized changes in production

We can think of software changes as streams feeding into our environments which are lakes. Change management puts a gate in the stream to control what flows into the lake, but it doesn’t monitor the lake. And there is always a means to make a change to production outside of the process. In fact, the only way to prove you don’t have unauthorized changes is with runtime monitoring. Knowing what’s coming down the stream isn’t enough. You need to be sure that nothing is entering the lake from elsewhere.

There are 3 pillars for ensuring that your runtime environments are properly secure:

1. Runtime Forensics: Continuous monitoring and recording of runtime workloads. A historic record for what has been actually running in production over time. (the lake)
2. Change Compliance Records: Records for all approved workloads. Attestations and evidence that every change has met your compliance and security requirements. (the stream)
3. Controls and Alerts: Notifications when non-approved or undocumented changes are found (action)

How Kosli detects unauthorized workloads in production

1: Runtime forensics

Kosli makes it easy to record any environment. Use our CLI to set up monitoring for production. Kosli can track container workloads, S3 buckets, serverless functions, server folders and more. Every time there is a change to an environment, Kosli records a snapshot of what is running. With this simple step, you now have a full picture of how your system changes over time. Now you can prove the negative: that you’ve never run an unauthorized workload.

2: Change compliance records

The next step is to ensure that every workload is compliant with your process. Kosli Flows make it easy to record attestations and evidence in your pipelines for your risk control steps such as code review, unit test evidence, security scans, and more. This codifies your change management documentation with cryptographic proof of compliance for every change.

3: Controls and Alerts

Now you know what’s in production, and if it is compliant. The next step is to control and alert for deviations. Kosli makes it easy to set up slack notifications and webhooks for events such as non-compliance. And, of course, the best approach is proactive. You can use policy as code approaches such as OPA or the kosli cli to make pre-deployment decisions in your DevOps.

Conclusions

Imagine trying to prove to an auditor that an unauthorized workload has never run in your environment. It’s practically impossible. You might be able to show that right now there are no unauthorized changes. You can maybe even show the evidence for every valid change. But you’d need a time machine to prove that only authorized compliant workloads have ever run.

But with Kosli, you get that time machine for your environments. You can easily prove how production has changed over time, and you can prove every change was legitimate. And most importantly, you can prove that you are in control.

If unauthorized changes are a threat to your organization, take some time to discuss your challenges with us.

Alternatively, if you’d like to explore the product under your own steam you can try our free tier.

And if you’re looking to join those high performing teams that are deploying with speed and compliance we have a free SDLC template that gives you a DevOps friendly process that your developers and auditors will love.