7 min read
Are you trying to do DevOps under regulation? If so, you’ll know the pain of change management. In this article we’ll look at how delivering software with DevOps is incompatible with old school ways of managing change with ITIL and how you can automate your change management process with a DevOps approach. As regulated industries speed up their DevOps processes they find that managing software releases with ITIL tickets and change meetings just doesn’t scale.
The ITIL change management framework made sense when we were releasing a relatively small number of changes in a quarterly release, but how do we manage change now that it’s possible to make hundreds of changes every day? Let’s dig in and find out!
What is the ITIL framework?
What does ITIL actually stand for and what does it do? The Information Technology Infrastructure Library is a framework for managing technology change in organizations. It is an attempt to standardize the overall lifecycle of IT infrastructure from initial procurement to implementation, maintenance, and eventual replacement.
It’s important to realize that ITIL is not a set of hard and fast rules. Instead, it offers IT managers a set of guidelines and best practices which they can use as a framework for their own policies and procedures. And one of the key aspects of the guidelines is the way changes are categorized.
What are the 4 types of ITIL changes?
Under the ITIL framework, changes are categorized into four categories according to the potential risk they pose to an organization. These categories are: emergency changes, major changes, minor changes, and standard changes. Let’s take a closer look at each category.
Emergency changes: These are (hopefully!🤞) rare events that require immediate action and the change approval process has to happen retrospectively. These fall outside the scope of how change is usually managed in ITIL. Emergency changes might be required for events like servers going down or security breaches - existential threats that need to be addressed without delay.
Major changes: Major changes are high risk, high impact changes that have the potential to severely impact your day-to-day operations. A major change might result in downtime and interruptions to live production environments. Examples might include significant changes to network infrastructure or large data migrations. These events require planning, preparation, and approval through a change advisory board (CAB) where risk has to be assessed and authorized by management before implementation can take place.
Minor changes: Minor changes are not as trivial as they sound. A request to make minor changes still needs to be made to the change board where approval has to be obtained before the changes can be made. Examples of minor changes include updates to websites, installation of new network devices, and deployments of security patches. These are low risk events, but under the ITIL framework they still require a change process similar to major changes.
NB: Major and minor changes are sometimes grouped together as “regular changes.”
Standard changes: Standard changes are low impact, low risk changes. An example might include an OS upgrade, firewall changes, or new DNS entries. For the purpose of this discussion the most important thing to know about standard changes is that they are pre-approved. That means they can be made without long lead times involving change requests or CAB meetings. Bear this in mind as we proceed!
Why are regulated DevOps teams still using ITIL?
The short answer is: history. This is how it’s always been. The IT department takes responsibility for managing change. This makes sense when you consider that for the most part IT is concerned with big changes that involve lots of risk and require preparation and approval. Traditionally, releasing software in big batches is considered in much the same way, so DevOps teams find ITIL thrust upon them.
A software release isn’t the “big bang” event it once was. But when you have established ITIL change management processes, with clearly defined change advisory roles, all built around enterprise software like Service Now, it’s really hard to imagine another way.
Even if your developers have made the move to infrastructure as code, reproducible builds, automated testing and (in theory!) continuous delivery, how to actually deploy changes to production without ITIL change approval isn’t obvious. And that’s because regulated teams must be able to prove what’s going on in their process. Or, they have to at least pretend they know they can with some sort of process involving paperwork and CABs.
Why ITIL change management is ineffective for regulated DevOps?
If your team is developing software in a regulated industry like finance, automotive, or healthcare you need a change management process to mitigate risk. Unlike your counterparts in unregulated environments, you have to document the changes to your software and you need to be able to prove that your software is being developed according to a defined process. And if you’re trying to do DevOps you’ll know this creates an ITIL shaped bottleneck just before the release.
Regulated DevOps teams find themselves in a frustrating situation where they have a highly dynamic way of developing software that has lots of automation for activities like build and test, but they still can’t deploy their changes without long lead times. At the release stage they meet a change management roadblock and it’s the complete opposite of DevOps: handovers, meetings, checklists, tickets, etc. We call this the DevOps Lite Trap.
The fact is DevOps produces high volumes of change to the software that ITIL was never equipped to manage in an effective way.
How to automate your change management needs with DevOps
But what if you could use DevOps to automate our change management requirements on a continuous basis along with everything else? We already learned this lesson for integration and delivery. Why not for compliance too? In a previous article we wrote about automating a secure chain of custody across your pipelines. It shows how you can execute an automated change management process with DevOps instead of ITIL.
- Define your SDLC. What does your process look like? What has to happen between the initial commit and the deployment for a release candidate to be considered compliant and ready for production? Which tests, scans, pull requests, etc. are required to mitigate your risk?
- Ensure the binary provenance of your artifacts. Take the SHA-256 cryptographic fingerprint for every artifact in the build process because it is a unique identifier. This means you can’t qualify one thing and deploy something else because the SHAs will be different.
- Attach evidence to the SHA-256 for every change that occurs to an artifact as it passes through your pipeline. So, every unit test, security analysis, pull request, etc. is recorded against the SHA-256 as and when it happens. It’s also important that this information is recorded in an append-only way. This means you can now automate your risk controls in the pipelines instead of trying to validate them with paperwork later on.
Human in the loop? 🙋 No problem. There still doesn’t have to be a high ceremony. The evidence for your process is being logged automatically, so you can quickly establish binary provenance for the artifact and check that it has gone through all of the necessary steps defined in your SDLC. If you can see that all the necessary evidence is recorded against the SHA-256 you can release the artifact to production because you know that it’s in compliance with the process.
Remember the standard changes we talked about earlier? 💡 By making your change management process part of your DevOps there’s no reason why individual releases to production shouldn’t be considered as pre-approved, standard changes. With automation you can verify each change without any batching, meetings, tickets, or delays. 🥳
Conclusion
Tl;dr - ITIL change management doesn’t work for DevOps because meetings and tickets can’t keep up with the number of changes. It’s simply unrealistic to expect a daily (hourly?) CAB meeting to approve a continuous delivery pipeline. What would that even look like? 🤷
Under ITIL some regulated teams have succeeded in reducing their lead time for changes to weeks, and in some cases even down to days. But to get from months to minutes - and that’s where the elite performers are heading - you’ll need to automate the ITIL change process for software releases with DevOps.
