7 min read
Until fairly recently, software releases happened once or twice a year, maybe once a quarter. This gave IT teams plenty of time to verify and manually sign off on every change before they were released in big batches during a bank holiday weekend or off-peak hours. Typically, they’d produce paperwork to show that all changes had been properly tested, and then those changes would be approved for release in a change advisory board meeting (CAB).
But these change management processes no longer work because today’s software teams are deploying multiple times per day. In fact, companies like Amazon and Airbnb are deploying over 100,000 code changes every day. How are manual change management processes supposed to keep up when DevOps teams are able to deliver features and changes faster than ever?
In this post we’ll talk about the challenges, pitfalls, and best practices of change management in DevOps. We’ll also cover how automation can streamline change management while improving security and compliance.
Understanding Change Management in DevOps
Every software change comes with some risk, whether it’s a negative impact to the user experience, the introduction of a security vulnerability, or a potential compliance violation. That’s why IT and system managers rely on a change management process to maintain software quality and control for risk in every change they make.
For example, they might decide that to control for risk every change must go through a unit test, a security scan, and a pull request before it can be deployed. The change management process is about gathering the evidence to show those steps have been completed so that the changes can be approved and deployed into production.
Change management for the release of software is usually interpreted as a “normal change” under the ITIL framework. Let’s quickly remind ourselves what that is and how it differs from a “standard change.” A normal change would include a change request, assessment and planning, and managerial approval before implementation. Standard changes — which are low risk and follow an automated procedure — can often bypass the lengthy change management workflow and be deployed immediately.
Classifying a software release as a normal change is why the process is still typified by paperwork and meetings. It’s also why many DevOps organizations want to leverage automation to achieve continuous compliance that turns deployments into standard changes.
In short, change management is crucial for ensuring software stability because every change can introduce bugs or vulnerabilities. Change management is also important in regulated industries that require companies to prove they’re following a defined process. The question is whether having a big ceremony around gathering evidence, raising change tickets, and sitting in meetings is effective for mitigating risk.
Challenges and Pitfalls of Change Management
The main challenge with traditional change management is that it seems to oppose key DevOps practices, such as very frequent deployments and continuous delivery. Even minor, low risk changes can be slowed down by change advisory board (CAB) meetings and other manual processes. The traditional change management approach doesn’t scale now that software teams are making hundreds of changes on a daily basis.
At the same time, many organizations in highly-regulated industries like finance, automotive, and healthcare assume that faster development velocity leads to increased risk. They feel reassured by delays and formal meetings and tend to assume that these are effective ways of reducing their risk exposure. The reality is that the opposite is true. More frequent changes are smaller and safer. Batched changes and “big bang” releases are often poorly vetted and introduce much greater risk. It’s much easier to track down the root cause of a failure within a single incremental change than a large batch of changes.
Research in Accelerate by Nicole Forsgren, Founder and CEO at DevOps Research Assessment (DORA) also found that slowing down and using manual processes like CABs is “worse than having no process at all!” An organization that pushes changes quickly can most likely pull back changes just as fast if there’s a problem. By taking an iterative approach, development teams can also get feedback faster, leading to better quality code in the long run.
The key to change management in DevOps is implementing an automated process for managing and mitigating risk. By eliminating manual CAB reviews and signoffs, and adopting modern change management best practices, software teams can shorten release cycles without sacrificing stability. In the following sections we’ll describe ways to shift from traditional change management to a DevOps-centric risk mitigation approach.
Best Practices for Change Management in DevOps
Here are some best practices for implementing a successful DevOps change management strategy.
Establish clear communication channels between stakeholders
The DevOps approach brings together development and operations teams to accelerate the software delivery process, but the audit and compliance teams are often still a barrier to deployment. Security has also traditionally been siloed until companies began adopting a DevSecOps approach, where DevOps and security teams collaborate to deliver more secure software.
Establishing clear communication channels and cross-functional teams with all stakeholders ensures the goals of each team are considered when implementing an effective change management process. Aim to bring governance tasks like audit and compliance fully into your DevOps value stream instead of having them siloed.
Automate change request workflows
Automating existing change request workflows is a great way to accelerate change management to keep pace with DevOps. For example, a Jira ticket or ServiceNow change request can be automatically created when a developer pushes a code change. This not only eliminates manual effort, but also makes it easier to document changes and approvals using data from an automated workflow.
Get the most out of your CI/CD tools
Many companies implement continuous integration and continuous delivery (CI/CD) tools like Jenkins and CircleCI when they adopt a DevOps strategy. These tools can automate and track every step of the software delivery process from initial code commit to deployment to production. From a change management perspective, CI/CD tools also collect valuable data that helps with documenting changes and generating an audit trail throughout the software delivery process.
Use configuration management tools
Besides source code changes, software deployments often involve infrastructure and configuration changes. Tools like Ansible, Chef, and Puppet help software teams manage the current state of applications and environments. Just like version control and CI/CD tools, configuration management tools make it easier to track configuration changes and roll them back when there are issues.
Compliance and Security Considerations
Automating change management can not only accelerate releases, but also improve DevOps security and compliance. It only takes one bad change to compromise the security and compliance of an application, so change management is an important aspect of maintaining a consistent cybersecurity posture.
From a security and compliance perspective, automated change management processes are useful for gathering evidence from every code commit, deployment, and release to build a transparent audit trail. This helps organizations pass security audits and demonstrate compliance with industry regulations, reducing the risk of fines, legal issues, and reputational damage.
It’s also important to remember that old school change management practices involving tickets and meetings can give companies a false sense of security, with unauthorized changes still slipping through and leading to a security failure. That’s why continuous monitoring in production is crucial for identifying rogue deployments, undocumented workloads, and other non-compliant changes.
Automated Change Management with Kosli
The speed at which product teams are able to ship new features is always increasing, so it’s important to modernize your change management processes to keep up with continuous integration, automated testing, and continuous delivery. Despite the inherent friction between change management and DevOps, these practices can co-exist using the right approach.
Kosli is a security and compliance platform that can help DevOps teams to automate all of the tasks they need to complete to meet their change management criteria. By replacing slow and manual change approvals with automated evidence gathering, Kosli allows you to continuously deploy compliant software without batching changes into large, risky releases. For example, Firi used Kosli to deploy over 100,000 changes in a year without any hand brakes or compliance issues,
With Kosli you can make continuous compliance and continuous security monitoring the center of your automated change management strategy. The reality is that traditional change management isn’t enough to fully protect applications from security failures, so it’s important to implement real time monitoring to detect and mitigate unauthorized changes in production.
As software development becomes more complex and fast-paced, it’s getting harder to stay on top of changes. Kosli gives you the confidence to ship faster while still prioritizing compliance and security — even in heavily regulated industries.
Reach out to Kosli to learn more about automating change management in DevOps.
