7 min read
The fast-paced nature of modern software development means developers are capable of deploying changes to production multiple times a day. But, while DevOps allows development teams to deliver new features faster, increased deployment frequency can make it more difficult to stay on top of security threats. It only takes one malicious or incompetent change to dramatically increase the risk exposure of an application.
DevOps environments are constantly changing, so it’s a challenge to achieve a consistent cybersecurity posture. Change management processes aim to prevent unauthorized code from reaching production, but it can still be difficult to determine whether every change made to runtime is compliant. Undocumented changes can still slip through if companies aren’t continuously monitoring production environments.
In this article, we’ll look at the common causes and consequences of security failures, the importance of DevOps security and compliance, and how Kosli can help you improve your security posture.
DevOps Security Failures
DevOps security failures can occur when unauthorized code changes are pushed into production. Security failures caused by internal problems — such as laziness, poor training, or improper security controls — can lead to vulnerable or noncompliant code being deployed.
The problem is that change management processes often give companies a false sense of security. Traditionally, IT teams are used to relying on manual change approvals involving tickets and meetings to ensure the security and compliance of a software release. But, with continuous deployments to cloud environments, those manual processes don’t scale and cannot be relied on. Eventually, noncompliant code will slip through undetected and put the production environment at risk.
The research confirms it. In Accelerate, Nicole Forsgren, Founder and CEO at DevOps Research Assessment, found that traditional change management processes involving the use of external bodies like change advisory boards (CABs) were worse for mitigating risk than having no change process at all. Without effective monitoring in place for software running in production, risky code changes could become a security incident.
There are many external threats that can exploit risky code changes or vulnerabilities within the software supply chain. A data breach could occur, where malicious actors take advantage of security vulnerabilities to steal sensitive or confidential information. DevOps security failures could also increase the risk of other cyberattacks, such as intellectual property theft or cyber espionage.
Consequences of Security Failures
The consequences of security and compliance failures vary depending on the type and severity of the incident. A minor compliance issue could result in a fine, while a large cyberattack could have a significant financial impact on the business and even lead to bankruptcy. There are also many non-monetary consequences, such as reputation damages, a loss of intellectual property, legal implications, and more.
For banks, an unauthorized change to an application can result in customers losing access to funds. For companies that deploy code to mission critical devices and applications in healthcare, the outcomes can be even worse. In some industries there is no room for mistakes.
An incident or an outage can not only lead to a loss of clients and revenue, it can end up in catastrophic legal and brand damage. One of the best ways to mitigate against these risks is to monitor, detect, and then notify engineers of unauthorized changes running in production as quickly as possible.
Let’s consider one of the most famous examples of an unauthorized change going to production, and how it went undetected until it brought down the whole organization.
The Impact of Knight Capital’s Security Failure
Knight Capital was a global financial services firm that suffered from a software error which led to a catastrophic trading loss and a collapse in its stock price. After employees made a system upgrade, faulty code resulted in the program placing millions of errant trades that ultimately lost Knight Capital over $440 million.
This incident could have been prevented if the company implemented sufficient risk management and security controls. If Knight Capital had continuous runtime monitoring with notifications in place, the bad change could have been identified and rolled back before it did any damage.
Unfortunately, Knight Capital’s change management process could not prevent the deployment of a disastrous change to production. No one noticed it until the company was hemorrhaging millions of dollars in trades, and by then it was too late to save it.
Security Compliance in DevOps Environments
In the DevOps world, security and compliance are two distinct concepts. This is because complying with regulations doesn’t necessarily mean an application is immune to security failures and cyber threats. Protecting an application (and preventing the negative consequences of a secure failure) requires adequate DevOps security and compliance together.
DevOps security involves monitoring and reviewing code changes and applications throughout the entire software development lifecycle. This includes everything from secure coding practices and scanning pull requests for potential vulnerabilities to penetration testing and threat detection after deployment. Many companies are also adopting a DevSecOps approach, where development, IT operations, and security teams come together to deliver more secure software.
DevOps compliance involves proving the software adheres to industry regulations through documentation, change management processes, frequent audits, and more. Compliance not only reduces the risk of a security failure, but is also a requirement in many industries. An automated approach to DevOps compliance can streamline change controls and evidence gathering throughout the CI/CD pipeline.
Implementing DevOps Security Compliance
When it comes to implementing DevOps security compliance, it’s important to define (and then actually follow) a secure software delivery life cycle (SDLC) that tracks everything deployed into production. For compliance purposes it’s also necessary to prove that you’re following it.
This means gathering compliance-related data for all actions between commit and deployment to build a transparent audit trail. Generating a provable record of a compliance-centric software delivery process reduces risk and helps organizations pass security audits.
Beyond reviewing changes for compliance before deployment, it’s also important to implement real-time monitoring in production. Rogue deployments, undocumented workloads, and other unauthorized changes to production environments can lead to security failures if they’re not discovered immediately.
Even fully-approved changes cause a security failure if there is a new vulnerability within the software supply chain that isn’t remediated quickly. Real-time monitoring can help security teams keep up with rapidly changing software and environments to prevent these types of security failures.
Fully-Automated DevOps Security Compliance
A fully-automated DevOps security compliance strategy is essential for all businesses, but implementing it from scratch can be challenging. That’s why companies should consider a DevOps security compliance solution that easily integrates with existing tools and workflows so that security and DevOps teams can quickly get started.
Kosli is a security and compliance solution that simplifies governance engineering for DevOps teams. By automatically tracking what’s running in production, how it got there, and whether it adhered to change management processes, Kosli helps secure DevOps environments in real time.
More specifically, Kosli provides continuous compliance and continuous security monitoring capabilities that reduce the risk of compliance issues and security failures. Kosli’s Evidence Vault also streamlines the collection and storage of compliance evidence, and creates a transparent audit trail to meet the requirements of SOC2, ISO 27001, GDPR, PCI DSS, and other regulations. This data is easily accessible for security audits and incident investigations.
How Kosli Could Have Prevented the Swedbank Outage
Swedbank is a Nordic-Baltic banking group that had a major outage caused by an unapproved change to its IT systems. This left almost one million customers with incorrect balances, and many were unable to make payments. The company was fined $85M for the incident, and likely suffered reputational damages that have had a lasting impact on the business.
Although Swedbank had a change management process in place, the incident was caused by not actually adhering to it. If Swedbank had implemented a solution like Kosli, the company could have identified the unauthorized change in production before it caused an outage. This is just one example that proves traditional change management processes are no longer enough to prevent security failures.
DevOps Security Compliance with Kosli
As you can see, DevOps security compliance requires monitoring how environments are changing and where those changes have come from. The only way to be sure about your security and compliance status is to have monitoring in place that can give you real time answers to three questions:
- What’s running in production?
- How was it deployed?
- Is it in compliance with our defined process?
Traditional change management processes may help companies comply with industry standards and regulations, but they do not eliminate the risk of security failures. Continuous compliance and security monitoring with Kosli helps DevOps teams ship new code with confidence. By automating software governance, organizations can deliver software faster while still prioritizing strong security and compliance from initial code commit to deployment in production.
Is security and compliance a pain in your DevOps? Find out how Kosli can help
