The high stakes of SDLC compliance: Lessons from EVE Online's battle of B-R5RB and Equifax

n our previous exploration of The Punchcard Paradigm, we traced the roots of modern compliance practices back to the early days of computing. We saw how the physical constraints of punchcards shaped programming practices and how those practices lingered long after the technology had evolved.

Now, let’s dive deeper into why modern compliance is more critical than ever in today’s digital landscape.

Why Compliance Matters

At its core, compliance is about ensuring the reliability, security, and trustworthiness of our systems and processes.

It’s not about ticking boxes or following rules for their own sake.

As described in The Punchcard Paradigm, compliance was achieved through physical checks and balances, like the visual review of punchcards against coding sheets. As systems grew more complex, manual processes like Change Advisory Boards (CABs) emerged to manage risk. However, these processes have struggled to keep pace with the speed and agility demanded by modern software development.

The Illusion of Control

Many organizations cling to the belief that centralized control and manual approvals are the best way to manage risk. The thinking goes that if we can just review and approve every change, we can prevent issues from reaching production. But is this really effective? Does having more people around the table make the process more robust?

The Accelerate study by Dr. Nicole Forsgren and her team found, this is often an illusion. In fact, the study showed that having an external approval process like a CAB performed worse than having no approval process at all in terms of lead time, deployment frequency, and restore time!

Why is this? CABs, with their high approval rates and superficial reviews, often fail to catch potential issues that could be identified through more integrated and automated processes.

As Dr. Forsgren puts it, “The illusion of control provided by CABs often leads to complacency. While these boards approve most changes, they rarely add value in terms of risk mitigation. Instead, they create delays and frustration within teams.”

The Battle of B-R5RB: A Tale of Cosmic Proportions

The potential consequences of small oversights and the illusion of control are vividly illustrated by the infamous Battle of B-R5RB in EVE Online. This massive multiplayer online game is a universe where players can forge alliances, wage wars, and engage in complex economic activities. It’s a world where virtual actions can have real-world value.

On January 27, 2014, the EVE Online universe witnessed the Battle of B-R5RB, a clash of epic proportions involving over 7,500 players from 717 corporations. The catalyst for this cosmic conflict? A missed payment.

The system of B-R5RB, used by the alliance Pandemic Legion as a staging area, became vulnerable when a sovereignty bill went unpaid. As reported by Bo Moore in Wired, “It appears someone forgot to check ‘Automatic Bill Pay’ in his game settings. Pandemic Legion insists the box was checked, raising the possibility an inattentive player, a spy or even a bug in the game was to blame. Whatever the case, the rent didn’t get paid.

The biggest battle in the history of forever started with a clerical error.”

The Battle of B-R5RB: The biggest online battle in the history of forever started with a clerical error

image credit to @CaptainBenzie

Seizing this opportunity, the opposing coalition CFC/RUS moved in, capturing the station. Pandemic Legion and their N3 allies countered, escalating the battle to a full-scale war. For 21 hours, the battle raged, with 10 seconds of real time equating to 1 second in-game due to server strain.

B-R5RB Initial CFC jump in 27/01/2014 #1

The battle saw the destruction of 75 titans, each worth about 100 billion ISK (roughly $3,000), and over 500 capital ships, with a total damage of 11 trillion ISK (equivalent to around $300,000), making it the costliest battle in EVE Online history at the time.

This cosmic clash, started by a clerical error, illustrates how in complex, interconnected systems, a single point of failure can have cascading consequences. As CCP Games, the game’s publisher, put it in their official report

“One pilot’s action (or inaction) had repercussions for the entire universe… a butterfly wing causing a massive typhoon of destruction.”

Now for EVE players, unpredictable gameplay is part of lure and the lore. And the Battle for B-R5RB would soon be eclipsed in the following decade by more epic battles.

Yet, the lesson underscores the need for robust, automated compliance processes, especially around critical resources, that can catch these oversights before they lead to catastrophe.

The Equifax Breach: A Down-to-Earth Disaster

The butterfly effect of compliance failures is not confined to the virtual world. In 2017, Equifax, one of the largest credit reporting agencies, suffered a massive data breach that exposed the sensitive information of 147 million people.

The cause was a failure to patch a known vulnerability in the Apache Struts web framework. Despite the patch being available for months, Equifax failed to apply it to their systems. This single oversight led to one of the most damaging data breaches in history, with costs estimated at hundreds of millions of dollars.

The former CEO of Equifax, Richard Smith, in his testimony before the U.S. House Committee on Energy and Commerce, stated, “Both the human deployment of the patch and the scanning deployment did not work. But the protocol was followed*.*”

The point is, following a protocol is not enough if that protocol is fundamentally flawed or inadequate. And clearly, in the case of Equifax, the protocol for patching known vulnerabilities and scanning for threats was not up to snuff.

It’s a stark reminder that compliance is not just about ticking boxes and following procedures.

From Virtual Space Wars to Real-World Breaches

The parallels between the B-R5RB battle and the Equifax breach are striking. In both cases, a seemingly minor oversight – a missed payment, a forgotten patch – had catastrophic consequences. And in both cases, these oversights could have been caught by robust, automated compliance processes.

Imagine if, in the EVE Online scenario, there was a report that could have validated that any crucial payments for station sovereignty had not gone through. But instead, the attacking coalition found out about the vulnerability before the defenders did, and it took them under 8 hours to launch an attack that changed the course of the war.

Similarly, if Equifax had a system in place to continuously monitor their environment for known vulnerabilities and validate that all necessary patches had been applied, could the breach have been prevented? The answer is likely yes.

These incidents highlight the critical importance of continuous compliance – of having automated checks and balances at every stage of the development process. By integrating compliance into the very fabric of the development process, by ensuring that every change is validated against predefined standards, we can catch these issues early, before they have a chance to cause harm.

Empowering Developers: Through Continuous Compliance

Continuous compliance isn’t just about catching bugs and vulnerabilities. It’s also about empowering developers to build security and reliability into their code from the start.

By codifying compliance requirements into automated tests and policies, we make compliance an integral part of the development workflow. Developers can get immediate feedback on whether their changes meet necessary standards, without having to wait for manual reviews.

If you’ve ever been the Dev who worried “Did I just break Prod?” with the last PR, or tasked as SRE to find out which of the myriad of changes results in alerts firing 🚨, this article tells the story of such a Dev + SRE combo.

This shift requires a change in mindset. Compliance needs to be seen not as a bureaucratic burden, but as a key enabler of quality and reliability. It needs to be embedded into the culture of the organization, with everyone taking responsibility for the integrity of the systems they build.

The Road Ahead

The shift to continuous compliance is a journey, not a destination. It requires changes in tools, processes, and most importantly, in culture. But it’s a journey we must undertake if we want to build systems that are truly trustworthy and resilient.

In the next article, we’ll explore the practical steps organizations can take to implement continuous compliance. We’ll look at the tools and practices that can help embed compliance into every stage of the software development lifecycle.

We’ll see how by embracing this new paradigm, we can not only improve the security and reliability of our systems but also unlock new levels of agility and innovation. The future of compliance is continuous, and it’s a future we must embrace if we want to thrive in the digital age.