8 min read
With software delivery, speed is everything. But how do you balance rapid delivery with quality, security, and compliance?
To answer this question, let’s embark on a journey - one that starts in a software factory to running on the production superhighway.
From Factory Floor to Open Road
Gene Kim’s “The Phoenix Project” introduced us to the software factory, applying lean manufacturing principles to code production. But what happens next?
DevOps orchestrates a vast network of these software app factories. Each produces its own fleet of vehicles – from sleek sports cars (microservices) to sturdy trucks (monolithic applications) and specialized vehicles for niche purposes (utility scripts and tools).
The challenge? Ensuring this diverse fleet can navigate from factory floor through Dev’s testing tracks, Staging’s parallel routes, and finally merge onto the Production Superhighway smoothly, safely, and in compliance with all the rules of the road.
STOP 🛑 Security Checkpoints + Compliance Tollbooth
As code “vehicles” leave the factory, they embark on a complex journey through what we might call the Security Checkpoint and Compliance Tollbooth.
This stretch of pipeline is notorious for its frequent stops and thorough inspections, with a series of toll booths at every on-ramp and off-ramp.
At each checkpoint, code changes must come to a complete stop. They undergo various inspections – some automated, some manual. It’s akin to paying a toll, getting your license plate scanned, and going through customs inspection all at once.
In many organizations, this journey is managed through a series of manual workflows, often documented in tools like Jira, Confluence, or ServiceNow. Tickets are administrative backbone of the tollbooth system, tracking each vehicle’s progress and storing the results of each inspection.
Each vehicle (code change) gets a “ticket” – a printout stuck to the car window – detailing all the necessary information and checks required before it can proceed to the next stage. This ticket might be a Jira ticket, a Confluence page, or a ServiceNow change request, depending on the organization’s toolset.
At first glance, this system might seem comprehensive and foolproof. After all, every change is thoroughly inspected, right? But this approach has significant drawbacks:
🚦Friction: Each stop dramatically slows down the delivery process. Imagine the frustration of developers, stuck in traffic at yet another tollbooth, waiting for approvals in Jira or sign-offs in ServiceNow.
🚦Inconsistency: Like border checkpoints relying on spot inspections, not every “vehicle” gets the same scrutiny. A QA spot check might thoroughly examine one change while barely glancing at another, leading to inconsistent documentation in Confluence.
🚦Bypasses: Just as some might find ways around physical checkpoints, determined developers might circumvent these gates, potentially introducing risk. They might skip updating a Jira ticket or bypass a required field in ServiceNow.
🚦Limited Visibility: Once a vehicle passes a checkpoint, there’s often no record of what was inspected or approved. The printout on the window (Jira ticket or ServiceNow history) might get outdated, leaving gaps in the audit trail.
🚦Resource Intensive: This system requires a lot of “toll booth operators” - reviewers, approvers, and security teams manually checking each change and updating various tools.
Humans-in-the-loop for code reviews and security evaluations is essential. But too often, the mandate is to slow down and build even more tollbooths or add more fields to Jira tickets.
Do more checkpoints solve the problem? Or does this create a frustrating, slow, and unreliable developer experience for all concerned?
Busy highways need automated systems to manage traffic flow efficiently, just like our software delivery pipeline needs to automate routine checks while still allowing for human expertise where it matters most.
What we need is a way to keep our “vehicles” flowing with all necessary safety and compliance standards, without getting bogged down in a maze of tickets, alerts, and change requests.
A FastPass for Code: Software Delivery Evidence Management (SDEM)
SDEM introduces a game-changing concept to our production superhighway: cryptographic snapshots at the point of deployment and upon every change to production.
Unlike the error-prone paper tickets of the old system, this digital “license plate” along with factory VIN number:
✅ Automatically records every checkpoint passed and every change made
🎞️ Contains cryptographic proof of all tests run, approvals given, and scans performed
🚙 Updates in real-time as the vehicle progresses through the system
🔐 Cannot be altered or forged without detection
Onto the Production Superhighway with Real-Time Monitoring
Real-time monitoring doesn’t stop at deployment. If you extend tracking to the production environment itself - it’s like adding a vehicle tracking system such as OnStar or Toyota Connected, but for your software.
Now imagine having a bird’s-eye view of every “vehicle” on your production highway at any given moment. An environment snapshots feature that records every change, like a file integrity manager. These snapshots are like traffic cameras that capture the state of your production environment at regular intervals, allowing you to:
📸 See exactly what version of each service is running at any given time
🚨 Identify unauthorized or unexpected changes immediately
📊 Track the health and performance of your services in real-time
Just as Toyota Connected can provide vehicle health reports, collision notifications, stolen vehicle alerts, destination assistance and roadside support.
Kosli can generate real-time compliance reports, with alerts for unexpected changes or deployments, and help SRE and Vuln teams quickly identify and resolve issues by tracing them back to specific changes
Under Construction 🚧 + Emergency Incidents 🚔
Once through Compliance Tollbooths, our code vehicles finally merge onto the Production Superhighway. In an ideal world, this would mirror the staging test track perfectly, with identical conditions and predictable traffic. But reality is far messier.
The Production Superhighway is always in motion, and this is where the true story of what’s running in production unfolds. It’s not just about the vehicles that successfully passed through all the tollbooths - it’s about how they behave on the open road, and how the road itself changes beneath them.
How do you keep track of every vehicle, every lane change, every repair in real-time across a vast, busy highway system? This is where SDEM truly shines, providing a comprehensive view of your entire production environment.
Emergency vehicles (hotfixes) race down the shoulder, responding to critical issues that can’t wait for the normal deployment process. These towtrucks and ambulances, bypassing the usual checkpoints to address urgent situations. Road crews (ops teams) are constantly at work, making minor repairs and adjustments, tweaking configurations and scaling resources to meet changing demands. They might leave behind temporary changes – think flares or traffic cones – similar to manual Kubernetes adjustments like scaling the number of pods. It’s as if the highway itself is alive, expanding and contracting lanes as needed.
Security alerts or a breach may require immediate attention calling in the police cars. With an SDEM tracking every code change, these emergency vehicles can race down the road, sirens blaring, pinpointing exactly where the vehicles with Vulns are located, and pull these compromised containers out of production.
The Security Vuln overload pileup with false positives without view of runtime production
Now you’ve tracked these emergency responses meticulously, every temporary measure is logged like authorized change from the factory. Because over time, these small changes - both planned and emergency - will add up, causing the production environment to drift from its expected state.
What was once a straight, well-defined road might gradually transform into a winding path with unexpected turns and merges. An immutable change history ensures you have the lights on, and not operating in the dark.
Bird’s Eye View: Satellite Imaging of Your Superhighway
SDEM doesn’t just monitor the vehicles on the road. It provides a panoramic view of the entire highway system. Think of it as having access to real-time satellite imagery of your production environment.
This runtime environment scanning gives you actual snapshots of what’s really happening across your entire system. Triggered by system changes, you always have a bird’s eye view of every modification throughout your infrastructure. Whether it’s a new deployment merging onto the highway, a configuration change adjusting the road signs, or an emergency response rerouting traffic, SDEM captures it all.
how did we ever get anywhere without traffic maps and routing?
This holistic view is invaluable for:
- Ensuring compliance across your entire system, not just at the tollbooths
- Quickly identifying unauthorized or unexpected changes, like vehicles that somehow bypassed the checkpoints
- Understanding the full context of your production environment at any given time, including how emergency changes have altered the landscape
- Facilitating faster, more accurate incident response and resolution by providing a complete map of your system’s current state
With SDEM, you’re not just managing a series of checkpoints - you’re overseeing a living, breathing superhighway system, with all its complexity and constant change. This comprehensive visibility ensures that you’re not caught off guard by the realities of production, no matter how chaotic things might get on the road.
Now Accelerate with Confidence
By implementing SDEM, you’re transforming a congested, stop-and-go journey into a high-speed, secure, and compliant superhighway.
Go from a series of manual tollbooths into a smooth, efficient fastpass to the production superhighway with real-time traffic management.
Why wait at checkpoints when you can start recording and zoom ahead with confidence? Embrace SDEM and merge onto the future of DevOps today!
