A few years ago when DevOps and microservices were all the rage, it seemed like the software industry had reached a consensus on what good practice looked like. No book captured this better than Accelerate, a favorite of mine that guided my own work and thousands of other professionals. Based on survey data, the book lists 24 “capabilities” that every high performing engineering org should aim for.
Re-reading the book now, I still think it is filled with good ideas, but I would be hesitant to describe them as “best practices”. Every capability contains an assumption, and some may not apply in all cases.
For example:
Team experimentation—cheap experiments foster innovation.
Assumes that there are material gains to be had from fostering innovation.
Loosely coupled architecture—independent components enable scaling.
Assumes the coordination costs of independent components don’t outweigh the benefits.
Continuous Delivery—deployment frequency correlates to quality.
Assumes frequent deployments are possible or even desirable.
The trouble with this advice is it does not distinguish between different engineering org objectives. Consider a government software contractor: there’s no possibility for a dev to launch an AB test and discover a new UI that “delights customers” leading to a revenue win. The client does not want ad-hoc deployments, except in the case of fixing a serious bug. Even monthly releases may be too frequent. The government software contractor’s engineering objective is to satisfy the contract while minimizing delivery cost and operational risk, not to maximize market learning.
By treating all organizations as benefiting from rapid learning and frequent delivery, Accelerate may drive you in the wrong direction.
DORA
These metrics work for any type of technology your organization is delivering, but are best suited for measuring one application or service at a time. Whether you are building large language models, retail banking applications, mobile food ordering applications, or mainframe-based travel systems, these five metrics can help you assess the delivery performance of your application.
dora.dev/guides/dora-metrics
The DORA metrics were developed alongside Accelerate as a way of measuring software delivery performance. Unlike the 24 capabilities, they don’t prescribe how to work; they define metrics correlated with high-performing organizations.
But metrics, like capabilities, encode assumptions. If the assumptions change, the metrics themselves may cease to measure what matters.
DORA assumes that deployment is the unit of work. In a CI/CD environment, that’s a reasonable assumption because deployments happen frequently and are intentionally kept small. But with a fixed release cadence, that no longer applies.
So let’s consider the DORA metrics and how we might adapt them to our government software contractor example.
Deployment Frequency
How often a team successfully releases software to production over a given time period. Deployment frequency is really a proxy for an organization’s ability to work in small, independently releasable increments. When release cadence is fixed externally, that proxy breaks down.
Instead let’s use the number of tickets released over a given time period. A quarterly release containing ten tickets represents a very different engineering capability from one containing two hundred, even though both deploy on the same day.
Lead Time for Changes
This is the amount of time it takes for a committed change to reach production. Unfortunately measuring until production incorporates customer scheduling and contractual delays that are outside the engineering org’s control.
Instead of deployment, we measure the time from a ticket entering development until it is approved for release—the throughput that the engineering org actually controls.
Mean Time To Recovery
This is how long it takes to restore service to normal after a deployment-related failure. With infrequent deployments, it may not tell us much.
Instead we can broaden the measure to be the median time from discovery of a production issue until the fix is deployed. That could be a deployment-related failure, but it could also be discovered later, such as a customer-reported defect.
Change Failure Rate
The percentage of deployments that cause a failure in production requiring a hotfix or rollback. With infrequent releases, deployments don’t occur often enough to accurately track engineering activity.
Instead we might track the rate of tickets that caused a failure in production requiring a hotfix or rollback. That’s a stability metric that can be tracked per team, and improved over time.
“Best” practices
Software engineering practices are solutions to optimization problems. Change the optimization objective, and the optimal practices may change too.
Accelerate never claimed to discover universal laws of software engineering. It identified practices correlated with success in organizations competing through rapid software delivery. The truth is that “best” is shorthand for “best under these assumptions”. By drawing out those assumptions and comparing them to our own situation, we may well do better.
Tags: software-engineering