Design to Be Operated (SRE)

Chapter Info

Calculating... Writing Progress: 20%

A system is not well designed if it works but cannot be operated reliably. Production exposes a different set of constraints: failures must be understood under pressure, recovery must be safe, reliability must be measurable, and the system must remain sustainable for the people who operate it.

Operability is therefore a design property, not something an operations team bolts on after release. Site Reliability Engineering (SRE) provides many of the engineering practices for achieving it — turning reliability into something that can be measured, operated, and continuously improved.

This chapter explores how reliability is defined, how systems are operated and recovered, how teams learn from failure, and how AI agents are changing operational work.

The Operability Mindset [Mindset] {1}

Operability is the discipline of making a system practical to run, change, and recover at any hour. It is designed into the system rather than added after release.

Building Is the Beginning, Not the End [Run Cost] {1}

The headline cost of software is what it takes to ship version 1. The real cost is everything that comes after: keeping it up, patching it, migrating it, debugging it under load, and supporting it through generations of engineers. Design decisions often treat shipping as the finish line. Operability starts from the opposite assumption: release is when the system begins its long life.

Operability vs Observability vs Availability [Distinctions] {1}

Three words, three different jobs. !!Observability is the ability to see what the system is doing. Availability describes whether users can successfully use it. Operability is the discipline of running, changing, debugging, and recovering it sustainably!!. A system can be observable yet difficult to operate, or available yet dependent on unsustainable human effort.

You Build It, You Run It [Build & Run] {1}

When the team that writes the code is also exposed to its operational consequences, design decisions change. The feedback becomes immediate: latency matters more, errors need to be readable, retries need boundaries, and deployments need safer defaults. Separating builders completely from operational consequences weakens this feedback loop. Giving engineering teams direct responsibility for how their systems behave in production makes operational quality part of design.

Site Reliability Engineering [SRE] {1}

Operability is the goal; Site Reliability Engineering is an engineering discipline for achieving it. SRE treats reliability as something that can be measured, budgeted, automated, and continuously improved. Service-level objectives define what reliable means, error budgets provide a quantitative basis for deciding how much operational risk the team can take, and operational work becomes input for improving the system rather than a permanent human burden.

Much of modern SRE practice was codified and popularized at Google, notably through Site Reliability Engineering (2016) and The Site Reliability Workbook (2018). These books helped establish many of the practices used throughout this chapter, including SLOs, error budgets, toil management, sustainable on-call, and learning from incidents.

Define Reliability [Define Reliability] {1}

Reliability Starts With the User [Start With the User] {1}

A system can have healthy CPU, memory, network, and processes while users are still failing to complete what matters. !!Reliability should therefore be defined from the outside in!!: start with the user journeys the service exists to support, then use infrastructure signals to explain why those journeys succeed or fail.

Measure What Users Experience [Measure UX] {1}

Reliability metrics should describe what users actually experience, not just summarize system behavior. For example, latency is better understood as a distribution of response times than as a single average. Averages can hide slow experiences for a meaningful share of users, while percentiles make those slow experiences visible.

Service Levels — SLI, SLO, SLA [Service Levels] {1}

Reliability becomes manageable once you can define it. An SLI measures what users experience, an SLO defines the target the team aims to meet, and an SLA turns reliability into an external commitment. Together, they replace vague expectations with measurable outcomes.

SLI — What You Measure [SLI] {1}

A Service Level Indicator measures a specific aspect of the service experience: the proportion of requests completed within a latency threshold, the fraction that return successfully, or the share of writes that durably persist. A useful SLI changes when the user experience changes. Infrastructure metrics such as CPU utilization are usually diagnostic signals, not user-facing SLIs. Pick a few user-facing measures and express them as a ratio of good events to total events.

SLO — The Target You Aim to Meet [SLO] {1}

A Service Level Objective is the internal target you set for an SLI over a window: "99.9% of requests complete within 300 ms, measured over 28 days." The measurement window and the definition of a good event make or break it. The window should match how the service is operated; rolling windows are often useful because they avoid calendar-boundary effects. Latency objectives should capture the proportion of requests that meet a meaningful threshold rather than relying on an average that can hide poor user experiences.

SLA — The Promise With Consequences [SLA] {1}

A Service Level Agreement is an external commitment tied to business consequences when it is missed. An SLA is a business promise; an SLO is an engineering target.

Keep Margin Between Internal Targets and External Commitments [Internal Margin] {1}

When an external SLA exists, an internal SLO can provide an earlier threshold for action. Setting the internal target above the contractual commitment gives the team room to respond before customer consequences begin. The size of that margin should reflect the service, the contract, and how quickly reliability can be restored.

Dependencies Put a Ceiling on Reliability [Dependency Math] {1}

No service is an island: yours calls a database, an auth service, internal APIs, and a cloud provider. For mandatory dependencies called in series without a fallback, availability multiplies: five dependencies at 99.9% each cap the path at 0.999⁵ ≈ 99.5% before your own code runs a single line. Redundancy, caching, graceful degradation, and asynchronous work can change that equation. The lesson is to set an SLO that reflects the path users actually take, including both its dependencies and the mechanisms that reduce their impact.

Error Budgets [Error Budgets] {1}

Every SLO implies some tolerated failure. Error budgets make that tolerance explicit so reliability and delivery risk can be discussed with numbers rather than opinions.

100% Is the Wrong Target [Error Budget] {1}

Perfect reliability is rarely the right target. The gap between 100% and the SLO is the error budget: the amount of unreliability the service can absorb while still meeting its objective. For an availability SLO of 99.9%, that corresponds to roughly 43 minutes of unavailable time in a 30-day month. That margin acknowledges that limited unreliability can occur while the service still meets its objective.

Use the Error Budget to Govern Risk [Budget Policy] {1}

An error budget turns a subjective debate about shipping risk into an explicit policy. !!When the budget is healthy, the team can take more risk; when it is exhausted, the team should reduce risk and prioritize restoring reliability!!. The exact response is a product decision — it might slow releases, require stronger safeguards, or focus capacity on stabilization — but it is agreed before the incident or the deadline. The number does not replace judgment; it gives that judgment a shared frame.

Critical Windows Change the Risk Policy [Critical Windows] {1}

Critical business periods can justify a more conservative operating posture even when the underlying SLO does not change. During events such as Black Friday, tax deadlines, or major launches, the cost of failure is higher, so teams may reduce deployment velocity, increase capacity margin, tighten change controls, and escalate sooner. !!The reliability target defines what users need; the operating policy defines how much risk the organization is willing to take at that moment!!.

Set Reliability Targets from Evidence [Evidence-Based Targets]

The right SLO is the lowest reliability target that still delivers an experience users consider acceptable. Start with historical performance and user evidence, identify the point where reliability begins to create meaningful user pain, and set the objective deliberately around that boundary. A service that genuinely requires 99.99% availability should be engineered for it; one whose users are satisfied with 99.9% should not absorb the additional cost and complexity without a reason. Reliability beyond what users need is not free: it consumes engineering complexity, infrastructure, and operational effort that could be spent elsewhere.

READS: The Signals That Explain Service Health [READS] {1}

SLOs define whether the service is meeting its reliability target; operational signals help explain why it is or is not. READS is a practical synthesis inspired by established models such as RED and Google's Golden Signals: Request Rate, Errors, Availability, Duration, and Saturation. Together, they connect user-visible health with the conditions inside the system.

Request Rate — How Much Traffic? [Request] {1}

Measure the rate of incoming work: requests, jobs, or messages. Traffic gives errors, latency, and saturation their context, while sudden changes can reveal problems before they appear elsewhere.

Errors — How Much Is Failing? [Errors] {1}

Measure the proportion of work that fails, including errors, timeouts, and incorrect responses. Prefer ratios over raw counts so failures remain meaningful as traffic changes.

Availability — Is It There at All? [Availability] {1}

Errors describe failed work; availability summarizes whether the service can still fulfill its primary function. Measure availability from the caller's perspective: a request that fails or times out is unavailable to the user even if the underlying process is running. Prefer request-based availability over simple uptime when partial failures are possible: a service can be running while a meaningful fraction of users still cannot complete their requests.

Duration — How Long Does It Take? [Duration] {1}

Measure how long work takes to complete and look at the distribution rather than only the average. Percentiles expose slow user experiences that averages can hide.

Saturation — How Close Are We to a Limit? [Saturation] {1}

Measure how close the service is to exhausting a constrained resource such as CPU, memory, connections, queues, or disk I/O. Saturation often provides warning before users experience failures.

Operate the System [Operate] {1}

Operational signals matter only when they inform a clear decision or action. Runbooks, on-call practices, safe production access, and toil reduction turn observation into reliable daily work.

Runbooks [Runbooks]

A runbook is the documented procedure for handling a known operational situation — a deployment, a recovery, a routine maintenance task, a frequent alert. Runbooks transform tribal knowledge into a documented, repeatable procedure. Without them, every incident is solved from scratch by whoever happens to be on call.

A Runbook Is a Promise to Your 3 AM Self [3AM Promise] {1}

A runbook is a letter you write to a future, exhausted, half-awake version of yourself — or to a colleague who has never seen this system before. The promise is simple: !!you should not have to reconstruct the procedure from memory — you should be able to follow a known path!!. A good runbook lists the symptoms, the verifications, the safe mitigation, and the escalation path. A runbook also exposes cognitive load: recurring steps are signals that some operational work may eventually be automated or designed away.

From Tribal Knowledge to Written Procedure [Tribal to Written]

Every team has a senior engineer who "just knows" how to fix the gnarliest production issues. That knowledge is a liability disguised as an asset. The day they leave — or the day they happen to be on vacation when the system breaks — the team rediscovers everything painfully and publicly. Runbooks turn individual operational knowledge into shared procedures, and shared procedures into repeatable systems. Critical operational knowledge should not remain dependent on one experienced engineer; it should be encoded into documentation, tooling, and automation.

The Test of a Good Runbook [Runbook Test]

A runbook should be usable under pressure, not merely readable. Have someone other than its author follow it against a real or simulated situation. Ambiguous steps, missing prerequisites, unsafe actions, or undocumented knowledge are failures in the runbook. Operational procedures should be exercised before they are needed under pressure.

Paging and On-Call [On-Call] {1}

On-call reveals the operational cost of shortcuts and deferred work, often at the worst possible hour. Designing it sustainably is an engineering concern: it determines whether a team can operate a system over the long term.

What Deserves a Page? [Page Discipline]

A page is a contract: it says "drop everything, this needs a human right now." Page when immediate human action is necessary to prevent or reduce meaningful service impact. Conditions that can wait, self-heal safely, or require no immediate decision belong in a ticket, dashboard, or asynchronous review.

On-Call Exposes Operational Debt [On-Call Tax] {1}

Repeated pages are feedback about the system's design and operational maturity: a missing retry, a misconfigured threshold, an automatable runbook, or an insufficiently resilient dependency. Page frequency, severity, and recurrence reveal the operational burden the system places on its team. When recurring pages also consume the service's error budget, the signals reinforce each other: user reliability is deteriorating while operational debt is consuming human attention. That is a strong reason to prioritize fixing the underlying cause. Reducing it usually means fixing the underlying design, not simply adding more responders.

Pager Fatigue Degrades Reliability [Pager Fatigue] {1}

Frequent night pages do not make a team better at responding; they erode judgment and attention. Sustainable on-call is a precondition for reliable software.

Design a Sustainable Rotation [Healthy Rotation]

A healthy rotation keeps paging rare, distributes the burden fairly, provides recovery after disruptive shifts, and gives the team enough capacity to eliminate recurring causes. Respecting the human cost is part of operational maturity.

Safe Production Debugging [Prod Debug]

Production contains conditions that pre-production environments cannot fully reproduce. Operability means making that environment inspectable without making investigation itself dangerous.

Make Production Safe to Explore [Prod Not Enemy] {1}

The instinct to keep engineers far from production comes from a good place — fear of damage — but it can leave a team unable to debug the environment that matters. Make production access safe by default with read-only defaults, audit trails, correlation IDs, scoped diagnostics, and controlled reversible actions.

Debug from Evidence, Not Guesswork [Evidence Debugging]

Start with evidence that can connect an outcome to a change: traces, logs, metrics, deployment history, configuration changes, and the caller's experience. Correlation IDs and time-bounded queries make that evidence usable under pressure. Good production debugging narrows the search space without turning an investigation into an untracked experiment.

Toil Reduction [Toil]

Repeated operational work should not become a permanent dependency on human attention. Toil reduction turns those recurring tasks into an engineering backlog.

What Is Toil? [Toil Definition] {1}

Toil is operational work that is manual, repetitive, automatable, and provides little enduring value. It often scales with service growth: restarting a flaky service, repeating configuration changes, or manually copying diagnostic information between systems.

Put a Ceiling on Toil [Toil Ceiling]

Set an explicit limit on the share of engineering time spent on toil, then treat sustained breaches as a signal to invest in elimination at the source. Google SRE famously uses 50% as an upper bound for toil in some SRE roles, not as a universal law.

Eliminate Toil at the Source [Automate Pain]

Target repeated work that produces little learning: manual restarts, bespoke certificate renewals, and recurring pages from the same alert. Prefer eliminating the need for the work over merely automating its execution. Then verify that the work has disappeared rather than merely moved to another queue.

Day 2 Operations [Day 2] {1}

A production system must evolve while it is running. Certificates expire, dependencies change, data moves, capacity grows, infrastructure is replaced, and exceptional maintenance eventually becomes necessary. Day 2 operations are the controlled changes and actions that keep a live system healthy throughout that evolution. They should be designed as first-class operational capabilities, not improvised when the need appears.

Make Day 2 Operations First-Class Capabilities [First-Class Operations] {1}

Production operations such as migrations, rotations, resets, failovers, repairs, diagnostics, and infrastructure changes should have explicit, supported workflows. !!Encoding operational intent as a defined operation separates what needs to happen from how a particular tool executes it!!, allowing the implementation to evolve without forcing every service owner to redesign the operation.

Operate Production Without Direct Access [Controlled Access] {1}

Routine production operations should not require engineers to log into hosts, clusters, databases, or cloud accounts and manipulate them directly. !!Expose supported operations through controlled interfaces with scoped permissions and known behavior!!. Direct production access should be the exception, not the operating model.

Match Authority to Operational Risk [Risk-Based Authority] {1}

Not every production operation deserves the same level of authority. Low-risk, well-understood operations with bounded impact can be pre-authorized or automated, while destructive, irreversible, or data-sensitive operations should require stronger controls and explicit approval. !!The execution path should reflect the potential blast radius of the operation!!.

Make Operational Workflows Safe by Construction [Safe Operations] {1}

Day 2 operations modify live systems and can have significant consequences. !!Build safety into the workflow!!: validate inputs, constrain permissions, make execution idempotent where possible, preserve durable state for long-running operations, define retry and recovery behavior, and record who requested the operation, what was authorized, what actually executed, and what outcome followed. Safety and traceability should come from the operational platform, not from an operator remembering every precaution.

Exercise Operations Before You Need Them [Exercise Operations] {1}

An operational workflow that exists but has never been exercised is only a hypothesis. Regularly test migrations, rotations, failovers, recovery procedures, and other critical Day 2 operations before they become urgent. !!Verify not only that they start, but that they complete safely, recover from interruption, and still work as the surrounding system evolves!!.

Respond to Failure [Respond] {1}

When a system fails in production, the next minutes determine the cost. Incident response is the choreography of detection, declaration, mitigation, and communication that turns chaos into a controlled operation.

Declare Before You Are Certain [Declare Early] {1}

A common failure in incident response is waiting too long to declare. Engineers want to be sure before raising the alarm, and "sure" can take an hour the system does not have. !!Declaring an incident is cheap; under-declaring is expensive!!. The right reflex is to declare on suspicion, organize the response, and stand down quickly if it turns out to be nothing. Standing down a false alarm is cheap; discovering customer impact late is not.

Stabilize First, Diagnose Second [Stabilize First] {1}

During an incident, reducing user impact comes before perfect understanding. Prefer the safest known action that can reduce impact quickly: rollback, disable a feature, fail over, shed load, or restore a known-good state. Deeper diagnosis matters, but it belongs after the system is stable enough for careful investigation. !!Mitigation buys the time that diagnosis needs!!.

Establish Clear Incident Command [Commander]

Under stress, a group of competent engineers can produce remarkable confusion: parallel investigations, contradictory mitigations, no one talking to the customer. The Incident Commander role creates the coordination structure needed to avoid this. The commander's primary responsibility is coordination: assigning roles, maintaining the timeline, escalating when needed, and protecting responders from distraction. This role transforms incident response from improvisation to operation.

Communicate on a Predictable Cadence [Comms]

While responders stabilize the system, someone must keep affected stakeholders informed. Customers, internal stakeholders, support teams, and executives need a single, calm update on an agreed cadence appropriate to the incident. Silence lets imaginations fill the gap with worst cases. State what is known, what is not, and when the next update will come, even when there is no news.

Learn and Improve [Learn] {1}

Postmortems and Learning [Postmortems] {1}

The incident is the price you paid. The postmortem is what you get in return. A team that does not learn systematically from incidents pays the price twice — once for the original failure, and again when the same systemic weakness contributes to another failure. The discipline of structured learning from failure turns incidents into improvements rather than isolated events.

Blameless Postmortems [Blameless] {1}

A blameless postmortem starts by understanding why the actions taken made sense in their context rather than searching for someone to punish. The question shifts from "who made the mistake?" to "what conditions allowed this failure, and what would make the system safer next time?" A blame-oriented process encourages defensive reporting and hides useful context; blamelessness makes complete evidence and systemic improvements more likely.

Look Beyond the Immediate Cause [Five Whys]

"The database went down" describes what happened, not why the system allowed it to happen. An expired certificate may explain the outage, but missing expiry tracking, absent automation, or persistent firefighting may explain why the certificate could expire unnoticed. Techniques such as the Five Whys help push the investigation beyond the immediate trigger toward weaknesses in the system, process, and tooling.

Action Items That Actually Happen [Action Items]

The point of a postmortem is not the document — it is the change. Postmortem action items can easily die in the backlog: too vague, too unowned, too large, or too far from current priorities. Good action items are specific, owned, prioritized, and tracked through completion. A postmortem that identifies needed changes but never executes them produces learning without improvement.

Operability in the AI Era [The AI Era] {1}

Operability is one of the disciplines most likely to be transformed by AI agents. Runbooks, on-call response, incident triage, postmortem drafting, and toil automation are all areas where AI can remove substantial mechanical work. They are also areas where poorly bounded authority can create significant operational risk.

The Investigation Bottleneck [Investigation Bottleneck] {1}

Detection is increasingly automated; investigation still requires humans to assemble context. After an alert fires, an engineer jumps between traces, metrics, logs, deployments, configuration changes, and previous incidents to understand what happened. The bottleneck is often not detecting the failure, but connecting the evidence that explains it. AI can reduce much of that context-gathering work by correlating evidence across operational systems.

From Runbooks to Operational Agents [AI Runbooks] {1}

A traditional runbook tells an engineer what to do. An operational agent can combine that procedure with the current state of the system — alerts, telemetry, recent changes, and previous incidents — and propose a plausible next step using the runbook as operational guidance and the current system state as context. The runbook does not disappear; it provides operational guidance, while policy and tooling define what the agent is actually allowed to do.

Agent-Assisted Mitigation [Auto-Mitigation] {1}

Operational agents can execute mitigations when the action is bounded, pre-authorized, observable, and recoverable. They might restart a known unhealthy worker, invoke an approved failover, or roll back through an established workflow. Every action should be disclosed and leave evidence behind. Automation should reduce incident impact without making recurring failures invisible.

Consequential Actions Need Explicit Guardrails [Scripts for Destroy] {1}

AI can reason probabilistically, but consequential actions need explicit boundaries. Deleting data, destroying infrastructure, rewriting history, or making other hard-to-reverse changes should pass through predefined, validated, bounded, and auditable workflows. The agent may select or invoke an approved operation, but its authority should be constrained independently of what the model decides.

The On-Call of the Future [Future On-Call]

The human role in on-call will move away from collecting evidence and typing routine commands toward judging intent, risk, and exceptions. Agents can assemble context, run bounded diagnostics, and execute approved mitigations. Humans remain responsible for decisions that fall outside the agent's explicit authority or require broader judgment.

The future of on-call is not the absence of humans. It is a smaller operational surface that requires their direct attention.

AI-Drafted Postmortems [AI Postmortems]

AI can reconstruct an incident timeline from alerts, chat, deployment history, and operational evidence, then produce a first postmortem draft. Engineers validate the narrative, challenge proposed causes, and decide which changes should follow. AI assembles the evidence; humans own the learning.

The goal of operability is not to remove humans from operations. It is to move their attention toward judgment, learning, and engineering improvements that make the next incident less likely.