Chapter 13 — DevOps, SRE, and Work That Lasts

“Everything fails, all the time.” — Werner Vogels, CTO of Amazon, on why you design for failure instead of pretending it away

“One who is faithful in a very little is also faithful in much, and one who is dishonest in a very little is also dishonest in much.” — Luke 16:10 (ESV)

Why This Matters

For twelve weeks you have built parts. A server. An identity store. A GPU partition. A backup. A cloud deployment with a bill attached. Each was a thing you could finish, point at, and call done.

This week is about the part you can never finish: keeping it all working, week after week, deploy after deploy, while the code changes under you and people sleep through the night trusting it will still be there in the morning. That discipline has two names that overlap. DevOps is the culture and the machinery — automate the path from a commit to production, and make that path so boring that shipping is a non-event. SRE — Site Reliability Engineering, Google’s invention — is the engineering of reliability itself: you decide, in advance and in numbers, how reliable a service must be, you measure whether it is, and you let that number govern whether you’re allowed to ship more or must stop and fix what you have.

Both threads of this book run straight through here, and this week they finally meet the same pipeline. AI as the tool you wield: the coding agent that turns an issue into a pull request overnight, the AI that drafts your tests and reviews your diff and writes the first cut of the postmortem. AI as the workload you run and govern: an AI application — a model behind an API, a RAG service answering real questions — that you now have to deploy, version, and keep reliable. And here is the twist that makes this the hardest reliability problem in the book: that workload is nondeterministic. The same input can yield a different output tomorrow than today. The model can drift — quietly get worse — while every server stays green, every CPU graph stays flat, and every classic health check says “up.” A service can be perfectly available and perfectly wrong.

So this week’s apologetic question is the one the whole craft circles back to: what makes work trustworthy over the long haul? Not impressive in the demo — trustworthy on the Tuesday eight months from now when nobody’s watching and the model has quietly slid two points of accuracy. Jesus’ words in Luke 16:10 are an engineer’s proverb before they are anything else: faithful in a very little, faithful in much. The discipline that keeps a one-line config change from taking down production is the same discipline that keeps a fine-tuned model from rotting in place. Trustworthiness is not a feature you add. It is a thousand small faithfulnesses, automated and measured, that compound.

The spine rule holds, and it bites harder here than anywhere yet. The human stays in the loop where the judgment lives. An AI agent can open the pull request, generate the tests, and propose the rollback. It cannot decide that 99.5% grounded is the right promise to make to a ministry’s volunteers, cannot own the postmortem when the promise breaks, and cannot be the name on the change. AI accelerates the typing and the triage. You own the SLO, the verification, and the account you give when it fails.

13.1 — DevOps Culture: The Path From Commit to Production

Before any tool, DevOps is a refusal. It refuses the wall between the people who write the change (Dev) and the people who run it (Ops) — the wall where a developer throws a release over to operations and washes their hands, and operations gets paged at 3 a.m. for code they never saw. DevOps says: the team that builds it runs it. You feel the pager, so you make the thing pageable as little as possible.

The mechanism that dissolves the wall is the pipeline — Continuous Integration and Continuous Delivery (CI/CD). Every commit triggers an automated path:

The shape of a good test suite is the testing pyramid: a wide base of fast, cheap unit tests; fewer integration tests; a thin top of slow, expensive end-to-end tests. Invert it — lots of slow end-to-end tests, few unit tests — and your pipeline becomes so slow nobody trusts it, so they route around it. The economics of which test to run first is the same right-tool instinct you’ve used all course: the cheapest gate that can catch the failure runs first.

GitOps is the modern expression of the culture: the desired state of your infrastructure lives in Git, and a controller continuously reconciles reality to match. The repo is the single source of truth; a deploy is a merge; a rollback is a git revert. You learned the idempotent, declarative mindset in Chapter 8 with Infrastructure as Code — GitOps is that mindset wired to a pipeline. The payoff is auditability: every change to production is a commit, with an author, a timestamp, a reviewer, and a one-command undo. When something breaks at 3 a.m., “what changed?” has an answer you can read instead of guess.

Two terms get muddled, so pin them. Continuous Integration is the discipline of merging small changes often into a shared branch, each one validated by the automated build-and-test gauntlet — it keeps the codebase always-releasable. Continuous Delivery extends that so any green build can be released to production at the push of a button; Continuous Deployment removes even the button, releasing every green build automatically. Most teams running an AI workload stop deliberately at Delivery — they keep the human button — precisely because the thing being shipped can be confidently wrong in ways a test suite doesn’t fully catch. That choice is the whole book in one configuration setting.

Coach’s Note — A pipeline is a runbook that cannot be skipped. Every “we’ll remember to run the tests” becomes “the tests ran, because they always run.” The whole point of automation here is not speed — it’s that the boring, easily-skipped, faithful-in-a-little steps stop depending on a tired human remembering them at 5 p.m. on a Friday.

13.2 — Releasing Without Breaking: Blue-Green, Canary, Rollback

Shipping is the riskiest thing you do, so the strategy is to ship in a way that limits the blast radius and lets you undo. Three patterns, each one you’ll recognize from earlier chapters:

• Blue-green. Run two identical production environments, blue (live) and green (new). Deploy to green, smoke-test it, then flip the load balancer. If green is bad, flip back — the rollback is instant because blue never went away. The cost is double the infrastructure during the cutover.
• Canary. Route a small slice of real traffic — 1%, then 5%, then 25% — to the new version while watching its metrics. If error rate or latency degrades, stop the rollout and revert before most users ever touched it. The canary is a measured rollout: you don’t promote on a feeling, you promote on numbers staying healthy.
• Rollback. The non-negotiable. Every deploy must be reversible, and reverting must be a practiced, one-command motion — not an improvisation under pressure. A deploy you can’t undo is a deploy you shouldn’t make.

There’s a third pattern worth naming: feature flags. Ship the code dark — present but switched off — and turn it on for a cohort with a config change rather than a deploy. This decouples deploy (the code is on the box) from release (users can reach it), so you can ship at noon and release at midnight, and kill a bad feature in seconds without a redeploy. For an AI app, a flag is often how you gate a new prompt or a new model behind a small audience while you watch its quality.

Hold onto canary and blue-green, because in §13.6 you’ll see that the exact same patterns — under the names shadow deploy and champion/challenger — are how you safely ship a new model. The sysadmin’s release toolkit and the MLOps engineer’s are the same toolkit, and a feature flag is how you route 1% of traffic to a challenger model without redeploying anything.

Coach’s Note — Notice the verb in every one of these patterns is limit or undo, never prevent. Mature operators have made peace with Vogels’ line — everything fails — and stopped trying to ship perfectly. They ship recoverably. The competence isn’t never breaking production; it’s making the breakage small, fast to see, and fast to reverse. That posture, applied to a model that can be subtly wrong, is the difference between an incident and a catastrophe.

13.3 — SRE: SLIs, SLOs, and the Error Budget

SRE makes reliability a number you manage instead of a virtue you hope for. Three terms, in strict order:

• SLI — Service Level Indicator. A measurement of how the service is doing. “The fraction of requests served in under 300 ms.” “The fraction of answers that are grounded.” An SLI is a number you actually compute from telemetry.
• SLO — Service Level Objective. A target for the SLI. “99.5% of requests under 300 ms, measured over 28 days.” This is the promise you make to yourselves and to users.
• SLA — Service Level Agreement. The contractual version of an SLO, with consequences (refunds, penalties) if you miss it. Most internal services have SLOs, not SLAs.

Before the budget, one more SRE staple: the four golden signals Google teaches as the minimum you watch on any service — latency (how long requests take), traffic (how much demand), errors (the rate of failures), and saturation (how full the system is). For a classic web service those map cleanly to metrics you already know. For an AI workload they need a translation, and the translation is the chapter:

Stare at the “errors” row. On a classic service, an error is an error — a 500 is unambiguous. On an AI service, the most dangerous error returns 200 OK: a fluent, confident, wrong answer. Your error signal has to include a quality measurement or it is blind to the failure you most fear. That single row is why this chapter exists.

The genius move is the error budget. If your SLO is 99.5%, then 0.5% failure is permitted — it’s budget you’re allowed to spend. Over a 28-day window serving 20,000 requests, that’s 100 failures you may have before the promise breaks. This reframes the eternal Dev-vs-Ops fight. Developers want to ship features (which adds risk); operations want stability (which means not shipping). The error budget settles it with arithmetic: while you have budget, ship; when the budget is spent, stop shipping and stabilize. No one has to win the argument. The number decides. code/error_budget.py computes exactly this — feed it an SLO, the request count, and the bad count, and it tells on-call whether they may keep shipping or must freeze.

# 0.5% of 20,000 = 100 allowed failures; we've had 140. Budget blown.
python error_budget.py --slo 0.995 --total 20000 --bad 140
# VERDICT: BUDGET EXHAUSTED -> freeze releases, stabilize, consider rollback/retrain.

Two more SRE load-bearing ideas:

• Toil is manual, repetitive, automatable work that scales with the size of the service and produces no lasting value — the same restart, the same disk-cleanup, the same ticket, over and over. SRE treats toil as a measured enemy: budget a fraction of the team’s time (Google’s convention is to cap it near 50%) to automating toil away. AI is the most powerful toil-killer to arrive in a decade — and §13.5 is where we hand it the dangerous parts carefully.
• Postmortems are blameless. When something breaks, you write up what happened, why, and what will change — focused on the system and process that let a human error become an outage, not on the human. The goal is a system where the next tired person at 3 a.m. cannot make the same mistake. (The apologetic section returns to this; blamelessness is a deeply theological posture.)

Coach’s Note — The error budget is the most important idea in this chapter and the one students under-use. It converts “are we being responsible?” — a question people argue about forever — into “do we have budget?” — a question telemetry answers. When an AI agent wants to auto-deploy, the error budget is the gate that tells it no, not this week. The number you set is a judgment only a human can make. The number you enforce is something a machine does tirelessly. That division of labor is the thesis of this book.

13.4 — On-Call, Toil, and the Cost of Keeping Watch

Someone carries the pager. When the SLO is at risk, an alert fires and a human responds. Good on-call is engineered, not endured: alerts must be actionable (every page corresponds to something a human can and must do now), tied to user-facing symptoms (alert on “answers are slow,” not “CPU is 80%” — CPU at 80% may be fine), and rare enough that responders aren’t numb. Alert fatigue — too many pages, most of them noise — is itself an outage risk, because the one real page gets lost in the false ones.

The single most useful alerting refinement for an SLO-driven service is burn-rate alerting. Instead of paging the instant any one request fails, you page when the error budget is being consumed too fast to last the window. A fast burn (you’ll exhaust a month’s budget in an hour) is an emergency and pages immediately; a slow burn (you’ll exhaust it in two weeks) is a ticket, not a 2 a.m. wake-up. This is how you alert on the promise, not on individual blips — and it maps directly onto the AI workload: page on “groundedness is collapsing fast enough to blow the month’s budget,” not on a single hallucination, which the error budget already accounts for. code/error_budget.py computes the burn so you can see the difference between “watch it” and “wake someone up.”

This is exactly where the AIOps agents from Chapter 9 earn their keep. As of 2026, nearly every major observability vendor ships a semi-autonomous SRE agent that reads telemetry and runbooks and posts a root-cause hypothesis to chat before a human even logs in — Datadog Bits AI SRE (GA), New Relic’s SRE Agent (preview, and explicitly recommend-only — it does not make production changes or bypass approvals), PagerDuty’s Advance SRE Agent, Microsoft’s Azure SRE Agent (GA, with a read-only Reader mode and a separate privileged mode). The agent triages; the human decides. That is the same recommend-don’t-act gating you’ll meet again as a design principle in Chapter 15.

Two cautions, because the AIOps agent is itself a workload you now pay for and govern. First, the billing is real and metered in ways that surprise teams: Azure’s SRE Agent bills in Azure Agent Units (AAU) and an alert storm that triggers many investigations can spend more than the incident cost — agentic triage is not free, and a runaway agent investigating a noisy alert is a cost incident of its own. Second, the vendor “X% faster resolution” numbers (New Relic’s report of roughly 27 minutes/issue versus 50, for instance) are marketing, not independent benchmarks; treat them as directional, not as the spec you build to. The agent that reads your runbook is genuinely useful and genuinely a thing you must size, budget, and keep on a leash.

13.5 — AI as the Tool: The Pipeline That Writes and Reviews Itself

Now the first AI thread, full strength. As of 2026, AI coding agents have moved from autocomplete to asynchronous teammate. You assign an issue; the agent — GitHub’s Copilot coding agent, Claude Code, Cursor — spins up a sandboxed environment, writes the change, and opens a pull request while you do other work. The agent generates unit tests. It drafts the diff. It writes the first version of the postmortem. It will even propose the rollback.

Here is where AI shows up across the DevOps lifecycle, and the verification each use demands:

Every row is a force multiplier. Every row is also a place a confident-wrong partner can lead you astray faster than you can catch it. The discipline is the same one the whole book teaches, applied to your own pipeline.

This is a real force multiplier and a real new attack surface, so it slots into the pipeline under one disciplined pattern that you should be able to recite: AI proposes, CI validates, the human gates.

• AI proposes. The agent’s work lands as a pull request on a non-protected branch. GitHub’s Copilot coding agent, for instance, is scoped to copilot/* branches — it cannot push to main or any protected branch. The change is a proposal, not a fait accompli.
• CI validates. Every agent-authored PR runs the same gauntlet a human’s would, and then some: CodeQL static analysis and secret scanning run automatically before review. AI-written code leaks API keys and ships SQL injection exactly like human-written code — the deterministic gates don’t care who typed it. This is the cheap, fast, certain layer catching what it can catch for free.
• The human gates. A person reviews and approves the merge. Not a rubber stamp — an actual read. The agent is a confident, fast, sometimes-wrong partner; the reviewer is the judgment.

code/ci.yml is a pipeline built to this shape. Notice the order: lint, unit tests, and secret scan — cheap, deterministic, fail-fast — run first; the slow, paid, nondeterministic LLM eval gate runs last, only on a PR that already passed everything free.

permissions:
  contents: read          # least privilege: this job cannot push to main

That one line is the whole §13.5 security lesson. The two OWASP-LLM risks you met in Chapter 8 govern any AI in your pipeline: Improper Output Handling (LLM05) — never feed an LLM’s output into a shell, an apply step, or a database without validation — and Excessive Agency (LLM06) — never grant the agent more permission than the task needs. A coding agent with write access to main and the ability to trigger a production deploy is a prompt injection away from being your adversary. Scope it down. Make it propose. Keep the gate human.

Coach’s Note — “AI wrote the tests” is a sentence that should make you more careful, not less. A model that writes both the code and the tests for that code can write tests that pass for the wrong reason — tests that assert the bug. Read AI-generated tests harder than AI-generated code. The test is your specification; if the agent writes the spec and the implementation, no one has checked the spec against reality but you.

13.6 — AI as the Workload: Reliability for a Service That Can Be Confidently Wrong

Here is the hardest reliability problem in the book, and the reason this chapter exists.

Everything you know about reliability assumes determinism: the same input gives the same output, so a test that passed yesterday passes today unless the code changed. An AI application breaks that assumption at the root. The same prompt can return different text run to run. Worse, the model’s quality can degrade with nothing in your code changing at all — because the world the model sees has changed. This is drift, and it comes in two flavors you must never conflate:

• Data drift (feature drift): the input distribution shifts. Your support bot was validated on questions about the 2025 product; now users ask about the 2026 product. The inputs moved off the distribution the model was good at.
• Concept drift: the input-to-output relationship shifts. “What is a reasonable response time?” meant one thing before the new SLA and another after, with identical inputs. The right answer changed even though the questions didn’t.

The standard detector for data drift is the Population Stability Index (PSI), with these rules of thumb: PSI < 0.1 no significant shift, 0.1–0.2 moderate, > 0.2 significant. code/psi.py computes it. Other statistical tests — the Kolmogorov–Smirnov (KS) test, Jensen–Shannon divergence, chi-square — do related jobs, and tools like Evidently, NannyML, and Alibi Detect package them. But here is the trap, and it is on the exam: PSI detects data drift only. It is blind to concept drift. Concept drift requires labels — actual outcomes — to detect, and those arrive late. Monitoring only the inputs is watching one of two doors.

A note on why AI services drift in ways traditional software doesn’t. Conventional code is static — it does tomorrow exactly what it did today unless someone edits it. An AI application has three moving parts that can each shift the output with zero code changes: the input distribution (users ask new things), the world the model was trained to describe (facts and policies change underneath a frozen model), and — for hosted models — the model itself, when a provider silently updates the endpoint behind a name. That last one is the strongest argument in the book for pinning a model ID, never a marketing name or latest. A version you didn’t choose is a deploy you didn’t make and can’t roll back.

For an LLM, the failure you fear most isn’t a distribution shift you can chart — it’s a hallucination: a fluent, confident, wrong answer. You can’t unit-test that away, because the same prompt may be right today and wrong tomorrow, and “wrong” is a judgment about meaning, not a string comparison. Quality monitoring for generative AI is therefore layered, and no single layer is sufficient:

code/judge.py is a minimal LLM-as-a-judge groundedness evaluator — the gate ci.yml calls. Read its docstring: the rubric is explicit and example-grounded, not “rate this 1–10,” and it warns you to validate the judge against human labels and watch its own scores for drift. A judge you don’t audit is just a second model you’ve decided to trust on faith.

The tooling here is real and worth knowing by name, with the caveat that it churns. MLflow (3.x, with tracing and evals) is the common backbone for tracking model versions, runs, and eval results. For LLM/RAG evaluation specifically: RAGAS scores groundedness and faithfulness; Arize Phoenix, Langfuse, and LangSmith capture traces and run LLM-as-a-judge evaluators over them. For drift, Evidently, NannyML, and Alibi Detect package PSI, KS, and friends. You don’t need all of them; you need to understand the jobs — version tracking, tracing, eval, drift — and pick one tool per job that’s healthy and standards-based. Because, again: tools are temporary. The job and the discipline last.

Now apply SRE to all this. Your SLI is no longer “is it up” — it’s a quality rate: grounded answers ÷ total answers, computed continuously over a window. Your SLO is a promise on that rate: “99.5% of answers grounded over 28 days.” Your error budget is the permitted hallucinations. When drift burns the budget, the standard model lifecycle gives you the safe path to a fix — and you will notice it is the §13.2 release toolkit wearing MLOps clothes:

train → evaluate (gate) → shadow deploy → canary → full production → monitor → retire

• Shadow deploy = the new model (the challenger) scores live traffic, but its outputs are only logged, never served. This is dark-launch / traffic-mirroring — the new release runs in the dark while the old one answers users.
• Canary = route 1% of real traffic to the challenger, watch its quality SLI, then ramp. Identical to §13.2’s canary.
• Champion/challenger = promote the challenger to champion only if it scores significantly better on the eval set. This is blue-green with A/B scoring bolted on.

And the connective tissue underneath all of it is OpenTelemetry (with the emerging GenAI semantic conventions / OpenInference). Model telemetry — latency, token throughput, GPU utilization, groundedness scores — converges onto the same tracing backbone as your app and infrastructure telemetry. One observability spine for the whole system. Bet on that open standard, because the specific “best” eval tool churns fast: in roughly the last year (as of 2026) WhyLabs wound down, Helicone moved to maintenance mode, and BentoML was acquired — a live demonstration of this chapter’s thesis that tools are temporary and disciplines last.

Coach’s Note — Reread that lifecycle and notice you already knew it. Shadow is a dark launch. Canary is a weighted rollout. Champion/challenger is blue-green with a scoreboard. You did not learn a new bag of tricks for AI — you learned that the reliability craft you’ve built all course transfers, with one addition: the thing you’re rolling out can be confidently wrong, so the metric you gate on is quality, not just uptime.

13.x — Interactive Lab: Model Drift Monitor

Below this chapter on the site is an interactive panel called the Model Drift Monitor. Open it now — reading about drift is not the same as watching it breach an SLO in front of you.

The widget simulates one AI service running over time. You set the SLO (say, 99.5% grounded), the error-budget window, and the alert thresholds. Then it plays the service forward day by day, drawing three signals — accuracy/groundedness, hallucination rate, and latency — against your thresholds, while a live error-budget meter drains. Watch what happens when drift creeps in: the model gets quietly worse, the hallucination line climbs, and the budget burns down until it breaches. At the breach, the panel forces the decision SRE forces in real life — roll back to the champion, or trigger a retrain? — and shows you the consequence of each.

Tune the alerts and you’ll feel the on-call tension from §13.4 directly. Set the thresholds too tight and you drown in false pages (alert fatigue); too loose and the breach is a surprise. The lab teaches three things at once: that an AI service can fail silently while every CPU graph stays flat, that the error budget converts that slow failure into a clear decision, and that you — not the dashboard — are the one who decides whether a degraded model keeps serving real users. That last decision is the human-in-the-loop, and the widget will not make it for you.

13.7 — The Steward and the Servant: Faithful in a Very Little

Return to Luke 16:10. “One who is faithful in a very little is also faithful in much.” It sits, in Luke’s Gospel, inside a parable about a manager — a steward — entrusted with his master’s affairs, and the point is about how a small thing reveals the shape of a large one. The engineer’s life is almost entirely “a very little”: one more test, one more line of the SLO doc, one more honest entry in the postmortem, one more boring deploy that goes off without incident. None of it is glamorous. All of it is where trustworthiness actually lives.

This is the answer to the week’s question — what makes work trustworthy over the long haul? Not the heroic 3 a.m. save. The heroic save is usually evidence that a thousand small faithfulnesses were skipped earlier. Trustworthy work is unheroic: the pipeline that always runs the tests, the rollback that’s been practiced so it’s boring, the drift monitor that catches the slide before a user does, the error budget honored even when the feature is exciting and the budget says no. Faithful in a very little, every day, is what produces a system anyone can trust in much.

The blameless postmortem is where this gets surprisingly Lutheran. It rests on a doctrine of human limitation that an LCMS reader will recognize immediately: people are finite and fallible, they will err, and a wise system is built knowing this rather than pretending it away. Blamelessness isn’t softness — it’s an honest anthropology applied to engineering. You stop asking “who is the bad person?” and start asking “what in our system let an ordinary tired person cause an outage, and how do we change it?” That is grace and structure at once: the person is not condemned, and the system is genuinely fixed so the next person is protected. A culture that hunts for someone to blame learns nothing and hides everything; a culture that owns the failure honestly and changes the system is the engineering image of confession and amendment of life.

And the AI workload sharpens the stewardship to a fine point. You are now keeping watch over a servant that can be confidently wrong — a model that will hand a hallucination to a real person with total fluency and no flicker of doubt. The drift monitor, the eval gate, the human-reviewed rollback: these are not bureaucracy. They are the watch you keep because you are accountable for what the servant does in your name. A computer cannot give an account of its management. You can. The whole apparatus of this chapter exists so that when the model fails — and Vogels is right, everything fails, all the time — a faithful steward is standing in the loop, having built the system so that the failure is small, seen early, and answered for honestly.

13.8 — Common Pitfalls

Pitfall: Treating an AI service’s reliability as “is it up?” Example: The model API returns 200 OK for every request, the dashboard is all green, and meanwhile the model has drifted two points and is hallucinating for 4% of users. Fix: Define a quality SLI (groundedness, accuracy) and an SLO on it. “Available” and “correct” are different promises; measure both.

Pitfall: Monitoring only data drift and calling it covered. Example: PSI on every input feature reads under 0.1, so the team relaxes — but the right answers changed (a new policy), and the model is confidently giving the old ones. Concept drift, invisible to PSI. Fix: Pair input-distribution monitoring (PSI/KS) with outcome-based evaluation that uses labels. Know that concept drift needs ground truth and arrives late; design for that lag.

Pitfall: Trusting the LLM-as-a-judge without validating it. Example: A groundedness judge scores everything 0.9+, the eval gate stays green, and three weeks later you discover the judge itself drifted and was rubber-stamping hallucinations. Fix: Validate the judge against a human-labeled set before trusting it, ground its rubric in concrete examples, and monitor the judge’s scores for drift like any other model.

Pitfall: Giving an AI coding agent more permission than the task needs. Example: The agent has write access to main and can trigger a production deploy “to move fast.” A prompt injection in an issue comment turns it into your adversary. Fix: Least privilege (OWASP LLM06). Scope agents to non-protected branches, gate every merge on a human review, and let CI (CodeQL, secret scanning) validate every PR regardless of author.

Pitfall: An inverted testing pyramid — and a slow, paid eval gate that runs first. Example: CI runs a 12-minute LLM eval before the 4-second linter, so every trivial PR costs twelve minutes and real money, and developers start skipping the pipeline. Fix: Cheap, deterministic, fail-fast gates first (lint, unit, secret scan); the slow nondeterministic eval gate last, only on PRs that already passed everything free. See code/ci.yml.

Pitfall: A deploy you can’t undo. Example: A model swap with no champion kept warm and no traffic-routing path back; when the challenger turns out worse in production, “rollback” becomes a frantic re-deploy under pressure. Fix: Keep the champion warm. Make rollback a practiced one-command motion. Use shadow/canary so the challenger proves itself on a slice before it ever owns 100% of traffic.

Pitfall: Betting the architecture on this quarter’s “best” eval tool. Example: The whole pipeline is wired to a single vendor’s SDK; the vendor is acquired or moves to maintenance mode (this happened repeatedly in 2025–2026), and the integration rots. Fix: Instrument with the open standard — OpenTelemetry / OpenInference, the OpenAI-compatible API — and treat specific eval vendors as swappable. Disciplines last; tools are temporary.

13.9 — Reps

Conditioning lives in the exercises. This week’s reps build the SRE-for-AI muscle by hand before you let an agent near the project — you can’t review a pipeline or an eval gate you’ve never typed. A preview:

• Compute an error budget from an SLO and decide, from the number alone, whether on-call may keep shipping (code/error_budget.py).
• Run PSI on a reference vs. a shifted window, interpret the 0.1/0.2 thresholds, then construct a case where PSI stays calm while the answers go wrong — concept drift PSI can’t see (code/psi.py).
• Write an explicit, example-grounded groundedness rubric for an LLM-as-a-judge and hand-grade five answers, then argue where the judge would disagree with you.
• Build the code/ci.yml pipeline so the cheap deterministic gates fail fast and the eval gate runs last; prove the ordering by timing a trivial PR.
• Map shadow / canary / champion-challenger onto blue-green and weighted-rollout — write the one-sentence equivalence for each.

AI policy for the reps (Phase 2): agentic AI is on for the course, but these reps are hand-built unless a rep says otherwise. You’re building the judgment that lets you review what an agent produces for the project. A short “Check Your Reps” quiz is embedded on this page — take it before you start, to find the gaps.

13.10 — This Week’s Project

The project is Project 13 — P13: “A Pipeline for an AI Application.” You’ll take a small RAG service and give it the full reliability apparatus this chapter describes: a CI/CD pipeline with deterministic gates and an LLM eval gate, a quality SLO with an error budget, drift and hallucination monitoring, and a safe deploy strategy (shadow → canary → promote) with a practiced rollback.

At a high level: Normal tier builds the pipeline and the eval gate and proves a regression fails the build. Medium tier adds drift/quality monitoring with an error budget and a shadow-vs-canary rollout. Hard tier demands the judgment an agent cannot supply — a written reliability memo that sets the SLO, justifies the number to a non-engineer, and makes the rollback-vs-retrain call with the evidence to back it. As a Phase 2 project, an agent-log.txt is required: what you delegated, what the agent built, where it was wrong, and where you intervened.

13.11 — Coach’s Final Word

You came into this week able to build things and finish them. You leave it able to keep them — which is the harder and more valuable craft, and the one almost nobody teaches well. DevOps gave you the pipeline that makes shipping boring. SRE gave you the error budget that turns “are we being responsible?” into a number telemetry can answer. And the AI workload gave you the chapter’s sharpest edge: a service that can be perfectly available and perfectly wrong, that drifts while every classic alarm stays silent — so you learned to monitor quality, not just uptime, and to keep a human hand on the rollback.

Hold the two threads together one more time, because they are the book in miniature. AI is your tool: it opens the PR, writes the tests, drafts the postmortem, triages the page. AI is your workload: a nondeterministic, drift-prone, confidently-wrong servant you deploy and govern with SLOs and eval gates. And between the tool and the workload stands you — setting the promise, reading the diff, owning the postmortem, deciding when a degraded model stops serving real people. That is not a constraint AI removes. It is the seat of judgment, and it is yours.

Faithful in a very little, faithful in much. The boring deploy, the honest postmortem, the budget you honor when the feature is exciting — those small faithfulnesses are what make work trustworthy over the long haul. Build that way, and you’ll build things that last.

See you next week.

Up next: Read the exercises and put the reps in your hands, then build Project 13 — Project 13: A Pipeline for an AI Application. Lean on Appendix A for the lab toolchain, Appendix B for pointing your code at a local or cloud model, and Appendix C for the agentic-AI rules your agent-log.txt must honor. Then on to Chapter 14 — Maintenance and the Lifecycle of Things, where the question turns from keeping work alive to knowing when it’s time to let it go.