Chapter 13 — Reps

Conditioning, not grading. This week the reps build one muscle: keeping a nondeterministic service trustworthy — error budgets, drift math, eval gates, and the deploy patterns that let you undo a bad release. You build them by hand so you can review the ones an agent writes for the project.

Ground rules:

1. Type every line yourself. No copy-paste of the YAML, the rubrics, or the drift math. The error-budget arithmetic and the PSI thresholds only become instinct if your fingers build them once.
2. Run everything. Run the Python against real numbers. Run the pipeline. Watch each gate pass and — more important — make one fail on purpose. A green pipeline you’ve never seen go red is a pipeline you don’t trust.
3. Predict before you measure. Before you run error_budget.py or psi.py, write down the answer you expect. When the tool disagrees, find out why before moving on. That gap is the rep.
4. AI/agents are ON for Phase 2 — but these reps are hand-built unless a rep explicitly says otherwise. You are training the judgment that lets you review the agent’s pipeline, eval gate, and rollback logic in the project. You cannot review a reliability apparatus you have never built.
5. No “latest.” Anywhere a rep touches a model, pin a model ID. “latest” is how a reliable service drifts without anyone choosing it.

The starter files live in code/. Toolchain (Python 3.12, git, a container runtime) is in Appendix A; pointing code at a local or cloud model is Appendix B.

Reps 1–3: SRE Arithmetic by Hand

Rep 1 — From SLO to Error Budget

Open code/error_budget.py and read it before you run it. Then, predicting each verdict first, run all three:

python error_budget.py --slo 0.995 --total 20000 --bad 80
python error_budget.py --slo 0.995 --total 20000 --bad 140
python error_budget.py --slo 0.99  --total 20000 --bad 140

The third one changes only the SLO from 99.5% to 99% and the verdict flips. In one sentence, explain why a less ambitious promise makes the same 140 failures acceptable — and why that is a real engineering decision, not a cheat.

Rep 2 — Burn Rate and the Freeze Decision

Still in error_budget.py, find a --bad value (for --slo 0.995 --total 20000) that lands exactly on the BURN RATE HIGH verdict but not BUDGET EXHAUSTED. Then write three sentences: what should on-call actually do in the “high burn” state, what they should do in the “exhausted” state, and why those are different responses rather than the same panic. Tie it to the chapter’s rule — while you have budget, ship; when it’s spent, stop.

Rep 3 — Pick an SLI for an AI Service

You are running a RAG support bot for a Christian school’s help desk. Write down, in plain English, three candidate SLIs — and for each, state whether it measures availability or quality. At least one must be a quality SLI (e.g. groundedness rate). Then pick the one you’d put an SLO on and defend it in two sentences. The point: “is it up” is the easy SLI and usually the wrong one to promise on for an AI service.

Reps 4–6: Drift You Can Compute

Rep 4 — Compute PSI

Create two small data files (one value per line) — a reference.txt of, say, 200 numbers clustered around 50, and a current.txt of 200 numbers clustered around 50 as well (barely shifted). Run:

python psi.py reference.txt current.txt

Confirm the verdict reads OK. Now make a shifted.txt clustered around 70 and run it against the same reference. Watch PSI cross 0.2 into ALERT. Write the two PSI values you got and, in one sentence, what the metric is telling you about your model’s inputs.

Rep 5 — The Drift PSI Cannot See

This is the most important rep of the week. Construct — on paper or in a short note — a scenario where PSI stays under 0.1 (calm) while the model is now wrong. (Hint: the inputs look identical to before, but the correct answer changed — a new policy, a new product version, a new SLA.) Name which kind of drift this is, and state precisely why no input-distribution metric like PSI or KS can catch it and what you’d need instead. If you can’t make PSI lie to you here, you haven’t understood the data-vs-concept distinction yet.

Rep 6 — KS vs PSI Over-Triggering

Read the chapter’s note that the Kolmogorov–Smirnov test over-triggers on very large samples. In three sentences, explain to a teammate why a drift alarm that fires constantly once your traffic grows is as useless as one that never fires — and connect it to the §13.4 alert-fatigue lesson. You don’t need to run KS; you need to explain the failure mode.

Reps 7–9: The Eval Gate and the Judge

Rep 7 — Write a Groundedness Rubric

Open code/judge.py and read the RUBRIC. Now write your own rubric for a different quality dimension — say, refusal correctness (did the bot correctly decline a question it shouldn’t answer?). It must be explicit and example-grounded: include one example that should score pass and one that should score fail. Then state, in one sentence, why “rate this answer 1–10” is a worse rubric than a binary pass/fail with a reason.

Rep 8 — Hand-Grade Against the Judge

Take five answers from any small RAG demo (or write five toy CONTEXT/ANSWER pairs yourself, at least two of which sneak in a fact not present in the context). Grade all five by hand with your groundedness rubric first, writing down your verdict and reason. Only then imagine the judge’s verdict. Where would the LLM judge plausibly disagree with you? Write the one case you’re least sure the judge would get right — that case is why you validate the judge against human labels before trusting it.

Rep 9 — Order the Pipeline

Build code/ci.yml in a scratch repo (you can stub the steps with echo so they run). Then deliberately break the ordering: move the LLM eval gate to run first, before lint and unit tests. Trigger a trivial one-line PR and compare wall-clock time and (conceptually) cost between the two orderings. Write one sentence on why “cheap, deterministic, fail-fast gates first” is an economic decision, not a stylistic one, and how a slow pipeline gets routed around by developers.

Reps 10–11: The Deploy Patterns Transfer

Rep 10 — Map the Lifecycle

In a small table, map each MLOps deploy pattern onto the classic sysadmin release pattern it equals, with a one-line justification each:

The chapter gives you the answers; the rep is forcing yourself to write the equivalence in your own words so the transfer sticks. The lesson: you didn’t learn a new bag of tricks for AI — your reliability craft transfers, with “quality” swapped in for “uptime.”

Rep 11 — Practice the Rollback

Take any small service you’ve containerized this course (or a trivial Flask/FastAPI app). Tag two image versions, v1 (good) and v2 (deliberately broken — make it return a wrong answer, not crash, to mirror a bad model). Write a one-command rollback from v2 back to v1 and run it. Time it. Then write two sentences: why a rollback that returns a wrong-but-200 result is the AI-workload version of this drill, and why you keep the champion warm.

Done? One Last Thing.

This capstone rep is the project in miniature — do it before you start P13.

Stand up the smallest possible RAG service you can (or stub one that returns canned answers from a fixed context). Then wire the whole apparatus around it, minimally:

1. A CI pipeline (code/ci.yml) with at least one deterministic gate (a unit test or lint) and the eval gate (code/judge.py-style groundedness over a fixed 5-case eval set), with the cheap gate first.
2. A quality SLO and an error budget computed with code/error_budget.py over your 5 cases.
3. One drift check (code/psi.py) on an input feature like prompt length.
4. A deliberately regressed version of your service (swap in answers that fail groundedness) and a demonstration that your eval gate turns the build red because of it.

Then write a five-sentence note: what your SLO is, why you chose that number, what your eval gate caught, and — the judgment call — whether you’d roll back or retrain if this happened in production. That last sentence is the part of this week an agent cannot do for you. Keep the note; it’s the seed of your project’s reliability memo.

Up next: Project 13 — Project 13: A Pipeline for an AI Application.