Project 13 — A Pipeline for an AI Application

“One who is faithful in a very little is also faithful in much, and one who is dishonest in a very little is also dishonest in much.” — Luke 16:10 (ESV)

Chapter: 13 — DevOps, SRE, and Work That Lasts Due: End of Week 13 Submit: A link to a public GitHub repository containing the AI application source, the CI/CD pipeline (.github/workflows/ci.yml or equivalent), your eval set and eval runner, the SLO/error-budget config, the drift and quality monitoring, a REPORT.docx, and (required) agent-log.txt. See Appendix A for the toolchain and git workflow, Appendix B for pointing your code at a local or cloud model, and Appendix C for the agentic-AI rules. Allowed tools: Python 3, your CI provider (GitHub Actions or equivalent), a container runtime, any OpenAI-compatible model endpoint (a local vLLM/Ollama server from Chapter 7, or a cloud API from Chapter 12), an eval/drift library if you want (Evidently, RAGAS, MLflow) or hand-rolled math, git, the textbook. AI — Phase 2 (agentic AI is ON): You may delegate implementation to a coding agent (GitHub Copilot coding agent, Claude Code, Cursor). An agent-log.txt is REQUIRED — every task you delegated, what the agent built, where it was wrong, and where you intervened. The SLO, the eval rubric, and the rollback-vs-retrain decision are yours. The agent may write pipeline YAML, glue code, and test scaffolding once you’ve specified the targets; it may not decide what “reliable enough” means for a service that answers real people. That number is the architecture, and the architecture is the part of this project an agent cannot do for you.

The Setup

The ministry from your earlier weeks has a new problem, and it’s a good one: their tools work well enough that people now rely on them. The help desk at the affiliated Christian school has stood up a small RAG support bot — it answers staff questions (“how do I reset a student’s account?”, “what’s the snow-day procedure?”) from a folder of school policy documents. It’s genuinely useful. It’s also, right now, a loaded gun.

Because nobody is watching its quality. The server is up, the dashboard is green, and last Tuesday the bot confidently told a new teacher the wrong lockdown procedure — fluent, certain, and pulled from nothing in the policy docs. A hallucination. No alarm fired, because every classic alarm was watching whether the service was up, and it was always up. The policy documents were also updated for the 2026 school year, and no one knows whether the bot’s answers drifted when they did.

Your job is to build the reliability apparatus this service should have had from day one: a pipeline that cannot ship a regression, a promise about quality with a budget behind it, monitoring that catches a silent slide before a teacher does, and a deploy path that lets you undo a bad model in one practiced motion. The bot answers real people about real safety procedures. A tool that can be confidently wrong, unwatched, is not done.

Setup (the starter)

Lean on this chapter’s code/ files as scaffolding and reference:

• code/ci.yml — a CI pipeline shaped the right way: cheap deterministic gates fail-fast, the LLM eval gate runs last. Adapt it; don’t just copy it.
• code/error_budget.py — turns an SLO + request counts into an error budget and a ship/freeze verdict.
• code/psi.py — Population Stability Index for data-drift monitoring (and a built-in reminder of what it can’t see).
• code/judge.py — a minimal, example-grounded LLM-as-a-judge groundedness evaluator — the gate your pipeline calls.

You supply the RAG service. It can be small: a fixed folder of “policy” documents, a retrieval step, and an OpenAI-compatible model call. If you’d rather not build retrieval, a stub that returns canned answers from a fixed context is acceptable for Normal tier — the reliability apparatus is the graded part, not the cleverness of the bot.

Toolchain — read this before you start. The CI gates here are exactly the ones the course toolkit installs. code/judge.py is now a runnable OpenAI-compatible eval runner (openai SDK; OPENAI_BASE_URL + OPENAI_API_KEY + MODEL_ID from the environment) — it’s the gate code/ci.yml calls. Install ruff, pytest, and gitleaks for the deterministic gates per Appendix A §A.11 (the “P13 (CI gates)” and “secret scan” rows), and let that section’s pip install openai pyyaml plus the ruff/pytest rows be the backbone of your repo’s requirements.txt. Pin your model ID (no latest).

Learning Targets

By completing this project, you will demonstrate that you can:

• Build a CI/CD pipeline for an AI application with the right gate ordering (deterministic gates first, the nondeterministic eval gate last) and prove a quality regression turns the build red.
• Define a quality SLI and SLO for a nondeterministic service, and compute and honor an error budget.
• Monitor for data drift (PSI/KS) and reason correctly about the concept drift those metrics cannot see.
• Run a groundedness/hallucination eval gate, and validate the judge rather than trusting it blindly.
• Deploy safely with shadow → canary → promote and execute a practiced one-command rollback.
• Direct a coding agent through the pipeline build while retaining the SLO, the rubric, and the rollback-vs-retrain decision as human judgment — documented in agent-log.txt.

Normal Tier

Goal: Give the RAG bot a CI/CD pipeline with an eval gate that blocks a quality regression. The pipeline must run deterministic gates first and the LLM eval gate last, and you must prove a regression fails the build.

Required features

1. A working (or stubbed) RAG service with a fixed context corpus and an OpenAI-compatible model call. Pin the model ID in config — no latest.
2. A CI pipeline (adapt code/ci.yml) that runs, in this order: build/install → lint → unit tests → secret scan → LLM eval gate. The expensive eval gate runs last, only on a PR that passed everything free.
3. An eval set of at least 8 fixed cases stored in the repo (JSONL), and an eval runner (adapt code/judge.py) that scores groundedness and fails the build below a threshold you set (e.g. < 0.85). code/judge.py scores each case from two fields — a context (what was retrieved) and an answer (what the bot said) — so those are the two every case must carry. Keep a question per case for your own readability if you like, but the judge ignores it; if you also store an expected answer for hand-labeling (Medium tier, judge validation), know that judge.py doesn’t read it either.
4. Proof of a caught regression. Commit a deliberately regressed version (answers that fabricate facts not in the context) on a branch and show the pipeline turning red at the eval gate — a screenshot or CI log link in REPORT.docx.
5. Least privilege. The CI job’s permissions are scoped (e.g. contents: read); document why an AI coding agent must not have write access to your protected branch.
6. REPORT.docx stating your groundedness threshold and why, plus the regression-caught evidence.

Normal-tier rubric (out of 100)

Medium Tier (+up to 25% extra credit)

Goal: Add the SRE layer — a quality SLO with an error budget — and monitoring that catches drift, plus a safe rollout.

1. Quality SLO + error budget. Define a quality SLO on your groundedness SLI (e.g. “≥99% grounded over the eval set / a rolling window”) and compute the error budget with code/error_budget.py. Show a “healthy” run and a “budget exhausted” run.
2. Drift monitoring. Instrument at least one input feature (e.g. question length, retrieved-chunk count) and compute PSI (code/psi.py) on a reference vs. a current window. Show a calm reading and an alerting reading.
3. The concept-drift writeup. In REPORT.docx, construct the case where your PSI stays calm but the bot is now wrong (the policy docs changed). Name the drift type and state what you’d need to detect it.
4. A safe rollout. Implement shadow deploy or canary for swapping the model/prompt: the challenger scores live (or replayed) traffic without serving it, or serves a small slice, and you promote on a metric, not a vibe.
5. Judge validation. Hand-label a handful of eval cases and compare your labels to the LLM judge’s; report agreement and what you’d do if the judge disagreed often.

Hard Tier (+up to 25% additional extra credit)

Hard tier is graded on judgment an agent cannot supply. Add a RELIABILITY_MEMO.docx (1–2 pages) that does the following, and means it:

1. Sets the SLO and defends the number to a non-engineer. Why 99% and not 99.9% or 95%? What does each “nine” cost to achieve, and what does a missed answer cost the school? An SLO is a value judgment about acceptable harm; make it, and own it.
2. Makes the rollback-vs-retrain call. Present a realistic drift/breach scenario from your monitoring and decide, with the evidence in hand, whether the right move is an immediate rollback to the champion or a retrain — and what you’d tell on-call to do at 2 a.m. Justify it; don’t hedge.
3. Writes a blameless postmortem for the original hallucinated-lockdown-procedure incident: timeline, contributing factors in the system (not the person), and the specific control you added so the next tired person can’t repeat it. Connect at least one fix to a control in your pipeline.
4. States the limits of your own monitoring. What can your apparatus not catch? Where does a human still have to look? Honesty here scores higher than false confidence.

A reader should finish the memo trusting you with the system — not because your pipeline is clever, but because your judgment is sound and you know where it ends.

Submission

Push everything to a public GitHub repo: the service, the pipeline file, the eval set + runner, the SLO/budget and drift config, REPORT.docx, RELIABILITY_MEMO.docx (Hard), and the required agent-log.txt. Include the CI run links/screenshots that prove the regression was caught. Submit the repo URL.

Turning the pipeline on — Appendix A §A.12.5. Dropping ci.yml at .github/workflows/ci.yml makes GitHub run it on every push (free minutes on public repos; watch runs in the Actions tab — a red run is the regression evidence this project asks for). Two things this project lives or dies on:

• Where the model key goes. The eval gate’s API key is an Actions secret, not a .env file: Settings → Secrets and variables → Actions → add OPENAI_API_KEY as a secret, and MODEL_ID / OPENAI_BASE_URL as variables. The workflow reads them as ${{ secrets.OPENAI_API_KEY }} and ${{ vars.MODEL_ID }}.
• A GitHub runner cannot reach your local Ollama. Hosted runners live in the cloud, so the eval gate in CI must point OPENAI_BASE_URL at a cloud OpenAI-compatible endpoint (Appendix B.5), not http://localhost:11434. Local Ollama is fine for developing the judge; it just can’t be what CI calls. This is why both judge.py and ci.yml take the endpoint from the environment rather than hardcoding one vendor. Note that a cloud endpoint needs a little API credit (most are paid); if cost is a concern, the Azure for Students account from Appendix A §A.2.1 gives you $100 of credit with no credit card, and it covers the Chapter 12 cloud-AI services this gate calls.

Use a branch + Pull Request for the regression demo (git checkout -b regression-demo, A.12.3). Per the A.12.6 submission checklist, this is a Phase 2 project, so the required AI artifact is agent-log.txt (not the Phase-1 AI_USAGE.txt).

Hints (Read Before You Begin)

• Build the eval set first, before any pipeline. The eval set is your specification of “good.” If you write it last, you’ll write it to pass.
• Make the gate fail before you make it pass. A green eval gate you’ve never seen go red is proof of nothing. Regress the service on purpose, watch the build go red, then fix it.
• Pin the model ID everywhere. A pipeline that says latest is a pipeline that drifts without anyone choosing it. (This is also the Chapter 14 deprecation lesson arriving a week early.)
• Instrument with OpenTelemetry / OpenInference if you can. Tools churn; the open tracing standard outlives them. Don’t marry a single eval vendor’s SDK.
• Order the pipeline for money, not style. The LLM eval gate costs real tokens and real minutes. Never let it run before the free linter.
• Let the agent build the glue; you own the targets. Delegate the YAML and scaffolding freely. Do not delegate the SLO number, the rubric, or the rollback decision — and log exactly where you drew that line.

What Mastery Looks Like (Beyond the Rubric)

Mastery is when your pipeline is boring. The regression gets caught automatically, the drift alarm fires before a teacher notices, the rollback is one command you’ve run before, and the reliability memo reads like it was written by someone who has been paged at 3 a.m. and learned from it. Anyone can build a service. The master keeps it trustworthy when the model drifts, the docs change, and no one is watching — and can give a clear account of every promise it makes.

Coach’s Note — The most valuable artifact you’ll produce this week is not the pipeline. It’s the RELIABILITY_MEMO.docx — the one document an agent could not have written, because it requires you to decide what “reliable enough” means for real people and to stand behind the number. Spend disproportionate care there. That’s the seat of judgment this whole book is training you for.

When You’re Done

• The RAG service runs and answers from a fixed context; the model ID is pinned.
• The pipeline runs deterministic gates first and the eval gate last.
• A deliberate regression provably turns the build red.
• (Medium) An SLO, error budget, and at least one drift check are in place, with a concept-drift writeup.
• (Medium) A shadow or canary rollout and a judge-validation comparison exist.
• (Hard) RELIABILITY_MEMO.docx sets the SLO, makes the rollback-vs-retrain call, includes a blameless postmortem, and names the limits.
• agent-log.txt records what you delegated, what was wrong, and where you intervened.
• REPORT.docx ties it together with evidence.

A theological footnote. “One who is faithful in a very little is also faithful in much.” This project is almost entirely “a very little”: one more eval case, one honest line in the postmortem, one boring deploy that goes off without incident, one error budget honored when the feature was exciting and the number said no. None of it impresses in a demo. All of it is where trustworthiness actually lives. You are keeping watch over a servant that can be confidently wrong — a model that will hand a hallucination to a real teacher with total fluency and no flicker of doubt — and the watch you keep is not bureaucracy; it is stewardship of work that real people depend on. The pipeline and the budget and the rollback are the small faithfulnesses that compound into a system anyone can trust. And the postmortem is the most Lutheran artifact in the course: it owns the failure honestly, refuses to hunt for someone to condemn, and changes the system so the next finite, tired person is protected. Faithful in a very little. That is what makes work last.

See you next week.