Project 1 — From Runbook to Copilot

“Moreover, it is required of stewards that they be found faithful.” — 1 Corinthians 4:2 (ESV)

Chapter: 1 — The Administrator in the Age of AI Due: End of Week 1 Submit: A link to a public GitHub repository containing your verified runbook, your AI-drafted runbook, your hardened verification tooling, your AI usage policy, and REPORT.docx. See Appendix A for lab and submission mechanics and Appendix B for wiring up your AI copilot. Allowed tools: A Linux lab box (VM or container), bash, systemd, python3, git, and an AI copilot of your choice (a cloud model such as Claude Sonnet 4.6 / Opus 4.8, or a local model via Ollama — see Appendix B). AI policy (Phase 1): AI is used and evaluated, not trusted. You must build the verified runbook by hand first, then bring in the copilot and critically compare. Every deliverable that touches AI carries an honest AI usage note. The human owns the verdict. (No agent-log.txt yet — that begins in Phase 2, Week 9.)

The Setup

You are the lone systems administrator for St. Augustine Lutheran School & Parish, a small LCMS congregation with an attached K–8 school. You run a modest stack: a parish website and giving portal (ministry-web), a student-information app, a file server, and the backups that protect all of it. There is no team. There is no on-call rotation. There is you, a pager, and a principal who expects the giving portal to work on Sunday morning.

Last Sunday it did not. ministry-web threw 502s for forty minutes during the offering, and you spent thirty of those forty minutes remembering how to diagnose it. You have decided two things. First, that procedure is going into a verified runbook so next time it takes five minutes, not forty. Second, you are going to evaluate — honestly, with your own eyes — whether the AI copilot everyone keeps recommending can be trusted to help you write and run procedures like this one. You are a steward of this parish’s systems. You will not hand the keys to a tool you have not tested.

Setup (the starter)

This chapter ships starter and reference files in code/:

• code/restart-web-service.sh — a verified, safe restart procedure in the confirm → capture → change → verify shape. Your model for “what good looks like.”
• code/ai-drafted-runbook.txt — an AI first draft with seeded hallucinations. Your comparison target and adversary.
• code/verify_runbook.py — a tiny regex linter that flags mechanical footguns. You will extend it.
• code/copilot_policy.yaml — a starter AI usage policy with an autonomy ladder. You will adapt it.

Fork or copy these into your project repo. Do not submit them unchanged — make them yours.

Learning Targets

You will demonstrate that you can:

• Capture a real operational procedure as a verified, runnable runbook that confirms, captures, changes, and verifies — not one that merely “starts” a service.
• Critically evaluate an AI copilot’s output for hallucinations, naming each error and its class (wrong identifier / invalid directive / absurd value / unsafe action).
• Build mechanical verification (a linter) and know precisely where it stops and human judgment must begin.
• Write a governance artifact — an AI usage policy with an autonomy ladder that names accountable humans — and defend a judgment call within it.

Normal Tier

Goal: Build one verified runbook by hand, get an AI to draft the same runbook, and produce an honest, line-by-line verdict plus a team AI usage policy.

Required features

1. runbook-verified.txt — Pick one real procedure you can actually run on your lab box (restart a service, rotate a log, add a least-privilege user, take a backup). Write it by hand in the confirm → capture → change → verify shape. It must include at least one pre-change confirmation, one state capture, the change, and a health verification that is distinct from “started.”
2. restart-service.sh (or equivalent) — Adapt code/restart-web-service.sh to your procedure. It must run on your lab box and exit non-zero when the post-change health check fails.
3. runbook-ai.txt — Ask a copilot to write the same runbook from a one-line prompt. Save it unedited, with a header naming the tool and model (e.g. “Claude Sonnet 4.6 (claude-sonnet-4-6), mid-2026”).
4. verdict.txt — A line-by-line table comparing the AI draft to your verified runbook. For each differing line: mark verified / wrong / unsafe, give the error class, and record the human’s final call. End with your name as the accountable steward.
5. verify_runbook.py — Run the linter against runbook-ai.txt and capture the output in REPORT.docx.
6. copilot_policy.yaml — Adapt code/copilot_policy.yaml to St. Augustine (or your own context): at least three task→rung mappings changed, and required_in_every_change naming an accountable human.
7. REPORT.docx — 600–900 words: what you built, where the AI helped, where it hallucinated (with examples), and your verdict on whether/where you would trust this copilot. Include the honest AI usage note.

Normal-tier rubric (out of 100)

Medium Tier (+up to 25% extra credit)

Make the verification real.

1. Extend the linter with at least three new, well-justified rules that catch hallucinations the shipped version misses (e.g., invalid Restart= values, world-writable chmod, deletion of evidence directories). Document each rule’s why.
2. Adversarial test. Generate a fresh AI runbook for a different procedure, run your extended linter against it, and report: what it caught, what it missed, and what it false-positived. A linter that flags safe lines is its own failure mode.
3. Two models, one prompt. Draft the same runbook with two different model tiers (e.g. Haiku 4.5 vs Sonnet 4.6, or a local Ollama model vs a cloud model). Compare hallucination rate and note the cost/quality tradeoff as a mid-2026 snapshot.

Hard Tier (+up to 25% additional extra credit)

This is the judgment the machine cannot make for you, and it is graded as such.

Write MEMO.docx (one page, addressed to “the principal / the parish council”) that makes and defends a single recommendation: which one of St. Augustine’s procedures (or your own org’s) should be allowed to climb to rung 2 — act-with-approval, and which must never leave rung 1 — suggest. You must:

• State the procedure, the rung, and the specific reversible, bounded properties that justify (or forbid) the climb.
• Name the failure mode you are accepting and the human who would answer for it.
• Tie the recommendation explicitly to the spine rule — the human stays in the loop where the judgment lives — and to at least one concrete control from your copilot_policy.yaml (audit log, second approver, never-delegate list).

A memo that says “automate everything” or “automate nothing” fails. The grade is in the discrimination: showing you can tell a rung-2 task from a rung-1 task and defend the line.

Submission

Push everything to a public GitHub repo:

project-01/
  runbook-verified.txt
  runbook-ai.txt
  verdict.txt
  restart-service.sh
  verify_runbook.py
  copilot_policy.yaml
  REPORT.docx
  MEMO.docx           # Hard tier only

Submit the repo link per Appendix A. Use placeholders for any host names, emails, or secrets — never commit a real credential.

Hints (Read Before You Begin)

• Do the hand version first. If you let the copilot write the verified runbook too, you have nothing to compare it against and the whole point collapses.
• Pick a procedure you can actually run. A runbook you cannot execute is a runbook you cannot verify. A throwaway nginx or a dummy systemd unit on a lab VM is plenty.
• “Started” is not “healthy.” The single most common bug in these runbooks (human or AI) is treating a successful start as success. Make your health check hit a real endpoint or check real output.
• Capture the AI draft before you fix it. The unedited draft is evidence. Editing it destroys the comparison.
• Your linter is not your judgment. It catches mechanical traps so you can spend attention on the subtle ones. Say in REPORT.docx exactly where that line falls.

What Mastery Looks Like (Beyond the Rubric)

A mastered submission is one where I can read your verdict.txt and MEMO.docx and tell that you are in the loop — that you caught the wrong unit name the linter missed, that you can name which procedure is safe to promote and why, and that your AI usage note is honest about where the copilot saved you time and where it nearly cost you. The runbook is table stakes. The governance and the judgment are the project.

Coach’s Note — The temptation this week is to make the AI draft look bad so your verdict looks smart. Resist it. The honest finding is more valuable than the flattering one — copilots genuinely help, and they genuinely hallucinate, and a steward reports both. Your credibility for the next fifteen weeks starts with whether I believe your Week-1 report.

When You’re Done (a short checklist)

• runbook-verified.txt runs as written and verifies health, not just start.
• The script exits non-zero on an unhealthy service (you tested this).
• runbook-ai.txt is unedited and names its tool + model.
• verdict.txt classes every hallucination and records a human call.
• You named at least one error your linter cannot catch.
• copilot_policy.yaml names an accountable person.
• (Hard) MEMO.docx defends one rung-2 promotion and one rung-1 hold.
• No real secrets committed.

A theological footnote. The steward in 1 Corinthians 4:2 is trusted with another’s house and asked for one thing: faithfulness. This project is the smallest possible rehearsal of that trust. The copilot will offer to take the work off your hands; the parish’s data, the family’s giving record, the principal’s Sunday morning are still entrusted to you. To accept a draft is fine. To sign it without reading it is to bury the talent and call it efficiency. Faithfulness here is unglamorous and exact: confirm, capture, change, verify — and put your name on the verdict. That is what it means to be a faithful steward of what you did not make.

See you next week.