Project 2 — Diagnose the Failure Twice

“And he is before all things, and in him all things hold together.” — Colossians 1:17 (ESV)

Chapter: 2 — The Machine Underneath: Operating System Architecture Due: End of Week 2 Submit: A link to a public GitHub repository containing your captured incident evidence, your AI transcript(s), the sizing analysis, and REPORT.docx. Set up your environment with Appendix A (the lab) and your local + cloud AI access with Appendix B. Allowed tools: A real Linux host or VM (and optionally Windows) per Appendix A; ps, top/htop, dmesg, journalctl, iostat, systemctl; a local model runner (Ollama / LM Studio / llama.cpp); one cloud or local LLM for the AI half; the textbook and code/ starters. AI policy — Phase 1 (Weeks 1–8): AI is studied and used, but you do the human diagnosis first. Every incident is diagnosed by hand and written down before you let an AI touch it. The AI is the thing under test, not your partner-in-charge. You own the verdict. There is no agent-log.txt this week (that’s Phase 2) — but there is a mandatory AI usage section in your report naming every model you used and exactly where it helped and where it lied.

The Setup

A small Lutheran relief organization runs a handful of servers: a donor-database box, a file server for case records, and — new this year — a single GPU workstation in the corner of the IT closet that runs a local language model so that volunteer-written case notes never leave the building. The lone administrator is competent but stretched thin, and lately three things have been going wrong on a rotation nobody can pin down: the donor database “randomly” dies overnight, the file server goes “slow” every afternoon, and the new model box keeps “crashing” whenever someone tries a bigger model.

The director, who just read a vendor white paper, wants to buy an AIOps assistant that will “automatically diagnose and fix” these incidents so the administrator can sleep. Your job, as the consultant they trust, is two-fold: diagnose what’s actually happening, and tell them honestly whether an AI should be allowed to act on it on its own. You will do the first by diagnosing a real captured failure twice — once with your own eyes, once with an LLM — and the second by writing the judgment the director’s white paper can’t.

Coherence is the whole game here. Three “random” failures are almost never random; each is one broken agreement in the machine underneath, inheriting a crack everything downstream falls through. Find the keystone. Then decide who is fit to be trusted to find it next time.

Setup (the starter)

This chapter’s code/ folder gives you everything you need even if you can’t reproduce a live failure:

• code/triage.sh / code/triage.ps1 — the four-question OS triage pass; run it to capture your own evidence.
• code/sample-incident.log, code/sample-dmesg.log, code/sample-ps.txt, code/sample-iostat.txt, code/sample-journal.log — pre-captured incidents you may use directly.
• code/vram_sizer.py — the model-sizing worksheet for the Medium tier.
• code/ask_local_model.py — the OpenAI-compatible client, if you want to triage with a local model instead of a cloud one.

You may use the provided sample incidents, or capture your own (better — reproduce an OOM, a crash loop, or an I/O bottleneck on a throwaway VM). Either way, the evidence goes in your repo.

Learning Targets

By completing this project, you will demonstrate that you can:

• Capture and read OS-level incident evidence — a process table, a kernel log, a service state, an I/O snapshot — and diagnose the root cause by hand.
• Identify the single offending line and distinguish a root cause from a symptom (OOM vs leak vs sizing; I/O-bound vs CPU-bound; first-failure vs crash-loop noise).
• Drive an LLM to interpret the same evidence under a cite-and-verify discipline, and catch its failure modes by name (hallucinated cause, misread line, destructive fix).
• Size an AI model as a workload and explain an OOM event as a sizing failure with the params × bytes-per-weight + overhead + KV-cache math.
• Write a defensible recommendation about how much diagnostic authority to grant an AIOps assistant — a judgment a tool cannot make for you.

Normal Tier

Goal: Diagnose one captured incident twice — by hand, then with an LLM — and adjudicate, in a REPORT.docx.

Required features

1. Capture or adopt one incident. Either reproduce a real failure (OOM kill, crash loop, or I/O saturation) on a throwaway VM and capture the evidence with code/triage.sh + dmesg/journalctl, or adopt code/sample-incident.log (+ code/sample-ps.txt). Commit the raw evidence files to the repo unmodified.
2. Diagnose by hand, written down first. Before any AI, produce a hand diagnosis:
   • The root cause (one sentence), and which of the four OS subsystems it lives in (CPU / memory / storage / services).
   • The single offending log line, quoted exactly from the evidence.
   • Your reasoning — which clues (a STAT letter, an anon-rss value, an await column, a first-failure line) led you there.
   • The fix you would actually run, and explicitly: is it a root-cause fix or a symptom patch?
3. Diagnose with an LLM, second. Hand the same raw evidence to a model (cloud or local) using a cite-and-verify prompt that demands quoted log lines, a confidence level, and one ruled-out alternative. Save the model’s full response verbatim to the repo (ai-transcript.txt).
4. Adjudicate. In REPORT.docx, compare the two diagnoses:
   • Where the AI agreed with your hand diagnosis.
   • Where it went wrong, naming the failure mode (hallucinated cause / misread line / destructive or unsafe fix / fabricated log line). Diff every quote the AI gave against the raw file and call out any that aren’t really there.
   • The verified verdict and fix — your final answer, the one you’d put in the runbook.
5. The AI usage section. Name the model and version, where it saved you time, and where you had to override it.

Example shape of the hand diagnosis

ROOT CAUSE:   The model server (PID 8423, ollama) was OOM-killed; the box
              lacked the RAM for the model that was loaded. Subsystem: MEMORY.
OFFENDING LINE: "Out of memory: Killed process 8423 (ollama) ...
                 anon-rss:15994208kB ... oom_score_adj:0"
REASONING:    anon-rss ~16GB at the moment of death matches an FP16 7B
              (~14-16GB). The box has 16GB RAM. It never fit.
FIX:          Root-cause fix, not a restart: run a Q4_K_M quant (~4-5GB) OR
              add RAM. Restarting just OOMs again on the next load.

Normal-tier rubric (out of 100)

Medium Tier (+up to 25% extra credit)

M1. A second incident, of a different subsystem

Diagnose a second incident that lives in a different one of the four subsystems than your first (if your Normal incident was memory, do an I/O or services one). Use code/sample-iostat.txt or code/sample-journal.log, or capture your own. Diagnose it twice the same way. The point is to prove you can tell a CPU problem from a memory problem from a storage problem from a dead service — and that the AI’s failure modes differ by subsystem.

M2. The sizing analysis (turn an OOM into arithmetic)

For your memory incident, write a short SIZING.txt that turns the OOM event into math. Using code/vram_sizer.py and the §2.7 rules:

• Compute the weights memory for the model that (probably) died, at FP16, INT8, and INT4. Drive the worksheet with --params and --quant (e.g. python3 code/vram_sizer.py --params 7 --quant fp16).
• Add framework overhead (~15–20%) and a KV-cache estimate for two context lengths, set with --context — run it twice, once at 4K and once at 32K (--context 4096 then --context 32768), and note how the KV-cache line grows. For a 7B that jump is from roughly half a GiB to a few GiB; at long context it can rival or exceed the weights themselves.
• State the smallest configuration (quant + context) that would have fit the box, and the smallest single GPU (16/24/48/80 GB) that would host it comfortably.

Include the worksheet output and two sentences relating it to the anon-rss value in the actual OOM line. This is the difference between “it crashed” and “it never fit, and here’s exactly why.”

Hard Tier (+up to 25% additional extra credit)

H1. The autonomy recommendation memo (the judgment piece)

Write MEMO.docx (half a page to a page) addressed to the relief org’s director, who wants to buy an AIOps assistant that automatically diagnoses and fixes these incidents. Using your own observed evidence from this project — including every place the AI got it wrong — answer:

1. For each of your diagnosed incident classes, recommend an autonomy level on this ladder: observe only → recommend to a human → act with human approval → act autonomously. Justify each with what you saw the AI do (and get wrong) on that class.
2. Cite at least two specific AI failures you observed this project (a hallucinated cause, a misread line, a destructive fix) and explain how each would have played out if the AI had been allowed to act unsupervised.
3. Name the one class of incident, if any, where you’d trust autonomous remediation today — and the guardrails (bounded actions, approval gate, audit log, blast-radius limit) you’d require even there.
4. State the constraint under which you’d change your recommendation (e.g. “if we add a verified-cause gate that diffs every AI quote against the raw log, I’d move I/O incidents from recommend to act-with-approval”).

This memo is the heart of the Hard tier. The diagnoses are necessary; the judgment about who may be trusted to act, and under what guardrails is the architect’s work — and it is precisely the work no tool does for you. A memo that just says “AI is great, automate it all,” or “never trust AI,” fails. The grade is in the nuance, defended with your own numbers and observed failures.

Submission

Submit one URL: a public GitHub repository.

What the repo must contain

1. evidence/ — the raw, unmodified incident files (your captures and/or the adopted samples).
2. ai-transcript.txt — the LLM’s verbatim diagnosis response(s), with the exact prompt you used.
3. REPORT.docx — the twice-diagnosis and adjudication (template below).
4. SIZING.txt (Medium) — the model-sizing analysis with worksheet output.
5. MEMO.docx (Hard) — the autonomy recommendation.
6. A README.txt pointing to each artifact and stating the tier you targeted.

# Project 2 — Diagnose the Failure Twice

**Tier targeted:**  Normal / Medium / Hard
**Incident(s) diagnosed:**  (subsystem(s))
**Root cause (verified):**  (one line each)
**Offending line(s):**  (quoted)
**Where the AI helped:**  (one or two lines)
**Where the AI lied:**  (name the failure mode(s) — this is the point of the week)
**Autonomy recommendation (Hard):**  (one line; full reasoning in MEMO.docx)
**What I learned:**  (one paragraph)
**AI usage:**  Model(s) + version(s) used: ____.  Used for: ____.  Overridden when: ____.
              Signed: <your name>

Hints (Read Before You Begin)

• Read the STAT column and the anon-rss number before anything else. A D-state pile is storage; a Z pile is a buggy parent; a fat anon-rss in an OOM line is a sizing story. The chapter’s §2.2–§2.3 are your cheat sheet.
• A crash loop hides its own cause. When systemctl status shows activating (auto-restart), do not read the latest restart — journalctl -u <unit> -b and find the first failure after the last clean boot.
• Make the AI cite, then verify the citation. Your single most important move is to diff every log line the model quotes against the raw evidence file. A cause with no real line behind it is a hallucination, however confident the prose.
• Never run an AI’s destructive fix on the premise alone. If the model says rm -rf something, confirm the problem it claims to fix actually exists before you’d ever run it. In this project, just flag such suggestions as a failure mode — do not run them.
• Size before you blame the software. Before you conclude “Ollama has a bug,” do the params × bytes-per-weight math. Most “model crashes” are the OS correctly killing a workload that never fit.
• Local vs cloud LLM is a real choice this week. Triaging with a local model via code/ask_local_model.py keeps the (possibly sensitive) logs on your machine — note in your report which you chose and why.

What Mastery Looks Like (Beyond the Rubric)

A great Project 2 reads like an incident review a senior SRE would sign. The hand diagnosis is crisp: one root cause, one offending line, one fix, each defended by a specific clue in the evidence. The AI adjudication is fearless — it names exactly where the model was brilliant and exactly where it confidently lied, with the diffed quote to prove it. And the Hard memo doesn’t take a slogan position; it grants autonomy where the evidence earned it, withholds it where the AI failed, and names the guardrail that would change the call. That last move — trust calibrated to observed behavior, under stated guardrails — is the entire human-in-the-loop thesis, graded.

Coach’s Note — The director wants a yes-or-no on automation. The white paper gave a slogan. Your value is that you ran the experiment: you watched this exact AI read this exact log and you saw where it broke. A recommendation grounded in observed failure beats a vendor’s confidence every time. That is what they’re paying the steward for — not faster typing, but calibrated trust.

When You’re Done

1. Re-open your raw evidence and confirm every line your REPORT.docx quotes is actually there — diff it. No fabricated quotes, including any you carried over from the AI.
2. Read your hand diagnosis and the AI’s side by side. Could a stranger see, from your write-up alone, where the AI helped and where you had to override it? If not, sharpen it.
3. (Medium) Confirm SIZING.txt’s numbers reconcile with the real anon-rss in the OOM line — the math should explain the death, not just sit beside it.
4. (Hard) Re-read MEMO.docx and ask: does each autonomy level have a reason tied to an observed AI behavior? Strip any sentence that’s a slogan.
5. Commit, push, submit the repo URL.
6. Read Chapter 3. Identity next — what it means to be truly known, and to guard against the impostor in a world of synthetic faces.

A theological footnote. “In him all things hold together” (Colossians 1:17, ESV) is a claim about coherence — that beneath the chaos there is a keystone, and the parts cohere in it. The administrator’s craft is a small, faithful echo of that confidence: the refusal to believe three failures are random, the conviction that there is a single broken agreement to be found. This week you also met the temptation of the age — to let a confident copilot hold the system together for you, to accept its verdict and move on. But a part cannot be the thing in which the parts cohere; the copilot is one more component, and the coherence — the judgment that verifies and decides — stays with the steward. You diagnosed twice precisely so you would never confuse the two. Keep the watch that holds the parts together.

See you next week.