Chapter 2 — Reps

Conditioning, not grading. This week’s reps are diagnostic reflexes on the machine underneath — process tables, memory and the OOM killer, storage I/O, services, AI-assisted log reading, and sizing a model so it fits.

Ground rules:

1. Type every command yourself. No copy-paste from the chapter. Your fingers have to learn journalctl -u <unit> -b and iostat -x cold, because at 2 a.m. you won’t be reading a textbook.
2. Run everything on a real machine. Linux (a VM, WSL2, or a cloud box per Appendix A) for the Linux reps; a Windows host or VM for the PowerShell ones. Reading about top is not the same as watching the load average move.
3. Predict before you measure. Every diagnostic rep: write your hypothesis first — which process, which line, which subsystem — then run the command and compare. The gap is the lesson.
4. AI policy — Phase 1, “by hand first.” For any rep that involves an AI interpreting evidence: do your own analysis and write it down before you ask the model. Then grade the model against your analysis. The human owns the verdict. End each AI-touching rep with an honest one-line AI usage note.

The chapter’s code/ folder has the starter scripts and sample logs referenced below. If you can’t run a tool locally, the sample files (e.g. code/sample-dmesg.log) let you do the reading reps anywhere.

Reps 1–3: Reading the Process Table

Rep 1 — The four-question triage pass

Run the bundled triage script and read its output as the OS’s answers to its own four questions (CPU, memory, disk, services):

bash code/triage.sh

(On Windows, run pwsh code/triage.ps1.) Predict first: before you run it, write down what you expect a healthy idle box to look like — roughly what load average, what %iowait, how many running processes. Then run it. Write two sentences: which of the four subsystems looks busiest on your box right now, and how you can tell.

Rep 2 — Name the sick process from STAT alone

Get a live process table sorted by CPU, then by memory:

ps -eo pid,stat,pcpu,pmem,rss,comm --sort=-pcpu | head -n 12
ps -eo pid,stat,pcpu,pmem,rss,comm --sort=-rss  | head -n 12

Open code/sample-ps.txt, which contains a captured table from a misbehaving host. Without looking anything up: which process is the problem, and what does its STAT letter tell you about why (is it pinning a CPU, stuck on I/O, or a zombie)? Write the PID, the state, and your one-sentence reasoning.

Rep 3 — Watch the load average and per-core split

Open top (or htop). Press 1 to break out per-core utilization, M to sort by memory, P by CPU. In another terminal, generate load:

# Pin one core for 20 seconds (Ctrl-C to stop early)
timeout 20 sh -c 'while :; do :; done'

Watch the 1-minute load average climb and one core go to 100%. Predict: by how much should the 1-minute load average rise for one pinned core? Then confirm. One sentence on the relationship between cores, load average, and “is this box overloaded?”

Reps 4–6: Memory and Storage — Where the Box Actually Hurts

Rep 4 — Find the OOM kill and translate it

Open code/sample-dmesg.log. Somewhere in it is an OOM-killer event. Find it by hand:

grep -i -E 'killed process|out of memory|oom' code/sample-dmesg.log

Read the matching line. Write down: which PID and command were killed, how much resident memory (anon-rss) it was holding when it died, and — the real question — translate that into a sizing story: what was probably running (use the §2.7 table) and why didn’t it fit? One sentence on why “just restart it” is the wrong fix.

Rep 5 — Watch memory pressure live (optional, safe-guarded)

On a VM you don’t care about, watch memory fill. Do not run this on a machine you need. Cap it so it can’t take the box down:

# Allocate ~500MB in a memory-capped scope so it can't OOM the whole host.
# Requires systemd; Ctrl-C to stop. This is a DELIBERATELY bounded test.
systemd-run --scope -p MemoryMax=600M --user \
  python3 -c "x=bytearray(500*1024*1024); input('holding 500MB, Enter to free')"

Watch free -h (or top, press M) in another terminal. Predict what happens if you push the allocation past MemoryMax. Then push it (edit to bytearray(800*...)) and observe the bounded kill. Write one sentence on why MemoryMax makes the failure predictable instead of letting the system-wide OOM killer pick a victim. AI usage: none required.

Rep 6 — I/O-bound vs CPU-bound, told apart in one screen

Run iostat -x 2 3 while generating disk load on a scratch directory:

# Generate sustained writes (delete the file after!). Adjust count for speed.
dd if=/dev/zero of=/tmp/iotest.bin bs=1M count=2048 oflag=direct status=progress
rm -f /tmp/iotest.bin

While dd runs, read iostat -x: %util, await, w/s. Predict first: will %iowait in top be high or low? Will %user? Now open code/sample-iostat.txt — two captures, one CPU-bound box and one I/O-bound box. Identify which is which and cite the exact columns that prove it. One sentence: why does adding CPU cores not help the I/O-bound box?

Reps 7–9: Services, and the AI That Reads Your Logs

Rep 7 — Debug a crash loop by finding the first failure

Inspect a service and read its journal scoped to the current boot:

systemctl status ollama        # or any installed unit; ssh, cron, etc.
journalctl -u ollama -b --no-pager | tail -n 40

If you have no flapping service, use code/sample-journal.log, which captures a crash-looping unit. By hand: find the activating (auto-restart) flicker, then scroll up to the first error after the last clean start. Write the first-failure line and explain why reading the latest restart would have misled you.

Rep 8 — Hand-size three models, then check yourself

No tools first — paper and head only. Using the §2.7 rule (~2 GB/B at FP16, ~1 at INT8, ~0.5 at INT4), fill in the memory for a 7B, 13B, and 70B model at FP16, INT8, and INT4. Add ~20% overhead to one of them by hand. Then check yourself:

python3 code/vram_sizer.py --params 7   --quant q4   --context 8192
python3 code/vram_sizer.py --params 70  --quant int4 --context 32768
python3 code/vram_sizer.py --params 13  --quant fp16 --context 4096

Where did your hand estimate differ from the script? Write down the smallest single GPU (pick from 16/24/48/80 GB) that fits a 70B INT4 at 32K context, and why the KV cache mattered to the answer. AI usage: none — this is the arithmetic you must own yourself.

Rep 9 — Make an AI read a log, then catch it

By hand first. Open code/sample-incident.log (a mixed kernel + journald capture). Write your root cause and the single offending line, with reasoning — before you open any AI.

Now ask a model. Use a tight, evidence-demanding prompt:

You are triaging a Linux incident. Below are kernel + journald logs.
Identify the single most likely root cause. Quote the exact log line(s)
that support it, state your confidence, and give one alternative you ruled
out. Do not speculate beyond the logs.

<paste code/sample-incident.log>

Compare the model’s answer to yours. Grade it: did it cite a real line (diff its quote against the file)? Did it agree with your root cause? Did it suggest anything destructive? Write three sentences — agreement, any hallucination or misread you caught, and your final verdict. AI usage: required; name the model and version.

Reps 10–11: Local Model on the Real Machine

Rep 10 — One client, the universal API

Install a local runner (Ollama is the quickest; Appendix B), pull a small quantized model, and hit it with the reference client:

ollama pull llama3.2:3b        # small; ~2GB, runs on modest hardware
ollama run  llama3.2:3b "Say hello in one short sentence."

# Now from code, via the OpenAI-compatible endpoint:
python3 code/ask_local_model.py --base-url http://localhost:11434/v1 \
  --model llama3.2:3b --prompt "Name the four jobs an OS does. One line each."

While it answers, watch top (press M). Write down: the model’s RSS while serving, and whether your machine used CPU or GPU (does ollama ps show a GPU?). One sentence relating the RSS you saw to the §2.7 sizing table.

Rep 11 — Same model as a managed service

Adapt code/model-server.service for your runner and install it as a user service so the model survives a logout and gets a bounded memory cap:

mkdir -p ~/.config/systemd/user
cp code/model-server.service ~/.config/systemd/user/
# edit ExecStart + MemoryMax for your box, then:
systemctl --user daemon-reload
systemctl --user enable --now model-server.service
systemctl --user status model-server.service

Predict: what happens if you set MemoryMax below the model’s needed memory? Then set it too low on purpose and read journalctl --user -u model-server.service for the bounded kill. One sentence on why a MemoryMax is safer than letting the host OOM killer choose. (Restore a sane value when done.)

Done? One Last Thing.

This is the project in miniature — diagnose a failure twice.

Open code/sample-incident.log and code/sample-ps.txt together as a single incident. Do all three steps:

1. By hand. Write the root cause, the single offending log line, and the fix you would actually run — with your reasoning. No AI.
2. With AI. Hand the same two files to a model under the cite-and-verify prompt from Rep 9. Save its full answer.
3. Adjudicate. In ~150 words: where did the AI agree with you, where did it go wrong (name the failure mode — hallucinated cause / misread line / destructive fix), and what is the verified verdict and fix? End with: would you let an AIOps assistant act on this automatically? One sentence, defended.

Keep this write-up. It’s the exact shape of the REPORT.docx the project asks for — you’ve already drafted it.

Up next: Project 2 — Project 2: Diagnose the Failure Twice.