Chapter 02 · Week 2

The Machine Underneath: Operating System Architecture

In what do all the parts hold together?

Chapter 2 — The Machine Underneath: Operating System Architecture

“Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.” — Brian Kernighan

“And he is before all things, and in him all things hold together.” — Colossians 1:17 (ESV)


Why This Matters

Last week you stepped into the role: the administrator in the age of AI, the steward of a system you did not build. You learned that the work is moving from manual administration to AIOps, and that a copilot can draft your runbook in seconds and lie to you in the same breath. You ended the week with a thesis you have to live by: the human stays in the loop where the judgment lives.

This week you go down a layer. Beneath every dashboard, every container, every cloud bill, there is a machine running an operating system, and the OS is doing four things, always, for everything: it schedules processes onto CPUs, it hands out and protects memory, it moves bytes to and from storage, and it keeps services alive. When a server is slow, when a deploy hangs, when the 2 a.m. page fires — the answer is almost always in one of those four. You cannot administer what you cannot diagnose, and you cannot diagnose what you do not understand at the level of the metal.

Here is what changed, and why a graduate administrator in 2026 has to own this cold. The OS is no longer just the thing your applications run on. It is also the thing your AI runs on. The moment you pull a model down with ollama run and watch your laptop’s fans spin up, you have made the OS host an AI workload — and that workload behaves nothing like a web server. It pins a CPU, it floods memory, it gets killed by the kernel’s out-of-memory reaper if you size it wrong. So this chapter teaches the OS twice: the OS you diagnose, and the OS that has become the substrate for AI itself.

Both AI threads run straight through the middle of this week. AI as the tool you wield: you will paste a process table, a dmesg dump, a Windows Event Viewer export into a large language model and let it interpret what you cannot read fast enough yourself — and you will learn precisely where that confident interpretation goes wrong, because a wrong root cause at 2 a.m. is worse than no root cause. AI as the workload you run and govern: you will learn why a “7-billion-parameter” model needs the memory it needs, what quantization actually buys you, and why CPU-only inference is a different animal from GPU inference. The full GPU and storage math comes in Chapters 4 and 6; this week you build the foundation.

The Christian question for the week is the one the whole machine begs: in what do all the parts hold together? A running system is a thousand moving pieces — schedulers, page tables, daemons, file handles — that must cohere or the whole thing falls down. Paul writes of Christ that “in him all things hold together” (Colossians 1:17, ESV). That is a claim about the cosmos, not about Linux. But it sharpens the engineering: the administrator’s craft is the search for the one thing that, when it fails, takes everything with it. Diagnosis is the discipline of finding what holds the parts together — and what, when it broke, let them fall apart.


2.1 — The Four Things the OS Does for Everything

Strip away the brand names. Linux, Windows, macOS, BSD — under the marketing they are all doing the same four jobs, and every problem you will ever diagnose lives in one of them.

  1. Process and CPU scheduling. A process is a running program with its own address space. The OS decides which process gets a CPU core and for how long, juggling far more processes than you have cores. A process is always in some state: running, runnable (ready but waiting for a core), sleeping (blocked on I/O or a timer), stopped, or — the one that bites you — zombie (finished, but its parent hasn’t reaped its exit status). When you read a process table, you are reading the scheduler’s ledger.
  2. Memory management. Every process thinks it owns a vast, private, contiguous address space. It doesn’t — that’s virtual memory, an illusion the OS maintains with page tables and the MMU, mapping virtual pages to physical RAM (or to disk, when it pages out). The illusion is glorious until physical RAM runs out. Then someone has to die. (See §2.3.)
  3. Storage and I/O. Files, block devices, the page cache, the journaling that keeps a filesystem consistent across a crash. Disk is thousands of times slower than RAM even on NVMe, so the OS works hard to avoid touching it — caching reads, buffering writes. When a box is “slow” but the CPU is idle, you are almost always I/O-bound, waiting on storage.
  4. Services and daemons. The long-lived background programs that are the server: the web server, the database, the SSH daemon, your model server. On Linux these are managed by systemd; on Windows by the Service Control Manager (and you inspect them with services.msc, Get-Service, or sc.exe).

Coach’s Note — When you get paged, do not start by reading the application’s logs. Start by asking the OS its four questions: Is a CPU pinned? Is memory exhausted? Is the disk the bottleneck? Did a service die? Four commands answer all four. The application log tells you what the program thinks went wrong; the OS tells you what actually ran out. Trust the OS first.


2.2 — Reading the Process Table (ps, top, Task Manager)

The process table is the single most useful diagnostic surface on any machine. Learn to read it the way a doctor reads a chart.

On Linux, the snapshot tool is ps and the live tool is top (or the friendlier htop). Here is the ps invocation worth memorizing — the processes sorted by CPU, then by memory:

# Top CPU consumers, with state, RSS (resident memory), and command
ps -eo pid,stat,pcpu,pmem,rss,comm --sort=-pcpu | head -n 12

# Top memory consumers (RSS in KB)
ps -eo pid,stat,pcpu,pmem,rss,comm --sort=-rss | head -n 12

The STAT column is the one beginners ignore and seniors read first. Its first letter is the process state:

STATMeaningWhat it tells you
RRunning or runnableUsing or waiting for a CPU
SInterruptible sleepBlocked on I/O or an event (normal idle)
DUninterruptible sleepStuck in a kernel I/O wait — often a sign of disk or NFS trouble
ZZombieFinished but unreaped; many Z means a parent isn’t cleaning up
TStoppedSuspended (e.g., SIGSTOP)

A pile of D-state processes is a storage problem wearing a CPU mask. A growing population of Z is a buggy parent process. Neither is obvious until you read the STAT column.

top shows the same data live, plus the load average — the three numbers (1-, 5-, 15-minute) that estimate how many processes are competing to run. The rule of thumb: a load average near your core count is healthy; far above it means a backlog. On a live top, press M to sort by memory, P by CPU, 1 to break out per-core utilization.

On Windows, the equivalents are Task Manager (the GUI), Resource Monitor (resmon), and — the one you will actually script — PowerShell:

# Top 10 processes by CPU time; WorkingSet64 is bytes of physical RAM
Get-Process | Sort-Object CPU -Descending |
  Select-Object -First 10 Name, Id, CPU,
    @{N='WS(MB)';E={[math]::Round($_.WorkingSet64/1MB)}}

The starter script code/triage.sh bundles the four OS questions into one pass; code/triage.ps1 does the same on Windows. Run them before you read another application log.


2.3 — Memory, Virtual Memory, and the OOM Killer

Memory is where the most violent failures live, because when RAM runs out the Linux kernel does not politely return an error. It picks a process and kills it. This is the OOM (out-of-memory) killer, and the first time it eats your database you will wish you had read this section.

Virtual memory is the foundation. Each process gets its own virtual address space; the kernel maps virtual pages to physical frames on demand (a page fault is the normal mechanism by which a page is first loaded, not an error). When physical RAM fills, the kernel can page out cold pages to swap. When swap is also exhausted — or when swap is off, as it often is on modern servers and in containers — the kernel invokes the OOM killer. It scores every process by an oom_score (roughly, how much memory it would free) and kills the highest. Your big-memory process — frequently the very thing you care about most — is the likeliest victim.

You read the verdict in the kernel ring buffer:

# Show OOM-killer events and what got killed
sudo dmesg -T | grep -i -E 'killed process|out of memory|oom'
# On systemd systems, the journal carries it too:
journalctl -k --since "1 hour ago" | grep -i oom

A real OOM kill looks like this in dmesg — and this exact line is the one your AI assistant will be asked to interpret in the lab:

[174523.881] Out of memory: Killed process 8423 (ollama) total-vm:18402112kB,
  anon-rss:15994208kB, file-rss:0kB, shmem-rss:0kB, oom_score_adj:0

Read it: PID 8423, named ollama, was holding ~16 GB of resident anonymous memory (anon-rss) when the kernel killed it. That is not a bug in Ollama. That is a sizing failure — someone tried to run a model that didn’t fit (we do that math in §2.7). The fix is not “restart the service”; the fix is “right-size the workload or add memory.” Knowing the difference is the whole job.

Coach’s Note — “It got OOM-killed” is a symptom, never a root cause. The root cause is always one of three: a genuine leak (memory grows without bound — watch RSS over time), a workload that never fit (you sized it wrong — see the model math below), or memory pressure from a neighbor (some other process ate the RAM and yours was the unlucky high scorer). The AI will happily tell you “increase the memory limit.” Sometimes that’s right. Sometimes it’s papering over a leak you’ll meet again next week. You decide which.


2.4 — Storage I/O: When “Slow” Isn’t the CPU

The most misdiagnosed performance problem in the world is the I/O-bound box that everyone insists is “out of CPU.” The tell is simple and you can check it in one command.

# %iowait is the share of time CPUs sat idle WAITING on disk.
# High %iowait + low %user/%system = a storage bottleneck, not a CPU one.
iostat -x 2 3

# Per-process I/O, the smoking gun:
sudo iotop -o -b -n 2

In iostat -x, the columns that matter are %util (how busy the device is — near 100% means saturated), await (average ms a request waits — single-digit ms is healthy for NVMe; tens-to-hundreds means the disk is drowning), and r/s/w/s (IOPS). A device pinned at %util 99 with a fat await is your bottleneck, full stop — and no amount of adding CPU will help. The processes you saw in D state back in §2.2 are queued behind exactly this.

Two numbers you must keep separate, because vendors blur them: IOPS (operations per second — what random small reads need, like a database) versus throughput (MB/s — what big sequential reads need, like loading a model file off disk). A model-serving cold start that reads a 16 GB weights file is throughput-bound; a busy transactional database is IOPS-bound. The same disk can be great at one and terrible at the other.

This matters directly for AI: loading model weights from disk into RAM (and then to VRAM) is a large sequential read, and on a slow disk it dominates your cold-start time. A 70B model’s ~40 GB of INT4 weights read at 500 MB/s is ~80 seconds just to read the file — before a single token is generated. The disk you put your weights on is a latency decision, not just a capacity one.


2.5 — Services and Daemons: systemd and Windows Services

A server is its services. The web server, the database, the model server — these are long-lived daemons, and managing them is a daily act. On modern Linux that means systemd.

# Is the service alive? (state, recent log tail, last restart)
systemctl status ollama

# Start / stop / restart / enable-at-boot
sudo systemctl restart ollama
sudo systemctl enable --now ollama        # start now AND on every boot

# Why did it die? The journal, scoped to the unit, since boot:
journalctl -u ollama -b --no-pager | tail -n 50

# Watch a flapping service restart over and over (the crash loop):
systemctl status ollama | grep -E 'Active:|Main PID:'

The state vocabulary matters: active (running) is healthy; failed means it died and systemd gave up; activating (auto-restart) is the crash loop — the service keeps dying and systemd keeps relaunching it, which masks the real failure behind a flicker of “it’s starting.” When you see auto-restart, stop watching status and go read journalctl -u <unit> to find the first failure, not the latest.

A minimal unit file — the kind you’ll write to run a local model server as a managed service — lives at code/model-server.service. Note the MemoryMax line: that is how you tell systemd to cap a workload’s memory so a runaway model gets killed in a bounded, predictable way instead of triggering the system-wide OOM killer and taking a neighbor down with it.

On Windows, the Service Control Manager is the analog, and the Event Viewer is where the bodies are buried:

Get-Service -Name 'W3SVC' | Select-Object Status, StartType, Name
Restart-Service -Name 'W3SVC'

# Pull the last 20 errors from the System log (the SCM writes here):
Get-WinEvent -FilterHashtable @{LogName='System'; Level=2} -MaxEvents 20 |
  Select-Object TimeCreated, Id, ProviderName, Message

2.6 — AI as the Tool: Reading Logs With an LLM (and Where It Lies)

Now the first AI thread. A modern administrator’s logs are too many to read by eye — a busy host emits thousands of journald lines an hour, and the Windows System log is a firehose. The 2026 move is to hand that firehose to a large language model and ask it, in English, what happened? This genuinely works, and it is genuinely dangerous. Both are true at once.

Here is the honest version of the workflow. Capture the evidence, then ask:

# Capture the last hour of kernel + service logs into one file
journalctl -k -u ollama --since "1 hour ago" --no-pager > /tmp/incident.log

# (Then paste /tmp/incident.log into your assistant with a tight prompt.)

A good prompt is specific, scoped, and asks for evidence, not just a verdict:

You are helping triage a Linux incident. Below are kernel and journald logs
from the last hour. Identify the single most likely root cause. For your
answer, quote the exact log line(s) that support it, state your confidence,
and list one alternative explanation you ruled out. Do not speculate beyond
the logs. If the logs are insufficient, say so.

<paste /tmp/incident.log>

Used this way, an LLM is a superb first-pass triage analyst. It correlates a dmesg OOM line with a service restart in the journal faster than you can scroll. It knows that STAT D plus high await means I/O. It can turn a wall of Windows Event IDs into a sentence. As of 2026, a capable model such as Claude Sonnet 4.6 or Opus 4.8 (or GPT-5.5, or Gemini 3 Pro) does this well enough to save you real minutes at 2 a.m.

And then there are the failure modes, which you must know by name because they are systematic, not random:

  • Confident hallucination of a cause. The model invents a plausible-sounding culprit (“the failure was caused by a corrupted inode in /var/lib/ollama”) that appears nowhere in your logs. It sounds like a senior engineer. It is making it up. This is why the prompt above demands quoted evidence — a cause with no log line behind it is fiction.
  • Misread of the actual offending line. The model fixes on a scary-looking but irrelevant WARNING and ignores the quiet Killed process line that is the real event. Recency and salience bias both apply.
  • Plausible-but-wrong remediation. “Run sudo rm -rf /var/lib/ollama/models to clear the corruption” — confidently destructive advice for a problem that doesn’t exist. Never run a destructive command an AI suggested without verifying the premise yourself.
  • The fabricated log line. Asked to “show the relevant lines,” a model will sometimes generate a log line in the right format that was never in your file. Always diff its quotes against the source.

Coach’s Note — The discipline is one sentence: make the model cite, then verify the citation against the raw log yourself. The AI accelerates your reading. It does not replace your reading. The instant you accept a root cause you have not confirmed in the actual log, you have handed your judgment to a confident stranger — and in this book, judgment is the one thing you never delegate. AI drafts the hypothesis; you own the verdict.


2.7 — AI as the Workload: Running a Model on the Machine Underneath

Now the second thread. When you run a model locally — and every administrator should, at least once, to feel what it costs — you are asking the OS to host a workload unlike any web service. Let’s size it honestly, because sizing is where the OOM kill in §2.3 came from.

The dev-tier runners you’ll meet first (full production serving is Chapter 7):

RunnerDefault endpointFormatBest for
Ollamalocalhost:11434/v1GGUFQuickest start; CLI + daemon
LM Studiolocalhost:1234/v1GGUF / MLXGUI, Apple-silicon MLX backend
llama.cpp (llama-server)localhost:8080/v1GGUFLowest-level, CPU/edge friendly

All three speak the OpenAI-compatible HTTP API (/v1/chat/completions), which means you change only the base_url and a dummy key and the same client code runs against any of them. (Versions move fast — as of mid-June 2026, Ollama is around v0.30.x; re-verify before you teach a lab.) The reference client in code/ask_local_model.py does exactly this.

Now the math that determines whether the OS hosts your model or the OOM killer eats it. Weights memory ≈ parameters × bytes-per-weight. The rough rules of thumb you should carry in your head:

  • ~2 GB per billion parameters at FP16 (16-bit, the unquantized baseline)
  • ~1 GB per billion at 8-bit (INT8 / FP8)
  • ~0.5 GB per billion at 4-bit (the GGUF Q4_K_M sweet spot)

So a 7B/8B model: ~14–16 GB at FP16, but only ~4–5 GB at 4-bit. A 70B model: ~140 GB at FP16, ~35–40 GB at INT4 — which is why a single 48 GB GPU is the practical floor for a 70B at modest context. A 405B model is ~810 GB at FP16, ~405 GB at FP8 — it fits one 8×H100 node (640 GB) only because of quantization.

ModelFP16 (~2 GB/B)INT8 (~1 GB/B)INT4 (~0.5 GB/B)
7B / 8B~14–16 GB~7–8 GB~4–5 GB
13B~26 GB~13 GB~6.5 GB
70B~140 GB~70 GB~35–40 GB
405B~810 GB~405 GB (FP8)~200 GB

That is weights only. Two things push the real number higher. First, framework and activation overhead adds roughly 15–20%. Second — and this is the one people forget — the KV cache grows with context length and batch size, and at 32K–128K context it can exceed the weights themselves. Real deployed memory is typically 15–40% above weights-only. The sizing worksheet code/vram_sizer.py does this arithmetic for you so you can stop guessing.

Quantization is the lever that makes any of this fit. It stores each weight in fewer bits — FP16 → INT8 → INT4 — trading a little accuracy for a lot of memory. The 2026 landscape, grounded in the fact brief:

  • Q4_K_M (a GGUF K-quant, ~4.5–4.9 bits/weight) is the default sweet spot for local serving — the knee of the quality-vs-size curve.
  • INT8 loses roughly 1–3% quality; INT4 noticeably degrades hard reasoning and math, so test on your task before trusting it.
  • FP8 (E4M3) is the 2026 production “near-lossless” workhorse, native on Hopper/Blackwell GPUs. On Blackwell, hardware-native 4-bit formats NVFP4/MXFP4 approach FP8 accuracy at INT4-class memory.
  • Tooling moves weekly; llm-compressor and GPTQModel are current, while the older AutoAWQ/AutoGPTQ are deprecated (archived in 2025). Pin versions and re-check.

Finally, CPU versus GPU. A model will run on CPU — llama.cpp and Ollama both support it — but it runs slowly, because inference is memory-bandwidth-bound and a GPU’s VRAM bandwidth dwarfs system RAM’s. CPU inference is fine for a single user, a small model, a dev box. The moment you need throughput for more than one or two concurrent users, you need a GPU, and you need the VRAM math above to fit on it. That is the bridge into Chapters 4 (storage) and 6 (GPU virtualization).

Coach’s Note — The OOM kill in §2.3 was an ollama process holding 16 GB. Now you can read it as a sizing story: someone ran an FP16 7B (~14–16 GB) on a box without the RAM, instead of a Q4_K_M quant (~4–5 GB) that would have fit four times over. The kernel didn’t malfunction. The administrator didn’t do the arithmetic. Do the arithmetic.


2.x — Interactive Lab: Log Triage Console

Below this chapter on the website, you’ll find the Log Triage Console — an interactive panel built for exactly the skill §2.6 just taught. Use it now; it’s part of the chapter, not an extra.

The console gives you a realistic process table (with STAT, %CPU, RSS, command) sitting beside a kernel/event log (a mixed stream of dmesg, journald, and Windows-Event-style lines). One of those log lines is the true offending event — an OOM kill, a crash-looping service, a disk saturating into D-state waits. Your job, in order:

  1. Form a hypothesis first, by hand. Read the process table and the log the way §2.2–§2.5 taught you. Which process is the problem? Which single log line is the root-cause event? Commit to an answer before you look at any AI output. This is the predict-before-you-measure discipline of Phase 1.
  2. Compare the candidate AI interpretations. The console shows you two or three AI-generated readings of the same logs — and at least one of them is confidently wrong in one of the §2.6 ways (a hallucinated cause, a misread line, a destructive “fix”). Pick the correct interpretation and, crucially, say why each wrong one is wrong.
  3. Locate the true offending line and the console scores you against ground truth.

What it teaches: that an AI reading of a log is a hypothesis to verify, not a verdict to accept — and that the verification is a concrete, learnable motion. You hand-read, the AI hand-reads, and you adjudicate. That is the loop you’ll run for real every time a server misbehaves for the rest of your career. Do it here, where being wrong costs nothing.


2.y — In What Do All the Parts Hold Together?

A running system is held together by invisible agreements. The scheduler trusts that a process will yield. Virtual memory trusts that page tables map truthfully. systemd trusts that a daemon, restarted, will recover. Storage trusts that a journal replay will leave the filesystem consistent. None of these is visible on the surface; all of them must cohere, every microsecond, or the machine falls down. The administrator’s craft is, in large part, the discipline of finding the one broken agreement when everything downstream of it has collapsed into noise.

Paul writes that in Christ “all things hold together” (Colossians 1:17, ESV). That is a cosmological claim, and I am not going to flatten it into a metaphor about init systems. But it does sharpen the engineering, the way the right question always does. Diagnosis is the search for what holds the parts together — and what, when it broke, let them fall apart. The OOM kill, the crash loop, the saturated disk: each is a story about a single agreement breaking and the rest of the system inheriting the crack. The mature administrator does not chase symptoms. They look for the keystone — the load-bearing thing that, when it failed, explains all the rest. That instinct is a kind of faith in coherence: a refusal to believe the failure is random, a confidence that there is a thing that holds the parts together, and that it can be found.

There is a vocational point underneath this, and it’s the one the LCMS tradition would press. You did not make this machine. You did not write the kernel, design the scheduler, author the model. You are a steward of what you did not make — and stewardship is precisely the willingness to understand a thing deeply enough to keep it whole. The temptation of the AI age is to let the confident copilot hold the system together for you — to accept its root cause, run its fix, and move on. But the copilot does not hold your system together. It cannot. It is one more part, and a part cannot be the thing in which the parts cohere. The coherence — the judgment that adjudicates, verifies, and decides — stays with you. That is not a limitation of today’s models you are waiting out. It is the shape of the vocation. The steward keeps the watch that holds the parts together.


2.z — Common Pitfalls

Pitfall: Treating “OOM-killed” as a root cause and just restarting. Example: A service dies nightly; the runbook says “restart it,” and it works until tomorrow. Fix: OOM is a symptom. Watch RSS over time (leak?), check the model’s sizing math (never fit?), and check neighbors (memory pressure?). Restarting a leak just delays the next death.


Pitfall: Blaming the CPU for an I/O-bound box. Example: “The server’s pegged, add cores!” — but top shows low %user and high %iowait. Fix: Read %iowait and iostat -x %util/await first. A saturated disk and a pile of D-state processes is storage, not CPU. Adding cores buys nothing.


Pitfall: Watching systemctl status during a crash loop and reading the latest restart instead of the first failure. Example: Status shows activating (auto-restart) and a fresh, clean-looking start; you conclude it’s recovering. Fix: A crash loop hides the real failure. Read journalctl -u <unit> -b and find the first error after the last clean boot — that’s the cause; everything after is the loop.


Pitfall: Accepting an AI’s root cause without verifying it against the raw log. Example: The model says “corrupted inode in /var/lib/ollama”; you believe it because it sounds authoritative. Fix: Demand a quoted log line for every claimed cause, then diff that quote against the actual file. A cause with no evidence in the log is a hallucination, however confident.


Pitfall: Running an AI-suggested destructive command without checking the premise. Example: “Run rm -rf on the model cache to fix the corruption” — for corruption that never existed. Fix: Never run a destructive command an AI proposed until you have independently confirmed the problem it claims to fix. Mark destructive steps; verify the premise; back up first.


Pitfall: Sizing a model in your head as “it’s only 7 billion parameters, it’ll fit.” Example: You run an FP16 7B (~14–16 GB) on a 16 GB box and get OOM-killed mid-load. Fix: Do the arithmetic: params × bytes-per-weight + ~15–20% overhead + KV cache. Reach for a Q4_K_M quant (~4–5 GB for a 7B) when memory is tight. The math is in §2.7 and code/vram_sizer.py.


Pitfall: Forgetting the KV cache and sizing only the weights. Example: A 13B INT4 (~6.5 GB) “fits” your 8 GB GPU — until you serve it at 32K context and the KV cache blows the budget. Fix: At long context the KV cache can exceed the weights. Budget 15–40% above weights-only, and compute the KV cache explicitly for your context length and batch size.


2.(z+1) — Reps

Open the exercises for the full set. This week’s reps build the diagnostic reflexes the project will grade: reading a process table cold, finding an OOM kill in dmesg, separating an I/O bottleneck from a CPU one, debugging a crash loop, and sizing a model so it actually fits.

This week’s AI policy (Phase 1): for the diagnostic reps you do it by hand first, write down your root cause, and only then ask an AI to interpret the same evidence — then you grade the AI against your own analysis. The human owns the verdict; the AI is the thing under test. Every rep ends with an honest “AI usage” line.

A preview:

  • Rep 2 — Read a real process table and name the offending process and its state from STAT alone.
  • Rep 4 — Find the OOM-killer line in a dmesg dump and translate anon-rss into “what didn’t fit.”
  • Rep 6 — Separate an I/O-bound box from a CPU-bound one using iostat -x columns.
  • Rep 8 — Hand-size a 7B/13B/70B model at FP16/INT8/INT4, then check yourself with code/vram_sizer.py.
  • Done? One Last Thing — diagnose a failure by hand, then with an AI, and write the verdict.

A short “Check Your Reps” quiz is embedded on this page, below the lab. Take it before you move on.


2.(z+2) — This Week’s Project

You’re ready for Project 2 — “Diagnose the Failure Twice”, in Project 2.

You’ll be given (and you’ll also generate) a captured incident: a process table, a kernel log with a real OOM event or crash loop, and a service state. You will diagnose it twice — once entirely by hand, writing down the root cause and the single offending line with your reasoning, and once by handing the same evidence to a large language model under the §2.6 cite-and-verify discipline. Then you’ll adjudicate: where did the AI agree with you, where did it go wrong, and what’s the verified verdict? The Normal tier diagnoses one incident both ways and produces a REPORT.docx. Medium adds a second incident and a model-sizing analysis that explains an OOM as a sizing failure. Hard demands the judgment piece — a memo recommending whether your team should let an AIOps assistant act on this class of incident automatically, defended with the failure modes you observed.

Like every Phase 1 project, it ends in a written verdict, not just a transcript. The diagnosis is half the grade; the honest accounting of where the AI helped and where it lied is the other half.


2.(z+3) — Coach’s Final Word

This week you went down a layer, to the machine underneath everything. You can now read a process table and name the sick process from its STAT column. You can find an OOM kill in dmesg and read it as a sizing story, not a mystery. You can tell an I/O-bound box from a CPU-bound one in one command. You can debug a crash loop by finding the first failure instead of chasing the loop. And you can size an AI model — params times bytes-per-weight, plus overhead, plus the KV cache everyone forgets — so the OS hosts it instead of killing it.

You also ran both AI threads through your own hands. You let a model triage a log and you caught it lying — a hallucinated cause, a misread line, a destructive fix offered with total confidence. And you sized the AI as a workload, learning why a 7B model needs the memory it needs and what quantization buys you when it doesn’t fit. That double vision — AI as the tool you wield, AI as the workload you govern — is the spine of this whole course.

In him all things hold together. The administrator’s version of that confidence is the refusal to believe a failure is random — the conviction that there is a keystone, a single broken agreement, and that you can find it. The copilot can help you look. It cannot be the thing that holds your system together, because a part is never the whole. The coherence stays with the steward. Keep the watch.

See you next week.


Up next: Read the exercises and complete every rep — they’re the conditioning. Then open Project 2 and diagnose the failure twice. Set up your environment with Appendix A (the lab) and your local + cloud AI access with Appendix B; the agentic-AI rules in Appendix C are still off this week — Phase 1. After that, Chapter 3 — identity and access in a world of synthetic faces, where we ask what it means to be truly known.

Previously: Chapter 1 — the administrator in the age of AI: from runbook to copilot.

Interactive Lab — Week 2
Log Triage Console

A node just went unresponsive. You have ps output and a kernel-log excerpt. Find the runaway process, then judge two AI interpretations against the evidence — because a confident answer is still a hypothesis until the log confirms it.

1 Pick the offending process
PID USER %CPU %MEM COMMAND
dmesg — last 4 lines
[ 8231.04] node invoked oom-killer: gfp_mask=0x..., order=0
[ 8231.04] Out of memory: Killed process 4471 (ingest-worker.js)
[ 8231.05]   total-vm:8412904kB, anon-rss:8090112kB
[ 8231.11] oom_reaper: reaped process 4471 (ingest-worker.js)

No process selected yet.

2 Judge the AI's interpretation

Two assistants summarized the same incident. Pick the one the evidence supports.

Try: Pick the wrong process first and read why it doesn't fit. Then pick Hypothesis B and notice it sounds authoritative but cites no line in the log — the OOM message names PID 4471 explicitly, and B never engages with it.
Check Your Reps

Check Your Reps — Chapter 2

Question 1 of 5
On a Linux host that feels "slow" while top shows low %user and high %iowait, what is the most likely bottleneck?
Why: High %iowait with low %user means the CPUs are sitting idle waiting on disk, so the bottleneck is storage I/O and adding cores buys nothing.
Question 2 of 5
In a process table, what does a process in STAT state 'D' indicate?
ps -eo pid,stat,pcpu,rss,comm --sort=-rss
Why: State 'D' is uninterruptible sleep — a process blocked in a kernel I/O wait, and a pile of them is a storage problem wearing a CPU mask.
Question 3 of 5
Using the chapter's rule of thumb, roughly how much memory do the weights of a 7B model need at FP16 versus 4-bit (Q4_K_M)?
Why: At ~2 GB per billion params (FP16) a 7B is ~14–16 GB, while ~0.5 GB/B at 4-bit makes it ~4–5 GB — the lever that lets a too-big model fit.
Question 4 of 5
When an LLM interprets your logs and names a root cause, what is the single most important verification discipline from the chapter?
Why: A claimed cause with no real supporting line is a hallucination however confident, so you demand a quoted line and verify it against the actual log before accepting the verdict.
Question 5 of 5
A systemd service shows 'activating (auto-restart)'. Where do you look to find the real cause?
journalctl -u model-server -b --no-pager
Why: A crash loop hides its cause behind repeated restarts, so you scroll up to the first failure after the last clean boot rather than reading the latest restart.
YOU FINISHED. NICE WORK.