Chapter 05 · Week 5

Network Services and the Watch on the Wire

Who keeps the gate, and how does a message travel faithfully?

Chapter 5 — Network Services and the Watch on the Wire

“The network is the computer.” — John Gage, Sun Microsystems

“So you, son of man, I have made a watchman for the house of Israel. Whenever you hear a word from my mouth, you shall give them warning from me.” — Ezekiel 33:7 (ESV)


Why This Matters

A server is a box of potential until it can talk. The network is where your systems actually become a system — where a DNS lookup turns a name into an address, where a TLS handshake turns a stranger into a trusted peer, where a firewall rule decides whether a packet lives or dies. For a system administrator, the network is not plumbing you delegate to “the network team.” It is the surface on which every failure, every breach, and every AI workload you run this semester will show up first. The wire tells the truth before the dashboard does.

Two things changed under your feet, and this chapter is about both.

First, AI moved onto the wire as a watcher. The old way to find a compromise was to write a rule — “alert if anyone connects to a known-bad IP” — and hope the attacker used a known-bad IP. The new way is a behavioral baseline: you teach a model what normal traffic looks like for this host, on this port, at this hour, and you flag what deviates. Network Detection and Response (NDR) and AI-enhanced flow analysis can surface beaconing, data exfiltration, and lateral movement that no static rule anticipated. That is the watchman of Ezekiel 33 rendered in software — set on the wall to see what is coming and to sound the horn. But a watchman who cries wolf at every shadow gets ignored, and an AI that flags 400 anomalies a night trains your team to click “dismiss.” False positives are not a cosmetic problem; they are how good detection dies.

Second, AI moved onto the wire as a workload — and it is the most bandwidth-hungry workload you have ever administered. When you train or serve a large model across more than one GPU, the bottleneck is almost never the math. It is the interconnect: how fast GPUs can shovel gradients and KV-cache between each other. This is why NVIDIA built NVLink and NVSwitch, why GPU clusters run InfiniBand or RoCEv2 instead of ordinary Ethernet, and why GPUDirect RDMA lets a network card write straight into GPU memory and skip the CPU entirely. As of 2026, an administrator who can size a vector database (Chapter 4) but cannot reason about east-west fabric will build an AI estate that is starved at the very point you spent the most money.

Here is the week’s question, and it is older than any protocol: Who keeps the gate, and how does a message travel faithfully? God appoints a watchman in Ezekiel 33 and makes the assignment heavy — the watchman who sees danger and stays silent is held to account for the blood. That is a startling claim about responsibility. The gate-keeping does not belong to the wall, or to the horn, or to the watcher’s eyesight. It belongs to the watchman. We will let an AI watch the wire this week. We will not let it keep the gate. Hold that distinction; we will earn it.

Coach’s Note — The most useless sentence in network operations is “the network is fine.” The network is never fine; it is within tolerance, which is a number you measured. Make people show you the number.


5.1 — The Core Services, Fast and Correct

Every network you will ever administer rests on a small set of services, and a graduate admin should be able to reason about each one’s failure mode in their sleep. You know what these are; the point here is to fix the administrator’s mental model — what breaks, and what it looks like on the wire.

ServiceJobClassic failureWhat you watch
DNSname → addressstale/poisoned records, slow resolver, TTL stampedequery latency, NXDOMAIN rate, unexpected resolvers
DHCPhand out addressespool exhaustion, rogue serverlease counts, duplicate offers
NTPagree on timedrift breaks TLS, logs, Kerberosoffset in ms; > a few hundred ms is trouble
HTTP(S) / reverse proxyfront the appcert expiry, upstream 502sTLS handshake errors, 5xx rate
VPNextend the perimeterMTU/fragmentation, authtunnel up/down, throughput
Firewall / segmentationallow/denyoverly broad rules, shadow rulesdeny logs, east-west attempts
Load balancerspread the workuneven hashing, dead backend in poolper-backend health, connection skew

Three of these deserve a sysadmin’s special suspicion, because they fail silently and globally. DNS is the one that takes down everything and looks like “the app is slow.” NTP is the one nobody checks until Kerberos tickets stop validating and TLS certificates appear invalid because the clock is wrong. And the reverse proxy’s certificate is the one that expires at 2 a.m. on a holiday. A quick liveness pass belongs in your runbook:

# Resolve + time the resolver (watch for slow or wrong answers)
dig +stats chapel.example.org @10.20.4.10

# Is the clock honest? offset should be small; large offset breaks auth + TLS
chronyc tracking | grep -E "System time|Last offset"

# Will the cert outlive the weekend? (days remaining)
echo | openssl s_client -connect app.example.org:443 -servername app.example.org 2>/dev/null \
  | openssl x509 -noout -enddate

Segmentation is the quiet hero of this chapter. The single highest-leverage network control you own is not a fancier firewall — it is east-west segmentation: the assumption that a host in your application subnet has no business opening an SSH session to a host in your finance subnet, and a rule that says so. When you build the AI estate later this term, the inference subnet should not be able to reach the backup vault, and the watchman’s first job is to notice when something tries.


5.2 — Segmentation and the Death of the Perimeter

The old network model had a hard outside and a soft inside: a firewall at the edge, and once you were past it, you could reach anything. That model died the day the first phished laptop walked through the front door already compromised. The modern posture — call it zero trust if you like the term, least-privilege networking if you don’t — assumes the attacker is already inside and asks a sharper question for every connection: should this particular host be allowed to reach that particular service on that particular port, right now?

For an administrator, this is concrete, not philosophical. It is segmentation: carving the network into zones by function and denying east-west traffic by default. The web tier reaches the app tier; the app tier reaches the database; nothing reaches the backup vault except the backup job. When you stand up the AI estate, the inference subnet (your vLLM hosts), the data subnet (pgvector, datasets), and the management subnet each get their own zone, and the default answer to “can A talk to B?” is no until you write the rule that says yes.

# A minimal, explicit deny-east-west posture (nftables sketch — adapt, don't paste blind):
#   inference hosts may reach the vector DB on 5432 and nothing else inbound from the app tier
nft add rule inet filter forward ip saddr 10.20.4.62 ip daddr 10.20.4.20 tcp dport 5432 accept
nft add rule inet filter forward ip saddr 10.20.4.0/24 ip daddr 10.20.9.0/24 drop  # app -> backup vault: never

Why does segmentation belong in a chapter about watching? Because segmentation and detection are the same discipline from two angles. A good segmentation policy is a declaration of what normal looks like — and the moment you have declared it, every violation becomes a high-signal alert. “Host in the app subnet just tried to SSH into the finance subnet” is only an anomaly if you decided, in advance, that it shouldn’t. The watchman watches against a standard, and segmentation is how you write the standard down.

Coach’s Note — The cheapest detection you will ever deploy is a drop rule with logging on a connection that should never happen. You don’t need a model to tell you the app server is scanning your domain controllers; you need a default-deny rule and a log line.


5.3 — Seeing the Wire: SNMP, Flow Logs, and Packet Capture

You cannot watch what you do not collect, and there are three altitudes of collection. Know which one answers which question.

  • SNMP / device telemetry — the counters. Interface utilization, error rates, CPU on the switch. Cheap, coarse, always-on. Answers “is the link saturated?”
  • Flow logs (NetFlow / IPFIX / sFlow, VPC Flow Logs in cloud) — the who-talked-to-whom. One record per conversation: source, destination, port, protocol, bytes, packets. No payload. This is the sweet spot for behavioral baselining, because it is small enough to keep for 30 days and rich enough to reveal a pattern. This is the data your AI watchman reads.
  • Packet capture (pcap via tcpdump/Wireshark) — the every-byte truth. Expensive, surgical, usually triggered on demand when a flow log says “look here.”
# Flow-level view of a conversation, no payload, cheap to keep:
#   one line ~= one flow record your baseline ingests
sudo tcpdump -nn -q -c 50 'host 10.20.4.62 and port 8080'

# Surgical capture to disk when a baseline flags 10.20.4.62 -> a public IP:
sudo tcpdump -nn -w /tmp/suspect.pcap 'host 10.20.4.62 and not net 10.20.0.0/16'

The sample flow files for this week — code/baseline_flows.csv and code/window_flows.csv — are exactly this flow-log altitude. Open them. The baseline carries a distilled 30-day profile per host (mean and p95 bytes out, the set of destinations it normally talks to). The window is a live hour with three anomalies hiding in it. That is the raw material for everything that follows.

A worked example of why altitude matters: DNS exfiltration. An attacker who can’t open an obvious egress channel will smuggle data inside DNS queries — encoding stolen bytes into long, weird subdomain names that get tunneled out to a resolver they control. At the SNMP altitude this is invisible (the link isn’t even busy). At the packet-capture altitude it’s obvious but you’d never know where to look. At the flow altitude it has a tell: one internal host issuing an abnormal volume of DNS queries to an unusual resolver, with high entropy in the names. The watchman who baselines DNS query rate per host catches it; the one who only blocks known-bad IPs never will. Pick your altitude to match the threat.

Coach’s Note — Keep flow logs longer than you think you need. The median dwell time of an intrusion is measured in weeks, not hours. If you only retain 24 hours of flows, you will discover the breach and have nothing to investigate it with.


5.4 — The Behavioral Baseline: What “Normal” Means

A static rule asks is this on the blocklist? A behavioral baseline asks is this normal for this host? The second question catches the attack nobody has seen before, which is the only kind that matters.

Building a baseline is honest, explainable statistics before it is ever “AI.” For each host you learn a profile: how many bytes it sends in a typical hour, its p95 (the busy-but-normal ceiling), the set of destinations it speaks to, the ports it uses, and the rhythm of its connections. Then you score a live window against that profile and flag the deviations. Three anomaly classes cover most of what an administrator catches on the wire:

  • Beaconing — a host phoning home on a fixed cadence with low variance in size. Malware check-ins look boring and regular; humans and healthy services are bursty. Tell: the same (src, dst, port) repeating every N minutes with near-identical byte counts.
  • Port scan / lateral movement — one source suddenly touching many destinations on admin ports (22, 3389, 445). Tell: fan-out to admin ports that this host never used before.
  • Exfiltration spike — bytes out far above the host’s baseline p95, often to a destination it never spoke to. Tell: a 1.8 GB upload from a host whose p95 hourly egress is 3 KB.

The reference detector for this week, code/flow_baseline.py, implements exactly these three checks in fewer than 80 lines — on purpose. You should be able to read every line of your detector, because the whole argument of this book is that you own the verdict, and you cannot own a verdict you cannot explain.

python3 code/flow_baseline.py code/baseline_flows.csv code/window_flows.csv
# [exfil    ] src=10.20.4.45    dst=198.51.100.42     ::  1887436800
# [port_scan] src=10.20.4.77    dst=6 hosts           ::  [22, 445, 3389, 5985, 5986]
# [beaconing] src=10.20.4.62    dst=203.0.113.77:8443 ::  4 hits, low jitter

Notice what the script prints last: a human confirms each before any block. That line is not decoration. It is the spine rule of the course, compiled into a detector.


5.5 — AI as the Tool: NDR and the AIOps Watch Layer

Now scale that up. The hand-rolled baseline in §5.4 is the math under a class of products you will meet in production. Network Detection and Response (NDR) platforms learn per-entity baselines across millions of flows, correlate them with identity and endpoint signals, and surface a ranked list of “this is weird.” As of 2026, the same agentic AIOps layer you will study in depth in Chapter 9 sits on top of network telemetry too: an investigation agent reads the flow logs, the firewall denies, and the DNS query history, then posts a root-cause hypothesis — “host 10.20.4.62 is beaconing to 203.0.113.77; recommend isolate and capture” — into the chat channel before a human has logged in.

The connective tissue under all of it is OpenTelemetry (OTel), which by 2026 carries network and service telemetry on the same backbone as everything else, so the watchman sees one correlated picture instead of five disconnected tools.

It helps to see the three generations of network watching side by side, because each later one layers on the earlier rather than replacing it:

GenerationAsksCatchesMisses / cost
Static rules / signatures”is this on the blocklist?“known-bad IPs, known CVE patternsanything novel; zero-day; the insider
Behavioral baseline (§5.4)“is this normal for this host?“beaconing, exfil, lateral movementneeds a clean baseline; false positives; a slow adversary under threshold
Agentic AIOps layer”what is the story across all signals?“correlated root-cause across flow + DNS + identityconfident-wrong narratives; alert fatigue; opaque reasoning

You do not pick one. A mature watch on the wire runs all three, and the administrator’s skill is knowing which one is lying in a given incident.

This is genuinely powerful. It compresses the triage that used to take an analyst an hour into a paragraph you can read in thirty seconds. And it is genuinely dangerous in three specific ways an administrator must name out loud:

  1. The confident-wrong hypothesis. The agent will write “this is benign backup traffic” in the same calm, fluent prose whether it is right or catastrophically wrong. Fluency is not evidence. The agent’s confidence is uncorrelated with its correctness, and your job is to verify the claim against the actual flows — not to ratify the paragraph.
  2. Alert fatigue and the silent watchman. Tune a baseline too tight and it screams; too loose and it sleeps. A model that flags 400 things a night is functionally a model that flags nothing, because your team learns to dismiss it. The Ezekiel problem in reverse: the horn that always blows is the same as no horn at all.
  3. The model is attackable. An adversary who understands you run a behavioral baseline will move slowly — drip exfiltration under your p95, beacon with deliberate jitter — to live below the threshold. AI on the wire raises the bar; it does not end the game.

Coach’s Note — Treat every AI-generated incident hypothesis as a witness statement, not a verdict. A witness can be sincere and wrong. You corroborate before you act, and you — not the model — sign the incident report.

The administrator’s discipline, then, is the same one this whole book teaches: let the AI do the triage, the first-pass ranking, the draft narrative. Keep for yourself the judgment — is this real, is it bad enough to act, and what is the blast radius of the action? The watchman uses better eyes. The watchman still answers for the warning.


5.6 — AI as the Workload: The Network Is the Bottleneck

Flip the chapter over. Now AI is not watching your wire — it is the traffic, and it will humble any network you didn’t design for it.

When a model fits on one GPU, networking barely matters. The moment it does not — a 70B model split across GPUs, a 405B model split across a whole node, training that synchronizes gradients across hundreds of GPUs — the GPUs must constantly exchange enormous tensors. This is east-west traffic (GPU-to-GPU), and it dwarfs the north-south traffic (users hitting your API). Get the east-west fabric wrong and you have bought a rack of the most expensive idle silicon on earth, because every GPU spends its time waiting for data instead of computing. Interconnect determines scaling, not FLOPs.

The fabric comes in layers, fastest and most local first:

TierTechnologyBandwidth (as of 2026)Scope
Intra-node, GPU↔GPUNVLink (5th-gen)up to ~1.8 TB/s per GPU bidirectionalGPUs in one server
Intra-rack, GPU↔GPUNVLink Switch / NVSwitch (GB200 NVL72)~130 TB/s aggregate in a 72-GPU domainup to 576 GPUs in one NVLink domain
Inter-nodeInfiniBand or RoCEv2 + GPUDirect RDMAhundreds of Gb/s per NICacross servers/racks
API egressordinary Ethernet / internetwhatever your uplink isusers and external calls

Two facts to carry out of that table. NVLink delivers roughly 2× the per-GPU bandwidth of the prior Hopper generation and more than 14× a PCIe Gen5 link — which is precisely why GPU servers do not just talk over PCIe. And GPUDirect RDMA lets the network card DMA directly into GPU memory, bypassing the CPU bounce buffer that would otherwise cap you; NVIDIA recommends roughly a 1:1 GPU-to-NIC ratio so each GPU has its own fast lane off the node. By 2026, disaggregated inference — splitting the prefill and decode phases onto different machines and shuttling the KV-cache between them over RDMA (NVIDIA’s NIXL transfer library, the llm-d and Dynamo projects) — made this east-west traffic a first-class design concern even for serving, not just training.

Here is where it bites the administrator, because the failure is rarely the silicon. On Kubernetes, RDMA breaks for boringly operational reasons: a missing nvidia_peermem kernel module, an absent RDMA/SR-IOV device plugin, memlock ulimits left at the default so memory registration fails under load, or RDMA GID resolution that doesn’t survive the pod’s network namespace. The check script code/gpu_fabric_check.sh walks exactly these prerequisites:

# Read the watch on the GPU cluster's wire (read-only).
bash code/gpu_fabric_check.sh
# == NVLink / intra-node fabric ==   (look for the ~1.8 TB/s/GPU links)
# == GPUDirect RDMA peer-memory module ==
#   MISSING: nvidia_peermem — RDMA will fall back to a CPU bounce buffer  <-- this is your bottleneck

When a distributed training job runs at a third of its expected throughput, the cause is far more often this list than the model. The wire, again, tells the truth first.

To make the path concrete, trace a single tensor as it travels between two GPUs in different servers during a distributed job — this is the journey every gradient and every shuttled KV-cache block makes, and every hop is a place an administrator can break it:

GPU memory (on node A) → GPUDirect RDMA: the NIC reads straight out of GPU memory, no CPU bounce buffer (this needs nvidia_peermem) → SR-IOV virtual function so a pod gets its own NIC lane → InfiniBand or RoCEv2 switch at the top of rack → back down through the peer node’s NIC → GPUDirect RDMA write directly into GPU memory (on node B). The CPU is never in the data path. That is the whole point.

Now read the prerequisite list against that path. No nvidia_peermem? The DMA can’t reach GPU memory, so traffic detours through a CPU bounce buffer and your “1.8 TB/s fabric” collapses to PCIe speeds. No RDMA device plugin or SR-IOV in the pod? The pod can’t claim a fast NIC lane at all. memlock left at the default? RDMA can’t pin the memory it needs and registration fails under load — intermittently, which is the worst way for anything to fail. Every one of these is an administrator’s responsibility, invisible to the data scientist who just sees “the job is slow.”

Coach’s Note — “We added more GPUs and it got slower” is a sentence you will hear, and it is almost always an interconnect story. More GPUs across a thin fabric means more synchronization stalls. Size the network with the GPUs, not after them. And remember the headline asymmetry: 5th-gen NVLink moves data between GPUs in the same box an order of magnitude faster than anything crossing between boxes — so where a job’s tensors physically live (intra-node vs inter-node) is itself a performance decision you make.


5.7 — Serving the API: North-South, Egress, and the Bill

Not all AI traffic is east-west. When you serve a model behind an API — your own vLLM endpoint, or a call out to a cloud model — you are back in familiar territory: a reverse proxy, TLS termination, a load balancer spreading requests across replicas, rate limiting so one tenant can’t starve the rest. The networking is ordinary. What is not ordinary is the egress.

If you call a hosted model, every token in and out crosses your uplink, and a chatty agent that makes ten model calls per user question multiplies your egress accordingly. If you self-host, you avoid the per-token network egress but inherit the east-west problem from §5.6. The administrator’s job is to make this a measured decision, not a vibe — which is exactly the cost-and-bandwidth reasoning we will formalize in Chapter 12. For this week, internalize the shape: east-west bandwidth is the scaling constraint when you run the model; egress bandwidth and token cost are the constraint when you call it. Both are network problems, and both are yours.

A reverse proxy in front of an inference service is the same nginx/Envoy pattern you already know — TLS, a health check, an upstream pool — and the load balancer’s only AI-specific wrinkle is that requests are long-lived and uneven (one prompt is 50 tokens, the next is 50,000), so naive round-robin produces ugly tail latency. Prefer least-connections or a queue-aware balancer for inference backends.


5.x — Interactive Lab: Traffic Baseline Explorer

Below this chapter on the site is the Traffic Baseline Explorer — your hands-on watchman’s wall for the week. It loads a baseline of normal flows on the left and a live window on the right, drawn from the same kind of data as code/baseline_flows.csv and code/window_flows.csv.

Your job is to play the watchman. Walk the live window and flag the anomalies before you reveal the answer — that “predict first” habit is the rep that builds the instinct. There are three to find, and they map exactly to §5.4: a beaconing host calling out on a fixed cadence with suspiciously regular byte counts, a port scan where one host fans out to admin ports it never used, and an exfiltration spike where bytes-out blow past a host’s baseline p95 to a destination it has never spoken to. Click each flow you believe is hostile, classify it, and then reveal the ground truth to score yourself.

The Explorer teaches three things the prose cannot. First, it shows you how boring an attack looks at the flow altitude — no skull-and-crossbones, just a number slightly out of place — which is why a baseline beats your eyes. Second, it lets you feel the false-positive tradeoff directly: tighten the sensitivity and watch legitimate bursts (a nightly git pull, a package-mirror sync) start tripping the horn. Third, when you reveal the answers, the Explorer shows the why for each flag, modeling the explainable-witness-statement discipline of §5.5. Do the Explorer before you start the reps; it is the warm-up that makes the rest of the week click.


5.8 — The Watchman and the Gate

“So you, son of man, I have made a watchman for the house of Israel.” — Ezekiel 33:7 (ESV)

The image God gives Ezekiel is precise, and precision is why it belongs in a networking chapter. A watchman is set on the wall not to fight the enemy but to see him coming and to sound the horn. The watchman’s competence is in perception and warning. And the assignment is grave: if the watchman sees the sword coming and does not blow the trumpet, the people’s blood is on the watchman’s hand. If the watchman does warn and the people ignore it, the blood is on their own heads. The structure of the passage is a structure of responsibility — who saw, who warned, who decided, who answers.

Map that onto the wire. The AI behavioral baseline is a magnificent set of eyes. It sees patterns across millions of flows that no human analyst could hold in their head, and it can sound the horn faster than any of us. We should be glad of it; refusing good tools is not faithfulness, it is sloth. But notice what the watchman does not do in Ezekiel 33. The watchman does not decide the city’s policy. The watchman does not unilaterally open or close the gate. The watchman reports to those who answer for the people. The eyes and the horn are delegated. The judgment and the accountability are not.

This is the whole human-in-the-loop argument, and Scripture states it more sharply than any vendor whitepaper. When you let an NDR platform watch your wire, you are deploying a watchman — and a watchman that, in 2026, can also be configured to act: to auto-quarantine a host, to push a firewall block, to kill a session. That capacity is a real gift and a real temptation. The temptation is to let the one who sees also keep the gate, because it is faster and you are tired. Ezekiel will not let you. The watching may be delegated to the wall; the keeping of the gate belongs to the one who answers for the city. An auto-block that takes down the chapel’s donation server on a Sunday morning because a generous donor’s bulk upload looked like exfiltration is your misjudgment, not the model’s — because the model was never given the authority to keep the gate. You were.

So set the watchman on the wall. Give it the best eyes you can afford. Tune the horn so it is neither deaf nor hysterical. And keep the gate yourself — or in the hand of a named human who answers for the outcome. That is not Luddism. It is the doctrine of vocation: God gives real authority to real persons who bear real accountability, and a tool, however capable, is not a person and cannot stand before the consequences. A computer can never be held accountable. The watchman’s eyes may be silicon. The watchman’s office is human.


5.9 — Common Pitfalls

Pitfall: Trusting the AI’s incident narrative because it is fluent. Example: An agent labels a 1.8 GB outbound transfer “scheduled off-site backup” in confident prose; it was exfiltration to a host the source had never contacted. Fix: Verify every hypothesis against the raw flows. Fluency is not evidence; corroborate the claim, then act.


Pitfall: Tuning the baseline so tight it cries wolf. Example: The detector flags the nightly git pull and the package-mirror sync every night; the team mutes the channel, then misses the real beacon. Fix: Baseline against p95 with a sane multiplier (the reference script uses 10×), exclude known-good periodic jobs, and measure your false-positive rate as a first-class metric.


Pitfall: Ignoring NTP until authentication breaks. Example: A node drifts 90 seconds; Kerberos tickets fail and TLS certs appear invalid, and the team chases a “certificate bug” for two hours. Fix: Monitor clock offset (chronyc tracking) and alert above a few hundred milliseconds. Time is a network service.


Pitfall: Adding GPUs without sizing the interconnect. Example: A distributed job runs at one-third of expected throughput; engineers blame the model, but nvidia_peermem was never loaded so RDMA fell back to a CPU bounce buffer. Fix: Run a fabric prerequisite check (code/gpu_fabric_check.sh) before training. Size the east-west network with the GPUs.


Pitfall: Confusing the three collection altitudes. Example: A team tries to baseline behavior from full packet captures (too expensive to retain) or to do forensics from SNMP counters (no per-flow detail). Fix: SNMP for counters, flow logs for behavioral baselining and retention, packet capture on demand when a flow points you somewhere.


Pitfall: Letting the watchman keep the gate. Example: Auto-quarantine takes the donation server offline on a Sunday because a donor’s bulk upload tripped the exfil threshold. Fix: Keep automated action behind a human approval gate for anything with real blast radius; the AI may sound the horn, a named human opens or holds the gate.


Pitfall: A flat network with no east-west segmentation. Example: A compromised web host opens SSH to the finance subnet and the backup vault; nothing said it couldn’t. Fix: Segment by function, deny east-west by default, and make “host in subnet A talking to subnet B on an admin port” a top-priority baseline alert.


5.10 — Reps

The keyboard is the gym, and this week the gym is the wire. The reps live in the exercises. A taste of what’s waiting:

  • Build a per-host baseline from a flow log by hand, then reproduce it with code/flow_baseline.py and reconcile the two.
  • Hunt the three anomalies in code/window_flows.csv — beaconing, port scan, exfil — and write the one-line “why” for each before you run the detector.
  • Time your DNS resolver, check NTP offset, and measure days-to-expiry on a TLS cert — the silent-and-global failures of §5.1.
  • Run code/gpu_fabric_check.sh (or read it line by line if you have no GPU node) and explain what each prerequisite guards against.
  • Take an AI-generated incident hypothesis, find one true claim and one unsupported claim in it, and write the corroboration step you’d run before acting.

AI policy for the reps (Phase 1, Weeks 1–8): do it by hand first, then bring in AI and grade it. You may use a copilot to draft a detector or interpret a flow, but every rep ends with your written verdict on what the AI got right and wrong. The human owns the verdict. End each rep with an honest “AI usage” line.

A short Check Your Reps quiz sits at the bottom of this page — five questions drawn straight from this chapter. Take it before you move on.


5.11 — This Week’s Project

This week’s project is P5 — “Read the Traffic”, specified in Project 5. You will take a baseline flow log and a live window, build a behavioral detector that finds beaconing, port-scan, and exfiltration anomalies, and — the part that matters most — write a defensible incident memo for each finding that a human could act on.

At a high level: the Normal tier asks you to build the per-host baseline and detector and correctly flag the three planted anomalies with evidence. The Medium tier adds a measured false-positive analysis — tune your thresholds and report the tradeoff with numbers, plus a GPU-fabric prerequisites audit. The Hard tier demands judgment an agent cannot supply: a one-page memo recommending which findings warrant an automated response and which must stay behind a human gate, and why — the watchman-and-gate decision of §5.8, graded. Use Appendix A for the lab environment and Appendix B for running a local model if you bring AI into the triage.


5.12 — Coach’s Final Word

The network is where your systems stop being a pile of computers and start being a system — and where AI shows up twice in the same breath, as the watcher on the wall and as the most demanding traffic that wall has ever carried. Learn to read the wire at all three altitudes. Learn to build a baseline you can explain line by line, because a verdict you can’t explain isn’t a verdict, it’s a guess with good lighting. And learn the east-west fabric, because the most expensive mistake in AI infrastructure is a rack of brilliant GPUs starved by a network nobody sized.

The watchman of Ezekiel 33 is given remarkable eyes and a grave assignment, and the two are not the same thing. We will gladly give the watching to a machine this week — better eyes than ours, tireless, fast. We will not give it the gate, because keeping the gate is an act of judgment, and judgment answers to someone, and a tool answers to no one. That is not a limit on the technology. It is the right ordering of it: the eyes delegated, the office retained. Set your watchman well. Then stand at the gate yourself.

See you next week.


Up next: Read the exercises and do every rep with your hands, then build Project 5 — Project 5: “Read the Traffic.” Use Appendix A for the lab, Appendix B for local + cloud AI, and Appendix C for the agentic-AI rules when you let a copilot touch your triage. Previously: Chapter 4 — Storage Administration and the Weight of Data. Then on to Chapter 6 — Virtualization and the Shape of the Machine.

Interactive Lab — Week 5
Traffic Baseline Explorer

Network Detection & Response (NDR) doesn't match signatures — it learns what "normal" looks like, then hunts for deviation. Below is a baseline for an internal subnet, then eight current flows. Flag the ones that break the baseline, then Analyze.

Learned baseline · internal subnet 10.0.4.0/24 (approx., 14-day window)
  • Talkers: hosts chat with the file server (10.0.4.10), DC (10.0.4.2), and proxy (10.0.4.3).
  • Ports: ~95% of bytes on 443/HTTPS, 445/SMB, 53/DNS, 389/LDAP.
  • DNS: a host issues roughly 50–400 queries/hr; query names are short.
  • Egress: outbound uploads are small and bursty; sustained large egress is rare.
  • Cadence: human traffic is irregular. Fixed-interval, identical-size connections are not.
Flag? Flow (src → dst:port) Volume Pattern
Try: Flag the host hitting one external IP every 60s with tiny identical connections — that cadence is beaconing (C2). But don't flag the 9 PM SMB backup to the file server just because it's big: it's on a known internal talker, a known port, on schedule. Volume alone is not an anomaly.
Check Your Reps

Check Your Reps — Network Services and the Watch on the Wire

Question 1 of 5
Which collection altitude is the right source for building a 30-day behavioral baseline of who-talked-to-whom, without storing payload?
Why: Flow logs record one small record per conversation (src, dst, port, bytes) with no payload, making them cheap enough to retain for weeks and rich enough to baseline behavior.
Question 2 of 5
In a behavioral baseline, what is the characteristic 'tell' of beaconing?
Why: Beaconing is malware phoning home: regular, low-jitter check-ins of near-identical size, which is exactly what the reference detector flags with its low-standard-deviation test.
Question 3 of 5
For distributed AI training/inference across multiple GPUs, what most determines whether the cluster scales well?
Why: GPUs constantly exchange large tensors, so the interconnect — not the math — gates scaling; a thin fabric leaves expensive GPUs stalling on synchronization.
Question 4 of 5
A distributed training job runs at one-third its expected throughput. According to the chapter, what is the most likely culprit?
Why: Slow distributed jobs are usually an interconnect story: a missing nvidia_peermem module, absent RDMA/SR-IOV device plugin, or low memlock ulimit forces traffic through a CPU bounce buffer instead of GPUDirect RDMA.
Question 5 of 5
How does the chapter's Ezekiel 33 watchman framing apply to AI-driven network detection?
Why: Ezekiel's watchman is given eyes and a horn but reports to those who answer for the city; likewise the AI can watch and warn, but the gate-keeping judgment and accountability stay with a human.
YOU FINISHED. NICE WORK.