Chapter 9 — Monitoring, Observability, and Keeping Watch

“What gets measured gets managed.” — attributed to Peter Drucker (management adage)

“And what I say to you I say to all: Stay awake.” — Mark 13:37 (ESV)

Why This Matters

Welcome to Week 9. You crossed a line at the midterm, and you should feel it. Phase 1 was do it by hand, then judge the AI. Phase 2 — the half of the course you are now in — is the AI is in the work, increasingly agentic, and you are accountable for it. From here on, every project ships an agent-log.txt. You delegate, the agent acts, and you own the verdict. Nowhere does that bargain bite harder than in monitoring, because monitoring is where you find out whether anything you built is actually true.

For most of computing history, watching a system meant knowing the questions in advance. You decided CPU and disk and error count mattered, you wired a dashboard, and you got paged when a threshold tripped. That is monitoring, and it is still necessary. But modern systems — distributed, containerized, partly nondeterministic — fail in ways you did not predict. You need to ask questions you did not pre-wire: why did checkout fail for these three users at 14:05 but not the fourth? The ability to answer arbitrary questions about a system from the data it already emits is observability, and its raw material is three streams: metrics, logs, and traces — the three pillars.

Here is where this book’s two AI threads both show up at once, and you have to hold them at the same time. AI is the tool: every major observability vendor now ships an agentic SRE that reads your telemetry and runbooks and posts a root-cause hypothesis to chat before a human even logs in — Datadog Bits AI SRE, New Relic’s SRE Agent, Microsoft’s Azure SRE Agent. You can type “why was latency high at 2am?” and get an English answer instead of writing a query. AI is also the workload: the inference services you stood up in Chapter 7 are themselves something you must now watch — token throughput, time-to-first-token, GPU utilization, KV-cache pressure — and these are not the metrics your old CPU/disk dashboard knows how to ask about.

And both threads sharpen the same blade: the AIOps agent is a confident, fast, sometimes-wrong partner. It will hand you a root-cause hypothesis with the same calm tone whether it is right or hallucinating a correlation. The natural-language query that “just works” is generating a real query underneath — and a wrong-but-plausible query returns wrong-but-plausible answers that look exactly like right ones. The watchman’s job did not get easier. It got faster and more dangerous.

Which is the week’s question. The watchman in Scripture is not asked to act; he is asked to see truly and report faithfully (Ezekiel 33; Mark 13). To keep watch is a discipline of attention — staying awake when nothing is wrong so you are not blind when something is. What does it mean to keep watch when the watching itself is increasingly done by a machine that never sleeps but also never quite understands? Hold that. We will earn it by the end.

Coach’s Note — Monitoring tells you that something is wrong. Observability lets you ask why without shipping new code first. If you can only afford one sentence on the difference, it is this: monitoring answers known questions; observability answers questions you didn’t know to ask. You need both, and the AI layer sits on top of both — it cannot conjure data your system never emitted.

9.1 — The Three Pillars: Metrics, Logs, Traces

Everything in this chapter rests on three data types. Learn what each is good at, because the most common rookie mistake is reaching for the wrong pillar and paying for it.

Metrics are your radar — cheap, always-on, great for that something is wrong. Logs are your transcript — the ground truth of what happened, but they get expensive fast, which is why you sample, structure, and tier them. Traces stitch a single request across service boundaries so you can see that the 5-second checkout spent 4.9 of those seconds waiting on the payments call.

The connective tissue across all three, as of 2026, is OpenTelemetry (OTel) — a vendor-neutral standard for emitting metrics, logs, and traces (and now, via its GenAI semantic conventions, model telemetry too). Instrument with OTel and you can change backends without re-instrumenting. That is not a small thing in a market where products get renamed and acquired every quarter — bet on the open standard, not the brand.

The teaching stack for this week lives in code/docker-compose.observability.yml: Prometheus and Grafana for metrics, OpenSearch and its dashboards for logs. Bring it up with one command:

docker compose -f code/docker-compose.observability.yml up -d
# Grafana    http://localhost:3001   (admin / admin on first run)
# Prometheus http://localhost:9090
# OpenSearch http://localhost:9200

Coach’s Note — Structure your logs. A line like payment failed for user 4419 is a sentence; a line like {"level":"ERROR","service":"payments","user":"u-4419","reason":"upstream_timeout"} is data. Structured logs are what let both a query language and an AI assistant filter precisely. Look at code/sample-logs.jsonl — every field is queryable. That is not an accident; it is the whole game.

9.2 — Metrics with Prometheus: The Pull Model

Prometheus is the de-facto open-source metrics engine. Its defining design choice is the pull model: targets expose a /metrics endpoint, and Prometheus scrapes them on an interval. Logs get pushed to you; metrics get pulled from your services. See code/prometheus.yml for a real scrape config.

global:
  scrape_interval: 15s          # sample every target every 15 seconds
scrape_configs:
  - job_name: inference
    static_configs:
      - targets: ["host.docker.internal:8000"]   # vLLM exposes /metrics

You query metrics with PromQL. A few graduate-level examples worth memorizing:

# 95th-percentile request latency over 5-minute windows, per service
histogram_quantile(0.95, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))

# error ratio for the payments service (the numerator of an SLO)
sum(rate(http_requests_total{service="payments",status=~"5.."}[5m]))
  / sum(rate(http_requests_total{service="payments"}[5m]))

The one trap that bites everyone: cardinality. Every distinct combination of label values is a separate time series. Put a user_id or a trace_id in a metric label and you will mint millions of series and melt your Prometheus. High-cardinality identifiers belong in logs and traces, never in metric labels. This is the single most expensive lesson in the chapter; learn it here for free.

9.3 — Logs and the Search Stack

When the radar (metrics) says something is wrong, you go to the transcript (logs) to find out what. The dominant open-source log stack is the ELK / OpenSearch family — an inverted-index search engine fronted by a dashboard. (OpenSearch is the Apache-2.0 fork the AWS ecosystem standardized on; Elastic’s own line, Elasticsearch + Kibana, is the original. As of mid-2026 Elastic’s current line is the 9.x series.)

The skill that matters is querying logs precisely. Given the structured lines in code/sample-logs.jsonl, the question “what failed in checkout at 14:05?” is a filter:

level: ERROR AND service: (checkout OR payments) AND ts: 2026-06-12T14:05*

That returns the cluster of upstream timeout → payment_declined lines that tell the story: Stripe started timing out at 5+ seconds, payments returned 504s, and checkout aborted for three users in a four-second window. Notice you needed two services correlated by time and by trace_id to see it. A single grep would have missed half the picture.

Coach’s Note — Sampling and retention are budget decisions disguised as engineering ones. You cannot keep every debug line forever; logging is often the single biggest line item in an observability bill. Tier it: hot/searchable for days, warm/cheap for weeks, cold/archive for compliance. Decide before the incident, because during the incident you will want everything and the meter will be running.

9.4 — SLIs, SLOs, and SLAs: Defining “Healthy”

You cannot keep watch over a thing you have not defined as healthy. Three terms, often confused, exactly distinct:

• SLI — Service Level Indicator. A measurement. “The fraction of requests served in under 300 ms.” A number you compute from metrics.
• SLO — Service Level Objective. Your internal target for that indicator. “99.9% of requests under 300 ms over 28 days.” This is the line that defines an alert.
• SLA — Service Level Agreement. A contract with a customer, with penalties. Always looser than your SLO, so you find out before they do.

The gap between your SLO and 100% is your error budget. A 99.9% availability SLO grants you roughly 43 minutes of downtime per 30-day month — a budget you spend on risky deploys and conserve when you’re close to breaching. This is the SRE idea we will build out fully in Chapter 13; here it matters because SLOs are what good alerts are made of. Alert on symptoms the user feels (SLO burn rate), not on every CPU spike. An alert that does not map to a customer-visible objective is usually noise.

9.5 — The Three Classic AIOps Jobs (Before the Agents)

Long before “agentic SRE,” AIOps platforms earned their keep on three unglamorous, well-defined jobs. Know these cold, because the flashy agent layer is built on top of them — and when the agent fails, it usually fails because one of these underneath it failed.

1. Anomaly detection. Learn a baseline of “normal” per metric and flag deviations — replacing brittle static thresholds with learned ones. Datadog’s Watchdog is the canonical ML-anomaly example.
2. Event / alert correlation (noise reduction). One root cause can fire hundreds of alerts. Correlation collapses an alert storm into one incident. BigPanda, PagerDuty AIOps, and Splunk ITSI built businesses on this.
3. Causal root-cause analysis. Walk the dependency/topology graph to find the upstream service that actually broke. Dynatrace Davis is known for a causal (not merely correlational) engine.

A fourth job, prediction/forecasting (“you will run out of disk in 9 days,” “this is trending toward a breach”), rounds out the classic set. These four — anomaly, correlation, causation, prediction — are the load-bearing walls. The natural-language and agentic features in the next section are a fifth layer wrapped around them.

Coach’s Note — “Event correlation” is where AIOps most reliably pays for itself, and it’s the least magical. Hand-correlate 500 raw alerts from one cascading failure into a single incident once (Rep 6 makes you do it) and you will understand exactly what BigPanda is automating — and why getting the topology wrong makes it confidently group unrelated failures.

9.6 — AI as the Tool: Agentic SRE and Natural-Language Query

Now the 2026 shift. The industry moved from “AI features bolted onto dashboards” to semi-autonomous SRE agents that read telemetry plus runbooks and post a root-cause hypothesis to chat before a human logs in. The reference points, as of mid-2026:

Two facts you must not gloss over. First, most of these agents are deliberately gated to “recommend, not act” on production — fully autonomous remediation is still early-access/preview territory in 2026. The vendors themselves are holding the human in the loop. Second, the billing is consumption-based and genuinely confusing: Azure’s SRE Agent bills in Azure Agent Units (AAU) — roughly 4 AAUs per agent-hour when always-on, with active investigation flows moved to token-based metering in April 2026 — while the separate Azure Monitor Observability Agent bills in Azure Agent Credit (AAC), where a single deep investigation can be capped at 500 AACs. Do not conflate those two products or those two units. When an alert storm hits and your agent investigates each alert, your bill investigates with it.

The other AI-as-tool capability is natural-language log query: you type English, the assistant emits a real query in the backend’s query language, runs it, and answers. Elastic’s AI Assistant turns English into ES|QL; Dynatrace has Davis CoPilot; Datadog has Bits. The crucial mental model — and the thing the lab below makes you feel in your hands:

Natural-language query is a layer over a real query language, not magic. The model translates English → query, your engine runs the query, and the answer is exactly as good as the query. A wrong-but-plausible query returns a wrong-but-plausible answer in the same confident voice as a right one.

The reference script code/ask_logs.py makes the architecture explicit on purpose: the model returns a JSON predicate, your code prints the predicate and then applies it itself. The model never touches the data; you can always read and audit the query it produced before you trust the answer. That print statement is not a debug aid — it is the human-in-the-loop control rendered in code.

9.7 — AI as the Workload: Observing the Models You Serve

The dashboard you built for web services does not know how to watch an LLM. The inference services from Chapter 7 emit a different vocabulary of health, and learning it is the bridge into the reliability work of Chapter 13. The metrics that matter for an AI workload:

Serving engines help you here: vLLM and friends expose a Prometheus /metrics endpoint with TTFT, throughput, and queue depth out of the box — which is exactly why code/prometheus.yml scrapes an inference target on :8000 and a gpu target (NVIDIA’s dcgm-exporter, typically :9400) for per-GPU utilization, memory, temperature, and power. Look back at code/sample-logs.jsonl: the inference lines carry gpu_util_pct: 97 and queue_depth: 18 — a model under KV-cache pressure. That is a saturation signal, and your old CPU dashboard would have called the box “healthy” the whole time because the CPU was idle while the GPU burned.

Coach’s Note — GPU utilization is the most-misread metric in AI ops. 97% util can mean “perfectly busy” or “thrashing on a too-small KV cache.” Util alone is necessary, not sufficient — pair it with TTFT and queue depth before you conclude anything. This is the AI-era version of “CPU at 100% isn’t automatically a problem”; the AI workload just hides it behind a different number.

There is a recursion worth naming. When the foundation model behind your AIOps agent slows down or its provider has an incident, your observability tool degrades — and the thing meant to watch your system is now itself a watched workload. The watcher needs watching. Hold that thought for the apologetic.

9.x — Interactive Lab: Natural-Language Log Query Console

Below this chapter on the website is an embedded interactive panel, the Natural-Language Log Query Console. Use it now — this is a hands-on section, not a reading one.

Type a plain-English question into the console — start with “why did checkout fail at 14:05?” — and watch it resolve, in two visible stages, into (1) the structured query/predicate it generated and (2) the matching log lines plus a correlated event timeline. Then do the thing the console is really built to teach: read the generated query before you read the answer. Ask a sloppy question (“what’s broken?”) and a precise one (“show ERROR lines from the payments service between 14:05:02 and 14:05:10”) and compare the queries each produces. Try a question whose answer is not in the data and see whether the console invents a plausible-looking result or correctly returns nothing — that contrast is the whole lesson.

What it teaches: natural-language query is a translation layer over a real query, and the human-in-the-loop control is your ability to inspect that translation. The console deliberately shows you the predicate so you build the habit of auditing it. It also contrasts NL query with raw keyword search so you feel where each wins — search finds strings; correlation finds stories across services and time. Carry that habit straight into code/ask_logs.py, which prints its AI-generated filter for exactly the same reason, and into Project 9, where you will be graded on whether you verified the query rather than just trusting the answer.

9.8 — The Steward and the Servant: What It Means to Keep Watch

The week’s question — what does it mean to keep watch? — is older than computing, and Scripture treats it as a vocation with a particular shape. In Ezekiel 33 the watchman is set on the wall not to fight but to see and to sound the trumpet faithfully; his guilt, if the city falls, is the guilt of having seen and not spoken, or of having slept. Jesus extends it to all of us in Mark 13:37 — “And what I say to you I say to all: Stay awake” (ESV). The watchman’s discipline is attention sustained when nothing is wrong, so that he is not blind when something is.

That is precisely the discipline monitoring asks of you, and precisely the one it is most tempting to outsource. An always-awake agent that never tires looks, at first, like the perfect watchman — it solves the one thing humans are worst at, staying alert at 3 a.m. But look at what the watchman is actually accountable for. He is accountable for seeing truly and reporting faithfully — and the AIOps agent is reliable at neither in the way a steward must be. It will report a correlation as a cause. It will translate your question into a subtly wrong query and answer it in a confident voice. It sees patterns; it does not see truth, and it cannot be held to account, because (as a 1979 IBM training note famously put it, and we will return to it in Week 15) a machine cannot be held accountable, and therefore must not make the decision.

So the LCMS frame is not “AI bad, human good.” It is vocation and the limits of delegated authority. You are a steward of a system you did not make, set to watch over what belongs, ultimately, to someone else — the users, the ministry, the people whose checkout failed at 14:05. You may delegate the watching to a tireless servant; you may not delegate the answering for it. The trumpet is yours to sound. When the agent posts its confident hypothesis to the channel, the act of keeping watch is no longer “did I notice the alert?” — the machine noticed it first. Keeping watch is now did I verify before I acted, did I read the query before I trusted the answer, did I stay awake to the ways my tireless servant is confidently wrong?

There is even a quiet mercy in the recursion we named: the watcher needs watching, the model that powers your observability is itself a workload that can fail, and so no created thing is the final watchman. The system does not, in the end, hold itself together by your vigilance or the agent’s — and an honest engineer feels the relief in that as much as the weight. You keep watch faithfully and you are not God. Both are true. Stay awake; you are not the one who never sleeps.

9.9 — Common Pitfalls

Pitfall: Confusing monitoring with observability and buying only the first. Example: A team wires beautiful CPU/memory dashboards, then a novel failure hits and they cannot answer “which users were affected and why” because they never emitted structured logs or traces. Fix: Instrument for unknown questions: structured logs with IDs, traces across service boundaries, and OpenTelemetry so you aren’t locked to one backend.

Pitfall: Putting high-cardinality identifiers in metric labels. Example: Adding user_id and trace_id as Prometheus labels mints millions of time series and brings the metrics store to its knees. Fix: Keep identifiers in logs and traces. Metric labels are for bounded dimensions (service, region, status class), never per-request IDs.

Pitfall: Trusting the AIOps root-cause hypothesis because it is confident and fast. Example: Bits AI / a SRE agent reports “root cause: payments service” and an engineer restarts payments — when the real cause was the upstream Stripe timeout payments was merely surfacing. Fix: Treat the hypothesis as a lead, not a verdict. Verify against raw telemetry and traces before you act. AI proposes; you confirm and own the call.

Pitfall: Trusting a natural-language query’s answer without reading the query. Example: “Show me the failed payments today” silently generates a filter on the wrong timezone or status field, returns 4 rows, and the on-call concludes the incident was small. Fix: Read the generated query (ES|QL/PromQL/predicate) every time — code/ask_logs.py prints it on purpose. The answer is only as right as the query under it.

Pitfall: Alerting on causes instead of symptoms. Example: Pages fire for every CPU spike and pod restart; the team tunes them out (alert fatigue), and the one alert that mapped to a real SLO breach gets ignored in the noise. Fix: Alert on SLO burn rate — user-visible symptoms. Route low-value signals to dashboards/digests, not pagers.

Pitfall: Watching the web tier but not the AI workload it now contains. Example: GPU is thrashing on KV-cache pressure, TTFT has tripled, users are furious — and every traditional dashboard shows green because CPU is idle. Fix: Add AI-native metrics: TTFT, tokens/sec, queue depth, GPU utilization + VRAM, KV-cache pressure. Scrape the serving engine’s /metrics and dcgm-exporter.

Pitfall: Ignoring the cost of the observability stack itself. Example: Log volume and an agent that investigates every alert produce a surprise five-figure monthly bill; nobody set retention tiers or AAU/AAC spend caps. Fix: Tier log retention, sample traces, and put spend caps on agentic investigation. Model the bill the way you model the workload — observability is a budget, not a free utility.

9.10 — Reps

Your conditioning for the week lives in the exercises — 9 reps plus a capstone. They are runnable and graduate-level; type the commands yourself and predict before you measure. A taste of what’s waiting:

• Stand up the Prometheus + Grafana + OpenSearch stack from code/docker-compose.observability.yml and confirm all four UIs.
• Write the PromQL for a p95-latency panel and an error-ratio SLI from scratch.
• Query code/sample-logs.jsonl by hand to reconstruct the 14:05 checkout failure — then ask an AI for the same query and diff the two.
• Hand-correlate a storm of alerts into a single incident, the way an event-correlation engine would.
• Compute an error budget and a burn-rate alert threshold from an SLO.

AI policy for the reps (Phase 2): AI is in the work now, not banned — but every rep where you use it requires you to do the manual version first or alongside, write down the AI’s query/answer, and record where it was wrong or imprecise. The verdict is yours. A short “Check Your Reps” quiz is embedded on this page; clear it before you start the project.

9.11 — This Week’s Project

Project 9 — “Ask Your Logs” (Project 9) — has you build a small but real observability pipeline and then put a natural-language query layer over your logs, in the spirit of code/ask_logs.py. You will instrument a service, ship metrics and structured logs into the Week-9 stack, answer a set of incident questions both by hand and via an AI assistant, and prove — in writing — that you verified the AI’s queries rather than trusting them.

At a high level: Normal stands up the stack and answers the incident questions with verified queries. Medium adds AI-workload observability (TTFT, queue depth, GPU/KV-cache metrics) and SLO-based alerting. Hard demands judgment an agent cannot supply — a one-page memo recommending where in your org you would let an AIOps agent act autonomously, where you would gate it behind human approval, and why, with the cost of being wrong on each side. As a Phase 2 project it requires an agent-log.txt: what you delegated, what the agent did, where it was wrong, where you intervened.

9.12 — Coach’s Final Word

Monitoring is the most honest discipline in this entire course. Everything else you build makes a claim — this is secure, this is fast, this will recover. Monitoring is where the system tells you whether the claim was true. That is why we put it here, at the hinge between the two phases: it is the practice that keeps every other practice accountable, and now it is the practice most eagerly being handed to a machine.

So take the handoff with both hands and clear eyes. Use the agent — it really will read your runbooks and beat you to the alert, and pretending otherwise is malpractice in 2026. But remember what the watchman is for. He is not the wall and he is not the army. He is the one who sees truly and reports faithfully, and that — seeing truly, reporting faithfully, answering for the call — is the one job you cannot delegate to something that cannot be held to account. The agent stays awake so you can sleep some nights. You stay awake to the agent.

Do the reps. Bring up the stack. Read the query before you trust the answer. Stay awake.

See you next week.

Up next: Read the exercises and clear the “Check Your Reps” quiz on this page, then build Project 9 — Ask Your Logs. Set up your lab with Appendix A, wire local or cloud AI with Appendix B, and read the agentic-AI ground rules in Appendix C before you delegate anything. Then on to Chapter 10 — Security Operations: The Adversary Who Disguises Himself.