Chapter 09 · Reps

Monitoring, Observability, and Keeping Watch — Reps

← Back to Chapter 9

Chapter 9 — Reps

The keyboard is the gym, and this week the gym is full of dashboards. These reps build the muscle to see truly: emit the three pillars, query them precisely, define healthy, and — now that you’re in Phase 2 — use an AI assistant on your telemetry while keeping the verdict in your own hands.

Ground rules

  • Type it yourself. No copy-paste of whole blocks. Your fingers learn PromQL and ES|QL-style filters by typing them.
  • Run everything. Bring up code/docker-compose.observability.yml and actually look at the UIs. A query you didn’t run is a query you don’t trust.
  • Predict before you measure. Before each query or alert fires, write down what you expect. The gap between prediction and result is the rep.
  • AI policy (Phase 2): AI is allowed and encouraged — but for every rep where you use it, do the manual version first or alongside, paste the AI’s generated query/answer into your notes, and write one line on where it was wrong, imprecise, or got lucky. The human owns the verdict. Read the AI’s query before you trust its answer.
  • No real secrets. Use placeholders. The teaching stack ships with security plugins disabled — never expose those ports.

Reps 1–3: Stand Up the Three Pillars

Rep 1 — Bring up the stack and confirm all four UIs

docker compose -f code/docker-compose.observability.yml up -d
docker compose -f code/docker-compose.observability.yml ps
curl -s localhost:9090/-/healthy        # Prometheus
curl -s localhost:9200 | head           # OpenSearch

Open Grafana (:3001), Prometheus (:9090), OpenSearch (:9200), and OpenSearch Dashboards (:5601). Reflection: which pillar does each tool serve — metrics, logs, or traces? Which pillar is not yet represented in this stack, and what would you add to get it?

Rep 2 — Make Prometheus scrape itself, then read the targets

Confirm the prometheus job in code/prometheus.yml is up under Status → Targets. The inference and gpu targets will show DOWN (nothing is serving on those ports yet). Reflection: is a DOWN target an error or expected here? Explain the difference between “the scrape failed” and “the target is unhealthy,” and why Prometheus models them separately.

Rep 3 — Ship the sample logs into OpenSearch

Bulk-load code/sample-logs.jsonl into an index and confirm the count:

# convert JSONL to the bulk format, then load (one approach; write your own)
while read -r line; do echo '{"index":{}}'; echo "$line"; done \
  < code/sample-logs.jsonl \
  | curl -s -H 'Content-Type: application/x-ndjson' \
      -XPOST 'localhost:9200/week9-logs/_bulk' --data-binary @- > /dev/null
curl -s 'localhost:9200/week9-logs/_count'

Reflection: you should have 14 documents. Why is loading structured JSON lines (vs. raw text) what makes the next reps possible at all?

Reps 4–6: Query Like You Mean It

Rep 4 — Write the p95 latency and error-ratio queries by hand

Without looking back at the chapter, write the PromQL for (a) p95 request latency per service over 5-minute windows and (b) the 5xx error ratio for the payments service. Then check yourself against §9.2.

# (a) p95 latency per service
histogram_quantile(0.95, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))

Reflection: which of these two is a candidate SLI, and why? What label would, if you added it, blow up your cardinality?

Rep 5 — Reconstruct the 14:05 checkout failure from logs alone

Using the index from Rep 3 (or grep/jq directly on the file), find every line involved in the checkout failures at 14:05. Correlate by ts and by trace_id.

jq -c 'select(.ts >= "2026-06-12T14:05:00Z" and .ts < "2026-06-12T14:05:10Z")' \
  code/sample-logs.jsonl

Reflection: write the two-sentence incident story (what failed, root cause). Note specifically why a single keyword grep for "checkout" would have missed the cause — and which field carried it.

Rep 6 — Hand-correlate an alert storm into one incident

The 14:05 failure produced multiple ERROR lines across payments and checkout. Imagine each line also fired a separate alert (6+ pages). On paper, group them into one incident with one root cause. Reflection: what signal did you use to group them (time? trace_id? service topology?)? This is exactly the job BigPanda / PagerDuty AIOps / Splunk Event iQ automate — and where it goes wrong when the topology is mis-modeled. Describe one way auto-correlation could wrongly merge two unrelated failures.

Reps 7–8: Define Healthy, Watch the AI

Rep 7 — Compute an error budget and a burn-rate threshold

For a 99.9% availability SLO over a 30-day month: (a) how many minutes of downtime is the error budget? (b) If you want a page when you’d exhaust the whole month’s budget in 1 hour at the current rate, what error ratio triggers it?

Reflection: show your arithmetic. Why do mature teams page on burn rate (budget spent per unit time) rather than on a raw error count? Tie your answer to alert fatigue.

Rep 8 — AI-assisted query vs. by-hand (the Phase 2 rep)

Point an OpenAI-compatible client at a local model and run code/ask_logs.py:

export OPENAI_BASE_URL=http://localhost:11434/v1   # Ollama, vLLM, etc.
export OPENAI_API_KEY=not-needed
python code/ask_logs.py code/sample-logs.jsonl "why did checkout fail at 14:05?"

Read the AI-generated filter it prints before you read the matching lines. Then ask it a question whose answer is not in the data (e.g. “show me the database deadlocks”). Reflection: Did the generated predicate match what you’d have written by hand in Rep 5? Where was it imprecise (wrong timezone? wrong field? too broad)? For the unanswerable question, did it correctly return nothing or invent a plausible filter? Record the verdict — this is the human-in-the-loop, written down.

Rep 9: Watch the Workload

Rep 9 — Read the AI-workload signals

Find the inference lines in code/sample-logs.jsonl. One shows gpu_util_pct: 97 and queue_depth: 18 with a kv cache pressure warning. Reflection: every traditional dashboard would call this box “healthy” — why? Name the three AI-native metrics you’d add to a Grafana dashboard to catch this, and explain why 97% GPU utilization is not by itself proof of either health or trouble.

Done? One Last Thing.

This is the project in miniature. Pick one real incident question — “why did checkout fail at 14:05?” — and answer it three ways, then judge:

  1. By hand: the precise log query/filter you wrote in Rep 5, and the answer.
  2. By AI: run code/ask_logs.py (or the embedded Natural-Language Log Query Console on the chapter page); paste the generated query and the answer.
  3. The verdict: in 3–4 sentences, state whether the AI’s query was correct, where it differed from yours, and whether you would have shipped a fix based on its answer alone. End with the one control that kept you safe: you read the query before you trusted the answer.

Write it up as keeping-watch.txt. That paragraph — the verdict, in your own words — is the muscle this whole week was building.

Up next: Project 9 — Project 9: Ask Your Logs.