Project 9 — Ask Your Logs

“And what I say to you I say to all: Stay awake.” — Mark 13:37 (ESV)

Chapter: 9 — Monitoring, Observability, and Keeping Watch Due: End of Week 9 Submit: A link to a public GitHub repository containing your observability config, your incident write-up (keeping-watch.txt), an agent-log.txt, and a REPORT.docx. See Appendix A for the lab and Appendix B for wiring local or cloud AI. Allowed tools: Docker / Docker Compose; Prometheus, Grafana, OpenSearch (the code/docker-compose.observability.yml stack); any OpenAI-compatible model endpoint (Ollama, vLLM, or a cloud API per Appendix B); Python. AI policy (Phase 2): AI is part of the work and may be agentic. An agent-log.txt is required — record what you delegated, what the agent did, where it was wrong, and where you intervened. This project is deliberately shaped so an agent cannot finish it alone: the Hard-tier memo is a judgment call no agent can make for you. Read Appendix C before you delegate.

The Setup

Cornerstone Faith Resources runs a small online store — Bibles, study guides, a few hundred SKUs — for a network of LCMS congregations and schools. Last Friday at 14:05, checkout failed for a cluster of customers for about ninety seconds, then recovered on its own. Nobody could say why until Monday, when an engineer finally grepped the logs by hand. The board has one question for IT: “Next time, can we just ask the system what happened?”

That is your charge. Stand up real observability for the storefront, reproduce the 14:05 incident, and put a natural-language query layer over the logs so the on-call can ask instead of grep — without surrendering the judgment that tells a true answer from a confident wrong one. The store also runs a small in-house LLM service (product Q&A) on a GPU, and the board wants that watched too, because it is the most expensive box in the rack.

Setup (the starter)

Everything you need is in this chapter’s code/:

• code/docker-compose.observability.yml — Prometheus + Grafana + OpenSearch + Dashboards.
• code/prometheus.yml — scrape config, including inference and gpu targets.
• code/sample-logs.jsonl — the storefront’s structured logs, including the 14:05 incident and the AI-workload lines.
• code/ask_logs.py — the natural-language-to-query bridge that prints its generated filter so you can audit it.

Fork or copy these into your repo. Bring the stack up, load the logs (see the exercises, Rep 3), and confirm all four UIs respond before you start.

Heads-up on ports. Grafana in the starter compose publishes on http://localhost:3001, not 3000 — Open WebUI from Appendix B already owns 3000. If your stack runs on a remote lab box, you won’t reach these UIs from your laptop’s browser directly: open an SSH tunnel with the ssh -L port-forward recipe in Appendix A §A.11 (The Course Toolkit), adding -L 3001:localhost:3001 for Grafana, then browse to localhost.

Learning Targets

You will demonstrate that you can:

• Stand up the three pillars (metrics + logs; bonus traces) and instrument a service into them.
• Write precise queries by hand (PromQL for metrics, a structured filter for logs) and correlate across services by time and trace_id.
• Define healthy with an SLI/SLO and build a symptom-based, burn-rate alert — not a cause-based one.
• Use an AI assistant on your telemetry and verify its generated query before trusting the answer.
• Observe an AI workload with AI-native metrics (TTFT, queue depth, GPU/KV-cache).
• Make and defend a human-in-the-loop autonomy decision an agent cannot make for you.

Normal Tier

Goal: A working observability pipeline that answers the 14:05 incident with verified queries.

Required features

1. Bring up the stack from code/docker-compose.observability.yml; load code/sample-logs.jsonl into OpenSearch; screenshot all four UIs healthy.
2. Build a Grafana dashboard (or Prometheus graph) with at least a p95-latency panel and an error-ratio panel, with the PromQL shown.
3. Answer three incident questions by hand, showing the exact query for each: why did checkout fail at 14:05? which users were affected? what was the root cause vs. the symptom? Correlate by ts and trace_id.
4. Run code/ask_logs.py (or the chapter’s Natural-Language Log Query Console) on the same questions. Paste the AI-generated filter it printed and the answer for each.
5. In keeping-watch.txt, write the verdict: for each question, did the AI’s query match yours? Where did it differ? Would you have shipped a fix on its answer alone?

Normal-tier rubric (out of 100)

Medium Tier (+up to 25% extra credit)

Goal: Watch the AI workload and define “healthy” rigorously.

1. AI-workload observability. Add a dashboard section for the in-house LLM service: TTFT, tokens/sec, queue depth, GPU utilization, and KV-cache pressure. Use the inference/gpu lines in the sample logs (and the dcgm-exporter/serving /metrics targets in code/prometheus.yml) as your signal set. Show the moment in the data where the box was “CPU-green but GPU-saturated.”
2. SLO + burn-rate alert. Define an availability SLO for checkout, compute the error budget for a 30-day window, and write a Prometheus alerting rule that fires on burn rate (symptom), not on a raw error count. Explain in one paragraph why this avoids alert fatigue.
3. Event correlation. Show, on paper or in code, how the multiple 14:05 ERROR lines collapse into a single incident, and name the signal you correlated on.

Hard Tier (+up to 25% additional extra credit)

Goal: The judgment an agent cannot supply.

Write a one-page memo to Cornerstone’s IT director titled “Where We Let the Agent Act — and Where We Don’t.” Using the real 2026 landscape (Datadog Bits AI SRE GA, New Relic SRE Agent preview/recommend-only, Azure SRE Agent’s Reader vs. privileged modes and AAU/AAC billing), recommend a concrete autonomy ladder for this org: which observability actions you would let an AIOps agent perform autonomously (e.g., summarize an alert episode), which require human approval (e.g., restart a service), and which it may never do unattended (e.g., scale down the GPU node) — and the cost of being wrong on each side. Include a short FinOps note: what happens to the bill when an agent investigates an alert storm, and what spend cap you’d set. The memo must commit to a recommendation, not survey options. This is graded on judgment, not coverage — an agent can draft it, but only you can stand behind it.

Submission

Push a public repo containing: the compose/config files, your dashboard export or screenshots, keeping-watch.txt (the incident verdict), the Hard-tier memo (if attempted), an agent-log.txt, and a REPORT.docx tying it together. Paste the repo link into the LMS. For how to get the repo onto GitHub (account, git push auth or the browser-only upload path) and the submission checklist, follow Appendix A §A.12 (Submitting Your Work) — note that this is a Phase 2 project, so the agent-log.txt is the required artifact (Phase 1 would be AI_USAGE.txt).

Hints (Read Before You Begin)

• Load the JSONL with the _bulk recipe from the exercises, Rep 3. Confirm 14 documents before you query.
• For ask_logs.py, any local model works; set temperature=0 for a query translator (it already does) and read the printed filter.
• The root cause at 14:05 is the upstream timeout, not the service that surfaced it. If your AI calls the wrong service the root cause, that is the lesson — write it down.
• Burn-rate math: a 99.9% SLO over 30 days ≈ 43 minutes of budget. A page that says “you’ll burn the month’s budget in an hour” is far more useful than “5 errors occurred.”
• GPU utilization at 97% is not automatically healthy. Pair it with TTFT and queue depth before you conclude anything.

What Mastery Looks Like (Beyond the Rubric)

A master doesn’t just get the AI to answer; they make the AI show its work and then check it. The strongest submissions read the generated query every single time, catch at least one place the AI was confidently imprecise, and say so plainly. The Hard memo, in a master’s hands, is not hedged — it draws the autonomy line, names the cost of being wrong, and signs it.

Coach’s Note — The deliverable the director will actually remember is not the dashboard. It’s the sentence in your memo where you said “the agent may recommend this, but a human approves it, and here’s what it costs us if we get that line wrong.” Dashboards are table stakes. Drawing the autonomy line and standing behind it is the job.

When You’re Done (a short checklist)

• All four UIs healthy; logs loaded and counted.
• Three incident questions answered by hand with exact queries.
• AI-generated filters captured and audited; verdict written in keeping-watch.txt.
• (Medium) AI-workload dashboard + burn-rate SLO alert.
• (Hard) Autonomy-ladder memo with FinOps note and a committed recommendation.
• agent-log.txt records what you delegated, what it got wrong, where you intervened.

A theological footnote. The watchman in Ezekiel 33 is not asked to fix the city; he is asked to see truly and sound the trumpet faithfully. This project hands the watching to a tireless servant — and then asks the one thing the servant cannot do: answer for the call. You may delegate the query; you may not delegate the verdict. When you read the AI’s filter before trusting its answer, and when you draw the autonomy line in your memo, you are doing exactly what “Stay awake” (Mark 13:37, ESV) means for an engineer — staying alert to the ways your tireless servant is confidently wrong, and keeping the accountability that a machine can never carry.

See you next week.