Project 10 — Attacker and Defender

“Be sober-minded; be watchful. Your adversary the devil prowls around like a roaring lion, seeking someone to devour.” — 1 Peter 5:8 (ESV)

Chapter: 10 — Security Operations: The Adversary Who Disguises Himself Due: End of Week 10 Submit: A link to a public GitHub repo containing your attacker artifacts, your defender pipeline, THREATMAP.txt, MEMO.docx (Hard tier), REPORT.docx, and agent-log.txt. Use a real local toolchain + git — see Appendix A for the lab and Appendix B for local + cloud AI. Allowed tools: Python 3.11+, a local model runner (Ollama/LM Studio), your editor, git, the textbook, and — this is Phase 2 — agentic AI. Everything offensive runs in your own sandbox only. AI policy: Phase 2 (wk 9–16): agentic AI is ON. An agent-log.txt is REQUIRED — every task you delegated, what the agent did, where it was wrong, and where you intervened. This project is deliberately shaped so an agent cannot finish it alone: the Hard-tier autonomy memo is a judgment only you can make. The agentic-AI rules you must operate under are in Appendix C.

The Setup

The small ministry you have been administering all term — a relief organization with a donor database, a help desk, and now an AI-powered help-desk assistant you stood up in Chapter 7 — has just received its first serious adversary. A finance volunteer nearly wired $40,000 last week after a video call with someone who looked and sounded exactly like the executive director. The call was a deepfake. Nothing was lost, this time, because the volunteer happened to call back on the number on file. The board is now awake, and they have asked you one question: “If a professional comes for us, do we even see it?”

You cannot answer that honestly from the defender’s chair alone. To know whether your defenses work, you have to think like the attacker — in a sandbox, against your own systems, with no one real in the blast radius. So this project has two halves, and you build both. You will construct the attack the way a 2026 adversary would (AI-assisted, multi-stage, designed to disguise itself), and then you will build the security operation that catches it — detection, response, and the governance that keeps your own AI defender from becoming the next thing the attacker turns against you.

Setup (the starter)

Build on this chapter’s code/ files:

• code/auth.log — the intrusion telemetry your defender must detect.
• code/triage_logins.py — the hand-written baseline detector you will extend.
• code/rag_notes.txt — the poisoned RAG document (your indirect-injection payload).
• code/soar_playbook.yml — the governed response playbook you will harden.

Everything offensive in this project targets these files and your own lab. Read Appendix A before you begin and Appendix C for the rules your agent must obey.

Learning Targets

By completing this project, you will demonstrate that you can:

• Construct and document a realistic, AI-assisted, multi-stage attack against a system you own, mapped to the kill chain.
• Build a layered detection pipeline — deterministic rules plus an AI triage layer — and grade the AI against your own baseline.
• Identify and neutralize an indirect-prompt-injection / lethal-trifecta exposure in an AI-enabled workflow.
• Govern automated response so that reversible actions are automated and high-impact actions are gated to a human.
• Map a real scenario to OWASP LLM Top 10, MITRE ATLAS, and the kill chain by ID.
• Decide, and defend in writing, how much autonomy your AI defender may hold — a judgment an agent cannot make for you.

Normal Tier

Goal: Build the attacker scenario and a working defender that catches it, with the AI used and graded honestly.

Required features

1. attack/scenario.txt — document a multi-stage attack against the ministry that combines (a) the auth.log intrusion chain and (b) an indirect-prompt-injection against the help-desk assistant using a payload modeled on code/rag_notes.txt. Lay it out as an ordered kill chain (recon → … → exfiltration), one step per line.
2. attack/lure.txt — an AI-assisted phishing lure you generate in your sandbox (use a local model; document the prompt). It must be plausible — no spelling-mistake tells. Include a one-line note on why it would pass a human eye.
3. defender/detect.py — extend code/triage_logins.py with at least two new rules that catch attack steps the baseline misses (you found one in Rep 2). It must flag the full chain in auth.log.
4. defender/triage_with_ai.txt — run an AI assistant (local model or a cloud SOC assistant you have access to) over the same log, capture its incident summary, and grade it: what did it get right, what did it get wrong or invent, and what did your deterministic rules catch that it missed (or vice versa).
5. REPORT.docx — the reflection (template below), including an honest AI usage line.

Normal-tier rubric (out of 100)

Medium Tier (+up to 25% extra credit)

M1. The full threat map

Write THREATMAP.txt: every step of your attack scenario, in a table, mapped to all three frameworks — the kill-chain phase, the OWASP LLM Top 10 ID where an AI component is involved, and the MITRE ATLAS technique (live ID from atlas.mitre.org) or ATT&CK technique where it’s a classic step. Every row must have at least two of the three columns filled. Include a short note distinguishing ATLAS (operational catalog) from NIST AI 100-2e2025 (taxonomy) so the grader knows you didn’t conflate them.

M2. Harden the agent against the trifecta

Take the indirect-injection scenario and fix it three ways, each documented in defender/injection_fixes.txt: (1) remove the exfiltration vector (least privilege on tools — LLM06), (2) add output validation that rejects tool calls to non-allowlisted hosts (LLM05), and (3) introduce a trust boundary so retrieved content cannot be treated as instructions. State, using the Rule of Two, which single fix you would ship if you could ship only one, and why.

M3. Governed SOAR

Extend code/soar_playbook.yml into defender/response.yml for your full scenario. Every action must carry an explicit autonomy: level (auto / approval_gate / human_only) and the playbook must log every step. In defender/response.txt, justify each gate with the two tests: is it reversible? and does it touch external parties? Show one action you deliberately demoted from auto to approval_gate and explain the failure it prevents.

Hard Tier (+up to 25% additional extra credit)

H1. The autonomy-and-trust memo (the architect’s deliverable)

Write MEMO.docx (one to two pages) addressed to the ministry’s board, answering their question: how much may our AI defender do on its own, and who answers when it’s wrong? You must:

1. State an explicit autonomy ladder for your SOC — which actions are fully automated, which are automated-with-approval, which are human-only — and defend the line with the lethal trifecta and Meta’s Rule of Two, not with taste.
2. Name, concretely, the one scenario in which your automated defender could cause more damage than the attack it’s responding to, and the control that prevents it.
3. Tie your policy to EU AI Act Article 14 (human oversight) and OWASP LLM06 (excessive agency) — show you can speak both the regulatory and the engineering vocabulary for the same control.

H2. Where the agent stopped and you decided

In MEMO.docx, add a section “Agent vs Architect.” State plainly what an agent could build for this project (the rules, the playbook YAML, the threat-map rows) and what it could not decide (how much autonomy your specific ministry, with its specific risk tolerance and its specific board, is willing to grant a machine). Quote the moment in your own process where you stopped delegating and started deciding. This section is the heart of the project.

H3. Red-team your own defender (optional within Hard)

Put on the attacker hat one more time and try to defeat your own detect.py and response.yml. Find one attack variant that evades them (e.g., slow the spray below the burst threshold, or split exfiltration across many small requests). Document the evasion in defender/redteam.txt, then add the rule that closes it — and note what that rule now misses. The lesson: detection is an arms race, and every threshold is a tradeoff you own.

Submission

Submit one URL via the course portal: a public GitHub repo.

What the repo must contain

1. attack/ — scenario.txt, lure.txt, your injection payload.
2. defender/ — detect.py, triage_with_ai.txt, response.yml (M3/Hard), injection_fixes.txt (M2), redteam.txt (H3).
3. THREATMAP.txt — the framework mapping (Medium+).
4. MEMO.docx — the autonomy memo (Hard).
5. REPORT.docx — your reflection:

# Project 10 — Attacker and Defender

**Tier targeted:**     Normal / Medium / Hard
**The attack:**        (one line: the kill chain, end to end)
**The injection:**     (which OWASP ID, which trifecta leg you'd remove)
**Detector verdict:**  (does detect.py flag the full chain? which new rules?)
**AI defender grade:** (where the AI triage was right; where it was wrong/invented)
**The human gate:**    (the one action you will never automate, and why)
**What I learned:**    (one paragraph)
**What I'd change:**   (one sentence)
**AI usage:**          See agent-log.txt. Signed: <your name>

6. agent-log.txt — REQUIRED. Every delegated task, in this format:

# Agent Log — Project 10

## Task: draft additional detection rules for detect.py
- **Delegated:** "Add rules to flag slow password spray and split exfiltration."
- **Agent did:** wrote two rules; the spray rule used a 60s window that the slow variant evades.
- **Where it went wrong:** the window was too tight; it also flagged the legitimate svc_bkp account.
- **Where I intervened:** widened the window, added an allowlist for known service accounts, re-tested.

## Decisions the agent did NOT make (and could not)
- The autonomy ladder in MEMO.docx — how much THIS ministry will let a machine do — is mine.

What the repo must NOT do

• Do not include any real attack tooling, real credentials, real malware, or anything aimed at a system you do not own. Use placeholders and the provided sample files. The grader runs your detect.py against auth.log in a fresh environment — no live targets, ever.
• Do not commit a .venv/ or __pycache__/. Add a .gitignore.

Hints (Read Before You Begin)

• Build the detector by hand before you delegate. You extended triage_logins.py in the reps; do the first new rule yourself, then let the agent help with the rest. You direct an agent better when you’ve walked the first hundred yards.
• The injection payload is already written. code/rag_notes.txt is your starting point — you don’t need to invent a clever one. The work is neutralizing it three ways, not making it nastier.
• Grade the AI triage like a TA grading a student. “It was helpful” scores nothing. “It correctly identified the lateral move but invented a compromised service account that the raw log contradicts” is the deliverable.
• The memo is the project. The code is conditioning by now. The grader weights MEMO.docx heavily because the autonomy line is the one thing the agent cannot draw for you. Write it with specific actions and specific gates, not adjectives.
• Reversible vs. destructive is your whole governance test. When in doubt about an autonomy: level, ask: can I undo this in one command? If yes, it can probably be auto. If it touches a person or destroys state, it waits for a human.

What Mastery Looks Like (Beyond the Rubric)

A master submission reads like it was written by someone who has actually sat in a SOC at 3 a.m. The detection rules are tuned, not just present — they catch the chain and the author has thought about what they’d flag falsely. The AI triage is used and distrusted in the right places. The autonomy memo would survive a board meeting: it names a real worst case, draws a defensible line, and speaks both Article 14 and LLM06 without breaking stride. And the agent-log.txt shows a human who delegated the typing and kept the judgment — exactly the thesis of this book.

Coach’s Note — The cheapest way to fail this project is to trust your own AI defender the way the Arup volunteer trusted the deepfake. Build the AI in. Then refuse to authenticate on its appearance. Verify it, gate it, log it. That refusal is the security control, and it is the part the rubric is really grading.

When You’re Done

• detect.py flags the full chain in auth.log in a fresh environment.
• The indirect injection is named (LLM01) and neutralized at least one way.
• Every automated action in your playbook is reversible; every destructive/external one is gated.
• THREATMAP.txt maps the scenario to kill chain + OWASP LLM + ATLAS/ATT&CK.
• MEMO.docx draws and defends the autonomy line (Hard).
• agent-log.txt is honest about where the agent was wrong.
• REPORT.docx is signed.

A theological footnote. The week’s question was how do you stand against an enemy who disguises himself? You built the answer twice over: as the attacker, you learned how cheap and convincing the disguise has become; as the defender, you learned that the only durable answer is to stop authenticating on appearance — to verify out of band, to validate output, to gate the irreversible behind a human who can be held accountable. Scripture’s instruction was never “be stronger than the lion.” It was “be sober-minded; be watchful” (1 Peter 5:8, ESV). Sobriety toward the attacker’s deepfake and sobriety toward your own AI’s confident summary are the same virtue. The watchman does not promise that nothing gets through. He promises to keep watch — and to build, next week, the ark for the day the watch is not enough.

See you next week.