Project 3 — Catch the Impostor

“I am the good shepherd. I know my own and my own know me.” — John 10:14 (ESV)

Chapter: 3 — Identity and Access in a World of Synthetic Faces Due: End of Week 3 Submit: A link to a public GitHub repository containing your detector code, your tuned configuration, the labeled output, and REPORT.docx. See Appendix A for the lab and Appendix B for running a local/cloud model. Allowed tools: Python 3 (standard library is sufficient; pandas/numpy permitted), a shell, and — for the marked steps only — a local or cloud LLM. AI policy (Phase 1, AI-OFF for the building): You build the detectors by hand. AI is permitted only for the two clearly-marked critique steps (log interpretation in Medium, and drafting you then rewrite in Hard), and every use must be disclosed in an AI usage section of REPORT.docx. An anomaly pipeline you cannot explain line-by-line is not yours, and it will not pass.

The Setup

You are the lone identity-and-access administrator for Shepherd’s Gate Health Cooperative, a regional network of three clinics, a billing office, and a small data team that runs an AI charting assistant. You have one directory, a few hundred human identities, a dozen service accounts, and — new this quarter — two AI agents with their own credentials. You do not have a 24/7 SOC. You have you, a stream of authentication logs nobody is watching, and a board that just read a news story about a firm that wired US$25.6 million to deepfaked criminals and wants to know “could that happen to us?”

It could. Your job this week is to build the thing that watches the stream you can’t, tune it like an adult who knows false alarms have a cost, and write the one document the board actually needs: where we set the dial, why, and what we do when a face on a screen asks for money.

Setup (the starter)

Clone this chapter’s code/ into your repo. You are given:

• code/auth_events.csv — a realistic authentication log: ts,user,event,result,role,country,city,lat,lon,src_ip. It contains normal traffic and several planted attacks.
• code/ground_truth.csv — the labels (ts,user,label) so you can score your detectors honestly.
• code/impossible_travel.py — a working reference detector for the impossible-travel case, to read and build on (not to copy blindly).

Learning Targets

You will demonstrate that you can:

• separate authentication from authorization in a real log and detect failures of each;
• implement anomaly detectors (impossible travel, off-hours privileged access, brute-force/password-spray, privilege escalation) from first principles;
• score a detector with precision and recall against ground truth, and tune it as a deliberate tradeoff;
• design phishing-resistant and out-of-band controls against the synthetic-face threat; and
• make and defend a judgment call about sensitivity for a named organization with named stakes — the thing the machine cannot do for you.

Normal Tier

Goal: A working, self-scoring anomaly-detection pipeline over the authentication log.

Required features

1. Parser. Load code/auth_events.csv into a clean structure; handle malformed lines without crashing.
2. Impossible-travel detector. For each user, flag successive successful logins whose implied speed exceeds a configurable threshold (default 900 km/h). Use the great-circle (haversine) distance.
3. Off-hours privileged-access detector. Flag role=admin events outside a configurable business window, without flagging a legitimate nightly service account whose baseline is off-hours.
4. Brute-force / password-spray detector. Flag (a) many failures against one account and (b) few failures each across many accounts from one source IP.
5. Self-scoring. Compare your alerts to code/ground_truth.csv and print precision and recall, plus the list of false positives and false negatives by name.
6. Reproducible run. A single command (python3 catch.py code/auth_events.csv --truth code/ground_truth.csv) produces the alert list and the score.

Normal-tier rubric (out of 100)

Medium Tier (+up to 25% extra credit)

Goal: Make the pipeline act with proportion, and prove you understand the false-positive tax.

1. Privilege-escalation detector. Flag an identity that assumes a role or accesses a resource class it has never used in its baseline (a simple per-user “seen-roles” set is enough).
2. Risk-based step-up logic. Instead of binary block, output one of three dispositions per alert — allow, challenge (require step-up / passkey), block — and justify the mapping from anomaly confidence to disposition.
3. [AI step — critique] Feed ~20 log lines to a local/cloud model (Appendix B) and ask it to identify the suspicious events. In REPORT.docx, document where it agreed with your detectors, where it produced a false negative, and where it hallucinated a concern unsupported by the data. Quote the wrong line.
4. False-positive analysis. Identify at least one benign event your detector flags (e.g., a traveling VP behind a VPN exit node) and propose a tuning or allow-list that spares it without opening a hole.

State your AI usage honestly. The AI is a junior analyst whose work you check — not the author of your verdict.

Hard Tier (+up to 25% additional extra credit)

Goal: The judgment the machine cannot make. This is graded hardest.

Write two documents in REPORT.docx:

1. The CISO memo (one page). Recommend where to set detection sensitivity for Shepherd’s Gate specifically — a clinic network where a locked-out clinician at 3 a.m. delays patient care. State the precision/recall you’d actually ship, which error you chose to tolerate (false positive vs. false negative) and why, and how risk-based step-up lets you avoid the worst of both. Then write one paragraph on how your answer would change for a brokerage instead of a hospital — proving the dial is a business decision, not a technical default.
2. The deepfake-fraud playbook. A numbered, out-of-band verification runbook for the finance office that would have stopped the Arup fraud — callback to a known number, dollar threshold for dual approval, a code word — with one sentence stating plainly why “I recognized the CFO on the video call” appears nowhere as a control. Add a short note recommending a path to phishing-resistant (FIDO2 / passkey) authentication for privileged accounts.

An AI may draft prose here, but the recommendation, the chosen tradeoff, and the accountability must be yours — and REPORT.docx must show what you changed and why. A memo an agent could have written alone earns zero on this tier.

Submission

Public GitHub repo containing: your detector code, the run output (alerts + score), your tuned config/allow-list, and REPORT.docx (with the AI-usage section, and for Hard the memo + playbook). Include the exact command to reproduce your run.

Hints (Read Before You Begin)

• Do the reps first. the exercises builds every piece of this — impossible-travel arithmetic, z-scores, the confusion matrix. The capstone rep is this project in miniature.
• Haversine, not Euclidean. Latitude/longitude on a sphere; a flat-earth distance will mis-flag.
• Precision and recall, not “accuracy.” On a stream where attacks are rare, accuracy lies. Report both, and read §3.8 again.
• Spare the service account. The off-hours detector that flags your legitimate nightly batch is the false-positive tax made personal. Baseline per identity.
• Don’t over-fit to the labels. Tuning your thresholds until they exactly match ground_truth.csv teaches you nothing; a real stream has no answer key. Tune for the reasoning, then report the score honestly.

What Mastery Looks Like (Beyond the Rubric)

A passing project finds the planted attacks. A masterful one knows what it costs to find them — it reports its precision and recall without flinching, names the benign events it would harm at full sensitivity, and makes a tuning recommendation a real CISO could sign because it is tied to real stakes. The masterful submission treats the AI critique step as a chance to catch the model lying, not to outsource the thinking. And its deepfake playbook reads like something a finance office could tape to the wall.

Coach’s Note — The board’s question — “could that happen to us?” — has a true answer, and it isn’t “no, we bought a tool.” It’s “not without defeating the following named controls, here’s what they are, and here’s the one a deepfake can’t touch.” Write the memo so that answer is unmistakable. That memo is the whole job.

When You’re Done

• All four (Normal) / five (Medium) detectors run from one command.
• Precision and recall print, with FP/FN named.
• The legitimate service account is not false-flagged.
• REPORT.docx has an honest AI usage section.
• (Hard) The CISO memo names a tradeoff and the deepfake playbook names the out-of-band controls.

A theological footnote. The week’s question is what does it mean to be truly known, and to guard against the impostor? You built a system to tell the shepherd’s own from the wolf in the flock — and you learned that the only reliable signal was never appearance. The man who recognized every face in the room was robbed by faces. The Good Shepherd knows His own by something a synthetic face cannot counterfeit, and says to the convincing impostor, “I never knew you; depart from me” (Matthew 7:23, ESV). Your detectors are a small, fallible echo of a true knowing — built, tuned, and answered for by a human who stays accountable when the machine is confident and wrong. Build to know your own. Guard against the one who only looks the part.

See you next week.