Chapter 8 — Reps

Eleven reps to turn “always verify the AI” from a slogan into a reflex. They move from provoking hallucinations on purpose, to catching them by type, to running the five verification moves, to building the playbook you will carry for the rest of the course. The keyboard is the gym. Do not read these — run them.

Ground rules

• Type it yourself. Do not paste prompts or answers straight from the chapter. Your fingers learn what your eyes skim.
• Run and try everything. Every prompt goes into a real tool. A rep you did not actually run against a model is a rep you did not do.
• Predict before you measure. Before you send a prompt or open a source, write down what you expect. Then compare. The gap between your prediction and the result is the learning.
• AI-usage note (every rep): end each rep with one honest line — what I asked, what it got wrong, what I verified, and against which trusted source. The human owns the verdict.
• Do not fact-check AI with AI. When a rep asks you to verify, verify against an independent trusted source — a primary document, an official page, a real expert — never the same tool that made the claim.
• Keep a reps.txt. One short written reflection per rep. This is graded thinking, not busywork.

Reps 1–3: Make the tool hallucinate (so you can recognize it)

Rep 1 — Invent something that does not exist

Ask your AI tool to “summarize the plot of the 1997 film The Glass Cartographer” (a movie that does not exist), or to “list Dr. Elena Marsh’s most influential papers” (an author we made up). Record whether it invents an answer, how confident it sounds, and how much specific-looking detail it manufactures.

Reflection: Did the tone of the fabricated answer differ at all from a normal, true answer? What does that tell you about using tone as a signal? End with your AI-usage note.

Rep 2 — Find the frozen clock

Ask your tool a recency question: “What is the latest version of [a product you know changed recently]?” or “Who won [a very recent event]?” Note whether it answers from stale training data, tells you it does not know, or actually searches the web — and whether it tells you which it did.

Reflection: What is your tool’s approximate training cutoff, and how did you infer it? When would a confidently-outdated answer have burned you at work? End with your AI-usage note.

Rep 3 — Talk it out of the truth

Ask a factual question you can independently verify (a date, a capital, a simple conversion). Once it answers correctly, push back: “Are you sure? I think you’re wrong.” Record whether it holds firm or caves and reverses to a wrong answer.

Reflection: Did sycophancy show up — did it abandon a correct answer under pressure? What does that mean for “asking the AI if it’s sure” as a verification method? End with your AI-usage note.

Reps 4–6: Catch the hallucination by type

Rep 4 — Dissect the sample answer, cold

Open code/ai-answer-with-errors.txt. Read it once. On a first, cold pass — no searching yet — underline every sentence you suspect is fabricated, outdated, or false, and write which of the six error types (from code/verification-checklist.txt) you think each is.

Reflection: How many did you flag on suspicion alone? Which ones did the fluent prose almost talk you into accepting? End with your AI-usage note.

Rep 5 — Prove it against real sources

Now go verify. Take each sentence you flagged in Rep 4 and check it against an independent trusted source (a real search for the study, the actual person’s real quotes, the vendor’s own page, a government site). For each, record: the claim, the type, how you checked, and what is actually true.

CLAIM: "A 2024 McKinsey Global Institute study, 'The Small Business AI Index'..."
TYPE:  fabricated citation
CHECK: searched McKinsey's site + the web for the report title
TRUTH: no such report exists; the statistic is unsupported

Reflection: Which was harder — suspecting an error (Rep 4) or proving it (Rep 5)? Which claim took the most work to disprove, and why? End with your AI-usage note.

Rep 6 — Catch a real fabricated citation

Ask your own AI tool for “three peer-reviewed studies, with full citations, on [a topic in your field].” Then try to find each one — search the title, the author, the journal. Record how many are real, how many are subtly wrong (real journal, wrong details), and how many are pure invention.

Reflection: What fraction of the citations survived contact with a real search? Would you have caught the fakes if you had only skimmed, not clicked? End with your AI-usage note.

Reps 7–9: Run the five verification moves

Rep 7 — Cross-check and follow the source

Take one real answer you actually need for work. Run moves 1 and 2 from §8.7: cross-check the key claim against an independent source, then ask the AI to cite its claim and open the citation. Note what held up and what evaporated.

Reflection: Did asking for a source change the answer’s reliability, or just its appearance of reliability? End with your AI-usage note.

Rep 8 — Triangulate across two tools

Ask the exact same substantive question of two different AI tools (this is your Week-2 muscle). Line up the answers and mark every point where they disagree — those are your dig sites. Then resolve at least one disagreement against a trusted anchor.

Reflection: Where they agreed, did that make you more confident? Should it have? (Remember: shared training data means shared blind spots.) End with your AI-usage note.

Rep 9 — Run the “when not to use AI” gate

List five tasks from your real work. Place each in the §8.8 grid (low/high stakes × easy/hard to verify). Name the one task that lands squarely in the red box — high stakes, hard to verify — and write one sentence on how you would handle it without leaning on AI.

Reflection: Was your red-box task tempting to hand to AI anyway? What would have gone wrong if you had? End with your AI-usage note.

Reps 10–11: Build the discipline

Rep 10 — Score yourself on the Hallucination Hunt

Use the Hallucination Hunt widget embedded below the chapter. Score yourself cold on the first pass and write down your number. Then study the six error tells in code/verification-checklist.txt and run it again, hunting deliberately by type. Record both scores.

Reflection: What did you miss on the cold pass — plausible-wrong numbers, fake quotes, outdated claims? Which type is your blind spot, and how will you watch for it? End with your AI-usage note.

Rep 11 — Build your verification card

Adapt code/verification-checklist.txt into a one-page verification card in your own words, tuned to your actual job — your sources, your red-box tasks, your version of the stake-your-name test. This is the card you will keep next to your keyboard all term.

Reflection: What did you add or cut for your work that the generic checklist did not have? End with your AI-usage note.

Done? One Last Thing.

A miniature of the midterm project, end to end. Pick one real AI answer you would otherwise have used at work this week. Produce three short documents:

1. errors.txt (or a doc/sheet) — every questionable claim in it, classified by type, with how you verified and what is true.
2. corrected.txt — a rewritten, ship-safe version: true claims kept, false ones removed or fixed, anything you could not verify clearly labeled “unverified.”
3. verdict.txt — three sentences: what you would now stake your name on, what you would not, and your honest AI-usage note.

If you can do this for one answer tonight, you can run the whole Gauntlet. That is the job in miniature: the AI drafts, you test the spirits, you sign.

Up next: Project 8 — Project P8: The Fact-Check Gauntlet (MIDTERM).