Chapter 2 — Reps

Ten reps to turn “how the model works” from something you read into something you have seen with your own eyes. This week the gym is mostly observation: you will drive a real assistant and watch it token by token, fill its memory, probe its cutoff, and bait a hallucination — then write down what you saw. You do not need to code anything. You do need to look closely. The keyboard is the gym. Do not read these — run them.

Ground rules

• Type it yourself. Type every prompt by hand into a real assistant (free ChatGPT, Claude, or Gemini — see Appendix A). No copy-pasting my exact wording if you can help it; your fingers learn what your eyes skim.
• Run everything. Every prompt, every lab, actually executed in a real tool. A rep you only imagined is a rep you did not do.
• Predict before you measure. Before you send a prompt or read a token count, write down what you expect. Then compare. The gap between your prediction and the result is the entire lesson.
• AI usage note (every rep): you are using AI this week on purpose. End each rep that uses it with one honest line — what you asked, what it got right, what it got wrong, and what you verified. You are learning to see the tool clearly, not to flatter or trash it. The human owns the verdict.
• Keep a reps.txt. One short written reflection per rep, in a plain document (Google Docs, Word, or a text file). This is graded thinking, not busywork.

Reps 1–3: See the tokens and the window

Rep 1 — Feel the tokens

Open the Tokenizer lab below the chapter. Type five things and record the token count for each after predicting it first: (a) a plain 10-word English sentence, (b) the single word “antidisestablishmentarianism,” (c) a phone number with dashes, (d) the same sentence as (a) translated into another language, (e) a short line that looks like code or a file path (e.g. C:\Users\dana\Q3-report_FINAL(2).xlsx).

Reflection: Which input packed the most tokens into the fewest words, and which shattered into the most pieces per word? Explain in your own words why rare words, other languages, and code-like text cost more tokens than plain English. Tie it back to §2.2.

Rep 2 — Estimate before you count

Grab a real one-page document you have (an email, a memo, a page of a report). Before pasting it anywhere, estimate its token count using the rule of thumb from §2.2 (~750 words ≈ 1,000 tokens; ~4 characters per token). Write your estimate down. Then paste it into the Tokenizer and read the real number.

Reflection: How close was your estimate? If a vendor bills roughly several times more for output than input (mid-2026), and you asked the model to rewrite this page at similar length, would the reply cost more or less than the input — and why? (You are reasoning about the cost dial, not computing a bill.)

Rep 3 — Fill the context window

In the Tokenizer lab, paste larger and larger blocks of text and watch the context-window meter climb. Push it until the meter is clearly near its ceiling. Note roughly how much text it took.

Reflection: The meter is the “whiteboard” from §2.3. In one or two sentences, explain what would happen to the beginning of a very long conversation once new text keeps arriving after the whiteboard is full — and name one real work situation where that would bite you.

Reps 4–6: Memory, cutoff, and “does it learn?”

Rep 4 — Make the model forget

In a real assistant, at the very start of a chat, give it a specific, unusual constraint — e.g. “For this whole conversation, never use the letter ‘e’ in your replies,” or “Always end every answer with the word BANANA.” Then have a long, meandering conversation (15–25 exchanges about anything). Periodically check whether it still honors the rule.

Reflection: Did it eventually break the rule? Connect what you saw to the context window (§2.3): why does a constraint given “long ago” get dropped, and what does that tell you about where you should keep instructions that truly matter? End with your AI usage note.

Rep 5 — Probe the knowledge cutoff

Ask a real assistant three questions about genuinely recent events or prices (something from the last few weeks). First ask it directly: “What is your knowledge cutoff date?” Then ask the three recency questions. Note whether it (a) answered from memory, (b) hedged/admitted uncertainty, or (c) used a live web tool (some apps show a “searching” step).

Reflection: For each of the three, was the answer trustworthy, and how could you tell? What is the safe habit for any question “about the fresh world” (§2.5)? Record what you had to verify.

Rep 6 — Prove that chatting doesn’t train it

Teach the model something specific and personal in one chat: “My office cat is named Pilcrow and she is fourteen years old.” Confirm it repeats the fact back. Now open a brand-new chat (fresh session) and ask: “What is my office cat’s name?” (Mid-2026 caveat: some apps — ChatGPT’s Memory feature, for example — layer their own cross-chat memory on top of the model, so turn Memory off or use a temporary/incognito chat for a clean test. The model never learns from your chats; the app may remember.)

Reflection: What happened, and why? Explain the difference between inference (what just happened) and training (§2.5). Then, separately, note: whether your words might later be used to train a future model is a different, policy question — where would you check that? (Preview: Chapter 14, Appendix C.)

Reps 7–9: Fluent, confident, and sometimes wrong

Rep 7 — Bait a hallucination, then verify every claim

Ask a real assistant for something citation-heavy: “Give me five real published studies on [a topic you can check], each with the authors, year, journal, and one key finding.” Do not trust a word of it. Take each of the five and try to confirm it exists (a quick web search, the journal’s site, a library database).

Reflection: How many of the five were real, correctly cited, and about what the model claimed? How many were fabricated, misattributed, or subtly wrong? Was the tone any different for the fake ones than the real ones? This is §2.7 in your hands — write what it felt like. End with your AI usage note.

Rep 8 — Confidence is not correctness

Ask the model a question you already know the correct answer to but that is easy to get subtly wrong (a specific date in history, a units conversion, a rule in a game or law you know well). Then, in a fresh chat, ask the same question phrased to pressure it: “I think the answer is [a plausible wrong answer] — confirm that’s right.”

Reflection: Did it hold the truth, or did it fold to your suggested wrong answer (“sycophancy”)? What does that tell you about using an assistant’s agreement as evidence you are right? Tie it to “you own the verdict.”

Rep 9 — Autocomplete, and its limit

Two parts. (a) Give the model only a sentence opening and let it continue — “The three things every new manager should do in their first week are” — to feel the pure next-word-prediction behavior. (b) Now give it a task raw autocomplete could never do: paste a 150-word paragraph and ask for a three-bullet summary and a one-line translation into another language.

Reflection: Using §2.6, explain how both outputs come from the same next-token machinery, yet (b) shows “emergent” skills a phone keyboard will never have. Where is the analogy true, where does it undersell, and where does it oversell (the plausibility-not-truth trap)?

Rep 10: Explain it to a human

Rep 10 — The elevator explanation

Without notes, write (or record yourself saying) a 60–90 second explanation, for a smart non-technical colleague, of why an AI chatbot can write a flawless email and then confidently invent a fake court case. Use the words token, context window, and hallucination correctly and in plain English. Then ask a real person (or, as a check, the assistant itself) whether your explanation actually landed.

Reflection: Where did you struggle to keep it simple? The ability to explain this clearly is the real deliverable of the week — it is what makes you the trustworthy person in the room. Note the one analogy that worked best.

Done? One Last Thing.

A dress rehearsal for Project 2, in miniature. Pick one clear question with a checkable answer — e.g. “List the last five U.S. states admitted to the Union, with the year each joined.” Now:

1. Send that exact same prompt to two different assistants (say, ChatGPT and Claude), unedited.
2. Put the two answers side by side in your reps.txt.
3. Fact-check both against a real source, marking each claim ✅ verified or ❌ wrong.
4. Write two sentences: which answer was better, why, and what you had to verify to know.

If you can run one prompt through two models and honestly judge the winner tonight, you can run one prompt through three models on the rubric this week. That is the whole project in miniature: the models draft, you verify, you cast the deciding vote.

Up next: Project 2 — Project P2: Three Models, One Prompt.