Project P2 — Three Models, One Prompt

“In the beginning was the Word, and the Word was with God, and the Word was God.” — John 1:1 (ESV)

Chapter: 2 — How AI Models Actually Work Due: End of Week 2 Submit: A link to a shared folder (Google Drive / OneDrive / Dropbox) or a single combined PDF containing your filled rubric, the raw outputs, and your write-up. Non-technical — no GitHub, no code. See Appendix A for setup and submission mechanics. Allowed tools: The three major consumer assistants on their free tiers — ChatGPT, Claude, and Gemini — from any browser or phone. Optional: a fourth tool of your choice. (Tool reference: Appendix B.) AI policy (Phase 1): You are using AI to measure and compare, not to obey. You will run one identical prompt across three models and score the answers against a shared rubric with your own eyes. Every deliverable ends with an honest AI usage note. Follow the hard rule from Week 1: never paste anything confidential, personal, or client-identifying into a consumer AI tool — use the safe fictional scenario below, and read Appendix C if you’re unsure.

The Setup

Meet Marcus Bell, the one-person communications shop at Harbor Light, a mid-sized food-bank nonprofit. Marcus writes everything — donor thank-yous, grant blurbs, the monthly newsletter, the awkward “we’re over capacity” notices. His board just asked him to “start using AI to move faster,” and handed him exactly zero guidance about which AI.

Marcus has heard that “they’re all basically the same.” You are going to find out whether that’s true — not by vibes, but by running the same prompt through three different models and scoring the results on the same rubric. This is the week you stop saying “the AI said” and start saying “the model I benchmarked said,” because you will understand — from Chapter 2 — why three models trained differently give three different answers to one identical prompt.

Setup (the starter)

This chapter ships two starter files in code/:

• code/benchmark-prompts.txt — five ready-to-paste prompts of different kinds (a factual question, a summarization task, a practical-planning task, a tone-rewrite task, and a format-following task). You’ll pick one — Prompt 1 is the recommended default — and run it, unchanged, through each model.
• code/model-compare-rubric.txt — a scoring rubric with four dimensions (accuracy, usefulness, tone, and format-following; honesty about uncertainty is scored inside accuracy). This is how you turn “I liked that one” into evidence.

Copy both into your project folder. Do not submit them blank.

Learning Targets

You will demonstrate that you can:

• Benchmark like a professional — run one identical prompt across three models and score them on a shared rubric instead of first impressions.
• Explain the “why” — connect an observed difference (a hallucinated fact, a longer answer, a refused request) back to a Chapter 2 concept: training data, context window, or the fact that a model predicts the next word rather than knowing things.
• Read for honesty — notice which models admit uncertainty and which confidently invent, and prize the former.
• Own the verdict — form and defend a judgment about which model fits which kind of task.

Normal Tier

Goal: Pick one of the five benchmark prompts, run it through three models, score all three results on the shared rubric, and write up what you found and why.

Required features

Produce a single document (or folder) named p2-three-models containing:

1. outputs.txt — the raw evidence. Your one chosen prompt — any of the five in the starter file, or your own equivalent — run unedited through ChatGPT, Claude, and Gemini (3 outputs total). Paste each reply verbatim under a header naming the tool and, if shown, the model (e.g., “Claude — claude-sonnet-5, mid-2026”). Do not clean them up.
2. scorecard.xlsx (or a table) — the filled rubric. Score each of the three outputs on every rubric dimension from code/model-compare-rubric.txt, with one sentence of justification per score. Total each model.
3. The “why” column. For at least three notable differences you observed, write one sentence tying it to a Chapter 2 concept — e.g., “Model X invented a statistic because it predicts plausible next words, not facts”; “Model Y lost the thread of the long pasted passage — a context-window effect.”
4. The hallucination catch. Fill in the rubric’s fact-check log: verify every checkable claim in all three outputs yourself. Identify every fabricated fact, fake citation, or invented detail you find, and mark which model produced it. (If you want to see a model tempted into a confident wrong answer, Prompt 1 is built for exactly that — which is why it’s the recommended pick.)
5. findings.docx / .pdf — the write-up (half to one page). Which model won on your prompt? Name one kind of task where you would expect a different model to win. Include the honest AI usage note (what you asked, what any model got wrong, what you verified).

Normal-tier rubric (out of 100)

Medium Tier (+up to 25% extra credit)

Push past the benchmark. Pick one or more:

1. Add a fourth contender. Run your same prompt through one more model — a different tier of the same family (a “mini” vs. a flagship), or a fourth provider. Does a smaller/cheaper model actually lose on your task, or is it “good enough”? (This previews Week 3.)
2. Same model, twice. Run your prompt through the same model twice. Note whether the answers differ, and explain — in Chapter 2 terms — why a next-word predictor need not be identical run to run.
3. Stretch the context window. Paste a long (invented, non-confidential) document and ask each model to use a detail buried near the end. Note which models “forget” the middle — a context-window effect you can now name.
4. Run the full battery. Run all five benchmark prompts through all three models (15 outputs), score every one on the rubric, and compare how the ranking shifts by task type — does the winner on facts also win on tone and format?

Hard Tier (+up to 25% additional extra credit)

This is the judgment the machine cannot make for you, and it is graded as such.

Write recommendation.docx — a one-page memo to Marcus that makes and defends a task-by-task recommendation: not “use Model X for everything,” but which model he should reach for by kind of task (drafting a donor letter vs. answering a factual question vs. summarizing a report), and why. You must:

• Back every call with your own scorecard — cite the specific outputs and scores that justify each recommendation. (Task-by-task calls need task-by-task evidence, so the Medium-tier full-battery option is the natural foundation for this memo.)
• Name the tradeoff. Where is your top pick weak? When should Marcus deliberately switch?
• State the verification step that stays constant no matter which model he uses — you choose the tool; you own the verdict.

A memo that says “they’re all the same” fails; your own scorecard already disproves it. The grade is in the discrimination — matching a model to a task from evidence.

Submission

Put everything in one shared folder (Google Drive / OneDrive / Dropbox) or export it as one combined PDF, and submit the link per Appendix A. A tidy folder looks like:

p2-three-models/
  outputs.txt                          — 3 raw, unedited outputs (1 prompt x 3 models)
  scorecard.xlsx     (or a table)      — the filled rubric with justifications
  findings.docx      (or .pdf)         — the write-up + AI usage note
  recommendation.docx  (Hard tier)     — the task-by-task memo

Do not submit anything containing real donor names, real financials, or confidential material. The benchmark prompts are deliberately generic and safe. When in doubt, re-read Appendix C.

Hints (Read Before You Begin)

• An identical prompt, or it’s not a benchmark. Paste the exact same text into each tool. If you improve the prompt for the second model, you’re comparing your prompts, not the models.
• Capture raw output first. The unedited reply is your evidence. Editing destroys the comparison.
• The factual prompt is a trap on purpose. Prompt 1 exists to see which model invents and which admits it doesn’t know — that’s why it’s the recommended pick. Reward the honest one in your scores.
• A longer answer is not a better answer. Score usefulness, not word count.
• Watch for the “confident wrong.” Fluency is not accuracy — that’s the whole Chapter 2 lesson. A wrong fact in a beautiful sentence is still wrong.

What Mastery Looks Like (Beyond the Rubric)

A mastered submission reads like a fair experiment. Your scorecard is filled in with evidence, not stars-because-I-liked-it. You didn’t just notice that the models differ — you can say why one invented a number and another said “I’m not sure,” in the vocabulary of tokens, training, and next-word prediction you learned this week. And your Hard-tier memo doesn’t crown one champion for all time; it matches the model to the task, the way a professional actually works.

Coach’s Note — The seductive finding this week is “wow, they’re all amazing.” They are. But your job is to be the person who ran the controlled test and can tell Marcus, from a scorecard, which amazing tool to trust with which job — and which one quietly made up a statistic for his grant application. Impressed and rigorous. That’s the posture.

When You’re Done (a short checklist)

• Your one prompt run unedited on all three models; 3 outputs saved, tool/model named.
• Rubric fully scored with a justification per score.
• Fact-check log filled in; every hallucination found and attributed to the right model.
• Three+ differences each tied to a real Chapter 2 concept.
• Write-up names a winner for your prompt and one task where a different model would win.
• Honest AI usage note; no confidential data anywhere.
• (Hard) recommendation.docx gives a task-by-task call backed by your scorecard.

A theological footnote. John opens his Gospel with a claim that should make anyone who works with language models stop: “In the beginning was the Word.” The Word — Logos — is not a clever arrangement of tokens; it is a Person, the One through whom all things were made, in whom meaning itself is grounded. A language model traffics in words about words, predicting what token comes next, and it can be dazzling — but it does not know what it says, because there is no knower there. That is not a flaw you will benchmark away as the models improve; it is the difference between a mechanism that produces language and a mind that means it. This week you weighed three machines’ words against each other and against the facts. Do it as someone who knows that words carry weight because behind true words there is always a knower who is answerable for them. The model is not answerable. You are.

See you next week.