Chapter 6 — Reps

This week’s reps do one thing: they drag your own project from a topic to an operationalized, falsifiable claim with a threats-to-validity table. Every rep feeds Project 6. Do them in your portfolio repo, in order.

Ground rules.

• Work on your real domain and your real gap — the one from your Week 4 comparison matrix and Week 5 AI-method proposal. No toy examples.
• Write everything down in version control as you go. A claim you cannot reconstruct next week is a claim you did not really make.
• AI policy: you may use an AI assistant to enumerate rival hypotheses, candidate confounds, and metric failure modes, and to stress-test your wording. You may not let it choose your metric or write your hypothesis for you — those are judgments you own and must be able to defend. Log any substantive AI use per Appendix C; you will disclose it in your proposal.
• Reflection is part of every rep. A rep done in three minutes with no written thought is not done.

Rep 1 — Climb the ladder

Take your project idea and write it at all four rungs from §6.1:

TOPIC:               (your area of interest)
QUESTION:            (a specific thing you don't know — must contain a comparison)
HYPOTHESIS:          (a predicted answer — what beats what, by which metric, roughly how much)
OPERATIONALIZED:     (the hypothesis with every term defined and measurable, plus split + seeds)

Reflection: Which rung did you skip on your first try? Most people jump from topic straight to “hypothesis” and land on a topic-with-a-verb. Name the gap you had to fill.

Rep 2 — Write the null

Write your hypothesis and its null side by side, using code/hypothesis-template.txt:

H1 (your claim):      ____ achieves higher ____ than ____ on ____, predicted direction: ____
H0 (the null):        ____ produces no difference in ____ vs. ____ on ____

Reflection: Was there a moment of reluctance writing H₀ — naming the result that would prove you wrong? If so, write one sentence about why. (Reluctance is information; §6.8.)

Rep 3 — Hunt your hedges

Paste your hypothesis into the Hypothesis & Validity Builder widget on the chapter page. Record every word it flags as non-falsifiable (“can,” “may,” “tends to,” “sometimes,” “under certain conditions”). Rewrite until the falsifiability checker is clean.

Reflection: List the hedge words you used. Each one was a way for your claim to not lose. Which was the hardest to give up, and why?

Rep 4 — The variables table

Fill in code/variables-worksheet.csv for your design:

role,variable,operationalization
IV,,
DV_primary,,
DV_secondary,,
controlled,,
controlled,,
controlled,,

You must have exactly one IV for a clean causal claim, at least one secondary DV that captures a tradeoff, and at least three controlled variables (base model + version, data/split, compute).

Reflection: What is the one secondary DV that keeps you honest — the cost your method trades against? If you can’t name a tradeoff, your hypothesis may be too good to be true.

Rep 5 — The skeptic’s sentence

For your hypothesis, complete this sentence with the nastiest plausible confound:

“A skeptic would say my result is really caused by ______, not by my independent variable.”

Use an AI assistant to generate five candidate confounds; then you pick the one that actually threatens your setup and explain why the AI’s other four don’t apply.

Reflection: Which control technique from the §6.4 table (hold constant / randomize / match / measure & adjust) closes your top confound? Write the specific control.

Rep 6 — Operationalize the metric

For your primary DV, work the four-step construct discipline from §6.5:

CONSTRUCT (what you actually care about):     ____
PROXY     (what you will measure):            ____
GAP       (where proxy and construct diverge):____
DIRECTION OF ERROR (over- or under-states?):  ____

Reflection: Name your construct-validity threat in one sentence. If your DV is an LLM judging correctness, note how you guard against fluent-but-wrong outputs (§6.5).

Rep 7 — Decide seeds and test now

Before any data exists, commit to:

NUMBER OF SEEDS:        (≥ 5; why this number?)
SIGNIFICANCE APPROACH:  (which test/effect-size you'll report — placeholder is fine)
TEST SPLIT TOUCHED:     (once — name the split you will not peek at)

This is statistical-conclusion validity, pre-paid. Cite Bouthillier et al. 2021 in your reasoning (single-run wins are statistically empty).

Reflection: Why is deciding this this week — not in Week 12 — the honest move? (Hint: HARKing and p-hacking live in the gap between “I’ll decide later” and the data.)

Rep 8 — The threats-to-validity table

Build your full four-category table using code/threats-to-validity.txt. One or two concrete threats per Cook & Campbell category, each with a concrete mitigation:

| Category                 | Threat (concrete) | Mitigation |
| Statistical conclusion   |                   |            |
| Internal                 |                   |            |
| Construct                |                   |            |
| External                 |                   |            |

No generic threats (“the sample might be biased”). Every threat must be specific to your design.

Reflection: Which category is your weakest? For most applied-AI projects it’s construct or external validity. Say which and why.

Rep 9 — Internal vs. external, the deliberate choice

In two or three sentences, state where your design sits on the internal-vs-external tradeoff (§6.6) and why you chose that point. Tight benchmark control (high internal, low external) or messier field data (the reverse)?

Reflection: A reviewer will ask “does this generalize?” Write your honest one-sentence answer now, before they do.

Rep 10 — Cross-examine a published paper

Take one paper from your Week 3–4 bibliography. Find its hypothesis (often implicit) and its threats-to-validity / limitations section. Map its stated threats onto the four Cook & Campbell categories. Find one threat the authors did not name.

Reflection: Did the paper state a falsifiable claim, or hedge it? What did you learn about your own draft by reading theirs critically? (This is Proverbs 18:17 in practice — be the one who examines.)

Done? One Last Thing.

Assemble Reps 1–10 into the four artifacts your deliverable needs and run the full thing through the Hypothesis & Validity Builder one final time:

1. Sharpened research question (Rep 1).
2. Falsifiable hypothesis + null (Reps 2–3).
3. Variables table — IV, DVs, controls (Reps 4–7).
4. Threats-to-validity table (Reps 8–9), informed by your critical read of a real paper (Rep 10).

Read the whole thing aloud once. If any sentence contains a hedge that lets the claim avoid losing, cut it. If you cannot point to the result that would refute your hypothesis, you are not done. When every term is defined, every threat has a mitigation, and you can state the null without flinching — you have a research claim. That is the foundation of everything from here to the symposium.

Up next: Project 6