Chapter 6 — Research Questions, Hypotheses, and Variables

“A theory that cannot be refuted by any conceivable event is non-scientific. Irrefutability is not a virtue of a theory (as people often think) but a vice.” — Karl Popper, Conjectures and Refutations (1963)

“The one who states his case first seems right, until the other comes and examines him.” — Proverbs 18:17 (ESV)

Why This Matters

You spent five weeks finding ground to stand on. You read like a scientist (Week 1), found a problem worth solving (Week 2), gathered the literature (Week 3), synthesized it into a comparison matrix that surfaced a gap (Week 4), and matched a modern AI method to your chosen domain (Week 5). You now have a direction. What you do not yet have is a claim — a single sentence you could be wrong about, that an experiment could decide.

That sentence is the hinge of the whole practicum. Everything before this week was about discovering what is true. Starting this week, you commit to a claim and then spend the rest of the semester trying — honestly, with everything you have — to break it. A research question that cannot fail is not a research question. It is an advertisement.

Here is the thing most first-year graduate students get wrong, and it is the most expensive mistake in the course. They write a hypothesis they cannot lose. “LLM agents can help with code review.” Can help? Under some conditions, for some tasks, by some metric, sure — that statement is true before you run a single experiment, which is exactly why it is worthless. A claim that is true no matter what your data says is unfalsifiable, and an unfalsifiable claim teaches you nothing. Popper’s whole point, in the epigraph, is that the risk of being wrong is what gives a claim its scientific value. You want a hypothesis that sticks its neck out.

The AI thread runs straight through this. Modern AI evaluation is drowning in construct-validity problems — the live, unsettled question of whether a benchmark measures what its name says it measures. Does a high MMLU score measure reasoning, or memorization of the test set? Does “the agent fixed the bug” mean the bug is fixed, or that the hidden test happened to pass? When you operationalize a metric for your applied-AI project this week, you are stepping into the single most contested methodological frontier in the field. You will use AI as a tool here too — to brainstorm rival hypotheses, to stress-test your wording — but the judgment about what your variable actually measures is yours, and no model can make it for you.

And here is the apologetic question this chapter exists to answer: what does it mean to make a claim you could be wrong about? Proverbs 18:17 says the one who states his case first seems right — until the other comes and examines him. The whole machinery of falsifiability, of independent and dependent variables, of threats to validity, is the disciplined practice of being the other who examines — of cross-examining your own case before a reviewer does it for you. It is an act of intellectual humility that has deep roots in the tradition. We will develop that late in the chapter. For now, hold the question.

This week you produce four artifacts: a sharpened research question, a falsifiable hypothesis, your variables (independent, dependent, controlled), and a threats-to-validity table. That is Project 6. Let’s earn it.

6.1 — From a Topic to a Question to a Claim

There is a ladder here, and most people try to skip the middle rungs. The rungs, bottom to top:

A topic cannot be tested. A question can be answered but not necessarily measured. A hypothesis predicts. An operationalized claim is the thing you actually run. The practicum lives on the top rung. Most of the work of this week is dragging your idea up the ladder one rung at a time, and refusing to start the experiment until you are standing on the top one.

Coach’s Note — The single most common feedback I write on Week 6 drafts is “this is still a topic.” Students hand in “I will study how RAG improves database query optimization” and think they have a hypothesis. That is a topic with a verb. A hypothesis names what beats what, by what measure, by roughly how much, under what conditions. If your sentence does not contain a comparison and a metric, you are not done climbing.

A useful sanity check from Week 2 still applies: run the question through FINER — Feasible, Interesting, Novel, Ethical, Relevant. But FINER vets the question. This week’s job is the hypothesis and the variables underneath it. Different rung, sharper tools.

6.2 — Falsifiability: The Claim You Could Lose

Falsifiability is the property that there exists some possible result of your experiment that would make you say, honestly, “my hypothesis was wrong.” If no possible outcome would make you say that, the hypothesis is not falsifiable, and it is not science — it is rhetoric.

Walk the test on three example claims:

The third is uncomfortable to write because it commits you. That discomfort is the feeling of doing it right.

A practical move: write the hypothesis, then write its null alongside it. The null is the boring “no difference” claim your experiment will try to reject.

• H₁ (your claim): Retrieval-augmented review raises defect recall vs. the no-retrieval baseline.
• H₀ (the null): Retrieval-augmented review produces no difference in defect recall vs. the baseline.

If you cannot state a clean H₀, your H₁ is not yet a hypothesis. The starter file code/hypothesis-template.txt forces you to write both, side by side, plus the direction of the predicted effect.

Coach’s Note — “Falsifiable” does not mean “false.” It means refutable in principle. Your hypothesis can be true and falsifiable at the same time — in fact, that is the goal: a true claim that risked being false and survived the test. The claims to fear are the ones that could never have lost.

6.3 — Independent, Dependent, and Controlled Variables

Every experiment has a structure: you change something, you measure something, and you hold everything else steady. Those are your three families of variables.

• Independent variable (IV) — what you deliberately vary. The thing whose effect you are testing. (Retrieval on vs. off. Model A vs. model B. Three prompt strategies.)
• Dependent variable (DV) — what you measure as the outcome. The number you predict the IV will move. (Defect-detection recall. Query latency. Task success rate.)
• Controlled variables — everything you hold constant so it cannot explain the result. (Same benchmark split, same hardware, same temperature, same number of seeds, same base model.)

The discipline is brutal in its simplicity: for a clean causal claim, change one IV at a time and control the rest. If you flip retrieval on and switch from GPT-5.5 to a fine-tuned model in the same comparison, and recall goes up, you cannot say which change caused it. You have confounded your own experiment.

A worked layout for an applied-AI project:

Notice the secondary DV. A good hypothesis often predicts a tradeoff: retrieval raises recall but may raise false positives. Measuring both keeps you honest — it stops you cherry-picking the one number that flatters your method. The template code/variables-worksheet.csv is a fill-in version of this table.

Coach’s Note — Write your variables down before you touch code. The number of graduate experiments I have seen collapse because the student “just tried a few things” and then could not reconstruct which knob caused which number is depressing. The variables table is your pre-commitment. It is also the spine of the experimental-design document you write next week, Chapter 7.

6.4 — Confounds and Controls: What Else Could Explain Your Result?

A confound is a variable you did not control that moves together with your IV and could be the real cause of your result. Confounds are how true-looking results turn out to be artifacts. They are the reviewer’s favorite kill shot, and you should be hunting them before the reviewer does.

The applied-AI canon is full of them. A few you will meet:

• Benchmark contamination. Your “better” model scores higher because the benchmark leaked into its training data, not because it reasons better. (We treat this rigorously in Chapter 7; for now, know it is a confound, and that contamination-resistant benchmarks like LiveCodeBench and LiveBench exist partly to fight it.)
• Data leakage. You scaled or selected features using the whole dataset before splitting train/test, so information from the test set leaked into training. Kapoor & Narayanan (2023, Patterns) found leakage across 294 papers in 17 fields — this is not a rare beginner mistake; it is endemic.
• Prompt-tuning on the test set. You iterated your prompt until accuracy went up — on the same data you then report. You have just turned your test set into a training set.
• Run-to-run variance. Deep-learning results swing by seed, init, and data order. Bouthillier et al. (2021, MLSys) show a single-run “we beat the baseline” is statistically empty. If you run once and win by a hair, the win may be luck.

The defense against a confound is a control: you either hold it constant, or you measure it and rule it out, or you design it away. The four moves:

Coach’s Note — Here is a discipline that pays off all semester: for every result you hope to claim, write the sentence “a skeptic would say my result is really caused by ___.” Then fill the blank with the nastiest plausible confound and design the control that closes it. If you cannot fill the blank, you are not thinking hard enough; ask an AI assistant to generate five rival explanations, then you decide which are real. The AI is good at enumerating confounds and bad at knowing which one actually applies to your setup — that judgment stays with you.

6.5 — Operationalizing the Metric (and the Construct-Validity Trap)

To operationalize a variable is to replace a fuzzy concept with a concrete, measurable procedure. “Code-review quality” is a concept. “Defect-detection recall on the c-CRAB gold labels” is an operationalization. The gap between the two is where your whole experiment can quietly go wrong.

This gap is construct validity (we formalize the term in 6.6), and in applied AI it is the live wire. The field’s running debate: does a high MMLU score measure reasoning, or memorization? When you pick a metric, you are claiming “this number stands in for the thing I actually care about.” State that claim out loud, because it is contestable.

A discipline for operationalizing well:

1. Name the construct. What do you actually care about? (“Whether the agent finds real bugs.”)
2. Name the proxy. What will you measure? (“Recall against benchmark gold defects.”)
3. Name the gap. Where could the proxy and construct diverge? (“Gold labels miss subtle logic bugs; benchmark PRs may not resemble my target codebase.”)
4. State the direction of error. Does the proxy likely over- or under-state the construct?

Watch for Goodhart’s law: when a measure becomes a target, it ceases to be a good measure. If you optimize your agent to maximize a metric, the metric stops measuring the underlying quality and starts measuring “how hard you gamed the metric.” This is exactly the prompt-tuning-on-test confound from 6.5, dressed in a proverb.

Two more cautions that bite applied-AI projects specifically:

• LLMs are fluent but can be unfaithful. A 172-billion-token study (arXiv:2603.08274) finds fabrication rises with context length — the output reads confidently and is wrong. If your DV is “did the LLM answer correctly,” fluency is not correctness, and you must measure correctness against ground truth, not vibes.
• A single metric hides tradeoffs. Report a primary DV and the cost it trades against (latency, false positives, compute, tokens). One number is a sales pitch; two numbers start to be science.

6.6 — The Four Threats to Validity (Cook & Campbell)

When you claim “my result means X,” there are exactly four families of reasons you might be wrong. The canonical taxonomy is Cook & Campbell (1979), Quasi-Experimentation, adapted for our field by Wohlin et al., Experimentation in Software Engineering. Learn these four words; reviewers speak them fluently and so should you.

(Cook & Campbell’s first category is precisely “statistical conclusion validity”; “conclusion validity” is the common shorthand — use the full name in a paper.)

A few sharpening notes:

• Internal vs. external is a tension, not a checklist. Tight lab control (same model, same split, fixed seeds) buys internal validity at the cost of external validity — the more you control, the less your result resembles the messy real world. Field studies trade the other way. You cannot maximize both; you choose, and you state the choice.
• Construct validity is the applied-AI weak point. Most reviewer fights about LLM papers are construct-validity fights in disguise. Take it seriously.
• Statistical conclusion validity is mostly downstream of design. If you decide this week to run 5+ seeds and pick a real test, you have pre-paid the bill that Week 12 (data analysis) would otherwise charge you.

Your deliverable this week includes a threats-to-validity table: one or two concrete threats per category, and what you will do to mitigate each. The reference layout is code/threats-to-validity.txt.

Coach’s Note — The mark of a maturing researcher is that they can name the threats to their own work before anyone else does. The first time you read a “Threats to Validity” section in a published paper, it feels like the authors are confessing weakness. They are not. They are demonstrating that they cross-examined themselves — which is the most credible thing a researcher can do. A paper with an honest threats section is more trustworthy, not less. (This is Proverbs 18:17 in methodological clothing; hold that thought for 6.8.)

6.7 — Interactive Lab: Hypothesis & Validity Builder

Below this chapter on the website you will find the Hypothesis & Validity Builder — an interactive panel built for exactly this week’s work.

Use it like this:

1. Declare your variables. Type your independent variable, your dependent variable(s), and your controlled variables into the fields. The tool lays them out in the IV/DV/control structure from 6.3.
2. Write your hypothesis. Paste your candidate hypothesis sentence. The builder scans it for non-falsifiable wording — hedges like “can help,” “may improve,” “sometimes,” “tends to” — and flags them, because each of those words removes a way for your claim to lose.
3. Read the surfaced threats. Based on the shape of your design (one IV vs. many; benchmark vs. field data; single seed vs. multi-seed; LLM-as-metric), the panel surfaces the likely validity threats across all four Cook & Campbell categories and suggests the control for each.
4. Iterate until clean. Rewrite until the falsifiability checker is satisfied and you have a mitigation noted for every surfaced threat.

What it teaches: the builder will not write your hypothesis — it cannot, because it does not know your domain or what you actually care about. What it does is make the structural weaknesses visible so you can fix them with judgment. Treat its flags as a checklist a skeptical reviewer would run, not as a grade. Then copy the cleaned-up version into Project 6.

6.8 — A Claim You Could Be Wrong About (the Apologetic)

“The one who states his case first seems right, until the other comes and examines him.” — Proverbs 18:17 (ESV)

Step back from the variables tables and ask the question this chapter has been circling: what does it mean to make a claim you could be wrong about?

Most of human persuasion is built to avoid that exposure. The salesman, the partisan, the confident undergraduate — each states a case so it seems right and hopes no one examines it. Proverbs 18:17 names the move and then names its undoing: until the other comes and examines him. The proverb is not cynical about the first speaker. It is realistic about the human tendency to mistake the seeming of rightness for rightness itself, and it prescribes the cure — cross-examination.

Falsifiability is that cure, formalized. A falsifiable hypothesis is a case built to be examined. When you write your null hypothesis, you are inviting the other to come. When you list your own threats to validity, you are doing what the proverb says wisdom does: you are being both the one who states the case and the one who examines it. That is not weakness. In the Christian tradition it is a particular form of humility — the willingness to hold your own claim loosely enough that the truth, rather than your reputation, gets the last word.

There is a deeper root. Research, in the frame of this book, is the disciplined search for truth — the glory of kings is to search things out (Prov 25:2). But the searcher is finite and fallible. To make a claim you could be wrong about is to confess, in the structure of your method, that you are not the author of truth, only its servant — that reality is what it is independent of what you wish your data said, and your job is to conform your claim to it, not the other way around. The temptation in research is always the opposite: to make the claim safe, to hedge it past the point of refutation, to protect the self by never risking being wrong. Scripture calls that bearing false witness (a theme we return to in Chapter 11, when you face your raw data). Falsifiability is its antidote — a methodological practice of telling the truth even when the truth might cost you your hypothesis.

So when this chapter demands a claim sharp enough to lose, it is not only good methodology. It is a small act of integrity: stating your case in a form that lets the truth examine you, and submitting to the examination. The researcher who does this in the design phase, before any data exists, has already won the more important contest. The one who only seems right until examined — and knows it — has not.

Coach’s Note — I tell every cohort the same thing here. The hypothesis you are most attached to is the one you should cross-examine hardest, because attachment is exactly what makes you stop examining. If you find yourself unwilling to write the null — unwilling to name the result that would prove you wrong — that reluctance is information. It usually means you want the answer more than you want the truth. Name the null anyway. Then go find out.

6.9 — Common Pitfalls

Pitfall: The unfalsifiable hypothesis. Example: “LLM agents can improve code review under certain conditions.” Fix: Strip every hedge (“can,” “certain conditions”). Name what beats what, by which metric, by roughly how much, under which fixed conditions. If you cannot write a clean null hypothesis to go with it, it is not falsifiable yet.

Pitfall: Changing two things at once. Example: Comparing “retrieval on + new model” against “retrieval off + old model” and crediting retrieval for the gain. Fix: One IV at a time. Hold the model, data, temperature, and seeds constant; vary only retrieval. If you must vary two factors, you need a factorial design — and you must say so explicitly.

Pitfall: Optimizing the metric on the test set. Example: Tuning your prompt until accuracy climbs on the same split you then report as your result. Fix: Tune on a validation split; report on a held-out test split you touch once. Goodhart’s law: the moment the metric becomes the target, it stops measuring quality and starts measuring how hard you gamed it.

Pitfall: Confusing fluency for correctness. Example: Scoring an LLM’s output as “correct” because it reads confidently, without checking against ground truth. Fix: Operationalize correctness against a gold standard, not against how persuasive the text sounds. Fabrication rises with context length (arXiv:2603.08274) — confident and wrong is the default failure mode, not the rare one.

Pitfall: A single seed, a single run. Example: Running the experiment once, winning by 0.7%, and declaring victory. Fix: Plan multi-seed runs now (Bouthillier et al. 2021). Decide your number of seeds and your significance test this week, in the design, so Week 12’s analysis is a formality, not a scramble.

Pitfall: No threats-to-validity thinking until the paper is due. Example: Writing the threats section the night before submission, listing one vague threat to look humble. Fix: Build the four-category table now, with concrete threats and concrete mitigations. The threats you name in Week 6 are the controls you build in Week 7 — they are not paperwork, they are the design.

Pitfall: Letting the AI choose your metric. Example: Asking a chatbot “what’s the best metric for code review?” and adopting its answer without interrogating the construct gap. Fix: Use the AI to enumerate candidate metrics and their known failure modes; then you decide which proxy best captures the construct you care about, and you state the gap. The choice of what a number means is a judgment, and judgment is the human’s job.

6.10 — Reps

Open the exercises for the full set. This week’s reps move your own project from a topic to an operationalized, falsifiable claim with a threats table. They are not generic drills; every one feeds Project 6.

A preview:

• Rep 1 — Climb the ladder: write your idea at all four rungs (topic → question → hypothesis → operationalized claim).
• Rep 2 — Write your H₁ and its null H₀, side by side, with predicted direction.
• Rep 4 — Fill the IV/DV/controlled-variables table for your design (code/variables-worksheet.csv).
• Rep 6 — Operationalize your metric: construct, proxy, gap, direction of error.
• Rep 8 — Build your four-category threats-to-validity table with a mitigation each.

Do not skip the on-page Check Your Reps quiz below the chapter — five questions that confirm you can tell a falsifiable claim from a comfortable one.

6.11 — This Week’s Deliverable

The deliverable is Project 6 — “Question, Hypothesis, and Variables.” You will hand in:

1. A sharpened research question for your domain (the gap from Week 4, now a precise question).
2. A falsifiable hypothesis with its explicit null and predicted direction.
3. Your variables: independent, dependent (primary + at least one secondary), and controlled.
4. A threats-to-validity table across all four Cook & Campbell categories, with a mitigation per threat.

This artifact is not graded on its own weight, but it is load-bearing: it is the methodological core that the Research Proposal (P8, 15%) is built on, and the experimental-design document next week (Chapter 7) executes. A weak hypothesis here propagates into a weak proposal, a weak design, and a weak paper. Get it right now while it costs only a week of thinking, not a semester of compute.

Set up your tools per Appendix A; keep the artifact in your portfolio repo and start dropping these definitions into your acmart/IEEEtran draft’s methodology section (Appendix B). For AI assistance and disclosure rules, Appendix C; for unfamiliar terms, Appendix D.

6.12 — Coach’s Final Word

This is the week the practicum gets sharp. Until now you have been gathering; from here on you commit. A hypothesis is a commitment to a claim that reality is allowed to refute — and the courage to write one is the difference between a student who studies research and a researcher who does it.

Do not soften your claim to make it safe. A safe claim is a worthless claim. Write the sentence that could lose, write the null that would beat it, name the four ways you might be fooling yourself, and then build the controls that close them off. That is the entire game, and you are now playing it.

The one who states his case first seems right — until the examination. Be the one who examines yourself first, hardest, and most honestly. The truth will keep. Your hypothesis might not, and that is exactly as it should be.

See you on Monday.

Up next: Read the exercises and complete every rep, then build Project 6 — your question, hypothesis, variables, and threats table. Prev: Chapter 5. Next: Chapter 7 — designing the reproducible experiment that tests the claim you just sharpened.