Chapter 13 — Discussion: Making Meaning

“The first principle is that you must not fool yourself — and you are the easiest person to fool.” — Richard P. Feynman, Caltech commencement address, 1974

“but test everything; hold fast what is good.” — 1 Thessalonians 5:21 (ESV)

Why This Matters

Last week you produced numbers. This week you have to say what they mean — and that sentence is the most dangerous one in the whole paper.

The Results section is bounded. A table is a table; a confidence interval is a confidence interval; a p-value is a p-value. You did the analysis in Chapter 12 and the discipline there was honesty about what happened. The Discussion is a different animal. Here you leave the safety of the table and start making claims: our method works because…, this generalizes to…, prior work missed…. Every one of those is a place to fool yourself. Feynman’s line is not a motivational poster; it is a description of the exact failure mode that kills practicum papers in peer review. You ran the experiment, you want it to mean something, and wanting is precisely the force that bends interpretation past what the data support.

The Discussion is where a reviewer decides whether you are a scientist or a salesperson. A good Results section with an overclaiming Discussion gets rejected — and it should. The numbers were honest; the meaning-making was not. So this week you learn the single skill that separates the two: for every claim you make, you can point to the specific result that licenses it, and you can say exactly how far that license extends — and not one inch farther.

That skill maps onto our AI thread from both sides. As a tool, an LLM will happily write you a fluent, confident Discussion section in thirty seconds — and the 172-billion-token Q&A study (arXiv:2603.08274) is a standing reminder that fluency and faithfulness are different things; fabrication actually rises with longer context. The model does not know which of your claims your data support. It pattern-matches against the discussion sections of papers that did have the support, and hands you their confidence without your evidence. As a workload, if your project used an LLM as an instrument, your Discussion must reckon with construct validity — did your benchmark measure reasoning, or memorization of a contaminated test set? The human stays in the loop exactly here: the AI can draft prose, but you decide what is true, and you are accountable for every sentence.

That brings us to the chapter’s question. Paul writes, “test everything; hold fast what is good” (1 Thess 5:21, ESV). Notice the order. You do not hold fast first and test later — that is confirmation bias with a Bible verse stapled to it. You test, and only what survives the test do you hold. The discipline cuts both ways: test the result you love as hard as the one you fear, and when an inconvenient finding survives the test, you are bound to hold it fast too. We will develop this in §13.7. For now, take it as the posture of the whole chapter. The Discussion is where you decide what you are allowed to believe.

This week’s deliverable is your Discussion Section (P13) — a draft that interprets your results without overclaiming, set against the prior work in your comparison matrix. Let’s build it.

13.1 — What a Discussion Section Is For

The Results section answered what happened. The Discussion answers what does it mean — and it does four distinct jobs, in roughly this order:

1. Interpret your headline results: what the numbers say in plain language, tied to your research question.
2. Compare against prior work — explicitly, paper by paper, using the comparison matrix you started in Chapter 4.
3. Explain the surprises: the negative result, the ablation that didn’t move, the baseline that beat you on one split. Honestly.
4. Bound the claim: state limitations and revisit your threats to validity (Chapter 6) so the reader knows exactly how far your conclusion travels.

A useful test: the Results section should contain no opinions, and the Discussion should contain no new numbers. If you are introducing a fresh measurement in the Discussion, it belongs in Results. If you are editorializing in Results (“impressively, our method…”), it belongs in Discussion. Keep the wall between them clean — reviewers read for exactly this separation.

Coach’s Note — The single most common structural mistake in a first practicum draft is a Discussion that just restates the Results in sentences. “Our method achieved 0.87 F1, the baseline achieved 0.81.” That is not discussion; that is the table read aloud. Discussion starts at the word because, however, or this suggests — and every one of those words is a claim you now owe evidence for.

13.2 — Back to the Comparison Matrix

You did not start your literature review for fun. The comparison matrix — rows are papers, columns are method / dataset / metric / result / limitation — was the instrument that found your gap in Chapter 4. Now it earns its keep a second time: it is the scaffold for the comparison part of your Discussion.

Add your own work as the bottom row. Then read down each column and write the sentences that fall out:

The matrix forces the honest comparison. Notice the trap it exposes: Smith reports 38.2% from a single run; you report 41.0% ±2.1 over five seeds. You cannot write “we beat Smith” as if it were a clean win — Smith’s number has no error bars, so the comparison is not apples-to-apples, and you must say so. Bouthillier et al. 2021 (“Accounting for Variance in ML Benchmarks,” MLSys; arXiv:2103.03098) is the citation here: single-run “we beat the baseline” claims are statistically empty, and a careful Discussion names that asymmetry rather than hiding behind it.

A starter matrix you can drop your own rows into lives at code/comparison-matrix.csv. The empty cells are not decoration — an unaddressed column is still telling you where the field has not looked, and that is future-work fuel.

13.3 — Claim → Evidence: The Core Discipline

Here is the move the whole chapter turns on. Take your draft Discussion and break it into atomic claims — one assertion per sentence. For each claim, name the specific result that supports it. Not “the experiments”; the specific table cell, figure, or test.

There are exactly three verdicts:

• Supported — a result in your paper directly licenses this exact claim.
• Over-reach — the result points this direction but the claim says more than the result earns (wrong scope, missing condition, causal language over correlational evidence).
• Unsupported — there is no result in the paper for this claim at all. (This is where a hallucinated comparison or a remembered-but-unmeasured intuition sneaks in.)

Worked example, from a code-review-agent project:

The fix for over-reach is to narrow the claim until the evidence covers it — “improves resolve rate on SWE-bench Verified.” The fix for unsupported is harsher: either run the measurement and move it to Results, or delete the sentence. You do not get to keep an unsupported claim because it feels true.

Coach’s Note — Reviewers do this exact pass on your paper, line by line, looking for the gap between the claim and the result. The only question is whether you do it first. The student who claim-evidence-maps their own Discussion before submission catches the over-reach a reviewer would have caught — and a caught over-reach is a revision, while an uncaught one is a “Reject: overclaims relative to evidence.”

13.4 — Statistical vs. Practical Significance

You learned the statistics in Chapter 12. The Discussion is where they get interpreted, and the trap is treating a p-value as a meaning when it is only a screening test.

The 2016 ASA Statement on p-Values (Wasserstein & Lazar, The American Statistician 70(2):129–133) lays out six principles; the 2019 follow-up editorial (“Moving to a World Beyond p < 0.05,” Wasserstein, Schirm & Lazar) goes further and urges dropping the phrase “statistically significant” entirely. The Discussion-level consequence: report and interpret the effect size and its confidence interval, not the p-value alone.

Two failure modes to interpret around:

• Statistically significant but practically trivial. With enough seeds or samples, a 0.3% improvement can clear p < 0.05. Cohen’s d (0.2 / 0.5 / 0.8 as rough small/medium/large landmarks — Cohen himself called them arbitrary and field-dependent) or a domain-calibrated yardstick tells you whether anyone should care. A reviewer in your domain will know if 0.3% matters; your Discussion must show you know too.
• Practically interesting but statistically uncertain. A big effect with a wide CI that crosses zero is a finding worth reporting honestly as uncertain, not a win to be dressed up. “Our method shows a promising +5 point trend, though the 95% CI ([-1, +11]) does not exclude no effect; a larger run is needed.”

Write the result sentence three ways and pick the most honest:

p-value form:    "Our method significantly outperformed the baseline (p = 0.011)."
CI form:         "Our method improved resolve rate by 2.8 points (95% CI [0.7, 4.9])."
effect-size form:"The improvement was small-to-moderate (Cohen's d = 0.41, 95% CI [0.10, 0.72])."

The CI and effect-size forms are honest because they carry magnitude and uncertainty. The bare p-value hides both. For multi-seed deep-learning comparisons specifically, the deep-significance package (v1.2.5; arXiv:2204.06815) implements the Almost Stochastic Order test, which makes weaker distributional assumptions than a t-test — useful when you have, say, five seed scores per condition and a t-test’s normality assumption is a stretch.

13.5 — Honest Handling of Negative and Unexpected Results

A negative result is not a failed experiment. A fabricated positive result is a failed career. Sit with that ordering.

The temptation arrives quietly. Your method did not beat the baseline on two of three benchmarks. The honest report has three benchmarks; the dishonest report has one, with a sentence explaining why the other two were “not the focus.” That second move — quietly dropping the runs that didn’t work — is on the spectrum that ends in falsification (the F in the FFP triad: fabrication, falsification, plagiarism, 42 CFR Part 93). You logged every run with provenance in Chapter 11 precisely so this temptation has nowhere to hide.

How to discuss an unexpected result honestly:

1. Report it plainly — the surprising number, in the table, with the same provenance as the others.
2. Offer candidate explanations, labeled as hypotheses — “one possible explanation is X; we did not test this.” Do not assert the explanation as if you measured it.
3. Resist the post-hoc story. HARKing — Hypothesizing After the Results are Known (Kerr 1998) — is when you invent the hypothesis your data happened to support and present it as your original question. The Discussion is the scene of the crime. Your preregistered question (Chapter 8) is your alibi: state what you hypothesized before the data, then what you found, then what you now suspect — three clearly separated things.

Coach’s Note — Some of the most-cited papers in CS are negative results: “this widely believed thing does not hold under these conditions.” A clean negative result, honestly bounded, is publishable and valuable. A dressed-up negative result is a landmine that detonates when someone tries to build on it. Choose to be the person whose negative results others trust.

13.6 — Limitations and Threats, Revisited

You wrote a threats-to-validity table back in Chapter 6. The Discussion is where you close the loop: which threats did your design control, which remain, and how far does each shrink your conclusion?

Walk Cook & Campbell’s (1979) four categories — adapted for software engineering by Wohlin et al., Experimentation in Software Engineering — and write one honest limitation per category:

Construct validity is the live applied-AI question of the decade: does a high benchmark score measure capability, or memorization of a contaminated test set? If your project used a public benchmark as a metric, your Discussion has to address contamination as a construct threat — were the test items plausibly in the model’s training data? Naming this is not weakness; it is the mark of someone who understands what their number does and does not certify.

A limitations section is not a confession that tanks your paper. It is a map of where you stopped, which is exactly what lets the next researcher start there. Limitations stated become future work; limitations hidden become rejections.

13.x — Interactive Lab: Claim–Evidence Mapper

Below this chapter on the site is the Claim–Evidence Mapper — use it now, before you write a word of your own Discussion.

Paste in (or use the seeded example of) a draft Discussion section. The Mapper splits it into atomic claim sentences and asks you, for each one, to attach the specific piece of evidence that licenses it — a table, a figure, a test, a CI — and to label the link Supported, Over-reach, or Unsupported. It then highlights every sentence with no evidence attached and every claim whose scope exceeds its evidence: the overclaiming a reviewer will catch.

What it teaches: the physical feeling of the claim-evidence gap. When you cannot find a table to attach to a sentence, you feel — viscerally — that you wrote a check the experiment can’t cash. That feeling is the instinct you are training. Run your own draft through it for this week’s deliverable, fix every flagged sentence, and keep the export; it doubles as a reviewer-readiness checklist. The discipline is the same one in code/claim-evidence-map.txt — the widget just makes the pass fast and unmissable.

13.7 — Test Everything, Hold Fast What Is Good

Our verse is short and easy to misread: “but test everything; hold fast what is good” (1 Thess 5:21, ESV). Read carelessly, it sounds like license to keep whatever you like. Read carefully, it is the discipline of the Discussion section.

Notice the sequence Paul gives. Test comes first; hold fast comes second, and it holds fast only to what is good — what survived the test. You are not permitted to hold fast to the result you wanted and then look for a test it can pass. That inversion — decide the conclusion, then hunt for support — is HARKing and p-hacking with the steps reversed, and it is exactly the self-deception Feynman warned about in the other epigraph. The two epigraphs are saying the same thing in two languages: you are the easiest person to fool, so submit your favorite finding to the test before you embrace it.

There is a deeper LCMS note here. The verse sits inside a passage about discernment in a community — test everything, then a few verses later, abstain from every form of evil. Truth-telling about data is not a private technical nicety; it is a form of love for the people who will trust your paper and build on it. To bear a false result is, in the plainest terms, to bear false witness against your neighbor — the researcher three years from now who wastes a year because your Discussion claimed more than your data showed. The integrity that says “our 95% CI crosses zero, so we cannot claim an effect” is the same integrity that refuses to fabricate. It is humble in the right place: confident about what the data say, restrained about what you wish they said.

And the verse cuts the other way, against false modesty too. When an inconvenient result survives the test — when your method genuinely beat the baseline, or when a beloved prior assumption genuinely failed — you are bound to hold it fast. You do not bury a real finding because it is awkward, any more than you inflate one because it is convenient. Honest interpretation is the same in both directions: say what is, neither more nor less, because the matter is not finally yours to flatter. “The glory of kings is to search things out” (Prov 25:2, ESV) — and the searching is only glory if the finding is true.

13.z — Common Pitfalls

Pitfall: Restating Results as Discussion. Example: A “Discussion” section that is three paragraphs of “Method A scored X, Method B scored Y, Method C scored Z.” Fix: Every Discussion sentence must add interpretation a table can’t: why, compared to whom, what it implies, how far it goes. If a sentence could live in a table caption, it isn’t discussion.

Pitfall: Scope creep in the claim. Example: Tested on one English benchmark, concluding “our method improves code review.” The evidence covers one benchmark in one setting. Fix: Narrow the claim until the evidence covers it: “improves resolve rate on SWE-bench Verified.” Reviewers reject the broad version; the narrow version is bulletproof and still a contribution.

Pitfall: Worshiping the p-value. Example: “The improvement was statistically significant (p = 0.03)” — with no effect size, no CI, and a real-world gain of 0.2%. Fix: Lead with effect size and confidence interval (ASA 2019). State magnitude and uncertainty. A significant-but-trivial result is reported as exactly that.

Pitfall: Letting the LLM write your Discussion. Example: Pasting your tables into a chatbot and asking for “a strong discussion section.” It returns fluent prose with confident causal claims you never measured — and possibly a comparison to a paper that does not exist. Fix: Use the model for phrasing, never for claims. Claim-evidence-map every AI-touched sentence against your real tables; verify every cited comparison in dblp or Semantic Scholar. You own every word (Appendix C).

Pitfall: Quietly dropping the runs that “didn’t work.” Example: Three benchmarks tested, two unfavorable, only the favorable one discussed — with a hand-wave about “focus.” Fix: Report all three with full provenance (Chapter 11). Discuss the negative ones honestly. Dropping inconvenient runs is falsification, full stop (42 CFR Part 93).

Pitfall: A limitations section that hides instead of maps. Example: “A limitation is that more experiments could always be done” — generic, contentless, protects nothing. Fix: One concrete limitation per Cook & Campbell category, each naming a specific threat your design did not control and how it bounds the conclusion. Specific limitations build trust; vague ones signal you didn’t look.

Pitfall: Post-hoc storytelling (HARKing). Example: Your data happened to favor long inputs, so you write the Discussion as if “the effect of input length” was your research question all along. Fix: Separate three things explicitly: what you preregistered (Chapter 8), what you found, and what you now hypothesize. Label the third as a new, untested hypothesis for future work.

13.(z+1) — Reps

The reps for this week are in the exercises, and they move your project’s Discussion forward — they are not toy problems. A preview:

• Rep 1 — Add your own work as the bottom row of your comparison matrix and write the three comparison sentences that fall out.
• Rep 3 — Claim-evidence-map your own draft Discussion; classify every sentence Supported / Over-reach / Unsupported.
• Rep 5 — Rewrite your headline result three ways (p-value, CI, effect size) and pick the most honest.
• Rep 7 — Write one honest limitation per Cook & Campbell validity category.
• Rep 9 — Red-team your Discussion as a hostile reviewer: find the one sentence you’d attack first.

Do the reps before the project; they are the project’s raw material. And take the on-page Check Your Reps quiz below the chapter — five questions to confirm the concepts landed before you write.

13.(z+2) — This Week’s Deliverable

This week you produce your Discussion Section (P13), specced in Project 13. It is a draft Discussion that interprets your results against prior work, handles your surprises honestly, and bounds every claim with a stated limitation — built so that every sentence survives a claim-evidence pass.

The Hard tier asks for the judgment an AI cannot supply: a one-page interpretation memo arguing what your central finding does and does not license a reader to conclude, and why a careful skeptic should believe it. That memo is the heart of being a scientist, and it is yours alone to write.

Bring your raw results (P11), your analysis (P12), and your comparison matrix. The starter files in code/ — the matrix template, the claim-evidence map, the three-ways significance script, and the limitations checklist — are your scaffolding.

13.(z+3) — Coach’s Final Word

The Discussion is where research stops being arithmetic and starts being testimony. You are standing up in front of your field and saying here is what I found, and here is what it means. Every reviewer, and every researcher who ever builds on your work, is trusting that the meaning you assigned does not exceed the evidence you gathered.

So do the unglamorous thing. Map every claim to its evidence. Narrow the sentences that reach too far. Report the runs that hurt. State the limitations that map where you stopped. Test everything — especially the finding you love — and hold fast only to what survives. That is not timidity. That is the confidence of someone who knows their conclusion will hold, because they already tried to break it and couldn’t.

Write the Discussion you’d be proud to defend. Then defend it.

See you on Monday.

Up next: the exercises for the reps, then Project 13 for the deliverable. Then Chapter 14, where we assemble the whole draft paper. Reference: Appendix B (researcher’s toolkit), Appendix C (using AI responsibly), Appendix D (glossary).