Chapter 12 — Analyzing the Data

“Absence of evidence is not evidence of absence.” — attributed to Carl Sagan / Martin Rees

“A false balance is an abomination to the LORD, but a just weight is his delight.” — Proverbs 11:1 (ESV)

Why This Matters

Last week you ran the experiment and you logged every number with its seed, its config, and its commit. You bore true witness to the raw record. This week you have to do the harder thing: decide what those numbers mean — and resist, with everything you have, the urge to make them say more than they do.

Here is the trap, and it is a good one, because it is invisible. You have a method you believe in. You have a baseline you want to beat. You have a deadline. And you have a spreadsheet with twelve metrics, four datasets, three random seeds, and a dozen subgroups you could slice. Somewhere in that grid, by pure chance, there is a comparison where your method “wins” with p < 0.05. If you go looking for it, you will find it. And if you report only that one, you will have lied — not with a fabricated number, but with an honest number ripped out of its context. That is p-hacking, and the field has a measured history of doing it without ever meaning to.

So this chapter is about the just weight. You will learn to ask not just “is there an effect?” (the p-value question) but “how big is it, and how sure am I?” (the effect-size and confidence-interval question). You will learn why deep-learning results that vary across random seeds make single-run “we beat the baseline” claims statistically empty (Bouthillier et al., 2021). You will learn to correct for the multiple comparisons you actually ran, and to confront the gap between a result that is statistically significant and one that is practically meaningful.

The AI thread runs both ways here, sharper than ever. As a tool, an LLM will happily write your analysis code, suggest “the right test,” and draft the prose of your Results section — and it will do all three with a confidence that is uncorrelated with correctness. It will pick a t-test where your data violates its assumptions; it will report a p-value and forget the effect size; it will narrate a story about your numbers that reads beautifully and isn’t supported by your confidence intervals. As a workload, if your study compares model variants across seeds, the variance is your data, and the wrong statistic will manufacture a finding out of noise. The spine rule does not bend: the human owns the claim. AI can run scipy.stats.permutation_test for you in a heartbeat. Whether the result means what you’re about to write — that is yours, and it is not delegable.

And underneath all of it sits the oldest balance there is. Proverbs 11:1 is not a verse about retail scales. It is a verse about whether the numbers a person reports can be trusted — whether the weight in the pan is the weight on the label. A p-value chosen from twenty is a false balance. An effect size you hid because it was embarrassingly small is a false balance. The apologetic question of this week is plain: what does a just weight and a false balance have to do with statistics? Everything. Let’s earn the right to the answer.

Coach’s Note — You are not analyzing data to win. You are analyzing data to find out what is true and report it so a stranger can check you. If that sounds less exciting than “beating the baseline,” good. The exciting version is how careers end.

12.1 — The Question Behind the Numbers: Estimation, Not Just Yes/No

A p-value answers one narrow question: if there were truly no effect, how surprising would data this extreme be? It does not tell you the effect is large, important, real, or reproducible. The American Statistical Association said this officially. The 2016 ASA Statement on p-Values (Wasserstein & Lazar, The American Statistician 70(2):129–133) lays out six principles, two of which you should tattoo on your forearm:

• A p-value does not measure the probability that the hypothesis is true, nor the size of an effect.
• “Statistical significance” is not the same as scientific or practical importance.

The follow-up — the 2019 editorial “Moving to a World Beyond p < 0.05” (Wasserstein, Schirm & Lazar) — went further and urged researchers to stop using the phrase “statistically significant” as a verdict at all. The replacement is estimation: report how big the effect is and how uncertain you are about it. That means three numbers travel together, always:

Lead with the effect and the interval. Let the p-value bring up the rear. The moment you write a sentence whose only quantity is a p-value, you have told your reader nothing about whether the result matters.

12.2 — Effect Size: The Number That Tells You If You Should Care

A difference can be statistically significant and trivially small. With enough samples, a 0.1% accuracy gain becomes “significant.” Nobody should adopt your method for it. Effect size is the antidote.

For a difference of means, the workhorse is Cohen’s d — the difference in means divided by the pooled standard deviation. Cohen offered rough benchmarks of 0.2 (small) / 0.5 (medium) / 0.8 (large) — and Cohen himself called them arbitrary and field-dependent. Take them as a starting vocabulary, not a verdict. A 2025 CHI meta-study (Ortloff et al.) argues for field-calibrated interpretation: a “small” d in one subfield is a headline result in another. Report the number; argue its importance in your own domain’s terms.

import numpy as np

def cohens_d(a, b):
    """Standardized mean difference (pooled SD)."""
    a, b = np.asarray(a, float), np.asarray(b, float)
    na, nb = len(a), len(b)
    pooled = np.sqrt(((na - 1) * a.var(ddof=1) + (nb - 1) * b.var(ddof=1)) / (na + nb - 2))
    return (a.mean() - b.mean()) / pooled

A fuller, runnable version — with bootstrap CIs and a permutation test wired together — is in code/analyze_results.py. The discipline: never report a p-value without the effect size beside it. If a reviewer reads only your effect sizes and CIs, they should still be able to judge whether your contribution matters.

Coach’s Note — When your effect is large and significant, say so plainly and move on. When it’s significant but tiny, the honest sentence is: “the difference, though significant, is small (d = 0.14) and unlikely to matter in practice.” Writing that sentence about your own method is one of the hardest reps in this book. Do it anyway.

12.3 — Confidence Intervals and the Bootstrap

A confidence interval is the honest face of uncertainty: a range that, under repeated sampling, would contain the true value a stated fraction of the time (typically 95%). A wide CI says you don’t know much yet; that is information, not failure.

When your data is messy — non-normal, small-n, a weird metric like F1 or BLEU where analytic formulas don’t apply — reach for the bootstrap: resample your data with replacement thousands of times and look at the spread of the statistic. SciPy ships it:

from scipy.stats import bootstrap
import numpy as np

scores = np.array([0.81, 0.83, 0.79, 0.84, 0.80, 0.82, 0.78])
res = bootstrap((scores,), np.mean, confidence_level=0.95,
                n_resamples=10_000, random_state=0)
print(res.confidence_interval)   # BootstrapResult: low, high

SciPy 1.18.0 (released 2026-06-19, as of this writing) also gives you scipy.stats.permutation_test for hypothesis testing without distributional assumptions, and scipy.stats.false_discovery_control for correction. Pin the version in your environment so a reader gets the same numbers — this is the reproducibility discipline from Chapter 9 cashing out in your stats.

12.4 — Many Seeds, Not One Run: Variance Is Your Data

In deep learning, a single training run is a sample, not the answer. Results vary substantially with random seed, data ordering, and initialization. Bouthillier et al. (2021), “Accounting for Variance in Machine Learning Benchmarks” (MLSys; arXiv:2103.03098) showed that a single-run “we beat the baseline” claim is statistically empty — the variation between seeds can dwarf the difference you’re claiming. (The field’s gallows-humor companion is Picard’s “torch.manual_seed(3407) is all you need,” arXiv:2109.08203 — a paper whose whole point is that you can fish a seed that wins.)

So: run multiple seeds, report the distribution, and compare distributions — not point estimates. For comparing two methods across seed-level scores in NLP/DL, the t-test’s assumptions are often violated. The purpose-built tool is deep-significance (v1.2.5; Dror & Reichart line; arXiv:2204.06815), which implements the Almost Stochastic Order (ASO) test over multi-seed score sets with weak distributional assumptions. The recommended acceptance threshold is τ = 0.2 (lower means method A more stochastically dominates method B).

This is also where the field’s stakes become concrete. The reproducibility crisis that frames this whole course — Baker (2016), Nature 533:452–454, surveying 1,576 researchers, found ~52% believed science faced a significant reproducibility crisis and over 70% had failed to reproduce another scientist’s experiment — is not just a humanities or psychology problem. (Baker is cross-disciplinary, not CS-specific; take it as framing, not a CS measurement.) In applied ML, a large part of the irreproducibility is exactly this: results reported from one lucky seed, with no error bars, that nobody else can hit. Multi-seed reporting is how you stop being part of that statistic.

from deepsig import aso
# scores_a, scores_b: lists of test scores, one per seed (≥5 seeds recommended)
min_eps = aso(scores_a, scores_b, seed=0)
print(min_eps)        # < 0.2  → A stochastically dominant over B

Coach’s Note — “I only had compute for one run” is a real constraint, not a sin. The sin is reporting that one run as if it settled the question. If you genuinely can’t afford five seeds, say so in the limitations, weaken the claim to match, and don’t draw error bars you didn’t earn. Honesty about what you couldn’t do is itself a just weight.

12.5 — Multiple Comparisons: The Correction You Owe

Here is the arithmetic that should scare you. If you run 20 independent comparisons at α = 0.05 and nothing is real, you expect one to come up “significant” by chance. Run enough metrics × datasets × subgroups and a false positive is not a risk — it’s a guarantee. If you report that lucky one without saying how many you ran, you have used a false balance.

The fix is multiple-comparison correction. statsmodels packages the standard family:

from statsmodels.stats.multitest import multipletests
pvals = [0.001, 0.013, 0.021, 0.04, 0.30, 0.44]
reject, p_adj, _, _ = multipletests(pvals, alpha=0.05, method="fdr_bh")

FWER (family-wise error rate) asks “what’s the chance of any false positive?” — strict. FDR (false discovery rate) asks “what fraction of my discoveries are false?” — generally higher power, the right call when you’re screening many comparisons. SciPy also exposes scipy.stats.false_discovery_control (bh/by). The non-negotiable: correct for the comparisons you actually ran, and count them honestly — including the ones you ran and didn’t report.

The practical rule of thumb: a confirmatory study with two or three planned comparisons should use a FWER method (holm — it dominates plain Bonferroni at no cost). A screening study sweeping dozens of metrics or hyperparameters should use FDR (fdr_bh), and if those comparisons are correlated (the same data sliced many ways), fdr_by. Whatever you pick, report it and report the count K — “we ran K = 36 comparisons; after Holm correction, 4 remain significant” is a sentence that builds trust. Hiding K is the false balance; stating K is the just weight made arithmetic.

12.6 — p-Hacking and HARKing: The Two Honest-Person Failure Modes

Neither of these requires a villain. Both happen to careful people under deadline.

p-hacking — researcher degrees of freedom — is exploiting the many small choices in an analysis (which outcome, which subgroup, when to stop collecting data, which outliers to drop) until something crosses p < 0.05. The foundational demonstration is Simmons, Nelson & Simonsohn (2011), “False-Positive Psychology” (Psychological Science 22(11):1359–1366): they showed that ordinary, defensible flexibility lets you “find” almost any effect in random data. The widget below lets you feel this.

HARKing (Kerr, 1998) — Hypothesizing After the Results are Known — is running the analysis, seeing what came up significant, and then writing your introduction as though that’s what you predicted all along. It converts an exploratory finding (a hypothesis-generating fishing trip, which is fine and valuable) into a confirmatory one (a hypothesis-testing result, which it is not), and it inflates false positives because the “prediction” was guaranteed to match.

The cures are structural, not willpower:

• Preregister your hypotheses, primary outcome, and analysis plan before you see the data. OSF (Open Science Framework, Center for Open Science — free, time-stamped, read-only once locked) and AsPredicted.org exist for exactly this. COS also promotes Registered Reports, where a venue accepts the plan before results exist.
• Separate exploratory from confirmatory in your write-up, explicitly. Exploratory analysis is honest and useful — as long as you label it and don’t dress it up as prediction.
• Report everything you ran, not just what won. The denominator is part of the result.

Coach’s Note — The tell of p-hacking in your own work is the sentence “if I just try it this way…” said for the fourth time about the same data. The moment you notice that sentence, write down what you’ve tried. The count is the truth. Hiding the count is the lie.

12.7 — Visualization That Doesn’t Mislead

A figure is an argument. The same data can tell the truth or flatter you, depending on how you draw it. Honest defaults:

• Show uncertainty. Bars without error bars (or CI bands) are a claim with the doubt erased. Plot the seed-level spread — a strip/box/violin over the points — not just the mean.
• Don’t truncate the y-axis to inflate a difference. A bar chart starting at 0.80 instead of 0 turns a 1% gap into a cliff. If you truncate, say so and have a reason.
• Match the geometry to the data. Don’t bar-chart a distribution; show the distribution. Don’t line-connect categorical points as if they’re a trend.
• Caption so the figure stands alone. A reviewer should understand the claim, the n, and the uncertainty from the caption without hunting through the text.

matplotlib/seaborn make the honest plot easy; the dishonest one usually takes extra effort (truncating an axis is a manual override). When AI generates your plotting code, the most common silent error is dropping the error bars — check for them every time. Use code/plot_with_ci.py as your honest-defaults starting point.

There is a deeper reason to show the seed-level points and not just a clean bar. A bar with an error band summarizes; a strip plot of the raw runs exposes. When a reader sees your seven seeds scattered over the baseline’s seven seeds — overlapping heavily — they understand instantly that your margin is thin, and they trust you more for showing it. The figure that hides the overlap behind a tidy mean is the visual equivalent of reporting a bare p-value. Let the reader see the spread and reach the cautious conclusion with you, rather than feeling sold to.

Coach’s Note — A good test: would your figure look the same if a reviewer who doubts you drew it from your data? If your honest figure and the skeptic’s figure diverge, the difference is the spin you smuggled into the geometry. Draw the skeptic’s figure. It’s the true one.

12.8 — Error Analysis: Where the Aggregate Number Lies

A single accuracy number is a summary, and summaries hide structure. Error analysis is the qualitative companion to your statistics: pull the cases your method gets wrong and read them. Patterns there are often more publishable than the headline metric — “our method fails systematically on long-context inputs” is a finding; “94.2% accuracy” is a number.

This is also where AI-as-instrument bites. If your experiment used an LLM as a component (a judge, an extractor, a labeler), its errors are not random — they’re correlated and fluent. A recent large study (arXiv:2603.08274) found fabrication in LLM outputs rises with context length, and the output stays grammatical the whole way down. So a 95%-accurate LLM judge is not a 95%-accurate measuring stick if its 5% of errors all break the same direction. Read the errors. Stratify your metric by the slices that matter (input length, class, domain). The aggregate is where bias goes to hide.

12.8b — Statistical vs. Practical Significance

These are different questions, and conflating them is one of the most common ways an honest paper overclaims. Statistical significance asks: is this difference distinguishable from noise? Practical significance asks: is the difference large enough that anyone should change what they do? With a big enough sample, a 0.1% gain is statistically significant and practically irrelevant. With a tiny sample, a 10-point gain might be practically huge and statistically inconclusive — the right response there is “we found a promising effect (d = 1.1) but our CI is wide; more data is needed,” not silence.

The discipline is to separate the two sentences and never let one stand in for the other. “Significant” in your Results section should mean the statistical test; “matters” is an argument you make in the Discussion (Chapter 13), in your domain’s terms, using the effect size. A latency reduction of 3ms is significant and matters in high-frequency trading; the same 3ms is significant and irrelevant in a batch ETL job. Same statistic, opposite practical verdict. Only you, who knows the domain, can make that call — which is exactly why an LLM cannot make it for you.

12.9 — The Stack: Python, R, and the Notebook Discipline

The applied-AI standard analysis stack, as of mid-2026:

One notebook discipline rule worth its own line: a notebook you ran out of order is not reproducible. Before you believe any number in a Jupyter notebook, Restart Kernel and Run All. Cells executed top-to-bottom on a fresh kernel are the only ones that count — hidden state from an out-of-order run is how a number nobody can reproduce ends up in a paper. Versions above are point-in-time; re-verify at authoring time and pin them in your requirements.txt. See Appendix B for the toolkit and Appendix A for the no-install Colab/Codespaces path.

12.x — Interactive Lab: Significance & p-Hacking Explorer

Below this chapter on the page is the Significance & p-Hacking Explorer. Go use it now — this is a section you do, not read.

The widget generates data from a world where there is no real effect — pure noise. Your job is to “find” a significant result anyway, the way an honest, motivated researcher accidentally does:

• Crank up the number of comparisons (metrics × subgroups × seeds) and watch significant p-values appear out of nothing. Count how many comparisons it takes before one crosses 0.05.
• Toggle “try another metric / stop when significant” (optional stopping) and watch the false-positive rate climb past the 5% you thought α bought you.
• Turn on multiple-comparison correction (Bonferroni / Holm / BH) and watch the spurious “findings” evaporate.
• Watch the effect-size readout sit near zero the whole time, even when the p-value “wins” — the tell that the result is noise.

What it teaches, in your hands and not just your notes: effect size + correction + preregistration are not bureaucratic hoops. They are the only things standing between you and a confident, publishable, false claim. When you can reliably manufacture a significant result from noise in the widget, you’ll recognize it instantly in your own analysis — and in the next paper you review.

12.10 — A Just Weight and a False Balance

What does a just weight and a false balance have to do with statistics? The whole chapter, it turns out.

Proverbs 11:1 — “A false balance is an abomination to the LORD, but a just weight is his delight” (ESV) — comes out of a marketplace where a merchant kept two sets of weights: a heavy one for buying and a light one for selling. The fraud was invisible to the customer, because each transaction looked honest. The number on the scale was real. What was false was the correspondence between the number and the truth it claimed to represent.

That is p-hacking exactly. The p-value you cherry-picked from twenty is a real number. The accuracy gain you reported without its confidence interval is a real number. The seed you fished for is a real run. None of them is fabricated — and that is precisely why this temptation is more dangerous than outright fabrication, which at least you’d recognize as a lie. The false balance is subtle. It passes every audit of the individual numbers. What it corrupts is the correspondence between what you report and what is true. The merchant didn’t invent a weight; he selected one. So do we.

The LCMS confession of vocation reframes the whole enterprise. You are not, in the end, the customer of your own research — you are the steward of it. The data is not yours to bend toward the conclusion you’d hoped for; it is a thing entrusted to you, to weigh justly and report faithfully, before God who sees the comparisons you ran and didn’t show. This is why preregistration is not merely good methodology — it is a discipline of honesty, a way of binding your future self before the temptation arrives, the way Ulysses bound himself to the mast. You declare the hypothesis before you can see which one the noise will reward.

And note what the verse promises on the other side: the just weight is His delight. Not merely permitted — delighted in. The careful, slow, honest analysis — the effect size you reported even though it was small, the correction you applied even though it killed your finding, the negative result you wrote up faithfully — that work is not a loss. It is the work that is delighted in, whether or not a reviewer ever sees the integrity that produced it. You search the matter out (Prov. 25:2) and you weigh it justly (Prov. 11:1), because the One you ultimately report to already knows the answer. The just weight is for Him.

Coach’s Note — The day your honest analysis kills your favorite hypothesis is the day you find out whether you’re a scientist. I have buried results I loved. It never stops stinging. It is also the only thing that makes the results I kept worth anything.

12.z — Common Pitfalls

Pitfall: Reporting a p-value with no effect size. Example: “Our method significantly outperforms the baseline (p = 0.03).” The reader has no idea if the gain is 0.2% or 20%. Fix: Lead with the effect and its CI: “Our method improves accuracy by 4.1 points (95% CI [1.2, 7.0]; d = 0.6; permutation p = 0.03).” Three numbers, every time.

Pitfall: Claiming a win from a single training run. Example: One seed, one number, “we beat SOTA.” Bouthillier et al. (2021) shows the seed-to-seed variance may exceed your margin. Fix: Run ≥5 seeds, report the distribution, and compare with a multi-seed test (deepsig.aso, τ = 0.2). If you can’t afford many seeds, say so and weaken the claim accordingly.

Pitfall: Forgetting to correct for the comparisons you ran. Example: Twelve metrics across four datasets; you report the three that hit p < 0.05 as if each were a standalone test. Fix: Count every comparison and apply multipletests (holm for FWER, fdr_bh for FDR). Report the count of tests in the paper.

Pitfall: Optional stopping (“just collect a bit more data”). Example: You peek at the p-value, it’s 0.08, so you run three more seeds and it dips to 0.04. You stop. You just p-hacked. Fix: Fix your sample size / seed count in advance (preregister it on OSF). Don’t let the data decide when to stop.

Pitfall: HARKing — rewriting the intro to “predict” what you found. Example: An exploratory sweep surfaces a subgroup effect; you write the paper as though that was your hypothesis all along. Fix: Keep a confirmatory/exploratory split. Label post-hoc findings as exploratory and hypothesis-generating. They’re valuable — labeled.

Pitfall: Trusting AI-generated analysis code or “the right test” uncritically. Example: An LLM hands you a Student’s t-test on heavily skewed, multi-seed scores and a Results paragraph that “narrates” the numbers. Fix: Verify the test’s assumptions against your data; cross-check against a permutation/ASO test; confirm the prose matches the CIs. The model accelerates; you decide, and you’re accountable (Appendix C).

Pitfall: The misleading figure (truncated axis, no error bars). Example: A bar chart starting at 0.85 makes a 1% gap look like a chasm, with no uncertainty shown. Fix: Start axes at a defensible baseline, always plot uncertainty (error bars / CI bands / the raw points), and caption so the figure stands alone.

12.(z+1) — Reps

The keyboard is the gym, and this week the gym is your own results file. Head to the exercises for the full set; they move your project forward, not a toy. A preview:

• Rep 1 — Write the three-number sentence for your primary comparison: effect size, bootstrap CI, and permutation p-value, computed together.
• Rep 3 — Re-run your headline comparison across ≥5 seeds and compare distributions with deepsig.aso.
• Rep 5 — Count every comparison you ran this semester and apply Holm and BH correction; report what survives.
• Rep 7 — Preregister your next confirmatory analysis on OSF before you touch the data.
• Rep 8 — Build one honest figure (uncertainty shown, axis defensible, caption stands alone) for the Results draft.

Don’t skip the on-page “Check Your Reps” quiz — five questions that catch the misconceptions this chapter is built to kill.

12.(z+2) — This Week’s Deliverable

This week you produce the Results Section Draft — see Project 12. It is the first piece of your paper that lives or dies on statistical integrity: honest figures, the right tests for your data, effect sizes beside every p-value, and the multiple-comparison correction you actually owe. You’ll feed it from your raw results dataset (the Chapter 11 deliverable) and it becomes the spine the Chapter 13 Discussion will interpret. Starter scaffolding lives in code/results_section_template.txt and the analysis script in code/analyze_results.py.

12.(z+3) — Coach’s Final Word

Anyone can find a p < 0.05. Give a motivated person a spreadsheet and an afternoon and they will hand you a significant result from a table of random numbers — you’ll prove that to yourself in the widget today. What separates a researcher from a salesperson is the willingness to count the comparisons, report the effect size even when it’s small, and keep the run that didn’t work in the record. That is the just weight. It is slower, it is less flattering, and it is the only thing that makes your numbers worth a stranger’s trust.

You logged the truth last week. This week, weigh it justly. Lead with the effect, carry the interval, and let the p-value bring up the rear. When you’re tempted to try it just one more way — write down the count instead.

See you on Monday.

Up next: the exercises for the reps, then Project 12 for the Results Section Draft. Then on to Chapter 13 — Discussion: Making Meaning, where you’ll interpret these numbers without overclaiming. Reference: Appendix B (toolkit), Appendix C (AI responsibly), Appendix D (glossary).