Chapter 12 — Reps

This week’s reps run on your own results file. Every rep moves the Results Section Draft forward — no toy datasets, no busywork. By the end you’ll have honest figures, the right tests, effect sizes, and a corrected multiple-comparison table.

Ground rules

• Use your real data from the Chapter 11 raw results dataset. If a run is missing, note it — don’t invent it.
• Three numbers travel together: effect size, confidence interval, p-value. Never report one alone.
• Pin your stack. Record SciPy / statsmodels / scikit-learn / deep-significance versions in your notebook header. Versions below are point-in-time (mid-2026) — confirm yours.
• Restart Kernel and Run All before you trust any notebook number. Out-of-order cells don’t count.
• AI policy: you may use an LLM to draft analysis code and explain a test, but you verify every assumption and every number against the primary tool/source. The human owns the claim (Appendix C). Log substantive AI use.
• Commit each rep. Tag your analysis commit so the figures trace back to a code state.

Rep 1 — The three-number sentence

Take your single most important comparison (your method vs. its strongest baseline) and compute all three numbers at once.

import numpy as np
from scipy.stats import bootstrap, permutation_test

mine = np.array([...])   # your method's scores
base = np.array([...])   # baseline's scores

d = (mine.mean() - base.mean()) / np.sqrt(
    ((len(mine)-1)*mine.var(ddof=1) + (len(base)-1)*base.var(ddof=1))
    / (len(mine)+len(base)-2))
ci = bootstrap((mine - base,) if len(mine)==len(base) else (mine,),
               np.mean, confidence_level=0.95, n_resamples=10_000, random_state=0)
pt = permutation_test((mine, base), lambda a,b: a.mean()-b.mean(),
                      n_resamples=10_000, random_state=0)
print(d, ci.confidence_interval, pt.pvalue)

Reflection (3–4 sentences): Write the one-sentence result that leads with the effect and carries the interval. Is the effect large enough to matter in your domain — independent of the p-value?

Rep 2 — Pick the right test, and justify it

Your data has a shape: paired or unpaired, normal or skewed, multi-seed or single-sample. Choose your test deliberately.

Do: For each comparison, state in one line: (a) paired or not, (b) does it meet normality, (c) which test you chose and why. Default to scipy.stats.permutation_test when assumptions are shaky.

Reflection: Where did the “obvious” t-test not fit your data, and what did you use instead?

Rep 3 — Many seeds, one verdict (ASO)

If your work compares model variants, re-run your headline comparison across ≥5 seeds and compare distributions, not point estimates.

from deepsig import aso
min_eps = aso(scores_a, scores_b, seed=0)   # < 0.2 ⇒ A stochastically dominant
print(min_eps)

Reflection: Did the multi-seed picture change your single-run conclusion? Reference Bouthillier et al. (2021) — was your margin larger or smaller than the seed-to-seed spread?

Rep 4 — Bootstrap a CI on your weird metric

Pick the metric where an analytic CI is awkward (F1, BLEU, a custom score). Bootstrap it.

from scipy.stats import bootstrap
res = bootstrap((scores,), np.mean, confidence_level=0.95,
                n_resamples=10_000, random_state=0)

Reflection: How wide is the interval? What does its width tell a reader about how much you actually know yet?

Rep 5 — Count and correct the comparisons

List every statistical comparison you’ve run this semester — metrics × datasets × subgroups × ablations. Then correct.

from statsmodels.stats.multitest import multipletests
reject_h, p_holm, _, _ = multipletests(all_pvals, method="holm")    # FWER
reject_b, p_bh,   _, _ = multipletests(all_pvals, method="fdr_bh")  # FDR

Reflection: How many “findings” survived Holm? How many survived BH? Which of your earlier “wins” did not survive, and how does that change the story?

Rep 6 — Build a p-hacked result on purpose (then bury it)

Use the Significance & p-Hacking Explorer widget on the chapter page (or random noise locally): manufacture a p < 0.05 from data with no real effect by cranking comparisons and using optional stopping.

Reflection: How many comparisons did it take? Watch the effect size — what did it read while the p-value “won”? Write the two sentences you’d use to catch this exact move in a paper you’re reviewing.

Rep 7 — Preregister your next confirmatory analysis

Before you touch the data for your next confirmatory test, write the plan and lock it on OSF (or AsPredicted.org): the hypothesis, the primary outcome, the test, the sample/seed count, the correction.

Reflection: What did writing the plan first force you to decide that you’d otherwise have decided after seeing the data?

Rep 8 — The honest figure

Build the primary results figure with: uncertainty shown (CI bands / error bars / raw points), a defensible axis range, and a caption that stands alone.

import seaborn as sns, matplotlib.pyplot as plt
ax = sns.barplot(data=df, x="method", y="score", errorbar=("ci", 95))
sns.stripplot(data=df, x="method", y="score", color="k", alpha=.4, ax=ax)

Reflection: Re-draw it the dishonest way (truncated axis, no error bars) and look at both side by side. Describe in one sentence how the dishonest one would mislead a skimming reviewer.

Rep 9 — Error analysis: read the failures

Pull the cases your method gets wrong. Stratify your metric by a slice that matters (input length, class, domain). If an LLM was an instrument in your pipeline, check whether its errors break in one direction.

Reflection: What systematic failure did the aggregate number hide? Is it itself a finding worth reporting?

Rep 10 — Three ways to say one result

Take one comparison and write the result sentence three ways: (1) p-value only, (2) confidence interval, (3) Cohen’s d + plain-language effect.

Reflection: Which version would survive the ASA 2019 “world beyond p < 0.05” standard? Which one is closest to honest? Keep that one.

Rep 11 — Notebook reproducibility check

Restart your analysis kernel and Run All top-to-bottom. Confirm every number in your draft regenerates from a clean run on pinned versions.

Reflection: Did anything change? Out-of-order cell state is how an unreproducible number sneaks into a paper — what did the clean run catch, if anything?

Done? One Last Thing.

Assemble Reps 1–11 into the first honest draft of your Results Section (Project 12). It must contain, for your primary claim: the effect size, its confidence interval, the right test’s p-value, the multiple-comparison correction over every comparison you ran, one honest figure, and one paragraph of error analysis. Then do the hardest rep in the book: read your own draft and find the one sentence that overclaims relative to what your CIs and effect sizes actually support — and rewrite it true. That sentence is the false balance hiding in plain sight. Fix it before a reviewer finds it.

Up next: Project 12