Appendix D — Glossary

The terms below are the working vocabulary of this practicum — the words you will read in papers, write in your draft, and hear in review. Each definition is short on purpose: enough to use the word correctly, not a treatise. Where a term is venue- or year-specific (review policies, page limits, model names), treat the definition as as of mid-2026 and re-verify against the primary source before you rely on it. Coach’s rule stands: never state as fact something you have not checked.

Ablation — Removing or disabling one component of your method and re-running the experiment to measure that component’s contribution. One ablation per claim is the discipline: it shows which part of your system actually earned the result.

Abstract — The short opening summary of a paper. Kent Beck’s four-sentence form is a reliable skeleton: (1) the problem, (2) why it’s a problem, (3) the startling/contribution sentence, (4) the implication.

Agent (AI agent) — An LLM-driven system that plans and takes actions through tools (search, code execution, APIs) in a loop, rather than emitting a single response. Built with frameworks such as LangGraph, the Claude Agent SDK, or CrewAI.

Annotated bibliography — A list of sources where each entry carries a short annotation: what the work claims, its method, and its relevance to your question — not just the citation.

arXiv — A free preprint server (the cs.* categories for computer science). Preprints are moderated, not peer-reviewed; as of Oct 31, 2025, CS review/survey/position papers must already be peer-review-accepted (with a journal/conference DOI) before they can be posted.

Artifact badge (ACM) — A badge awarded after independent evaluation of a paper’s artifacts. Three independent families exist: Artifacts Available, Artifacts Evaluated (Functional/Reusable), and Results Validated (Reproduced/Replicated).

ASO (Almost Stochastic Order) test — A significance test for comparing two methods across multiple random seeds under weak distributional assumptions (unlike the t-test). Implemented in the deep-significance package; a violation score τ ≤ 0.2 is the recommended acceptance threshold.

Baseline — The established method your work must beat (or match) to count as a contribution. A result without a credible baseline is not yet a finding.

Belmont Report — The foundational U.S. research-ethics document for human subjects, built on three principles: respect for persons, beneficence, and justice.

Benchmark — A standard dataset-and-metric package used to compare methods on equal footing (e.g., SWE-bench Verified for code tasks). A benchmark is only as honest as its freedom from contamination.

Benchmark contamination — When evaluation data has leaked into a model’s training data, inflating its score. N-gram decontamination (e.g., 13-gram for GPT-3, 40-gram for GPT-4) catches verbatim overlap but is evaded by paraphrase; contamination-resistant benchmarks include LiveCodeBench, LiveBench, MMLU-Pro, and FrontierMath.

Bonferroni correction — The simplest multiple-comparison correction: divide your significance threshold by the number of tests. It controls the family-wise error rate (FWER) but is conservative (low power). See Holm, FDR.

Bootstrap — A resampling method that estimates the variability (e.g., a confidence interval) of a statistic by repeatedly resampling your data with replacement. Available as scipy.stats.bootstrap.

Camera-ready — The final, formatted version of an accepted paper, submitted after revision for publication. At this stage double-blind anonymity is removed and venue page limits apply (e.g., NeurIPS 2026 = 10 content pages max excluding references).

CARS (Create A Research Space) — Swales’ (1990) rhetorical model for an introduction, in three moves: establish the territory → establish a niche (the gap) → occupy the niche (your contribution). Move 2 is where you operationalize “the gap.”

Cohen’s d — A standardized effect size: the difference between two means in units of standard deviation. Conventional 0.2/0.5/0.8 = small/medium/large, but Cohen himself called these arbitrary and field-dependent — interpret in context, not by the label.

Common Rule (45 CFR 46) — The U.S. federal policy governing human-subjects research, including eight exempt categories at §46.104(d). Whether your study is exempt is determined by the IRB or an institutional official, not self-certified.

Conference — In computer science, often the top venue (above journals), with peer review and selective acceptance (frequently under 20%). Contrast with workshop (on-ramp) and journal (archival/extended).

Confidence interval (CI) — A range, computed from your data, that quantifies the uncertainty around an estimate (e.g., a mean difference). Report the CI alongside or instead of a bare p-value; it shows the magnitude and precision, not just “significant or not.”

Confound — A variable you did not control that offers an alternative explanation for your result. Confounds are the enemy of internal validity.

Construct validity — Whether your metric actually measures the thing you claim. The live applied-AI version: does a high MMLU score measure reasoning, or memorization?

Control (control group / control condition) — The baseline condition held fixed so that the effect of your manipulated variable can be isolated. Without a control, you cannot attribute a change to your intervention.

CORE rankings — A community ranking of CS venues into tiers A*/A/B/C, used as a prestige proxy. Latest finalized round as of mid-2026 is ICORE 2026; verify a specific venue’s rank on core.edu.au.

CRediT — The Contributor Roles Taxonomy (ANSI/NISO Z39.104-2022): 14 standardized roles (conceptualization, methodology, software, writing, etc.) for stating “who did what” on a paper.

Cross-validation — A model-evaluation procedure that rotates which fold of the data is held out for testing (e.g., k-fold). In scikit-learn, StratifiedKFold is the default for classifiers because it preserves class proportions in each fold.

Data leakage — When information from outside the training set (often the test set) contaminates training, producing inflated, irreproducible results. The classic bug is scaling or feature-selecting before the train/test split. Documented across 294 papers in 17 fields by Kapoor & Narayanan (2023).

dblp — A free (CC0) computer-science bibliography maintained by Schloss Dagstuhl (~8.6M publications as of mid-2026), best for authoritative metadata; it labels arXiv-only items “informal publications.”

Distillation — Training a smaller “student” model to imitate a larger “teacher” model, trading some capability for lower cost. The last rung of the Prompt → RAG → Fine-tune → Distill decision ladder.

DOI (Digital Object Identifier) — A persistent identifier for a published artifact (paper, dataset, software deposit). Zenodo mints a DOI per deposit; a DOI is what makes a reference stably citable.

Double-blind (double-anonymous) review — Review in which authors and reviewers are mutually anonymous. Used by most top CS venues; de-anonymization happens only at camera-ready. Requires scrubbing names, affiliations, grant IDs, and identifying links.

Effect size — A measure of the magnitude of an effect (e.g., Cohen’s d), independent of sample size. The ASA’s guidance: report effect size and CI, because a small p-value can accompany a trivially small effect.

External validity — Whether your result generalizes beyond your specific sample, setting, and conditions to the populations and contexts you care about.

Fabrication / Falsification / Plagiarism (FFP) — The U.S. federal triad defining research misconduct (42 CFR Part 93): inventing data, manipulating data/results, and using others’ work or words as your own. Honest error and differences of opinion are explicitly excluded.

Falsifiability — The property of a hypothesis that it could, in principle, be proven wrong by evidence. A claim you can’t be wrong about is not a scientific hypothesis.

FDR (False Discovery Rate) — A multiple-comparison framework that controls the expected proportion of false discoveries among rejected nulls (e.g., Benjamini–Hochberg, fdr_bh). Generally higher power than FWER methods like Bonferroni.

FINER criteria — A checklist for a good research question: Feasible, Interesting, Novel, Ethical, Relevant.

Fine-tuning — Continuing to train a pretrained model on task-specific data to change its behavior. In 2026, full fine-tuning is rarely justified (cost, catastrophic forgetting); parameter-efficient methods like LoRA dominate. Rule of thumb: retrieve (RAG) for facts, fine-tune for behavior/form/style.

FWER (Family-Wise Error Rate) — The probability of making at least one false-positive across a family of tests. Controlled by Bonferroni and Holm; stricter (and lower-power) than FDR.

Gap (research gap) — A specific, citable absence in prior work that your study would fill (CARS Move 2). A gap must be backed by a cited absence, not a hunch.

GraphRAG — A RAG variant that retrieves over a knowledge graph to support multi-hop reasoning across linked facts. Contrast with naïve, hybrid, and agentic RAG.

GRPO (Group Relative Policy Optimization) — A reinforcement-learning algorithm for LLMs popularized by DeepSeek-R1, which estimates advantages by comparing a group of sampled responses rather than using a separate value model.

h5-index — A Google Scholar Metrics venue measure: the largest h such that h articles published in the last five complete years each received at least h citations (reported with an h5-median). A prestige proxy for venues.

HARKing — “Hypothesizing After the Results are Known” — presenting a post-hoc hypothesis as if it had been predicted in advance (Kerr, 1998). Preregistration is the standard guard against it.

Holm correction — A step-down multiple-comparison method that controls FWER but is uniformly more powerful than Bonferroni (holm in statsmodels).

Hypothesis — A falsifiable, testable statement predicting a relationship between variables, derived from your research question.

IMRaD — The standard paper structure: Introduction, Methods, Results, and Discussion — expanded in CS to abstract / introduction / related work / method / results / discussion / future work.

Independent variable (IV) / Dependent variable (DV) — The IV is what you deliberately manipulate; the DV is what you measure as the outcome. Your hypothesis predicts how the IV affects the DV.

Internal validity — Whether the observed effect is genuinely caused by your manipulation rather than by a confound. One of Cook & Campbell’s (1979) four validity categories.

IRB (Institutional Review Board) — The committee that reviews and approves human-subjects research and determines exemptions under the Common Rule. Exemption is the IRB’s call, not the researcher’s.

Journal — A peer-reviewed, archival publication venue. In CS it often sits below top conferences in prestige; a common pipeline is workshop → conference → extended journal version.

LLM (Large Language Model) — A neural network trained on large text (and increasingly multimodal) corpora to predict and generate language; the engine behind agents, RAG, and most applied-AI methods in this book. An LLM can never be an author at any major CS venue.

LoRA (Low-Rank Adaptation) — A parameter-efficient fine-tuning method that freezes the base model and trains small low-rank adapter matrices instead of all weights — far cheaper than full fine-tuning.

Multimodal — A model or system that handles more than one input/output modality (text, image, audio, video). Frontier models in mid-2026 are natively multimodal.

Multiple-comparison correction — Adjusting significance thresholds when running many tests, because each test carries its own false-positive risk. Methods: Bonferroni, Holm (FWER); Benjamini–Hochberg/Yekutieli (FDR).

NeurIPS Paper Checklist — A mandatory checklist (descended from Pineau’s ML Reproducibility Checklist) covering error bars, data splits/hyperparameters, reproducibility, and compute; a missing checklist is grounds for desk rejection. Useful to complete during the pilot, not just at submission.

OpenReview — A free, nonprofit open-review platform that runs review for NeurIPS, ICLR, and ICML (configurably open or closed). Its public review threads are a goldmine for seeing what reviewers flag as missing.

Operationalize — To turn an abstract construct (e.g., “code quality”) into a concrete, measurable metric. The measurement you choose is a construct-validity decision.

p-hacking — Exploiting “researcher degrees of freedom” — trying outcomes, subgroups, or stopping rules until something crosses p < 0.05 (Simmons, Nelson & Simonsohn, 2011). It manufactures false positives.

p-value — The probability of observing data at least as extreme as yours if the null hypothesis were true. It is not the probability the hypothesis is true; the ASA (2016) cautions against treating p < 0.05 as a verdict and (2019) against the phrase “statistically significant” altogether.

Peer review — Evaluation of a submission by independent experts for novelty, methodology, and clarity, used to decide acceptance and to improve the work.

Permutation test — A non-parametric significance test that builds the null distribution by shuffling labels/pairings of your actual data, making few distributional assumptions (scipy.stats.permutation_test).

Power (statistical power) — The probability that a test detects a real effect when one exists (1 − β). Underpowered studies miss true effects and produce unreliable estimates; power depends on effect size, sample size, and α.

Preregistration — Publicly recording your hypotheses and analysis plan, time-stamped, before collecting data (e.g., on OSF or AsPredicted.org). It curbs HARKing and p-hacking by fixing the analysis in advance.

Provenance — The full record of where each result came from: seed, configuration, dataset version, and code commit. Provenance is what lets a number be re-traced and a run be re-created.

QLoRA — LoRA plus 4-bit quantization of the frozen base model, enabling fine-tuning of large models on a single GPU.

RAG (Retrieval-Augmented Generation) — Grounding an LLM’s output in documents retrieved at query time, so it answers from a real index rather than parametric memory. Retrieval — not generation — is the dominant failure mode; variants include hybrid, agentic, and GraphRAG.

Rebuttal — The authors’ written response to reviews during the review period (NeurIPS 2026: up to 10,000 characters per review, plain text/Markdown, no file uploads). It addresses reviewer concerns; the original submission still anchors the decision.

Related work — The paper section that positions your contribution against prior research, showing both what exists and the gap you fill. Synthesis, not a list of summaries.

Replication — Independently obtaining consistent results without the original team’s artifacts (ACM “Replicated”). Contrast with Reproduction. The reproducibility crisis (Baker, 2016) is the backdrop: most surveyed scientists had failed to reproduce others’ — and their own — results.

Reproduction — Independently obtaining the same results using the original team’s artifacts (ACM “Reproduced”). The weaker but essential first bar; replication is the stronger one.

Reproducibility checklist — A set of items (environment pinned, seeds set, data versioned, README runs, code public) that answers the question “could a stranger rerun this?”

RLHF (Reinforcement Learning from Human Feedback) — Fine-tuning a model using a reward signal learned from human preference comparisons, to align behavior with what people prefer.

RLVR (Reinforcement Learning from Verifiable Rewards) — RL using automatically checkable rewards (e.g., math correctness, passing unit tests) rather than learned human preferences. An open debate as of mid-2026 is whether RLVR expands reasoning beyond the base model or merely sharpens sampling — cite both sides.

Seed (random seed) — The value that fixes a pseudo-random sequence so a run can be repeated. Deep-learning results vary substantially by seed (Bouthillier et al., 2021), so report results over multiple seeds, never a single lucky run.

Significance (statistical vs practical) — Statistical significance asks whether an effect is distinguishable from noise; practical significance (effect size) asks whether it’s big enough to matter. The two are not the same, and a small p-value does not guarantee a meaningful effect.

Snowballing — Expanding a literature search by following references backward (a paper’s citations) and citers forward. Supported by tools like Connected Papers, Research Rabbit, and scite.

Threats to validity — The Cook & Campbell (1979) taxonomy of four ways a study’s conclusions can be wrong: statistical conclusion, internal, construct, and external validity (adapted for software engineering by Wohlin et al.).

Three-pass method — A reading strategy: pass 1 for the gist (title/abstract/headings), pass 2 for the content and figures, pass 3 for full, critical depth. It lets you triage a large literature efficiently.

Train/validation/test split — Partitioning data so the model learns on train, is tuned on validation, and is evaluated once on a held-out test set. Touching the test set during development is data leakage.

Workshop — A higher-acceptance, lower-stakes venue (often co-located with a conference) for work-in-progress and student research. Student Research Workshops (SRWs) accept short WIP papers with a student first author — the realistic practicum target.

Zenodo — A free archival repository (CERN/OpenAIRE) that mints a DOI per deposit and, since Oct 21, 2024, auto-deposits public source code to Software Heritage with a bidirectionally linked SWHID.

Zotero — A free, open-source reference manager (300 MB free storage; no native AI in 2026, only plugins). The practicum’s tool for collecting, organizing, and citing sources.