Chapter 4 — Reading, Synthesis, and the Comparison Matrix

“It’s not the writing part that’s hard. What’s hard is figuring out what you know.” — Wendell Berry, on the discipline of seeing before you speak

“This is the third time I am coming to you. Every charge must be established by the evidence of two or three witnesses.” — 2 Corinthians 13:1 (ESV)

Why This Matters

Last week you hauled in twenty-plus sources and sorted them into a Zotero library. You have an annotated bibliography. That is a stack of witnesses. It is not yet a case.

Here is the trap almost every first-year master’s student walks into. They confuse a literature review with a book report. They write: “Smith (2024) did X. Then Jones (2025) did Y. Garcia (2026) did Z.” Twenty paragraphs of that, each one a faithful little summary, and at the end the reader has no idea what you think, where the field actually stands, or — the only thing that matters — what nobody has done yet. That is a summary. A summary is a list of what each witness said. A synthesis is the verdict you reach after you cross-examine all of them together.

This is the week the practicum earns its name. You are going to read papers the way a working scientist reads them — fast, structured, and ruthless about extracting the four things you actually need: method, data, metric, limitation. You are going to lay those extractions side by side in a comparison matrix until the empty cell in that matrix stops being empty space and becomes your research gap — the thing you will spend the next twelve weeks doing something about. And you are going to turn all of that into the first graded writing of the course: a Literature Review Draft worth 15% of your grade, the single largest deliverable before the proposal.

The AI thread runs straight through this chapter from both sides. As a tool you wield: grounded extraction engines — Elicit, Consensus Study Snapshots, scite — can pre-fill a matrix column faster than you can read, and you will use them. As a workload you govern: the very LLMs that help you read are documented to fabricate citations at rates that will end your academic career if you trust them blind — Walters & Wilder (2023) found 55% of GPT-3.5 and 18% of GPT-4 citations entirely fabricated. The spine rule of this whole book lives in that gap. AI accelerates the extraction. You verify every cell against the primary source, and you are accountable for every claim. The human stays where the judgment lives.

And that brings us to the question under the chapter. Paul tells the Corinthians that every charge must stand on the evidence of two or three witnesses (2 Cor 13:1, ESV). It is a courtroom principle older than the church — it appears in Deuteronomy 19:15 — and it is, quietly, the epistemology of the entire scientific enterprise. What does it mean to weigh many witnesses? Not to count them. Not to let the loudest or the most recent win. To examine them against one another, to notice where they agree and where one stands alone, and to render an honest verdict about what the body of testimony actually establishes. A literature review is a courtroom. This week you learn to preside over one.

Coach’s Note — If you remember one sentence from this chapter, make it this: the goal of a literature review is not to prove you read the field — it is to prove the field has a hole in it, and that you can see the hole. Everything below serves that one move.

4.1 — Summary Is Not Synthesis (Say It Until It’s Reflex)

Let me make the distinction concrete, because your grade depends on it.

A summary answers: what did this paper say? It is organized by source — one paragraph per paper.

A synthesis answers: what does the field collectively know, disagree about, and leave undone? It is organized by idea — one paragraph per theme, drawing on many papers.

The tell is the topic sentence. If your paragraph opens with an author’s name, you are summarizing. If it opens with a claim about the field — “Early work on X assumed Y, an assumption that two later studies undermine” — and then marshals the witnesses to support that claim, you are synthesizing. Rewrite every author-first topic sentence into an idea-first one. That single edit is worth more than any other this week.

4.2 — The Three-Pass Method: Reading Without Drowning

You cannot read twenty papers the way you read a novel. You will burn out at paper four. The standard discipline in CS — codified by S. Keshav in his widely circulated note “How to Read a Paper” — is to read in three passes, each one a decision about whether to spend more time.

Pass 1 — The five-minute bird’s-eye (do this for all twenty+ papers). Read the title, abstract, and section headings. Read the introduction. Read the conclusion. Glance at the figures. Read the references and check off the ones you already have. At the end, you should be able to answer the five C’s: Category (what type of paper — measurement? system? theory?), Context (what other work it relates to), Correctness (do the assumptions look valid?), Contributions (what’s the main claim?), Clarity (is it well written?). Then make a triage decision: full read, skim later, or discard. Most papers get triaged out. That is the point — Pass 1 is how you protect your hours.

Pass 2 — The one-hour read (do this for the ~8–12 papers that survive triage). Read the whole thing, but ignore proofs and heavy derivations. Look hard at the figures, tables, and graphs — are the axes labeled? what’s the baseline? where are the error bars? Mark references you should chase. After Pass 2 you can summarize the paper to a colleague with evidence. This is the pass where you fill your matrix row.

Pass 3 — The re-implementation read (do this for your 3–5 keystone papers). Virtually re-create the work in your head. Challenge every assumption. Identify what you’d do differently. This is how you find the methods you’ll baseline against and the limitation that becomes your gap. Budget several hours per paper. You only do this for the papers your own work will directly stand on.

Coach’s Note — Pass 1 is a gatekeeper, not a courtesy read. Its job is to let you say “not this one” with confidence and move on. A student who refuses to triage reads ten papers deeply, runs out of time, and never sees the shape of the field. Triage is not laziness. It is how you afford to be thorough where thoroughness pays.

4.3 — The Extraction Sheet: Five Fields, Every Paper

For every paper that survives to Pass 2, you fill one row. Discipline yourself to extract these five fields in the paper’s own terms first, then in your words:

1. Problem — what question does it actually answer? (Often narrower than the title implies.)
2. Method / approach — what did they build or do? Name the architecture, algorithm, or technique.
3. Dataset / benchmark — what did they evaluate on? A result is meaningless without knowing this.
4. Metric & result — what number, on what measure, against what baseline? “92% accuracy” is noise without the baseline and the metric definition.
5. Limitation / threat — what does the paper not show? Read the “Limitations” and “Future Work” sections like a prosecutor. This column is where your gap is born.

A starter sheet lives at code/extraction-sheet.txt — copy it once per paper during Pass 2. The reason you extract into a fixed schema is the same reason a clinician uses a chart: it forces you to notice what’s missing. The paper that reports accuracy but never names its baseline has a hole. You only see the hole if every other row has a baseline filled in.

Coach’s Note — Your gap almost always comes from column 5. Mine the “Future Work” and “Limitations” sections of your keystone papers — that is where authors, under the honesty peer review enforces, confess what they didn’t do. CARS Move 2 (Swales) calls this “establishing a niche”: you back your gap with a cited absence, not a hunch. “Nobody has done X” is a claim you must be able to footnote.

4.4 — From Extraction Sheets to the Comparison Matrix

Now you transpose. Each extraction sheet becomes one row of a single table. The columns are your five fields (plus a citation key). Lay all of them in one spreadsheet and something happens that cannot happen when papers sit in separate documents: the columns become arguments.

A worked, fillable template — the exact one this week’s deliverable uses — is in code/comparison-matrix.csv. Here is the shape, populated with a tiny illustrative slice for a student working on LLM agents for automated code review (one of the course’s seed examples):

Read down the columns, not across the rows. When you read down the “Dataset” column and every paper uses the same three benchmarks, you’ve found a contamination/over-fitting risk worth flagging. When you read down “Metric” and nobody measures the thing you care about — say, whether developers accept the agent’s review comments — you’ve found a gap. The empty cell is not missing data. The empty cell is your research question wearing a disguise.

This is precisely what the chapter widget makes you feel; see §4.8.

4.5 — Grounded AI for Extraction (and Where It Lies)

You will not hand-read every cell from scratch, and you shouldn’t pretend to. Several tools extract structured fields and can pre-fill matrix columns. Use them as a fast first pass — then verify. As of 2026, the useful, grounded ones:

The workflow that keeps you honest: auto-extract one column with Elicit, then hand-verify every cell against the PDF. Run three of your keystone references through scite and reclassify how the field actually treats them — you will sometimes find a paper everyone “cites approvingly” is in fact contrasted by the citers.

Now the non-negotiable. These tools are grounded — they retrieve against a real index — so they hallucinate less than a raw chatbot. They do not hallucinate zero. And if you ever ask a bare LLM (“give me five papers on X”) to seed your review, treat its output as unverified leads, not citations. The numbers, again: Walters & Wilder (2023) — 55% of GPT-3.5 citations and 18% of GPT-4 citations entirely fabricated; among the real ones, 43%/24% had substantive metadata errors. Bhattacharyya (2023) found 87% of citations to real works still carried at least one metadata error. A 2026 study of a 172-billion-token Q&A corpus (arXiv:2603.08274) found fabrication rises with context length — the longer the conversation, the more the model invents.

Verification protocol, every reference, no exceptions: confirm it exists and the metadata is correct against dblp (free, CC0, ~8.6M CS publications), Semantic Scholar, or the publisher’s page. If you cannot find it in a real index, it does not go in your review. See Appendix C for the full responsible-AI discipline and venue disclosure rules, and Appendix B for the citation-hygiene workflow.

Coach’s Note — ICCV has rejected papers containing non-existent citations without review. arXiv announced (May 2026) a one-year submission ban for “incontrovertible evidence” of unchecked LLM content — hallucinated references, leftover meta-comments. The cost of one fabricated citation is not a lost point. It is your reputation, which you only get to spend once. Verify.

4.6 — Citation Mechanics: Zotero → BibTeX → Your Manuscript

The synthesis is intellectual; the plumbing is mechanical, and broken plumbing sinks good work. Your Zotero library from Chapter 3 is the source of truth. Export a BibTeX file and let the manuscript pull from it — never hand-type a citation.

The no-admin way (do this if you’re on the browser path): from the Zotero web library (zotero.org/mylibrary), select your collection → the ⋯ (More) menu → Export → choose BibTeX → save it as references.bib next to your .tex file and upload it to Overleaf. No install, no admin rights, as of 2026. This is the default for this course, and it produces a fully valid .bib.

Zotero web library → select collection → ⋯ (More) → Export
   → format: BibTeX → save as references.bib → upload to Overleaf.

The optional desktop convenience: if you already have the Zotero desktop app (which needs admin to install — the local path), the Better BibTeX add-on auto-syncs your .bib and pins stable, human-readable cite keys (e.g., jimenez2024swebench) that won’t churn on re-export. It’s a nice-to-have, not a requirement — the plain web-library export above earns full marks. On the no-admin path you simply pick a sensible key once and leave the item’s author/year alone so the key stays put. Full setup, both paths, is in Appendix A.

In your ACM acmart or IEEE IEEEtran manuscript (the one you started in Week 1 on Overleaf), you cite by key and let LaTeX + BibTeX assemble the reference list:

% acmart loads natbib-style citation commands by default.
Recent agent benchmarks evaluate task completion but not
review quality~\cite{jimenez2024swebench, ccrab2026}.

% ... at the end of the document:
\bibliographystyle{ACM-Reference-Format}
\bibliography{references}   % pulls from references.bib

A minimal, real-shaped references.bib you can adapt is in code/references.bib. Two rules that save careers: (1) the .bib file is generated from Zotero, so Zotero stays the single source of truth — fix metadata there, re-export, never edit the .bib by hand; (2) every entry in your .bib corresponds to a paper you have opened and verified, not a key an LLM suggested. The portfolio you’ve been building since Week 1 (Zotero library + Git repo + the Overleaf manuscript) is exactly the pipeline that makes this one command instead of one hour. Setup details: Appendix A.

4.7 — Structuring the Review: Themes, Not a Pile

With the matrix built and the gap visible, you write. The structure of a literature review is thematic, and it makes one argument: here is what the field knows, here is where it disagrees or stops, and here is the specific hole my work will fill. A reliable skeleton:

1. Opening frame (1 paragraph). What is the problem space and why does it matter? Orient the reader. (Swales CARS Move 1: establish the territory.)
2. Thematic body (3–5 subsections). One subsection per theme you found reading down the matrix columns — e.g., “Benchmarks for code-review agents,” “Human-acceptance metrics,” “Retrieval strategies.” Each synthesizes several papers and ends by naming what that theme leaves open.
3. The gap (1–2 paragraphs). Pull the threads together. Name the unaddressed column or empty cell explicitly, backed by cited absences. (CARS Move 2: establish the niche.)
4. Your foothold (1 paragraph). One sentence: how your planned work occupies that niche. (CARS Move 3.) You are not proposing the experiment yet — that’s Chapter 8 — but the reader must finish your review knowing exactly what you’re going to do something about.

A short annotated outline is in code/lit-review-outline.txt. One caution grounded in current policy: a student literature review is not an arXiv-postable survey. As of October 31, 2025, arXiv requires CS review/survey/position papers to already be peer-review-accepted at a journal or conference (workshop review is explicitly insufficient; a DOI is required). Your review is a section of your eventual paper, not a standalone survey you can post. Keep it in your manuscript.

4.8 — Interactive Lab: Comparison Matrix Builder

Open the Comparison Matrix Builder embedded directly below this chapter on the site.

Load the small starter set of papers (or paste in your own keystone five). For each paper, fill the five field columns — problem, method, dataset, metric, limitation. As you fill, do the thing this lab is built to teach: read down the columns. Toggle a column to “highlight gaps” and watch the tool flag the cells that stay empty across many rows and the columns no paper addresses. That highlighted emptiness is the visual form of your research gap.

What it teaches: synthesis is a spatial act before it is a writing act. When the papers sit in one grid, the field’s blind spot becomes something you can point at. Export your filled matrix as CSV when you’re done — it drops straight into this week’s deliverable and into the matrix file in your portfolio. Do not leave this lab until at least one column or cell is conspicuously, defensibly empty. That emptiness is your week’s payoff.

4.9 — Weighing Many Witnesses: The Apologetic

Return to the verse over the chapter. “Every charge must be established by the evidence of two or three witnesses” (2 Cor 13:1, ESV). Paul is quoting a rule embedded in the Law itself — Deuteronomy 19:15 — and applying it to his own coming judgment among the Corinthians. The principle is ancient and it is precise: a single witness is not enough to establish a charge. Testimony must be corroborated.

Notice what that rule is not. It is not “believe whoever speaks last.” It is not “count the witnesses and let the majority win.” It is a demand that evidence be weighed against itself — that you examine the witnesses together, attend to where they agree, and notice with care where one stands alone. This is, almost exactly, the discipline of a literature review. A single paper claiming a result is one witness. You do not build your work on it. You ask: who corroborates? who contradicts? what does the body of testimony, weighed honestly, actually establish? The comparison matrix is the two-or-three-witnesses rule rendered as a spreadsheet. Each row a witness. The verdict is what survives cross-examination.

There is a deeper LCMS instinct underneath. Confessional Lutheranism reads truth as something given — not manufactured by the cleverness of the reviewer but discovered by the patient, humble examination of what is actually there. The reviewer who cherry-picks the witnesses that flatter a thesis, or who lets a fabricated citation stand because it sounds right, has not made a small methodological error. He has borne false witness — the same sin the Eighth Commandment names, the same one Exodus 20:16 forbids, the one Chapter 11 of this book will put under your own data. The honesty you owe your sources is the honesty you owe your neighbor, because a fabricated citation defrauds every reader who trusts your review to walk the path behind you.

And so the LLM that hallucinates a citation is not merely wrong. It cannot bear witness at all, because it cannot be held accountable for what it says — which is exactly why no major venue will let it be an author. Accountability is the thing a witness brings that a tool cannot. You are the witness. The matrix is your testimony. Weigh it the way you would want to be weighed.

Coach’s Note — The humility here is load-bearing, not decorative. “Establish the charge on two or three witnesses” is a guard against the most natural sin in research: deciding what you want to find and then enlisting the sources that agree. Let the witnesses speak before you render the verdict. The field is allowed to surprise you. The good reviewer wants to be surprised.

4.10 — Common Pitfalls

Pitfall: Writing a summary and calling it a synthesis. Example: Twenty paragraphs, each opening “Author (year) did…,” with no claim about the field tying them together. Fix: Make every topic sentence an idea, not an author. Organize by theme, cite several papers per paragraph, end each theme by naming what it leaves open.

Pitfall: Trusting an AI-extracted matrix cell or an AI-suggested citation without checking the PDF. Example: Elicit fills “92% F1” into your metric column; the paper actually reports 92% recall with no F1, against an unstated baseline. Fix: Auto-extract for speed, then hand-verify every cell against the primary source. Confirm every citation exists in dblp / Semantic Scholar / the publisher before it enters the review.

Pitfall: Reading every paper at Pass-3 depth. Example: You re-implement ten papers in your head, run out of time at paper eleven, and never see the field’s shape. Fix: Triage hard at Pass 1. Reserve deep reading for your 3–5 keystones. Thoroughness is a budget — spend it where it pays.

Pitfall: Claiming a gap by assertion (“nobody has studied X”). Example: “To our knowledge, no prior work addresses Y” — with nothing backing it. Fix: Back every gap with a cited absence. Show the matrix column that’s empty and the searches that came up empty. CARS Move 2 demands evidence, not a hunch.

Pitfall: Comparing results that aren’t comparable. Example: Putting “92% accuracy” and “0.88 F1” in the same column and concluding one method “wins.” Fix: Record the metric definition, the dataset, and the baseline in the matrix. Numbers are only comparable within the same metric on the same data. When they aren’t, say so.

Pitfall: Hand-editing the .bib file. Example: You fix a typo directly in references.bib, then re-export from Zotero and silently overwrite the fix — or worse, the two drift apart. Fix: Zotero is the single source of truth. Fix metadata there, re-export (web-library BibTeX export, or Better BibTeX if you’re on the desktop app), never touch the .bib by hand.

Pitfall: Treating your review as a postable survey. Example: Planning to put your literature review on arXiv as a standalone survey paper. Fix: As of Oct 31, 2025, arXiv-CS won’t take a survey without prior peer-review acceptance + DOI. Your review is a section of your paper, not a separate artifact. Keep it in the manuscript.

4.11 — Reps

The reps for this week live in the exercises. They are not busywork — every one of them moves your own literature review forward, so that by Friday the deliverable is assembled rather than started. A preview of what’s waiting:

• Three-pass triage your full source set: Pass-1 all twenty, mark which survive to Pass 2.
• Fill five extraction sheets from your keystone papers using code/extraction-sheet.txt.
• Build the comparison matrix in the widget and export it to code/comparison-matrix.csv shape.
• Run the AI-extraction-vs-verification drill: auto-fill one column with Elicit, then catch its errors against the PDFs.
• Citation-context audit: run three keystone references through scite and reclassify supporting/contrasting/mentioning.

Don’t skip the on-page Check Your Reps quiz below the chapter — it’s five quick questions that confirm the distinctions stuck before you write.

4.12 — This Week’s Deliverable

This week produces the Literature Review Draft — worth 15% of your course grade, the largest single deliverable before the proposal. The full spec, tiers, and rubric are in Project 4. In short: a thematically organized review of 20+ sources, plus the comparison matrix that surfaces your gap, ending in one paragraph that names the niche your work will occupy. It goes into your portfolio and becomes the Related Work section of your eventual paper. Read the spec before you start writing — the rubric rewards synthesis and integrity, and it penalizes the book-report failure mode hard.

4.13 — Coach’s Final Word

Here is the shift I want you to feel this week. You walked in with a stack of papers. You walk out with a verdict — a defensible claim about where your field stands and exactly where it stops. That verdict is the foundation everything else rests on: the proposal, the experiment, the paper. Get the gap wrong and you spend twelve weeks answering a question someone already answered. Get it right and you have a thesis.

Synthesis is hard because seeing is hard. Wendell Berry was right — the writing isn’t the hard part; figuring out what you actually know is. The matrix is your instrument for seeing. Fill it honestly, read down the columns, verify every cell, and let the witnesses render the verdict together. Then write it plain.

Weigh the witnesses. Find the hole. Point at it.

See you on Monday.

Up next: the exercises for the reps · Project 4 for the Literature Review Draft · Chapter 5 — AI Across the Computer Science Domains. Previously: Chapter 3.