What Is Research, Really?
What does it mean to search a matter out?
Chapter 1 — What Is Research, Really?
“If we knew what it was we were doing, it would not be called research, would it?” — widely attributed to Albert Einstein (attribute cautiously; the line is unsourced in his papers)
“It is the glory of God to conceal things, but the glory of kings is to search things out.” — Proverbs 25:2 (ESV)
Why This Matters
Welcome to the practicum. For the next sixteen weeks you are not going to study research from the outside, the way an undergraduate surveys a method and writes a proposal nobody executes. You are going to do it. By the end of this course you will walk out holding the foundation of a publishable paper in a computer science domain you choose this week and keep all semester: a defined problem, a systematic literature review, a reproducible design, preliminary results, a conference-style draft, and a roadmap to a real venue. A practicum is the first half of a thesis. Treat it that way from today.
Here is the thing nobody tells you on day one: most graduate students cannot actually say what research is. They can say “I’m researching LLM agents” — but that is a topic, not research. Research is something specific and demanding. It is a systematic departure from accepted practice to produce new, verifiable knowledge. Every word in that sentence carries weight. Systematic — not poking around. Departure from accepted practice — you must know the accepted practice first, which is why Weeks 3–4 are a literature review. New — novelty, or you are reinventing. Verifiable — and this is the one that defines computer science research in 2026: someone else, somewhere else, with your paper in hand, must be able to reproduce it.
That last word is where the stakes live. In 2016, Nature surveyed 1,576 scientists across disciplines (Baker, Nature 533:452–454): roughly 52% said science faces a significant reproducibility crisis, more than 70% had failed to reproduce another scientist’s experiment, and more than half had failed to reproduce their own. That survey is broad and cross-disciplinary — cite it as framing, not as a CS measurement — but the direction is unmistakable, and computer science is not exempt. The whole apparatus of this course — version control, seeds, baselines, honest statistics, citation hygiene — exists to put you on the right side of that crisis.
Now the AI thread, which runs through every chapter of this book from both directions. AI is, at once, the most powerful research instrument you have ever held and the fastest way yet invented to fabricate knowledge that does not exist. You will use it as a tool — to find papers, draft prose, extract a comparison matrix, write analysis code — and you will study it as a workload, because your semester project investigates how a modern AI method advances your chosen domain. Both halves sit under one spine rule you will hear me repeat until it is reflex:
The human stays in the loop where the judgment lives. AI accelerates the search, the draft, the extraction. You own the claim, the verification, and the account you will give for every number and every citation. An LLM can be an instrument in your method; it can never be an author, and “the model said so” is never a defense.
And then the week’s real question — the one we will engineer toward, not preach about. Solomon writes that “it is the glory of God to conceal things, but the glory of kings is to search things out” (Proverbs 25:2, ESV). The verse frames research as something noble — the dignified labor of searching out what is hidden. So the question for Week 1 is exactly that: what does it mean to search a matter out? Hold it. We will earn an answer by the end.
Coach’s Note — If you are coming from the other books in this library, welcome back; if not, here is the house rule. Learning is a sport. The keyboard is the gym. You do not become a researcher by reading about research any more than you get fit reading about squats. Every chapter ends with Reps and a graded deliverable. This week’s deliverable starts your portfolio and your paper. Show up. Do the work.
1.1 — Knowledge, Not Opinion: What Counts as Research
Start by drawing a line you will defend all semester: the line between a claim you can be wrong about and an opinion you cannot test. Research traffics only in the first kind.
“LLM agents are the future of software engineering” is not a research claim. It is a forecast, an opinion, a vibe. You cannot run an experiment that falsifies it. Compare: “On SWE-bench Verified, an LLM agent with retrieval-augmented context resolves more GitHub issues than the same agent without retrieval, measured by test-suite pass rate.” Now we are talking. That sentence names a benchmark (SWE-bench Verified — 500 human-validated GitHub tasks where success means the hidden test suite passes), a method, a baseline, and a metric. It could be false, and an experiment could show it. That is research.
The discipline of distinguishing the two is the first thing the practicum drills. As you read this week’s three papers, ask of every sentence in the abstract: is this a claim I could falsify, or an opinion dressed as one? The empirical sciences — and CS is overwhelmingly empirical — earn their authority precisely by making claims that risk being wrong and then surviving the test.
Coach’s Note — Beginning researchers confuse “I have a strong opinion” with “I have a hypothesis.” A hypothesis is an opinion that has agreed to lose if the data says so. If you cannot describe the result that would prove you wrong, you do not yet have a research question. Hold that thought; we sharpen it formally in Chapter 6.
1.2 — The Scientific Method, in a CS Key
The textbook scientific method — observe, question, hypothesize, predict, test, analyze, conclude — was written for wet labs, but it maps cleanly onto computer science once you translate the nouns. Here is the loop you will spend the semester walking, and the week each leg gets built:
| Classical step | In a CS practicum | Built in |
|---|---|---|
| Observation | ”Retrieval keeps failing on multi-hop questions.” | Week 2 |
| Question | ”Does GraphRAG beat naïve RAG on multi-hop QA?” | Week 2 |
| Literature | What has prior work already settled or left open? | Weeks 3–4 |
| Hypothesis | A falsifiable prediction with named variables | Week 6 |
| Design | Baselines, ablations, metrics, seeds, splits | Week 7 |
| Pilot | A small run to prove the pipeline works | Week 10 |
| Execution | The real runs, with full provenance | Week 11 |
| Analysis | Honest statistics — effect size, not just p | Week 12 |
| Conclusion | Discussion that interprets without over-claiming | Week 13 |
| Communicate | The paper, the talk, the venue | Weeks 14–16 |
Notice the loop is iterative, not linear. A pilot that fails sends you back to design. A literature review that finds your question already answered sends you back to the question. That is not failure; that is the method working. Research that never loops back is usually research that never risked anything.
Walk one example all the way around, because the loop is abstract until a real question is standing on it. Suppose you observe that a retrieval-augmented assistant keeps stumbling on questions that need two hops of reasoning — “who advised the advisor of X?” Your question sharpens: does a graph-structured retriever (GraphRAG) beat a naïve top-k retriever on multi-hop QA? The literature tells you the dominant failure in RAG systems is retrieval, not generation — so your money is on the retriever, and surveys like arXiv:2501.09136 (“Agentic RAG”) map what has already been tried. Your hypothesis commits to a falsifiable prediction with named variables (independent: retriever type; dependent: answer accuracy on a multi-hop benchmark). The design names baselines and an ablation per claim; the pilot runs on 10% of the data to prove the pipeline is wired correctly before you burn a weekend of compute; execution runs it for real with every seed and config logged; analysis asks not “is p < 0.05?” but “how big is the effect, and does it survive across seeds?”; and the discussion says only what the numbers support. That is one trip around the loop, and it is the spine of your next fifteen weeks.
The single most important leg for a computer scientist is the one the classical method underweights: reproducibility. A chemistry result that cannot be reproduced is suspect. A CS result that cannot be reproduced is not a result at all — it is an anecdote with a graph. This is why your portfolio (next section) starts in Week 1 with a Git repository, and why we will pin environments, set seeds, and document datasets with the seriousness other fields reserve for safety protocols.
1.3 — Basic vs Applied, Research vs Development
Three distinctions sort almost any CS work you will read. Get them straight now, because your Week-2 problem and your whole project depend on placing yourself correctly.
Basic vs applied research. Basic (or fundamental) research seeks knowledge for its own sake — a new complexity bound, a theorem about what is learnable. Applied research aims that search at a concrete problem — does this method improve congestion control on real networks? This practicum is overwhelmingly applied: you will investigate how a modern AI method advances a chosen domain. That is a feature, not a limitation. Applied work that is rigorous is publishable; applied work that is sloppy is a product demo.
Research vs development. This is the one students miss most. Development builds a system that works. Research produces generalizable, verifiable knowledge — a claim that holds beyond your one machine. Building a slick RAG chatbot for your lab is development. Measuring, with baselines and statistics, whether and when retrieval augmentation reduces hallucination on a defined task — and reporting it so another team can confirm it — is research. The deliverable of development is an artifact. The deliverable of research is a claim that survives scrutiny. You may build a system this semester, but the system is your apparatus, not your contribution.
| Development | Research | |
|---|---|---|
| Goal | A working artifact | Verifiable, generalizable knowledge |
| Success | ”It runs / ships" | "The claim survives peer review and reproduction” |
| Output | The system | The paper (and the artifact that backs it) |
| The question | ”Does it work?" | "Is it true, and why, beyond this one case?” |
Coach’s Note — The fastest way to turn a development project into a research project is to add a baseline and a measurement. “I built X” becomes “X beats the standard approach by this much on this benchmark, and here is the evidence.” The system was always the easy part. The measured comparison is the research.
1.4 — What Makes Work Publishable: The Four Pillars
A program committee accepts or rejects your paper on four questions. Learn them in Week 1 and you will read every paper — and write your own — with the reviewer’s eye.
- Novelty. Is it new? Not “new to me” — new to the field. This is why the literature review is non-negotiable: you cannot claim novelty against work you have not read. A reviewer’s most common objection is some variant of “this was already done by [citation].”
- Rigor. Is the method sound? Proper baselines, controls, enough runs to distinguish signal from luck, the right statistical test, honest treatment of confounds. A single-run “we beat the baseline” is rigor-free; deep-learning results vary substantially by seed and initialization (Bouthillier et al., Accounting for Variance in ML Benchmarks, MLSys 2021), so one run tells a reviewer almost nothing.
- Reproducibility. Could another team rerun it and get the same answer? Code, data, environment, seeds. The ACM now issues independent Artifact Review badges — Artifacts Available, Artifacts Evaluated, Results Validated — precisely because reproducibility became a first-class scholarly value.
- Significance. Does it matter? A perfectly rigorous, reproducible, novel result about something nobody cares about is a curiosity. Significance is the “so what” — who benefits, what changes, what becomes possible.
These four are not equal partners. Reproducibility quietly governs the other three, because a result you cannot reproduce is one whose novelty, rigor, and significance you cannot actually trust — and this is the pillar the field has historically been worst at. It is why the ACM, NeurIPS, and the major venues spent the last decade building machinery around it: artifact-evaluation tracks, mandatory reproducibility checklists (the NeurIPS Paper Checklist descends from Joelle Pineau’s ML Reproducibility Checklist), and DOI-minting archives like Zenodo. When you rate a paper’s reproducibility this week, you are rating it on the dimension the whole field is most anxious about — and learning, by contrast, exactly what your own work will have to do to clear the bar.
Hold all four against this week’s three papers using code/structural-analysis-template.txt. Rate each pillar 1–5 and — this is the part that builds the muscle — back the rating with a specific line, table, or figure from the paper. “Feels rigorous” is not analysis. “Reports five seeds with 95% confidence intervals in Table 3” is.
Coach’s Note — When you write your own paper, a reviewer mentally checks these four boxes in this order: new? sound? reproducible? matters? A “no” on any one can sink you. The good news is that three of the four are in your control from Week 1 — rigor, reproducibility, and a clearly-argued significance are discipline, not luck. Only novelty depends on the field, and that is precisely what the literature review (Weeks 3–4) exists to establish. Earn the three you control and you have already out-prepared most submissions.
1.5 — The Five Kinds of CS Research
Computer science is not one method; it is several, and a paper that is excellent as one kind would be rejected if judged as another. Know which kind you are doing, because it sets the bar you will be held to.
| Type | The contribution is… | A 2026 applied-AI example | What reviewers demand |
|---|---|---|---|
| Empirical | A measured finding about how something behaves | ”RLVR sharpens vs expands LLM reasoning” (cite both sides: arXiv:2506.14245 and the “Limit of RLVR” line) | Baselines, multiple seeds, significance |
| Systems | A built system that does something new or better | Aurora RL congestion control (Jay et al., ICML 2019); NVIDIA Programmable CC (arXiv:2207.02295) | A working system + an evaluation against alternatives |
| Theory | A proof, bound, or formal model | Sample-complexity or convergence guarantees | Correct proofs; clearly stated assumptions |
| HCI / user study | A finding about how people use computing | A study of AI-generated feedback on programming assignments | Study design, participants, ethics (IRB), statistics |
| Measurement | A characterization of a system or phenomenon at scale | A 172-billion-token study of how Q&A hallucination rises with context length (arXiv:2603.08274) | Representative data, sound methodology, honest limits |
Most of you will do empirical or systems work — applied AI in a domain lives there. But read across types this week. A measurement paper and a theory paper look completely different on the page, and learning to recognize which game a paper is playing is half of reading efficiently.
Coach’s Note — Match your ambition to your type. If you choose a systems project, a reviewer wants the system and an honest evaluation against a real baseline — building it is necessary but not sufficient. If you choose empirical, the build can be small but the measurement must be airtight. Pick the type that fits the question you actually care about, then meet that type’s bar.
1.6 — Choose Your Domain (and the AI Question Inside It)
This is the most consequential decision you make all term, and you make it this week. Pick a CS domain and keep it all semester. Everything downstream — your literature search, your matrix, your experiment, your paper — lives inside it. The menu:
AI · cybersecurity · networking · software engineering · databases · HCI · computer graphics · AR/VR · robotics · cloud · HPC · computer architecture · data science
And the cross-cutting mandate of this course: investigate how a modern AI method advances that domain. You are not just studying your domain; you are studying the seam where AI meets it. Some seeds, each anchored to a real, verifiable paper you can chase this week:
| Domain | The AI-meets-domain question | A seed to verify and snowball |
|---|---|---|
| Software engineering | Do LLM agents improve code review? | c-CRAB code-review-agent benchmark (arXiv:2603.23448); SWE-bench Verified |
| Networking | Can RL learn better congestion control? | Aurora (Jay et al., ICML 2019); arXiv:2207.02295 |
| Databases | Can an LLM optimize SQL queries? | LLMSTEER query optimization (arXiv:2411.02862) |
| Security | AI vs traditional phishing/prompt-injection detection | arXiv:2602.05484 |
| HCI / edtech | Is AI-generated assignment feedback as good as a TA’s? | (anchor to a verified study in your lit search) |
| HPC | Can RL schedule Slurm jobs better than heuristics? | confirm the DOI/arXiv id before citing — these were not re-verified in the brief |
How do you choose well? Three tests, applied honestly. Curiosity — can you stay interested in this for sixteen weeks, through the boring middle when the pilot keeps breaking? Tractability — can you actually do it with the time, data, and compute you have? A 405-billion-parameter model you cannot fit on any GPU you can rent is not a tractable project; a careful measurement on a public benchmark is. A live AI seam — is there a real, current question where AI meets this domain, or are you retrofitting AI onto a domain that does not need it? The strongest projects sit where all three overlap: a domain you care about, a question you can finish, and a genuine open seam between AI and that domain.
A warning that will save you a week: a gap — the open problem you will eventually fill — is rhetorically the “establish a niche” move (Swales’ CARS). It is easy to assert and hard to verify. Do not write “no one has studied X” on a hunch. Every gap claim you make from here on must be backed by a cited absence — a line in a real paper that says, in effect, “we did not address Y.” We hunt those gaps formally in Chapter 2; this week, just notice them in the Future Work sections as you read.
Coach’s Note — Do not over-optimize this choice. Students freeze for two weeks trying to find the perfect domain and lose more than they would have lost picking a good-enough one and starting. Any domain on that menu, attacked rigorously, will teach you to do research — which is the actual point of a practicum. Pick the one you would happily read about on a Saturday, and commit by Friday.
1.7 — Start the Portfolio and the Paper This Week
Here is what separates this practicum from a class: you begin producing the thesis foundation on day one. Two things get built this week, both in Appendix A.
The research portfolio. A single Git repository that will grow all semester. By Week 1 it holds at least: a README.txt, a lit/ folder (your comparison matrix lives here later), a notebooks/ folder, a data/ folder with a DATASETS.txt, and paper/ holding your template. Commit early, commit often — your commit history is part of the reproducibility story. See Appendix A for the no-install, browser-based path (GitHub + Overleaf + Google Colab / Codespaces) if you are on a locked-down laptop, and the local path if you want it.
The paper template — yes, in Week 1. You will start the manuscript in a real venue template now, so the practicum becomes a submission, not a class project. The two standard choices:
| Template | Class | Use when | Version (as of mid-2026) |
|---|---|---|---|
| ACM acmart | \documentclass[sigconf]{acmart} | Targeting an ACM venue (most CS) | acmart.cls v2.18, dated 2026/06/01 |
| IEEE IEEEtran | \documentclass[conference]{IEEEtran} | Targeting an IEEE venue | IEEEtran.cls v1.8b |
Both are pre-loaded on Overleaf (the no-install path; Appendix A). For ACM, you submit single-column for review (\documentclass[manuscript,review]{acmart}, plus anonymous for double-blind venues, which renders “ANONYMOUS AUTHOR(S)”), then switch to sigconf two-column for camera-ready via ACM’s TAPS pipeline. Copy code/paper.tex into a new Overleaf project today and compile it. It already stubs the right sections and — critically — puts the AI-disclosure where ACM wants it (the Acknowledgements). Verify the literal version string in Overleaf’s package list before you lean on any feature; the brief confirmed v2.18 via search, not a direct fetch.
Coach’s Note — Students resist starting the paper in Week 1 because “there’s nothing to write yet.” Wrong instinct. An empty, correctly-structured template is a map of the work. Every week you fill one more section. By Week 14 the “drafting” is mostly assembly, because you have been writing the paper since the day you read these words.
1.x — Interactive Lab: Paper Anatomy Explorer
Below this chapter on the website you will find an interactive panel called the Paper Anatomy Explorer. Go use it now — it is not optional flavor; it is the rep that wires today’s lesson into your hands.
The Explorer walks you through a sample paper one section at a time — abstract → introduction → related work → method → results → discussion → future work — and for each part shows the job that section does and the one question you should ask of it. Click a section and you see what a reviewer is looking for there: in the abstract, can you restate the contribution? In Related Work, whom does this beat? In Results, are there baselines and error bars, or one lucky run?
Then it flips the script: it hands you an unlabeled paper with the section headings stripped, and asks you to identify each part by its job. You will mislabel some. Most people put a paragraph of Related Work in the Introduction, or read a Discussion limitation as a Result. That confusion is exactly the point — naming the parts by their function, not their position, is the skill that makes the three-pass reading method (next section) actually fast.
Run it before you read your three papers cold, then keep code/paper-anatomy.txt open beside you as your reading template. The Explorer teaches the shape; the template makes you apply it.
1.8 — Reading Like a Scientist: The Three-Pass Method
You cannot read a research paper like a textbook chapter, front to back, every word. You will drown. Use Keshav’s three-pass method — it is the single most valuable reading skill in this course, and this week’s deliverable is named for it.
- Pass 1 (~5 minutes). Title, abstract, intro, section headings, conclusion; skim the references. Goal: decide relevance and answer the five Cs — Category (what type of paper?), Context (what does it relate to?), Correctness (do the assumptions hold?), Contributions (the main claim in one sentence), Clarity. Many papers stop here — relevant enough to cite, not enough to read deeply.
- Pass 2 (~1 hour). Read the body, look hard at every figure and table (results live there), but skip proofs and the finest detail. Mark references you must chase. After this pass you can summarize the paper to someone else.
- Pass 3 (re-implement in your head). Challenge every assumption. Could you reproduce this? Where would it break? This is the pass that surfaces the gap you will eventually occupy — and the pass that catches the over-claim a reviewer would catch.
A crucial skill hides inside Pass 1: the skill of declining to read deeply. Most papers you encounter deserve only five minutes — relevant enough to know exists, not relevant enough to spend an hour on. Beginners feel guilty skimming; experts know that the hour they did not spend on a tangential paper is the hour they spent on the three that actually anchor their work. Reading efficiently is mostly reading selectively. The three-pass method is as much a triage protocol as a reading method.
Use code/paper-anatomy.txt as the column-by-column map and code/structural-analysis-template.txt as the form you fill out per paper. Three papers, three passes each, three filled templates — that is the core of this week’s deliverable.
1.9 — AI as a Research Instrument (and Its Sharpest Failure Mode)
Now the tool half of the thread, applied to this week’s work. AI will genuinely accelerate your reading: tools like Semantic Scholar (TLDR summaries), Elicit, and Consensus can surface and summarize papers fast, and we map the whole literature toolkit in Chapter 3 and Appendix B. Use them. But understand the knife attached, because in research it is uniquely dangerous.
LLMs fabricate citations. This is not an occasional glitch; it is a measured, structural failure mode. Walters & Wilder (2023, Scientific Reports) found 55% of GPT-3.5 and 18% of GPT-4 citations were entirely fabricated; among the citations that pointed to real works, 43% (GPT-3.5) and 24% (GPT-4) still had substantive errors. A separate study (Bhattacharyya 2023, Cureus) found 87% of citations to real works carried at least one metadata error. Grounded/RAG tools hallucinate less — but still not zero, and they must be verified per-citation against a real index. These percentages come from 2023-era models and shift as retrieval improves; treat them as illustrative and point-in-time, but treat the risk as permanent.
The consequence is professional, not academic. At ICCV, a paper with a non-existent citation can be rejected without review. arXiv announced (as of May 2026, via secondary reporting of a moderator announcement — verify the exact terms) a possible one-year submission ban for incontrovertible evidence of unchecked LLM content: hallucinated references, leftover meta-comments like “would you like me to make any changes?”, unreplaced placeholder data. “The model gave me that citation” is not a defense anywhere.
So we build a habit and a tiny tool. code/verify_citation.py queries the free Semantic Scholar API for a title and shows you the closest matches so you can verify by eye:
$ python3 code/verify_citation.py "Attention Is All You Need"
Top matches for "Attention Is All You Need" (verify by eye — do not trust blindly):
- Attention is All you Need (2017)
Ashish Vaswani, Noam Shazeer, Niki Parmar | NIPS | no DOI/arXiv id
A human still owns the verdict on every reference in your bibliography.
It does not prove a citation is correct — only you, against the publisher, dblp, or Semantic Scholar, can do that. It clears the obvious case (no match at all is a red flag) so your judgment can spend itself on the subtle one: the plausible title with the wrong year or the wrong authors — exactly the record an LLM invents. That division of labor — machine handles the mechanical, human owns the meaning — is the whole spine rule in one script.
Coach’s Note — Run every citation an LLM hands you through a real index before it touches your bibliography. Not most. Every one. This is the single habit that most reliably separates a graduate researcher from someone who got burned in their first submission. The full venue disclosure rules — who is never an author, where disclosure goes for ACM/IEEE/NeurIPS/arXiv — live in Appendix C. Read it before you submit anything anywhere.
1.10 — Searching a Matter Out: Research as a Faithful Vocation
Now the week’s question, given its due. What does it mean to search a matter out?
The verse is Proverbs 25:2: “It is the glory of God to conceal things, but the glory of kings is to search things out” (ESV). Read it slowly, because it dignifies the entire enterprise you just signed up for. The world is made with things concealed in it — structure that is real but not yet seen. And the searching-out of that hidden order is not idle curiosity or mere ambition; the text calls it glory, and it sets it beside the work of kings. To investigate creation carefully is a noble calling, not a lesser one. The Christian tradition has long held that studying the world is a way of thinking God’s thoughts after Him — that the orderliness which makes science possible is not an accident to be exploited but a gift to be received with care.
That framing changes how you do the work, not just why. If the truth you are searching out is real and given — not constructed by you, not yours to bend — then your relationship to your data is one of receiving, not manufacturing. This is the deepest reason research integrity is the spine of this course and not a footnote. Fabricating data is not merely against the rules; it is a refusal to search the matter out — it is replacing what is with what you wish were so. Cherry-picking results, hiding the runs that “didn’t work,” reporting a p-value you fished for, citing a paper you never read, letting an LLM invent a reference you never checked — each is a small lie about what is actually there. And the kingly dignity of the work is forfeited the moment you stop telling the truth about the world.
The AI era sharpens this rather than dissolving it. The copilot offers to launder your accountability — “the model wrote it,” “the agent found the citation,” “the tool ran the analysis.” But a steward of the truth cannot speak that way. Every major venue agrees an LLM can never be an author, because authorship requires accountability a tool cannot bear (ICMJE’s four criteria; ACM, IEEE, NeurIPS, ICLR all concur). When your name is on the paper, the model’s confidence is not your defense; your verification — or its absence — is what you will answer for. LCMS theology calls honest work like this a vocation: a calling through which you serve your neighbor. Your neighbor here is the next researcher who builds on your result, trusting it is true. To search a matter out faithfully is to hand that neighbor knowledge they can stand on. That is the glory the verse is talking about, and it is the standard this course will hold you to for sixteen weeks.
1.11 — Common Pitfalls
Pitfall: Confusing a topic with a research question. Example: “I’m researching LLM agents for code review.” That is a topic; it names no claim, no baseline, no metric, nothing that could be wrong. Fix: Push it to a falsifiable claim: “Do LLM agents resolve more SWE-bench Verified issues than a non-agentic baseline?” If you cannot state the result that would prove you wrong, you do not yet have a question. (Formalized in Chapter 6.)
Pitfall: Mistaking development for research. Example: Building a polished RAG chatbot and calling it your contribution. Fix: A built system is your apparatus, not your finding. Add a baseline and a measurement: whether and when retrieval reduces hallucination on a defined task, reported so another team can confirm it.
Pitfall: Trusting an AI-supplied citation because the title and authors look right.
Example: You paste five references from a chatbot into your bibliography; two are fabricated and one has the wrong year.
Fix: Verify every citation against dblp / Semantic Scholar / the publisher before it enters your portfolio. Run code/verify_citation.py as the first pass; a no-match is a red flag, but a plausible near-miss is the real danger. (55% of GPT-3.5 / 18% of GPT-4 citations were fabricated — Walters & Wilder 2023.)
Pitfall: Asserting a research gap on a hunch. Example: Writing “no one has studied X” because you, personally, have not seen it. Fix: Back every gap with a cited absence — a line in a real paper admitting Y was not addressed. “Gap exists” is easy to claim and hard to verify; require evidence of yourself the way a reviewer will. (Gap-hunting is Chapter 2.)
Pitfall: Reading every paper front-to-back, every word. Example: Three hours into one paper, you have read nothing else and missed the field’s shape. Fix: Use the three-pass method (§1.8). Pass 1 in five minutes decides relevance; most papers earn only a Pass 1. Reserve deep reading for the few that anchor your work.
Pitfall: Judging (or building) a paper by the wrong research type’s bar. Example: Dismissing a measurement paper for “not proposing a new method,” or shipping a systems project with no baseline because “the system is the contribution.” Fix: Identify the research type first (§1.5), then hold it to that type’s standard. A systems paper needs an evaluation against alternatives; an empirical one needs seeds and significance.
1.12 — Reps
The work is in the exercises. The keyboard is the gym; this is where Week 1 gets into your hands and your project starts to move. A preview of what is waiting:
- Choose your domain and write the one-sentence AI-meets-domain question you will chase all semester.
- Read three recent conference papers with the three-pass method and fill
code/structural-analysis-template.txtfor each — verifying every citation first. - Score a paper on the four pillars (novelty / rigor / reproducibility / significance), backing each rating with a specific line.
- Run
code/verify_citation.pyagainst five LLM-supplied citations and report how many were fabricated or wrong. - Initialize your portfolio repo and your Overleaf paper from
code/paper.tex, and commit.
A short Check Your Reps quiz is embedded on this page, right under the chapter — five questions, grounded in exactly what you just read. Take it before you move on.
1.13 — This Week’s Deliverable
Your first deliverable is P1 — “Read Like a Scientist,” specified in Project 1. You will choose and commit to a domain, set up your research portfolio and ACM/IEEE paper template, read three recent conference papers in your domain, and produce a structural analysis of each plus a reflection on your research interests. It is the foundation everything else is built on.
At a high level: Normal tier delivers the domain choice, the live portfolio + template, three verified-citation structural analyses, and the reflection. Medium tier deepens the analysis — snowball one paper’s references, rate all four pillars with evidence, identify a candidate gap from a Future Work section. Hard tier asks for a one-page memo that takes a position: which one of the four pillars is the real bottleneck for progress in your chosen domain right now, and what that implies for where you should aim your own project. That memo is the judgment no tool can make for you, and it is where the thesis of the course gets graded.
1.14 — Coach’s Final Word
Here is what I want you to carry out of Week 1. Research is not a topic you have feelings about; it is a systematic search for verifiable knowledge, and it lives or dies on four pillars and one virtue. The pillars are novelty, rigor, reproducibility, and significance — you now read every paper, and will write your own, through that lens. The virtue is integrity, and in the AI era it is under more pressure, not less, because the tools that accelerate the search are the same tools that can fabricate its findings without blinking.
You have a domain now — or you will by the end of the reps. You have a portfolio repo and an empty, correctly-shaped paper that is already a map of the next fifteen weeks. You have a reading method that turns a terrifying stack of papers into a fast, repeatable craft. And you have a habit that will save your career more than once: verify every citation, every number, every claim — yours and the machine’s — against what is actually there.
That habit is the whole of this week’s question. To search a matter out is to insist on what is true over what is convenient, to receive the order in the world rather than invent a flattering version of it, and to put your name on knowledge your neighbor can stand on. The AI will draft, summarize, and suggest. You decide, you verify, and you answer for it. The glory of kings is to search things out. Do it like it is glory.
Now go do the reps. The Paper Anatomy Explorer is waiting right below this page, the templates are in code/, and Project 1 is where it all comes together.
See you on Monday.
Up next: Read the exercises and do all of Week 1’s reps, then build Project 1 — P1: Read Like a Scientist. Set up your portfolio and Overleaf template from Appendix A, get to know the literature toolkit in Appendix B, and read Appendix C for the AI-in-research integrity rules you will live under all semester (glossary in Appendix D). Then Chapter 2 — Finding a Problem Worth Solving.