Chapter 11 — Testing, Quality, and the Defect Log

“Program testing can be used to show the presence of bugs, but never to show their absence.” — Edsger W. Dijkstra, Notes on Structured Programming (1970)

“Search me, O God, and know my heart! Try me and know my thoughts! And see if there be any grievous way in me, and lead me in the way everlasting!” — Psalm 139:23–24 (ESV)

Why This Matters

Last week you built the core. It works — on your machine, with your data, when you drive it the way you meant it to be driven. That is not the same as working, and this is the week you find out the difference.

This week you wear the tester’s hat, and it is the hardest hat in the set, because you are testing something you built. The developer in you wants the software to pass. The tester in you is paid to make it fail. Same person, same fifteen hours, opposite incentives. In industry these are two people precisely because the conflict is real. You do not get two people. You get discipline instead — a written plan, a matrix that does not care about your feelings, and a log that records every fault you find under your own name.

Where you are in the life cycle: verification and validation. You have finished requirements (Weeks 3–4), design (Weeks 5–6), planning (Week 7), and a first pass of construction (Weeks 9–10). Verification asks did we build the thing right — does the code satisfy the specification? Validation asks did we build the right thing — does the specification satisfy the user? Both are your job. Both produce artifacts that are graded in Week 16: a test plan, a test suite that runs from one command, a traceability matrix, results, and a defect log. Those are not paperwork about the work. In the Week-16 rubric they are work, worth real points, and there is no way to reconstruct them in Week 15 from memory.

Here is the sentence that should make you uncomfortable. Right now, in your repository, there are defects you already suspect. The edge case you skipped because the demo did not need it. The error path that returns a 200 with an empty body. The date arithmetic that you know is wrong at a boundary but has never come up. You are not going to find those with a debugger. You are going to find them by writing down what you promised in Week 3 and then, one promise at a time, trying to prove you did not keep it.

The AI thread runs hot this week, from both sides. As a tool, an assistant is genuinely excellent at generating test cases and edge cases from a requirement — it is one of the strongest uses in this entire course, and you should use it. It is also, with total confidence and perfect syntax, going to assert the wrong thing: it will test what your code currently does rather than what your requirement says, and a suite full of those is worse than no suite, because it turns every future fix into a red build. As a workload, if your project calls a model, you now have to test something that does not return the same answer twice. Deterministic assertions do not survive that. Property assertions, golden sets, and tolerances do.

And the question underneath the week, which is older than software: why does an honest builder look for his own faults? Every instinct says to look away. The psalmist does the opposite — he asks to be searched. We will take that seriously in 11.13, because it is the difference between a tester and someone performing testing.

11.1 — The Tester Hat, Worn by the Person Who Wrote the Bug

Three roles are fighting for your fifteen hours this week, and you are all of them.

The failure mode of a solo capstone is that the developer wins all three arguments. You “test” by using the app the way you always use it, find nothing, and conclude there is nothing. Confirmation bias is not a character flaw here; it is the predictable result of asking the author to grade the paper. Three structural fixes, all cheap: write the test cases before you run anything, derived from the requirement text in docs/requirements.md rather than from the code — read the code first and you will write tests that agree with it. Open each session by reading your own exit criteria out loud, which replaces “does it seem fine” with “have I met the number I committed to.” And count finds, not passes — a session where everything passed means either the software is good or the testing was, and you should assume the second until the matrix says otherwise.

Coach’s Note — The tester’s hat has one job the developer’s hat cannot do: it wants the bad news early. Every hour a defect survives undiscovered, it gets more expensive — code gets built on top of it, documentation describes it, and eventually a stranger meets it in Week 16 with your name on the commit. Find it now, while it is cheap and nobody is watching.

11.2 — The Test Plan: Scope, Levels, Environment, Entry and Exit Criteria

A test plan is a short document that answers five questions before you write a single assertion. Yours goes in docs/test-plan.md, and the template is code/test-plan-template.md.

1. Scope — which requirements, at which build. Name the commit SHA. “The system” is not a scope.
2. Out of scope — what you are deliberately not testing, and why. This section separates a professional from a student: it is the only place you get to be honest about your limits in advance rather than apologizing for them afterward.
3. Levels — unit, integration, acceptance: what runs at each level, by what command, roughly how many.
4. Environment — runtime version, OS, database, third-party services (real, sandboxed, or stubbed), and the data set.
5. Entry and exit criteria — when you may begin, and when you are allowed to say done.

The classic outline descends from IEEE 829, the old standard for software test documentation; the ISO/IEC/IEEE 29119 family has since taken over that ground. You need neither document to pass this course — but when an employer hands you a test plan template in your first job, that is the lineage it came from, and you will recognize every heading.

Exit criteria are the whole point of the document, so let us do this book’s signature move and put the bad one next to the good one.

BAD EXIT CRITERIA
Testing is complete when the application is stable and the major
features work as expected with no significant bugs remaining.

GOOD EXIT CRITERIA
Verification is complete when ALL of the following hold at the tagged build:
  1. 100% of Must requirements have >= 1 passing acceptance test (per docs/traceability.csv).
  2. Zero open defects at severity S1 or S2.
  3. Every open S3/S4 defect is in docs/defect-log.md with a stated workaround.
  4. Every defect fixed in this iteration has a named regression test that fails
     against the pre-fix commit.
  5. The full suite runs from `make test` and passes in CI on a clean checkout.
  6. p95 response time for the dashboard route is <= 800 ms on the seed data set (NFR-02).

The bad version has four undefined words — stable, major, as expected, significant — and every one of them will be defined, on the last night, by a tired person who wants to go to bed. That is not a criterion. That is permission. In the good version you can be wrong about criterion 6; you cannot be vague about it. Every line is something a stranger could check without asking you a question, which is exactly the test your Week-16 grader will apply.

Coach’s Note — Write the exit criteria on Monday, before you know how the week is going. Criteria written on Friday are always, mysteriously, criteria the current build happens to meet.

11.3 — Unit, Integration, Acceptance: What Each Level Can and Cannot Tell You

Levels exist because different mistakes hide at different scales. A test at the wrong level does not just cost more — it gives you the wrong information.

Mike Cohn’s test pyramid (Succeeding with Agile, 2009) is the shape to aim at: many fast unit tests at the base, fewer integration tests in the middle, a thin layer of acceptance tests on top. The inverted version — a handful of unit tests holding up a mountain of slow, flaky end-to-end tests — is a well-known anti-pattern for the obvious reason: the suite gets so slow and so unreliable that you stop running it, and a suite you do not run is a suite that does not exist.

For a capstone at your scale, a defensible target is roughly 30–60 unit tests, 5–15 integration tests, and one acceptance test per Must requirement — not sacred numbers, just the shape you should have to argue your way out of. And the seam question matters more than the count: an integration test is only worth its runtime if it crosses a seam where you actually drew a boundary in Week 6 — your code to the database, to the third-party API, to the model provider. Mocking both sides of a seam and then testing the mock is theater.

PantryPilot, concretely. FR-11 is barcode lookup via a third-party product API. The unit test covers the parser: given a JSON payload shaped like the vendor’s, produce the right product record — including the payload where brand is missing. The integration test covers the client against a stub server that returns a real captured response, a 429, and a timeout; it answers does my code do something sane when the vendor misbehaves? The acceptance test covers FR-11-AC1 end to end: scan a known barcode, see the item appear in the pantry with the right name and unit. Three tests, three different failures, none of which the other two would have caught.

TraceLens, for contrast. A command-line log parser has no UI and no auth, so its acceptance tests are golden-file tests: feed a fixed input log, compare stdout byte-for-byte against a committed expected file. Nearly all its weight sits in units and goldens; the pyramid is squatter. Same discipline, different shape — and your project will have its own.

11.4 — From Acceptance Criteria to Test Cases, One to One

This is the load-bearing idea of the week, and it is why Chapter 3 made you write acceptance criteria in the first place.

Every acceptance criterion becomes at least one test case. Every test case names the criterion it verifies. If a criterion has no test, you have not verified that requirement — no matter what your coverage tool says.

That mapping is the traceability matrix, and you started one in Chapter 4. This week you do not start a new one — you expand the file you already have into the instrument that tells you whether you are done. The Week-4 matrix carried one row per requirement, with placeholders in the design and test columns that you promised to fill in later. This is later. What changes is the grain: one row per acceptance criterion instead of one per requirement, so a requirement with three criteria becomes three rows and each row can carry its own verdict. A worked sample of the expanded shape lives at code/traceability-matrix.csv — read it for the shape, then grow your own file. Do not copy it over docs/traceability.csv; that file has twelve points of Week-4 work in it.

Here is exactly how the two line up.

coverage_report.py looks up the columns it needs by name and ignores every other column in the file, so the Week-4 columns ride along at no cost. One consequence to plan for: if you wired Week 4’s check-traceability.py into CI, it will now report a duplicate identifier on every requirement that has more than one criterion, because repeated req_id values are the whole point of the new grain. Point it at a per-requirement export, or retire it this week and let the coverage script be the gate.

Here is a bad test case and a good one, for the same criterion — a requirement from PantryPilot’s SRS.

THE REQUIREMENT
FR-07  The system shall list every pantry item whose expiry date falls within
       the next 3 days, inclusive of today.
       FR-07-AC1: An item expiring in 2 days appears in "Expiring soon".
       FR-07-AC2: An item expiring today appears in "Expiring soon".
       FR-07-AC3: An item expiring in 4 days does NOT appear in "Expiring soon".

BAD TEST CASE
TC-021: Test the expiry warning.
  Steps: Add some items. Look at the dashboard.
  Expected: The right items are shown.

GOOD TEST CASE
TC-021  Verifies: FR-07-AC2      Level: unit      Data: seed set `small`
Preconditions: system date fixed at 2026-11-06 (clock injected, not the OS clock).
Steps:
  1. Insert item "milk", expiry = 2026-11-06.
  2. Call expiring_soon(window_days=3).
Expected: the returned list contains exactly one item, name == "milk".
Fails if:  the list is empty (off-by-one: window starts tomorrow).

Nobody but the author can run the bad one, and the author cannot run it twice the same way: no data, no boundary, no oracle. “The right items” is a memory, not an expectation. Four things made it good: it names the criterion it verifies, it pins the clock so it cannot pass in November and fail in December, it states the expected result as an observable value, and it names the specific failure it is hunting. A test that does not know what bug it is looking for usually is not looking.

Boundaries are where the bugs live. For any requirement with a number, a date, or a length in it, do equivalence partitioning and boundary value analysis — techniques that go back to the classic testing literature and have never stopped earning their keep. Partition the input space into classes that should behave the same, then test the edges of each class, because that is where the < should have been a <=.

Five boundary tests for one requirement. That is not excessive; that is the requirement, tested. FR-07-AC2 — the today case — is exactly the one a hurried developer skips, and it is exactly the one that becomes DEF-014 in code/defect-report-template.md.

Coach’s Note — Line coverage tells you which lines a test touched. It does not tell you whether the test asserted anything true — you can reach 90% line coverage with a suite that calls every function and asserts nothing. Verified-requirement coverage — the fraction of your requirements whose every acceptance criterion has a passing test — cannot be gamed that way, because each point of it is a promise you kept. Report that number in your milestone. Report line coverage second, if at all.

Run python3 coverage_report.py from code/coverage_report.py against the sample matrix and read the output carefully. It computes verified-requirement coverage, and then does the more useful thing: it lists the criteria with no test case at all, the ones tested and failing, and the ones where a test exists but was never run. That last bucket is the one that quietly lies to people. A test that has never executed is not coverage. It is an intention.

11.5 — Test Data That Resembles Reality

Most student test suites pass because they are fed data that could not exist. Names are "test", quantities are 1, dates are today, and every string is ASCII. Then a real user types an apostrophe into a surname and the whole thing folds.

Build three families of test data and check all three into the repository.

Four rules that will save you real pain.

• Generate it, do not hand-type it. A small script or a faker library (Faker for Python, faker-js for Node, and equivalents in most ecosystems) gives you 50 records in a minute and lets you regenerate deterministically from a fixed seed. Commit the seed, and the data is reproducible on the grader’s machine.
• Never use real personal data. Ever. Not your roommates’ real names, not a scraped customer list, not a screenshot with an email address in it. If your project touches real people — and PantryPilot does, it has actual roommates — your test data is synthetic and your NFRs from Chapter 4 say so in writing. This is not squeamishness; it is the privacy requirement you already committed to, being enforced by you, on yourself, when it is inconvenient.
• Fix the clock. Any test that depends on “now” must inject a fixed date. A suite that passes today and fails on the 1st of March is a suite nobody will trust, and you will spend a Week-14 evening on it.
• Make setup one command. If seeding takes six manual steps, you will stop reseeding, your tests will start depending on leftover state, and you will get failures that vanish when you look at them.

11.6 — Writing a Defect Report Someone Else Could Act On

A defect report is a message to a stranger. That stranger might be a teammate, a grader, a future employer reading your public repository — or you, in Week 15, at eleven at night, with no memory of what you meant.

The bar: someone who has never seen your project should be able to reproduce the defect from the report alone. Seven fields, every time. They are laid out in code/defect-report-template.md: ID and title, environment, steps to reproduce, expected vs. actual, evidence, severity and priority, traceability and status.

BAD DEFECT REPORT
Expiry thing is broken. Doesn't work right sometimes. Need to fix before demo.

GOOD DEFECT REPORT
DEF-014 — Expiry warning misses items expiring today
Environment: commit a1b2c3d, macOS 15, seed data set `small`, clock pinned 2026-11-06
Requirement: FR-07 (FR-07-AC2)      Test case: TC-021

Steps to reproduce:
  1. Load seed data set `small`.
  2. Add item "milk" with expiry date = 2026-11-06 (today).
  3. Open the dashboard.
Expected: "milk" listed under "Expiring soon" (FR-07-AC2 — inclusive of today).
Actual:   "Expiring soon" is empty; "milk" appears only under "All items".
Evidence: TC-021 fails. Log: `expiry_window: start=2026-11-07 end=2026-11-09`
          — the window starts tomorrow, not today.
Severity: S2 — a Must requirement is unmet and the feature's entire purpose fails.
Priority: P1 — on the demo path; suspected one-line off-by-one.
Status:   Open. Owner: <you>.

Of the bad one, ask: what build? Broken how? Which items? What did you expect instead? “Sometimes” is the most expensive word in a bug tracker, because it means the reporter did not find the condition and now the next person starts from zero. The good report costs four extra minutes and saves an hour — and it does something the bad one cannot: it narrows the bug. Writing “expected vs. actual” in the observable world, with the log line beside it, is frequently the moment you realize you already know the fix.

Coach’s Note — Log the defect before you fix it, even when the fix is thirty seconds away. The log is graded, and more importantly the log is the only evidence that you found things. A repository with an empty defect log tells a reviewer one of two things: the software is flawless, or the testing was. They will not guess in your favor.

11.7 — Severity Is Not Priority: Triaging With a Deadline

Students collapse these two into one number and then make bad calls all week. They are different axes with different owners.

Severity is a fact about the product: how much damage does this do to a user? Priority is a decision about the calendar: when do we fix it, given what remains? In industry a tester or product owner sets severity and an engineering lead sets priority; here you set both, so state which hat you were wearing on each line. Your Week-16 reviewer will notice if every S4 is also a P1 on the demo path.

The two combinations that teach the distinction are the off-diagonal ones. High severity, low priority: your admin export crashes on data sets over 100,000 rows — S1, it is a crash, but P3, because no user in your scope has more than 800 rows and you will say so in the log. You ship it, documented. Low severity, high priority: the project’s name is misspelled on the landing page — S4, nothing is broken, but P1, because it is the first thing the grader and your audience will see and the fix is ten seconds.

A defect that is S1/P1 does not need a decision. Everything else does, and the decision is worth more than the fix, because it is the one thing here an assistant cannot make for you: it depends on who your users are, what your demo shows, and how many hours you actually have left.

BAD TRIAGE — everything urgent, so nothing is; no reasons, so you re-litigate it all next week
DEF-004  Sev: High   Pri: High   "Add item validation broken"
DEF-011  Sev: High   Pri: High   "Accessibility"
DEF-014  Sev: High   Pri: High   "Expiry off by one"

GOOD TRIAGE
DEF-014  S2 / P1  Must requirement FR-07 unmet; on the demo path; ~1 line.  -> fix first
DEF-004  S2 / P2  FR-03 rejects valid units ("dozen"); workaround: type "12". -> fix this week
DEF-011  S2 / P2  NFR-05: item form inputs have no labels; screen reader unusable;
                  it is a Must NFR and a real user harm.                      -> fix this week
DEF-022  S3 / P3  Sort is unstable for equal expiry dates; cosmetic ordering.  -> ship if needed
DEF-025  S4 / P4  Footer year hard-coded. Will not fix in v1.0; noted in log.  -> ship

Each line carries the reason, so a stranger — and your Friday self — can audit the judgment rather than just the outcome.

11.8 — Regression Discipline and the Test That Must Never Go Red Again

A regression is a defect in something that used to work. They are the most demoralizing bugs in a capstone because they cost you twice: once when you built it, once when you broke it. And in Weeks 12–14, when you are integrating and refactoring under time pressure, they arrive in clusters.

The discipline is one non-negotiable rule. When you fix a defect, first write the test that fails because of it. Then fix it. Then watch the test go green. Then name that test in the defect log.

Not “write a test afterward.” First. A test written after the fix proves only that the code does what it currently does. A test written before the fix proves that it catches this specific bug — you watched it catch it. That test is now permanent. It must never go red again, and if it does, you have reintroduced a bug you already paid for.

# 1. Reproduce as a failing test (RED) — commit it on its own
git commit -m "test: TC-021b reproduces DEF-014 (expiry window excludes today)"

# 2. Fix (GREEN) — the fix commit references the defect
git commit -m "fix(FR-07): make expiry window inclusive of today — closes DEF-014"

# 3. Prove the regression test is real
git stash && <run TC-021b>   # expect RED against the pre-fix code
git stash pop && <run TC-021b>  # expect GREEN

That third step is the one people skip, and it is the one that catches the embarrassing case where your new test passes against the broken code too — meaning it never tested the bug at all. Two more habits keep the suite trustworthy:

• Zero tolerance for flaky tests. A test that passes 90% of the time is worse than a deleted test, because it trains you to re-run and shrug. Fix it (usually: a fixed clock, a seeded random, an explicit wait instead of a sleep, an isolated database) or delete it and log why. Do not leave it there blinking.
• The suite runs on every push. You built CI in Chapter 9. This is the week it starts earning its keep. A red main branch is a stop-the-line event, not a background condition.

Boris Beizer named the trap you will hit around Week 13: the pesticide paradox — run the same tests long enough and the bugs they can catch are already gone, so the suite keeps passing while the remaining defects are the ones your methods are blind to. The counter is to periodically change the kind of testing you do: exploratory sessions, hostile data, a new boundary, a different level. A green suite means “the bugs I know how to look for are absent.” It never means “there are no bugs.”

11.9 — Testing What Isn’t Deterministic

If your project calls a language model — for PantryPilot’s recipe suggestion (FR-14), for a summary, for a classification — you now own a component that returns a different answer to the same input. Every assertion you know how to write breaks. Students respond by not testing it, and then it is the one feature that fails live in Week 16. You cannot assert equality. You can assert four other things.

1. Structure and contract. The most valuable assertions, and they are fully deterministic. The output parses as JSON. It has the required keys. servings is an integer between 1 and 12. Every ingredient named appears in the pantry list you passed in. If the model returns prose instead of JSON, your code raises a typed error rather than crashing on undefined.

2. Properties, not values. Property-based testing (Hypothesis for Python, fast-check for JS/TS, jqwik for the JVM, and QuickCheck’s descendants elsewhere) generates many inputs and checks an invariant that must hold across all of them. For a recipe suggester: no suggestion ever names an ingredient the household does not have. That property is true for every valid output, and it is exactly the failure a user would actually notice.

3. A golden set with tolerance. Fix 15–25 representative inputs with expected characteristics — not exact strings. Run them, record results, and set a threshold you commit to in the test plan: “at least 13 of 15 golden inputs produce a valid, in-pantry recipe.” Re-run when you change the prompt or the model. This turns “it seems better now” into a number, which is the entire point.

4. Failure behavior — this is the one that saves your demo. Test the paths that do not depend on the model at all:

All five are deterministic tests against a stubbed client. Write them. In Week 16, the network in the presentation room will be someone else’s, and the feature that degrades gracefully looks engineered while the one that throws a stack trace looks unfinished.

Coach’s Note — Never let your automated suite make live paid API calls. Stub the client by default and put the small number of real-provider tests behind an explicit flag you run by hand. Otherwise CI will run them on every push, and you will learn about your free-tier limits from a bill or a lockout at the worst possible moment. Model pricing and free tiers change constantly — check your provider’s current terms before you rely on any of it.

11.10 — Letting an Assistant Write Your Tests (and Catching What It Gets Wrong)

Generating test cases from a requirement is one of the highest-yield uses of an assistant in this whole course. It is fast, it is broad, and it will name edge cases you did not think of — Unicode, empty collections, timezone boundaries, concurrent writes. Use it. Then audit it, because the failure mode is specific and expensive.

Prompt it with the requirement, not the code. This is the whole trick:

Here is requirement FR-07 and its acceptance criteria, verbatim: <paste>.
Do NOT look at my implementation.
1. List the equivalence partitions and the boundary values for this requirement.
2. For each acceptance criterion, write one test case: preconditions, steps,
   expected result, and the specific bug it would catch.
3. List three edge cases my acceptance criteria do not currently cover, and say
   what the requirement should say about each.

Step 3 is worth the whole exercise. Roughly a third of the time it will surface a genuine hole in your requirement, which is a Chapter 3 defect discovered in Week 11 — cheap now, expensive later. Log it as a defect against the specification and take it to change control in Chapter 12.

Now the failure modes, in the order they will bite you.

• The change detector. Show the model your code and it writes tests that assert what the code does. If the code has an off-by-one, the generated test asserts the off-by-one — confidently, with a green check mark and a name like test_expiry_window_correct. Now your suite actively defends the bug, and fixing it turns the build red. This is why you prompt with the requirement and keep the implementation out of the context.
• The empty assertion. Tests that call a function, wrap it in a try/except, and assert nothing meaningful. They raise line coverage and verify nothing. Read every generated assertion and ask: what value of what variable is being checked, and would a real bug change it?
• Invented API. Generated tests call methods that do not exist, fixtures you never defined, or a library you are not using. Cheap to catch — it will not run — but it eats time.
• Confident wrong oracles. The model will state an expected value derived from the sound of the requirement rather than its text. For FR-07 it may cheerfully assert that “within 3 days” excludes today. Only your specification decides that, and only you can check it.

The rule for the week: an assistant may propose test cases; only you may approve an assertion. Every assertion in your suite is a claim about what your software is supposed to do, and you signed the requirements. Record what you used, and for what, in docs/ai-usage.md — the log you started in Chapter 10, which the final package requires.

Coach’s Note — Here is a cheap, honest metric worth putting in your milestone. Take one requirement, have the assistant generate ten test cases, then review each against your specification. Count how many asserted the wrong thing. That number is your review burden, measured rather than guessed — and it is the number that tells you how fast you are actually allowed to accept generated tests.

11.11 — Good Enough: Exit Criteria and Shipping With Known Defects

You will not ship zero defects. Nobody does. Dijkstra’s line at the top of this chapter is not pessimism — it is arithmetic: testing samples the input space, and the space is effectively infinite. So “quality” cannot mean “no bugs.” It has to mean something you can actually check.

Quality means: every promise you wrote down is verified, and every defect you know about is written down, classified, and decided. That last clause is the professional part. Shipping with known defects is not a failure; shipping with unrecorded defects is. A release with a documented list of open S3s and their workarounds is a mature release. A release that claims everything works, with a defect log containing three entries all marked “fixed,” is a release nobody experienced believes.

So your release notes in Week 14 will carry a Known Issues section, and it will be honest:

Known issues in v1.0
  DEF-022 (S3): Items with identical expiry dates sort inconsistently between
    page loads. Workaround: sort by name. Fix planned for v1.1.
  DEF-025 (S4): Footer copyright year is hard-coded to 2026.
  DEF-031 (S3): Barcode lookup returns "Unknown product" for regional items not
    present in the vendor catalog. Item can still be added manually (FR-03).

Three lines. They cost nothing and buy enormous credibility, because they prove you knew your own system well enough to find its edges. And when the numbers say you are not done, the honest options look like this:

Cutting scope on the record is a legitimate engineering decision, and Week 12 gives you the machinery for it. Quietly not testing something is not a decision at all. It is just a thing that happened to you.

11.12 — Interactive Lab: The Traceability Tester

Below this chapter on the website is The Traceability Tester. Do the lab before you build your own matrix; it is twenty minutes and it will change how you fill in the spreadsheet.

The first panel is a coverage matrix. You map requirement identifiers to test cases and mark each one pass, fail, or not run. The widget renders the matrix and then does the thing your spreadsheet will not do for you: it highlights every requirement with no test at all and every acceptance criterion with no matching assertion, and it computes verified-requirement coverage — the fraction of requirements whose criteria are all covered by passing tests. Watch what happens to that number when you mark a test “not run.” It drops, as it should. That is the honesty line-coverage tools do not give you.

The second panel is a defect triage queue. Incoming defects arrive with a description; you assign each a severity and a priority. The widget computes the resulting fix order and — this is the part worth sitting with — shows you what ships with known defects, because something always does. Try assigning P1 to everything and watch the queue become useless. Then try being honest about your remaining hours and watch a real release plan appear.

What it teaches: that coverage is a claim about promises rather than lines, that severity and priority are genuinely different axes, and that “done” is a set of numbers you commit to in advance rather than a feeling you arrive at on Friday.

11.13 — Why Does an Honest Builder Look for His Own Faults?

“Search me, O God, and know my heart! Try me and know my thoughts! And see if there be any grievous way in me, and lead me in the way everlasting!” (Psalm 139:23–24, ESV)

Read that as a tester and it is startling. The psalmist does not ask to be told he is fine. He asks to be searched, tried, and shown the grievous way. He is requesting the defect report. Volunteering for the audit. And he does it while already knowing — the whole psalm is about this — that he is completely known already. There is nothing to hide, so the only reasonable posture is to stop hiding.

That is the psychological engine of good testing, and it is why testing your own code is so much harder than testing someone else’s. The natural instinct is self-protection: run the happy path, see it work, feel good, move on. The instinct is not stupidity. It is the same instinct that makes us defensive when a friend tells us something true. Jeremiah puts it bluntly: “The heart is deceitful above all things, and desperately sick; who can understand it?” (Jeremiah 17:9, ESV). We are not neutral observers of our own work. We have a stake. And a stake, in testing terms, is a bias in the measuring instrument.

The Christian answer to that bias is not “try harder to be objective.” It is ask to be searched. Psalm 19:12 — “Who can discern his errors? Declare me innocent from hidden faults” (ESV) — names the real problem exactly: the faults that matter most are the ones you cannot see, which means no amount of sincerity will find them. You need something outside yourself. A test suite is a small, mechanical version of that: a set of claims you wrote when you were being honest, run by a machine that has no stake in the answer, reporting to you whether the thing you built matches the thing you promised. It cannot be flattered. That is its entire value.

There is a second reason this matters beyond craft, and it is the one that separates an engineer from a technician. The faults you do not look for do not disappear; they get transferred. They move to your user, who loses the food they thought PantryPilot was tracking. They move to your successor, who inherits the off-by-one at two in the morning. They move to your employer, who ships it. To decline to look is not neutral — it is a decision that someone else will pay. The Week-4 question, who is my neighbor when I write software, comes back with sharper teeth here: your neighbor is the person who meets your untested edge case.

So the discipline of this week has a moral shape underneath the technical one. Writing exit criteria on Monday is a way of binding your future self, who will be tired and motivated, to a standard set by your present self, who is not. Logging a defect before you fix it is a refusal to quietly erase evidence. Reporting verified-requirement coverage instead of line coverage is choosing the number that can hurt you over the number that flatters you. Publishing a Known Issues list is confession, in the plain old sense — saying the true thing about yourself out loud before someone else has to.

And notice how the psalm ends. Not “search me and condemn me” but “lead me in the way everlasting.” The searching is not the point; the searching is for the leading. The defect log is not a monument to your failures. It is the map of what to fix next. An honest builder looks for his own faults because he intends to do something about them — and because a fault he refuses to see is one he has quietly decided to keep.

That is why your log is graded. Not because we enjoy counting your bugs. Because a builder who can face his own work honestly is the only kind anyone should hire.

11.14 — Common Pitfalls

Pitfall: Testing the code instead of the requirement. Example: Reading expiring_soon() and writing a test that asserts exactly what the function currently returns — including the off-by-one. Fix: Derive every test case from the acceptance criterion text before you open the implementation file. If you have already read the code, have someone else — or the assistant, prompted with the requirement only — write the case.

Pitfall: Reporting line coverage as if it were verification. Example: “We achieved 87% test coverage” in the milestone, while FR-16 has no test case at all. Fix: Report verified-requirement coverage from your traceability matrix first. Run code/coverage_report.py. Line coverage is a secondary diagnostic for finding untested code, not a claim that the software works.

Pitfall: Collapsing severity and priority into one field. Example: Every defect marked “High/High,” so the fix order is whatever you happen to open first. Fix: Two columns, always, each with a one-line reason. Severity describes damage to the user; priority describes your calendar. Practice the split in the widget’s triage panel until high-severity/low-priority feels natural.

Pitfall: Fixing the bug before writing the test, and logging it afterward from memory. Example: A one-line fix committed with the message fix expiry bug, no test, defect log entry written on Friday. Fix: Red first. Commit the failing test on its own, then the fix, then verify the test actually fails against the pre-fix commit. Name the regression test in the defect log — the milestone rubric checks for it. Log every defect when you find it, including the ones you fix in thirty seconds; a log of three retrospective entries is evidence of no testing at all.

Pitfall: Tests that depend on the current date, the network, or leftover database state. Example: The suite is green all week, then fails on the first of the month and again in the presentation room’s Wi-Fi. Fix: Inject the clock. Seed randomness. Stub the network by default. Recreate the database from the checked-in seed data at the start of every run. If a test cannot be run twice in a row with the same result, it is not a test yet.

Pitfall: Deferring the non-deterministic feature because it is “hard to test.” Example: FR-14 (recipe suggestion) has zero tests, and in Week 16 it returns malformed JSON on stage. Fix: You cannot assert the model’s words, but you can assert structure, properties, a golden set with a committed threshold, and — most importantly — every failure path. Those are deterministic tests against a stub. Write them this week.

11.15 — Where Your Hours Went This Week

Roughly fifteen hours, spent about like this. If yours looks very different, that is information — write it in the hours log with a sentence about why.

The number that surprises people is the 2.0 on the first acceptance pass. Running your own software deliberately, against a written script, trying to break it, is slow — and it is where most of your defects will come from this week. Do not let it get squeezed.

11.16 — Reps

The reps for this week are in the exercises, and this week they are not warmups. They assemble the milestone piece by piece. A preview:

• Rep 1 — write your scope and out-of-scope paragraphs, then your six exit criteria as checkable numbers.
• Rep 3 — take one requirement with a number in it and produce the full partition/boundary table.
• Rep 4 — build your traceability matrix and run code/coverage_report.py against it. Report the real number.
• Rep 7 — rewrite a bad defect report into a good one; then rewrite one of your own.
• Rep 10 — have an assistant generate ten test cases for one requirement, then count how many asserted the wrong thing.

Take the on-page Check Your Reps quiz when you finish the chapter — it is the ungraded rehearsal for Week 11 Quiz in Canvas, and the early-warning system for whether this week actually landed.

11.17 — This Week’s Milestone

Milestone 11 — Milestone 11: Test Plan, Test Suite & Defect Log. You will ship a test plan with real exit criteria, a traceability matrix from requirements to test cases, an automated suite that runs from one command in CI, a test-data set, and a defect log with at least eight honestly-classified entries and a regression test named for every fix.

Where it lands: docs/test-plan.md, docs/traceability.csv, docs/test-results.md, docs/defect-log.md, and tests/ in your repository. It feeds the test suite + results and defect log lines of the Week-16 rubric — see Appendix C for exactly how many points that is, and Appendix B for templates and worked good/bad examples. Milestone 11 is graded twice, and you know by now what that means: the milestone points are earned this week or not at all, and the same artifacts come around again in the Week-16 rubric, to be scrambled for in a week when there is no time left to spare.

11.18 — Coach’s Final Word

Something changes in a student the week they find their first real bug in their own code on purpose.

Up to that moment, testing feels like a tax — a thing you do after the fun part, to satisfy a rubric. Then you write the boundary case for a date window, watch it go red, and realize the software you were about to demo has been wrong since Week 9 in a way no amount of clicking around would have shown you. That is the moment testing stops being paperwork. It is not the work after building. It is how you find out whether you built the thing you said you would.

You spent Weeks 3 and 4 making promises in writing. This week you go back and check every one of them, on purpose, hunting for the ones you did not keep. That is uncomfortable, and it is supposed to be. The alternative is to find out in Week 16, in front of a room, with the projector on.

Search your own work. Write down what you find, all of it, in your own name. Fix what matters, decide about the rest in the open, and ship the list. That is what the honest builder does — and it is, not incidentally, the whole reason anyone will trust the next thing you make.

See you on Monday.

Up next: the exercises builds the suite rep by rep · Milestone 11 is Milestone 11 · then Chapter 12 — integration, the hard parts you deferred, and change control. Reference: Appendix B (test plan, defect report, and defect log templates with worked examples), Appendix C (the grading contract), Appendix E (glossary: regression, traceability, definition of done). Previous: Chapter 10.