Project 6 — The Catechism Data Pipeline

“Take heed therefore unto yourselves, and to all the flock… feed the church of God, which he hath purchased with his own blood.” — Acts 20:28

Chapter: 6 — Files, Data, and Persistence Due: End of Week 6 Submit: A link to your code — an OnlineGDB project URL or a public GitHub repo URL — with Pipeline.java, supporting source files, and a README.txt. See Coding 1’s online-coding workflow appendix for the full workflow — Coding 1 and Coding 2 share the same OnlineGDB submission model. Allowed tools: the Java compiler, JUnit, a non-AI editor, the textbook. Not allowed (Phase 1 — AI is OFF): AI assistants of any kind. The careful work of data stewardship has to come from your own hands here, or it won’t translate to careful review of AI-generated data code in Phase 2.

The Setup

A small ministry has hand-transcribed Luther’s Small Catechism — the questions for the Six Chief Parts — into a CSV file. They want to publish the catechism on their website and in their study app. The website wants JSON. The app wants a queryable interface (“give me the answer to question 17”). And the ministry, having been bitten once by a corrupted database backup, is adamant: no write to the catechism file should ever risk corrupting the previous good copy.

You are the engineer they hired. Build the pipeline.

The CSV they handed you is real — a real transcription of Luther’s Small Catechism, one row per question/answer pair, in the format described below. Some of the rows are clean. Some have stray whitespace, smart quotes that should be normalized to ASCII, or missing answers. Your pipeline must:

• Read the CSV faithfully (including fields with embedded commas).
• Validate every row.
• Normalize whitespace and quotation marks.
• Write the cleaned data as well-formed JSON.
• Never corrupt the file even on a mid-write crash.
• Report every problem it finds in the input, with a useful line-number-and-cause message.

This is real data engineering at small scale. The scale is intentional — the discipline of careful handling is identical at any size.

Setup

A starter zip is provided on the course portal. Inside:

• data/catechism.csv — the real input, 30+ rows. Don’t edit by hand. (download catechism.csv)
• data/catechism.bad.csv — a deliberately-corrupted version with 5 malformed rows. Used by Medium and Hard tier. (download catechism.bad.csv)
• expected/catechism.expected.json — what your output JSON should look like for the clean input. (download catechism.expected.json)
• README.template.txt — fill in your tier, approach, and AI honesty line, then save your filled copy as README.txt for your submission. (download README.template.txt)

Reference implementation — The chapter’s §6.8 end-to-end example is the seed shape for this project. You can download it and its supporting modules to study the joints (then build your own, decomposed further): (download CatechismPipeline.java), (download SimpleCsv.java), (download JsonOut.java), (download SafeWrite.java).

You will write:

• Pipeline.java — the main program.
• Supporting files of your choosing — SimpleCsv.java, JsonOut.java, SafeWrite.java, Entry.java, etc. Project 6 rewards good decomposition.
• A test class for any tier that requires tests.

CSV format

number,question,answer
1,"What is the chief end of man?","Man's chief end is to glorify God, and to enjoy him forever."
2,"What rule has God given to direct us how we may glorify and enjoy him?","The Word of God, which is contained in the Scriptures of the Old and New Testaments, is the only rule to direct us."
...

Field rules:

• number is a positive integer. Unique within the file.
• question is a non-empty string that ends with ?.
• answer is a non-empty string.
• Fields may be wrapped in double quotes. Embedded commas in quoted fields are not separators. Embedded double-quotes are escaped as "".
• The first row is a header (number,question,answer). Skip it.
• Lines may have trailing whitespace; trim it.

Learning Targets

By completing this project, you will demonstrate that you can:

• Read and write files using java.nio.file cleanly.
• Parse CSV correctly, including quoted fields with embedded commas.
• Validate input at the door using record-style constructor validation.
• Hand-roll a small but correct JSON writer.
• Use the atomic-write (temp-file-plus-rename) pattern for persistence.
• Separate parsing, validation, and persistence into distinct, testable modules.
• Report errors clearly enough that the person who handed you the data can fix the source.

Normal Tier

Goal: Read catechism.csv, validate, normalize, and write catechism.json atomically. The output must match expected/catechism.expected.json byte-for-byte (modulo final newline tolerance).

Required features

1. CSV parser that respects quoted fields and embedded commas. Either your own (the Chapter 6 careful parser) or you can use a library if you can get one on the classpath — your call, but document the choice.
2. Validation of each Entry:
   • number ≥ 1.
   • question non-blank, ends with ?.
   • answer non-blank.
   • No duplicate number across the file.
3. Normalization applied before write:
   • Trim leading/trailing whitespace from question and answer.
   • Collapse runs of internal whitespace to single spaces.
   • Replace smart quotes (" " ' ') with ASCII (" " ' ').
4. JSON output written to catechism.json:
   • One JSON array.
   • Each element is an object with keys number, question, answer, in that order.
   • The exact shape — spacing, key order, where the commas and newlines go — is defined by the provided expected/catechism.expected.json (it is one object per line, two spaces of indent, a space after each : and inside each { }). Match it; don’t invent a different convention. This is what “byte-for-byte” in the goal above means.
   • All required JSON escaping handled (quotes, backslashes, newlines).
5. Atomic write — your write must use the temp-file-plus-rename pattern. The grader will inspect your code to confirm this.
6. Error handling — every exception that could plausibly happen (file not found, malformed line, invalid entry) is caught, reported with file name + line number + cause, and the program exits with a non-zero status code. No bare Exception stack traces in the output.
7. Decomposition — at least three distinct classes / records, each with a single responsibility (parser, validator/model, writer/persister). Not one giant Pipeline.java file.

Example output (catechism.json)

This is the exact shape your output must match — one object per line, two-space indent, a space after each : and inside each { }, a trailing comma on every object but the last. The full, authoritative copy lives in expected/catechism.expected.json (and you can download catechism.expected.json) — when in doubt, the file wins, not this snippet. Here are its first two rows so you can see the format at a glance:

[
  { "number": 1, "question": "What is the chief end of man?", "answer": "Man's chief end is to glorify God, and to enjoy him forever." },
  { "number": 2, "question": "What rule has God given to direct us how we may glorify and enjoy him?", "answer": "The Word of God, which is contained in the Scriptures of the Old and New Testaments, is the only rule to direct us." }
]

The chapter’s §6.4 JsonOut.writeAll writer already produces exactly this shape; if you start from it, your catechism.json will diff clean against the expected file. (Don’t reformat it into multi-line objects — that looks prettier but no longer matches byte-for-byte.)

Normal-tier rubric (out of 100)

Medium Tier (+up to 25% extra credit)

M1. Query interface

Add a Query.java companion program (or a --query flag on the main Pipeline). It loads the JSON you just produced and exposes a small interactive interface:

> 1
1. What is the chief end of man?
   To glorify God, and to enjoy him forever.

> 17
No entry with number 17.

> q
goodbye.

Read lines from stdin until the user types q or EOF. For each numeric input, look up the entry and print it. For unknown numbers, print a clean “not found” message. For non-numeric input, print “please type a number or q.”

M2. JUnit tests

Write JUnit tests for the pipeline. At minimum:

• One test that confirms a well-formed CSV produces the expected JSON.
• One test per validation rule (proves bad input is rejected with the right message).
• One test for the round-trip: write a known data set as JSON, then your CSV writer (if you have one) or a hand-rolled equivalent, parse again, confirm equality.
• One test for the atomic write: write twice to the same file, confirm no .tmp file is left behind.

At least 8 tests total.

M3. Multi-error reporting

Instead of crashing on the first bad row in catechism.bad.csv, collect all the errors, then report them at the end:

Loaded 28 of 33 rows. 5 errors:
  Line 7: question must end with '?', got "What is grace"
  Line 12: duplicate number 8
  Line 15: answer is blank
  Line 22: number must be >= 1, got -3
  Line 30: expected 3 fields, got 2

Then exit non-zero. Don’t write the JSON if any errors were found.

Hard Tier (+up to 25% additional extra credit)

H1. Bidirectional transformer

Add a JSON parser. (You may write it by hand — it’s a real exercise — or use Jackson or another library, with the library setup documented in your README.) Once you have one, build the inverse pipeline: read JSON, write CSV.

Then prove round-trip idempotency:

• csv → json → csv produces the same bytes as the original CSV (modulo documented normalizations).
• json → csv → json produces the same bytes as the original JSON.

Submit both round-trip tests, with the documented normalizations listed in your README.

The round-trip is the canonical test for any data transformation. It is the same logic the church uses to verify a manuscript copy: copy it again from the copy, compare the second copy to the first, look for drift. Drift detected at this level is a tiny number of pixels (or bytes) at a time, and over generations it adds up. The round-trip test catches drift.

H2. CLI flags

Make the pipeline accept command-line arguments:

java Pipeline --in catechism.csv --out catechism.json
java Pipeline --in catechism.json --out catechism.csv         # H1
java Pipeline --in catechism.bad.csv --report-errors          # M3
java Pipeline --in catechism.csv --validate-only              # no write

Format auto-detected from the extension. Document the CLI in your README.

H3. Performance and correctness on a larger corpus

Build a larger corpus of your own — no file is provided for this tier; assembling the bigger dataset is part of the rep. Extend the provided catechism.csv up to a few hundred rows: duplicate-and-vary the existing entries, add the Lord’s Prayer and the Apostles’ Creed explanations, or script the expansion. Run your pipeline on it. Time the run. Report any rows your pipeline rejects, and either fix the pipeline (if the rejection is a false positive) or document why each rejection is correct.

This is the rep where you discover the edge cases your test data didn’t have. Real data always exceeds the test set.

Submission

Submit one URL via the course portal:

• OnlineGDB project link (recommended for Coding 2).
• GitHub repo link (optional).

What the linked project must contain

1. Pipeline.java and any supporting source files you wrote (SimpleCsv.java, JsonOut.java, SafeWrite.java, etc.).
2. data/catechism.csv (the input — same one you started from, unchanged).
3. catechism.json — the output of one successful Normal-tier run. Include this in the repo so the grader can see what your code produced.
4. tests/ — for Medium and Hard, the JUnit test files. Confirm they pass before submitting.
5. README.txt — your reflection:

# Project 6 — Catechism Data Pipeline

**Tier targeted:**  Normal / Medium / Hard
**Features done:**  (list)
**Module breakdown:**  (one line per class — what it does)
**Atomic write:**  yes / no — and where in the code
**Library choice:**  (none / Jackson / opencsv / commons-csv — if any, why)
**What I learned:**  (one paragraph)
**What I'd change:**  (one sentence)
**AI usage:**  NONE — Phase 1.  Signed: <your name>

6. Reflection comment block at the top of Pipeline.java — same fields as the README, condensed.
7. The program left runnable — when the grader runs java Pipeline, your pipeline executes against the provided input and produces the expected output.

Hints (Read Before You Begin)

• Build the pieces, then assemble. Get the CSV parser working alone first (Rep 4 from the chapter). Then the JSON writer alone. Then the atomic write. Then wire them together in Pipeline. Each piece is small. Each piece is independently testable.

• Trim everywhere. Whitespace bugs are the most common single category of data-pipeline mistake. Trim every field after parsing. Trim trailing newlines after reading. Be aggressive.

• Don’t write your own JSON parser unless you’re going Hard tier with intention. Writing JSON by hand is fine; parsing JSON by hand is a much bigger project than this chapter assigned. Pick your battles.

• Test atomic write by inspection. Run your pipeline. List the directory contents. Confirm there is no .tmp file lingering. Then deliberately throw an exception in the middle of your writeAtomic (between writing the temp file and the move). Confirm the original file is still intact. Then remove the deliberate failure.

• Error messages are part of the deliverable. “An error occurred” is not an error message. “Line 17: question must end with ’?’, got ‘What is grace’” is. Imagine the person reading your error message has to fix the source CSV by hand, with no access to your code.

• Records save you typing. record Entry(int number, String question, String answer) {} plus a compact constructor for validation is the entire model layer. Don’t write classes with manual constructors and getters and equals/hashCode when a record will do.

What Mastery Looks Like (Beyond the Rubric)

A great Project 6 reads like a small but real piece of professional software. The classes are named for what they do. The methods are short. The error messages would help a human fix the source. The atomic write is in there even though no one will ever crash it on the grader’s machine — because it’s the right shape and you ship right shapes.

A great Project 6 has a Pipeline.java main that is short — maybe 20 lines. Most of it is delegation to the small classes you wrote. The work happens in the small classes. main is the conductor.

A great Project 6 fails informatively. You handed the broken CSV to the pipeline; the pipeline told you, in plain English, every problem it found. The person who wrote the broken CSV can now go fix it. That’s data stewardship serving the data’s owner.

Coach’s Note — When I have students do this project well, they often ship more code than they expected — 200+ lines across 5 files. That feels like a lot for “just reading a CSV.” It is also exactly how much code real data pipelines require. The discipline is in every one of those lines being short, named, and tested. Long code is not bad code. Long unstructured code is.

When You’re Done

1. Run the pipeline against the clean CSV. Diff your output against expected/catechism.expected.json.
2. Run the pipeline against the bad CSV (Medium tier). Confirm the error report is useful.
3. Delete catechism.json and re-run. Confirm regeneration.
4. Read your own Pipeline.java main slowly. Could a stranger read it and understand the whole pipeline shape from main alone? If not, refactor.
5. Submit.
6. Read Chapter 7. Recursion next — including how to walk a directory tree of catechism files recursively, which the project here only handled at single-file scale.

A theological footnote. The catechism in your data file was preserved across nearly five hundred years by people who cared about every word. The discipline of careful textual transmission in the church and the discipline of careful data engineering in software are the same discipline, applied to different artifacts. When you wrote the atomic-write pattern, you joined that tradition — a small joining, but a real one. Steward the data well.

See you next week.