Files, Data, and Persistence
How is data the steward of truth?
Chapter 6 — Files, Data, and Persistence
“Heaven and earth will pass away, but my words will not pass away.” — Matthew 24:35
“All the great religions of the world are also great traditions of textual care.” — every honest historian of writing
Why This Matters
Up to this point, your programs have lived in RAM. You start them, they compute something, they print, they exit, and everything they ever knew dies with the process. The next program you run begins from zero.
That is not how real software works.
Real software remembers. The roster you built in Project 13 of Coding 1 forgot its members the moment you closed the terminal. The next Account class you ship in your career will not. Real software persists. It reads files at startup; it writes files at shutdown; it parses other people’s data; it produces data other people will parse. The discipline of doing this carefully — knowing exactly what’s on disk, recovering gracefully from a partial write, validating every byte that comes in — is the discipline this chapter is about.
The Christian framing is on the nose. The church has cared deeply about the transmission of texts for two thousand years — the manuscript tradition that brought us the New Testament, the catechetical tradition that brought us the Small Catechism, the careful copying of liturgies and confessions and hymnals across centuries. How is data the steward of truth? It is the steward when the people who handle it are careful — when the copy is checked against the original, when the partial write is not allowed to corrupt the whole, when the parser refuses input it cannot trust. The discipline of careful data handling in code is exactly the discipline of careful textual transmission in the church, scaled and automated.
Project 6 — The Catechism Data Pipeline — has you read a CSV of catechism questions, normalize it, validate it, and write it back out as JSON. The data is real. The discipline is the deliverable.
6.1 — java.nio.file: the Modern Java File API
For decades Java had java.io.File, with its checked-exception-everywhere style and clunky API. Since Java 7, the modern way is java.nio.file, and as of Java 11, two methods on java.nio.file.Files cover 90% of what you need: readString and writeString.
import java.nio.file.Files;
import java.nio.file.Path;
public class FileBasics {
public static void main(String[] args) throws Exception {
Path p = Path.of("hello.txt");
Files.writeString(p, "Grace and peace.\n");
String contents = Files.readString(p);
System.out.println(contents);
}
}A few things to notice immediately.
Path.of("hello.txt") — a Path is Java’s way of representing a filesystem path. Path.of is the modern constructor (older code used Paths.get(...); they do the same thing). Relative paths resolve against the program’s working directory; absolute paths start with / on Unix or C:\ on Windows.
Files.writeString(p, "...") — writes the entire string to the file. If the file doesn’t exist, it’s created. If it does, it’s overwritten — no warning. To append instead, pass StandardOpenOption.APPEND as a third argument.
Files.readString(p) — reads the entire file into memory as a String. This is fine for files up to a few megabytes. For huge files (logs, datasets), use Files.lines(p) to stream line-by-line instead.
throws Exception — almost every Files method throws IOException. We declared throws Exception on main for brevity. In real code (Project 6 specifically), you will handle these exceptions with try/catch, applying the lessons of Chapter 3.
Reading lines as a list
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
List<String> lines = Files.readAllLines(Path.of("catechism.txt"));
for (String line : lines) {
System.out.println(line);
}readAllLines returns a List<String> — one entry per line, with the trailing newline removed. The list type is mutable (an ArrayList under the hood, but coded against the List interface, which Chapter 8 will treat properly).
Writing lines from a list
List<String> output = List.of(
"What is the chief end of man?",
"Man's chief end is to glorify God and to enjoy him forever."
);
Files.write(Path.of("answers.txt"), output);Files.write(Path, Iterable<? extends CharSequence>) writes one line per element, with the platform line separator between them.
Coach’s Note — For Coding 2, default to
Files.readString,Files.writeString,Files.readAllLines, andFiles.write. They cover almost every project task in this book. Streaming withFiles.linesis a Coding 3 topic when memory pressure becomes real.
6.2 — Path Hygiene
A Path is more than a string. It knows how to navigate, resolve, and compare itself against other paths.
Path data = Path.of("data"); // relative: data/
Path file = data.resolve("catechism.csv"); // data/catechism.csv
Path abs = file.toAbsolutePath(); // /Users/maya/proj/data/catechism.csv
Path back = file.getParent(); // data/
System.out.println(Files.exists(file)); // true if it's there
System.out.println(Files.isReadable(file)); // true if you can read
System.out.println(Files.size(file)); // size in bytesThe key methods:
resolve(child)— join a parent path with a child segment. Use this instead of+ "/litman-books/" +string concatenation, which breaks on Windows.getParent()— go up one level.toAbsolutePath()— resolve a relative path against the current working directory.getFileName()— just the last segment (catechism.csv).Files.exists(path)— does it exist?Files.isDirectory(path)/Files.isRegularFile(path)— kind?
Path hygiene matters because your code will run on machines you don’t own. Hardcoded "data\\catechism.csv" works on your laptop and breaks on the grader’s Linux box. Hardcoded "/litman-books/Users/maya/proj/data/catechism.csv" breaks for anyone whose name isn’t Maya.
Coach’s Note — Build paths up with
.resolve(...)from a known root (e.g., the program’s working directory or a path passed as an argument). Never hardcode absolute paths. Future-you, on a different machine, will thank you.
6.3 — CSV: Comma-Separated Values, the Honest Version
CSV looks dead simple — one record per line, fields separated by commas. So it is almost dead simple, until you meet a field that contains a comma. Or a newline. Or a literal quote character. Then the format reveals its small-but-real subtlety.
Here is a real CSV from this week’s project:
number,question,answer
1,"What is the chief end of man?","Man's chief end is to glorify God."
2,"How many gods are there?","One — the LORD, our God, is one LORD."
3,"What is the Word of God?","The Word of God is the Bible, the Old and New Testaments."Note: the values are wrapped in double-quotes because some of them contain commas (and apostrophes don’t matter, but commas would, and the writer wanted consistency).
The simple parser (when you control the data)
For tightly-controlled data — data your own program wrote, or data you cleaned by hand — a one-line split is fine:
String[] fields = line.split(",");This breaks the moment a field contains a comma. Don’t ship this for data you didn’t author.
The careful parser
For real CSV — data from the wild, from another program, from an export, from a user — write the rules explicitly. A minimal but correct CSV parser respects three things:
- Fields can be wrapped in double quotes.
- A quoted field can contain commas (which are then not field separators).
- A quoted field can contain a literal double quote, escaped as two double quotes (
"").
Here is a small hand-rolled parser, sufficient for the well-formed data Project 6 ships:
import java.util.ArrayList;
import java.util.List;
public class SimpleCsv {
public static List<String> parseLine(String line) {
List<String> out = new ArrayList<>();
StringBuilder field = new StringBuilder();
boolean inQuotes = false;
for (int i = 0; i < line.length(); i++) {
char c = line.charAt(i);
if (inQuotes) {
if (c == '"') {
// peek ahead for escaped quote
if (i + 1 < line.length() && line.charAt(i + 1) == '"') {
field.append('"');
i++;
} else {
inQuotes = false;
}
} else {
field.append(c);
}
} else {
if (c == ',') {
out.add(field.toString());
field.setLength(0);
} else if (c == '"') {
inQuotes = true;
} else {
field.append(c);
}
}
}
out.add(field.toString());
return out;
}
}That’s about 25 lines, handles the three rules above, and is yours. Read every line. This is the kind of small, careful, well-tested utility code that a working programmer writes constantly.
A few caveats it does not handle: embedded newlines inside quoted fields (which true RFC-4180 CSV permits), Unicode BOM at the start of a file, and inconsistent line endings. For Project 6 you don’t need these. For production, you’d reach for a library — opencsv or commons-csv — and that’s an honest choice.
Writing CSV — the inverse
Writing is easier than reading because you control the format. The rule: if a field contains a comma, a double-quote, or a newline, wrap it in quotes and double any embedded quotes.
public static String escapeField(String s) {
boolean needsQuotes = s.contains(",") || s.contains("\"") || s.contains("\n");
if (!needsQuotes) return s;
return "\"" + s.replace("\"", "\"\"") + "\"";
}
public static String writeLine(List<String> fields) {
StringBuilder sb = new StringBuilder();
for (int i = 0; i < fields.size(); i++) {
if (i > 0) sb.append(',');
sb.append(escapeField(fields.get(i)));
}
return sb.toString();
}A round-trip test — parse(write(parse(write(...)))) — should produce the same data both times. Project 6 Hard tier asks you to prove that.
6.4 — JSON: When You Need Structure
CSV is for flat tables. JSON (JavaScript Object Notation, but used by everyone now) is for structured data — objects with named fields, lists nested inside objects, objects nested inside lists.
A CSV row:
1,"What is the chief end of man?","To glorify God."The same record as JSON:
{
"number": 1,
"question": "What is the chief end of man?",
"answer": "To glorify God."
}And a whole catechism as an array of those:
[
{ "number": 1, "question": "...", "answer": "..." },
{ "number": 2, "question": "...", "answer": "..." }
]Two routes: a library or by hand
For production Java, the standard library is Jackson (com.fasterxml.jackson). It’s three lines to read a JSON file into Java objects:
import com.fasterxml.jackson.databind.ObjectMapper;
ObjectMapper m = new ObjectMapper();
List<Catechism> entries = m.readValue(
Files.readString(Path.of("catechism.json")),
new com.fasterxml.jackson.core.type.TypeReference<>() {}
);Jackson is industry standard. If your project setup includes it (Coding 3 will properly), use it. For Coding 2 in OnlineGDB, getting Jackson on the classpath is annoying, so we’ll teach a tiny hand-rolled JSON writer (we won’t write a parser by hand for Coding 2 — that’s a multi-week project on its own).
A tiny hand-rolled JSON writer
For the structured, well-controlled data we produce, hand-writing JSON output is fine and totally legitimate. The rules:
- Strings: wrap in
"...", escape backslashes and quotes (\\,\"), and escape newlines (\n). - Numbers: just print them.
- Booleans:
trueorfalse. - Arrays:
[ a, b, c ]. - Objects:
{ "key": value, ... }.
Here is a minimal writer that handles the catechism case:
import java.util.List;
public class JsonOut {
public static String escape(String s) {
StringBuilder sb = new StringBuilder("\"");
for (char c : s.toCharArray()) {
switch (c) {
case '\\' -> sb.append("\\\\");
case '"' -> sb.append("\\\"");
case '\n' -> sb.append("\\n");
case '\r' -> sb.append("\\r");
case '\t' -> sb.append("\\t");
default -> sb.append(c);
}
}
sb.append("\"");
return sb.toString();
}
public record Entry(int number, String question, String answer) {}
public static String writeAll(List<Entry> entries) {
StringBuilder sb = new StringBuilder("[\n");
for (int i = 0; i < entries.size(); i++) {
Entry e = entries.get(i);
sb.append(" {");
sb.append(" \"number\": ").append(e.number()).append(",");
sb.append(" \"question\": ").append(escape(e.question())).append(",");
sb.append(" \"answer\": ").append(escape(e.answer()));
sb.append(" }");
if (i < entries.size() - 1) sb.append(",");
sb.append("\n");
}
sb.append("]\n");
return sb.toString();
}
}About 30 lines. Handles the realistic catechism shape. If you want to read this JSON back in Java without Jackson, you’d need to write a parser — which is a worthy exercise but bigger than this project. For Coding 2, write JSON from Java, read JSON elsewhere (browser, Python, another Java program with Jackson). That’s a perfectly common production split.
The record keyword we used here is Java 14+‘s way of declaring a simple value-bearing class. record Entry(int number, String question, String answer) {} gives you a class with three immutable fields, a constructor, accessors (number(), question(), answer()), and sensible equals, hashCode, toString. It’s exactly the shape we need for a CSV row or a JSON object.
Coach’s Note — Records are the Java answer to “I just need to bundle three things.” Use them aggressively for parsed data. You’ll have less code, fewer bugs, and cleaner method signatures.
6.5 — Validation: Trust Nothing the File Says
Data from a file is data from outside your program. Treat every byte as suspect until you’ve checked it.
What does validation look like for a catechism CSV?
public record Entry(int number, String question, String answer) {
public Entry {
if (number < 1) {
throw new IllegalArgumentException("number must be >= 1, was " + number);
}
if (question == null || question.isBlank()) {
throw new IllegalArgumentException("question must be non-blank for entry " + number);
}
if (answer == null || answer.isBlank()) {
throw new IllegalArgumentException("answer must be non-blank for entry " + number);
}
if (!question.endsWith("?")) {
throw new IllegalArgumentException("question for entry " + number + " must end with '?'");
}
}
}The unusual block — public Entry { ... } — is the compact constructor for a record. It runs the validation as part of constructing the record, so you cannot construct an invalid one. The check is centralized; every code path that produces an Entry is covered.
Validation is the same discipline as scriptural copyediting: every transmission of the text gets a check against the rules of the text. The bad transmission is caught before it propagates. The careful copyist catches the dropped word in line 3 before it ends up in seventeen later copies.
Where validation belongs
Three reasonable layers:
- In the constructor (above). The strongest. The object cannot exist in an invalid state.
- In the parser. Validate as you parse — reject malformed lines with a useful error.
- In a separate
validate()pass. Useful when you want to accept everything, report all errors, and then decide what to do.
Project 6 Normal tier asks you to do layer 1 (constructor validation) and layer 2 (parser validation). Medium adds layer 3 — collect every error in the file before reporting, so the user can fix many things in one pass rather than fixing-one-and-retrying.
6.6 — Atomic Writes: Don’t Corrupt the Data You Came to Steward
Here is a scenario. Your program reads catechism.json, processes it, and writes the cleaned version back to the same file. Halfway through writing, the process crashes — power loss, kill signal, your code throws an exception. Now catechism.json is half the old data, half nothing, or worse, garbled.
You have just destroyed the file you came to steward.
The fix is the atomic write pattern, sometimes called “write-then-rename”:
- Write the new data to a temporary file in the same directory:
catechism.json.tmp. - Once the write completes successfully, atomically rename the temp file over the original.
The rename is atomic at the filesystem level — there is no moment when the file exists in a half-written state. Either the original is still there, or the new one is there. Never an in-between.
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
public class SafeWrite {
public static void writeAtomic(Path target, String contents) throws Exception {
Path tmp = target.resolveSibling(target.getFileName() + ".tmp");
Files.writeString(tmp, contents);
Files.move(tmp, target, StandardCopyOption.REPLACE_EXISTING,
StandardCopyOption.ATOMIC_MOVE);
}
}Three lines do the work:
resolveSibling(...)gets a path next to the target, in the same directory (atomic rename only works within a single filesystem).Files.writeString(tmp, contents)writes to the temp file. If this throws, the original is untouched.Files.move(tmp, target, REPLACE_EXISTING, ATOMIC_MOVE)is the atomic rename.
This pattern is on every senior engineer’s reflex. Use it any time the file you are writing to is important. For your scratch output.txt file, fine, write directly. For data that represents weeks of work — always atomic.
Coach’s Note — I have seen this bug ship in production at multiple companies. Someone writes a config file or a database snapshot directly, and a crash mid-write corrupts the last good copy. Three lines of
Files.move(... ATOMIC_MOVE)would have prevented every one of those incidents. Learn the move now.
6.7 — Try-with-Resources: Close What You Open
When you open a file with anything other than the one-shot Files.readString family — for example, a BufferedReader for streaming, or a Scanner reading from a file — you must close it when done. Otherwise the OS file handle leaks, and on long-running programs that adds up.
The clean way is try-with-resources:
import java.io.BufferedReader;
import java.nio.file.Files;
import java.nio.file.Path;
try (BufferedReader r = Files.newBufferedReader(Path.of("big.txt"))) {
String line;
while ((line = r.readLine()) != null) {
// process line
}
} // r is automatically closed here, even if the body throwsThe resource declared in the try (...) header is closed when the try block exits, whether normally or by exception. No finally block needed. This is Java’s answer to the C++ RAII (Resource Acquisition Is Initialization) idea you met at the end of Coding 1: scope the resource’s life to the block.
Any class that implements AutoCloseable works here — readers, writers, scanners, database connections, network sockets, custom resources you write yourself.
6.8 — A Small End-to-End Example
Here is the smallest realistic version of what Project 6 asks you to build. Read a CSV, validate each row, write JSON.
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
public class CatechismPipeline {
public record Entry(int number, String question, String answer) {
public Entry {
if (number < 1) throw new IllegalArgumentException("number");
if (question == null
|| !question.endsWith("?")) throw new IllegalArgumentException("question");
if (answer == null
|| answer.isBlank()) throw new IllegalArgumentException("answer");
}
}
public static List<Entry> readCsv(Path csv) throws Exception {
List<String> lines = Files.readAllLines(csv);
List<Entry> out = new ArrayList<>();
for (int i = 1; i < lines.size(); i++) { // skip header row
List<String> fields = SimpleCsv.parseLine(lines.get(i));
if (fields.size() != 3) {
throw new IllegalStateException(
"Line " + (i + 1) + ": expected 3 fields, got " + fields.size());
}
int n = Integer.parseInt(fields.get(0).trim());
out.add(new Entry(n, fields.get(1).trim(), fields.get(2).trim()));
}
return out;
}
public static void writeJson(Path out, List<Entry> entries) throws Exception {
List<JsonOut.Entry> j = new ArrayList<>();
for (Entry e : entries) {
j.add(new JsonOut.Entry(e.number(), e.question(), e.answer()));
}
String contents = JsonOut.writeAll(j);
SafeWrite.writeAtomic(out, contents);
}
public static void main(String[] args) throws Exception {
List<Entry> entries = readCsv(Path.of("catechism.csv"));
System.out.println("Loaded " + entries.size() + " entries.");
writeJson(Path.of("catechism.json"), entries);
System.out.println("Wrote " + entries.size() + " entries to catechism.json.");
}
}(download CatechismPipeline.java — depends on SimpleCsv.java, JsonOut.java, and SafeWrite.java above)
That’s the whole pipeline. Less than 50 lines, including validation, CSV parsing, JSON writing, and atomic write. Notice the separation — each tiny module does one thing: SimpleCsv parses, Entry validates, JsonOut formats, SafeWrite persists. Clean joints. Easy to test. Easy to replace any one piece.
That separation is itself the lesson. The architecture chapter (Chapter 12, deep in Phase 2) will name this pattern out loud. For now, just notice that the careful version is also the version that’s easier to test, easier to read, and easier to change.
6.9 — Common Bugs (Week 6 Edition)
Bug: NoSuchFileException when the file is “right there.”
What it means: The path is relative and you’re not running from where you think you are. The “working directory” of the JVM may not be the directory the source file is in.
Fix: Print Path.of(".").toAbsolutePath() to see what directory you’re actually in. Pass paths in as arguments, or use an absolute path you constructed deliberately.
Bug: Output file is encoded weirdly; characters with accents are garbled.
What it means: Default character encoding mismatch. Java 17 defaults to UTF-8, but older versions or some platforms default to the OS encoding.
Fix: Be explicit: Files.writeString(p, s, java.nio.charset.StandardCharsets.UTF_8). Same for read.
Bug: CSV parsing breaks on a row that contains a comma inside a field.
What it means: You used .split(",") on a real CSV. The split takes everything.
Fix: Use the careful parser from §6.3, or a library.
Bug: Wrote the file successfully but a crash mid-process corrupted the previous version. What it means: You wrote directly to the target path. There was a window of vulnerability. Fix: The atomic write pattern from §6.6. Always for important data.
Bug: “Stream is closed” or “Cannot read from closed reader.”
What it means: A BufferedReader or Scanner was closed before you finished reading from it. Often because of misplaced try-with-resources scoping.
Fix: Make sure the consumer of the reader is inside the try-with-resources block, not after it.
Bug: Reading numeric fields with Integer.parseInt throws NumberFormatException on perfectly numeric-looking strings.
What it means: Trailing whitespace, a leading BOM character, or an unusual minus sign (en-dash vs ASCII hyphen) is sneaking in.
Fix: .trim() the string before parsing. For BOM, strip it explicitly: if (s.startsWith("")) s = s.substring(1);.
Bug: Files.write(...) truncates an existing file instead of appending.
What it means: Default open options include CREATE and TRUNCATE_EXISTING.
Fix: Pass StandardOpenOption.APPEND if you want to append.
6.10 — Reps
Open the exercises for the full set. Highlights:
Rep 1. Write a string to a file. Read it back. Print the round-trip.
Rep 3. Parse a CSV line that contains a quoted field with a comma. Confirm naive .split(",") is wrong.
Rep 5. Round-trip a record-shaped entry: CSV → object → JSON → object (mentally) → CSV.
Rep 7. Atomic write: simulate a mid-write crash and verify the original file is intact.
Full set in the exercises.
6.11 — This Week’s Project: The Catechism Data Pipeline
You’re ready for Project 6: The Catechism Data Pipeline, in Project 6.
You’ll receive a CSV file of catechism question/answer pairs. Your job: read it carefully, normalize it (whitespace, quotation marks, validation), and write it back as well-formed JSON — atomically, with full error reporting on malformed input.
Three tiers:
- Normal — read CSV, validate, write JSON, with proper exception handling and atomic write.
- Medium — add a query interface: read a question number from stdin, print the answer. Add JUnit tests.
- Hard — build a bidirectional transformer: CSV → JSON and JSON → CSV. Add round-trip tests proving idempotency.
6.12 — Coach’s Final Word for Week 6
Data persistence is where amateur programs become real ones. The discipline you built today — validate at the door, write atomically, separate parsing from validation from persistence — is the discipline of every working system you’ll ever interact with. The reason your bank balance doesn’t randomly corrupt is that thousands of programmers, across decades, did exactly this kind of careful work.
The church’s long tradition of careful textual transmission has a contemporary cousin in the discipline of careful data engineering. Both crafts know that the value of the artifact depends on the care of its handlers. A sloppy scribe and a sloppy data pipeline produce the same outcome: a derived copy that drifts from the original until no one can tell what the original said.
Be the careful copyist. Validate. Atomically write. Test the round-trip.
See you next week. Chapter 7 is recursion — when does the part contain the whole? — and you will use it, among other places, to walk a directory tree and find every catechism file in it.
Up next: Read the exercises and run every rep. Then open Project 6 and build the pipeline. After that, Chapter 7 — Recursion.