Chapter 11 · Reps

Backup, Recovery, and the Ark You Build Before the Flood — Reps

← Back to Chapter 11

Chapter 11 — Reps

Conditioning, not grading. This week you build backups and — the part that counts — you restore them. A backup you have never restored is a rumor. Make every rep produce a verified recovery.

Ground rules:

  1. Type every command yourself. No copy-paste of restore commands especially. The one place a fat finger is unforgivable is the recovery you run during a real outage; build the muscle now, on scratch data.
  2. Run everything for real. Against real files, a real Postgres, a real bucket (or a local MinIO / LocalStack stand-in). Watch each backup land and each restore come back. “It should work” is not a rep.
  3. Predict before you measure. Before every restore, write down your predicted RTO. Then time it. The gap between your guess and the clock is the lesson.
  4. AI is ON (Phase 2) — but every command an agent gives you is suspect until you run it on a scratch target. You will use AI to draft runbooks here on purpose. You will also catch it being confidently wrong. Both are the rep.
  5. Mark destructive commands. Anything that deletes, drops, or syncs-with-delete gets a comment saying so, and a WHERE/scope/--dryrun first. Never point a restore at production until you’ve proven it on scratch.

Most reps use standard tools you already have or can install in five minutes: tar, sha256sum, git, the aws CLI (or awslocal against LocalStack), Postgres with pgvector, and Python 3 with pyyaml. Toolchain setup is in Appendix A.

Reps 1–3: The Backup Trinity, By Hand

Rep 1 — Full, Incremental, Differential

Create a small “estate” and back it up three ways so you feel the trinity, not just read the table.

mkdir -p estate/{prompts,datasets} && echo "v1 prompt" > estate/prompts/system.txt
echo "row1" > estate/datasets/labels.csv

# FULL: everything, every time.
tar -czf full-1.tgz estate/

# Change one file, then take an INCREMENTAL (only what changed since last backup) using a snapshot file.
echo "v2 prompt" >> estate/prompts/system.txt
tar --create --gzip --listed-incremental=snap.snar --file inc-1.tgz estate/

# Change again; DIFFERENTIAL is "everything since the last FULL" — reset the snapshot from the full each time.
echo "row2" >> estate/datasets/labels.csv
cp snap-from-full.snar diff.snar 2>/dev/null || tar -g full.snar -czf /dev/null estate/  # seed once

Write down, in one sentence each: which backup is cheapest to write, which is cheapest to restore, and why the restore cost is the one you should optimize for.

Rep 2 — Restore and Prove It

Delete the estate (yes, on purpose — it’s scratch) and bring it back from the full.

rm -rf estate                       # DESTRUCTIVE — scratch data only
mkdir restored && tar -xzf full-1.tgz -C restored
diff -r restored/estate <(echo)     # or eyeball: ls -R restored/estate

Confirm the files are back. Now restore from the incremental chain instead (full, then each incremental in order) and confirm you get the latest version. Predict first: what happens if you apply the incrementals out of order, or skip one? Try it. Write the failure mode in one sentence — this is why incremental chains are fragile.

Rep 3 — Checksums Catch Silent Corruption

A restore that “succeeds” but returns subtly wrong bytes is worse than one that fails loudly. Build a checksum manifest and use it.

sha256sum estate/datasets/*.csv > manifest.sha256   # capture at backup time
# ...later, after a restore into ./restored ...
( cd restored/estate && sha256sum -c ../../manifest.sha256 )

Now corrupt one restored byte (printf 'x' >> restored/estate/datasets/labels.csv) and re-run the check. Watch it report FAILED. Write one sentence on why a checksum manifest belongs in every dataset backup.

Reps 4–6: 3-2-1, Immutability, and the Air Gap

Rep 4 — Set Object Lock and Feel It Refuse You

Create a versioned, object-locked bucket (use real S3 with a throwaway bucket, or awslocal against LocalStack), upload a “fine-tuned weights” file, then try to delete it.

aws s3api create-bucket --bucket dr-rep-$RANDOM --object-lock-enabled-for-bucket
# put a retention on the object (Governance mode so you can clean up later in a lab):
aws s3api put-object --bucket "$B" --key weights.bin --body weights.bin \
  --object-lock-mode GOVERNANCE --object-lock-retain-until-date 2026-12-31T00:00:00Z
# Now TRY to delete it — expect AccessDenied until the retention expires:
aws s3api delete-object --bucket "$B" --key weights.bin --version-id "$VID"

Write one sentence on the difference between Governance mode (privileged users can override) and Compliance mode (no one, including root, can), and which one you’d use for an irreplaceable model and why.

Rep 5 — Classify an Estate (Reproducible vs. Irreplaceable)

Open code/dr-asset-register.yaml. For each of the six assets, cover the class: line and decide for yourself: reproducible or irreplaceable? Then reveal and check. For every one you’d back up, write a one-sentence justification; for every one you’d rebuild instead, write what it’s derived from.

Add a seventh asset of your own — say, a RAG document corpus the embeddings were derived from — and classify it. (Hint: the embeddings are reproducible from the corpus, so what does that make the corpus?)

Rep 6 — Estate RPO/RTO From the Register

Run the calculator and read its output against the chapter’s §11.3.

pip install pyyaml
python code/rpo_rto.py code/dr-asset-register.yaml

Now break the plan: edit the register so vector-db-hnsw-index has strategy: rebuild-on-recovery and an rto_minutes of 240, and set an aggressive estate goal of 60 minutes in your head. Re-run. Explain in two sentences why a “rebuild the index on recovery” choice can blow an RTO target, and what you’d change to hit 60 minutes (cache the index? back it up? accept a higher RTO?).

Reps 7–9: Real Recovery and the AI Tool

Rep 7 — pgvector Snapshot, Corrupt, Restore-vs-Rebuild

Stand up Postgres with pgvector (Docker is fine), load a few hundred fake vectors, build an HNSW index, then dump it.

CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE documents (id bigserial PRIMARY KEY, embedding vector(384));
INSERT INTO documents (embedding)
  SELECT array_agg(random())::vector FROM generate_series(1,384), generate_series(1,500) g GROUP BY g;
CREATE INDEX documents_embedding_hnsw ON documents USING hnsw (embedding vector_cosine_ops);
pg_dump --format=custom --file=vectors.dump ragdb     # the source of truth

Now drop the index (simulating corruption) and recover two ways: (a) restore the whole dump into a scratch DB; (b) keep the source rows and rebuild the index with REINDEX. Time both. Write one sentence on when you’d restore vs. rebuild — and why you must never let the only copy of your knowledge live in an index you can’t regenerate.

Rep 8 — Run the Restore Test Script

Read code/restore_test.sh before you run it — confirm it restores into a scratch target and never touches production. Then run it against a backup you made in Rep 1/4.

Note that it REINDEXes and times the rebuild, and that it uses mktemp -d + trap so the scratch is always cleaned up. Predict the index-rebuild time, run it, and record the measured number back into your asset register’s rto_minutes. Write one sentence on why measured RTO beats estimated RTO every time.

Rep 9 — Make the AI Draft a Runbook, Then Catch It

Ask an AI assistant (Claude, or whatever your lab provides) to draft a restore runbook for the estate in the asset register: pgvector source, fine-tuned weights served by vLLM, prompts in git, object-locked S3. Require ordered steps with exact commands and a flag on any step needing human sign-off.

Then audit every command before running any of them. Specifically hunt for: a sync running the wrong direction (prod → backup overwriting your good copy), a --delete you didn’t want, a restore targeting the live DB instead of a scratch one, and an RTO estimate that counts only the download and not the engine warm-up + index rebuild. Run the corrected runbook against scratch. In your write-up, paste the one command the AI got wrong (or note that it got them all right this time — and say how you verified that, since you can’t tell by reading) and what running it would have cost you.

Reps 10–11: Continuity and Provenance

Rep 10 — Replication Faithfully Copies Your Mistakes

Set up a primary and a streaming replica (Postgres physical replication, or just two directories you rsync). Insert a row on the primary, watch it appear on the replica. Now run a destructive statement on the primary:

DELETE FROM documents;   -- DESTRUCTIVE — watch it replicate in milliseconds

Confirm the rows vanish on the replica too. Write two sentences: what replication does protect you from (hardware/host failure), and what it cannot protect you from (logical corruption, malice, ransomware) — and therefore why you still need an immutable historical backup.

Rep 11 — Provenance at Recovery Time

Write a minimal model-card.txt for a fictional fine-tuned model: owner, approver, base model + pinned digest, training-dataset reference + checksum, eval results, and the date. Back it up alongside the weights.

Now simulate a recovery where you find two candidate weight files in backup and only one matches the card’s digest. Write one sentence on how the model card lets you certify you restored the right model and not a stale or tampered one — and why “recovery without provenance is just hoping.”

Done? One Last Thing.

A miniature of the project. Take the estate in code/dr-asset-register.yaml and produce a one-page DR plan that:

  1. Classifies every asset reproducible vs. irreplaceable (you did this in Rep 5).
  2. Assigns each a strategy — 3-2-1 + immutable for the irreplaceable tier, rebuild-on-recovery for the reproducible.
  3. Backs up the irreplaceable tier with code/backup_ai_estate.sh (or your own version) to an object-locked target.
  4. Restore-tests at least one asset with code/restore_test.sh and records the measured RTO.
  5. States the estate-wide RPO/RTO from code/rpo_rto.py and names the single weakest link on the critical path.

Then drop one disaster on your own plan in writing — ransomware, region loss, or accidental destroy — and answer in three sentences: what did you lose, how long were you down, and would you stake your name on those numbers? If you wouldn’t, fix the plan until you would. That sentence — “I would stake my name on this” — is the judgment an agent cannot make for you, and it is the whole point of the project.

Up next: Project 11 — Project 11: DR for the AI Estate.