Project 11 — DR for the AI Estate

“The prudent sees danger and hides himself, but the simple go on and suffer for it.” — Proverbs 22:3 (ESV)

Chapter: 11 — Backup, Recovery, and the Ark You Build Before the Flood Due: End of Week 11 Submit: A link to a public GitHub repository containing your backup/restore scripts, the asset register (dr-asset-register.yaml), a measured restore-test transcript, a DR-PLAN.txt, and the required agent-log.txt. See Appendix A §A.11 for the lab toolkit (MinIO/object-lock, the aws CLI, Postgres + pgvector, PyYAML) and Appendix A §A.12 for how to submit (the GitHub repo, push auth, and the Phase 1 = AI_USAGE.txt / Phase 2 = agent-log.txt checklist). See Appendix B for the local + cloud AI you’ll be protecting. Allowed tools: Bash, the aws CLI pointed at MinIO with object-lock (free, no AWS bill — see Appendix A §A.11) or real S3, Postgres + pgvector (A.11), Python 3 with PyYAML (A.11), git, Docker, and an AI assistant per the policy below. AI — Phase 2 (agentic AI is ON): You may delegate scripting and the first draft of the runbook to an agent. An agent-log.txt is REQUIRED: every task you delegated, what the agent produced, where it was wrong (especially any restore command), and where you intervened. The classification and the RPO/RTO decisions are yours. An agent may write the tar/aws/pg_dump mechanics once you’ve specified the asset tiers; it may not decide which assets are irreplaceable or what data loss you’ll accept. That judgment is the architecture, and the architecture is the part of this project an agent cannot do for you. The project is deliberately shaped so an agent cannot finish it alone — the graded core is the memo and the measured restore, not the scripts.

The Setup

Grace Fellowship runs a small but real AI estate. Last year a volunteer team built a RAG assistant that answers congregants’ questions about ministries, events, and the church’s positions — grounded in a curated corpus the team spent three months assembling and labeling. It runs on a fine-tuned open-weight model (the fine-tune sharpened its tone and taught it the church’s specific answers), a pgvector store holding the embeddings and source documents, a prompt library the team has tuned through dozens of revisions, and a small dataset of question/answer pairs used for evaluation. It serves a few hundred people a week. It is loved, and it is completely unprotected.

Then, on a Friday afternoon, the IT volunteer’s laptop is compromised through a phishing email. By Monday, ransomware has encrypted the serving host and the single external drive where someone had been manually copying “the AI files” — because that drive was mounted and writable. The fine-tuned weights, the vector store, the prompts, the eval set: all encrypted, with a ransom note. The team has no clean copy.

Your job is to make sure this cannot end that way — to build the ark before this flood, for an estate exactly like Grace Fellowship’s. You will classify every asset, design a DR plan with real RPO/RTO targets, implement 3-2-1 backups with immutability where it counts, and — the part that earns the grade — prove you can recover by running a real restore test and measuring it. And because you are now an architect, you will write the memo that makes the judgment calls and defend the numbers you’d stake your name on.

Setup (the starter)

This chapter’s code/ directory is your starting kit:

• code/dr-asset-register.yaml — the asset register to adapt; classifies each asset and sets its strategy/RPO/RTO.
• code/backup_ai_estate.sh — a reference 3-2-1 backup of the four AI asset classes; adapt it to your estate.
• code/restore_test.sh — restores into a scratch target, verifies checksums, rebuilds + times the index. This is how you earn the word “backup.”
• code/rpo_rto.py — computes estate-wide RPO/RTO from the register so your numbers are honest.

Stand up a local estate to protect: Postgres + pgvector in Docker (install per Appendix A §A.11) with a few hundred fake vectors and a documents table; a small “weights” file standing in for a fine-tune; a git repo for the prompt library; and a couple of dataset files with a checksum manifest. rpo_rto.py needs PyYAML — also from A.11 (pip install pyyaml in your venv). For the WORM/object-lock tier, don’t pay an AWS bill: stand up MinIO with an object-lock bucket and point the aws CLI at it with --endpoint-url http://localhost:9000 (LocalStack or real S3 work identically) — just keep object lock real so you can prove a locked object refuses deletion. (Create that locked bucket with the aws CLI as shown in feature 2 below; A.11’s mc mb --with-lock local/wormbucket is the alternative if you install the MinIO mc client.)

Note — the scripts use the Postgres client tools. restore_test.sh and backup_ai_estate.sh call pg_dump, pg_restore, createdb, dropdb, and psql as host binaries. A.11 runs Postgres only as the pgv Docker container, so install the client tools with sudo apt-get install -y postgresql-client, or run each command inside the container, e.g. docker exec pgv pg_dump ... / docker exec -i pgv psql ... (the docker exec pgv pattern A.11 already uses).

Learning Targets

By completing this project, you will demonstrate that you can:

• Classify every asset in an AI estate as reproducible vs. irreplaceable, and defend each call.
• Implement a 3-2-1 backup with immutability (object lock / WORM) on the irreplaceable tier.
• Distinguish a backup from a snapshot, a replica, and a sync — and architect accordingly.
• Set and measure RPO and RTO for each asset and for the estate, accounting for AI-specific warm-up and index-rebuild cost.
• Run a real restore test into a scratch target and prove recovery with checksums and row counts.
• Use an AI assistant to draft a DR runbook and critically verify every command before trusting it.
• Make and defend an architecture-level DR recommendation — the judgment an agent cannot make.

Normal Tier

Goal: Classify the estate, back it up 3-2-1 with immutability on the irreplaceable tier, and pass a real, measured restore test.

Required features

1. An asset register you own. Adapt dr-asset-register.yaml to the Grace Fellowship estate: fine-tuned weights, vector-DB source, vector-DB HNSW index, prompt library, eval datasets, base-model checkpoint, configs. For each: class (reproducible/irreplaceable), strategy, immutable, rpo_minutes, rto_minutes. Every classification is justified in DR-PLAN.txt.
2. A working 3-2-1 backup of the irreplaceable tier: 3 copies, 2 media/stores, 1 offsite. The irreplaceable tier lands in an object-locked bucket (Compliance or Governance mode — state which and why). You do not need an AWS account or bill for this: use the MinIO object-lock stand-in from Appendix A §A.11 and drive it with the aws CLI (already installed in A.11) plus --endpoint-url. Create the locked bucket with the aws CLI exactly as the chapter shows: aws s3api create-bucket --bucket wormbucket --object-lock-enabled-for-bucket --endpoint-url http://localhost:9000. (Prefer the MinIO mc client? mc mb --with-lock local/wormbucket does the same — but it needs you to install mc and run mc alias set local http://localhost:9000 admin devpass123 first; A.11 ships only the MinIO server.) Real S3 Object Lock works identically — the lock semantics are the same, and either path earns the full points.
3. Reproducible assets are NOT backed up — they’re cached/pinned (base checkpoint by digest) or marked rebuild-on-recovery (HNSW index). Your register and script must reflect this; the grader will check you didn’t pay to store rebuildable bytes.
4. Checksum manifests on datasets, captured at backup time and verified on restore.
5. A passing restore test. Run restore_test.sh (or your own) into a scratch target. Capture the transcript: checksum verification, recovered row count, and the measured index-rebuild time. Commit it as restore-test.log.
6. Estate RPO/RTO computed, not guessed, via rpo_rto.py, with the output committed and the weakest link on the critical path named in DR-PLAN.txt.

Example session

$ python rpo_rto.py dr-asset-register.yaml
Estate RPO (max data loss):    60 min
Estate RTO (serial recovery):  300 min
Irreplaceable (back these up): fine-tuned-weights, vector-db-source, prompt-library, eval-datasets
Reproducible (rebuild, don't pay to store): vector-db-hnsw-index, base-model-weights

$ ./restore_test.sh s3://gf-ai-backups 2026-06-19T03-00-00Z
PASS datasets: checksums match
Vector source rows recovered: 512
real 41.20        # measured index-rebuild time -> this IS the RTO for the index

Normal-tier rubric (out of 100)

Medium Tier (+up to 25% extra credit)

Goal: Survive a real disaster scenario and prove your judgment about AI-drafted recovery.

1. Ransomware game day. Simulate the Grace Fellowship incident: encrypt (or delete) the live estate and a writable backup copy. Recover only from the immutable copy. Document what survived, what was lost, the measured RTO, and the achieved RPO. Show that the object-locked copy refused the “attacker’s” delete.
2. AI-drafted runbook, critically corrected. Have an AI assistant draft the full restore runbook. Then audit it: paste the runbook, annotate every command you verified, and call out at least one dangerous or wrong instruction (wrong-direction sync, --delete, restore-into-prod, optimistic RTO) — or, if it got them right, explain exactly how you verified that rather than assuming. Commit as runbook-review.docx.
3. Restore-vs-rebuild decision for the vector DB, with both times measured and a one-paragraph recommendation.

Hard Tier (+up to 25% additional extra credit)

Goal: Make the architecture call. This is graded on judgment, not scripts.

Write DR-MEMO.docx (1–2 pages) to Grace Fellowship’s board that:

1. Recommends a DR tier (hot/warm/cold) per asset with a stated, measured RPO/RTO for each and an estate-wide target — and names, in dollars-and-time terms, where you are deliberately accepting data loss and why (tie this forward to the cost reasoning of Chapter 12).
2. Defends the single hardest tradeoff you made — e.g., not backing up the HNSW index and accepting a longer RTO, or paying for Compliance-mode immutability you yourself cannot override.
3. States the limits of automation here: which recovery steps you would let an agent execute automatically, which require human sign-off, and why automatic failover for an AI service can be the wrong move when the cause was a poisoned input.
4. Ends with the sentence a steward signs: “I would stake my name on these recovery numbers because we have measured them,” followed by the evidence.

A clean memo that makes a defensible call an agent could not make earns more than a perfect script. This tier is where the human-in-the-loop thesis is graded.

Submission

Push a public GitHub repo containing: dr-asset-register.yaml, your backup/restore scripts, restore-test.log, the rpo_rto.py output, DR-PLAN.txt, agent-log.txt (required), and — for the extra-credit tiers — runbook-review.docx and DR-MEMO.docx. Include a top-level README.txt with the exact steps a grader runs to reproduce your restore test. For the GitHub account, git push auth (or the browser-only upload path), and the submission checklist — this is Phase 2, so agent-log.txt is required (Phase 1 would be AI_USAGE.txt) — follow Appendix A §A.12.

Hints (Read Before You Begin)

• Classify before you script. The whole project goes sideways if you start backing up bytes before you’ve decided which bytes are irreplaceable. Fill in the register first.
• Object lock cuts both ways. Compliance mode means you can’t delete it either — perfect for irreplaceable weights, painful if you lock something you’ll need to rotate. Use Governance mode where you need an escape hatch and say so.
• The index is derived state. Don’t back it up out of habit. Back up the source embeddings; rebuild the HNSW index on recovery and measure that rebuild — it’s your real RTO for that asset.
• A restore command is destructive until proven otherwise. Run every one against a scratch DB/bucket first. The restore_test.sh trap/mktemp pattern keeps you honest.
• Predict your RTO, then measure it. The gap is the most useful sentence in your memo.

What Mastery Looks Like (Beyond the Rubric)

Mastery is a board member who can’t read code coming away certain the AI estate is safe — because your DR-MEMO.docx told them, in plain language, what would be lost in each disaster and how long they’d be down, and because you backed every number with a measured restore, not a hope. Mastery is knowing the difference between the weights you can re-pull and the fine-tune you cannot, and spending the church’s money accordingly. Mastery is using the AI to write the runbook in twelve seconds and then refusing to trust a single line of it until you’d run it yourself.

Coach’s Note — The grade you should care about isn’t the rubric. It’s whether, six months from now, when the real Friday afternoon comes, the people who trusted this system get it back. Build it like they will.

When You’re Done (a short checklist)

• Every asset is classified reproducible/irreplaceable, and you can defend each in one sentence.
• The irreplaceable tier is 3-2-1 and immutable; you tried to delete a locked object and it refused.
• You ran a real restore test into a scratch target and committed the transcript with a measured RTO.
• Estate RPO/RTO are computed, not guessed, and the weakest link is named.
• agent-log.txt honestly records what you delegated and where the agent was wrong.
• You would stake your name on the recovery numbers — because you measured them.

A theological footnote. Joseph did not stop the famine; he stored grain so the years of want did not become the years of death (Genesis 41). That is this project. You will not stop disasters — that power was never yours. You can be the steward who, in the years of plenty when no one was watching and everyone thought you were paranoid, built the storehouse anyway. “The prudent sees danger and hides himself” (Proverbs 22:3). The reward for a perfect backup is that nothing happens, and no one thanks you. Do it anyway. The flood is coming, and the grain you store now is the difference for people who trusted you with what they could not replace.

See you next week.