Chapter 14 — Maintenance and the Lifecycle of Things

“There is nothing so permanent as a temporary solution.” — engineering adage, on the migrations that never came

“For everything there is a season, and a time for every matter under heaven.” — Ecclesiastes 3:1 (ESV)

Why This Matters

Last week, in Chapter 13, you built a pipeline that ships an AI application and watches it for drift. You learned the standard lifecycle — train, evaluate, shadow, canary, promote, monitor. This week we finish that sentence. Monitor, and then — what?

Because everything you have stood up across thirteen weeks is now aging. The reverse proxy fronting your inference API has a CVE this morning. The GPU driver you pinned in Week 6 is two production branches behind. The base model your RAG service depends on has a retirement date you have not read. The vector database you sized in Week 4 is now 40% larger than the disk you provisioned. None of this is a failure. It is the normal weight of running things — and the discipline of carrying that weight well, on purpose, is maintenance.

Here is the uncomfortable truth a graduate administrator has to make peace with: most of the cost of a system is paid after it works. The build is a week. The operating life is five years. And the thing that kills systems is almost never the dramatic outage — it is the slow accumulation of unpatched packages, undocumented configs, deferred upgrades, and “temporary” solutions that outlive the people who wrote them. Technical debt is not a metaphor. It compounds, it accrues interest, and one day it comes due all at once, usually at 3 a.m.

Both AI threads run straight through this week. AI as the tool you wield: it can triage a 200-CVE feed in seconds, spot end-of-life dependencies across a sprawling estate, and draft an upgrade plan — and it will do all of that with a confidence that is sometimes precisely wrong, recommending a patch that breaks a downstream service it never knew existed. AI as the workload you govern: the models themselves are now the most demanding assets in your lifecycle. They version, they deprecate, they retire on the vendor’s schedule, not yours. A fine-tuned derivative can become irrecoverable the day its base model is pulled. Governing that — model registries, model cards, AIBOMs, audit trails, and the three big GRC frameworks bearing down on all of it — is the new core of administration.

This week’s Christian question is the plainest one in the book, and the hardest: to everything there is a season — when is it time to let go? We are good at building and bad at burying. We keep the legacy box running “just in case,” we leave the deprecated model resolvable “for now,” we defer the retirement because the migration is tedious. Ecclesiastes will not let us off: there is a time to keep, and a time to cast away. Maintenance is mostly the discipline of knowing which season you are in — and acting on it before the season decides for you.

14.1 — Maintenance Is the Job (Not the Afterthought)

Let us define the work precisely, because “maintenance” is a word people use to mean “the boring part.” It is not boring. It is the part where the system either earns its trust over years or quietly rots. Five disciplines sit under the word, and a graduate administrator owns all five:

Notice the shape: every one of these is a control loop. Observe the real state, compare it to the intended state, decide, act, record. You already know this loop — it is the SRE loop from last week and the monitoring loop from Week 9. Maintenance applies it not to a live request but to the fleet itself, over months.

Coach’s Note — The single most useful sentence I can give you about maintenance: the work is to make the boring things boring. A patch should be a non-event. A config change should be a reviewed, reversible, logged non-event. When patching is exciting, it means you skipped the discipline that makes it dull, and excitement in operations is just unmanaged risk wearing a costume.

14.2 — Patch and Vulnerability Management: Triage, Then Act

You will never patch everything at once, and you should not try. A real estate produces dozens to hundreds of advisories a week. The job is not “patch all the things.” The job is triage: rank by real risk, patch the dangerous few now, schedule the rest, and record the decision — including the decision to defer.

Risk is not the CVSS number alone. A 9.8 on an internal box behind two firewalls with no exploit in the wild can wait behind a 7.5 that is internet-facing and on CISA’s Known Exploited Vulnerabilities list. A defensible ranking weighs at least: severity (CVSS), active exploitation (the KEV catalog), exposure (internet-facing?), and whether a fix even exists yet.

The reference scorer in code/patch_triage.py makes that ranking deterministic and auditable — no model in the loop, just a transparent rule you can defend in a change review:

python3 code/patch_triage.py code/cve_feed.json

CVE                CVSS  KEV  TIER
------------------------------------------------------------
CVE-2026-10122      9.8  yes  EMERGENCY (patch in 24h)
CVE-2026-10733      9.1  yes  HIGH (patch this week)
CVE-2026-10477      7.5   no  MEDIUM (next maintenance window)
CVE-2026-10610      4.3   no  LOW (track; defer)
CVE-2026-10588      8.1   no  LOW (track; defer)

Look at that last row carefully — it is the whole point. CVE-2026-10588 is CVSS 8.1, higher than the 7.5 above it, yet it ranks lowest. Why? It has no fix yet (has_fix: false), and the scorer adds a penalty for that: you cannot “patch fast” what has no patch. The right move there is mitigate — restrict exposure, add a compensating control — not rush. A naive sort by CVSS would have inverted this and sent you chasing a fix that does not exist.

AI as the tool here is genuinely strong. Point a copilot at the same feed and it will summarize each advisory in plain English, cross-reference your asset inventory, and draft the maintenance-window plan in a minute. Use it for that. But watch the failure mode precisely: an LLM will confidently tell you a patch is “low risk to apply” when it has no knowledge of the undocumented service three hops downstream that depends on the old behavior. It hallucinates dependency graphs it cannot see. The rule is the spine of this whole book: AI ranks and drafts; the human owns the schedule and signs the risk acceptance. The deterministic scorer is what you defend in the audit; the AI summary is what you read to understand it.

Coach’s Note — Keep one column in every triage table that AI is forbidden to fill: risk accepted by (human name) on (date). The moment you defer a patch, a person — not a model — has accepted that risk on behalf of real users. That signature is the difference between a decision and an accident.

14.3 — Change and Configuration Management: No Surprises, No Snowflakes

A change you cannot explain or undo is not a change — it is a liability with a timestamp. Change management is the boring, vital practice of making every modification to production reviewed, reversible, and recorded. Who, what, why, when, and how to roll back. That is it. The maturity of a shop is measured almost perfectly by how completely it can answer “what changed?” after an incident.

Configuration management is the sibling discipline: defining the intended state of a system and detecting when reality has drifted from it. You already met this in Chapter 8 — Infrastructure as Code is configuration management with a version-control system underneath. The enemy is the snowflake server: a box hand-tuned over years until no one can reproduce it and everyone is afraid to touch it. The antidote is drift detection — periodically diffing the running state against the declared state and treating any gap as a defect to investigate.

# Detect config drift against your declared IaC state (Ansible example).
# --check is a dry run; it reports what WOULD change without changing it.
ansible-playbook site.yml --check --diff
# Exit nonzero / nonempty diff == the box has drifted from intended state.

Both AI threads appear again. As the tool: AI is good at explaining a noisy drift diff and proposing the remediation. As the workload: AI infrastructure drifts in ways traditional servers do not — a GPU driver minor bump, a quantization-library version, a serving-engine flag — and these silently change model behavior, not just system state. A drifted vllm flag can alter throughput; a bumped quantization library can shift output quality. Configuration management for AI means versioning the whole stack that produces an output, because the output is the product.

14.4 — Technical Debt and the Capacity Curve

Technical debt is the gap between the system you have and the system you would build today if you started clean. Some debt is fine — taken deliberately, tracked, paid down on a schedule, exactly like financial leverage. The dangerous kind is the undocumented, unowned debt: the “temporary” auth shim from 2024, the pinned-and-forgotten driver, the model nobody dares retire. The epigraph is not a joke. Temporary solutions are the most permanent things in any estate, because nothing forces their removal.

The administrator’s tools against debt are unglamorous and they work: a debt register (write it down, with an owner and an interest estimate), a refresh cadence (hardware and dependencies have lifespans — plan the replacement before the failure), and capacity planning (watch the growth curve, project the wall, provision before you hit it). That last one bites AI estates hardest. Your vector database in Chapter 4 grows with every document ingested; your model catalog grows with every version you keep “just in case.” Storage and VRAM are finite. Capacity is a maintenance discipline, not a one-time sizing.

Coach’s Note — Every model version you keep around “in case we need to roll back” is real disk, real catalog entries, real audit surface, and real confusion about which one is live. Keeping the last two is prudence. Keeping the last twelve is hoarding. A time to keep, and a time to cast away is a capacity-planning verse before it is anything else.

Here AI as the tool earns its keep: EOL and dependency detection across a sprawling estate is exactly the kind of tedious, wide scan a copilot does well. Point it at your manifests, lockfiles, IaC, and model registry and ask it to surface what is approaching end-of-life — OS releases past their support window, libraries pinned to archived projects, base models with published retirement dates. As of 2026 this is a live problem on the AI side specifically: serving engines and quantization tooling churn weekly (the fact brief notes AutoAWQ and AutoGPTQ were both archived in 2025), so an AI estate accrues EOL dependencies faster than a traditional one. Let the AI build the candidate list. Then a human confirms each item against the real upstream notice — because the model will, on a bad day, confidently report a retirement date that does not exist. The scan is delegable; the confirmation is not.

14.5 — The AI Asset Lifecycle: From dev to retired

Now the workload thread takes center stage, because the hardest things in your modern lifecycle are not packages — they are models, datasets, and prompts, and they move through stages with governance gates between them.

The lifecycle, stated as stages:

dev  ->  staging  ->  production  ->  deprecated  ->  retired
        (gate)       (gate)         (set retire date)  (archive + audit)

A model does not slide from one stage to the next because someone clicked a button. It is promoted through a gate — and a gate that anyone can wave a model through is not a gate. The gates that matter:

The registry that enforces this is the modern equivalent of a package repository. MLflow Model Registry is the common reference. Notice one important 2026 detail: MLflow moved away from the old fixed Staging/Production/Archived stages to named aliases and tags — you tag a version @champion or @challenger and move the alias, rather than mutating a global stage. It is a small change with a big lesson: a “stage” is a pointer, and promotion is re-pointing it after a gate passes. The starter code/model_card.yaml shows the alias-based shape and the governance fields a real card carries.

AI versioning has one trap traditional packages do not. Bump the prompt and you have changed the system’s behavior as surely as bumping the weights — but the weight file is byte-identical, so naive versioning misses it. Version the whole behavior-determining bundle: weights ID + prompt + retrieval config + eval results. The model card in the starter does exactly this.

And rollback is not symmetric the way it is for code. Re-pointing the @champion alias back to the previous version is fast — but if the previous version depended on a base model that has since retired, “roll back” may mean “roll back to something that no longer runs.” This is why §14.8’s discipline matters: a deprecation you have not migrated through quietly removes your rollback target. Keep the last one or two versions resolvable and runnable; archive the rest with intent, not by accident.

14.6 — Documentation as Governance: Model Cards, Datasheets, AIBOMs

In traditional administration, documentation is a courtesy. In AI administration, it is a control — increasingly a legally required one. Three artifacts carry the weight:

• Model cards (Mitchell et al., FAccT 2019): the label on the model — intended use, training data references, evaluation metrics, safety results, owner, approver. Required for any model promoted past development. No card, no promotion.
• Datasheets / data cards (Gebru et al.): the same idea for the dataset — provenance, consent, PII class, known gaps and biases. This is where data governance from Chapter 4 becomes lifecycle policy.
• AIBOM (AI Bill of Materials): the ingredients label for the whole system. The two live formats as of 2026 are CycloneDX ML-BOM (v1.7, standardized as ECMA-424) and SPDX 3.0 with its AI and Dataset profiles. An AIBOM inventories weights, datasets, configs, and dependencies so that when a base model gets a deprecation notice, you can grep your estate for the dependency instead of discovering it on retirement day.

# An AIBOM is just signed, structured inventory. Validate it like any artifact.
cyclonedx validate --input-file aibom/ministry-rag-summarizer-2.3.0.cdx.json

The audit trail underneath all of it must be immutable — WORM storage (object-lock S3 or equivalent), which you built for backups in Chapter 11. Here it serves a second master: the EU AI Act’s Article 12 requirement that high-risk systems automatically log over their lifetime. Same storage primitive, two jobs — protection and provenance.

14.7 — The Three Frameworks Bearing Down on Your Estate

You cannot govern an AI lifecycle in 2026 without a working map of three regimes. They are layers, not competitors — a US voluntary framework, an EU binding law, and an internationally certifiable standard.

A few load-bearing facts a graduate must carry, with the hedges they require:

• The EU AI Act timeline is the single most volatile fact in this book. Prohibitions and AI-literacy duties have been in force since Feb 2, 2025; GPAI and governance obligations since Aug 2, 2025. Most high-risk obligations were statutorily due Aug 2, 2026 — but the 2026 “Digital Omnibus” (provisional political agreement May 7, 2026) defers standalone Annex III high-risk to Dec 2, 2027 and product-embedded Annex I to Aug 2, 2028. As of mid-2026 that Omnibus is not yet formally adopted in the Official Journal, so the original Aug 2, 2026 date technically remains the legal default until adoption. Phrase this as “deferred, pending formal adoption,” and confirm the live status before you rely on it. The lesson is bigger than the date: regulatory timelines shift, so your governance must track them as a maintenance task, not assume them fixed.
• GPAI systemic-risk threshold: a rebuttable presumption at >10²⁵ FLOP of cumulative training compute (Art. 51). Fines for prohibited practices reach up to €35M or 7% of global turnover (Art. 99).
• There is also state-level movement — Texas’s TRAIGA took effect Jan 1, 2026 — a reminder that “AI regulation” is not one thing in one place.

AI as the tool, even here: a copilot is genuinely useful for first-pass mapping a system to NIST RMF functions or drafting an ISO 42001 policy snippet. AI as the thing being governed: the model is the regulated object. And the spine rule holds with full force — an AI cannot tell you, authoritatively, whether your system is “high-risk” under a law that changed three weeks ago. That classification is a human, accountable judgment with legal consequences. Use AI to draft; never let it sign.

Coach’s Note — Do not memorize the EU AI Act dates as if they were physical constants. Memorize the posture: pin the version, read the primary source, date your claim, and build a recurring task that re-checks the regulation the way you re-check a CVE feed. A governance program that assumes the law sits still is already out of date.

14.8 — Deprecation and Retirement: A Model EOL Is an EOL

Bring the workload thread home. When a base model you depend on gets a retirement date, treat it exactly like an operating-system or package end-of-life: a dated, owned, tested migration — not a panic on the last day.

The real Claude retirement schedule makes a clean teaching example (verify the live dates before acting — they move): as of mid-2026, Anthropic published retirements such as Claude Opus 4.1 retiring 2026-08-05, Opus and Sonnet 4.0 retiring 2026-06-15, and Haiku 3 retired 2026-04-19. Pin the exact model ID, read the official deprecation notice, and write the migration runbook. The starter code/model-migration-runbook.txt is that runbook: inventory every reference, flag fine-tuned derivatives first, choose and eval the successor, stage and gate, canary the cutover, and actually retire the old one — archive to WORM, confirm zero references, close the change record.

One trap deserves its own sentence, because it ends careers quietly: a fine-tuned model can be inference-only until its base model is retired — and then irrecoverable. If you fine-tuned on a base that gets pulled and you did not migrate in time, the derivative may simply stop working with no path back. Find those derivatives first in any deprecation. They are the assets with the shortest fuse.

# Step 0 of any model EOL: you cannot migrate what you cannot find.
grep -rn "claude-opus-4-1\|claude-sonnet-4-0" . \
  --include="*.py" --include="*.yaml" --include="*.tf" --include="*.json"

14.x — Interactive Lab: Model Lifecycle Board

Below this chapter on the site is the Model Lifecycle Board — an interactive panel where you drag a model through the five stages (dev → staging → production → deprecated → retired), and a governance gate sits between the stages. Do not skim past it; it is the chapter’s central idea made tactile.

Here is the drill. Take the model and try to drag it straight from dev to production. The board will stop you — because the gates have not passed. Now work each gate honestly: run the eval, draft the model card, obtain the (simulated) human sign-off, clear the compliance check. Watch a promotion fail when the eval threshold is not met, and watch it succeed only when every gate is green. Then carry a model all the way to retired and notice what the board forces you to do at the end — set a retirement date, archive, confirm no live references.

What it teaches, in your hands: promotion is not movement, it is a gate that passed. A stage is a pointer; advancing it is an accountable, recorded decision with a human on the hook. By the time you have moved three models across the board — one that sails through, one that bounces off a failed eval, one fine-tuned derivative racing a base-model retirement — the abstract lifecycle in 14.5 will feel like muscle memory. That is the point. The board is a low-stakes gym for the highest-stakes loop in your estate.

14.y — A Time to Keep, and a Time to Cast Away

The whole week sits on one verse: “For everything there is a season, and a time for every matter under heaven” (Ecclesiastes 3:1, ESV). A few lines later the Preacher names our exact discipline — “a time to keep, and a time to cast away” (Ecclesiastes 3:6, ESV). This is not a poem about feelings. It is, read plainly, a maintenance philosophy: the wisdom of an administrator is largely the wisdom to know which season a thing is in and to act on it.

We are, by temperament, hoarders of running systems. We keep the legacy server alive “just in case.” We leave the deprecated model resolvable “for now.” We defer the retirement because the migration is tedious and nothing is forcing it today. Every one of those is a refusal to admit a season has ended. And the cost is not neutral — kept past its time, a system stops being an asset and becomes a liability that consumes the attention owed to living things. The unpatched box is the perimeter’s weakest point. The un-retired model is the audit’s biggest question mark. The undocumented snowflake is the outage no one can explain.

The LCMS frame sharpens it. This is vocation and stewardship: you are entrusted with what you did not make, and faithfulness includes the unglamorous, terminal acts — the patch, the migration, the retirement. Letting go well is a form of care, not neglect. The steward who buried his talent to keep it “safe” was the one condemned (Matthew 25:14–30); refusing to act on a season is its own kind of unfaithfulness. There is also a quiet humility here that the engineering needs: we did not build the season, and we do not get to suspend it. The regulation will change on its schedule. The model will retire on the vendor’s date. The hardware will age on physics’ clock. Our authority is real but bounded — we govern within seasons we did not set. The administrator who accepts that builds migration runbooks early and sleeps the night before a deprecation. The one who denies it is always surprised by a calendar.

There is a time to cast away. Knowing when — and doing it on purpose, documented, before it is forced — is the maturity this week is teaching.

14.z — Common Pitfalls

Pitfall: Patching by CVSS number alone. Example: A team patches every 9.x in the catalog and leaves a 7.5 internet-facing, actively-exploited bug for “next month.” The 7.5 owns the perimeter by Friday. Fix: Triage on real risk — CVSS and active exploitation (KEV) and exposure and fix availability. See code/patch_triage.py.

Pitfall: Trusting an AI’s “safe to upgrade” verdict on production. Example: A copilot calls a driver bump low-risk; it breaks an undocumented downstream service it could never see, and the model’s throughput silently halves. Fix: AI drafts the plan and the rationale; a human tests in staging and signs the change. The model cannot see your whole dependency graph — you must.

Pitfall: Versioning weights but not prompts. Example: Someone edits the system prompt; the weight file is byte-identical, the “version” never changes, and a behavior regression ships with no record of what moved. Fix: Version the whole behavior bundle — weights ID + prompt + retrieval config + eval results — in one model card. See code/model_card.yaml.

Pitfall: Treating the EU AI Act dates as fixed. Example: A 2025 compliance plan hard-codes “high-risk obligations apply Aug 2, 2026” and never re-checks; the Digital Omnibus deferral (pending adoption) makes the plan both wrong and untracked. Fix: Pin the version, cite the primary source, date the claim, and schedule a recurring re-check of the regulation — governance is a maintenance task.

Pitfall: Never actually retiring anything. Example: Twelve model versions linger in the registry “for rollback”; nobody knows which is live; the audit takes a day to answer “what’s in production?” Fix: Set a retire_after date at deprecation, keep the last two for rollback, archive the rest to WORM, and confirm zero references before retiring.

Pitfall: Forgetting fine-tuned derivatives in a base-model EOL. Example: A fine-tuned summarizer was inference-only on a base that retires; the migration plan missed it, and on retirement day it is simply gone, unrecoverable. Fix: In any deprecation, grep for derivatives first; they have the shortest fuse. See code/model-migration-runbook.txt.

14.aa — Reps

This week the reps are the graded lab — there is no separate project. Read the exercises and do every rep at the keyboard. They build, in order, toward a real model lifecycle and maintenance plan you could hand to a team. A preview:

• Run the deterministic patch-triage scorer, then re-triage the same feed with an AI copilot and write up where the two disagreed and which you trusted.
• Detect config drift with an Ansible --check --diff dry run and explain every line of the diff.
• Fill out a governance-grade model card for a real model and validate it.
• Build a minimal AIBOM (CycloneDX ML-BOM) for a small AI service and diff it against an SPDX 3.0 view.
• Write a base-model deprecation migration runbook from the starter and dry-run the inventory grep.

AI policy for the reps (Phase 2): AI is part of the work and increasingly agentic. Use it — and keep an honest agent-log.txt noting what you delegated, what it got wrong, and where you intervened. Several reps are deliberately shaped so an agent cannot finish them alone: the sign-offs, the risk acceptances, and the regulatory classifications are human judgments you own. A short “Check Your Reps” quiz is embedded on this page below the lab — take it after the reps, not before.

14.bb — This Week’s Project

There is no new project this week. Week 14 is a drill-and-lab week: the substantial graded work lives in the exercises, which becomes this week’s assignment. Treat the capstone lab there — the lifecycle and maintenance plan — as the deliverable.

Use this week to get ahead on capstone prep. The Week 16 final (Chapter 16) asks you to architect an AI-native organization, and a complete governance framework is one of its graded deliverables. Everything you build this week — model cards, AIBOMs, the lifecycle board’s gate model, the three-framework map, WORM-backed audit trails — is a piece of that capstone. Start the governance section of your final now, while it is fresh. See Appendix A for the lab environment, Appendix B for local and cloud AI setup, and Appendix C for the agentic-AI rules your agent-log.txt must satisfy.

14.cc — Coach’s Final Word

Most of this book has been about building and running. This week was about the part nobody puts on a slide: the patch, the drift check, the migration, the retirement. It is the least glamorous discipline in administration and the one that most separates a professional from an amateur. Amateurs build. Professionals build and keep — and know when to let go.

The dual-AI thread has been with us all week, and it lands hard here. AI will triage your CVEs, draft your runbooks, map your frameworks, and version your models faster than you can. Wield it — that is the whole point of the tool. But the patch schedule, the risk acceptance, the regulatory classification, the decision to retire — those carry a human name and a date, because real people depend on the system and someone must answer for it. A confident, fast, sometimes-wrong partner is wonderful for the first draft and dangerous for the final signature. Keep the judgment where the judgment lives.

And keep the season in view. To everything there is a season, and a time for every matter under heaven. You will be tempted to keep things past their time, because letting go is tedious and nothing forces it today. Resist it. Build the migration runbook early. Set the retirement date at deprecation. Cast away on purpose, documented, before the season casts away for you. That is stewardship, and it is the quiet center of this whole craft.

See you next week.

Up next: Do the lab in the exercises — it is this week’s graded assignment and your first real piece of the capstone governance deliverable. Reference Appendix A for the environment, Appendix B for AI setup, and Appendix C for the agentic rules. Then Chapter 15 — AIOps and Autonomous Operations: How Much May We Entrust?