CONTINUITY — Backup, Restore Drills, Rollback Tiers

Status: DRAFT v2.0 — command-center review pending

Descends from: SYSTEM.md §5 (the ledger, everything-as-arithmetic-over-it), bounded by VALUES.md V4. The prototype's only durability story was implicit — content lives in a git repo — and its first real deployment's recovery plan for the agent layer was literally "reinstall if recreated." Nothing covered config, cron schedules, skills, memories, or channel maps; no restore had ever been rehearsed. v2 ships durability as protocol plus workers: massive-failure recovery is a rehearsed procedure with a measured time-to-restore, not an improvisation.

1. Everything-as-code

All system state that can live in the repo does. Directives, doctrine, gate scripts, worker configurations, playbooks, pattern libraries, the ledger, channel and schedule maps as data files — versioned, diffable, restorable. If a piece of state matters and is not in the repo, that is a defect: either move it in, snapshot it in (§2), or inventory it in the secrets ledger (§3).

Everything-as-code is what makes the other three mechanisms cheap:

• Rollback tiers (§4) are only meaningful over versioned state.
• Snapshots (§2) only need to cover what the repo cannot natively hold.
• A restore drill (§5) is only mechanical if the system's shape is fully declared on disk.

The repo is the system's body; the host is a place the body happens to run. Any fresh host plus the repo plus the secrets ledger's restore procedures must equal the system.

2. Nightly sanitized agent-layer snapshots

The agent layer — the machinery that runs the workers, distinct from the content they produce — is snapshotted nightly by a backup worker (a script: cheap, mechanical, silent when unchanged):

• Covered: agent configuration, the cron/schedule registry, skills, memories, channel maps and directory — everything the agent layer needs to be itself.
• Sanitized: secrets are excluded, always. The snapshot is committed to the repo (or a paired private ops repo) with a dated tag. A leaked snapshot must be an inconvenience, not a compromise — no key, token, or credential value ever enters a snapshot. Sanitization is a checked property, not a habit: the backup worker's output passes a dumb secret-pattern scan before commit, and a match blocks the commit and raises a finding.
• Dated tags make every night a named restore point; the snapshot series is itself ledgered (snapshot id + timestamp + content hash), so gaps are detectable by timestamp arithmetic.

3. The secrets ledger

Secrets are the one class of state that must never be machine-restorable — credentials are granted, never derived (V4), and restoring them is a human provisioning act by design.

What ships instead is a secrets ledger: the inventory in git, the values never. One entry per key:

The ledger makes a total-loss restore a checklist ML can execute rather than an archaeology project. The prerequisite check from launch (seo/ONRAMP.md §2) doubles as the restore-time verifier: it reads the secrets ledger, tests every listed credential, and reports exactly which grants are missing.

4. Three rollback tiers

Rollback exists at three named tiers; naming them is what makes the middle and top tiers exist at all (the prototype had only the first, implicitly).

1. Change-level. The finest grain: a committed change (directive update, playbook merge, pattern addition, worker change) is rolled back through the same gate and review machinery that admitted it — reviewer-triggered rollback on regression, and retire-with-record for workers: retirement carries the full record of what the worker was, why it retired, and what replaced it, so removal is reversible and auditable. Removing capacity demands stronger evidence than adding it.
2. Commit-level. Any gated change or publish is a repo commit; git revert restores the prior state of any artifact, page, or configuration without touching anything else. The everything-as-code rule (§1) is what guarantees this tier's coverage.
3. System-level. Total loss of the host: restore the agent layer from the latest dated snapshot (§2) plus the repo onto a fresh container/host, then re-grant secrets per the ledger (§3). The system-level tier is the one that must be rehearsed, because it is the one that is never exercised naturally (§5).

Each tier's trigger, actor, and procedure live in the ops runbooks (ops/DEPLOYMENT.md); this document fixes the doctrine that all three exist and are tested.

5. Restore drills — an untested backup is a hypothesis

A backup that has never been restored is not a backup; it is a hypothesis about a backup. The drill machinery makes it a verified claim:

• A periodic restore-drill worker restores the latest snapshot plus repo into a scratch environment — a fresh container that touches nothing live — and verifies: the system boots; configuration, cron registry, skills, memories, and channel maps are present and coherent; a dry-run of representative loops executes end to end (no publishing, no external contact — the scratch environment holds no real keys, which the drill also verifies: a snapshot that restores a working credential has failed sanitization).
• The drill emits a typed outcome record: snapshot id + restore result + **measured time-to-restore** + timestamp. Time-to-restore is a metered series with a drift alarm — a restore that has been getting slower is a finding before it is an outage.
• A failed drill is bad news and survives compression up the funnel intact (V3): a deployment whose restore drill fails is running without a proven recovery path, and the executive brief says exactly that.
• The drill worker is a silent-failure detector's inverse — its subject (the snapshot series) is permanent, so the worker is permanent. It is never retired on silence, and its own function is canary-verified: periodically a deliberately-broken snapshot must be caught as a failed restore. A drill that passes everything is indistinguishable from a drill that checks nothing.

Drill cadence starts at build settings (frequent, while the snapshot machinery itself is young) and graduates like any gate — on evidence that consecutive drills pass and the snapshot format is stable, never on the calendar.

6. Scope notes

• Published content is durable through the site repo and its host platform's history; the content registry and redirect topology are repo state under §1. The live site can be rebuilt from the repo by the same deployment mechanics that publish to it.
• The ledger is append-only repo state and therefore covered by tiers 2–3; its integrity (no rewritten history) is a monitoring-layer check, not a continuity one.
• What is deliberately not restorable: secret values (§3, by design) and in-flight sandbox exploration (disposable by law — anything worth keeping crossed a commit boundary and is therefore in the repo).

7. What changed from the prototype

8. Open items

• Snapshot destination policy — same repo vs paired private ops repo — and whether the channel directory contains anything sensitive enough to force the private option. [OPEN — flag for command center]
• Drill cadence numerics (build setting and graduated setting) — joins the constants-to-be-measured ledger. [OPEN — flag for command center]
• Time-to-restore target (the acceptable ceiling before it is graded a finding). [OPEN — flag for command center]
• Whether the ledger's append-only integrity check belongs here or wholly in the monitoring layer — drafted as monitoring-owned (§6). [OPEN — flag for command center]