The Resilience Doctrine
The grid dies at 3 a.m. in a January ice storm. The Mini keeps thinking — for a while. Every doctrine before this one quietly assumed that morning never comes: that the lights stay on, that a cloud seat is one handshake away, that a human will notice the red dashboard. On a bad enough day — grid down, uplink dead, HuggingFace gone, operator unreachable, Montreal a no-go — each of those is a single point of failure nobody was watching.
The Resilience Doctrine removes those assumptions: seven small, fail-safe, mostly-dormant pieces, each giving the haus a sense it lacked — of its power, its uplink, its mortality, and its seed. None acts irreversibly unless two independent gates agree, and most ship disarmed by design, logging exactly what they would do so the first real action is never a surprise.
The seven pieces
Section titled “The seven pieces”| Piece | Daemon / tool | Cadence | What it gives the haus |
|---|---|---|---|
| Power-sense + graceful shutdown | haus.sanctum.power |
20s loop | a sense of grid-vs-battery; locks SanctumVault and halts cleanly before the cell dies |
| Operator dead-man’s-switch | haus.sanctum.deadman + deadman-autobeat |
6h check / 30m HID beat | a sense of its own mortality — conscious-intent HID/SSH beats, never passive beacons |
| WAN sentinel | haus.sanctum.wan-sentinel |
every 60s | a sense of its uplink; auto-flip to local seats |
| Offline-mode failover | sanctum-offline |
manual | re-point every cloud seat onto the cathedral, reversibly |
| R2 break-glass backup | sanctum-r2-breakglass |
daily | a geo-redundant rebuild seed on free-tier Cloudflare R2 |
| Cold model-weights archive | sanctum-cold-archive |
manual | a cold, integrity-verified copy of the kyber |
| Continuity paper layer | ~/.sanctum/continuity/ |
paper | a floor that needs no power |
A sense of its own power
Section titled “A sense of its own power”The EcoFlow Delta 3 Max Plus backing the Mini passes mains through, so macOS only ever sees “AC Power” — blind to grid-vs-battery. sanctum-power.py (com.sanctum.power, KeepAlive, 20s POLL_INTERVAL) gives the haus that missing sense, reading state-of-charge over the EcoFlow IoT Open API (GET https://api-e.ecoflow.com/iot-open/sign/device/quota/all, HMAC-SHA256 signed).
The state machine is GRID → BATTERY → CRITICAL (thresholds WARN_SOC=40, CRITICAL_SOC=25, SHUTDOWN_SOC=15, env-overridable). The destructive halt — the most-gated action in the campaign — needs four conditions at once:
| Halt gate | Condition |
|---|---|
| State | CRITICAL (SoC ≤ SHUTDOWN_SOC) |
| Below-shutdown debounce | sustained past SHUTDOWN_DEBOUNCE (60s) |
| Off-mains debounce | sustained past ON_BATTERY_DEBOUNCE (45s) |
| Armed | POWER_ARMED is true |
The shutdown is a bounded ordered sequence — quiesce the cathedral, stop the proxyd writers (signal-cli), /opt/homebrew/bin/limactl stop the VM, cryptographically lock /Volumes/SanctumVault via sanctum-vault-armor lock so the powered-off chassis rests sealed in BFU, sync, and halt cleanly via Aqua System Events shut down (falling back to sudo -n /sbin/shutdown -h now) — each step timeout-bounded, so a hung step is skipped and the run always reaches the protective sync and halt. pmset is corroboration only — under pass-through it always reads “AC Power,” so it never vetoes the EcoFlow sensor.
A sense of its own uplink
Section titled “A sense of its own uplink”Nothing watched the WAN, so a dead uplink used to be noticed only when a cloud council call hung mid-answer. sanctum-wan-sentinel.py (com.sanctum.wan-sentinel.plist, one tick every 60s) closes that. It TCP-probes three anchors on three operators — Cloudflare 1.1.1.1:443, Google 8.8.8.8:53, Quad9 9.9.9.9:443 — plus a real recursive resolve of one.one.one.one. Verdict is UP if any signal succeeds, so no one provider’s bad day reads as a WAN loss (a RST counts as UP — the packet still round-tripped).
The clever part is the WAN-vs-LAN distinction, made via the default gateway: a blackout plus a reachable gateway means WAN_DOWN (uplink dead, LAN fine); an unreachable gateway means LAN_DOWN; an unreadable gateway means UNKNOWN, which holds the prior state. A down verdict needs a 2-cycle debounce; recovery is trusted immediately. When armed (SANCTUM_WAN_AUTO_OFFLINE in {1,true,yes}) and WAN_DOWN, it calls sanctum-offline on; on LAN_DOWN it pages only — local seats cannot help an islanded box.
A sense of where to think when the cloud is gone
Section titled “A sense of where to think when the cloud is gone”sanctum-offline is the hands the WAN sentinel reaches for — a manual, reversible toggle, not a daemon. It snapshots the live proxyd config (~/.sanctum/sanctum-proxy/config.yaml) to config.online-snapshot.yaml, atomically rewrites every cloud seat and the fallback graph onto the cathedral (qwen3.6-35b-a3b-4bit at https://127.0.0.1:1337, mtls); proxyd hot-reloads on the change — no restart. The code seat (Devstral on :3301) is already local and is left untouched; off restores the snapshot bytes verbatim, then deletes it.
A “cloud” seat is one whose provider is openrouter or whose api_base carries :3456 (claude) or :6543 (gemini). Rather than half-migrate, it refuses:
| Refusal | Why |
|---|---|
| exit 5 | generated config still references a cloud backend |
| exit 4 | a snapshot already exists with live cloud refs — won’t clobber the true online seed |
| exit 2 | live config missing or unparseable (off also refuses a bad snapshot) |
It does not health-probe the local backends — it audits config text, not silicon — so a down cathedral still toggles green, then answers nothing.
A sense of its own mortality
Section titled “A sense of its own mortality”Every key in the haus gates on Bert; if he becomes unavailable, the Sanctum is a black box the family cannot open. sanctum-deadman.py (haus.sanctum.deadman, StartInterval 21600s ≈ 6h) closes that gap, paired with sanctum-deadman-autobeat.py (haus.sanctum.deadman-autobeat, every 30m). Rather than forcing manual calendar rituals that drift into false alarms, deadman-autobeat calls sanctum-alive only on genuine conscious physical or terminal intent — kernel IOHIDSystem HIDIdleTime < 3600s at the desk or an active remote SSH TTY idle < 3600s, never passive phone Wi-Fi beacons or background cron jobs. Any real beat holds the ladder at LIVE; true absence climbs it:
| Tier | Days silent | What fires |
|---|---|---|
| LIVE | beat fresh | nothing |
| REMIND | 3d | self-reminder to Bert (warn) |
| CHALLENGE | 6d | out-of-band liveness challenge (critical) |
| ALERT | 9d | louder; executor heads-up is staged, not sent (critical) |
| FLAG | 12d | raise deadman-PRESUMED-ABSENT.flag + audit bundle (critical) |
It is a bell, not a gavel: notify-only, never auto-contacting family — the flag itself says “This is NOT proof. Follow the Letter of Instruction.” Liveness is not gated on power or WAN; a missed beat with everything else up still climbs. The audit at ~/.sanctum/state/deadman-audit.jsonl is append-only (UF_APPEND). An alternate --cancel-code channel resets the ladder when the CLI is out of reach; that code lives only on a gitignored paper card in the fire-safe — never quoted here.
A sense of its own seed
Section titled “A sense of its own seed”Two cold layers survive Montreal itself. sanctum-r2-breakglass ships the minimal rebuild seed — secrets, certs, runbooks, scripts, sentinels, SOPS keys, memory — via restic to a free-tier Cloudflare R2 bucket (sanctum-breakglass), client-side encrypted with the same keychain passphrase (sanctum-backup-key) as every other sanctum repo. Sized to fit R2’s free 10 GB ceiling (~135 MB raw, well under 1 GB after dedup), it alerts past FREE_TIER_WARN_GB=8; a companion sanctum-r2-mint provisions and SigV4-verifies the S3 creds. The full 25 GB restic repo deliberately stays off R2 — it lives on the T9 SSD and Google Drive.
sanctum-cold-archive copies the kyber itself — the model weights the Castellan guards in normal times. sanctum-backup.sh excludes them as “re-downloadable” — but in a HuggingFace-down world they are not, and the council goes mute. This tool tars each cached model (~/.cache/huggingface/hub, ~/.cache/lm-studio/models) through zstd -19, optionally age-encrypts it, and records a sha256 in MANIFEST.json so --verify can prove every archive later. It is resumable and never mutates its sources.
The floor that needs no power
Section titled “The floor that needs no power”Under all of it is ~/.sanctum/continuity/ — a printed, secret-free kit in the fire-safe: a Letter of Instruction (whose Section 4 is the data-recovery procedure), a family one-pager, a service-restart cheatsheet, blank ICE medical cards, and a wartime doctrine. Every credential is a pointer (a 1Password item name, a keychain service name), never a value. The one secret-bearing document, the dead-man cancel card, is paper-only and gitignored — never quoted, photographed, or committed.
The wartime doctrine codifies the rails the whole campaign obeys (Council ruling, 2026-06-19): there is no automated wipe, and if one were ever built it must require two hands, a cooldown to abort, default to safe on uncertainty, and never fire on ambiguity.
Dormant by design — what arms each piece
Section titled “Dormant by design — what arms each piece”Most pieces ship inert. The gate is always a credential or a flag only Bert should set, so nothing acts until he says so.
| Piece | Dormant state | What arms it |
|---|---|---|
sanctum-power |
no EcoFlow keys → UNKNOWN, silent |
Armed live (GRID, 99% SoC) via ecoflow-{access-key,secret-key,sn} in /Volumes/SanctumVault/secrets/ |
sanctum-power halt |
logs the plan, never halts | Armed live (POWER_ARMED=true in haus.sanctum.power.plist) |
sanctum-wan-sentinel |
logs disarmed:page-only |
Armed live (SANCTUM_WAN_AUTO_OFFLINE=true + sanctum-offline verified) |
sanctum-deadman |
manual beat only | Live (tier: LIVE) + haus.sanctum.deadman-autobeat (IOHIDSystem / SSH TTY < 3600s) |
sanctum-r2-breakglass |
logs “DORMANT”, exits 0 | Active daily (sanctum-breakglass R2 bucket + symlink-resolved /Volumes/SanctumVault/secrets) |
sanctum-cold-archive |
dry-run only | COLD_ARCHIVE_DEST on a mounted, writable drive |
| continuity kit | [FILL-IN] placeholders |
Bert hand-fills paper templates; live cancel card sealed inside /Volumes/SanctumVault/continuity/ |
How it composes into resilience
Section titled “How it composes into resilience”The reflexes are no longer merely wired on paper — the core loop is live and armed on manoir. EcoFlow telemetry polls every 20 seconds with low-battery vault-lock and Aqua shutdown armed, WAN loss flips cloud seats onto the local cathedral automatically, deadman-autobeat tracks real human keyboard/SSH presence without crying wolf, and Sanctum Vault Armor seals every credential behind 2-of-3 blinded threshold witnesses before power leaves the rail. That is the same fail-safe instinct the Capacity Doctrine applies to RAM: every irreversible gate requires independent proof, and family contact stays human-gated by council non-negotiable.
What changed is the floor. Before this campaign every one of those failure modes was silent; now each has a sense, a sentinel, and a printed page in a fire-safe that survives even the loss of the box holding everything else.
See Also
Section titled “See Also”- The Capacity Doctrine — the same fail-safe instinct, applied to RAM
- Force Flow — the alert bus every piece pages through
- The Castellan — kernel-as-arbiter keeper of the kyber
- Drift Sentinel — Windu’s silent-failure watcher, hardened in this campaign