Skip to content
Le Sanctum

Upgrade Path

Upgrade Path — a staircase climbing out of a darkened cellar into a lit hall, each step a small brass plate; a scientist’s notepad firmly closed on a side table because this is not a research project today.

Sanctum upgrades should be boring. The system is now strong enough that upgrades should look like a bounded operator procedure, not a spiritual trial.

The Rule

Section titled “The Rule”

Do not upgrade on top of known drift.

Before touching anything, run:

Terminal window
python3 tools/sanctumctl.py doctor --quick

If that fails, fix the drift first. Upgrading a machine that is already lying to you is how one problem becomes three.

Standard Upgrade Flow

Section titled “Standard Upgrade Flow”
  1. Pull the checked-in changes.
  2. Run python3 tools/sanctumctl.py render to sync generated surfaces.
  3. Run python3 tools/sanctumctl.py doctor --quick.
  4. Run python3 tools/sanctumctl.py verify.
  5. If docs changed, run pnpm build in sanctum-docs.
  6. Only then consider the upgrade complete.

Minimal command sequence:

Terminal window
git pull
python3 tools/sanctumctl.py render
python3 tools/sanctumctl.py doctor --quick
python3 tools/sanctumctl.py verify
cd sanctum-docs && pnpm build

Five commands, in order. If any one of them goes red, stop. An upgrade rolled on top of a lie just produces a more elaborate lie, and the truth still has to be paid for eventually — usually at 3 AM, usually on a Sunday.

Automated Sunday Cadence

Section titled “Automated Sunday Cadence”

In addition to manual upgrades, the Mac Mini executes weekly automated Tier A upgrades every Sunday at 09:30 via haus.sanctum.sunday-upgrade (~/.sanctum/bin/sanctum-sunday-upgrade.sh):

  • OpenClaw: Validates 48-hour soak time and zero open crash issues before upgrading Mini CLI, VM CLI, and VM gateway.
  • Homebrew: Runs brew update && brew upgrade with post-upgrade checks for Tailscale skew, pinned dylib linkage, and 1Password CLI authorization. brew upgrade skips a pinned formula but still upgrades its dependencies. When one of them changes its dylib name, the pinned binaries can no longer start. The linkage check reads every binary and module of each pinned keg; a pinned Python links its dependencies from its modules. If it finds such a pin, it pages the operator through Force Flow on Signal and the iPhone push, naming the missing dylib and the fix (brew unpin X && brew upgrade X && brew pin X). It also pages when otool cannot run, because a blind check must never read as clean. A warning in the log alone once left a broken pinned Node unnoticed for 11 days. sanctum-sunday-upgrade.sh --check-pins runs that check by itself.
  • Python & MLX Ecosystem: Upgrades standalone uv tool CLI binaries (mlx-lm, mlx-vlm, mlx-whisper), refreshes ~/.cathedral-venv and ~/Projects/mlx-finetune/.venv, and validates import smoke tests.
  • Airgapped VM Ubuntu Security (--patch-yoda-only): Invokes Windu’s ephemeral ports.ubuntu.com loopback proxy over SSH -R (windu-threat-watch.py --patch-yoda-only) so yoda receives GPG-verified Ubuntu apt and SHA-256-verified /opt/node26 point updates without opening a persistent internet route.
  • Daily Security Lane (07:40): Between Sundays, haus.sanctum.windu-threat-watch refreshes tap indexes across manoir, mbp, and chalet, patches unpinned and same-major pinned security runtimes (HOMEBREW_NO_INSTALL_CLEANUP=1 + brew linkage --test + --check-pins), updates Home Assistant Green (add-ons, Core, and OS with --backup), and verifies macOS SoftwareUpdate security pillars — see 2026-10-24: The Tap That Froze the Watch.
  • Verification Sentinel: haus.sanctum.sunday-upgrade-check triggers every Monday at 10:00 to verify upgrade logs and surface parity. Its alert leads with a broken-pin line when the log has one, instead of whichever warning came first.

What render Actually Syncs

Section titled “What render Actually Syncs”

python3 tools/sanctumctl.py render reconciles the generated surfaces that most often drift during upgrades:

  • checked-in workspace manifests
  • runtime manifests in ~/.sanctum/services
  • synced agent capabilities
  • agent markdown
  • runtime calibration artifacts

This is the line between “new code exists in git” and “the running machine is using the new shape.”

If The Upgrade Touches Runtime Services

Section titled “If The Upgrade Touches Runtime Services”

Pay extra attention when changes affect:

  • sanctum/runtime_catalog.yaml
  • tools/render_runtime_services.py
  • launchagent templates
  • health or remediation scripts

Those changes alter the machine contract, not just repo logic. Treat them as operational changes, not cosmetic ones.

Rollback Posture

Section titled “Rollback Posture”

Sanctum does not currently have one universal rollback command. The practical rollback posture is:

  • keep upgrades small
  • keep generated surfaces reproducible
  • re-run render to restore canonical generated state
  • use subsystem-specific recovery paths from Troubleshooting
  • rely on existing backup and restore guidance in Backup & Restore

That is not yet perfect product-grade rollback. It is, however, explicit.

When To Delay An Upgrade

Section titled “When To Delay An Upgrade”

Delay the upgrade if any of these are true:

  • the stability window is active and you are measuring calm runtime behavior
  • doctor --quick is already failing
  • the machine has unresolved runtime drift
  • the docs build is already red

The right time to upgrade is when the current state is legible. Upgrades are not a substitute for diagnosis.