Secret Rotation

Every credential in Sanctum has one of three rotation stories — automatic via management API, half-automatic with a browser shim, or fully manual via the provider’s web UI. The rotator (tools/secret-rotator/rotate.py) is the single tool for all three. Whichever path a secret lives on, the rotator handles atomic propagation, verification, and the JSONL audit log. The human touches as little as the provider permits.

The doctrine

Scan daily, auto-rotate where it’s safe, notify everywhere else. Three LaunchAgents on the Mini implement this:

Schedule	Agent	What it does
06:00 ET	com.sanctum.secret-rotation-scan	secret-rotation.sh --scan — greps git remotes + file content for ghp_…, sk-or-…, AKIA…, etc. Detect-only. Has run clean for weeks.
06:15 ET	com.sanctum.secret-age-check	rotate.py check-ages --notify — reads the audit log, posts a Force Flow warn on anything past rotation_max_age_days. Never rotates.
06:30 ET	com.sanctum.openrouter-auto-rotate	rotate.py rotate openrouter_api_key --if-older-than 85 — silent no-op until the key crosses 85 days, then mints + propagates + revokes via OpenRouter’s management API.

The split is deliberate. Most providers don’t have an API clean enough to trust with silent rotation. The age-check is the human signal — when it fires a notification, you know it’s actually time, and you run the rotator yourself with a real value.

The three rotation kinds

Each entry in tools/secret-rotator/providers.yaml declares one auto_rotate.kind:

kind: openrouter — fully automatic

OpenRouter exposes /api/v1/keys for keys-of-keys management. The rotator pre-flights /auth/key to confirm the configured openrouter-mgmt-key keychain entry is actually a provisioning key (not a regular API key — common misconfig that fails 401 after the fact), then POSTs a new key with the configured label, propagates the new value to every consumer (MBP keychain, VM openclaw.json), HTTP-verifies the new key works, and DELETEs every prior key sharing the label. Old key dies only after the new one is verified — never both keys dead at once.

kind: driver — script-emitted

For providers without a usable management API but with a browser-driveable web UI. The driver script (drivers/voipms.py is the live example) generates new values locally with secrets.token_urlsafe, drives the provider’s site through agent-browser, and emits a JSON bundle. Multiple secrets that share a session use bundle_id/from_bundle so one login covers both — voip.ms’s dashboard and API passwords come out of one browser run.

This is the path most likely to break, because providers do change UIs. Each driver writes a screenshot to /tmp/<driver>-rotator/<ts>-<stage>.png on any failure, so the next person to look at it sees exactly where the form changed. voip.ms specifically sits behind Cloudflare and currently fails the headless probe — the driver structure is in place for the day Cloudflare relents or a CDP-attached Chrome is wired up.

kind: http_manual — paste-and-propagate

The default. The rotator opens the provider’s revoke and create URLs in the browser, reads the new value from getpass (or --from-clipboard, or --from-env for CI), validates against the declared regex / min_length, runs the optional HTTP verify, then propagates. The new value never lands in argv or shell history.

The atomic write path

Whatever the rotation kind, the consumer-write path is identical:

consumers:
  - type: keychain
    service: openrouter-api-key
    account: sanctum
  - type: vm_json
    host: ubuntu@10.10.10.10
    path: /home/ubuntu/.openclaw/openclaw.json
    json_path: models.providers.openrouter.apiKey
  - type: sops
    file: ~/.openclaw/secrets.enc.yaml
    path: voipms.dashboard_pass

Each consumer writes through a structured dot-path walk that refuses to traverse non-dicts. No exec, no shell interpolation, no string templating into shell commands. SOPS edits go through sops set so the existing recipient list is preserved; JSON files are rewritten atomically with a timestamped .bak-<ts> next to them.

A failed verify aborts before any consumer is touched. A failed write to one consumer does not roll back the others — by design, because half-rotated state is detectable in the audit log and recoverable from the backups, while a “rolled-back” mutex would just mask the failure.

The audit log

Every action lands in ~/.sanctum/state/secret-rotation.jsonl, append-only, JSONL with UTC timestamps. Six event types:

• rotation_started / rotation_finished — bookends, success flag on the latter
• verify — HTTP verify result
• consumer_write — one per consumer, with ok and any error
• rotation_skipped — self-throttle from --if-older-than
• manual_rotation — recorded when the rotation happened outside the rotator (provider UI), with a reason field
• openrouter_created / openrouter_revoked — auto-rotate provenance

rotate.py audit prints the last 50 entries. rotate.py check-ages walks the same log to compute “days since last rotation” per secret.

Doing a rotation

Most days, nothing. The auto-rotate either skips or silently re-keys; the age-check exits 0; the scan prints Remote scan clean. When something is overdue, you’ll see a Force Flow notification. The flow:

# Show me everything tracked + when each was last rotated
python3 tools/secret-rotator/rotate.py list
python3 tools/secret-rotator/rotate.py audit | tail -20

# One specific secret, interactive
python3 tools/secret-rotator/rotate.py rotate <name>

# Batch every secret in providers.yaml, auto-rotatable first
python3 tools/secret-rotator/rotate.py rotate all --from-clipboard

# After voip.ms rotated via the provider UI: write the new values into SOPS,
# audit-log a manual_rotation event so the age-check picks it up
sops set ~/.openclaw/secrets.enc.yaml '["voipms"]["api_pass"]' '"<value>"'
echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"manual_rotation","secret":"voipms_api_pass"}' \
  >> ~/.sanctum/state/secret-rotation.jsonl

What’s tracked

Secret	rotation_max_age_days	Path
openrouter_api_key	90	auto via management API
voipms_dashboard_pass	365	manual via provider UI (Cloudflare)
voipms_api_pass	365	manual via provider UI

GitHub authentication is intentionally not in this table — per-repo SSH deploy keys don’t have a rotation surface. Sanctum’s mTLS PKI rotates via tools/mtls-pki/, not the secret-rotator.

Caution

The auto-rotate’s existence is conditional on the provider’s management API behaving. The pre-flight /auth/key check exists because a regular API key under openrouter-mgmt-key will look correct until the first POST /api/v1/keys returns 401. If you ever swap the provisioning key, the rotator will refuse to mint until the new one passes is_provisioning_key.

When something breaks

Each backup file (<file>.bak-<ts>) is a complete pre-rotation state. To roll back: mv <file>.bak-<ts> <file>. The rotator never deletes its own backups — that’s a manual cleanup decision, not an automatic one.