Health Monitoring & Watchdog

The watchdog is the part of Sanctum that doesn’t sleep — a long-running Rust daemon (sanctumd) that wakes every ten minutes, sweeps every service, fixes what it can, and tells you about whatever it couldn’t. It is, literally, a program that watches other programs and judges them. If it ever develops opinions about your uptime, unplug everything.

How It Works

1. Check health — service-graph.py check-all --json probes every manifest, reporting per-service status plus root causes.
2. Evaluate — The graph separates root causes from downstream casualties, so dependants aren’t restarted blindly.
3. Auto-remediate — With auto_fix on, each root cause goes to service-graph.py remediate <svc>, bounded by a restart budget and circuit breaker.
4. Settle — Waits settle_delay for services to stabilize.
5. Re-check — Runs check-all again.
6. Notify — Alerts on anything still failed.

┌──────────────┐     ┌──────────────┐     ┌─────────────┐
│ check-all    │────→│ Root causes? │─No─→│ All Clear   │
└──────────────┘     └──────────────┘     └─────────────┘
                           │ Yes
                           ▼
                    ┌──────────────┐
                    │ remediate    │
                    │ <svc>        │
                    └──────────────┘
                           │
                           ▼
                    ┌──────────────┐
                    │ Settle Delay │
                    └──────────────┘
                           │
                           ▼
                    ┌──────────────┐     ┌─────────────┐
                    │ Re-check     │─OK─→│ Fixed       │
                    └──────────────┘     └─────────────┘
                           │ Still failing
                           ▼
                    ┌──────────────┐
                    │ Notify       │
                    └──────────────┘

Most days the path is: check, all clear, sleep. On the bad days it finds a root cause, remediates, re-checks, and either nods or tells you on Signal your evening is about to get worse.

Schedule

The watchdog is not a periodic LaunchAgent — it’s a continuously-running daemon that loops on its own timer. launchd’s only job is to keep it alive:

Property	Value
Label	com.sanctum.watchdog
Plist	/Library/LaunchDaemons/com.sanctum.watchdog-rust.plist
Program	sanctumd (sanctum-rs release build)
KeepAlive	true (relaunched immediately if it ever exits)
Check interval	WATCHDOG_CHECK_INTERVAL, default 600 (seconds)
Health API	port 2187

It starts at boot and runs forever, one sweep every check_interval seconds. Being a single long-lived process is why the dedup state below lives in memory.

Notification Deduplication

Nothing helps you fix a problem like your phone buzzing about it every ten minutes — so once a service fails and notifies, further failures of that same service are suppressed for the dedup_window. State is an in-memory map of service name to last-notified time, no on-disk file — so a restart wipes it, and the first failure after a relaunch always notifies.

First failure of "home-assistant" → Notify ✓
Same failure 5 min later           → Suppressed (within 30 min window)
Same failure 35 min later          → Notify ✓ (window expired)
Different service fails             → Notify ✓ (independent key)

Danger

If the watchdog process crashes, KeepAlive relaunches it in seconds — but that’s a resurrection, not a logic check. Nobody watches the watchmen’s judgment. Check ~/.openclaw/logs/watchdog-rust.log if you suspect missed alerts. If you find yourself building a watchdog for the watchdog, step outside and look at a tree.

Configuration

All settings live in instance.yaml under services.watchdog:

services:
  watchdog:
    enabled: true
    interval: 600            # seconds between check sweeps
    settle_delay: 15         # seconds to wait after a fix attempt
    auto_fix: true           # enable manifest-driven auto-remediation
    dedup_window: 1800       # seconds (30 min) to suppress repeat alerts
    notify_channels:
      - macos
      - signal
      - dashboard

Setting	Type	Default	Description
enabled	boolean	true	Enable or disable the watchdog entirely
interval	number	600	Seconds between check sweeps
settle_delay	number	15	Seconds to wait after repair before re-checking
auto_fix	boolean	true	Whether to run manifest remediation on root causes
dedup_window	number	1800	Seconds to suppress duplicate notifications

Notification Channels

The watchdog delivers alerts through three channels, routed by severity in ~/.sanctum/lib/notify.sh — the louder the problem, the more channels it lights up:

Severity	macOS	Dashboard	Signal	Channel detail
ambient				Log + chitti samskara only
warn	yes	yes		osascript Notification Center + dashboard banner
error	yes	yes	yes	Full fan-out
critical	yes	yes	yes	Full fan-out (memory kills land here)

The dashboard banner appends to ~/.sanctum/alerts.json, which the command center polls. Signal goes to a configured group via the apple-toolkit skill, reserved for error and critical — and nothing quite says good evening like a Signal from your haus at midnight telling you a service is down.

Remediation

The repair engine is the service graph, not a separate doctor process. With auto_fix enabled, each root cause goes to service-graph.py remediate <svc>, running that service’s manifest-defined action — a heal_launchagent.sh restart, a systemd kick over SSH, a Docker bounce. Blunt, but most failures here respond to “have you tried turning it off and on again.”

Two guards keep a flapping service from turning the watchdog into a respawn machine: a per-service restart budget (max_restarts_per_hour, default 3) and a circuit breaker that halts remediation when too many fail at once.

Tip

The service-doctor.sh script still exists as a manual tool — run it bare for a report, or --fix to remediate every failing service at once. No longer the watchdog’s engine, but it’s what you reach for when the haus is broken and you don’t want to wait ten minutes.

Health Test Suite

The suite isn’t a hardcoded list — it’s whatever manifests live in ~/.sanctum/services/. Each *.yaml declares how its service is probed, and check-all walks every one (roughly 37 today). Add a manifest and the watchdog picks it up next sweep, no code change. A probe is one of a few types:

Probe type	What It Checks	Example
port	TCP port is open (nc -z) — not an HTTP code	home-assistant :8123, sanctum-tts :8008
process	A named process is alive	sanctum-tts (tts_server)
command	A command exits zero	tailscale (tailscale status)
interface	A network interface holds its expected IP	bridge interface

Probes return pass/fail, latency, and a diagnostic message; the graph follows requires edges so a fallen dependency takes the blame, not everything downstream. Roughly 37 services every ten minutes is about 5,300 probes a day. Your haus gets more checkups than you do.

Caution

Two things deliberately absent: the DenchClaw gateway on port 1977 (retired 2026-04-23 — a standing security exception now, not a health test) and LM Studio. The MLX seats actually watched are council-mlx and mlx-builtin. If a doc or dashboard still lists :1977 as a live check, it’s lying to you.

Memory Sentinel

On April 23, 2026, a runaway session pushed the 64 GB Mac Mini into OOM until WindowServer missed its watchdog checkin and the kernel panicked — twice in one evening. The service graph kept dutifully checking ports while the system asphyxiated, because knowing a service is up doesn’t help when there’s no RAM left to run it.

The memory sentinel exists because of that night. It runs before the service graph — which itself spawns Python, threads, and SSH connections, all of which cost memory. Running diagnostics on a starved system is the monitoring equivalent of operating on a burning patient.

How It Works

The sentinel doesn’t reason about whole-system free memory. It reasons about Sanctum’s own services, one at a time: it enumerates every running com.sanctum.* job, reads each PID’s RSS via ps -o rss=, and compares it to a per-service limit in megabytes. Limits are tuned per service — mlx-builtin gets 20 GB because it holds a model in RAM; a tunnel gets 64 MB, anything over is a leak. Override any limit under services.<key>.memory_limit_mb. The worst grades set the overall status:

Level	Trigger	Action
OK	Every service under 80% of its limit	Exit 0, all clear
Warning	At least one service at or above 80%	Exit 0, log only
Critical	At least one service over 100% of its limit	Exit 1, kill the worst offender
Emergency	Three or more services over their limits	Exit 1, kill the worst offender

The “worst offender” is the service the most megabytes over its own limit. On a kill it sends SIGTERM, pauses, then SIGKILLs — and since every victim is a com.sanctum.* job, launchd restarts it clean.

Caution

The flip side: the sentinel only touches com.sanctum.* jobs. A rogue process that isn’t one of yours — a browser tab eating 30 GB — is invisible to it. It guards Sanctum from its own leaks; for everything else, you’re still the watchdog.

Watchdog Integration

The sentinel runs as Step 0.5 — the first thing each sweep does, before any service-graph work:

Watchdog check starting
  → Step 0.5: Memory sentinel pre-flight (--json)
    → If emergency/critical: --kill, notify, sleep 3s
    → If warning: log only
  → Step 1: validate manifests + service-graph check-all

On a kill it fires a critical notification and sleeps three seconds for the system to reclaim memory — so the check-all that follows sees a machine that has exhaled.

Configuration

The real knobs are the per-service memory_limit_mb overrides under each service’s key. The dedicated instance.yaml block also carries some forward-looking scaffolding:

services:
  memory_sentinel:
    enabled: true
    auto_kill: true
    kill_grace_seconds: 5
    thresholds:
      warning_free_pct: 15
      critical_free_pct: 8
      emergency_free_pct: 5
    safelist:
      - "com.apple.Virtualization.VirtualMachine"
      - "sanctum-mlx"
      - "WindowServer"
      - "kernel_task"

Caution

Honesty about what’s wired: memory-sentinel.py grades by per-service RSS vs. limit, so the thresholds and safelist keys above aren’t read yet, and kill_grace_seconds is ignored for a hardcoded 2-second pause before SIGKILL. (The killable set is implicitly safelisted anyway — com.sanctum.* only.) Flagged as code fixes below.

Tip

Run it by hand with python3 ~/.sanctum/scripts/memory-sentinel.py --status for a table, or --json. --kill only acts when a service is over its limit — on a healthy system it reports “no services over their memory limit” and exits without touching anything.

Agent Integration

The health agent can invoke the sentinel through the service-doctor skill’s memory-check.sh, checking pressure and resetting a runaway service without SSH to the host.

Caution

That wrapper still points at a memory-sentinel.sh that no longer exists (the live script is .py), so the agent path errors until it’s repointed — and the Rust daemon hits the same dangling default, silently skipping the pre-flight. Both are flagged as code fixes below.