The Smart Router Cathedral

The Smart Router started as a glorified match statement — model field in, backend out. It ended as a cathedral. Five tiers of defense between a client request and a model response, each tier addressing a specific failure mode the system used to quietly absorb.

This page is the canonical reference. If you want a five-minute tour instead of a first-principles read, skip to the matrix at the end.

Tier 0 — Routing Correctness

The model field the caller sends is a nickname. The upstream needs an ID. sanctum-mlx-devstral doesn’t accept coder; the Gemini :6543 proxy doesn’t accept spatial; Anthropic doesn’t accept cloud. Tier 0 added default_model per backend in instance.yaml so every outbound request carries the canonical upstream ID:

coder:
  url: http://127.0.0.1:3301/v1
  default_model: Devstral-Small-2-24B-Instruct-2512-4bit
cloud:
  url: http://127.0.0.1:3456/v1
  default_model: claude-opus-4
spatial:
  url: http://127.0.0.1:6543/v1
  default_model: gemini-3.1-pro-preview

Without this, every non-council backend 404’d. With it, routing is actually correct.

Tier 1 — Four Backends, End to End

Four production backends wired through sanctum-server on 127.0.0.1:8055:

Backend	Upstream	Route patterns	Auth
council-secure	Qwen3.6-35B-A3B on :1337 mTLS	yoda, mothma, windu, cilghal, mundi, jocasta, quigon	per-client cert
coder	Devstral-Small-2-24B-Instruct-2512-4bit via sanctum-mlx-devstral on :3301 (native MLX seat)	coder, code-*, *coder*, ahsoka	loopback
cloud	Claude Opus (Max subscription — the unpinned claude-opus-4 alias serves the latest Opus, 4.8 today) via claude-max-api-proxy on :3456 (runs on the Mini; the MBP reaches it over a persistent SSH tunnel on :3457)	opus, claude-*, escalation	Max OAuth (proxy-mediated)
spatial	Gemini 3.1 Pro via the local :6543 OAuth proxy (AI Ultra subscription — the metered API key is reserved for deliberate multimodal work)	gemini, windu-spatial, spatial	AI Ultra OAuth (proxy-mediated)

Each backend declares HA fallback_urls. cloud has exactly one fallback — OpenRouter — for when the Max proxy is unreachable; council-secure falls over from Mini mTLS to the MBP shadow seat, then to a plain-HTTP loopback as a last resort.

Note

Ahsoka is a reserved seat for the 16 GB M1 at the chalet — node offline, agent not yet deployed. Her pre-wired route points at coder rather than council-secure because the 35B doesn’t fit in 16 GB. The router is size-aware via config, not convention.

Tier 2 — Observability

/metrics serves Prometheus text at :8055/metrics. Every request, every decision, every backend call is a counter or histogram:

• sanctum_server_http_requests_total{route,method,status} — RED at the edge
• sanctum_server_backend_requests_total{backend,outcome} — RED per backend, outcome in ok / connect_error / parse_error / breaker_open / budget_exhausted / quality_issue. (A 4xx never reaches this counter — a bad request is the caller’s bug, not the backend’s, so it bails before recording.)
• sanctum_server_routing_decisions_total{from_model,to_backend,rule} — which Jedi went where, via which tier
• sanctum_server_backend_duration_seconds — latency histogram for p50/p95/p99 PromQL
• sanctum_server_backend_tokens_in_total / _out_total — what flowed
• sanctum_server_backend_fallback_total — when HA kicked in

Structured JSON logs (SANCTUM_LOG_FORMAT=json) carry a per-request request_id via axum middleware. Inbound X-Request-ID is honoured; otherwise a UUIDv4 is minted and echoed on the response. jq '.span.request_id=="…"' pulls a whole request’s trace.

The command center dashboard’s RouterPanel scrapes /api/router/status every 10 s and renders the backend and decision tables live. (sanctum-server itself only speaks /v1/*, /health, and /metrics; the dashboard backend on :1111 folds the raw Prometheus text into that JSON.)

Tier 3 — Self-Healing via State

Three mechanisms that watch for known-broken states and route around them:

Circuit breaker. Per backend. Closed → Open after 5 consecutive failures → HalfOpen after 30 s → Closed on canary success or Open again on canary failure. Lock-free on the hot path (atomic u32 failure counter); mutex only on state transitions. Complementary to HA fallback: HA handles per-URL failures within a request, the breaker handles sustained per-backend outages across requests.

Breaker-aware routing. Router::select skips backends whose breaker is Open at every tier (direct, pattern, intent). Default backend is always returned as a last resort so callers get a deterministic error instead of hanging.

Token budget. Per backend, per day. daily_cap_completion_tokens in instance.yaml caps billable output; when exhausted, Budget::allow() returns false and the router skips the backend exactly the way it skips Open breakers. The counter resets on a rolling 24-hour window measured from process start — not wall-clock midnight — and isn’t persisted across restarts, so a restart both zeroes the count and resets the clock. Good enough for a single-binary deployment that spends money in completion tokens, not in clever bookkeeping.

Tier 4 — Self-Healing via Speed

Backends that answer every request correctly but in 45 s instead of 3 s are degraded, not broken. Tier 4 detects and trips on them.

Latency EMA. Exponential moving average (α = 0.2, ~5-sample half-life) of completed-request seconds per backend. AtomicU64-packed f64 bits for lock-free reads. Exposed as sanctum_server_backend_latency_ema_seconds{backend} gauge and rendered in the Router panel’s EMA column.

Slow-backend trip. When the EMA crosses a configurable slow_threshold_s for slow_trip_count consecutive successful requests (default 3), the backend’s breaker gets a synthetic failure. The standard Open → HalfOpen → Closed cycle handles the rest.

cloud:
  url: http://127.0.0.1:3456/v1
  slow_threshold_s: 15.0   # Opus (Max, latest) p95 normal ~3s; 15s = broken
  slow_trip_count: 3

Tier 5 — Self-Healing via Quality

The final tier handles a failure mode that HTTP can’t see: the backend returned 200 OK with a response that’s garbage. Four detectors, each one earned by a pathology we actually saw in production:

Detector	Verdict	Trips breaker	Example
empty_content	Broken	yes	Google AI Studio at low max_tokens — “done” with zero tokens emitted
pure_repetition	Broken	yes	Qwen3.5 conv_state bug — “190/ 190/ 190/” for 80 tokens
think_tag_leak	Suspicious	no	Reasoning model leaked <think> past the stream filter
truncated_tiny	Suspicious	no	finish_reason=length and content < 4 bytes

Every successful response accumulates its content across the stream and runs the detectors at completion. A Broken verdict calls breaker.record_failure() and flips the metric outcome to quality_issue — one strike toward the breaker’s five-failure threshold. The current caller still sees whatever was streamed; if the garbage keeps coming, the breaker opens and everyone after gets routed away. The first users are the canary; the breaker is the guarantee.

Tip

Sophisticated scoring — LLM-as-judge, logprobs, semantic similarity — is Tier 6 R&D territory. Tier 5 is the cheap stuff that catches 80% of real-world degeneracy with one regex and a byte-window scan.

The five-tier matrix

Tier	Watches for	Signal	Action
0	wrong model ID	config	pin default_model
1	backend dead	HA fallback_urls	try next URL
2	—	/metrics + JSON logs	observe
3	sustained backend failure	error rate (breaker) / budget cap	route around — breaker opens, or budget skips
4	degraded backend (slow)	latency EMA > threshold	trip breaker → route around
5	garbage output	sanity verdict = Broken	trip breaker → route around

Thirteen stones across five tiers. 48 tests. The cathedral routes, observes, recovers, and reroutes on its own. The caller just sends {"model":"yoda"} and gets an answer — from wherever the answer is currently cheapest, fastest, and least likely to be nonsense.