Health System — The Brain

Overview

health_check.py pings every registered provider/model on an hourly schedule (Windows Task Scheduler). Results are appended to stats/health_log.json — never overwritten or averaged. Model state (active/paused/disabled) is written to stats/model_state.json and read by the orchestrator at routing time.

Probe Design

Each model receives a randomized math question (e.g. "What is 7+7? Reply with only the number.") from a pool of 10 prompts. The expected numeric answer is checked in the response. Using a varied prompt pool prevents cached responses from masking a broken model.

Models of the same provider are spaced 90 seconds apart to avoid triggering rate limits mid-cycle.

Model State Machine

State	Routing	Health checks	Transition
`active`	Included	Tested every cycle	→ paused on temporary failure; → disabled on 3+ hard failures
`paused`	Excluded	Tested every cycle	→ active automatically on first passing probe
`disabled`	Excluded	Skipped entirely	Manual edit of `stats/model_state.json` to re-enable

Failure Classification

failure_type	Cause	Action
`timeout`	Request did not complete within deadline	Pause model
`rate_limit`	HTTP 429 or "rate" in error message	Pause model
`http_error`	HTTP 5xx server error	Pause model
`wrong_answer`	Response succeeded but answer was incorrect	Increment failure count; disable after 3
`error`	Any other exception	Increment failure count; disable after 3

Log Format — stats/health_log.json

Append-only JSON array. One entry per model per check cycle:

{
  "ts":           "2026-05-02T14:00:00Z",
  "provider":     "mistral",
  "model":        "mistral-small-latest",
  "ok":           1,
  "latency_ms":   639,
  "failure_type": ""
}

Model State File — stats/model_state.json

{
  "mistral::mistral-large-latest": {
    "state":                "paused",
    "reason":               "timeout",
    "since":                "2026-05-02T15:00:00Z",
    "consecutive_failures": 1
  }
}

Effect on Routing

BaseAdapter.list_models() filters out paused and disabled models before returning the model list to the orchestrator. Paused/disabled models are never chosen for a delegate call — no routing code change required.

session_start.py Integration

session_start.py reads health_log.json and prints a 24h uptime/latency summary at session start. It also reads model state and emits routing advisories for any provider with uptime below 75% or a last status of error.

health_rollup.py

brain/health_rollup.py aggregates raw log entries into per-provider uptime, avg_latency_ms, and sample_count for the README health block and session briefing. Handles both the current format (ok field) and the legacy averaged format for backward compatibility.

Manual Invocation

python health_check.py