Your backup models may all be dead — and nothing tells you until the outage


It is 2am and the primary model starts answering 429. You are not worried — you set up a fallback chain months ago: another provider first, a local model after that. Hermes tries both and both fail, and the task dies in the middle. In the morning you open the config and see it: the first provider’s key was rotated last month, and the second provider renamed its model two releases ago. The chain was still there. It had simply never been checked.

That is not carelessness. It is what “backup” always does: nothing exercises it, so nothing notices when it rots, until the one minute you need it.

From “should work” to “does work”

hermes fallback check is a new subcommand that answers one question: can each entry in your chain actually be used right now? It does not start an agent session and does not spend conversation context.

hermes fallback check          # config and credentials only
hermes fallback check --live   # send one minimal request per entry
hermes fallback check --json   # structured output for scripts

If any entry is not ready, the command exits with status 1. That is the whole point: you can wire it into cron, into CI, into a pre-deploy checklist — instead of finding out at 2am.

Start by looking at what you have:

hermes fallback            # same as hermes fallback list

The chain lives in config.yaml under fallback_providers, an ordered list of {provider, model, base_url?, api_mode?} entries. The older fallback_model key is merged in automatically, so there is one source of truth. hermes fallback add uses the same picker as hermes model.

The static check: what the switch itself will see

Without --live, the command resolves every entry with the same kwargs the fallback switch uses when it fails over. Three things come out of that.

Does the provider exist? A misspelled provider is a config error, not a credential problem — the report keeps those apart, because the fix is different.

Are there usable credentials? This asks the question the switch asks: would it skip this entry for lack of auth? Using the same predicate means the check cannot say “ready” about an entry the real failover would skip.

Do the resolved endpoint and wire protocol look right? base_url and api_mode are printed, so an entry still pointing at an endpoint you migrated away from is visible at a glance.

The output is a table:

  #  Provider / model                            Config  Auth    Live    Latency   Status
  1  openrouter / gpt-x                         ok      ok      ok      210 ms    ready
  2  anthropic / claude-sonnet                  ok      FAIL    —       —         auth
  3  local / qwen3:8b                           ok      ok      ok      88 ms     ready

The three columns are independent. Config failing means the provider is unknown. Auth failing means the provider is fine but there is no credential for it. In static mode the Live column is a dash, because no inference request was sent.

--live: actually say something

A static check catches “not configured”. It cannot catch “configured and broken” — exhausted quota, a suspended account, a retired model name, a 502 behind the endpoint. Only a real request tells you that.

--live sends one minimal completion to every entry that passed the static check: the prompt Reply with exactly: OK, at most 512 tokens, the provider’s own request timeout (30 seconds if none is known), with SDK-level retries disabled. Entries are probed in parallel, so the wall-clock cost is roughly the slowest one.

One design decision is worth knowing: the probe deliberately does not go through call_llm. call_llm has a recovery ladder — it retries, and it can answer from another provider — which would hide exactly the thing you are trying to find out. A health check wants the bad news, not a rescued one.

Verdicts fall into a few classes:

  • ready — both stages passed.
  • auth — no usable credential; the switch would skip it too.
  • live_* — the request failed, with the classifier’s reason as the suffix (rate limit, timeout, auth, model not found, and so on).
  • live_response — HTTP 200 with no usable completion inside: a failed status, or partial text with nothing behind it.
  • model_mismatch — something answered, but not the model you asked for.

That last one is more common than it sounds. Plenty of aggregator gateways quietly substitute a different model when the requested one is unavailable; the request succeeds and the quality silently drops. The check compares what you requested with what the provider says answered: a dated snapshot (gpt-4o → gpt-4o-2024-08-06) counts as a match, a -latest pointer counts, a :tag must match. And where the reported model is not evidence at all — local endpoints, Azure’s deployment-vs-base-model naming, router meta-models like openrouter/auto — it does not pretend to know.

Reading the JSON, gating a pipeline

Humans read the table; scripts read the JSON:

hermes fallback check --live --json > fallback-report.json

The shape, per entry:

{
  "profile": "default",
  "live": true,
  "ready": false,
  "checked_at": "2026-11-17T03:12:44+00:00",
  "results": [
    {
      "provider": "openrouter",
      "model": "gpt-x",
      "base_url": "https://openrouter.ai/api/v1",
      "api_mode": "chat_completions",
      "auth_source": "env",
      "configured": true,
      "auth_ready": true,
      "inference_ready": true,
      "latency_ms": 210,
      "model_returned": "gpt-x-2026-09-01",
      "error_class": "",
      "error": "",
      "ready": true,
      "status": "ready"
    }
  ]
}

Error text is redacted: known credential values are replaced literally first, then the support redactor runs. So the file is safe to paste into a ticket without leaking a key by accident.

Wiring it up is then trivial:

# every morning at 8: fall out loud instead of failing quietly
hermes fallback check --live || hermes send --to ops "fallback chain is not ready"

Or gate a release on it. A chain that is down should stop the deploy, not surface as the first incident afterwards.

Edges worth knowing

auto and moa stay static. auto picks its backend per call and can walk the chain itself; moa is a fan-out preset that only an agent can bind. One request from a health check proves nothing about either, so they report ready (static) and are never probed.

--live costs money. One minimal request per entry — tiny, but not zero. On a long chain checked very often, consider running the live probe only when it matters.

Passing means “right now”. Quotas, rate-limit windows and model availability move. Treat the check as a physical, not insurance.

With no chain configured, you get No fallback providers configured. Add one with: hermes fallback add rather than an empty table.

If you have not built a chain yet, start with how Hermes layers fallbacks, then use the hermes fallback reference to add entries, then run hermes fallback check --live. Five minutes of work buys you the one thing a backup is for.

The short version

A fallback chain does not usually break when you configure it. It breaks during the months nobody looks at it. hermes fallback check closes that gap: static checks for config and credentials, --live for real availability, an exit code automation can act on. Run it after every hermes update — cheaper than any amount of “it should be fine”.

Full release notes: v0.23.2.