Classifying failures before deciding how to recover from them

The problem with retrying everything the same way

A naive retry policy treats every provider failure the same: something went wrong, wait a moment, try again. That's the wrong response for most of the ways a request to a hosted model actually fails. A rate limit is genuinely transient: the same request will very likely succeed seconds later. An authentication failure won't fix itself no matter how many times it's retried; it needs a different credential. A context-window overflow won't succeed on retry either, because the input that broke it hasn't gotten any smaller. Retrying the second and third case wastes the request budget and delays the failover that would have actually worked, and failing over immediately on the first case abandons a request that a short backoff would have rescued.

One router, a wide provider list

Synapse's router sits in front of How I measured LLM providers routed: 17Methodprovider keys registered in _KEY_MAP in workspace/sci_fi_dashboard/llm_router.py: anthropic, openai, gemini, groq, openrouter, mistral, togetherai, xai, cohere, minimax, moonshot, zai, volcengine, huggingface, nvidia_nim, qianfan, deepseek. Routed through litellm.Router with a per-role fallback table; Bedrock, Vertex, Ollama and OAuth paths are additional and not counted hereDate2026-08-11Sourcegithub.com/UpayanGhosh/Synapse-OSS/blob/main/workspace/sci_fi_dashboard/llm_router.py, each registered by its own environment-variable key and dispatched through litellm.Router with a fallback table keyed by request role. Synapse's own README describes this more conservatively, as six providers: that's not wrong, it's just the smaller, curated list a reader would notice first. The router's own key map is the larger, literal count, and I've cited that one here specifically because it's the number a reader can count directly against the source rather than take on faith.

Classifying the failure before choosing the response

The router sorts an incoming provider error into roughly seven recovery paths before deciding what to do next. A context-window overflow triggers a compaction pass and a retry. A rate-limit response triggers exponential backoff on the same provider rather than an immediate failover. An authentication failure rotates to the next configured credential and retries. A model-not-found error gets exactly one fallback attempt and no retry loop, because retrying a request against a model that doesn't exist is never going to succeed. A malformed request or a billing failure raises immediately, because no amount of retrying or rerouting fixes a request that was wrong to begin with.

def recover(error: ProviderError) -> Recovery:
    if is_context_overflow(error):
        return Recovery.COMPACT_AND_RETRY
    if is_rate_limit(error):
        return Recovery.BACKOFF_AND_RETRY
    if is_auth_failure(error):
        return Recovery.ROTATE_CREDENTIAL_AND_RETRY
    if is_model_not_found(error):
        return Recovery.FALLBACK_ONCE
    if is_format_or_billing(error):
        return Recovery.RAISE
    return Recovery.FALLBACK_ONCE

Simplified, but the shape is real: the classification happens once, up front, and every recovery path downstream of it is a direct consequence of what kind of failure this was, not a single generic retry loop with exceptions bolted on.

The honest cost

Every one of those classifications runs as substring matching against the provider's own error text, not against a structured error code. That's a real, specific fragility: providers don't publish a stable contract for their error strings, so a wording change upstream (a provider rephrasing its rate-limit message, say) can misclassify a failure the router used to handle correctly, silently, until something downstream notices the wrong recovery path firing. Some providers behind this router do expose structured error types; not all seventeen do, and not consistently enough to build the classifier against structured codes alone without leaving gaps for the providers that don't. Substring matching was the pragmatic choice across a provider list that wide, and it is a choice with a real, ongoing cost, not a solved problem I'm rounding up to sound finished.

Reproducing this

The provider key map and the router's fallback table live in workspace/sci_fi_dashboard/llm_router.py on the Synapse-OSS repository.

The full case study is at /work/synapse.