Routing and fallback

Providers fail in ordinary ways: a subscription hits its limit, a key gets rate-limited, a model is down for a minute. OpenLLM lets you name an ordered list of models under one alias, so a request that fails on one model moves to the next before your tool sees an error. This page explains how that works and how to configure it in Configuration.

Model ids

Every model has an id in the form provider/model, such as openai/gpt-5, claude_code/claude-opus-5-5, local_ollama/llama3.2, or custom:home-lab/qwen3-coder. To list the ids available to you, call GET https://www.openllm.sh/v1/models.

What you put in the model field decides what runs:

Aliases

An alias is a name for a fallback chain. Every account starts with three built-in aliases:

After the included models, each built-in chain lists suitable models from the providers you've connected. Subscription models come before the same models on an API key. Until you edit them, the built-in chains update on their own when you connect or remove a provider. On Configuration they carry an Auto default badge.

• Edit a built-in chain and OpenLLM saves your version as your own chain. It stops updating automatically.
• Delete a built-in chain and OpenLLM remembers it. It doesn't come back when your providers change.
• Create your own alias with any name. It works exactly like the built-in ones.

Fallback chains

OpenLLM uses at most the first 16 models in a chain and drops repeats, so each model appears once. A model can still be called twice in one request when OpenLLM retries it after a connection or server error.

Create or edit a chain

1. Open Configuration and select the Fallback chains tab.
2. To create a chain, type a name in the new-alias field and select New chain, or select a suggestion next to quick start.
3. Add models: drag them from Available models, or use + Add a fallback model… at the bottom of the chain. The picker lists only models you can use now.
4. Drag models to reorder them. The top model is tried first.
5. To remove a model, select Remove from chain. To delete the whole chain, select Remove chain.
6. Select Save changes in the bar at the bottom. Chains, model settings, and the search provider save together. Discard drops your edits.

There's no CLI command for editing chains.

Rate limits (429), overloads and errors: what moves a request on

OpenLLM moves on only while nothing has been sent to your tool. Once a model starts producing output, OpenLLM stays on it. A failure after that point reaches your tool as an error.

A 400 stops the chain because every model would reject the same malformed request. Fix the request and retry. If every model fails, you get the last model's error. See Errors.

Cooldowns

A model in cooldown is left out of later requests until the cooldown ends. On the hosted gateway, cooldowns apply per sk-llm key, so one client's rate limit doesn't affect another client using a different key. If every model in a chain is in cooldown, OpenLLM tries the full chain anyway.

The daemon also keeps its own cooldown list for the models it calls. That list covers every client on the device, whatever key it uses, and it never skips the last model in a chain.

When a subscription reports a usage window as used up, the daemon also skips it until that window resets. See Subscriptions.

Examples

A subscription runs out. Your plus chain is claude_code/claude-opus-5-5, then anthropic/claude-opus-5-5, then openllm/qwen/qwen3.7-plus. Claude Code calls the daemon. Your Claude Max window is used up, so the first model returns a usage-limit error. The request moves to your Anthropic API key and succeeds. For the next 3 minutes, and until the window resets, requests on that key start with the API key.

A provider is down. Your chain is openai/gpt-5, then google/gemini-3.5-flash. OpenAI returns 503. OpenLLM retries openai/gpt-5 once, gets 503 again, and moves to Gemini. The response carries x-openllm-chain: openai/gpt-5,openai/gpt-5,google/gemini-3.5-flash and x-openllm-attempted-fallbacks: 2.

A request is malformed. A tool sends a request with an invalid parameter, and the first model returns 400. OpenLLM returns that error right away, without trying the other models.

Where the chain runs

Subscription and local models only run on your devices. When a request comes to the hosted gateway, the first remaining model in the chain decides where it runs:

• If it's a subscription or local model, the gateway hands the whole request to an online device that can serve it. The device runs the chain in order and sends API-key and included models back through the gateway.
• If it's any other model, the gateway runs the chain and skips every subscription and local model, even when a device is online.

A request sent straight to the daemon at http://127.0.0.1:8787/v1 uses every model in the chain in order. See Which source serves a request.

Pin a model

To send a request to exactly one model, set the x-openllm-pin-model request header:

curl https://www.openllm.sh/v1/chat/completions \
  -H "Authorization: Bearer $OPENLLM_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-openllm-pin-model: anthropic/claude-opus-5-5" \
  -d '{"model": "plus", "messages": [{"role": "user", "content": "Hi"}]}'

The pin replaces whatever the model field says. The request runs on that one model with no fallback.

Sending a concrete id in model has the same effect. Use the header when you can't change the model field, for example when a tool always sends an alias. A common pattern is to read x-openllm-resolved-model from the first response in a conversation and pin later turns to it, so the conversation stays on one model.

Response headers

Context overflow

This setting decides what happens when a request is larger than a model's context window. It's the switch at the top of the Fallback chains tab in Configuration. Switched on, it reads Hop to larger context. Switched off, it reads Compact in place. Select Save changes after changing it.

Hop to larger context (default)

1. Before calling a model that isn't last in the chain, OpenLLM estimates the request size. If it's clearly larger than that model's window, OpenLLM skips the model.
2. If a model rejects the request as too long, OpenLLM jumps to the next model in the chain whose window is large enough.
3. If no model fits, OpenLLM shortens the request as a last resort and retries on the model with the largest window, up to 3 times.

Compact in place

1. OpenLLM doesn't skip models for size.
2. If a model rejects the request as too long, OpenLLM shortens the request to fit that model and retries the same model, up to 3 times.
3. It never moves to a different model because of size. If shortening isn't enough, you get the provider's error.

When OpenLLM shortens a request, it first trims the middle of older tool outputs, replacing the cut with …[openllm: output truncated to fit context]…. If that isn't enough, it drops the oldest messages. It never changes the system prompt, your latest message, or messages marked for prompt caching. Shortening changes what the model sees, so most tools work better when they compact their own history. OpenLLM reports each model's context window in GET /v1/models so tools can do that.

Model settings

Some models have settings you can set once for every request that reaches them, including through fallback chains. Open Configuration and select the Model settings tab. Only models with something to configure are listed.

Priority processing can cost more on OpenAI, and on Codex it uses more of your ChatGPT allowance. Context window sets the size OpenLLM uses for its context checks and Context overflow. It doesn't change what the provider accepts.

When more than one value applies, the first one wins:

1. The value in the request, such as "service_tier": "priority" in the body. This applies to service_tier only; there's no request field for the context window.
2. Your setting on Model settings.
3. The model's default. For the context window, that's the latest window the provider reported, or the catalog value if there's no report.

Select Save changes after editing. If a model you configured disappears from your catalog, its setting is kept under Unavailable models.

Limitations

• A fallback can't happen after a model starts streaming output.
• The hosted gateway uses subscription and local models only when they come first in the chain.
• Shortening a request removes content the model would otherwise see.

Further reading

Last updated on