How it works
How OpenLLM routes a request through the hosted gateway or the daemon on your device, falls back along a chain, and what its servers can and cannot see.
Every AI tool wants its own key, base URL, and model names, and some models can only be reached from your own machine: a Claude or ChatGPT subscription runs through the vendor's CLI, and a local model server listens on localhost. OpenLLM puts all of them behind one endpoint and one key, and decides for each request where the work runs.
The three parts
- Hosted gateway (
https://www.openllm.sh/v1). It checks your key, resolves model names and chains, serves API-key and included models, records usage, and hosts the dashboard. - Local daemon. One per device, built into the
openllmbinary and run as a background service. It serves your subscriptions and local model servers, and listens onhttp://127.0.0.1:8787/v1by default. It keeps a connection to the gateway so the dashboard can manage it and the gateway can hand requests to it. - CLI and MCP server (
openllm). It installs the daemon, starts clients with OpenLLM applied, and runs an MCP server for your clients. On a Mac, the Mac app can set up the daemon instead. See Install.
The request path
- Check the key. Your tool sends an OpenAI- or Anthropic-shaped request with your
sk-llm-…key inAuthorization: Bearerorx-api-key. The key also unlocks your vault for this one request. - Resolve the model. A full id such as
provider/modelnames one model. An alias such asultra, or a chain you made in Configuration, expands to an ordered list. Models you can't use right now drop out: a model cooling down after a rate limit, or an included model whose monthly allowance is spent. - Choose where it runs. If the chain starts with a subscription or local model and one of your online devices can serve it, the gateway hands the whole request to that device. Otherwise the gateway runs it.
- Walk the chain. OpenLLM tries the first model. On a rate limit, an exhausted quota, an auth error, or a missing model, it moves to the next one. On a network or server error it retries the same model once first.
- Stream the answer back in your tool's format. The
x-openllm-resolved-modelheader names the model that answered, andx-openllm-chainlists the models tried. - Record usage. After the response, OpenLLM records one row of metadata: model, provider, token counts, latency, and status. When a device served the request, the device records it.
Hosted gateway and your devices
| Model source | Hosted gateway | Your device (daemon) |
|---|---|---|
| API keys | Runs the request | Sends the hop to the gateway; credentials are not decrypted on the device |
| Included models | Runs the request | Sends the hop to the gateway |
| Subscriptions | Never | Runs the request through the vendor's own CLI |
| Local model servers | Never | Runs the request against the server on the device |
The gateway never holds a subscription login or a local server's key, so those models are only available while one of your devices is online. When a hosted request's chain starts with one of them, the gateway hands the request to an online device that can serve it. The device runs the subscription and local models itself and sends any API-key models in the chain back to the gateway.
If no online device can serve those models, or the chain starts with an API-key model, the
gateway skips subscription and local models and continues with the rest of the chain. If nothing
is left, the request fails with 403 and
subscription_requires_daemon or
local_requires_daemon.
A client pointed at the daemon on its own machine skips the hand-off. openllm claude does this
whenever the daemon is running. The daemon asks the gateway for a routing plan, sending the model name and
a token estimate but not the prompt, and runs the plan locally. If the plan has no subscription or
local model, the daemon passes the whole request to the gateway instead.
Fallback
Fallback happens only before a response starts. Once a model begins streaming output, OpenLLM stays on it, and a failure after that point reaches your tool as an error. A request that every model would reject, such as a malformed or oversized one, fails on the first model without moving on. To edit chains and context-overflow behavior, see Routing and fallback.
What OpenLLM's servers see
| Data | What OpenLLM's servers see |
|---|---|
| API-key provider credentials | Ciphertext encrypted in your browser. Decrypted in memory for one request, then wiped. |
| Subscription logins | Never. They stay in the vendor CLI on your device. |
| Local model servers | Never their keys. Only the list of models an online device reports. |
| Recovery phrase | Never. |
| Vault key | Only wrapped copies at rest. Unwrapped in memory for one request, using your sk-llm key. |
Your sk-llm keys | Each request's key in its header. At rest, only a copy encrypted under your vault key. |
| Prompts and completions on hosted requests | In memory while forwarding. Not logged, except a response cached for 10 minutes when you send an Idempotency-Key. |
| Prompts sent straight to your daemon that run on a subscription or local model | Never. The prompt goes from your device to the vendor or the local server. |
| Request metadata | Stored: model, provider, token counts, latency, status, key id, and time. |
- Recovery phrase: the servers never see it
- Encrypts provider credentials with the vault key
- Unlocks with a PIN or passkey
- Stores ciphertext and wrapped copies of the vault key
- Per request: your sk-llm key unwraps the vault key in memory, one credential is decrypted, then both are wiped
- Prompts pass through in memory; not logged
- Stores request metadata
- Subscription logins stay in the vendor CLI
- Local server keys stay on the device
- Runs subscription and local models
OpenLLM does not use your prompts or completions to train models. Retention periods are on Privacy.
Limitations
- Subscription and local models need one of your devices online with the daemon running.
- After a model starts streaming, there is no fallback.
- Your
sk-llmkey carries a wrapped copy of your vault key, so anyone holding it can decrypt your stored provider credentials. If a key leaks, delete it in Devices & keys. See If a key leaks. - Hosted requests pass through OpenLLM's servers in memory on the way to the provider.
Further reading
- Security and vault: how the vault, keys, and passkeys work.
- Privacy: what OpenLLM stores and for how long.
- Routing and fallback: chains, aliases, and model settings.
- Devices: add, manage, and remove devices.
Last updated on
Quickstart
Create an OpenLLM account, set up your vault, copy your key, send your first API request on the included models, and run Claude Code through OpenLLM.
Install OpenLLM
Install the openllm CLI and daemon on macOS or Linux with one curl command, or install the Mac app, then check, update, or uninstall OpenLLM.