How it works

OpenLLM sits in one place: between the tool making a request and the provider that answers it. Everything else follows from that.

The request path

1. Your tool sends a normal OpenAI- or Anthropic-shaped request to your gateway, authenticated with your sk-llm-… key.
2. OpenLLM reads the model you asked for. If it is a tier alias like plus, it looks up your fallback chain to pick the first provider to try.
3. It unlocks the credential for that provider in memory, calls the provider with your own account, and streams the answer straight back to your tool.
4. If that provider fails, rate-limits, or runs out of context, OpenLLM moves to the next provider in the chain and retries, without dropping your request.
5. The request, its cost, and whether it fell back are recorded so you can see them in the dashboard.

Two kinds of credential, two paths

OpenLLM handles two very different things, and the difference matters for both security and compliance.

API keys (Anthropic, OpenAI, Gemini, Kimi, Alibaba, Bedrock, and similar) are stored encrypted in the gateway. On each request the gateway decrypts the key in memory, uses it, and wipes it. See Security.

Subscription accounts (Claude Code, ChatGPT, Kimi Code over OAuth) run through a small local daemon on your own machine, not the cloud. When a request needs one of these, it hops to your machine, runs against the official vendor CLI using its own credentials, and returns the result. Those subscription credentials never route through the OpenLLM cloud. See Providers and Compliance.

Why one endpoint helps

Because the surface is standard OpenAI and Anthropic, any tool that already speaks either protocol works without a plugin. You change one base URL and one key, and the tool is now routing through your accounts with fallback and a usage dashboard behind it. Nothing about your prompts or your code has to change.