Use local models

Use models that run on your own hardware alongside your other providers. The daemon finds OpenAI-compatible model servers on the device and lists their models under your account. Available on every plan. You need the daemon running on the machine that runs the model server. See Devices.

Supported servers

The daemon checks these ports on 127.0.0.1:

A model id is the prefix plus the model name the server reports, for example local_ollama/llama3.2. llama.cpp and LocalAI share port 8080, so the daemon tells them apart by the server's response. A server it can't identify, and any server on a port you add yourself, gets the prefix local_port_<port>/, for example local_port_8080/ or local_port_9000/.

Make a server discoverable

1. Start the model server on the same machine as the daemon, on its default port.
2. Make sure the server answers GET http://127.0.0.1:<port>/v1/models with a list of models.
3. Open Providers and select the Local tab. If you have more than one device, select the device.
4. Select Rescan, or wait. The daemon checks again every 60 seconds and when it starts.
5. Check the server's row. Reachable with a model count means its models are ready to use.

The daemon only probes 127.0.0.1. It doesn't find servers on other machines on your network.

Use a different port

To add ports beyond the defaults, list them in OPENLLM_LOCAL_PORTS in the daemon's settings file, ~/.openllm/.env, then restart the daemon:

OPENLLM_LOCAL_PORTS=9000,9001

openllm restart

Models on those ports get the local_port_<port>/ prefix. There is no setting for this in the app.

Add a server API key

If the server asks for a key, its row shows Needs API key.

1. On the server's row, enter the key in Optional API key.
2. Select Set. The row shows Key set.

Your browser encrypts the key for that device, so it isn't readable on the way. The daemon saves it in ~/.openllm/.env on the device, and it never reaches OpenLLM's servers. To remove it, select the clear button next to the field.

Send requests to a local model

You can reach local models two ways.

From the daemon on the same machine. Point your tool at http://127.0.0.1:8787/v1. The daemon serves local models itself and can use them anywhere in a fallback chain. It only listens on 127.0.0.1 and doesn't check for an API key, because the person calling it owns the machine. The openllm client launchers use this endpoint. See Clients.

curl http://127.0.0.1:8787/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "local_ollama/llama3.2", "messages": [{"role": "user", "content": "Hi"}]}'

From the hosted gateway. Send the request to https://www.openllm.sh/v1 with your sk-llm key. When the first model in the chain is a local model, the gateway hands the request to an online device that lists that model. It tries the device the calling key belongs to first. The device then runs the chain. When the chain starts with any other model, the gateway skips local models. Embeddings, images, audio, and video never reach local models through the gateway.

Local models appear in GET /v1/models only while their device is online. When the device goes offline, they drop out of the list, and requests that need them fail with 403 local_requires_daemon unless another model in the chain can answer.

On the Free plan, local chat requests count toward the monthly token cap like any other non-included model. See Plans and limits.

Use a local model in a client

The openllm launchers use the daemon when it runs on the same machine, so a client started with openllm <client> can use local models anywhere in a chain. The daemon serves local models over the Chat Completions, Messages, and Responses APIs, so Claude Code and Codex work with them as they are.

To pick a local model, pass its full id:

To keep a client on an alias instead, add the local model to that alias's chain in Configuration, for example at the top of lite. See Fallback chains.

Coding agents send long prompts with tool definitions. Choose a local model whose context window and tool support fit the agent. When a request is too large for the model, the context overflow setting decides what happens.

Check that it works

1. On the Local tab, confirm the server shows Reachable and a model count.
2. Call GET https://www.openllm.sh/v1/models and look for ids that start with local_.
3. Send a request to one of them through the daemon, as above, and check the x-openllm-resolved-model response header.

Common problems

For more, see Troubleshooting.

Related

Last updated on