Last updated
What is a custom LLM used for in a Bolna voice agent?
A custom LLM lets a Bolna voice agent run on any endpoint that speaks the OpenAI chat completions API, whether that is an inference provider or a server you run yourself. Bolna calls it the same way it calls OpenAI, only at your URL and with your key. Use it for open-weight or fine-tuned models, very fast inference hardware, or traffic that has to stay on infrastructure or in a region you control.
How do I set up a custom LLM with Bolna?
- Check that your endpoint serves
POST {base_url}/chat/completionswith streaming responses ending indata: [DONE], and acceptsAuthorization: Bearer <key>. - In the Bolna dashboard, go to Settings → Integrations, filter by LLM and click Add model on the Custom LLM card. You can also pick + Add custom LLM from the LLM Provider dropdown on an agent's Intelligence tab.
- Pick a Provider preset, or Other (OpenAI-compatible) for any other server. The preset only changes the hints in the dialog.
- Enter the Model name your endpoint serves (for example
Qwen/Qwen2.5-3B-Instruct), the Endpoint URL without a trailing slash (usually ending in/v1), the API key and the Family (for exampleqwen,llamaormistral). - Save. Bolna checks the endpoint, stores the key encrypted and shows the model as its own card.
- On the agent's Intelligence tab choose custom, pick the model, save and place a web call before going live.
The custom LLM setup guide in the Bolna docs covers the save-time checks, extra request params and the API.
Which inference providers have presets in Bolna?
| Provider | Best for | Endpoint URL |
|---|---|---|
| Baseten | Model APIs, or your own Truss deployment on dedicated GPUs | https://inference.baseten.co/v1 |
| Cerebras | Wafer-scale hardware, around 3,000 tokens/s on gpt-oss-120b | https://api.cerebras.ai/v1 |
| Fireworks AI | Serverless open models plus on-demand GPU deployments | https://api.fireworks.ai/inference/v1 |
| Groq | Fast LPU inference for Llama and GPT-OSS | https://api.groq.com/openai/v1 |
| Modal | Your own vLLM server on serverless GPUs, in a region close to your callers | The URL modal deploy prints, plus /v1 |
| Nebius | Token Factory models, with dedicated endpoints in a region you choose | https://api.tokenfactory.nebius.com/v1 |
| Simplismart | Optimised open models on shared or dedicated endpoints | https://api.simplismart.live |
| Together AI | A large serverless catalog of open models | https://api.together.ai/v1 |
Anything else that is OpenAI-compatible, such as vLLM, SGLang, TGI, Ollama or LiteLLM, goes under Other (OpenAI-compatible).
What can you build with a custom LLM and Bolna?
Agents with the shortest pause before replying. The Bolna docs point to Groq or Cerebras when the silence before the agent speaks matters most.
Agents on a fine-tuned or private model. Serve your own weights on Baseten dedicated deployments, Fireworks AI on-demand GPUs, or your own vLLM server on Modal.
Agents with data residency needs. Keep inference in a fixed region with Nebius dedicated endpoints, Simplismart dedicated deployments or Modal's routing_region.
Multi-tenant setups. A model added on the main account is available to all sub-accounts, while a model added inside a sub-account stays there. If you would rather not run inference at all, compare a hosted option such as OpenRouter and check pricing.