> ## Documentation Index
> Fetch the complete documentation index at: https://www.bolna.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom LLMs for Bolna Voice Agents

> Point a voice agent at your own self-hosted or third-party OpenAI-compatible model, with your endpoint and your key.

Bring your own model by pointing Bolna at any endpoint that speaks the OpenAI chat completions API. Your endpoint can run anywhere: vLLM, Ollama, SGLang, TGI, a LiteLLM proxy, or a hosted gateway. Bolna calls it exactly as it calls OpenAI, only at your URL and with your key. Use this when you need a fine-tuned or open-weight model, when the weights have to stay on your own infrastructure, or when you already pay a provider directly and want the tokens billed there.

***

## Quick config

```json theme={"system"}
"llm_agent": {
  "agent_type": "simple_llm_agent",
  "agent_flow_type": "streaming",
  "llm_config": {
    "provider": "custom",
    "model": "Qwen/Qwen2.5-3B-Instruct",
    "base_url": "https://your-host/v1",
    "family": "qwen",
    "max_tokens": 150,
    "temperature": 0.2
  }
}
```

Register the model once under **Integrations → LLM → Custom LLM** in the dashboard, then select it on the agent's **LLM** tab. See [Using custom LLMs](/docs/customizations/using-custom-llm) for the walkthrough and the API.

***

## What your endpoint must support

<Note>
  Bolna sends requests with the OpenAI SDK, so the endpoint has to behave like OpenAI's chat completions API.
</Note>

| Requirement                        | Detail                                                                                                                                                                                                        |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `POST {base_url}/chat/completions` | Give Bolna the base URL including the version segment, for example `https://your-host/v1`. Bolna appends the rest of the path.                                                                                |
| Streaming responses                | OpenAI delta format, terminated by `data: [DONE]`. Tokens stream straight into speech synthesis, so an endpoint that only replies once the full answer is ready adds audible silence before the agent speaks. |
| Bearer authentication              | Your key is sent in the `Authorization` header.                                                                                                                                                               |
| A publicly resolvable host         | Private, loopback and link-local addresses are refused, both at registration and on every call.                                                                                                               |
| Streamed `tool_calls`              | Only if your agent uses tools. Support varies between self-hosted servers, so test it before relying on it.                                                                                                   |

Your model id is passed through unchanged, so it must match whatever your server expects.

***

## Key settings

| Setting           | Type    | Recommended                       | Description                                                    |
| ----------------- | ------- | --------------------------------- | -------------------------------------------------------------- |
| `provider`        | string  | `"custom"`                        | Provider name                                                  |
| `model`           | string  | e.g. `"Qwen/Qwen2.5-3B-Instruct"` | The model id your endpoint serves, passed through as-is        |
| `base_url`        | string  | e.g. `"https://your-host/v1"`     | Your OpenAI-compatible base URL, including the version segment |
| `family`          | string  | e.g. `"qwen"`                     | Model family, used for prompt formatting hints                 |
| `max_tokens`      | integer | `150`                             | Cap on response length, keep it short for voice                |
| `temperature`     | float   | `0.2`                             | Lower is more consistent                                       |
| `agent_flow_type` | string  | `"streaming"`                     | Always `"streaming"` for voice                                 |

***

## When to use a custom LLM vs a hosted provider

Use a **custom LLM** when:

* You run a fine-tuned or open-weight model that no hosted provider offers
* The weights or the traffic have to stay on your own infrastructure
* You already have a contract with a provider and want the tokens billed to it directly

Use a **hosted provider integration** when:

* You want a frontier model without operating an inference server
* You need Bolna's per-provider features, such as Azure data residency or provisioned throughput
* You would rather not own capacity planning, cold starts and upgrades

***

## FAQ

<AccordionGroup>
  <Accordion title="Does Bolna use the OpenAI Responses API for custom models?">
    No. Custom models always use chat completions, because the Responses API's server-side conversation chaining is specific to OpenAI and a self-hosted server cannot resolve a previous response id.
  </Accordion>

  <Accordion title="What happens if my endpoint has scaled to zero?">
    The first call pays the cold start, which the caller hears as silence before the agent speaks. Keep a warm instance if first-token latency matters. Registration tolerates an unreachable endpoint, so scaling to zero does not block you from adding the model.
  </Accordion>

  <Accordion title="Who bills me for the tokens?">
    Your provider. Bolna records a custom model's token usage against your own endpoint, so that model's inference is not billed by Bolna.
  </Accordion>

  <Accordion title="Can a sub-account have its own custom model?">
    Yes. A model added on the account is available to it and to all of its sub-accounts. A model added inside a sub-account stays in that sub-account, invisible to the main account and to sibling sub-accounts.
  </Accordion>

  <Accordion title="Can I point an agent at a model on a private network?">
    No. The endpoint must resolve to a public address. Expose it through a public hostname, with your key as the bearer token.
  </Accordion>
</AccordionGroup>

***

## Related

* [Using custom LLMs](/docs/customizations/using-custom-llm): register, validate, manage and delete a custom model
* [LLM Tab](/docs/agent-setup/llm-tab): configure LLM in the dashboard
* [OpenAI](/docs/integrations/llm-model/openai): direct OpenAI integration
* [OpenRouter](/docs/integrations/llm-model/openrouter): one key for many hosted providers
* [Prompting Guide](/docs/prompting/introduction): write effective prompts for voice