Custom LLMs for Bolna voice agents

Bring your own model by pointing Bolna at any endpoint that speaks the OpenAI chat completions API. Your endpoint can run anywhere: vLLM, Ollama, SGLang, TGI, a LiteLLM proxy, or a hosted gateway. Bolna calls it exactly as it calls OpenAI, only at your URL and with your key. Use this when you need a fine-tuned or open-weight model, when the weights have to stay on your own infrastructure, or when you already pay a provider directly and want the tokens billed there.

Quick config

"llm_agent": {
  "agent_type": "simple_llm_agent",
  "agent_flow_type": "streaming",
  "llm_config": {
    "provider": "custom",
    "model": "Qwen/Qwen2.5-3B-Instruct",
    "base_url": "https://your-host/v1",
    "family": "qwen",
    "max_tokens": 150,
    "temperature": 0.2
  }
}

Register the model once under Settings → Integrations → LLM → Custom LLM in the dashboard, then select it on the agent’s Intelligence tab. See Add a custom model in the dashboard for the walkthrough and Add a custom model over the API for the API.

What your endpoint must support

Bolna sends requests with the OpenAI SDK, so the endpoint has to behave like OpenAI’s chat completions API.

Requirement	Detail
`POST {base_url}/chat/completions`	Give Bolna the base URL including the version segment, for example `https://your-host/v1`. Bolna appends the rest of the path.
Streaming responses	OpenAI delta format, terminated by `data: [DONE]`. Tokens stream straight into speech synthesis, so an endpoint that only replies once the full answer is ready adds audible silence before the agent speaks.
Bearer authentication	Your key is sent in the `Authorization` header.
A publicly resolvable host	Private, loopback and link-local addresses are refused, both at registration and on every call.
Streamed `tool_calls`	Only if your agent uses tools. Support varies between self-hosted servers, so test it before relying on it.

Your model id is passed through unchanged, so it must match whatever your server expects.

Add a custom model in the dashboard

Open Integrations

Go to Settings → Integrations in the dashboard and filter by LLM. Find the Custom LLM card and click Add model.

Integrations page filtered to LLM providers, showing the Custom LLM card with an Add model button — The Custom LLM card on Settings → Integrations, under the LLM filter

Fill in the details

Model name: the model id your endpoint serves, for example Qwen/Qwen2.5-3B-Instruct
Endpoint URL: your OpenAI-compatible base URL, including /v1
API key: the bearer token your endpoint expects
Family: the model family, for example qwen, llama or mistral

Add custom model dialog with fields for model name, endpoint URL, API key and family — The Add custom model dialog

Save

Bolna checks the endpoint before saving it. Your key is stored encrypted and is never shown again.

Each model you add appears as its own card on the Integrations page, showing its endpoint, with a delete action.

Integrations page showing a registered custom model card with its model name, family tag, endpoint URL and a delete action — A registered custom model, shown as its own card alongside the built-in providers

What is checked when you save

Registration is refused when Bolna can prove the configuration is wrong:

The endpoint URL must resolve to a public address. Private, loopback and link-local addresses are rejected.
If the endpoint rejects the API key, you get The endpoint rejected this API key.
If the endpoint publishes a model list that does not include your model name, the error names what it does serve.
A model name must be unique within the scope that owns it, because an agent resolves its custom model by name. A duplicate is refused before the endpoint is contacted.

If the endpoint cannot be reached at all, for example because a serverless deployment has scaled to zero, the model is still saved. Verification is best effort and cannot guarantee that a model will be reachable later.

Use it on an agent

Open the agent, go to the Intelligence tab, choose custom as the provider, then pick your model. Save the agent. If your server needs parameters beyond the OpenAI standard, such as turning a model’s thinking off, switch on Extra request params on the same tab and enter them as JSON. They are sent on every request to your endpoint. See Extra request params for examples and the rules.

Key settings

Setting	Type	Recommended	Description
`provider`	string	`"custom"`	Provider name
`model`	string	e.g. `"Qwen/Qwen2.5-3B-Instruct"`	The model id your endpoint serves, passed through as-is
`base_url`	string	e.g. `"https://your-host/v1"`	Your OpenAI-compatible base URL, including the version segment
`family`	string	e.g. `"qwen"`	Model family, used for prompt formatting hints
`max_tokens`	integer	`150`	Cap on response length, keep it short for voice
`temperature`	float	`0.2`	Lower is more consistent
`agent_flow_type`	string	`"streaming"`	Always `"streaming"` for voice
`extra_body`	object	e.g. `{"chat_template_kwargs": {"enable_thinking": false}}`	Extra fields your server accepts beyond the OpenAI standard, sent on every request. See Extra request params

Extra request params

Many servers accept parameters that are not part of the OpenAI API, such as turning a model’s thinking off. Set them in extra_body and Bolna adds them to every request it sends to your endpoint.

"llm_config": {
  "provider": "custom",
  "model": "Qwen/Qwen3-8B",
  "base_url": "https://your-host/v1",
  "extra_body": {
    "chat_template_kwargs": { "enable_thinking": false }
  }
}

Each key is sent as a top-level field of the request body, exactly as the OpenAI SDK’s extra_body does, so the request above reaches your server as {"model": "Qwen/Qwen3-8B", "messages": [...], "chat_template_kwargs": {"enable_thinking": false}, ...}. Common examples:

Server	Goal	`extra_body`
vLLM or SGLang with Qwen3	Turn thinking off	`{"chat_template_kwargs": {"enable_thinking": false}}`
OpenRouter	Turn reasoning off	`{"reasoning": {"enabled": false}}`
vLLM	Sampling beyond the OpenAI set	`{"top_k": 20, "repetition_penalty": 1.05}`

It is set per agent, so two agents on the same model can differ. The params go on every request to your endpoint: conversation turns, graph agent routing, and the end-of-call and voicemail checks. Turning thinking off therefore speeds up routing as well as replies. In the dashboard, open the agent’s Intelligence tab with your custom model selected and switch on Extra request params. Type the JSON, or start from one of the example buttons.

Model Parameters card with the Extra request params toggle switched on, a JSON box containing chat_template_kwargs with enable_thinking set to false, and example buttons for Thinking off (vLLM / SGLang) and Reasoning off (OpenRouter) — Extra request params under Model Parameters, shown when a custom model is selected

Saving the agent is refused when:

extra_body is not a JSON object
the agent’s provider is not custom, since hosted providers never receive extra params
it sets a field Bolna controls: model, messages, stream, stream_options, tools, tool_choice, parallel_tool_calls or response_format

Bolna cannot know which params your server accepts. A misspelled or unsupported key is saved, and your server then either ignores it or rejects every request with a 400, which leaves the agent silent. Test a new extra_body with a web call before putting the agent on live traffic.

Manage your models

Delete a model from its card on the Integrations page. A model that is still selected on an agent cannot be deleted: the error names the agents using it, so you can point them at another model first. This applies to every agent config, including graph agent nodes. Deleting a model also removes the stored API key.

Sub-accounts

A custom model belongs to the scope it was created in:

A model added on your main account is available to the account and to all of its sub-accounts.
A model added inside a sub-account is available only in that sub-account. It is not visible to the main account or to other sub-accounts, and neither can delete it.

Add a custom model over the API

curl --request POST \
  --url https://api.bolna.ai/user/model/custom \
  --header 'Authorization: Bearer <BOLNA_API_KEY>' \
  --header 'Content-Type: application/json' \
  --data '{
    "custom_model_name": "Qwen/Qwen2.5-3B-Instruct",
    "custom_model_url": "https://your-host/v1",
    "custom_model_key": "<YOUR_ENDPOINT_KEY>",
    "custom_model_family": "qwen"
  }'

curl --request GET \
  --url https://api.bolna.ai/user/model/custom \
  --header 'Authorization: Bearer <BOLNA_API_KEY>'

curl --request DELETE \
  --url https://api.bolna.ai/user/model/custom/<MODEL_ID> \
  --header 'Authorization: Bearer <BOLNA_API_KEY>'

See Add a custom model for the full request and response schema.

When to use a custom LLM vs a hosted provider

Use a custom LLM when:

You run a fine-tuned or open-weight model that no hosted provider offers
The weights or the traffic have to stay on your own infrastructure
You already have a contract with a provider and want the tokens billed to it directly

Use a hosted provider integration when:

You want a frontier model without operating an inference server
You need Bolna’s per-provider features, such as Azure data residency or provisioned throughput
You would rather not own capacity planning, cold starts and upgrades

FAQ

Does Bolna use the OpenAI Responses API for custom models?

No. Custom models always use chat completions, because the Responses API’s server-side conversation chaining is specific to OpenAI and a self-hosted server cannot resolve a previous response id.

How do I turn thinking off on my reasoning model?

Set it in extra_body with the parameter your server uses. On vLLM or SGLang with Qwen3 that is {"chat_template_kwargs": {"enable_thinking": false}}. Thinking adds silence before every reply, so voice agents usually want it off.

Does Bolna send service_tier to my endpoint?

No. service_tier is an OpenAI-only field that strict OpenAI-compatible servers reject, so it is never sent to a custom endpoint.

What happens if my endpoint has scaled to zero?

The first call pays the cold start, which the caller hears as silence before the agent speaks. Keep a warm instance if first-token latency matters. Registration tolerates an unreachable endpoint, so scaling to zero does not block you from adding the model.

Who bills me for the tokens?

Your provider. Bolna records a custom model’s token usage against your own endpoint, so that model’s inference is not billed by Bolna.

Can a sub-account have its own custom model?

Yes. A model added on the account is available to it and to all of its sub-accounts. A model added inside a sub-account stays in that sub-account, invisible to the main account and to sibling sub-accounts.

Can I point an agent at a model on a private network?

No. The endpoint must resolve to a public address. Expose it through a public hostname, with your key as the bearer token.

Intelligence Tab: configure LLM in the dashboard
OpenAI: direct OpenAI integration
OpenRouter: one key for many hosted providers
Prompting Guide: write effective prompts for voice

Telephony

AI models

No-code automation

Choose providers

Custom LLMs for Bolna voice agents

Quick config

What your endpoint must support

Add a custom model in the dashboard

What is checked when you save

Use it on an agent

Key settings

Extra request params

Manage your models

Sub-accounts

Add a custom model over the API

When to use a custom LLM vs a hosted provider

FAQ

​Quick config

​What your endpoint must support

​Add a custom model in the dashboard

​What is checked when you save

​Use it on an agent

​Key settings

​Extra request params

​Manage your models

​Sub-accounts

​Add a custom model over the API

​When to use a custom LLM vs a hosted provider

​FAQ

​Related

Quick config

What your endpoint must support

Add a custom model in the dashboard

What is checked when you save

Use it on an agent

Key settings

Extra request params

Manage your models

Sub-accounts

Add a custom model over the API

When to use a custom LLM vs a hosted provider

FAQ

Related