Self-host an LLM on Modal for Bolna voice agents

​

What to enter in Bolna

​

What you need

• A Modal account and the Modal CLI (pip install modal, then modal setup)
• The Hugging Face model you want to serve
• A Bolna account with access to Settings → Integrations

​

Deploy and connect

1

Deploy a vLLM server

Save this as bolna_llm.py and run modal deploy bolna_llm.py. It is a trimmed version of Modal’s vLLM inference example, with the settings a voice agent needs. Swap in your own model and the GPU it needs.

bolna_llm.py

import json
import subprocess

import modal

MODEL = "google/gemma-4-26B-A4B-it"

image = (
    modal.Image.from_registry("nvidia/cuda:12.9.0-devel-ubuntu22.04", add_python="3.12")
    .entrypoint([])
    .uv_pip_install("vllm==0.21.0")
    .env({"HF_XET_HIGH_PERFORMANCE": "1"})
)
app = modal.App("bolna-llm")


@app.server(
    image=image,
    gpu="H200",
    port=8000,
    startup_timeout=10 * 60,
    min_containers=1,  # a Server with no running container rejects requests
    target_concurrency=100,  # requests per container before Modal adds another
    routing_region="ap-south",  # Mumbai; pick the region closest to your callers
    volumes={
        "/root/.cache/huggingface": modal.Volume.from_name("huggingface-cache", create_if_missing=True),
        "/root/.cache/vllm": modal.Volume.from_name("vllm-cache", create_if_missing=True),
    },
)
class Server:
    @modal.enter()
    def start(self):
        self.process = subprocess.Popen([
            "vllm", "serve", MODEL,
            "--served-model-name", "llm",
            "--host", "0.0.0.0", "--port", "8000",
            "--limit-mm-per-prompt", json.dumps({"image": 0, "video": 0, "audio": 0}),
            "--enable-auto-tool-choice", "--tool-call-parser", "gemma4",
        ])

    @modal.exit()
    def stop(self):
        self.process.terminate()

modal deploy prints the server URL, for example https://your-workspace--bolna-llm-server.ap-south.modal.direct. The dialog’s placeholder shows the older *.modal.run shape that Modal web functions use; both work the same way. Paste the URL you were given and add /v1.

2

Create a proxy token

A Modal Server rejects requests without a proxy token by default, with a 401. Create one under Settings → Proxy Auth Tokens in Modal, or run modal workspace proxy-tokens create. Join the token ID and secret with a period, wk-<id>.ws-<secret>, and use that as the API key. Modal accepts it as a bearer token, which is what Bolna sends.If your workspace uses role-based access, scope the token to the environment the app is deployed in, or requests are denied with a 403.

3

Check the endpoint

curl https://<your-modal-url>/v1/models \
  -H "Authorization: Bearer wk-<id>.ws-<secret>"

The id in the response is the model name to register, llm in the example above. To also measure time to first token, run the streaming test.

4

Choose Modal in the Add custom model dialog

In Bolna, go to Settings → Integrations, filter by LLM, and click Add model on the Custom LLM card. You can also open the same dialog from an agent: on the Intelligence tab, open the LLM Provider dropdown and choose + Add custom LLM.Pick Modal as the provider. The endpoint field only shows the shape of a Modal URL as a placeholder, because your URL is unique to your workspace and app.

5

Register the model

• Dashboard
• API

Fill in the values from the table above and click Add model.

curl --request POST \
  --url https://api.bolna.ai/user/model/custom \
  --header 'Authorization: Bearer <BOLNA_API_KEY>' \
  --header 'Content-Type: application/json' \
  --data '{
    "custom_model_name": "llm",
    "custom_model_url": "https://<your-modal-url>/v1",
    "custom_model_key": "wk-<id>.ws-<secret>",
    "custom_model_family": "gemma"
  }'

6

Use it on an agent

On the agent’s Intelligence tab, choose custom as the provider and select your model. If you added it from the agent’s dropdown, it is already selected. Save the agent and place a web call before you send live traffic to it.

​

Things to know

• Keep a container warm. A Modal Server with no running container rejects requests with a 503 instead of queuing them, so a call that arrives after the app has scaled to zero gets no reply at all. Keep min_containers=1 for any agent that takes live calls. You pay for that GPU while it idles.
• Region. routing_region sets where Modal’s proxy accepts requests: us-east (default), us-west, ca-central, eu-west, ap-south (Mumbai) or ap-southeast-2. To also keep the GPUs in that region, set compute_region, which adds a cost multiplier.
• Model name. Bolna checks the model name against your server’s /v1/models when you add it. vLLM lists every value given to --served-model-name, so enter one of them exactly.
• Tool calling. If your agent uses tools, vLLM needs --enable-auto-tool-choice and the parser for your model, for example gemma4 for Gemma 4, hermes for Qwen2.5 or llama3_json for Llama 3.1. See vLLM’s tool calling guide.
• Thinking. Qwen3 models think before every reply by default, which the caller hears as silence. Turn it off per agent with the extra request params {"chat_template_kwargs": {"enable_thinking": false}}. Gemma 4 has thinking off by default.
• Using vLLM’s own key instead. If you deploy with unauthenticated=True, Modal’s proxy lets every request through. In that case, protect the server with vLLM’s --api-key flag, read from a Modal Secret, and register that key in Bolna instead of a proxy token.

​

Tuning a Modal deployment for voice

​

Troubleshooting

​

Related

• Custom LLM inference: every field, the save-time checks and extra request params
• Modal vLLM inference example, Modal Servers and proxy auth tokens
• Baseten: managed dedicated deployments if you’d rather not maintain the server code