Modal self-hosted LLM for Bolna AI voice agents

What is Modal used for in a Bolna voice agent?

Modal is where you run your own LLM server for a Bolna voice agent: you write a small Python app, run modal deploy, and Modal gives you a public HTTPS URL backed by GPUs. Unlike the other inference providers, Modal has no model catalog or shared endpoint, so you run the inference server. Choose it for a fine-tuned checkpoint, a model no hosted API offers, a GPU region close to your callers, or full control over the serving engine.

How do I set up Modal with Bolna?

1. Install the Modal CLI with pip install modal, then run modal setup.
2. Deploy an OpenAI-compatible vLLM server with modal deploy, based on Modal's vLLM inference example. Keep min_containers=1 and set routing_region close to your callers.
3. Create a proxy token under Settings → Proxy Auth Tokens in Modal, or with modal workspace proxy-tokens create.
4. Check the endpoint with a request to /v1/models. The id in the response is the model name to register.
5. In Bolna, go to Settings → Integrations, filter by LLM, click Add model on the Custom LLM card and pick Modal.
6. Enter the model name (a value you gave vLLM's --served-model-name, such as llm), the URL modal deploy printed followed by /v1, the proxy token as wk-<id>.ws-<secret>, and a family such as gemma or qwen.
7. On the agent's Intelligence tab choose custom, select the model, save and place a web call.

The Modal setup guide in the Bolna docs includes the full example app.

Which Modal settings matter for a voice agent?

Setting	Where	Why it matters on a call
min_containers	@app.server	Keep at least 1; with none running, requests get a 503
target_concurrency	@app.server	Concurrent requests per container before Modal adds another
routing_region	@app.server	us-east (default), us-west, ca-central, eu-west, ap-south (Mumbai) or ap-southeast-2
GPU type	gpu=	Larger GPUs cut time to first token for bigger models
--max-model-len	vLLM flags	Smaller context windows boot faster and leave memory for concurrent calls
--enable-auto-tool-choice	vLLM flags	Needed with your model's tool call parser if the agent uses tools

What can you build with Modal and Bolna?

Agents on your own fine-tuned model. Serve any Hugging Face checkpoint with vLLM and use it as the agent's LLM, with tokens billed by your Modal account rather than Bolna.

Lower latency for callers in India. Put the endpoint in ap-south to cut network time on every turn.

Open models with thinking turned off. Qwen3 models think before every reply by default. Turn it off per agent with the extra request param {"chat_template_kwargs": {"enable_thinking": false}}.

Full control over serving. Pick the engine, GPU and context length yourself. For managed dedicated deployments without server code, see Baseten, and for every provider preset see the custom LLM page.