> ## Documentation Index
> Fetch the complete documentation index at: https://www.bolna.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Self-host an LLM on Modal for Bolna voice agents

> Deploy an open-weight model with vLLM on Modal serverless GPUs, protect it with a Modal proxy token, keep it warm, and connect it to a Bolna voice agent as a custom LLM.

[Modal](https://modal.com) is a serverless GPU platform: you write a small Python app, run `modal deploy`, and Modal gives you a public HTTPS URL backed by GPUs. Unlike the other providers in this section, Modal has no model catalog and no shared endpoint. **You run the inference server.** Choose Modal when you want a fine-tuned checkpoint, a model no hosted API offers, a GPU region close to your callers, or full control over the serving engine.

This guide deploys an OpenAI-compatible [vLLM](https://docs.vllm.ai) server on Modal and adds it to Bolna as a [custom LLM](/docs/custom-llm-inference).

## What to enter in Bolna

| Field | Value |
| - | - |
| **Provider** | Modal |
| **Model name** | A value you gave vLLM's `--served-model-name`, for example `llm` |
| **Endpoint URL** | The URL `modal deploy` prints, followed by `/v1` |
| **API key** | A Modal proxy token, written as `wk-<id>.ws-<secret>` |
| **Family** | The model family, for example `gemma` or `qwen` |

***

## What you need

* A Modal account and the Modal CLI (`pip install modal`, then `modal setup`)
* The Hugging Face model you want to serve
* A Bolna account with access to **Settings → Integrations**

***

## Deploy and connect

<Steps>
  <Step title="Deploy a vLLM server">
    Save this as `bolna_llm.py` and run `modal deploy bolna_llm.py`. It is a trimmed version of Modal's [vLLM inference example](https://modal.com/docs/examples/vllm_inference), with the settings a voice agent needs. Swap in your own model and the GPU it needs.

    ```python bolna_llm.py theme={"system"}
    import json
    import subprocess

    import modal

    MODEL = "google/gemma-4-26B-A4B-it"

    image = (
        modal.Image.from_registry("nvidia/cuda:12.9.0-devel-ubuntu22.04", add_python="3.12")
        .entrypoint([])
        .uv_pip_install("vllm==0.21.0")
        .env({"HF_XET_HIGH_PERFORMANCE": "1"})
    )
    app = modal.App("bolna-llm")


    @app.server(
        image=image,
        gpu="H200",
        port=8000,
        startup_timeout=10 * 60,
        min_containers=1,  # a Server with no running container rejects requests
        target_concurrency=100,  # requests per container before Modal adds another
        routing_region="ap-south",  # Mumbai; pick the region closest to your callers
        volumes={
            "/root/.cache/huggingface": modal.Volume.from_name("huggingface-cache", create_if_missing=True),
            "/root/.cache/vllm": modal.Volume.from_name("vllm-cache", create_if_missing=True),
        },
    )
    class Server:
        @modal.enter()
        def start(self):
            self.process = subprocess.Popen([
                "vllm", "serve", MODEL,
                "--served-model-name", "llm",
                "--host", "0.0.0.0", "--port", "8000",
                "--limit-mm-per-prompt", json.dumps({"image": 0, "video": 0, "audio": 0}),
                "--enable-auto-tool-choice", "--tool-call-parser", "gemma4",
            ])

        @modal.exit()
        def stop(self):
            self.process.terminate()
    ```

    `modal deploy` prints the server URL, for example `https://your-workspace--bolna-llm-server.ap-south.modal.direct`. The dialog's placeholder shows the older `*.modal.run` shape that Modal web functions use; both work the same way. Paste the URL you were given and add `/v1`.
  </Step>

  <Step title="Create a proxy token">
    A Modal Server rejects requests without a proxy token by default, with a `401`. Create one under [Settings → Proxy Auth Tokens](https://modal.com/settings/proxy-auth-tokens) in Modal, or run `modal workspace proxy-tokens create`. Join the token ID and secret with a period, `wk-<id>.ws-<secret>`, and use that as the API key. Modal accepts it as a bearer token, which is what Bolna sends.

    If your workspace uses role-based access, scope the token to the environment the app is deployed in, or requests are denied with a `403`.
  </Step>

  <Step title="Check the endpoint">
    ```bash theme={"system"}
    curl https://<your-modal-url>/v1/models \
      -H "Authorization: Bearer wk-<id>.ws-<secret>"
    ```

    The `id` in the response is the model name to register, `llm` in the example above. To also measure time to first token, run the [streaming test](/docs/custom-llm-inference#test-your-endpoint-before-you-register-it).
  </Step>

  <Step title="Choose Modal in the Add custom model dialog">
    In Bolna, go to **Settings → Integrations**, filter by **LLM**, and click **Add model** on the **Custom LLM** card. You can also open the same dialog from an agent: on the **Intelligence** tab, open the **LLM Provider** dropdown and choose **+ Add custom LLM**.

    Pick **Modal** as the provider. The endpoint field only shows the shape of a Modal URL as a placeholder, because your URL is unique to your workspace and app.

    <Frame caption="Modal selected: the endpoint placeholder shows the URL shape, the hint says to use the URL from modal deploy">
      <img src="https://mintcdn.com/bolna-54a2d4fe/0EXypen329Eb5XE2/images/custom-llm/providers/modal-selected.png?fit=max&auto=format&n=0EXypen329Eb5XE2&q=85&s=2b800720a839256685b7cbb1ef20ef63" alt="Bolna Add custom model dialog with Modal selected as provider and an empty endpoint URL field showing a Modal URL placeholder" width="1024" height="1220" data-path="images/custom-llm/providers/modal-selected.png" />
    </Frame>
  </Step>

  <Step title="Register the model">
    <Tabs>
      <Tab title="Dashboard">
        Fill in the values from [the table above](#what-to-enter-in-bolna) and click **Add model**.

        <Frame caption="A filled-in Modal registration">
          <img src="https://mintcdn.com/bolna-54a2d4fe/aV72vktown5Rn3f1/images/custom-llm/providers/modal-filled.png?fit=max&auto=format&n=aV72vktown5Rn3f1&q=85&s=c694afc737e83456ca3817d180d59c5f" alt="Add custom model dialog with Modal selected, a model name, a Modal endpoint URL, a masked API key and a family" width="1024" height="1172" data-path="images/custom-llm/providers/modal-filled.png" />
        </Frame>
      </Tab>

      <Tab title="API">
        ```bash theme={"system"}
        curl --request POST \
          --url https://api.bolna.ai/user/model/custom \
          --header 'Authorization: Bearer <BOLNA_API_KEY>' \
          --header 'Content-Type: application/json' \
          --data '{
            "custom_model_name": "llm",
            "custom_model_url": "https://<your-modal-url>/v1",
            "custom_model_key": "wk-<id>.ws-<secret>",
            "custom_model_family": "gemma"
          }'
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Use it on an agent">
    On the agent's **Intelligence** tab, choose **custom** as the provider and select your model. If you added it from the agent's dropdown, it is already selected. Save the agent and place a web call before you send live traffic to it.
  </Step>
</Steps>

***

## Things to know

* **Keep a container warm.** A Modal Server with no running container rejects requests with a `503` instead of queuing them, so a call that arrives after the app has scaled to zero gets no reply at all. Keep `min_containers=1` for any agent that takes live calls. You pay for that GPU while it idles.
* **Region.** `routing_region` sets where Modal's proxy accepts requests: `us-east` (default), `us-west`, `ca-central`, `eu-west`, `ap-south` (Mumbai) or `ap-southeast-2`. To also keep the GPUs in that region, set `compute_region`, which adds a [cost multiplier](https://modal.com/docs/guide/region-selection#pricing).
* **Model name.** Bolna checks the model name against your server's `/v1/models` when you add it. vLLM lists every value given to `--served-model-name`, so enter one of them exactly.
* **Tool calling.** If your agent uses tools, vLLM needs `--enable-auto-tool-choice` and the parser for your model, for example `gemma4` for Gemma 4, `hermes` for Qwen2.5 or `llama3_json` for Llama 3.1. See vLLM's [tool calling guide](https://docs.vllm.ai/en/latest/features/tool_calling/).
* **Thinking.** Qwen3 models think before every reply by default, which the caller hears as silence. Turn it off per agent with the [extra request params](/docs/custom-llm-inference#extra-request-params) `{"chat_template_kwargs": {"enable_thinking": false}}`. Gemma 4 has thinking off by default.
* **Using vLLM's own key instead.** If you deploy with `unauthenticated=True`, Modal's proxy lets every request through. In that case, protect the server with vLLM's `--api-key` flag, read from a [Modal Secret](https://modal.com/docs/guide/secrets), and register that key in Bolna instead of a proxy token.

## Tuning a Modal deployment for voice

| Setting | Where | Why it matters on a call |
| - | - | - |
| `min_containers` | `@app.server` | Keep at least `1`; with none running, requests get a `503` |
| `target_concurrency` | `@app.server` | Concurrent requests per container before Modal adds another; size it to your peak concurrent calls |
| `routing_region` | `@app.server` | Put the endpoint close to your callers to cut network time on every turn |
| GPU type | `gpu=` | Larger GPUs cut time to first token for bigger models |
| `--max-model-len` | vLLM flags | Smaller context windows boot faster and leave more memory for concurrent calls |

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Calls get no reply, or the first call after a quiet period fails">
    The app scaled to zero, and Modal rejected the request with a `503`. Registration still saves a model it can't reach, so this first shows up on a call. Set `min_containers=1` and redeploy.
  </Accordion>

  <Accordion title="The endpoint rejected this API key">
    Check that the key is the token ID and secret joined with a period, `wk-<id>.ws-<secret>`, with no spaces. A `403` means the token isn't scoped to the app's environment. If you use vLLM's `--api-key` instead, make sure the Secret is attached and redeploy after changing it.
  </Accordion>

  <Accordion title="Model not found">
    The model name must match a `--served-model-name` value exactly. Check `GET /v1/models`.
  </Accordion>

  <Accordion title="404 on chat completions">
    The URL is missing `/v1`, or it points at a different Modal app. Use the URL that `modal deploy` printed for the server, followed by `/v1`.
  </Accordion>
</AccordionGroup>

***

## Related

* [Custom LLM inference](/docs/custom-llm-inference): every field, the save-time checks and extra request params
* [Modal vLLM inference example](https://modal.com/docs/examples/vllm_inference), [Modal Servers](https://modal.com/docs/guide/servers) and [proxy auth tokens](https://modal.com/docs/guide/webhook-proxy-auth)
* [Baseten](/docs/custom-llm-inference/baseten): managed dedicated deployments if you'd rather not maintain the server code


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.