> ## Documentation Index
> Fetch the complete documentation index at: https://www.bolna.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Connect Baseten Model APIs or dedicated deployments to Bolna

> Use Baseten Model APIs or your own Truss deployment on dedicated GPUs as the LLM for a Bolna voice agent. Endpoints, model slugs, Fast variants and API keys.

[Baseten](https://www.baseten.co) is a production inference platform with two ways in, and they need different values in Bolna:

* **Model APIs**: a supported set of popular open models that Baseten runs for you, called by model slug at one shared URL. Nothing to deploy.
* **Dedicated deployments**: you deploy any model, including fine-tunes, to GPUs reserved for you with a short `config.yaml` and the Truss CLI. Each deployment gets its own URL.

Choose Model APIs to get running in minutes. Choose a dedicated deployment when you need a model Model APIs doesn't offer, your own weights, or capacity nobody else shares.

***

## Get a Baseten API key

Create a key on the [API keys](https://app.baseten.co/settings/api_keys) page of your Baseten workspace settings. Bolna sends it as `Authorization: Bearer <key>`, which is how Baseten's OpenAI-compatible endpoints authenticate.

***

## Option A: Model APIs

<Steps>
  <Step title="Pick a model slug">
    Open Baseten's [Model APIs](https://docs.baseten.co/development/model-apis/overview) list and copy the slug, for example `zai-org/GLM-5.3`. Some models also have a **Fast** variant: the same weights with dedicated capacity for higher sustained throughput, under a separate slug with its own pricing and rate limits.
  </Step>

  <Step title="Choose Baseten in Bolna">
    Open **Settings → Integrations → LLM → Custom LLM → Add model**, or choose **+ Add custom LLM** from the **LLM Provider** dropdown on an agent. Select **Baseten**.

    <Frame caption="Baseten selected: Model APIs endpoint as placeholder, hint for dedicated deployments">
      <img src="https://mintcdn.com/bolna-54a2d4fe/0EXypen329Eb5XE2/images/custom-llm/providers/baseten-selected.png?fit=max&auto=format&n=0EXypen329Eb5XE2&q=85&s=318bf2ac02cf9a045f32ddd2f4fe5385" alt="Bolna Add custom model dialog with Baseten selected and inference.baseten.co/v1 shown as the endpoint placeholder" width="1024" height="1220" data-path="images/custom-llm/providers/baseten-selected.png" />
    </Frame>
  </Step>

  <Step title="Register the model">
    <Tabs>
      <Tab title="Dashboard">
        * **Model name**: the slug, for example `zai-org/GLM-5.3`
        * **Endpoint URL**: `https://inference.baseten.co/v1`
        * **API key**: your Baseten API key
        * **Family**: for example `glm`

        <Frame caption="Baseten Model APIs, filled in">
          <img src="https://mintcdn.com/bolna-54a2d4fe/aV72vktown5Rn3f1/images/custom-llm/providers/baseten-filled.png?fit=max&auto=format&n=aV72vktown5Rn3f1&q=85&s=c4a9f3983487cd096ec8123ab4be2195" alt="Add custom model dialog with Baseten selected, model zai-org/GLM-5.3, endpoint inference.baseten.co/v1, masked key and family glm" width="1024" height="1172" data-path="images/custom-llm/providers/baseten-filled.png" />
        </Frame>
      </Tab>

      <Tab title="API">
        ```bash theme={"system"}
        curl --request POST \
          --url https://api.bolna.ai/user/model/custom \
          --header 'Authorization: Bearer <BOLNA_API_KEY>' \
          --header 'Content-Type: application/json' \
          --data '{
            "custom_model_name": "zai-org/GLM-5.3",
            "custom_model_url": "https://inference.baseten.co/v1",
            "custom_model_key": "<BASETEN_API_KEY>",
            "custom_model_family": "glm"
          }'
        ```
      </Tab>
    </Tabs>
  </Step>
</Steps>

<Note>
  Baseten's context and output limits for a Model API can be lower than the model's advertised maximum. Check the limits in the Model APIs table before writing a very long system prompt.
</Note>

***

## Option B: A dedicated deployment

<Steps>
  <Step title="Install Truss and sign in">
    ```bash theme={"system"}
    pip install --upgrade truss
    truss login --browser
    ```
  </Step>

  <Step title="Write a config.yaml">
    Follow Baseten's [Deploy your first model](https://docs.baseten.co/examples/deploy-your-first-model) guide. The config names the Hugging Face checkpoint (`checkpoint_repository`), the GPU and the inference engine. With the TensorRT-LLM engine, no Python code is needed.
  </Step>

  <Step title="Deploy">
    ```bash theme={"system"}
    truss push
    ```

    Truss prints a model ID and a deployment ID. Wait until the deployment shows **Active** in the Baseten dashboard.
  </Step>

  <Step title="Register the deployment in Bolna">
    Choose **Baseten** in the Add custom model dialog and replace the placeholder with your deployment's OpenAI-compatible URL:

    ```text theme={"system"}
    https://model-<model_id>.api.baseten.co/environments/production/sync/v1
    ```

    Use the model name from the deployment's example request in the Baseten dashboard, and your Baseten API key.
  </Step>
</Steps>

Because the URL points at the **production** environment, you can promote a new deployment to production in Baseten and your Bolna agents switch to it without a change.

***

## Finish on the agent

On the agent's **Intelligence** tab choose **custom**, pick the Baseten model, save and place a test call.

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="401 Invalid or missing API key">
    Paste a key from **Settings → API keys** in the same Baseten workspace that owns the model or deployment.
  </Accordion>

  <Accordion title="Model not found on Model APIs">
    Use the slug exactly as listed, including the organisation prefix. A Fast variant has its own slug.
  </Accordion>

  <Accordion title="Slow first reply on a dedicated deployment">
    The deployment scaled down and is cold-starting. Set a minimum replica count of at least 1 in the deployment's autoscaling settings for live agents.
  </Accordion>
</AccordionGroup>

***

## Related

* [Custom LLM inference](/docs/custom-llm-inference)
* [Modal](/docs/custom-llm-inference/modal): write and run the serving code yourself
* [Fireworks AI](/docs/custom-llm-inference/fireworks): serverless models plus on-demand GPUs


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.