> ## Documentation Index
> Fetch the complete documentation index at: https://www.bolna.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Connect Groq to Bolna for low-latency voice agents

> Run Llama and GPT-OSS on GroqCloud from a Bolna voice agent for fast time to first token. API key, production model IDs, unsupported OpenAI fields and rate limits.

[GroqCloud](https://console.groq.com) runs models on Groq's LPU hardware and is known for very fast token output, which makes it popular for real-time voice. A fast LLM shortens the gap between the caller finishing a sentence and the agent starting to speak, and that gap is what callers notice most.

***

## Production models to start with

Groq separates **production** models, which are meant for production traffic, from **preview** models, which are for evaluation and can be discontinued at short notice. Use production models for live agents.

| Model ID | Speed (tokens/s) | Good for |
| - | - | - |
| `llama-3.1-8b-instant` | \~560 | Short, scripted flows where speed matters most |
| `llama-3.3-70b-versatile` | \~280 | General-purpose agents that need better reasoning |
| `openai/gpt-oss-20b` | \~1000 | Fast open-weight model with tool use |
| `openai/gpt-oss-120b` | \~500 | Harder conversations and tool-heavy agents |

Speeds are from Groq's [supported models](https://console.groq.com/docs/models) page at the time of writing. Note the `openai/` prefix on the GPT-OSS models.

***

## Set up Groq

<Steps>
  <Step title="Create an API key">
    Sign in to the [GroqCloud console](https://console.groq.com) and create a key on the [API Keys](https://console.groq.com/keys) page.
  </Step>

  <Step title="Check your organisation's limits">
    Open [Settings → Limits](https://console.groq.com/settings/limits) to see your requests and tokens per minute and per day. Limits apply to the whole organisation, not to each key, so every agent sharing the org shares them.
  </Step>

  <Step title="Choose Groq in Bolna">
    Go to **Settings → Integrations → LLM → Custom LLM → Add model**, or choose **+ Add custom LLM** in an agent's **LLM Provider** dropdown. Select **Groq**.

    <Frame caption="Groq selected in the Add custom model dialog">
      <img src="https://mintcdn.com/bolna-54a2d4fe/0EXypen329Eb5XE2/images/custom-llm/providers/groq-selected.png?fit=max&auto=format&n=0EXypen329Eb5XE2&q=85&s=9572ba8bd6ddef937dabfed69d0511eb" alt="Bolna Add custom model dialog with Groq selected and api.groq.com/openai/v1 shown as the endpoint placeholder" width="1024" height="1140" data-path="images/custom-llm/providers/groq-selected.png" />
    </Frame>
  </Step>

  <Step title="Register the model">
    <Tabs>
      <Tab title="Dashboard">
        * **Model name**: a Groq model ID, for example `llama-3.3-70b-versatile`
        * **Endpoint URL**: `https://api.groq.com/openai/v1`. The `/openai` segment is required.
        * **API key**: the key from the console
        * **Family**: for example `llama` or `gpt-oss`

        <Frame caption="Groq with llama-3.3-70b-versatile">
          <img src="https://mintcdn.com/bolna-54a2d4fe/aV72vktown5Rn3f1/images/custom-llm/providers/groq-filled.png?fit=max&auto=format&n=aV72vktown5Rn3f1&q=85&s=10cc8248247d8d241873a097ecbc4e79" alt="Add custom model dialog with Groq selected, model llama-3.3-70b-versatile, endpoint api.groq.com/openai/v1, masked key and family llama" width="1024" height="1092" data-path="images/custom-llm/providers/groq-filled.png" />
        </Frame>
      </Tab>

      <Tab title="API">
        ```bash theme={"system"}
        curl --request POST \
          --url https://api.bolna.ai/user/model/custom \
          --header 'Authorization: Bearer <BOLNA_API_KEY>' \
          --header 'Content-Type: application/json' \
          --data '{
            "custom_model_name": "llama-3.3-70b-versatile",
            "custom_model_url": "https://api.groq.com/openai/v1",
            "custom_model_key": "<GROQ_API_KEY>",
            "custom_model_family": "llama"
          }'
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Select it on the agent">
    Choose **custom** on the **Intelligence** tab, pick the Groq model and save.
  </Step>
</Steps>

***

## OpenAI fields Groq handles differently

Groq is OpenAI-compatible, with some differences to know before you add **Extra request params**:

* `logprobs` and `top_logprobs` aren't supported. Sending them returns a `400` error.
* A `temperature` of `0` is converted to `1e-8`. Use a value above 0 and up to 2 if you see odd results.

See Groq's [OpenAI compatibility](https://console.groq.com/docs/openai) page for the full list.

***

## Rate limits during campaigns

Groq measures limits in requests per minute and per day, and tokens per minute and per day. Some organisations also have separate input and output token limits. You hit whichever limit you reach first. A batch of many short calls can hit the requests-per-minute limit long before the tokens limit. [Cached tokens](https://console.groq.com/docs/prompt-caching) don't count towards rate limits, so a long, stable system prompt shared by all calls costs less of your quota.

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="404 on chat completions">
    The endpoint must be `https://api.groq.com/openai/v1`. `https://api.groq.com/v1` doesn't work.
  </Accordion>

  <Accordion title="400 error after adding extra params">
    Remove `logprobs` or `top_logprobs` from **Extra request params**; Groq rejects them.
  </Accordion>

  <Accordion title="429 errors">
    You reached an organisation limit. Check **Settings → Limits** in the console, pace batches, or upgrade your Groq plan.
  </Accordion>

  <Accordion title="A model stopped working">
    Preview models can be removed at short notice. Move the agent to a production model.
  </Accordion>
</AccordionGroup>

***

## Related

* [Custom LLM inference](/docs/custom-llm-inference)
* [Cerebras](/docs/custom-llm-inference/cerebras): another very high-throughput provider
* [Intelligence Tab](/docs/agent-setup/llm-tab)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.