Connect Groq to Bolna for low-latency voice agents

GroqCloud runs models on Groq’s LPU hardware and is known for very fast token output, which makes it popular for real-time voice. A fast LLM shortens the gap between the caller finishing a sentence and the agent starting to speak, and that gap is what callers notice most.

Production models to start with

Groq separates production models, which are meant for production traffic, from preview models, which are for evaluation and can be discontinued at short notice. Use production models for live agents.

Model ID	Speed (tokens/s)	Good for
`llama-3.1-8b-instant`	~560	Short, scripted flows where speed matters most
`llama-3.3-70b-versatile`	~280	General-purpose agents that need better reasoning
`openai/gpt-oss-20b`	~1000	Fast open-weight model with tool use
`openai/gpt-oss-120b`	~500	Harder conversations and tool-heavy agents

Speeds are from Groq’s supported models page at the time of writing. Note the openai/ prefix on the GPT-OSS models.

Set up Groq

Create an API key

Check your organisation's limits

Open Settings → Limits to see your requests and tokens per minute and per day. Limits apply to the whole organisation, not to each key, so every agent sharing the org shares them.

Choose Groq in Bolna

Go to Settings → Integrations → LLM → Custom LLM → Add model, or choose + Add custom LLM in an agent’s LLM Provider dropdown. Select Groq.

Bolna Add custom model dialog with Groq selected and api.groq.com/openai/v1 shown as the endpoint placeholder — Groq selected in the Add custom model dialog

Dashboard
API

Model name: a Groq model ID, for example llama-3.3-70b-versatile
Endpoint URL: https://api.groq.com/openai/v1. The /openai segment is required.
API key: the key from the console
Family: for example llama or gpt-oss

Add custom model dialog with Groq selected, model llama-3.3-70b-versatile, endpoint api.groq.com/openai/v1, masked key and family llama — Groq with llama-3.3-70b-versatile

curl --request POST \
  --url https://api.bolna.ai/user/model/custom \
  --header 'Authorization: Bearer <BOLNA_API_KEY>' \
  --header 'Content-Type: application/json' \
  --data '{
    "custom_model_name": "llama-3.3-70b-versatile",
    "custom_model_url": "https://api.groq.com/openai/v1",
    "custom_model_key": "<GROQ_API_KEY>",
    "custom_model_family": "llama"
  }'

Select it on the agent

Choose custom on the Intelligence tab, pick the Groq model and save.

OpenAI fields Groq handles differently

Groq is OpenAI-compatible, with some differences to know before you add Extra request params:

logprobs and top_logprobs aren’t supported. Sending them returns a 400 error.
A temperature of 0 is converted to 1e-8. Use a value above 0 and up to 2 if you see odd results.

See Groq’s OpenAI compatibility page for the full list.

Rate limits during campaigns

Groq measures limits in requests per minute and per day, and tokens per minute and per day. Some organisations also have separate input and output token limits. You hit whichever limit you reach first. A batch of many short calls can hit the requests-per-minute limit long before the tokens limit. Cached tokens don’t count towards rate limits, so a long, stable system prompt shared by all calls costs less of your quota.

Troubleshooting

404 on chat completions

The endpoint must be https://api.groq.com/openai/v1. https://api.groq.com/v1 doesn’t work.

400 error after adding extra params

Remove logprobs or top_logprobs from Extra request params; Groq rejects them.

429 errors

You reached an organisation limit. Check Settings → Limits in the console, pace batches, or upgrade your Groq plan.

A model stopped working

Preview models can be removed at short notice. Move the agent to a production model.

Custom LLM inference
Cerebras: another very high-throughput provider
Intelligence Tab

Telephony

AI models

No-code automation

Choose providers

Connect Groq to Bolna for low-latency voice agents

Production models to start with

Set up Groq

OpenAI fields Groq handles differently

Rate limits during campaigns

Troubleshooting

​Production models to start with

​Set up Groq

​OpenAI fields Groq handles differently

​Rate limits during campaigns

​Troubleshooting

​Related

Production models to start with

Set up Groq

OpenAI fields Groq handles differently

Rate limits during campaigns

Troubleshooting

Related