A/B test voice agent prompts and versions

To A/B test a Bolna voice agent, save each variant as an agent version, send part of your calls to each version, then compare the results by agent_version_id and make the winner live. Your live agent keeps answering every other call while the test runs. A version is a full copy of the agent’s setup, so you can test any change: a new prompt or welcome message, a different voice, LLM or transcriber, or other call settings. Each call runs on exactly one version, and records it as the call’s agent_version_id.

How to A/B test a voice agent

Save the variants. Edit the agent and save each change as a new version. The live version is usually your control.
Split the calls. Pass agent_version_id on each POST /call, or split a batch across 2 to 4 versions by percentage.
Compare the results. Filter call history and executions by version, and compare answer rate, duration, cost and extracted data. See Compare the results.
Make the winner live. Every call that doesn’t ask for a version then uses it. See Pick a winner.

There are two ways to split calls across versions:

	Individual calls	Batches
How	Pass `agent_version_id` on each `POST /call`	Set the split when you upload the batch, in the dashboard or with `version_split` on `POST /batches`
Who decides each call’s version	You, in your own code	Bolna, by the percentages you set
Versions	Any number	2 to 4
Best for	Calls you trigger from your own system	Outbound campaigns from a CSV

Calls that don’t ask for a version, including all inbound calls, keep using the live version.

Save each variant as its own version first, see Save as a new version. You’ll need each version’s id, from the version switcher or from List agent versions.

Split individual calls

Make call takes an optional agent_version_id. The call runs on that version’s prompts, voice, models and settings, and calls without it run on the live version. To A/B test, choose a version for each call in your own code and pass it. Assign each contact a version once and reuse it, so a contact called again gets the same version. Hashing a stable ID does this without storing anything:

Pick a version per contact

import hashlib
import os

import requests

VERSIONS = [
    ("0c6a5e0e-7b8f-4d1a-9f2e-3a4b5c6d7e8f", 70),  # control, 70%
    ("9e8d7c6b-5a4f-4e3d-8c2b-1a0f9e8d7c6b", 30),  # challenger, 30%
]

def version_for(contact_id: str) -> str:
    bucket = int(hashlib.sha256(contact_id.encode()).hexdigest(), 16) % 100
    for version_id, weight in VERSIONS:
        if bucket < weight:
            return version_id
        bucket -= weight
    return VERSIONS[-1][0]

requests.post(
    "https://api.bolna.ai/call",
    headers={"Authorization": f"Bearer {os.environ['BOLNA_API_KEY']}"},
    json={
        "agent_id": "123e4567-e89b-12d3-a456-426655440000",
        "agent_version_id": version_for("+919876543210"),
        "recipient_phone_number": "+919876543210",
    },
)

A version that doesn’t exist, belongs to another agent or was deleted returns 404 with Agent version not found.

Split a batch

A batch can split its contacts across two to four versions. Each contact is assigned a version when the batch is uploaded, in proportion to the percentages you set, and its retries keep that version.

Dashboard
API

On the Batches page, select your agent and click Upload Batch. Turn on A/B test versions, then pick a version and a traffic share for each variant. See Test agent versions against each other.

Upload Batch dialog with A/B test versions turned on

Each row is a variant: choose its version and its share of the batch’s contacts. The shares must add up to 100%, and Split evenly resets them to equal shares.

A/B test versions with v1 and v2 at 50% each — Two variants, the live v1 and v2, each getting 50% of the contacts

Pass version_split on Create batch:

curl https://api.bolna.ai/batches \
  -H "Authorization: Bearer $BOLNA_API_KEY" \
  -F "agent_id=123e4567-e89b-12d3-a456-426655440000" \
  -F "file=@recipients.csv" \
  -F 'version_split={"arms":[{"agent_version_id":"0c6a5e0e-7b8f-4d1a-9f2e-3a4b5c6d7e8f","weight":70},{"agent_version_id":"9e8d7c6b-5a4f-4e3d-8c2b-1a0f9e8d7c6b","weight":30}]}'

Weights are whole numbers from 1 to 99 that add up to 100, and each arm needs a different version.

All versions in the split need the same telephony provider as the live version, and batches with a workflow can’t be split.

Compare the results

Every call records the version it ran on, however it was started.

Dashboard: in call history, select the agent and use the Version filter and column. Download Records includes an agent_version_id column.
API: filter Get agent executions with agent_version_id. Each execution also returns its agent_version_id.

Compare outcomes such as answer rate, call duration, cost and your extracted data between versions.

Pick a winner

Make the winning version live with Make live in the version switcher, or with Set current version. From then on, every call that doesn’t ask for a version, including inbound calls, uses it.

FAQ

What can I A/B test on a voice agent?

Anything saved in an agent version: the prompts, the welcome message, the voice, the LLM and transcriber models, and the other call settings. Change one thing per version so you know which change made the difference.

Does an A/B test change my live agent?

No. Calls that don’t ask for a version, including all inbound calls, keep running on the live version. A version only takes over once you make it live.

How many versions can I test at once?

A batch splits across 2 to 4 versions. With individual calls there’s no limit, because your code picks the version for each call.

Can I A/B test inbound calls?

No. Inbound numbers always answer with the live version. To compare versions on inbound traffic, make one version live for a period, then the other, and compare the two periods.

Do retries and repeat calls use the same version?

In a batch, yes: each contact gets its version when the batch is uploaded, and its retries keep it. With individual calls, it’s up to your code, so assign each contact one version and reuse it, as in the example above.

Can I delete a version while it's being tested?

Not while it’s in the split of a batch that hasn’t finished. Delete agent version returns 409 until that batch completes or is stopped. The live version can’t be deleted either, so make another version live first. Calls that already ran on a deleted version keep their transcripts and still show it as their version.

Can I A/B test a batch that uses a workflow?

Not yet. Batches with a workflow can’t be split across versions.

Next steps

Agent versions

Save, edit and manage versions

Batch calling

Upload and run batches

Build

Deploy

Monitor

Go live

A/B test voice agent prompts and versions

How to A/B test a voice agent

Split individual calls

Split a batch

Compare the results

Pick a winner

FAQ

Next steps

Agent versions

Batch calling