When to use it
You’ve saved a new version of an agent, such as a shorter greeting, a Hindi-first opening or a different offer, and want to know whether it does better than the version you run today. An A/B test on an agent node splits that node’s calls between two or more saved versions of its agent, by weight. Every call records which version ran, so you can see it on the contact’s journey, in the execution webhook, and side by side on the campaign page. An agent node without an A/B test calls the agent’s live version. This page covers workflow agent nodes. For the other ways to A/B test an agent, individual calls and batches, and for picking a winner, see A/B test voice agent prompts and versions.Compared with a batch split
Before you start
The agent needs at least two saved versions. Create one in the agent editor with Save agent as a new version (see Agent versions), or with the agent versions API. The versions you test don’t have to be live.Turn on an A/B test
- Dashboard
- API
1
Open the agent node
Select the agent node in the workflow editor. The A/B test section sits below Agent in the Inspector.
2
Turn on A/B test versions
Two variants appear: A with the live version at 50%, and B with an empty Choose a version picker at 50%, and the panel shows Choose a version for B until you pick one. The picker lists the agent’s versions with the live one marked Live, and a version already used in another row can’t be picked twice. An arm whose version was deleted reads Version deleted.
3
Set the weights
Type a whole-number percentage for each version; a weight outside 1 to 99 shows Weights are whole numbers from 1 to 99. The total beside the split bar, such as Total 100%, must reach 100. When it doesn’t, the panel shows the gap, such as 10% short or 10% over, and offers one fix, such as Add 10% to A or Take 10% from C. Split evenly divides 100 across the rows, with any remainder going to A (34 / 33 / 33 for three versions). Add variant adds one until every version of the agent has a variant, and once there are more than two, each one can be removed.
4
Choose whether contacts keep their version
Keep each contact on one version is on by default, captioned Retries and repeat visits use the same version. Turned off, the caption reads Each call picks a version again. See Keep each contact on one version for what it changes.
5
Publish
Validate and publish the workflow. A draft saves even when the weights don’t add up to 100, but it can’t be published until they do.
Settings
Rules
- At least two versions, and a version chosen for every arm.
- Weights are whole numbers from 1 to 99 and add up to exactly 100.
- Each version appears once, even through two different ids of it.
- Every version is one the agent’s versions list shows, for the node’s agent (
agent_id).
2 (B):
The first five checks report only the first problem they find, so after fixing it, validate again to see whether another arm needs attention. The split follows the weights approximately: each call is assigned a version by weight, so over many calls the shares come close to the weights, while a small number of calls can be uneven.
Keep each contact on one version
With Keep each contact on one version on ("sticky": true), each contact gets one version at this node for its whole journey. Retries and later visits to the node use the same version, so every call a contact receives from the node sounds the same, and the comparison can credit the journey’s outcome to that version.
With it off ("sticky": false), every visit to the node picks a version again by weight, so a contact retried three times can hear different versions. Use this when you are comparing single calls, such as how many complete or how long they last, rather than whole journeys. The comparison then shows call results only.
Each A/B-tested node picks on its own, and a contact who starts a new journey, for example in another campaign, can get a different version.
Versions with different call settings
A call runs entirely on the version it was assigned. When the versions in a test differ in their telephony provider, Total Call Timeout, Voicemail Detection or calling hours, each call uses the settings of the version that runs it.Prompt variables
All versions in the test share the node’s Variables. Map every variable that any of the versions uses: validation warnsunmapped_agent_variable for each one without a mapping and names the version that uses it, for example agent prompt variable 'offer_code' used by version 'v2' has no mapping on this node. In the editor, the node’s Variables list every variable any version uses, and a variable that only some versions use is captioned with them, such as Used by B only or Used by B and C only.
When the other versions’ variables can’t be read yet, the list shows the live version’s variables, with one line at the top saying why:
- Showing the live version’s variables. Finish every A/B split in this workflow to see the rest. This node’s split, or another A/B split in the workflow, isn’t complete yet.
- Showing the live version’s variables while the other versions load.
- Showing the live version’s variables. Couldn’t load the other versions’ variables. It tries again on the next save.
- Finish the split to check which versions use these. The split still has an empty row, a weight outside 1 to 99, or a total other than 100.
- Finish every A/B split in this workflow to check which versions use these. Another node’s A/B split isn’t complete yet.
- Checking which versions use these… The check is running.
- Couldn’t check which versions use these. It tries again on the next save.
Changing versions during a test
The test follows each version by its name, so ordinary work on the agent doesn’t break it.Deleting a version in a test
Delete agent version returns409 while a workflow call that has been accepted but hasn’t connected yet is waiting on a save of that version, for example one held until calling hours, with the message This version has workflow calls waiting to be placed. Take it out of the workflow and let those calls connect first. It also returns 409 while an A/B batch that hasn’t finished uses the version. Try again once those calls have connected.
After a version in a test is deleted:
- New calls on its arm run the live version instead. Workflows already published, and campaigns already running, keep going.
- The workflow can’t be published until the node is fixed: validation reports
unknown_agent_versionat that arm’sconfig.version_split.arms[<i>].agent_version_id, such asA/B arm 2 (B): agent version '<id>' not found. Choose another version for the arm, or turn the test off. - Calls that ran on the live version in its place show in the comparison as their own row for that arm, under the live version’s name, captioned for example Ran live. B’s version was deleted.
To take a version out of a test without deleting it, choose another version for its arm or turn the test off, and publish. A campaign keeps the split it was created with, so its calls on that arm keep using the version.
Running campaigns keep their split
The split is part of the published workflow version. A campaign keeps the version it was created with, so changing the weights or the versions in the test and publishing again applies to new campaigns and new runs, not to campaigns already created. Edits saved into the versions themselves do reach running campaigns, as described above.Branch on the version
Every agent node, with or without an A/B test, writes two more outputs whenever it places a call:
Branch on the name. Each save into a version gives it a new
id, while its name stays the same until you rename it:
node.<agent_node_id>.call.agent_version_name, even after another agent node has run. See Conditions and variables.
See which version a contact got
- Journey timeline. Open a contact’s journey from the campaign page. A step of an A/B-tested node shows a chip with the arm’s letter and the name of the version that ran, for example B, v2-short-greeting. The chip always names the version that ran: a call placed on the live version in place of a deleted one shows the live version’s name, with no separate label. Steps of other agent nodes have no chip; their version is in the step’s outputs.
- Execution API. The node’s attempt in Get workflow execution carries the two
call.*outputs. - Execution webhook. The node’s entry in the webhook
trailcarries them too, and on an A/B-tested nodeinputsaddsversion_split.agent_version_id:
inputs.version_split.agent_version_id is the version the node picked, exactly as it appears in arms. The outputs are the version that ran: the picked version’s latest save, so its id can differ from the one in arms after edits, or the live version when the picked one was deleted.
Compare the versions
On the campaign page, each A/B-tested node has its own card, titled A/B test: and the node’s name, such as A/B test: Qualification call. Open it to load the numbers, and click Refresh to update them; Updated shows how long ago they were loaded. Set shows the split you configured and, once calls have been placed, Actual shows the split calls really took, each with its shares such as 50 / 30 / 20. Until then the card reads No calls yet. Results appear as calls finish. The table has one row per version, and each column sorts:
Hover a percentage to see the count behind it, such as 731 of 1,204 calls. A cell with nothing to divide by, such as Avg call for a version with no completed calls, shows NA.
Call columns update as calls finish. Journey columns count settled journeys only: a contact still partway through the workflow counts once its journey ends. Each contact counts toward the version of its first visit to the node, and only when that visit placed a call: a contact whose first visit placed none, for example because the dial was rejected, isn’t counted in the journey columns, even if a later visit called. A journey that stopped without reaching an end node, such as one you canceled, is settled but has no outcome, so the three outcome columns can add up to less than all settled journeys. The card’s footer reads Calls update as they finish. Journey results count settled journeys. With Keep each contact on one version off, the journey columns are left out and the footer reads Calls update as they finish. followed by Journey results need ‘Keep each contact on one version’, because a journey can use several versions.
Workflow calls are agent calls, so the other ways to compare versions work for them too: the Version filter and column in call history, the
agent_version_id filter on Get agent executions, and the agent_version_id column in Download Records. See Compare the results.
Comparison API
The same numbers come fromGET /workflow-campaigns/{campaign_id}/ab-tests, worked out each time you call it:
All values are counts. Work out percentages and averages from them: the completion rate is
completed / calls, and the average call is duration_s / completed.
Next steps
Agent versions
Save, edit and test versions of an agent
Workflow nodes
Every agent node setting and output
Campaigns
Run the workflow over your contacts
Execution webhook
Get each contact’s journey, version included, when it finishes