Skip to main content

When to use it

You’ve saved a new version of an agent, such as a shorter greeting, a Hindi-first opening or a different offer, and want to know whether it does better than the version you run today. An A/B test on an agent node splits that node’s calls between two or more saved versions of its agent, by weight. Every call records which version ran, so you can see it on the contact’s journey, in the execution webhook, and side by side on the campaign page. An agent node without an A/B test calls the agent’s live version. This page covers workflow agent nodes. For the other ways to A/B test an agent, individual calls and batches, and for picking a winner, see A/B test voice agent prompts and versions.

Compared with a batch split


Before you start

The agent needs at least two saved versions. Create one in the agent editor with Save agent as a new version (see Agent versions), or with the agent versions API. The versions you test don’t have to be live.

Turn on an A/B test

1

Open the agent node

Select the agent node in the workflow editor. The A/B test section sits below Agent in the Inspector.
2

Turn on A/B test versions

Two variants appear: A with the live version at 50%, and B with an empty Choose a version picker at 50%, and the panel shows Choose a version for B until you pick one. The picker lists the agent’s versions with the live one marked Live, and a version already used in another row can’t be picked twice. An arm whose version was deleted reads Version deleted.
3

Set the weights

Type a whole-number percentage for each version; a weight outside 1 to 99 shows Weights are whole numbers from 1 to 99. The total beside the split bar, such as Total 100%, must reach 100. When it doesn’t, the panel shows the gap, such as 10% short or 10% over, and offers one fix, such as Add 10% to A or Take 10% from C. Split evenly divides 100 across the rows, with any remainder going to A (34 / 33 / 33 for three versions). Add variant adds one until every version of the agent has a variant, and once there are more than two, each one can be removed.
4

Choose whether contacts keep their version

Keep each contact on one version is on by default, captioned Retries and repeat visits use the same version. Turned off, the caption reads Each call picks a version again. See Keep each contact on one version for what it changes.
5

Publish

Validate and publish the workflow. A draft saves even when the weights don’t add up to 100, but it can’t be published until they do.
Turning A/B test versions off removes the A/B test from the node, which then calls the live version. When the agent has only one version the switch is unavailable, with the message Save the agent as a new version to A/B test it against the live version. If the versions can’t be loaded, the panel shows Couldn’t load this agent’s versions. with Try again. Choosing a different agent on the node clears the versions. On the canvas, a node with an A/B test shows its split bar and the weights under the agent name, such as A/B 50 / 30 / 20.

Settings

Rules

  • At least two versions, and a version chosen for every arm.
  • Weights are whole numbers from 1 to 99 and add up to exactly 100.
  • Each version appears once, even through two different ids of it.
  • Every version is one the agent’s versions list shows, for the node’s agent (agent_id).
Validation reports a broken rule as an error, so the workflow can’t be published until it’s fixed. Each message names the arm by its position and letter, such as 2 (B): The first five checks report only the first problem they find, so after fixing it, validate again to see whether another arm needs attention. The split follows the weights approximately: each call is assigned a version by weight, so over many calls the shares come close to the weights, while a small number of calls can be uneven.

Keep each contact on one version

With Keep each contact on one version on ("sticky": true), each contact gets one version at this node for its whole journey. Retries and later visits to the node use the same version, so every call a contact receives from the node sounds the same, and the comparison can credit the journey’s outcome to that version. With it off ("sticky": false), every visit to the node picks a version again by weight, so a contact retried three times can hear different versions. Use this when you are comparing single calls, such as how many complete or how long they last, rather than whole journeys. The comparison then shows call results only. Each A/B-tested node picks on its own, and a contact who starts a new journey, for example in another campaign, can get a different version.

Versions with different call settings

A call runs entirely on the version it was assigned. When the versions in a test differ in their telephony provider, Total Call Timeout, Voicemail Detection or calling hours, each call uses the settings of the version that runs it.

Prompt variables

All versions in the test share the node’s Variables. Map every variable that any of the versions uses: validation warns unmapped_agent_variable for each one without a mapping and names the version that uses it, for example agent prompt variable 'offer_code' used by version 'v2' has no mapping on this node. In the editor, the node’s Variables list every variable any version uses, and a variable that only some versions use is captioned with them, such as Used by B only or Used by B and C only. When the other versions’ variables can’t be read yet, the list shows the live version’s variables, with one line at the top saying why:
  • Showing the live version’s variables. Finish every A/B split in this workflow to see the rest. This node’s split, or another A/B split in the workflow, isn’t complete yet.
  • Showing the live version’s variables while the other versions load.
  • Showing the live version’s variables. Couldn’t load the other versions’ variables. It tries again on the next save.
Mappings that no version in the test uses are listed under No longer used by these versions, where you can remove them. Until the versions have been checked they are listed under Already mapped and can’t be removed yet, with one of these lines saying why:
  • Finish the split to check which versions use these. The split still has an empty row, a weight outside 1 to 99, or a total other than 100.
  • Finish every A/B split in this workflow to check which versions use these. Another node’s A/B split isn’t complete yet.
  • Checking which versions use these… The check is running.
  • Couldn’t check which versions use these. It tries again on the next save.

Changing versions during a test

The test follows each version by its name, so ordinary work on the agent doesn’t break it.

Deleting a version in a test

Delete agent version returns 409 while a workflow call that has been accepted but hasn’t connected yet is waiting on a save of that version, for example one held until calling hours, with the message This version has workflow calls waiting to be placed. Take it out of the workflow and let those calls connect first. It also returns 409 while an A/B batch that hasn’t finished uses the version. Try again once those calls have connected. After a version in a test is deleted:
  • New calls on its arm run the live version instead. Workflows already published, and campaigns already running, keep going.
  • The workflow can’t be published until the node is fixed: validation reports unknown_agent_version at that arm’s config.version_split.arms[<i>].agent_version_id, such as A/B arm 2 (B): agent version '<id>' not found. Choose another version for the arm, or turn the test off.
  • Calls that ran on the live version in its place show in the comparison as their own row for that arm, under the live version’s name, captioned for example Ran live. B’s version was deleted.
To take a version out of a test without deleting it, choose another version for its arm or turn the test off, and publish. A campaign keeps the split it was created with, so its calls on that arm keep using the version.

Running campaigns keep their split

The split is part of the published workflow version. A campaign keeps the version it was created with, so changing the weights or the versions in the test and publishing again applies to new campaigns and new runs, not to campaigns already created. Edits saved into the versions themselves do reach running campaigns, as described above.

Branch on the version

Every agent node, with or without an A/B test, writes two more outputs whenever it places a call: Branch on the name. Each save into a version gives it a new id, while its name stays the same until you rename it:
A later node reads the value as node.<agent_node_id>.call.agent_version_name, even after another agent node has run. See Conditions and variables.

See which version a contact got

  • Journey timeline. Open a contact’s journey from the campaign page. A step of an A/B-tested node shows a chip with the arm’s letter and the name of the version that ran, for example B, v2-short-greeting. The chip always names the version that ran: a call placed on the live version in place of a deleted one shows the live version’s name, with no separate label. Steps of other agent nodes have no chip; their version is in the step’s outputs.
  • Execution API. The node’s attempt in Get workflow execution carries the two call.* outputs.
  • Execution webhook. The node’s entry in the webhook trail carries them too, and on an A/B-tested node inputs adds version_split.agent_version_id:
inputs.version_split.agent_version_id is the version the node picked, exactly as it appears in arms. The outputs are the version that ran: the picked version’s latest save, so its id can differ from the one in arms after edits, or the live version when the picked one was deleted.

Compare the versions

On the campaign page, each A/B-tested node has its own card, titled A/B test: and the node’s name, such as A/B test: Qualification call. Open it to load the numbers, and click Refresh to update them; Updated shows how long ago they were loaded. Set shows the split you configured and, once calls have been placed, Actual shows the split calls really took, each with its shares such as 50 / 30 / 20. Until then the card reads No calls yet. Results appear as calls finish. The table has one row per version, and each column sorts: Hover a percentage to see the count behind it, such as 731 of 1,204 calls. A cell with nothing to divide by, such as Avg call for a version with no completed calls, shows NA. Call columns update as calls finish. Journey columns count settled journeys only: a contact still partway through the workflow counts once its journey ends. Each contact counts toward the version of its first visit to the node, and only when that visit placed a call: a contact whose first visit placed none, for example because the dial was rejected, isn’t counted in the journey columns, even if a later visit called. A journey that stopped without reaching an end node, such as one you canceled, is settled but has no outcome, so the three outcome columns can add up to less than all settled journeys. The card’s footer reads Calls update as they finish. Journey results count settled journeys. With Keep each contact on one version off, the journey columns are left out and the footer reads Calls update as they finish. followed by Journey results need ‘Keep each contact on one version’, because a journey can use several versions. Workflow calls are agent calls, so the other ways to compare versions work for them too: the Version filter and column in call history, the agent_version_id filter on Get agent executions, and the agent_version_id column in Download Records. See Compare the results.

Comparison API

The same numbers come from GET /workflow-campaigns/{campaign_id}/ab-tests, worked out each time you call it:
All values are counts. Work out percentages and averages from them: the completion rate is completed / calls, and the average call is duration_s / completed.

Next steps

Agent versions

Save, edit and test versions of an agent

Workflow nodes

Every agent node setting and output

Campaigns

Run the workflow over your contacts

Execution webhook

Get each contact’s journey, version included, when it finishes