Guides
A/B testing your agent
Switchback testing, and how to read its results honestly.
Testing here is a switchback test, and the distinction matters for reading your results.
One variant is live at a time#
The exposure window alternates on a timer, and every call is attributed to whichever arm was live when it started. Two callers ringing at the same moment hear the same thing.
Reading the verdict#
- No verdict is offered before 30 calls per arm.
- Confidence intervals are Wilson intervals, which stay inside 0–100% at the small numbers a single business produces.
- Average handle time is shown but deliberately not tested — a p-value we did not compute would be worse than none.
Worth knowingStopping a test publishes the untouched configuration; it is not an undo that has to remember what it changed. Only one test can run per agent at a time.