How do I A/B test a subject line or email in a campaign?
On the campaign's Sequence tab, open an email step and click Test another version of this email. You then have version A and version B side by side, and you can add up to four versions in total with Add version C and Add version D. Warmerly sends the versions in rotation, and once every version has had enough sends it picks a winner and uses only that one from then on.
Steps
- Open the campaign's Sequence tab and expand the email step you want to test.
- Click Test another version of this email.
- Edit each version's subject and body. Change one thing between versions (the subject, or one part of the body) so you can tell what made the difference.
- Set Decide after N sends each. Every version must reach this number of sends before a winner can be chosen. Larger is more reliable and slower.
- Launch, or keep the campaign running. Progress is shown under the versions.
How versions are handed out
Each new lead gets whichever version has been sent the least so far, so the split stays close to even across however many versions you have.
While a test is running, the versions are the step's copy. What you typed in the step before enabling the test becomes version A. If you turn the test off, the winner (or version A if there is none yet) goes back into the step.
How the winner is picked
- Every version must have reached the Decide after number of sends.
- The last email of each version must be at least 24 hours old, so recipients have had time to answer.
- Replies decide it. As soon as any version has a reply, the highest reply rate wins. Only genuine replies count, not out-of-office auto-replies.
- Opens are the fallback. If no version has a reply yet but some have opens, the highest open rate wins. Opens are only recorded when Track opens is on for the campaign, and they are unreliable, which is why replies come first. See Open and click tracking.
- If there are no replies and no opens at all, nothing is picked. Warmerly will not crown a winner with no evidence.
When a winner is chosen, the step shows "Version X won" and every later send uses it. The losing versions can no longer be edited. Results per version (Reply rate, Open rate and a Winner badge) appear on the campaign's Overview tab under Variant performance.
What it cannot do
- The winner is simply the highest rate. It is not a statistical significance test, so a small sample can pick a version that is not really better. Use a sample size big enough to trust: a rate off a handful of sends is noise.
- If you change both the subject and the body between versions, a winner tells you which email did better, not which change did it. The editor warns you about this, and warns if two versions are identical.
Common problems
The test never picks a winner. Check that every version has reached the sample size, that at least one reply exists (or that open tracking is on and at least one open exists), and that 24 hours have passed since the last send of each version.
Open tracking is off, so the test is slow. Then replies decide it. That is the better signal, it just needs more sends to settle.
I need the copy in one place. After a winner is chosen, turn the test off and the winner goes back to being the step's own copy.