What is A/B Testing?

A/B testing is a controlled experiment that sends two versions of one thing, such as a subject line, an email opener or a landing page, to randomly split audiences, then compares a result like replies or booked meetings.

At a glance

• Change one thing per test and keep everything else the same, including the list and the send times.
• Pick the result that matters before you start, ideally replies, meetings or pipeline, not opens.
• Decide the sample size in advance, then wait for it. Stopping as soon as a result “looks significant” produces false winners.
• Small B2B lists can only detect big differences. Test bold changes, not commas.
• Running more than two versions at once is sometimes called A/B/n or A/Z testing.

How A/B testing works

You write a hypothesis (“naming the prospect’s trigger event in line one will lift replies”), build two versions that differ only in that line, and split the list at random. Both go out at the same time so day and time don’t bias the result. When each version reaches the planned sample size, you compare them.

Evan Miller’s widely used rule of thumb for the sample size per version is n = 16 × p(1 − p) ÷ δ², where p is your current rate and δ is the smallest change you want to detect.

A worked example

Example (illustrative): your cold email gets a 5% reply rate. To detect a two-point change (5% to 7%), the rule gives 16 × 0.05 × 0.95 ÷ 0.02², or about 1,900 sends per version. To detect a five-point change, it’s about 300 per version.

• That’s why small outbound lists should test big ideas: a new angle, a new offer, a new persona.
• Small tweaks need samples most B2B lists don’t have.
• If the list can’t reach the sample, run the test longer or across several sequences with the same change.

Common A/B testing mistakes

• Peeking and stopping early. Miller shows that checking results repeatedly and stopping at the first significant reading can push the false positive rate from 5% to 26.1% in a worst case. Fix the sample size first.
• Changing two things at once. If the subject and the opener both changed, you don’t know which one worked.
• Judging on open rate. Opens are noisy. Judge on reply rate, positive replies or meetings.
• Splitting lists that aren’t alike. If one version goes to a better segment, you’re testing the list, not the message.
• Never writing down results. A test nobody records gets run again next quarter.

How A/B testing shows up in a revenue system

Message testing is part of the weekly work on outbound: a new opener goes to part of a list, the current one goes to the rest, and replies and meetings decide which stays. See how our sales pods work.

Terms that sit next to A/B testing

Most outbound tests run inside a sequence of cold emails, and the result is read in reply rate and conversion rate. Tests only mean something if email deliverability holds steady, because a version that lands in spam loses for reasons that have nothing to do with the copy.