How to Run an A/B Test on Your Website or Emails (Without the Jargon)

You change your homepage headline, and sales go up the next week. Was it the headline? Or payday, the weather, a competitor running out of stock or a mention on a local news site? Without a controlled test, you can’t tell.

An A/B test, also called a split test, removes the guesswork. You show 2 versions of a page or email to randomly chosen groups at the same time, then compare how each group behaves. Because the groups are random and run side by side, outside factors affect both equally, so the difference you see is far more likely to be caused by your change.

This guide walks through how to run one properly, from hypothesis to verdict, with as little statistics jargon as possible.

What you can A/B test

Almost anything a visitor or subscriber sees can be tested. Common starting points:

For emails, judge on clicks or sales rather than opens. Apple’s Mail Privacy Protection can load email content in the background, which makes open counts less reliable. Our guide to writing email subject lines has ideas worth testing.

Key takeaway: A good A/B test changes 1 thing, measures 1 main goal, splits people randomly and runs until a sample size you chose in advance. Deciding the stopping point before you start is what keeps you from fooling yourself.

Step 1: Pick one goal metric

Choose the single number that decides the winner before you start. Researchers call it the overall evaluation criterion. For a landing page, it might be completed bookings. For an email, it might be clicks to your offer.

Pick a metric close to revenue. A new button color might raise clicks while lowering purchases if it attracts curious but unready visitors. You can track secondary metrics too, but agree up front which one decides the result.

If you’re not sure which numbers matter most for your business, start with our checklist of marketing KPIs.

Step 2: Write a hypothesis

A hypothesis is a short, testable prediction with a reason behind it. Use this template:

“Because [what we observed], we believe [change] will cause [effect on goal metric] for [audience].”

Example: “Because visitors tell us they don’t know if we serve their area, we believe adding a service-area line under the headline will increase quote requests from mobile visitors.”

The “because” matters. It forces you to base tests on evidence such as customer questions, analytics or session recordings instead of hunches. That evidence-first approach is the heart of conversion rate optimization.

Step 3: Estimate how many visitors you need

This is the step most small businesses skip, and it’s why many tests “find” winners that aren’t real.

The plain-English version: the smaller the improvement you want to detect, the more people you need. And it isn’t a straight line. Halving the change you want to detect roughly quadruples the visitors you need.

A widely cited guide to online experiments by Ron Kohavi and colleagues at Microsoft gives a rule of thumb for planning, assuming 95% confidence and 80% power (the usual standards). For a conversion rate, it works out to:

Visitors per version ≈ 16 × p × (1 − p) ÷ (change to detect)²

Here p is your current conversion rate as a decimal, and the change is the absolute difference you want to be able to spot.

The paper’s own example: with a 5% conversion rate, detecting a 5% relative change (from 5% to 5.25%) needs just under 122,000 users per version. Detecting a 20% relative change (from 5% to 6%) needs about 7,600 per version.

What this means for a small site:

• Test bold changes, not tiny tweaks. A new offer or page layout can move results enough to detect. A slightly different shade of blue usually can’t.
• Test high-traffic pages or big email lists first.
• Use an online sample size calculator or your testing tool’s built-in calculator to plan, and write the number down.

If your traffic is too low to reach the sample size within a few weeks, skip formal A/B testing for that page. Make the change based on customer research, then watch your numbers.

Step 4: Build the variant and split traffic randomly

Create version B with the single change from your hypothesis. Then split people randomly, usually 50/50, so each group is comparable.

For websites, use an A/B testing tool or the experiment feature in your website platform. Google retired its free tool, Google Optimize, on September 30, 2023, so you’ll need a third-party testing tool or one built into your site builder or ecommerce platform. Make sure each visitor keeps seeing the same version on repeat visits.

For emails, most email marketing platforms have a built-in A/B or split test feature. A common setup sends both versions to a test portion of the list, then sends the winner to the rest. Make sure the test portion is large enough to be meaningful.

Protect your search rankings. Google’s Search Central guidance for website testing says to avoid cloaking (showing Googlebot different content from users), to use rel=”canonical” on alternate test URLs pointing to the original, to use temporary 302 redirects rather than permanent 301s and to remove test code once you’re done.

Before launch, run a quick check: preview both versions on mobile and desktop, submit test forms and confirm your analytics records conversions for each version.

Step 5: Run the test for the full planned time

2 rules matter most here.

Run for whole weeks. Kohavi’s guide strongly recommends running experiments for at least 1 to 2 weeks, then in multiples of a week, because weekday and weekend visitors often behave differently. A test that runs Monday to Thursday can mislead.

Don’t peek and stop. Results swing wildly in the first days. If you check every morning and stop as soon as B looks ahead, you’ll often crown a winner that was just random noise. Decide the sample size and end date in advance, and stick to them.

Also avoid changing anything else mid-test, such as pausing ads to one page or launching a sale that only one version mentions.

Step 6: Read the results honestly

When the test reaches its planned size, compare the goal metric for each version. Your testing tool will usually report a confidence or probability figure.

In plain terms:

• Clear winner: B beats A and the tool reports high confidence (commonly 95%). Roll out B.
• No clear difference: The versions performed about the same. That’s a useful result. It means the change didn’t matter much, so pick the version that’s easier to maintain and test something bolder next.
• B lost: Keep A, and write down what you learned.

Common traps when reading results:

• Looking at too many metrics. Check 20 metrics and 1 will look “significant” by chance. The goal metric decides.
• Slicing until something wins. “B won on Tuesday for Android users in Ohio” is usually noise. Treat segment findings as ideas for the next test.
• Ignoring tracking problems. If one version shows far more visitors than the other in a 50/50 test, something is broken. Fix it and restart.
• Forgetting the novelty effect. Returning visitors may click a new design just because it’s new. Kohavi’s guide notes that some experiments need to run multiple weeks for this reason.

Where to start: your first A/B test this month

1. Choose 1 high-traffic page or your largest email list. Check traffic in GA4. Our guide to reading GA4 as a business owner shows where.
2. Find 1 problem from customer questions, reviews or where visitors drop off.
3. Write a hypothesis using the template above.
4. Estimate the sample size and how many weeks it will take.
5. Build the variant, QA it on mobile and set the split to 50/50.
6. Run it for full weeks without peeking-and-stopping.
7. Record the result in a simple test log: date, hypothesis, metric, sample size, result and next idea.

A test log becomes one of your most valuable marketing assets over time, because it stops your team from re-testing the same ideas.

Conclusion

A/B testing is simply a fair race between 2 versions. Pick 1 goal, write down why you expect a change to work, plan your sample size, split people randomly, run whole weeks and only then read the result. Small sites should test bold ideas on their busiest pages and lists. Done this way, every test teaches you something about your customers, even when the new version loses. For definitions of terms like statistical significance, visit the glossary.

Sources

• Kohavi, Longbotham, Sommerfield and Henne: Controlled experiments on the web: survey and practical guide (Data Mining and Knowledge Discovery, 2009) — sample size rule of thumb, 7,600 and 122,000 examples, full-week guidance, A/A tests, novelty effects
• Google Search Central: Minimize A/B testing impact in Google Search — no cloaking, rel=canonical, 302 redirects, removing test code
• Google Analytics Help: Google Optimize sunset — Google Optimize and Optimize 360 ended on September 30, 2023
• Apple: Mail Privacy Protection and Privacy — remote email content loaded privately in the background