A/B experiment plan (A/B test plan)
Agree what you are testing, how you will measure it and when you will stop, before a single person sees either version.
- Time
- 45 min
- People
- 3–5
- Kind of work
- Test
In short
An A/B experiment plan is a short, agreed document written before you compare two versions of a service with real users. It states one hypothesis, the two variants, the main metric, how people are allocated between versions, and the rule for stopping. Writing it first protects the result from wishful reading later.
What it is
An A/B experiment shows version A to one group of people and version B to another, at the same time, and compares what they do. Because people are assigned at random, a difference in the outcome can be traced to the difference between the versions. The plan is the part you do before any of that: the decisions that make the result trustworthy.
Most failed experiments fail in planning. Teams change two things at once, pick the metric after seeing the data, or stop the test the day it looks good. A one-page plan with five fields prevents most of this. It also forces a useful conversation with an analyst about whether you will have enough data to learn anything at all.
When to use it
Use it when
- You have two credible versions of a page, message or step and want to know which works better for real users.
- The service has enough traffic or volume for a difference to show within a few weeks.
- A decision is contested and the team wants evidence it agreed on in advance.
- You can measure the outcome you care about automatically and reliably.
Skip it when
- Volumes are small, such as a service used by a few dozen people a month; the test will not reach a clear answer.
- You do not yet know why people struggle; run a moderated usability test first to find the problem.
- Showing one group a worse version could cause real harm, such as missed medical or legal deadlines.
You need
- Time
- About 45 minutes to write the plan together, plus time with an analyst to check data needs.
- People
- Three to five: the service owner, a designer or content writer, someone who can build the variants and an analyst.
- Materials
- The worksheet with Hypothesis, Variants, Metric, Allocation and Stop rule; current figures for the metric; a description or mock-up of each version.
How to run it
Step 1: Write one hypothesis
State the change and the effect you expect, and why. For example: "If we show the total fee before the form starts, more applicants will complete payment, because surprise at the fee is the main reason people leave." One hypothesis per experiment keeps the result readable.
Step 2: Describe the two variants
Write what A (the current version) and B (the new one) look like, side by side. Change one meaningful thing only. If B differs in wording, layout and timing at once, you will not know which change made the difference.
Step 3: Choose the main metric and one guard metric
Pick the single measure that decides the result, such as the share of started applications that reach payment. Add one guard metric that must not get worse, such as calls to the help line. Write the current value of each.
Step 4: Agree allocation with an analyst
Decide how people are split, usually half and half at random, and who is included. Ask the analyst how many people each version needs to detect the smallest difference worth acting on, and how long that will take at current volumes.
Step 5: Set the stop rule
Write when the test ends: a fixed date or a fixed number of people, whichever the analyst recommends. Add the conditions for stopping early, such as the guard metric getting worse by a set amount. Do not stop because the result looks good on day three.
Step 6: Decide what each result will mean
Before launch, write what you will do if B wins, if A wins, and if there is no clear difference. This turns the experiment into a decision and stops the team from rerunning it until it gives the answer they wanted.
Step 7: Get sign-off, then launch
Share the one-page plan with the decision maker and the analyst and freeze it. Any change after launch, such as a new metric, is recorded as a change with the reason.
Worked example
A retail bank wanted more customers to finish opening a savings account online. About 40 percent left at the step asking for proof of address. The digital team believed people did not know which documents counted.
Hypothesis: if the step lists accepted documents with small pictures, more applicants will upload one and continue, because uncertainty about what counts is the main reason they leave.
Variant A was the current upload step. Variant B added a list of four accepted documents with images. Nothing else changed. The main metric was the share of people reaching the step who completed it; the guard metric was the share of uploads rejected by staff.
The analyst estimated that with around 3,000 people reaching the step each week, a 50/50 split would need three weeks to detect a five-point improvement. The stop rule was three full weeks, with an early stop if rejected uploads rose by more than a quarter.
After three weeks, completion rose from 60 to 66 percent in B and rejected uploads did not change. As agreed in the plan, the team rolled out B to everyone and moved on to the next step in the journey.
IllustrativeTips and mistakes
Tips
- Write the plan in plain language that a manager outside the team could read in two minutes.
- Run both versions over full weeks, because behaviour on weekends and weekdays often differs.
- Check that the split works before you start by sending a small share of people through both versions.
- Keep a simple log of experiments and their results so the organisation stops retesting old ideas.
Common mistakes
- Choosing the metric after seeing the data, which almost always finds something that looks like a win.
- Stopping early because the result looks good, before enough people have seen both versions.
- Changing several things in B, so a win cannot be explained or repeated.
- Running a test with too little traffic and treating no clear difference as proof the change does not matter.
What you walk away with
You walk away with a signed-off, one-page plan: a single hypothesis, two clearly described variants, a main and a guard metric with their current values, an allocation agreed with an analyst, and a stop rule with the decision each result will trigger. Once the experiment runs, record the outcome against the plan. If you are unsure which metric really drives the outcome, build a KPI tree first, and if the change is still a guess, frame it as a hypothesis test.
Questions people ask
How long should an A/B test run?
Long enough to reach the number of people your analyst calculated, and at least one or two full weeks to cover normal weekly patterns. Agree the length before launch. Ending the test as soon as one version looks ahead is the most common way to get a false result.
What is the difference between an A/B test and a hypothesis test?
A hypothesis test is any small test of one belief, and it can be an interview, a prototype or a count. An A/B experiment is one specific kind: two versions shown to randomly split groups of real users and compared on a metric. Use it when you have the volume.
Do I need a statistician to plan an A/B test?
You need someone who can estimate how many people each version needs and how to read the result. That may be an analyst rather than a statistician. The rest of the plan, the hypothesis, variants and decisions, belongs to the service team.
Tool card
The worksheet your team fills in, one box per field.
- Hypothesis
- Variants
- Metric
- Allocation
- Stop rule