01 / UNDERSTAND
What is A/B testing?
A/B testing randomly assigns eligible units to a control or changed experience, then compares outcomes over a defined period. Random assignment supports causal interpretation, provided implementation and analysis are appropriate.
The unit might be a person, account or organisation. Returning users usually retain their assigned experience. A shared workspace may require organisation level assignment because the change can influence other people.
Why use A/B Testing?
An experiment replaces a behavioural forecast with an observed comparison. It estimates whether a change improves the chosen outcome, harms another or remains too uncertain to justify action.
When to use it
USE IT WHEN
The proposed change has a measurable outcome that would justify implementation.
Traffic and measurement can support the required sample and duration.
Both experiences are acceptable, with a response to operational harm.
When to choose another method
Use Usability Testing to investigate why people struggle.
Use First Click Testing to examine an initial navigation choice.
Use HEART Framework to connect metrics with experience goals.
What you’ll need
Preplan the hypothesis, population, assignment unit, primary metric, guardrails, analysis and stopping rules. Estimate baseline and variability from reliable data. Define the smallest worthwhile effect and calculate sample size for agreed error rate and power. Power is a planning assumption, not a guarantee of significance. Allow outcomes to mature and relevant usage cycles to occur.
What makes an experiment interpretable
Keep the elements that help answer your question and support a decision.
The change, proposed mechanism and predicted measurable consequence.
The stable entity randomly allocated, with possible interference considered.
One preplanned metric with an explicit event, denominator and observation window.
Outcomes that could make the change unacceptable despite a primary improvement.
A justified sample requirement, duration and outcome maturation period.
How effect size, uncertainty, operational evidence and costs will inform action.
02 / PUT IT INTO PRACTICE
How to plan and run an A/B test
Keep evidence and assumptions distinct. Revisit your work when new information changes your understanding.
01
State the decision
Define worthwhile benefit and unacceptable harm. Connect measurement to the user’s goal.
Count confirmed bookings rather than button clicks.
02
Choose the comparison
Describe control and treatment. Record other releases or experiments that could interact.
Changing wording and layout together estimates their combined effect.
03
Specify assignment
Choose a stable assignment unit and eligibility rule. Check interference between units.
Assign shared household accounts together.
04
Plan measurement and power
Lock metrics, sample calculation, duration and analysis.
Specify whether the target difference is absolute or relative.
05
Validate implementation
Check allocation, exposure and logging, using an A/A test where appropriate.
Reconcile events with actual booking confirmations.
06
Run integrity checks
Monitor harm and data quality; investigate sample ratio mismatch. Follow the stopping method.
Stopping when a conventional p value looks favourable can invalidate inference.
07
Interpret and decide
Report effect size, uncertainty and guardrails. Distinguish improvement, harm and inconclusive evidence.
A wide interval does not establish equivalence.
Example: making a collection booking clearer
Fictional scenario: a collection service replaces “Continue” with wording explaining the next step. It predicts fewer abandoned bookings and consistently assigns resident accounts to current or revised wording.
The preplanned primary outcome is a confirmed collection per eligible assigned account. Support requests and cancellation behaviour are guardrails. Baseline data determines sample and duration; this example invents no conversion rate.
Suppose clicks increase but confirmations remain uncertain. The team would inspect the flow rather than declare success from clicks. These proposed events and decisions are illustrative, not findings.
03 / BELOW THE SURFACE
Below the surface: the metric chooses the kind of improvement
The experiment can estimate an effect correctly while the decision still rewards the wrong behaviour.
VALUE
What makes this effect worth acting on?
Name the smallest practical benefit and its implementation cost.
HARM
How could the metric improve while the experience worsens?
Define an undesirable path and a guardrail.
INFERENCE
What conclusion does this design actually support?
Limit the claim to its population, treatment and observation period.
A metric is an agreement about value. More confirmations might reflect clearer information, or the removal of information residents need to choose appropriately.
Ask what an apparent improvement could conceal. For the fictional service, residents might confirm unsuitable dates and then contact support. That suggests a guardrail and a qualitative question.
The average can also conceal a smaller group’s difficulty. Preplan consequential subgroup questions where the sample permits credible analysis. Label later exploration as exploratory. Avoid turning many slices into selective proof; connect the shipping decision with the service’s responsibilities.
A/B Testing companion worksheet
The original two page companion worksheet documents an experiment plan and evidence review. It connects assignment, metrics and timing with integrity checks, effect estimates, guardrails and a decision.
Practical tips
Review event definitions with logging and analysis owners.
Document every material deviation from the plan.
Treat inconclusive results as information about uncertainty.
Common mistakes
Changing the primary metric after seeing which result looks strongest.
Counting repeated visits as independent units when assignment occurs by person.
Calling an insignificant difference proof that the experiences are equivalent.
What you’ll get
A treatment effect estimate with uncertainty and limitations. The useful result is a defensible decision, including further investigation when the experiment cannot resolve the question.