A/b testing: the data dictator
The brutal reality: your opinion is irrelevant
In a high-growth company, the CEO's “gut feeling” is the enemy of progress.
The Conflict: Humans are terrible at predicting what users want.
We fall in love with our own bad ideas.
The Truth: A/B testing is a battle between two versions of reality.
The data is the only judge.
The Fix: Kill your darlings.
If Version B increases conversion by 0.5%, Version A is dead.
No arguments.
1. velocity over accuracy
The company that runs the most experiments wins.
Don't wait 4 weeks for a “perfect” test.
Run 10 “messy” tests a week.
Even a failed test is a victory because it tells you exactly what not to do next time.
2. the big rocks
Don't waste time testing the color of a button.
Test the Offer, the Price, and the Headline.
Changing a color gives a 1% lift.
Changing an offer can give a 300% lift.
Smart words
A/B TESTING
The “Duel.” Comparing two versions of a page to determine which one wins in the real world.
STATISTICAL SIGNIFICANCE
The “Certainty.” The mathematical proof that a result wasn't just a lucky coincidence.
CONTROL GROUP
The “Baseline.” The group that sees the original version of the product.
Tactical directives
1. The Headline Test: Start an A/B test on your primary landing page headline by the end of the day.
2. The Goal: Commit to running at least 3 experiments per week, every week.
3. The Failure Log: Keep a public log of every failed test.
Document the lesson learned so you don't repeat the mistake.
Optional vocabulary