What is statistical significance in A/B testing?

Statistical significance is the probability that an observed difference between variants isn't due to chance. The standard threshold is 95% confidence (p < 0.05). Most teams call winners far before they reach significance — which is why most A/B test 'wins' don't replicate.

Why are most A/B tests underpowered?

Because the sample size needed to detect realistic lifts (1–5%) is much larger than teams expect — often tens of thousands of conversions per variant. Underpowered tests either miss real effects or declare noise as signal. The CRO chapter shows the math.

What does the CRO chapter mini-game (Landing Page Lab) do?

You ship landing page variants against a baseline, watch real-time conversion data accumulate, and decide when to call a winner. The game exposes how easy it is to declare false wins on small samples — and how patient you actually need to be.

What's the difference between optimization and conversion theater?

Optimization is changing things that meaningfully move the metric, validated by powered tests. Theater is shipping tweaks (button colors, hero copy) that feel productive but can't statistically move the needle. Most CRO programs are mostly theater.

Chapter 06 · CRO — Growth Marketing

Growth / A Story

Ch. 06 · CRO

Loading run_