A/B testing mostly doesn't work the way most people do it. When the thing you're looking for is rare (a meaningful conversion lift), false positives are more common than real positives — even at 95% statistical confidence. The Shopify CTO confirmed this. Read the book.