Menu
Cart

Napkin Folding — A/B testing

Bayesian Methods for Hackers release!

Posted by Cameron Davidson-Pilon at

Finally, after a few years writing and debugging, I'm proud to announce that the print copy of Bayesian Methods for Hackers is released! It has update content, including a brand new chapter on A/B testing, compared to the online version.      You can purchase it on Amazon today! 

Read more →

The Class Imbalance Problem in A/B Testing

Posted by Cameron Davidson-Pilon at

Introduction If you have been following this blog, you'll know that I employ Bayesian A/B testing for conversion tests (and see this screencast to see how it works). One of the strongest reasons for this is the interpretability of the analogous "p-value", which I call Confidence, defined as the probability the conversion rate of A is greater than B, $$\text{confidence} = P( C_A > C_B ) $$ Really, this is what the experimenter wants - an answer to: "what are...

Read more →

The Binary Problem and The Continuous Problem in A/B testing

Posted by Cameron Davidson-Pilon at

Introduction I feel like there is a misconception in performing A/B tests. I've seen blogs, articles, etc. that show off the result of an A/B test, something like "converted X% better". But this is not what the A/B test was actually measuring: an A/B test is measuring "which group is better" (the binary problem), not "how much better" (the continuous problem). In practice, here's what happens: the tester waits until the A/B test is over (hence solving the binary problem),...

Read more →