By Scott Watermasysk
We just shipped KickoffLabs Smart A/B Testing! This is going to double the number of conversions captured for customers running A/B tests. Want to learn how? Read on…
Smarter A/B Testing is now the default!
We were very reluctant to add A/B testing to KickoffLabs. It’s a pretty recent addition for us.
We firmly believed in the principals behind A/B testing and run experiments at KickoffLabs all the time. However, we’ve always had three major concerns that we don’t believe any solution has really addressed.
When we finally released A/B testing at KickoffLabs we intentionally kept it simple. Our goal was not to compete head to head with dedicated A/B platforms like VWO and Optimizely. We simply wanted to see how our customers would use the tests and how we could improve their results over time.
Turns out that our customers love the feature, but a lot of them still see the three issues listed above. People see favorable results and call tests too early that are not statistically relevant and they also lose a lot of conversions with bad tests.
Fast forward six months. Josh and I were both in attendance at Microconf 2015 when the always excellent Joanna Wiebe of CopyHackers was dropping some impressive copy knowledge. In passing she mentioned Multi-Arm Bandit Experiments.
The name “multi-armed bandit” describes a hypothetical experiment where you face several slot machines (“one-armed bandits”) with potentially different expected payouts. You want to find the slot machine with the best payout rate, but you also want to maximize your winnings. The fundamental tension is between “exploiting” arms that have performed well in the past and “exploring” new or seemingly inferior arms in case they might perform even better. There are highly developed mathematical models for managing the bandit problem, which we use in Google Analytics content experiments.
We (ok, I) quickly read the research and knew this is exactly what we needed to help customers make the most of A/B testing with limited traffic.
So we built it!
Every lead is vitally important to your business. You work very hard to get those potential leads to your landing pages. A/B testing does an excellent job of helping you optimize your conversion process. However, an unfortunate consequence of this is that some of your potential leads are lost in the validation process.
Using the Multi-Arm Bandit algorithm helps minimize this waste. Our early calculations proved that it could lead to nearly double the actual number of conversions for our customers running A/B tests.
You can now have your cake and eat it too.
Nothing. Simply create a new A/B test and keep all the defaults.
No problem. Just click the ‘use manual weighting’ checkbox and you are back in the drivers seat.