By Josh Ledgard
Just under a third of companies with 1-10% growth preferred experimentation over other methodologies, such as statistical analysis, to identify revenue and operational improvement opportunities. Approaching half (46%) of companies with 11-20% growth prefer it, while well over half (56%) of companies with more than 20% growth do. – HBR Blog – 8 Ways to Democratize Experimentation
More experiments generally equate to more growth and the HBR blog explains 8 tips for making experiments more successful. But I think they missed one critical tip:
You need to experiment in public to accurately gauge customer reaction to the experiments.
Here’s why the internal only test is invalid.
Consider the case of Google Wave. I’m sure it went over like gangbusters inside google. The type of person they hire would LOVE it. But those people are not their best paying customers. If they had gotten information about Wave out there sooner they would have realized that a market didn’t really exist for it.
I’m also willing to bet there has been a lot of lost opportunity inside of Google because of ideas that were killed by a negative selection bias.
I realize that transparency is a scary thing… but what’s more scary: Being transparent or wasting millions of dollars and 3 years of 30 people developing a failed product for an echo-chamber of executives that thought they knew better?
Real democratization happens in public. Start public, stay public, and ship with the public.