I love this discovery I found on the Harvard Business Review blog.
Just under a third of companies with 1-10% growth preferred experimentation over other methodologies, such as statistical analysis, to identify revenue and operational improvement opportunities. Approaching half (46%) of companies with 11-20% growth prefer it, while well over half (56%) of companies with more than 20% growth do. – HBR Blog – 8 Ways to Democratize Experimentation
More experiments generally equate to more growth and the HBR blog explains 8 tips for making experiments more successful. But I think they missed one critical tip:
You need to experiment in public to accurately gauge customer reaction to the experiments.
Here’s why the internal only test is invalid.
- Your company will produce an echo-chamber of self congratulatory praise on similar ideas. Most companies preach “diversity”, but miss hiring for diversity of thought.
- Without potential customer praise you don’t know if an experiment will really lead to growth.
- Internal only testing with strong management presumes that company management knows better what the market needs than customers. But if a corporate manager knew how to predict the market they wouldn’t be a middle manager in corporate America.
- Internal only experimentation was the Enron policy. That worked out great for them.
Consider the case of Google Wave. I’m sure it went over like gangbusters inside google. The type of person they hire would LOVE it. But those people are not their best paying customers. If they had gotten information about Wave out there sooner they would have realized that a market didn’t really exist for it.
I’m also willing to bet there has been a lot of lost opportunity inside of Google because of ideas that were killed by a negative selection bias.
I realize that transparency is a scary thing… but what’s more scary: Being transparent or wasting millions of dollars and 3 years of 30 people developing a failed product for an echo-chamber of executives that thought they knew better?
Real democratization happens in public. Start public, stay public, and ship with the public.