How to make experimentation work for product managers
If you work in a medium to large company, your CEO wants the organization to be more innovative. All the surveys about such things tell us this is the case. If you are in a smaller organization or on your own, you still care about innovation. But, how can you bring about more innovation, or let’s face it, in some cases, anything that looks like innovation?
In some form, this is what company leaders have talked with me about the most in the last year — incorporating more innovation into their work. It is something we especially need now.
I believe a key paving stone on the path to better innovation – aka, more products customers love — is adopting an experimentation mindset and conducting fast low-cost experiments. And that is what our guest, Dr. Stefan Thomke, Harvard Business School professor and researcher, is discussing with us. He is an authority on the management of innovation and the use of experimentation.
Summary of some concepts discussed for product managers
[2:29] Why is it important for innovative organizations to run experiments?
One of the big challenges in innovation is uncertainty. When we rely on our experience, we’re wrong 9 out of 10 times. Big data only shows correlations and not causations. The answer to uncertainty is experiments. Experimentation is the engine of innovation.
[6:53] How should experimentation be used in innovation?
It’s not experimentation versus data or experimentation versus experience. It’s experience plus data plus experiments. Companies that experiment still do normal research activities, but they don’t run to market with the research before it’s tested. Instead, they use research to generate hypotheses which they then test with experiments.
[10:53] What is the structure of a good experiment?
In an experiment, the tester wants to separate an independent variable from a dependent variable while holding everything else constant. The independent variable is the presumed costs, such as a bonus for the sales team, and the dependent variable is the observed effect, such as the revenue of the sales team. You change the independent variable, observe changes in the dependent variable, and make a conclusion about cause and effect.
To minimize bias, ideally an experiment should be blind and randomized. A blind experiment in which the person doesn’t know they’re being experimented on ensures that their knowledge of the experiment doesn’t change their behavior. Randomization distributes random variables among all the people so that bias is evened out.
[14:21] What’s an example of using experimentation?
Imagine you have a restaurant and want to find out if a new menu will increase your revenue. The wrong way to set up an experiment is to run the normal menu for a month and then the new menu for a month and compare your revenue. There are too many variables that could have changed from one month to the other. The right way is to create a randomized experiment by running both menus at the same time. As customers come into your restaurant, they’re randomly assigned to one menu or the other. Other variables, such as the chefs in the kitchen, are also randomized.
[18:55] How do we identify the hypotheses that need to be tested?
Hypotheses can come from many places, such as qualitative research, customer insights, or data mining, or even intuition. Good hypotheses should be grounded in facts, identify possible causes and effects through clearly measurable variables, and be able to be falsified.
[23:04] How does experimentation give us confidence?
Experimentation gives us confidence to be bold. Without experiment, you’re just gambling. Many times your intuition can get in the way. For example, Ron Johnson, who created the Apple store, was asked to become the CEO of JCPenney. He did everything that had worked at Apple, and a few months later JCPenney was fighting for survival, and Ron Johnson was out of his job. He had relied on his experience without testing.
[27:12] What are the steps of testing hypotheses?
First, the executive or leader sets the grand challenge–what’s the big question you’re trying to answer? Then people can break the big question down into testable hypotheses that you can individually pursue. You may test multiple touchpoints in isolation first, then bring them together to consider the interaction between them.
Next, you run experiments. In a digital setting, you need a good tool that allows you to write a hypothesis in the morning and run the test by lunch. You may have to wait a couple of weeks before you have enough results, but if you keep launching tests, you’ll have a steady stream of results coming in.
[32:23] What are the two main kinds of experiments?
Exploratory and optimization. Exploratory experiments give directionality and general insight. Optimization experiments optimize touchpoints and give you insight about cause and effect.
What are some myths about experiments?
One myth is that experimentation kills intuition. Actually, experimentation complements intuition. Testing tells you where you’re wrong and where you’re right, but if you rely on intuition alone, you’re more likely to be wrong than right.
Another myth is that you don’t need experiments if you have big data. The problem is, you often don’t have data when you’re trying to do something new, and data only shows correlations, not causations. The only way to determine causality is by running experiments.
Some people believe that experiments have to be time-consuming and expensive. Experiments can be very fast and inexpensive, especially using simulation tools available today.
Brick-and-mortar companies often think they can’t do experiments. Many brick-and-mortar companies are already doing it. There are great tools available for experimentation in a brick-and-mortar environment. And even if you’re brick-and-mortar, you’ll have some digital touchpoints with customers, and you’ll have to do experimentation to design those touchpoints. As Mark Okerstrom said, if you don’t do large scale experimentation, you’re dead.
- Stefan’s book, Experimentation Works
- Connect with Stefan through his LinkedIn profile
- Stefan’s Harvard profile
“It doesn’t make any difference how beautiful your guess is, it doesn’t make any difference how smart you are, who made the guess, or what his name is. If it disagrees with experiment, it’s wrong. That’s all there is to it.” – Richard Feynman
Thank you for being an Everyday Innovator and learning with me from the successes and failures of product innovators, managers, and developers. If you enjoyed the discussion, help out a fellow product manager by sharing it using the social media buttons you see below.