Deliberate experimentation is more important than deliberate practice in a rapidly changing world.
Most people think that Edison invented the first light bulb.
They’re wrong.
In fact, Edison was spectacularly late to the game.
In 1878, when the 36-year-old inventor decided to focus on building a light bulb, 23 others had already invented early versions called arc lamps, some of which were being used commercially to light streets and large buildings.
So how did Edison win in such a crowded field when he was so far behind?
He and his team spent a year working day and night doing thousands of experiments. On October 21, 1879, they succeeded, creating a light bulb for everyday use in the home.
Edison would go on to pioneer five different multibillion-dollar fields with his invention factory: electricity, motion pictures, telecommunications, batteries, and sound recording. In today’s terms, you can think of Edison as Elon Musk, Jeff Bezos, and Mark Zuckerberg all rolled into one.
What was the key to Edison’s incredible success? In two words — deliberate experimentation. For Edison, building a company was synonymous with building an invention factory.
The technique is just as powerful today. “Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day,” Jeff Bezos has claimed. In a recent interview, Mark Zuckerberg explained, “One of the things I’m most proud of that is really key to our success is this testing framework … At any given point in time, there isn’t just one version of Facebook running. There are probably 10,000.”
Bezos and Zuckerberg aren’t saying that experimentation is one of many strategies. They are saying it is THE strategy. In this article, you’ll see how luminaries across many fields use deliberate experimentation and how you can use it to increase your odds of success in your professional and personal life.
Why 10,000 Experiments Beat 10,000 Hours
Perhaps the most popular current success formula is the 10,000-hour rule popularized by Malcolm Gladwell. The idea is that you need 10,000 hours of deliberate practice to become a world-class performer in any field.
Research now tells us, however, that this formula is woefully inadequate to explain success, especially in the professional realm. A 2014 review of 88 previous studies found that “deliberate practice explained 26% of the variance in performance for games, 21% for music, 18% for sports, 4% for education, and less than 1% for professions. We conclude that deliberate practice is important, but not as important as has been argued.”
This chart summarizing the results should cause any ardent believer in the 10,000-hour rule to pause:
This means that deliberate practice may help you in fields that change slowly or not at all, such as music and sports. It helps you succeed when the future looks like the past, but it’s next to useless in areas that change rapidly, such as technology and business.
What Edison and others (see more examples below) teach us is that we should maximize the number of experiments, not hours. Instead of the 10,000-hour rule, we need what I call the 10,000-experiment rule.
Throughout history, the scientific method has arguably produced more human progress than any other philosophy. At the heart of the scientific method is experimentation: develop a hypothesis, perform a test to prove the hypothesis right or wrong, analyze the results, and create a new hypothesis based on what you learned. The 10,000-experiment rule takes this proven power of experimentation out of the lab and into day-to-day life.
Following the 10,000-experiment rule means starting your day with not just a to-do list but a “to-test” list like Leonardo Da Vinci. According to Walter Isaacson, one of Da Vinci’s biographers, “Every morning his life hack was: make a list of what he wants to know. Why do people yawn? What does the tongue of a woodpecker look like?”
As you go through your day, following the 10,000-experiment rule means constantly looking for opportunities to collect data rather than just doing what you need to do. It means adding a deliberate reflection process based on reviewing data before the day ends.
For example, do you want to improve your sales results by asking a new question at the end of sales calls? Now every sales call becomes an opportunity to ask that question and collect data so that you can learn how to make better sales calls in the future. Do you want to sleep better so that you can have more energy during the day? You can research all the best practices for falling asleep, turn the most compelling ones into a routine, use a sleep tracker to get objective data on the quantity and quality of your sleep, and then make adjustments to your routine to improve the results.
To achieve 10,000 hours of deliberate practice requires three hours of deliberate practice per day for 10 years. I argue that the 10,000-experiment rule is just as difficult, yet doable, requiring three experiments per day.
Why 10,000 Experiments Yield Success According To Decades Of Academic Research
If Edison’s approach is universal, you would expect it show up repeatedly among top performers. As it turns out, the academic world has been studying the phenomenon for decades, and that’s exactly what they’ve found.
Researcher Dean Keith Simonton has spent his career studying the world’s preeminent creative achievers and painstakingly dissecting their careers to find patterns. To share his findings, he has published more than 340 academic articles and 13 books, including Greatness: Who Makes History And Why and Origins of Genius: Darwinian Perspectives on Creativity.
Two fascinating insights have emerged from Simonton’s (and others’) research. The first is that most innovative ideas are generated by a small number of superstars. In any given field, the top 10% of performers produce more than 50% of breakthroughs.
Why are these superstars so much more successful? Is it because their ideas are just superior from the get go? Here’s what is really fascinating: The answer is no. The second lesson to learn from Simonton’s research is that superstars produce just as many bad ideas as everyone else — they just produce more ideas overall. Having many more ideas means they have more failures but also more hits.
“What is especially fascinating is that creative individuals are not apparently capable of improving their success rate with experience or enhanced expertise,” Simonton has written. “Creative persons, even the so-called geniuses, cannot ever foresee which of their intellectual or aesthetic creations will win acclaim.”
In other words, the key to maximizing creative success, according to the theory, is producing more experiments.
Editorial Note: For a more nuanced understanding of the Blind Variation And Selective Retention, you can read Simonton’s 2011 Academic Paper, Creativity and Discovery as Blind Variation: Campbell’s (1960) BVSR Model After the Half-Century Mark.
From Health to Stand-up Comedy: The 10,000-Experiment Rule Applies Across Fields
When you consider many of the most important achievements across different fields, you often see this theory at play.
A Fast Company article written by advertising legends Ben Clarke and Jon Bond points out that thanks to a combination of new technologies and lean business approaches, the world’s most innovative businesses are running thousands of experiments more annually:
In academia, Einstein is best known for his paper on relativity, but he published 248 other papers. Paul Erdos coauthored more than 1,500 mathematical research articles during his career. 1,500! As you might expect, Erdos made significant contributions, and although most of his papers have been forgotten, a handful of them made him one of the most influential mathematicians of the 20th century! Now consider that fewer than 1% of scientists publish a paper every year.
In the world of entertainment, SNL, one of the longest-running TV shows in history, has a grueling weekly experimentation process of brainstorming, researching, and rewriting scripts. Only a tiny percentage of sketch ideas ever air. The iconic cartoons published by The New Yorker are the result of a process in which 50+ freelancers submit up to 10 sketches each for consideration per week:
Pixar, one of the most successful movie studios in history, developed 100,000+ storyboards (i.e., step-by-step plot sequences) for the film Wall-E’s ultimate plot. 100,000!
Those who enthusiastically embrace experimentation in their personal lives tend to reap significant benefits as well. Take, for example, Shonda Rhimes, producer and writer of Grey’s Anatomy, Scandal, and other hit shows. She set up an experiment she called The Year of Yes to confront her debilitating social anxiety, limit her workaholism, and accept herself. Instead of continually saying no to social experiences, she committed to saying yes for an entire year. Among the many lessons she learned from the experience was that to know what to focus on you first need to try many things.
Entrepreneur Jia Jang took something most of us fear — rejection — and made it into an experiment with his 100 Days of Rejection project. Every day for 100 days he forced himself to do something socially awkward, where the result was likely to be rejection (i.e., asking to play soccer in someone’s backyard), all while video-recording himself. Journalist Elizabeth Gilbert quit her job and marriage and then spent a year traveling the world to discover herself. She divided the year into three experiments: eat, pray, and love. Her experience turned into a best-selling book and movie. Young entrepreneur Ari Meisel used data and experimentation to cure his Crohn’s disease, which his doctors said could not be cured.
Understanding The Math: If You Can Do Enough Experiments, Success Is Virtually Guaranteed
If experimentation is so powerful, why don’t more people do it?
I say that there are a few reasons…
First, we live in a culture that is obsessed with productivity: getting more done and less time, systematizing, automating, and even outsourcing. If one has a frame of short-term productivity, then taking time out of your day to nurture a creative process with unpredictable results that don’t pay off immediately is extremely hard. What is productive in the long-term often feels not productive in the short-term.
Also, performing experiments is time intensive. To squeeze out some deliberate learning every day requires at least 15 minutes, but even more challenging is that most experiments fail. While failure is increasingly celebrated in our society, most people still have a visceral feeling of shame and disappointment that comes from it.
It wasn’t until I understood the math behind experimentation that I was able to get past my fear of failure.
1. If you do enough experiments, the odds are in your favor. The quality of each subsequent experiment increases because you will tend to apply the lessons learned from previous experiments. Those lessons make your success curve exponential rather than linear.
2. One big winner pays more than enough for all the losing experiments, as Jeff Bezos explained in a recent SEC filing:
Given a ten percent chance of a 100 times payoff, you should take that bet every time. But you’re still going to be wrong nine times out of ten. We all know that if you swing for the fences, you’re going to strike out a lot, but you’re also going to hit some home runs. The difference between baseball and business, however, is that baseball has a truncated outcome distribution. When you swing, no matter how well you connect with the ball, the most runs you can get is four. In business, every once in awhile, when you step up to the plate, you can score 1,000 runs.
3. Today’s tools allow anyone to increase their quantity of experimentation by an order of magnitude. A new breed of affordable apps, services, and trackers help us learn about what works for other people, collect data on ourselves, interpret it, stay accountable, and track our progress in real time. In the health space, for example, these new tools have led to the biohacking and quantified self-movements where people use blood glucose levels, sleep, activity, heart rate, gut biome, and genetics to facilitate their experimentation. Similar experimentation explosions are happening in the world of relationships, sexuality, intelligence, happiness, productivity, and personal finances.
If so many people across so many fields can incorporate deliberate experimentation, so can you!
What We Can Learn From Edison About Actually Living the 10,000-Experiment Rule
Edison didn’t intend to just be more inventive. He created an experimentation factory to guarantee that he and his team consistently released new inventions. Edison’s goal, for example, was one minor invention every 10 days and one major invention every six months. When he was on the edge of a major breakthrough, such as the light bulb, he had a unique process called drag hunts in which he would generate and test hundreds, even thousands of possibilities.
So what could the 10,000-experiment lifestyle look like for you?
I recommend taking two steps now that could change everything for you.
First, identify at least one jackpot experiment that could change your life. The road to deliberate experimentation starts with one experiment, but not all experiments are created equal. Some experiments are extremely time and money intensive. Some create incremental changes, while others could be life changing. Some have a 1% chance of success. Others are a sure bet. When picking a first jackpot experiment to pursue, you want to pursue an experiment that is easy to do monetarily and timewise, has the potential to be life changing, and carries a reasonable probability of paying off. I call these Jackpot experiments.
Second, I recommend running three experiment tests each day. When you start the day, identify three tests you want to perform. Collect data throughout the day, and before the day ends, analyze the results.
Try this for one month, or 30 tests, and see the difference it makes!
Interested in really taking action to become a deliberate experimenter? After studying how dozens of the world’s most prolific experimenters work, we spent dozens of hours creating a free mini-course that includes several email lessons and a webinar to help you be successful with the 10,000-experiment rule. It includes case studies of experimenters who improved their health, confidence, and lifestyle, tips on how to perform a valid experiment, details on the nuances of Blind Variation And Selective Retention and much more.
Click here to sign up for the free course >>
Special thanks to Eben Pagan for reviewing the article and coining the phase ‘10,000-experiment rule.’
This article was written with love and care using the blockbuster mental model.
If there’s a link to an Amazon book, it’s an affiliate link, which means I get a small amount of compensation when you buy the book. This compensation does not influence the specific books I recommend, as I only recommend books that I read and love.