Survivorship bias, or Why you can’t say you can’t

Yesterday, I had a meeting with the president of the company that I work for (it’s a small company) and the managing director. I quit. Nicely, of course, but I quit. I’m launching a startup.

(Yes, I know — who isn’t?)

Part of my preparation has been reading everything ever written, and several things that weren’t, about entrepreneurship and what it takes to start a company. Here are some of the things I’ve learned:

• You have to be young – less than 28 is best
• You have to be single
• You have to be older: two or more prior startups is best
• You have to have a PhD
• You have to live in the Valley
• You should drop out of school
• You need a professional manager with a business degree
• You’ll sink if you have an MBA on board
• You should all be coder rock stars
• You should have a diverse set of skills on board

etc., etc. My conclusion is, of course, that it’s all stuff and nonsense.

Here’s why: survivorship bias.

Suppose you’re a financial analyst, and your Grandma wants to retire from the textile factory. You’re going to invest her money for her, and since she’s family, you decide to actually do some math to see which portfolio is better: one made up of growth stocks or value stocks. (Never mind how you tell one from another — that’s a fairy tale for a different day.) You dial up your Internet connection, log into AOL and tabulate the results of the top 100 value stocks, and the top 100 growth stocks over the last 10 — wait, it’s Grandma — last 25 years. You conclude that you should put your money into…

It doesn’t matter. You’re wrong. You’re wrong because you’ve fallen victim to survivorship bias. Your sample of 200 stocks only included those that survived — the companies that went belly-up were not included in your study. If you did that study today, your list of stocks wouldn’t include Enron, WorldCom, or a host of other “solid” companies that went quietly into that good night. To eliminate the bias you would have to examine what stocks were available at the time, and track those through 25 years, including the occasional bankruptcy in your calculations.

If you think such an error is juvenile and obvious, think again — it’s made time and time again in the popular and academic press. It’s made by academics and political pundits and sports commentators and bloggers. It’s not a quaint, old-fashioned mistake: its all over the New York Times today and it will be all over the Wall Street Journal tomorrow. It’s the first thing to check for when your B.S. spidey-sense is tingling but you don’t know why.

Survivorship bias is especially prevalent when people analyze companies or people. Books that profile successful firms or individuals and draw conclusions about what qualities contributed to their success ignore the possibility that there are many firms or people with exactly those qualities that did not succeed, or did succeed but not as brilliantly. So books like Built to Last and Secrets of the Millionaire Mind are entertaining to read, but not particularly illuminating. After all, there are tens of thousands of hardworking, frugal people who pay close attention to the bottom line but do not get rich.

So we come to theories about what it takes to successfully launch a startup. Folks who say, for example, that you’re really better off being under 30 and single to launch a tech startup probably feel like they have good data: they look around and see that most hot new Web 2.0 companies seem to be headed by young men. (And it is, sadly, mostly men.) They may even do some counting and put some numbers to the claim. But to prove the point, one would have to look at 1000 or so startups as they began, and see if having a young founder improved the odds of success.

Say for example that out of 20 successful startups, 15 have founders under 25. That alone proves nothing; if 1000 startups were launched to produce those 20 success stories and 750 of them had founders under 25, then age would have zero influence on success rates. If only 200 of the 1000 had founders under 25, though, you’d really be onto something. One could do a similar experiment with claims about geographic location: I suspect (but don’t know) that many more startups are launched in the Bay area or one of the other startup hubs like Boston or Seattle than in other cities. It would not be surprising that so many more successes came from there. To determine if geography was a factor in success, though, one would actually have to do some serious counting and tracking. Statistics on the table, as Stephen Stigler says.

The position I’ve been arguing against is admittedly a straw-man: nobody actually claims that you cannot start a tech company if you’ve seen 30 Christmases. There are, however, plenty of people saying you’re bucking the odds if you don’t pass this or that filter. But there are enough factors conspiring to discourage would-be entrepreneurs from launching their ideas without adding spurious ones to the mix. So my advice is: when you read something that suggests that qualities you have are required for success, believe the writer wholeheartedly – you’re going to need all the optimism you can get. But when you read that you lack some necessary ingredient for success, check whether the writer has done any counting. Then go start your company.

Explore posts in the same categories: Entrepreneurship, Startups, Statistics