Harvard and McKinsey's Bad Statistics

I recently came across a Harvard Business Review article that claims "Your Company is Too Risk Averse". Normally, when I come across something wrong on the internet I tend to ignore it, but HBR and McKinsey impact a lot of people by virtue of their alumni and thinkers being in leadership positions. So I figured it's worth a deeper dive to understand what exactly is wrong, a deeper dive into the veneer of math and scientism (looks like science but isn't) used to justify faulty conclusions. Here's the premise:

Here's the premise:

In theory, companies are supposed to create value for stakeholders by making risky investments. And as long as no single failure will sink the enterprise, those investments may be quite large. It won’t matter if even a significant percentage of them fail so long as the success of other bets compensates, which usually happens. [...]

In current practice, however, executives in large corporations are reluctant to propose and advocate for risky projects. They quash new ideas in favor of marginal improvements, cost-cutting, and “safe” investments.

Let's leave aside for a moment all of the problems with that starting premise, which has caused all sorts of problems in the world. Let's dive into the math they're using to justify this. So, building off of the authors' own 2012 McKinsey executive survey, a scenario is presented:

You are considering a $100 million investment that has some chance of returning, in present value, $400 million over three years. It also has some chance of losing the entire investment in the first year. What is the highest chance of loss you would tolerate and still proceed with the investment?

The conclusion reached by the authors is – to them – shocking:

A risk-neutral manager would be willing to accept a 75% chance of loss and a 25% chance of gain; one-quarter of $400 million is $100 million, which is the initial investment, so a 25% chance of gain creates a risk-neutral value of zero. Most of the surveyed managers, however, demonstrated extreme loss aversion. They were willing to accept only an 18% chance of loss, much lower than the risk-neutral answer of 75%. In fact, only 9% of them were willing to accept a 40% or greater chance of loss.

What’s more, the size of the investment made little difference to the degree of loss aversion. When the initial investment amount was lowered to $10 million, with a possible gain of $40 million, the managers were just as cautious: On average, they wouldn’t make the investment if the chance of losing was higher than 19%. And once again, only 9% were willing to accept a chance of loss greater than 40%. This indifference to the size of the investment seems perverse, because a relatively small investment is unlikely to present an existential threat to the enterprise and should, therefore, give managers scope to assume more risk.

Let's unpack that for a moment:

• You're a manager within an organization...
• ... tasked with an option to invest $100 million...
• ... the option having X chance of resulting in $0 in the first year...
• ... and Y chance of resulting in $400 million three years later.

At what odds do you gamble?

You Must First Survive

The first problem with experiments like this is that they are viewed as one-time hypotheticals, while reality is a time-based series, with multiple variations and minute changes over time. And in such scenarios, as Warren Buffet and Nassim Nicholas Taleb like to say, "In order to make money, you must first survive".

Here's Taleb on that topic:

[An] absorbing barrier is a point that you reach beyond which you can’t continue. You stop. So, for example, if you die, that’s an absorbing barrier. So, most people don’t realize, as Warren Buffett keeps saying, he says in order to make money, you must first survive. It’s not like an option. It’s a condition. So, once you hit that point, you are done. You are finished. And that applies in the financial world of course to what we call ruin, financial ruin.

Behavioral economists have something called mental accounting, which states exactly what you just said: that treating money according to the source is irrational because these are one-period models. That’s how they view the world, as a one-shot experiment. They don’t view the world as repetition. A repetition of bets. So, if you look at the world as repetition of bets, under condition of survival, then mental accounting is not only not irrational but is necessary. Any other strategy would be effectively irrational.

Viewing these experiments in a series of bets, each one impacted by the previous one, is essential to having the absorbing barrier come into play. And that's important because – as Keynes pointed out in the 1930s – "Markets can stay irrational longer than you can stay solvent.”

The second problem is that the financial-ruin endpoint of a manager (or executive) in an organization tends to be 'losing a large sum of money'. That endpoint comes doubly quick if it's a large sum of money lost on a stupid (risky) bet.

Let's define "stupid" in this context as anywhere where your chance of success is no better than a coin flip (50%). It's a rigorous definition because if you're tasked with investing $100 million (or any amount of money) and what you bring to the table is no better than flipping a coin, then both you and your hiring manager should be fired on the spot and replaced with that said coin. After all, a key role of an investor is not just to distribute money... it's to manage and improve risk scenarios.

To digress with Warren Buffet for a moment: his investments are based on intrinsic value and investing in companies undervalued by the market, where current stock price as evaluated by a market doesn't match the core earning value of the business. ¹ The whole point is not to take risky bets, but to wait for market conditions to give you a good offer. Or for the rest of us, it's akin to waiting for that thing we want to finally go on sale.

So when the authors write that "a risk-neutral manager would be willing to accept a 75% chance of loss and a 25% chance of gain", they're advocating for an entirely reckless and irresponsible sort of math for the real world. They're calculating "neutral" as the odds of the bet returning mathematical parity... but the manager really should be trying to ensure that the outcome is more (rather than less) likely to return something positive.

In fact, the only notable thing to come out of the data is that 91% of surveyed managers are not complete idiots, at least when it comes to making investment decisions using other people's money: "They were willing to accept only an 18% chance of loss, much lower than the risk-neutral answer of 75%. In fact, only 9% of them were willing to accept a 40% or greater chance of loss." ²

Not Even With Caveats

There are a few interesting caveats mentioned in the article, none of which stand up to closer scrutiny.

The biggest one is that "as long as no single failure will sink the enterprise, those investments may be quite large. It won’t matter if even a significant percentage of them fail so long as the success of other bets compensates, which usually happens."

There's definitely something to be said about this line of thinking. Venture capital has been the most vocal proponent of this idea recently – that one big "10x" win will make up for a number of bad bets over a period of time.³ But admitting a level of truth in this observation ends up highlighting the broken premise of the experiment as a one-off gamble, instead of as a series of bets distributed over time.

The other problem is with the math missing the real world of ancillary expenses; This includes accounting for salary, the cost of doing the investment (lawyers, accountants), and inflation over time.⁴

In the author's scenario, to do right by the real world, anyone willing to accept over a 75% chance will net out negative once those ancillary expenses are brought into play. If you estimate the cost of business at 5% the cost of the investment, then you'd need to accept 70% chance of failure to just to break even over multiple rounds. If you factor in inflation over time, then your investments need to show net growth higher than inflation (currently 2.3%)... so an aggregated 63.1% risk of failure if the return per investment is in 3 years.⁵ And – if you want to keep your job – you want to be sure to be meaningfully contributing to the bottom line over a long period of time. Aggregated investments that net out neutral mean that you haven't actually contributed anything. So, let's cut that in half and assume 31.5% failure rate, if you want to be employee of the year with those gambles.

Third – and more broadly – if you're a manager in a company, you're looking out for your individual successes and failures, not those of your company in aggregate. So if you fail and your colleague succeeds, you'll probably get fired and they will get promoted. Attrition matters here. And the math on this is pretty clear too. With neutral odds, you have a 45% chance of making a bad investment 3 times in a row (.75^3) or a 32% chance of making four back investments in a row. Can you survive those odds?

To be fair, the authors dutifully point out that the math makes sense almost only in aggregate, and not in the reality of how mid-level managers make decisions:

We would expect that senior executives will be more open to investing in small projects than lower-level managers are. And that does appear to be the case. In a recent workshop, Nobel laureate Richard Thaler asked 22 heads of magazines owned by a large publisher if they would accept a hypothetical 50-50 investment that would pay $2 million to the parent company if it was successful or lose $1 million if unsuccessful. Only three said they would accept the investment; the rest declined. In contrast, the CEO “wanted them all” to be accepted; he had a broader view of the possibilities and risks and realized that when the investments were pooled together, the risk profile was much more attractive.

Of course, the CEO can comfortably close down an underperforming magazine in a portfolio when he's focused on aggregate success... but the head of that magazine would face a terminal event from which there would be no further bets.

Another caveat (well, more of a surprise to the authors) is that this data holds true whether you're gambling with $100 million or $10 million. I suppose those numbers might be chump change for the roughly 2000 billion-dollar-annual-revenue businesses, but how applicable is it to us more common folk (or even managers within departments in those huge businesses)?

Let's say you're investing in a business, where you could have some reasonable expectation of control over your investment. (Because if you're investing in environments where you have a low probability of success and you have no control over that outcome, then why just switch to Russian routlette and get it over with?). A quick search will tell you that typical investments range anywhere from 10K on the low end to $1 million on the high end. If we take today's angel and venture capitalism as an example, raising $100 million is the result of multiple investors pooling together for a round.

Only Masayoshi Son of SoftBank seems to be doling out $100 million - $1 billion investments... and even with a $1 billion investment being only 1% of the total Vision Fund, no one seems to be very happy for any individual investment to not be bringing in multiples back. Just a few high-profile failures have been enough to completely derail his entire investment strategy.  (See: Uber, WeWork, that robot pizza maker company).

To prove the point, I wrote up a simulation of the "Four Losses and You're Fired" scenario that you can play: https://github.com/sharedphysics/Experiments-in-Probability/blob/master/25-percent-investment-calculator-with-absorbing-streak.py

I set the absorptive capacity to 4 losses in a row. See how many investments it takes for you to hit the terminal point, and how much you end up making.

These Aren't Real Investments Anyway

The last point worth making about this whole debacle is about how contrived it is.

Not to keep harping on a theme, but investments tend to not be all-or-nothing. I'm fairly confident that if the question was asked with probabilities that included a probability that included a net-0 option ($100 million invested = $100 million returned), then you'd have many more people willing to take the risks.

Also: many ideas are "execution dependent", as they say in Hollywood:

“Execution dependent” means that the best version of the movie is a hit, while a mediocre incarnation is worth vastly less. It’s not a diss. Most films that win Academy Awards are execution dependent, as are many blockbusters.

For example, Slumdog Millionaire is completely execution dependent. If it didn’t fire on all cylinders, you would never have heard of it. It would have been another ambitious indie failure.

Raiders of the Lost Ark is also extremely execution dependent. There have been countless movies with adventurers seeking treasure, but the combination of elements in Raiders just clicked. If Raiders were twenty percent less awesome, it wouldn’t have a place in film history.

Here’s a good test for whether a project is execution dependent: How many different directors could you imagine making it?  If there are five or fewer directors on your list, that’s a highly execution dependent project.

Going back to the example of investing in a company, not all companies are equal. Some have better business models than others (or in the case Masa-san, some companies simply have business models while others don't). And as Paul Graham, founder of Y-Combinator wrote in regards to starting a business in the height of the 2008 crisis:

If we've learned one thing from funding so many startups, it's that they succeed or fail based on the qualities of the founders. The economy has some effect, certainly, but as a predictor of success it's rounding error compared to the founders.

Which means that what matters is who you are, not when you do it. If you're the right sort of person, you'll win even in a bad economy. And if you're not, a good economy won't save you. Someone who thinks "I better not start a startup now, because the economy is so bad" is making the same mistake as the people who thought during the Bubble "all I have to do is start a startup, and I'll be rich." ⁶

So the success of your investment is dependent on:

• The person running the company/unit you're investing in,
• The general economic climate (I write this just days after the COVID-19 pulled a grey-swan level event and ended the longest bull market),
• What the competition and market is doing,
• And on many other factors beyond simple probability that make calculating your odds of success calculable in hindsight more than anything. If there were surefire investments that would guarantee such payoffs as such odds, business would be so much easier!  

There's an oft-repeated phrase that I find helpful when reading academic research like this: "If theory, there's no difference between theory and practice. In practice, there is."

So while the authors might argue for a more probabilistic investment philosophy, I recommend staying clear. Unless you're the CEO of a privately held company willing to give your team a wide leeway in success vs. failure, this is just bad real-world math and bad advice.

Footnotes

¹ - Here's Buffett himself in an early letter to shareholders:

We select our marketable equity securities in much the same way we would evaluate a business for acquisition in its entirety. We want the business to be (1) one that we can understand, (2) with favorable long-term prospects, (3) operated by honest and competent people, and (4) available at a very attractive price. We ordinarily make no attempt to buy equities for anticipated favorable stock price behavior in the short term. In fact, if their business experience continues to satisfy us, we welcome lower market prices of stocks we own as an opportunity to acquire even more of a good thing at a better price.

...

Our experience has been that pro-rata portions of truly outstanding businesses sometimes sell in the securities markets at very large discounts from the prices they would command in negotiated transactions involving entire companies. Consequently, bargains in business ownership, which simply are not available directly through corporate acquisition, can be obtained indirectly through stock ownership. When prices are appropriate, we are willing to take very large positions in selected companies, not with any intention of taking control and not foreseeing sell-out or merger, but with the expectation that excellent business results by corporations will translate over the long term into correspondingly excellent market value and dividend results for owners, minority as well as majority.

He turns down most investments that don't match his criteria, and takes the opportunity to buy when market valuation doesn't match his calculated intrinsic value. It's like a sale on stock. As an old investing aphorism goes, "In bear markets, stocks return to their rightful owners."

² - All that being said: I'm not sure how they calculated 19%, but that does seem a bit low, for reasons I get into later.

³ - There's an interesting part in Tim Ferris's "Tools of Titans" that gets into the difference between venture capitalism (represented by Marc Andreesan) and value investing (Buffet):

Tim: [joking]: “You’re not going to invest in See’s Candies [one of Buffett’s investments]?”

Marc: “No. No. Absolutely not. Furthermore, every time I hear a story like See’s Candies, I want to go find the new scientific superfood candy company that’s going to blow them right out of the water. We’re wired completely opposite in that sense. Basically, he’s betting against change. We’re betting for change. When he makes a mistake, it’s because something changes that he didn’t expect. When we make a mistake, it’s because something doesn’t change that we thought would. We could not be more different in that way. But what both schools have in common is an orientation toward, I would say, original thinking in really being able to view things as they are as opposed to what everybody says about them, or the way they’re believed to be.”

⁴ - If you're not an investment banker, your ancillary costs of an investment will also include all of the operating costs expenses.

⁵ - And with regards to inflation, I can't say it better than Buffet himself and won't even try:

Inflation acts as a gigantic corporate tapeworm.  That tapeworm preemptively consumes its requisite daily diet of investment dollars regardless of the health of the host organism.  Whatever the level of reported profits (even if nil), more dollars for receivables, inventory and fixed assets are continuously required by the business in order to merely match the unit volume of the previous year.

⁶ - Warren Buffet has made a similar claim in evaluating his insurance investments in a letter to shareholders:

Insurance companies offer standardized policies which can be copied by anyone. Their only products are promises. It is not difficult to be licensed, and rates are an open book. There are no important advantages from trademarks, patents, location, corporate longevity, raw material sources, etc., and very little consumer differentiation to produce insulation from competition. It is commonplace, in corporate annual reports, to stress the difference that people make. Sometimes this is true and sometimes it isn’t. But there is no question that the nature of the insurance business magnifies the effect which individual managers have on company performance. We are very fortunate to have the group of managers that are associated with us.