When “Big Data” Fails

The focus on more data

More data is all the rage. An article in the Harvard Business Review opined that “[T]hriving as a mainstream company today means being data driven.”

Big companies are paying heed.

A recent survey found that 92% attribute their “principal challenge to becoming data-driven”. The authors of the study concluded these results “strongly suggest the need for change in the focus of data executives toward programs that address data culture, literacy, and decision-making – even at the expense of technology initiatives.”

Should you rush to jump on the data bandwagon?

The importance of instinct

Data is important – often critically so – but the appeal of data-driven decisions may cause you to overlook another factor that should play an important role in your decision making: your gut instinct.

A recent study co-authored by academics at King’s Business School and the University of Malta (discussed here) found reliance on crunching data might impede the decision-making process by delaying the decision without improving accuracy.

The authors surveyed 122 companies and asked executives the role data, instincts and other mental strategies played in their decision making on a recent innovation project.

They found managers relied on their instincts as much as data. One of the co-authors wrote that the experience of executives can be more effective than data. He noted that, in situations of extreme uncertainty, data “could act as a hindrance rather than a driver of success.”

This conclusion is supported by many other studies (discussed here) demonstrating the accuracy of gut feelings, unsupported (and often contradicted) by data.

One study found that, in some situations, our gut instincts may be more accurate than sifting through vast amounts of data.

In one fascinating experiment, German and U.S. students were asked to guess whether Detroit or Milwaukee was larger.

The Germans had heard more about Detroit than Milwaukee. Based solely on that input, 90% of them accurately guessed it was larger. Only 60% of the Americans, who had much more data, made the correct choice.

When you are confronted with a difficult decision, consider this advice from Richard Branson, the billionaire founder of Virgin: “I rely far more on gut instinct than researching huge amounts of statistics.”

Dan trains executives and employees in the lessons based on the research on his latest book, Ask: How to Relate to Anyone. His online course, Ask: Increase Your Sales. Deepen Your Relationships, is now available. It’s rapidly being adopted by individual advisors, advisory firms and corporations as an impactful, cost-effective training tool.

Read more articles by Dan Solin