Analytical Rigor Trumps Big Data

The Silicon Valley brain trust, from VCs to entrepreneurs to business executives, are all agog with the relatively new phenomenon of big data. It’s clearly an important technology trend at the intersection of internet’s ability to generate massive amounts of data and cost efficiencies involving the storage and processing large data sets.

Big Data

Image by Rachel Jones of Wink Design Studio using:

Funds are flowing into many big data start-ups which are creating powerful systems and tools for enabling new types of decisions aided by big data collection, analysis, workflow and communication. Established tech companies are building connectors to big data systems. And, not to be left out, mainstream businesses are launching internal big data analysis projects.

The big data excitement is clearly a new phenomenon. Its roots come from applied mathematics, forecasting and econometrics. Statistical analysis has looked as data samples, built models, tested hypotheses and simulated outcomes for many years. Businesses have tested everything from pricing proposals to demographics to feature lists with applied math. Weather forecasts and evaluating baseball talent routinely use applied mathematics. Google’s successful business strategy is a triumphant use of applied mathematics.

What’s changing is that technology makes it efficient to collect the entire universe rather than samples. Size has its advantages. Analysis of larger samples improves accuracy. Collecting large samples quickly speeds decision-making.

But this is where we need to be careful. I believe that, in most cases, more value is created through rigorous analysis rather than collecting larger samples. While the cost of data collection is lower than ever, the labor pool capable of rigorous analysis remains fixed in size. Yes, these brains are increasingly working with larger data samples or the full universe.  Their tools and intellect are data size independent.

Invest in Analytical Rigor

Before jumping head-long into big data, I recommend investing in the brain power needed to do rigorous analysis. Analytical rigor is hard…and it takes a major investment of time, personnel and leadership to accomplish. This is a very different effort that making capital investments in computer hardware, software and processes. Creating a culture of analytical decision-making has paid off for Google and the Democratic National Committee.

Anyone can buy big data tools (heck, anyone can download HadoopR and many other big data management and analysis tools for free) and tell their investors they’re using big data. The missing piece that is the commitment to rigorous analysis, to building a team that has the brain power needed to collect the right data, to building valid predictive models that enable profitable decisions.

Once your organization excels at analytical decision-making, expanding into big data is a no-brainer. If you choose to invest in big data without having a foundation analytical rigor, results will likely be illusive.