The Demise of Big Data?

Making Intelligent Insights at the Edge

The transcript of a talk I gave both at the TechExeter conference in Exeter, and O’Reilly Media’s Strata conference in New York, during September 2019.

In our industry we tend to think far more about the future, than the past. The very nature of what we do means that we obsess about now and next, rather than putting things into their proper historical context, and that’s a mistake.

So every once in a while it’s worth it to take a step back, and look at history, then decide whether we want to repeat our mistakes and our triumphs, just one more time, or decide instead that we should be doing something different this time around.

Because we’ve been here before, back in 2011. Which was, by any measure, the dawn of the Big Data era. There were new tools appearing, new levers to move the world. We all got a bit excited.

I talked at one of the first big, Big Data, conferences, which not coincidentally was also in 2011. Although I didn’t talk about big data, mainly because I’m not entirely sure any of us knew what it was yet.

Instead I talked about Machine Learning. Nobody was all that excited about Machine Learning back then, I guess it’s funny how things turn out. The next year I keynoted the same conference, but I didn’t talk about Big Data then either. Instead I talked about small data, and all these years later it turns out that the small data systems might well be about to take over.

Because lets be totally honest with ourselves, as an industry we have failed to bring people along with us. We have engineered increasingly intricate data collection, storage, and correlation systems. Our carefully crafted websites harvest data as a matter of course. We’re proud of how much analytic data we can squeeze out of the people viewing our sites.

“…our industry is universally viewed as a dumpster fire.”

We have dashboards, lakes, and silos, and it doesn’t matter. Because outside of this industry, our industry is universally viewed as a dumpster fire, and nobody is coming to put it out.