Race Alongside the Machine

If there was one thing that kept coming up at this week’s O’Reilly Strata conference, it was the inevitable tension between human intuition and the harsh light of data. It informed everything: Wide-ranging, speculative keynotes about the future of abundance; furtive hallway conversations about the NSA; businesses trying to strike a balance between algorithms and boardroom gut decisions. There was even a formal, Oxford-style debate on whether technology creates more jobs than it destroys.

Data-driven decision making promises optimization and efficiency. Algorithms predict outcomes, organizations make better decisions, waste is reduced: Competitors fall. But when everyone is optimizing everything, we’re all racing toward what mathematicians call a “local maximum.” It’s the best score we can hope for within the rules of the game we’re playing.

What if you want to play by a different set of rules? What if you want to do something the data doesn’t support, something suboptimal at the outset? Something that’s literally a leap of faith, because the facts won’t back you up?

This is a problem Clay Christensen talked about in “The Innovator’s Dilemma,” and it’s one that Geoffrey Moore reprised in his opening remarks. Sustaining innovation is about doing more of the same, better. But disruptive innovation — the “moon shots” that raze industries and raise societies — is about doing something differently. Machines are for optimization; humans are for inspiration.

In some ways, we’re victims of our own success. Just a few years ago, Strata was focused tightly on the technology needed to answer questions that were previously unanswerable. Having now armed ourselves with an arsenal of colorfully-named technologies that can make sense of petabytes of information, we find ourselves in the uncomfortable situation of not knowing what questions to ask.

Maybe the problem is that we aren’t asking big enough questions. Google’s Larry Page echoed these remarks recently, saying that “We should be focusing on building the things that don’t exist.”

Big Data will revolutionize everything from medical care to education to conflict mitigation to waste management to materials science to geo-sensing — all the subjects of case studies at the conference. New things are possible, and we need to ask more of ourselves as a result.

Or maybe the power of existing institutions makes them too hard to change. Touching on this in his Strata talk, science historian James Burke observed that representative democracy was designed to solve the problem of bad roads in the 17th century — yet despite advances in telecommunications, we haven’t thrown it out.

Perhaps, as closing speaker David McRaney argues, we’re just not that smart. After all, as a species, we’re still guided by our genes, running on jungle-surplus hardware that only just clawed its way out of the primordial ooze, trying to overcome cognitive biases and simple errors that served us well in caves, but cripple us in the face of hard facts. Confronted by strong scientific evidence, we cling to our old beliefs, undermining any advantage the data might convey.

Strata is unique in the tech-conference firmament, in part because it focuses on the places where the rubber of technology hits the road of humanity. As one session put it, “Soylent Mean: Data Is People.”

What’s certain is that a conference ostensibly about the future of Big Data, new interfaces and ubiquitous computing has morphed into something larger: The sometimes uncomfortable, always seductive and wholly inextricable future as a hybrid of human and machine.

Co-chair of O’Reilly’s Strata conference, Alistair Croll has been an entrepreneur, author, and public speaker for nearly 20 years, working on a variety of topics from Web performance to big data to cloud computing to startups. “Lean Analytics” is his fourth book on analytics, technology, and entrepreneurship. Reach him at Solve For Interesting.

This article originally appeared on Recode.net.