Big data, artificial intelligence and visualization may be nearing an emperor has no clothes intersection as we regular humans---business managers---start believing in beautiful but statistically insignificant graphics and magic boxes that are marketed as experts systems.
Simply put, the whole big data and analytics movement may be hitting Gartner's trough of disillusionment. You knew the trough was coming. Consider the following developments:
Today, you can't escape the Internet of things and all that data that'll be thrown off from devices. Let's table the discussion about what we'll do with that information. We'll figure it out at some point, say business execs.
Sure they will.That backdrop is one reason why my conversation with BeyondCore CEO Arijit Sengupta was so timely. BeyondCore's product is 18 months old and has landed 23 Fortune 100 clients. BeyondCore's lead product, which takes any data with at least 1,000 rows (the more rows the better) and took eight years of research to develop, aims to tell the average business user a statistically significant narrative without a human question to initiate the discussion.
The key words in that previous sentence are statistically relevant and narrative. Sengupta's beef is that today's artificial intelligence is taking human intuition out of the equation. That movement means two things: First, we'll need data scientists to tell us what to do. Second, we'll be marketed magic boxes that aim to lead us. Either way we're outsourcing way too much intelligence and instead of becoming data savvy we'll just become visualization slaves. After all, we all look smarter with a fancy graphic to back us up. The catch is visualization isn't analysis.
Also: Big data initiatives not quite delivering yet, survey shows | Samsung at CES 2015: How enterprise fits it in with Internet-of-Things | The Internet inside the enterprise: We don't have it, and we need it | Lowe's at CES 2015: Smart homes are about lifestyles, demystifying home automation | Welcome to the dystopian Internet of Things, powered by and starring you | Five years until the Internet of Things arrives? Why I hope it's a lot, lot longer | Cisco's next stop on Internet-of-Everything roadmap: Connected analytics
"If artificial intelligence (AI) systems can't let a business user understand it, the insight will remain the domain of data scientists or machines," said Sengupta, an Oracle and Microsoft alum. "We're just getting to the point where the market is beginning to understand how dangerous believing in dashboards and magical boxes."
In other words, we can't let the data scientists and engineers run away with big data or we'll have information haves and have nots.
BeyondCore's technology, which I played with, provides text summaries and animated briefings. Computers compute and humans overlay their knowledge. But the real win is that all the narratives are built on statistically sound data---there's even a regression proof available (view source for data scientists)---so the business user can keep the wonks at bay. "Humans look at graphs and get excited," quipped Sengupta. "The solution to the world's problems is not more data scientists. The communication chasm between the business user and analytics is a problem."
Specifically, BeyondCore's algorithms spin Hadoop's approach around. Hadoop is optimized for search and reducing data to something manageable via MapReduce. The emphasis is on the Map part of the equation. BeyondCore is optimized for the reduce part of the big data. "So far big data has been about bragging about how much stuff you had stored in the attic," said Sengupta. "The promise of big data is broken."
Sengupta has a lot of examples of how the new approach to artificial intelligence and data needs to work. For instance, BeyondCore highlighted how young women with diabetes had a 49 percent rehospitalization rate. Out of 30 million patients, the natural conclusion was that there was some gender issue or drug reaction. Turns out that many women weren't taking insulin because they gain weight. A box that requires you to ask a question probably couldn't deliver that insight. "How would a researcher know to look at that question?" asks Sengupta.
Due to the current state of big data and the intersection with artificial intelligence, Gartner created a category dubbed "smart pattern discovery." These vendors are likely to spring up to make sense of data lakes. These vendors are also likely to be acquired because there just aren't enough people to manage the data or interpret it. Five years from now we're going to need to bring statistically relevant data to business users.
"I don't believe in magic models. Prove it to me," said Sengupta.
ZDNet's Monday Morning Opener is our opening salvo for the week in tech. As a global site, this editorial publishes on Monday at 8am AEST in Sydney, Australia, which is 6pm Eastern Time on Sunday in the US. It is written by a member of ZDNet's global editorial board, which is comprised of our lead editors across Asia, Australia, Europe, and the US.
Previously on Monday Morning Opener: