Soon, machine learning agents behind every application

Many IT and business professionals are just starting to get their feet wet with machine learning, which promises to enable systems to essentially program themselves, based on constantly refreshed data. It's catching on fast -- a survey of 360 organizations by the Economist Intelligence Unit and SAP finds that 68 percent are already using machine learning to enhance their business processes.

If machine learning isn't showing up in applications or systems, it's likely to show up soon, according to Joe Hellerstein, co-founder and CSO of Trifacta, and a professor of computer science at the University of California-Berkeley. I recently caught up with Hellerstein, who predicts machine learning will soon be commonplace -- embedded within the tools and applications used on a daily basis within data-driven businesses.

"Machine learning technology is difficult for enterprises to engineer, manage and deploy in-house," he explains. "As a result, its adoption in the enterprise context has been slower than the hype might suggest. But machine learning intelligence is increasingly being embedded by software vendors into more traditional user-facing data management products, and serving as an augmentation to the domain knowledge and human intelligence of end users."

Many user-facing tools now claim some use of embedded machine learning assistance, Hellerstein states. "As machine learning intelligence makes it ever easier for humans to work with data, we're going to see the barriers to entry for users drop significantly. This will enable a broader set of analysts and business people work with data in ways they never before thought possible."

Embedded machine learning is appearing within the next generation of data and analytics applications, serving to "guide users through the process of discovering, preparing, and analyzing data, as well as selecting models to run against that data," Hellerstein explains. "Machine learning technology is incorporated into rich, predictive user interactions: providing suggestions to users to accelerate and guide traditionally manual processes, while keeping humans in the loop to maintain control and context over tasks."

The human element is essential, as machine learning is "inherently imperfect, and will in some contexts make incorrect or inappropriate predictions," he adds. "Even in the context of predictive interaction, although the machine learning only makes suggestions to users, it can mislead users to draw incorrect conclusions about their data and in some cases lead to poor human decisions. This can undermine trust." The key, he says, "it's very important for the machine learning to be wedded to powerful and transparent user interfaces that help users understand what is being suggested or predicted, why it was chosen, and what the alternatives decisions may be."

Eventually, embedded machine learning "will be very common in user-facing tools for core functions like data discovery, preparation and analysis," Hellerstein says. "Enterprise users will have come to expect user experiences that are intelligent, and will be frustrated by tools that require them to initiate every step without some data-driven guidance from a machine learning agent."