Boon has turned to bane, as the explosive growth in the size of neural networks presents the AI community with ungainly computer workloads that tax existing resources.
Happily, the new year has kicked off with a plethora of solutions to make neural networks more manageable, both for training them, and when they are running on devices to answer questions, known as the "inference" phase of machine learning.
Also: AI startup Petuum aims to industrialize machine learning
A bevy of papers at the end of December and this week propose a variety of solutions to make networks more manageable. They include, in no particular order:
As one paper puts it, by Charbel Sakr and Naresh Shanbhag of the University of Illinois at Urbana-Champaign, "Deep neural networks' enormous computational and parameter complexity leads to high energy consumption, makes their training via the stochastic gradient descentalgorithm very slow often requiring hours and days, and inhibits their deployment on energy and resource-constrained platforms such as mobile devices and autonomous agents." Many of the papers offer similar concerns.
Also: Capital One AI chief sees path to explainable AI
The search for solutions to making neural nets more efficient to process goes back decades. It was an issue when pioneers Rumelhart and McClelland wrote their massive treatise Parallel Distributed Processing in 1986, and when Facebook's Yann LeCun proposed "Optimal brain damage" in 1990, as a way to reduce the number of parameters needed to be calculated. Work by Google Brain researchers Jeff Dean and others grappled with the problem again as deep learning took off, in 2012.
The search for improvement has been constant but the problem seems only to be getting worse, as the success of deep learning makes networks bigger and bigger in the areas where machine learning has really taken off, such as image recognition and natural language processing.
The authors of the various reports all claim to reduce in some way the computing intensity, including how long it takes to train a neural net, or use it to infer things, or both. They're all burdened by the daunting trade-off that neural networks always labor under: if you dump too much of the complexity of a neural network, you may improve its performance by hours or days, but you may also lose the accuracy of the network. All the authors claim to have successfully navigated that fearsome trade-off.
Here, in no particular order, are the optimizations:
The over-arching impression given by all these reports is that today's neural networks have not changed from a decade ago in many fundamental aspects; the same problems of scale that were confronted by Google and others back then still apply. And many of the designs of these neural networks -- their "architecture" -- is inefficient, in the sense of having a lot of redundant information.
Another conclusion left lingering is whether all this work will lead to better neural networks. The notion of "representing" the world in a neural network has always rested on the idea of in some way "constraining" that neural network to force it to find higher levels of abstraction.
The limits of neural networks are explained in another paper out this week, sponsored by the Defense Advance Research Projects Agency. Pierre Baldi and Roman Vershynin, professors at UC Irvine. In their paper, "The Capacity of Feedforward neural networks," they endeavor to describe just how much information about the world a neural network can realize. As they put it, "The capacity of a network can be viewed as an upperbound on the total number of bits that can be stored in a network, or the number of bits that can be "communicated" from the outside world to the network by the learning process."
While these research papers are dealing with how to manage complexity in a practical sense, it's conceivable the things they find may eventually have an impact on the theoretical problem Baldi and Vershynin are contemplating: Whether those representations in machine learning can be made better —that is, more sophisticated, closer to some high level of understanding about the world by computers.
What is AI? Everything you need to know
An executive guide to artificial intelligence, from machine learning and general AI to neural networks.
What is deep learning? Everything you need to know
The lowdown on deep learning: from how it relates to the wider field of machine learning through to how to get started with it.
What is machine learning? Everything you need to know
This guide explains what machine learning is, how it is related to artificial intelligence, how it works and why it matters.
What is cloud computing? Everything you need to know about
An introduction to cloud computing right from the basics up to IaaS and PaaS, hybrid, public, and private cloud.