AI training needs a new chip architecture: Intel

The reason why industry needs new architecture for neural networks becomes apparent when you work with GPUs at a low level, according to recently appointed chief technology officer of Intel Artificial Intelligence Product Group, Amir Khosrowshahi.

Khosrowshahi was previously co-founder and CTO of Nervana Systems, which was purchased by the chip giant for an undisclosed amount in August last year, and whose technology quickly found itself at the centre of Intel's plans for AI.

The CTO detailed how his former company used GPUs because "that was the state of the art", but the company replaced the standard Nvidia assembler with one of its own because it was regarded by Nervana as generating "sub-optimal" instructions.

"Early on in the history of Nervana we did this, partly for our own development efforts, but then realised it was about two or three times faster than Nvidia's own libraries, so we released it as open source," Khosrowshahi told ZDNet on Thursday.

Nervana's efforts did not stop at software alone, with the company also creating its own silicon targeting neural network training.

"Neural networks are a series of predetermined operations, it's not like a user interacting with a system, it's a set of instructions that can be described as a data flow graph," he said.

According to Khosrowshahi, some of the features that help a graphics processing unit do its graphics rendering job -- such as a large amount of cache, and handling vertices, rendering, and textures -- is superfluous.

"There is so much circuitry in a GPU that is not necessary for machine learning ... this is crud that has accumulated over time, and it's quite a lot of stuff," he said.

"You don't need the circuitry which is quite a large proportion of the chip and also high cost in energy utilisation."

"Neural networks are quite simple, they are little matrix multiplications and non-linearities, you can directly build silicon to do that. You can build silicon that is very faithful to the architecture of neural networks, which GPUs are not."

The answer Khosrowshahi had a hand in developing is now called Lake Crest, a discrete accelerator which Intel is set to roll out to select customers this year, but over time, it will become more closely tied to Xeon processors.

"It's a tensor processor, it deals with instructions that are matrix operations," Khosrowshahi explained. "So the instruction set is matrix 1 multiplied by matrix 2, go through a lookup table, and these big instructions that are high-level.

"In a GPU it is a register and a register, move into [another] register, do an element-by-element multiplication, it's fairly low-level."

Whereas the CTO said Nvidia have made an effort to make their GPUs more neural network friendly in recent years, their AI chips still carry a lot of graphics functionality with them.

"Being part of a chip effort myself, I can see why that would be hard for Nvidia to do," he said.

On the other hand, Intel shifted the way it approached AI through purchases.

"It's a challenge in the chip industry to come up with an entirely new architecture; what Intel has done is they have purchased it," Khosrowshahi said. "They've bought FPGAs [field-programmable gate array], so they bought Altera, which is this really cool architecture that is very neural networky, so FPGA fabric is pretty interesting for neural networks... and of course the Nervana chip is very much a neural network focused engine, but slightly removed from neural networks."

When thinking about neural networks, Khosrowshahi said it is wrong to think about etching neural networks in silicon, as much of that functionality remains in software.

"Quite a lot of it is in software, so even in the case of Lake Crest, the instructions [for Lake Crest] are not 'neural network, do this', it's multiply this matrix by this matrix."

"Outside the chip there is some software that knows it is a neural network, and it is training, and the user is asking for different things and searching for parameters -- all the sorts of things you have to do when you have a neural network system."

With a background in neuroscience, Khosrowshahi believes the point of artificial intelligence is not to recreate the human brain, but to go beyond it.

"The brain is an example of an AI, but it is a certain limited AI in the sense that my visual system sees the physical world, and it knows to understand the statistics of the world," he said.

"If you look around you, you see a lot of edges, you see a lot of surfaces, you see shaded regions and so forth, and if you actually look in the brain ... [in] the primary visual cortex, there are neurons that are sensitive to these features, so your AI has learned the statistics of the world and is able to do inferences on them -- like I'm going to crash into this, or this is a cup, I want to hold the cup."

But the data within enterprises is much different from that humans interact with, Khosrowshahi said.

"It's rows of tables, of people clicking on stuff, potentially the statistics are very non-intuitive, so the idea of having intelligence working on this data is kind of a different form of intelligence."

"I try to explain this to people because they think we are recreating a brain, we want to go beyond that, we want to create a new kind of AI that can understand the statistics of data used in business, in medicine, in all sorts of areas, and that data is very different in nature than the actual world."

One of the AI architectures Intel will go up against is Google's custom-built Tensor Processing Unit, which the search giant said on Wednesday would be 15 times to 30 times faster than contemporary GPUs and CPUs at inference, as well as being 30 times to 80 times more power efficient.

Also on Wednesday, IBM and Nvidia announced Big Blue would offer Tesla P100s within its IBM Cloud from May.

Disclosure: Chris Duckett attended Intel AI Day as a guest of Intel