X
Tech

Server makers push next-gen InfiniBand

Seeking a better way to make high-end database servers from low-end parts, tech giants are working on a less expensive alternative to the InfiniBand high-speed networking technology.
Written by Stephen Shankland, Contributor
Seeking a better way to make high-end database servers from low-end parts, Hewlett-Packard, IBM and others have begun working on a less expensive alternative to the InfiniBand high-speed networking technology.

The technology, called Remote Direct Memory Access (RDMA), borrows heavily from the years of work that went into creating InfiniBand, according to engineers involved in the work. While RDMA will arrive later and perform slower than InfiniBand, it will have the advantage of using conventional networking hardware based on the TCP/IP and Ethernet standards used to build the Internet.

"We're pushing hard on RDMA," Mary McDowell, the head of HP's Industry Standard Server group, said in an interview. (She spoke days before Friday's announcement that after years running Intel-based servers for Compaq Computer and then HP, McDowell is taking a sabbatical.)

If it meets expectations, RDMA links will be used to link small servers into a collection that can handle a large database that today must be housed on a much more expensive server with dozens of processors. That vision, long touted by companies such as Microsoft and Intel with strength in lower-end servers, is getting closer with database software such as Oracle's 9i RAC and IBM's DB2 EEE, but many still believe it's still years away from broad use.

RDMA took a significant step closer to reality last week as the RDMA Consortium released the second of three sets of specifications needed to build the technology into hardware and software. And it will take another step this week when Microsoft announces additional backing at its Windows Hardware Engineering Conference (WinHEC).

"You're going to hear a lot more of the pieces of this from Microsoft," said Karl Walker, chief technology officer for HP's Intel-based server group. Linux programmers also are actively involved, Walker said.

On Wednesday, Microsoft plans to announce that RDMA will be necessary to take advantage of forthcoming 10 gigabit-per-second Ethernet networks and will discuss how RDMA supports a company plan code-named Chimney to enable easy use of network-accelerating hardware called TCP/IP offload engines (TOEs), the WinHEC conference agenda said.


Reader Resources
Inifiniband
ZDNet White Papers

IBM is another backer. "If you line up RDMA and TOEs and 10-gigabit Ethernet, you have a very competitive technology. That's what InfiniBand does today," said Tom Bradicich, chief technology officer of IBM's Intel-based server line and one of the key engineers behind the InfiniBand technology.

The RDMA technology opens a new chapter in a long debate about the best way to make big servers. Traditionally, systems that can handle hundreds of thousands of transactions per minute are made with single, expensive machines with dozens of processors--"scale up," in industry parlance. But many advocate a less proven method, sharing the job over inexpensive machines joined with high-speed links--"scale out."

The debate pits industry stalwarts such as IBM and Sun Microsystems--with expertise in massive systems--against comparative upstarts such as Dell Computer and Microsoft, though most companies embrace both strategies in varying degrees.

RDMA, like InfiniBand, is a tool that could make the scale-out approach more feasible. "If you want no-compromise performance, InfiniBand is one of your best answers. But if you want scale-out for the masses, RDMA over TCP is an awfully compelling story," Walker said.

"The scale-out versus scale-up debate been going on in computer architecture circles for about 30 years," Walker said. "I don't think this debate is going to be over tomorrow, but we think the needle is moving...to shift to scale-out apps."

Companies vary on when they expect RDMA to arrive in the marketplace.

"I believe the deployment in product is in the 2005 timeframe," Bradicich said. He believes the technology won't catch on until 10-gigabit Ethernet using copper wires arrives, which he expects in 2004.

But Walker said HP believes RDMA-enabled products--operating systems and network cards--will be on sale in 2004. "We are architecting it so the technology is applicable to 1 gigabit," which is in widespread use today, not just 10-gigabit Ethernet coming later.

RDMA lets one computer directly read from or write to the memory of another without having to go through the time-consuming process required today, in which a request for data must move through the operating system and other layers of software. That delay, or latency, is the single most important issue for RDMA and the extent to which it will be able to succeed as a mainstream technology where InfiniBand failed.

The InfiniBand latency is about eight to 12 microseconds, Walker said, and a reasonable hope for RDMA using 1 gigabit-per-second Ethernet is in the range of 15 to 18 microseconds.

"If they can get 15 to 18 microseconds, that's pretty good," said Illuminata analyst Jonathan Eunice. "With the RDMA folks, the risk is that they shoot for 15 and get 35 or 40. The InfiniBand folks, for example, shot for two to five (microseconds), and they got about 10."

In any case, InfiniBand likely will have at least 18 months with the market to itself while RDMA details are worked out, Bradicich said.

If RDMA catches on, it won't be the first time a slower technology that benefits from the widespread use of Internet Protocol challenges a faster but comparatively nonstandard technology. In storage, the iSCSI protocol for sending storage data over IP networks is gaining on the incumbent technology, Fibre Channel, despite the latter's maturity and speed.

"Fibre Channel had a 10-year lead, then woke up one morning, looked out the rear-view mirror and saw iSCSI coming up its tailpipe," Walker said.

RDMA matches up well with iSCSI, with the ability to make the storage transfer standard more efficient, Walker said.

Much of RDMA technology came directly from work pioneered by InfiniBand engineers, Bradicich and Walker said. For example, InfiniBand has a "verbs" specification to ensure that software such as an operating system is aligned with hardware features. The RDMA Consortium followed suit with its own very similar verbs specification last week.

InfiniBand also influenced other technology, such as the PCI Express expansion slots for next-generation PCs as well as high-speed cabling technology used in IBM's x440 Intel servers, Bradicich said.

InfiniBand has met with modest success in some high-performance niches, though it didn't become the ubiquitous technology early advocates had in mind. First Intel and then Microsoft retreated from InfiniBand plans. In addition, IBM backed off plans to produce standard InfiniBand chips that companies could employ in their products, although it has continues its business selling customized InfiniBand chips for specific customers, a spokesman said.

InfiniBand is moving ahead, with IBM selling a database product using the technology and supercomputer makers interested as well. Several server makers rallied around InfiniBand in December; Sun in particular has high hopes for the technology.

The RDMA Consortium is building a specification intended to be mature enough for widespread use, but in the longer term, the group wants to hand over standardization to the Internet Engineering Task Force, which governs many Internet Protocol-related technologies and with which the RDMA Consortium already is working closely.

"The RDMA Consortium doesn't want to exist long term as another standards body," Walker said. The IETF is expected to charter its RDMA over Internet Protocol Suite Working Group in coming months, the RDMA Consortium said.

In addition to IBM, HP and Microsoft, the founding members of the RDMA Consortium are Intel, Dell, Cisco, EMC, Adaptec, and Network Appliance.

Seeking a better way to make high-end database servers from low-end parts, Hewlett-Packard, IBM and others have begun working on a less expensive alternative to the InfiniBand high-speed networking technology.

The technology, called Remote Direct Memory Access (RDMA), borrows heavily from the years of work that went into creating InfiniBand, according to engineers involved in the work. While RDMA will arrive later and perform slower than InfiniBand, it will have the advantage of using conventional networking hardware based on the TCP/IP and Ethernet standards used to build the Internet.

"We're pushing hard on RDMA," Mary McDowell, the head of HP's Industry Standard Server group, said in an interview. (She spoke days before Friday's announcement that after years running Intel-based servers for Compaq Computer and then HP, McDowell is taking a sabbatical.)

If it meets expectations, RDMA links will be used to link small servers into a collection that can handle a large database that today must be housed on a much more expensive server with dozens of processors. That vision, long touted by companies such as Microsoft and Intel with strength in lower-end servers, is getting closer with database software such as Oracle's 9i RAC and IBM's DB2 EEE, but many still believe it's still years away from broad use.

RDMA took a significant step closer to reality last week as the RDMA Consortium released the second of three sets of specifications needed to build the technology into hardware and software. And it will take another step this week when Microsoft announces additional backing at its Windows Hardware Engineering Conference (WinHEC).

"You're going to hear a lot more of the pieces of this from Microsoft," said Karl Walker, chief technology officer for HP's Intel-based server group. Linux programmers also are actively involved, Walker said.

On Wednesday, Microsoft plans to announce that RDMA will be necessary to take advantage of forthcoming 10 gigabit-per-second Ethernet networks and will discuss how RDMA supports a company plan code-named Chimney to enable easy use of network-accelerating hardware called TCP/IP offload engines (TOEs), the WinHEC conference agenda said.


Reader Resources
Inifiniband
ZDNet White Papers

IBM is another backer. "If you line up RDMA and TOEs and 10-gigabit Ethernet, you have a very competitive technology. That's what InfiniBand does today," said Tom Bradicich, chief technology officer of IBM's Intel-based server line and one of the key engineers behind the InfiniBand technology.

The RDMA technology opens a new chapter in a long debate about the best way to make big servers. Traditionally, systems that can handle hundreds of thousands of transactions per minute are made with single, expensive machines with dozens of processors--"scale up," in industry parlance. But many advocate a less proven method, sharing the job over inexpensive machines joined with high-speed links--"scale out."

The debate pits industry stalwarts such as IBM and Sun Microsystems--with expertise in massive systems--against comparative upstarts such as Dell Computer and Microsoft, though most companies embrace both strategies in varying degrees.

RDMA, like InfiniBand, is a tool that could make the scale-out approach more feasible. "If you want no-compromise performance, InfiniBand is one of your best answers. But if you want scale-out for the masses, RDMA over TCP is an awfully compelling story," Walker said.

"The scale-out versus scale-up debate been going on in computer architecture circles for about 30 years," Walker said. "I don't think this debate is going to be over tomorrow, but we think the needle is moving...to shift to scale-out apps."

Companies vary on when they expect RDMA to arrive in the marketplace.

"I believe the deployment in product is in the 2005 timeframe," Bradicich said. He believes the technology won't catch on until 10-gigabit Ethernet using copper wires arrives, which he expects in 2004.

But Walker said HP believes RDMA-enabled products--operating systems and network cards--will be on sale in 2004. "We are architecting it so the technology is applicable to 1 gigabit," which is in widespread use today, not just 10-gigabit Ethernet coming later.

RDMA lets one computer directly read from or write to the memory of another without having to go through the time-consuming process required today, in which a request for data must move through the operating system and other layers of software. That delay, or latency, is the single most important issue for RDMA and the extent to which it will be able to succeed as a mainstream technology where InfiniBand failed.

The InfiniBand latency is about eight to 12 microseconds, Walker said, and a reasonable hope for RDMA using 1 gigabit-per-second Ethernet is in the range of 15 to 18 microseconds.

"If they can get 15 to 18 microseconds, that's pretty good," said Illuminata analyst Jonathan Eunice. "With the RDMA folks, the risk is that they shoot for 15 and get 35 or 40. The InfiniBand folks, for example, shot for two to five (microseconds), and they got about 10."

In any case, InfiniBand likely will have at least 18 months with the market to itself while RDMA details are worked out, Bradicich said.

If RDMA catches on, it won't be the first time a slower technology that benefits from the widespread use of Internet Protocol challenges a faster but comparatively nonstandard technology. In storage, the iSCSI protocol for sending storage data over IP networks is gaining on the incumbent technology, Fibre Channel, despite the latter's maturity and speed.

"Fibre Channel had a 10-year lead, then woke up one morning, looked out the rear-view mirror and saw iSCSI coming up its tailpipe," Walker said.

RDMA matches up well with iSCSI, with the ability to make the storage transfer standard more efficient, Walker said.

Much of RDMA technology came directly from work pioneered by InfiniBand engineers, Bradicich and Walker said. For example, InfiniBand has a "verbs" specification to ensure that software such as an operating system is aligned with hardware features. The RDMA Consortium followed suit with its own very similar verbs specification last week.

InfiniBand also influenced other technology, such as the PCI Express expansion slots for next-generation PCs as well as high-speed cabling technology used in IBM's x440 Intel servers, Bradicich said.

InfiniBand has met with modest success in some high-performance niches, though it didn't become the ubiquitous technology early advocates had in mind. First Intel and then Microsoft retreated from InfiniBand plans. In addition, IBM backed off plans to produce standard InfiniBand chips that companies could employ in their products, although it has continues its business selling customized InfiniBand chips for specific customers, a spokesman said.

InfiniBand is moving ahead, with IBM selling a database product using the technology and supercomputer makers interested as well. Several server makers rallied around InfiniBand in December; Sun in particular has high hopes for the technology.

The RDMA Consortium is building a specification intended to be mature enough for widespread use, but in the longer term, the group wants to hand over standardization to the Internet Engineering Task Force, which governs many Internet Protocol-related technologies and with which the RDMA Consortium already is working closely.

"The RDMA Consortium doesn't want to exist long term as another standards body," Walker said. The IETF is expected to charter its RDMA over Internet Protocol Suite Working Group in coming months, the RDMA Consortium said.

In addition to IBM, HP and Microsoft, the founding members of the RDMA Consortium are Intel, Dell, Cisco, EMC, Adaptec, and Network Appliance.

Editorial standards