Prepare for Gigabit Ethernet networking

If you’ve been working with networks as long as I have, you probably remember the days when 10-Mbps Ethernet seemed fast. And when 100-Mbps "Fast" Ethernet debuted--Wow! Just when it seemed as if Ethernet couldn’t get any faster, along came Gigabit Ethernet with speeds up to 1,000 Mbps (1 Gbps).

Even if you’re not deploying Gigabit Ethernet right now, eventually you will. After all, just look at all of the Fast Ethernet equipment in your wiring closet. However, just because you’re not deploying Gigabit Ethernet today doesn’t mean you can’t get your network ready to handle it. In this article, I’ll show you what you need to do to prepare for Gigabit Ethernet.

What’s so great about Gigabit Ethernet?
Although Gigabit Ethernet is growing in popularity, few people I know are actually running it yet. When I ask why, many explain that they’d been burned by ATM and weren’t ready to go through that whole experience again. ATM was originally touted as the replacement for Ethernet. It was faster than Ethernet and avoided some of Ethernet’s initial problems, such as packet collisions.

However, ATM can be costly. It’s also difficult to set up and doesn’t work well in environments that require routing between many subnets. In the end, ATM’s performance can be disappointing.

This is what makes Gigabit Ethernet such a great technology. While Gigabit Ethernet is a high-speed medium, it’s still Ethernet-based. This means that setting up a Gigabit Ethernet network isn’t much different from setting up any other type of Ethernet network.

Best of all, since Gigabit Ethernet is a true Ethernet medium, it will interface with your existing Ethernet network. A Gigabit Ethernet switch can move packets between 10/100-Mbps and 1,000-Mbps networks without any type of packet translation. This not only leads to better efficiency than you’d get with ATM, but it also means less complexity and therefore less chance that something will go wrong.

How fast is it really?
Because Gigabit Ethernet supports transmission speeds of roughly 1,000 Mbps, many network gurus assume that installing a few Gigabit Ethernet NICs and attaching the cabling and a high-speed switch will make their networks perform at warp speeds. However, for several reasons, the actual network speed will probably be a bit less.

Suppose for a moment that you were to install Gigabit Ethernet NICs in a server and a workstation and then connect the two machines with a gigabit switch. Assuming there is no other traffic on the network, you’d probably expect traffic to flow between the two machines at 1,000 Mbps. Unfortunately, you’d be sadly disappointed. The truth is that, in most installations, Gigabit Ethernet implemented in the manner I’ve just described doesn’t even come close to reaching gigabit speeds. In the example above, the best you could hope for is typically between 700 and 800 Mbps.

While 700 Mbps is a huge improvement over the 100-Mbps speed of Fast Ethernet, you’re probably wondering why traffic is flowing at 200 to 300 Mbps below its potential. There are several factors that reduce Gigabit Ethernet’s performance. One of the most common factors is the cabling.

Cabling considerations
One of the biggest issues to take into account when implementing Gigabit Ethernet is cabling. When you first read the specs on Gigabit Ethernet, it sounds like an ideal technology, in part because it’s compatible with the Category-5 (Cat-5) cable that you already have. However, just because you can use your existing Cat-5 cable for Gigabit Ethernet, it doesn’t necessarily mean that you should.

Most big companies will probably be OK with existing copper Cat-5 cable. However, I’ve done network repair for many small organizations, and the cabling just wasn’t up to par in more of them than I can count. For example, in one location, most of the PCs had 10/100 NICs, but the company was still using a 10-Mbps hub. I swapped the hub out for a 10/100 model, and all of the PCs with 100-Mbps cards began to fail. Upon closer examination, I found that voice-grade phone cabling had been used instead of Cat-5 cable. When I explained the problem to the facility’s manager, he said that he had told the installers to use phone cable because it was cheaper than Cat-5 cable.

This is an extreme example, but other cable issues tend to be much more common. For instance, although Cat-5 cable has eight wires, 10/100-Mbps Ethernet uses only four of them. I’ve seen quite a few organizations in which the cable installer saved time by connecting only the four wires that were actually used. In most cases like this, the other four wires are simply cut off. But I’ve also seen situations in which the additional wires were used to attach a second PC to the network or as wiring for a phone jack.

Even if your cable installer didn’t use cheap cable or neglected to connect half of the wires in the cable, other Cat-5 issues could cause problems when you use Gigabit Ethernet. For example, 10/100-Mbps Ethernet standards require that no cable run exceed 100 meters. However, Ethernet and Fast Ethernet tend to be very forgiving when cable lengths are exceeded, so many organizations ignore the limit. Ignoring the 100-meter limit will come back to bite you when you implement Gigabit Ethernet.

Likewise, I’ve seen many Ethernet and Fast Ethernet networks in which Cat-5 cable segments are spliced together. While this isn’t supposed to work, it does when they're spliced well. But splices tend to cause big problems for Gigabit Ethernet.

Before implementing Gigabit Ethernet, I strongly recommend making sure that your cabling is up to the job. I suggest running three types of tests on your cabling. First, run a continuity test to verify that all eight wires are properly connected. Next, run a far-end crosstalk test. Finally, run a return signal loss test. If your cabling passes all three tests, the cable is adequate for Gigabit Ethernet use.

Speeding things up
As you can see, the condition and length of your Cat-5 cable can greatly affect your network’s performance. One way of getting around the problems caused by poor cable conditions or longer cable runs is to use fiber-optic cable instead of copper cable. Gigabit Ethernet is designed to work with either fiber or copper cable. However, merely switching to fiber-optic cable won’t solve all your problems.

Fiber-optic cable is just as susceptible to problems as copper cable. Although it’s true that fiber is more secure than copper and supports higher data speeds and longer runs, it's also much more delicate than copper. Things like excessive epoxy on the cable ends, poorly made splices, too many splices, cable damage, and excessive cable bends can dramatically slow or even stop network traffic flowing through fiber. I personally prefer using fiber over copper in many environments, but even using good quality, properly installed fiber will only help you so much.

The reason is that most computers are incapable of producing packets at gigabit rates. Currently, most PCs produce packets at the CPU level and then pass them through the PCI bus to the NIC and across the network. There are processors that can produce packets at gigabit speeds, but it’s important to remember that most of the time, your processors are busy doing things other than producing packets. For example, in addition to producing packets, a server’s processor is managing memory, running services, and maintaining the user interface.

Windows 2000 is more efficient than Windows NT and can produce packets faster, but this is still typically not enough to utilize a gigabit NIC’s full potential. There are several new technologies that will help you get the most out of Gigabit Ethernet and may soon allow 10- or even 100-gigabit connections.

The PCI-X bus and other new technologies
One such technology is the new PCI-X bus, which is similar to the PCI bus but isn’t as bandwidth-intensive when used in conjunction with PCI-X cards. This means that when a PCI-X bus and a PCI-X-based NIC are in use, packets can flow between the CPU and the NIC much more quickly than they could on a purely PCI machine.

Although faster bus speeds will usually allow a gigabit connection to utilize its full potential, other new technologies are at work. Some companies are developing NICs that have built-in microprocessors. The idea is to have a CPU dedicated to the sole task of generating packets. Because this CPU is integrated into the card, it’s guaranteed to be fast enough to produce packets at speeds for which the card is intended (1, 10, or 100 Gbps).

Still another experimental technology is bonding. Several NIC manufacturers have prototype NICs with multiple onboard fiber-optic ports. These NICs use several cables at once for parallel traffic flow, and they have an onboard microprocessor that does IP processing at the card level.

Currently, Gigabit Ethernet tends to be a bit pricey, and it probably won’t deliver true gigabit performance across your network. I still believe that implementing Gigabit Ethernet is worthwhile in any organization with growing bandwidth needs. After all, a well-installed Gigabit Ethernet connection will perform at least seven times better than a standard Fast Ethernet connection.

The challenge when implementing Gigabit Ethernet is getting the most performance for the least cost. I recommend beginning the rollout by replacing your existing switches with switches that support 10/100/1000-Mbps connections and that support both fiber and copper.

Once you’ve replaced the switches, the next trick is to figure out where to begin implementing gigabit connections. I suggest using Gigabit Ethernet for all connections between switches. Remember that at any given time, a lot of traffic is probably flowing between your switches. Placing gigabit connections between the switches will prevent the switches from becoming a bottleneck.

Best of all, most switches aren’t PCI-based and therefore can achieve true gigabit speeds. This means that, assuming your cabling is good, the traffic flowing between your switches can actually flow at 1,000 Mbps, regardless of the limits on other gigabit connections on your network.

The next thing I’d recommend is installing two gigabit NICs in each server. One of the gigabit NICs should be attached to one of your switches. Since workstations are also connected to the switches (but at lower speeds), this allows traffic to flow between the workstations and the servers. The reason for implementing this architecture is that it prevents a server’s network interface from becoming a bottleneck.

For example, suppose that a workstation with a 100-Mbps NIC began a network-intensive operation, such as copying a huge file off a server. If the server and the workstation both have 100-Mbps NICs, it would be possible for the workstation to consume most of the server’s available bandwidth. Of course, there are situations that would prevent this from happening, such as when traffic is already excessively high or when QoS is in use. But generally speaking, if the workstation consumes most of the server’s bandwidth, there’s little left for anything else. However, if the workstation has a 100-Mbps NIC and the server has a gigabit NIC, the workstation won’t even come close to consuming all of the available bandwidth.

I’d recommend using the server’s second NIC to connect to a dedicated switch that is linked only to servers (not workstations). Having a dedicated backbone between the servers makes it possible for server-related traffic, such as that generated by replication and other network functions, to flow through a dedicated network without placing any traffic on the main network.

I’d also suggest implementing specific gigabit connections with fiber-optic cable. Use fiber for any connection to a switch or server, and for any gigabit connection that requires a cable run exceeding 100 meters or that flows through an area in which radio interference, crosstalk, or attenuation might be a problem.

You should run your workstations at 100 Mbps for reasons that I explained earlier. As Gigabit Ethernet NICs become cheaper, you may later want to upgrade your desktop computers to gigabit speeds. If you do, however, I strongly advise that you run copper cable to the desktops. I’ve seen far too many cases of cable abuse over the years to recommend running a fiber-optic cable to someone’s desk. Fiber-optic cable is too delicate to survive the abuse that users can subject a cable to.

Pump up the volume
Although relatively few people are using Gigabit Ethernet compared to other forms of Ethernet, it's quickly becoming more affordable, and eventually, it will be as widely used as Fast Ethernet is now. Planning for Gigabit Ethernet today will save you money and effort when you eventually make the switch. Before you know it, you’ll be running your network at gigabit speeds and wondering how it ever worked when it was slower.