Linux to face off against NT again?
A third-party testing lab run by Neal Nelson & Associates is preparing for an August rematch between the two server operating systems. And, as was the case with the last PCWeek Labs-sponsored Mindcraft benchmark, representatives from Microsoft Corp. and the open-source community are both expected to participate in the testing process.
Nelson, who has made a name for himself in government-related benchmarking, has invited Microsoft, Red Hat Software Inc. and other open-source vendors to his Chicago testing lab. Nelson will act as referee, but a committee will decide on the specifics of the tests, he says. The panel will be comprised of 25 percent Microsoft, 25 percent Linux, 25 percent industry press and 25 percent "other" (academic, industry and possibly government).
Nelson declined to make public names of all of those who have agreed to participate, but did say that the Red Hat manager of IHV technical relations, Doug Ledford, has agreed to be on the panel. A Microsoft spokesman said Microsoft has no plans to participate on the panel or the benchmarking tests itself.
Nelson says his premise going into the tests will be that NT is best suited for some tasks, and Linux for others. Nelson's lab features 96 Pentium computers preconfigured to run Windows, Windows NT and Linux. The computers can support more than 5,000 live client sessions; more than 10 gigabits per second of network bandwidth; isolated power feeds; and the capacity to run 24 hours a day during high intensity tests.
The focus of Nelson's tests, like the two Mindcraft tests before them, will be on performance, he says. But unlike previous Linux vs. NT benchmarks, which as Nelson said, have "been oriented towards Microsoft clients," the forthcoming benchmarks will likely look at the performance of Berkeley Standard Distribution and Network File System clients. "The fact that it's not Microsoft's lab or a publication's facility will be different, too," Nelson said.
Nelson expects to test the two operating systems configured in multiple ways: as NFS file servers, SMB (Samba) file servers, e-mail servers, FTP servers and Web servers. He expects to be examining them under different workloads and on machines that were configured differently (i.e, single process, multiprocessor, fast and slow CPUs, small and large memory size, RAID or non-RAID disk configurations).
Microsoft sponsored the first Linux vs. NT benchmark performed by Mindcraft at Microsoft's own facilities in April. PCWeek Labs stepped in to host Mindcraft's second benchmark earlier this month. In both Mindcraft benchmarks, NT Server was found to outperform Linux -- a finding that set off howls of protest among Linux loyalists. Neal Nelson & Associates has no particular vendor affiliation, says Nelson. Its clients do include FedEx., IBM Corp., McDonald's, Northern Telecom, Sprint and major government agencies including the Internal Revenue Service, U.S. Department of the Air Force, U.S. Department of the Navy and U.S. Department of the Army.
Nonetheless, some open-source companies say they are gun-shy to be part of yet another benchmark (or, as some call it, benchmarketing) exercise. "The tests have been measuring speed only," says Andy Kaufman, communications director for Penguin Computing Inc., a Linux hardware vendor. "But the real world wants measures of reliability, expandability and scalability.
"How much do benchmarks really tell you anyways?" Kaufman continues. "Linux is meeting companies' very immediate business concerns. Our benchmark is our customers."
Take me to the Linux Lounge