Linux vendors hit back at Ballmer's email

Novell and Mandrakesoft have accused Microsoft's chief executive of being selective with the facts in his latest email attacking Linux

In a 2,600-word email sent to Microsoft customers and partners on Wednesday, Microsoft's Steve Ballmer drove home the key themes of Microsoft's "Get The Facts" campaign. Bill Gates' second in command claimed that Windows was a better choice than Linux in terms of security, total cost of ownership (TCO) and protection against legal action over patent violations.

"And it's pretty clear that the facts show that Windows provides a lower total cost of ownership [than Linux]; the number of security vulnerabilities is lower on Windows; and Windows responsiveness on security is better than Linux; and Microsoft provides uncapped IP indemnification of their products, while no such comprehensive offering is available for Linux or open source," he stated.

In the aftermath of Ballmer's mail, ZDNet UK sought out a reaction from the various Linux vendors to Microsoft's allegations that the open-source OS is expensive and untrustworthy.

Novell vice-president of strategic marketing John Hogan's first response to the various points in the Ballmer email is that Microsoft has cherry-picked sections of the reports to back up its 'facts'. Novell bought SuSE Linux in January 2004.

"Not surprisingly, the points made by Mr Ballmer leverage only those statements in its commissioned studies that reflect most positively on Microsoft," says Hogan. "A broader look paints a much more objective picture, one more favourable to Linux."

French Linux vendor Mandrakesoft levels the same criticism at Microsoft. "Microsoft has a well-know tendency to somewhat stretch the definition of 'fact'," says a spokesman from Mandrakesoft. "The 'facts' referred to in the above [Ballmer's] quote originally appeared in Microsoft-funded studies, the independence of which is, at best, doubtful."

Objectivity
Ballmer attempted to head off claims of bias in his email, claiming that the research cited is independent. "In each case, the research methodology, findings and conclusions were the sole domain of the analyst firms. This was essential: we wanted truly independent, factual information."

Novell's Hogan says that Microsoft has often specified the exact system configuration in benchmark tests. For example, in two Veritest studies which compare Windows 2003 Server with Linux, Hogan asserts that Veritest fine-tuned the Windows set-up but did not do the same for Linux.

"The test used Windows protocols only, while Linux had to emulate the Windows protocols using Samba," says Hogan. "As far as we can see, the testers did not even make the smallest optimisation for this Linux/Samba setting, while Microsoft helped Veritest fine tune on Windows."

Hogan also claims that Microsoft turned off the Windows 8.3 file-naming convention and made tweaks to the TCP stack on the client machines and to the buffer-cache pool on the server. "Obviously, Microsoft invested considerable time and effort in finding the best possible configuration," he says.

Total cost of ownership
In the email, Ballmer attempts to reinforce Microsoft's claim to lower TCO with findings from a Yankee Group from April 2004, entitled "Linux, UNIX and Windows TCO Comparison". He claims the study concludes that upgrading Windows was cheaper than switching to Linux for large enterprises.

But according to Novell, the Yankee Group study also found that the relative TCO of Microsoft and Linux varies according to situation and that the TCO of Linux is considerably lower in, for example, small firms, organisations with customised vertical applications and in "greenfield" sites

An alternative report on TCO carried out by Research and Markets found that Linux had a 30 percent lower TCO than Windows, according to Mandrakesoft.

Training
In the section of the email on TCO, Ballmer also raised issues of the cost and availability of trained Linux resources to support Linux deployments, citing a Forrester Report, from early 2004, entitled The Costs and Risks of Open Source.

The Microsoft chief executive claimed Forrester found that training for IT employees was 15 percent more expensive for Linux than for Windows due to a lack of internal knowledge and a lack of availability of training materials.

But according to Novell's Hogan, the Forrester report also states that the added training costs for Linux are transitory and will reduce as companies gain more experience with the OS.

Security
The Ballmer memo also quotes data from a Forrester study entitled Is Linux More Secure than Windows?. It states that according to the analysts' study the four major Linux distributions have a higher incidence and severity of vulnerabilities, and are slower than Microsoft to provide security updates.

Representatives from Red Hat, Novell and Mandrakesoft claim the Forrester report was flawed.

Earlier this summer, Mark Cox from Red Hat's security response team told ZDNet UK that his firm had worked closely with Forrester, and that these findings were flawed because the analyst group had just taken a simple average of the data.

"An average is not representative. Red Hat fixes issues which other operating systems wouldn't fix, such as temporary file vulnerabilities," said Cox, adding that the report also failed to take into account the severity of the issues.

"A vulnerability which could allow a remote attack on Windows was considered in the same light as a file vulnerability on Linux which makes the system slow down," said Cox.

Novell's Hogan agrees with Cox that the report fails to take into account severity. "Mr Ballmer failed to mention that the study found Microsoft had the highest number of critical flaws," he says.

Hogan also says that the study measured the time to fix a flaw from the time it is made public, which is different for Microsoft and open source. "In open source, this is immediate, so a fix can be generated quickly. Microsoft delays making the existence of a flaw known as long as possible, unless your company has signed a special non-disclosure agreement with them," he says.

Indemnification
In the email, Ballmer wrote that "it is rare for open-source software to provide customers with any indemnification at all."

The response from Red Hat and Novell SuSE executives is that both companies provide indemnification against intellectual property.

Red Hat provides a warranty to Red Hat Enterprise Linux customers which guarantees to replace software if there is an intellectual property issue so they can continue using the product without interruption. It also has a fund which assists companies with any legal expenses associated with litigation related to the development of software under an open-source licence.

Novell provides indemnification to customers of versions 8 and 9 of SuSE Linux Enterprise Server. It states that it will pay damages up to $1.5m for an allegedly infringing product. On top of this it will pay legal defence fees.

UNIX Migration
On the subject of migrating enterprise resource planning systems from UNIX to Windows or other platforms, Ballmer highlighted a survey purporting gains in performance by moving to Windows and suggests that Windows outperforms Linux in UNIX migration scenarios.

But Novell claims that an alternative analysis by Flexdata comparing Windows 2003 with SuSE Enterprise Server 9 (SLES9) found that it performed better on the same hardware.

Despite the distinct lack of common ground, there are some points that Microsoft and Novell agree on. At the end of Ballmer's email, he directs customers toward the "Get the Facts" section on the Microsoft Web site. Novell's Hogan agrees that people should read these reports to get a more complete picture.

"Read the complete reports on Microsoft's site, not just Microsoft's chosen sound bites," he says.

Gael Duval, the founder of Mandrakesoft, is less forgiving and sums up Microsoft's email as an attempt to spread confusion.

"We think that Microsoft is trying a new strategy to fight against Linux by spreading much FUD (fear, uncertainty and doubt) about Linux' strongest points."

Novell's full response follows

Novell's response
The following comments follow the flow of Mr. Ballmer's letter. Not surprisingly, the points made by Mr. Ballmer leverage only those statements in its commissioned studies that reflect most positively on Microsoft. A broader look paints a much more objective picture, one more favorable to Linux. 1. OBJECTIVITY: In teeing up the research results, Mr. Ballmer states that "In each case, the research methodology, findings and conclusions were the sole domain of the analyst firms. This was essential: we wanted truly independent, factual information." This is somewhat at odds with what transpired. Microsoft generally specified the configurations to be used. As an example... Based on two studies on Microsoft's "Get The Facts" website entitled "Windows Server 2003 Outperforms Linux for File Serving" and "Windows Beats Red Hat in Multiple Configuration Web Server Benchmark Tests" (Veritest 2003 and 2004), Microsoft concludes that Microsoft Windows 2003 Server has higher performance than Linux as a file- or web server. However, the test used Windows protocols only, while Linux had to emulate the Windows protocols using Samba. As far as we can see, the testers did not even make the smallest optimization for this Linux/Samba setting, while Microsoft helped Veritest fine tune on Windows. Microsoft provided a registry setting that turns off the standard Windows 8.3 file-naming convention. Another tweak was made to the TCP stack on the client machines. Yet another tweak was made to the buffer-cache pool on the server. Obviously, Microsoft invested considerable time and effort in finding the best possible configuration. 2. TOTAL COST OF OWNERSHIP: To support his TCO arguments, Mr. Ballmer quotes extensively from Yankee Group's report entitled "Linux, UNIX and Windows TCO Comparison" Yankee Group, April 2004". That report, available on Microsoft's site, also states the following, which Ballmer did not cite:
  • "...corporate customers report Linux provides businesses with excellent performance, reliability, ease of use and security. Yes, Linux is a viable alternative to UNIX and Windows. In addition, Linux is the most serious competition to Microsoft’s dominance in the server operating system market to date."
  • "Linux shows measurably improved TCO compared with UNIX and Windows in small firms, in organizations with customized vertical applications and in "greenfield" networking situations where there is no existing software infrastructure."
  • "The ability to modify and customize the Linux source code affords customers the most intriguing possibilities for custom application development. This ability stands in stark contrast to the closed or proprietary nature of the Windows operating system. In recent years, Microsoft has opened up Windows to a limited extent and released numerous APIs. This enables third-party ISVs to efficiently produce interoperable applications that more easily integrate with Windows. However, this is nothing like the changes developers can make with Linux, where there is total access. The open source philosophy is deceptively simple: allowing developers, programmers and engineers to read, modify and redistribute the source code via standardized Linux interfaces spurs software development and evolution."
  • "In summary, the Yankee Group’s TCO survey found that Linux does offer compelling cost savings, economies of scale and technical advantages, as many a satisfied user will attest. However, the cost savings and benefits are not automatic; they are not achieved without customer due diligence and they do not necessarily apply in every user scenario. Ultimately, the TCO and ROI of Linux may be less than, comparable to, or more expensive than UNIX or Windows depending on the individual corporate deployment circumstances."
3. TRAINED RESOURCES: Mr. Ballmer brings up the issue of the cost and availability of trained Linux resources to support Linux deployments, citing a Forrester Report titled "The Costs and Risks of Open Source." However, that study concludes, "We found the adoption of Linux and other open source components is accelerating in key areas of the enterprise." This acceleration of growth wouldn't be possible if lack of availability of Linux resources were truly a mitigating factor for customers. Linux expertise is extensive, and growing rapidly. Computer science graduates today have grown up on open source, not Windows. Evans Data Corporation, in their Linux Development Survey dated Summer, 2004, shows that there are 1.2 Million Linux developers and growing. Evans further states that "For the first time, the developers we surveyed are actually using and targeting Linux and UNIX at an almost equal rate with Windows, and subtle but meaningful trends indicate that the day is coming very soon when Linux will dominate software development." Forrester makes a clarifying comment regarding those companies who were expending more effort in their Linux deployments: "This is not unexpected, since most of these firms are just beginning to establish operating procedures and practices for open source — for many, their Linux projects served as the catalyst for this effort. These preparation and planning activities took 5% to 25% longer for Linux than Windows. This should change, of course, as companies gain more experience with the platform." The report concludes that these costs are transitory.

4. SECURITY: Mr. Ballmer brings up the issue of security, which admittedly must be much on his mind. He states "I think it's fair to say that no other software platform has invested as much in security R&D, process improvements and customer education as we have at Microsoft." Novell applauds Microsoft's continued efforts to improve their product quality. Novell deplores any malicious attack or any company or any software. But the reality is that the financial impact to the economy and to customers of the malicious attacks on Microsoft products has run into the billions.

Mr. Ballmer further states "We believe in the effectiveness of a structured software engineering process that includes a deep focus on quality, technology advances, and vigorous testing to make software more secure". We cannot argue that point, but isn't this the same process used in developing the products that have been plagued by malicious attacks? Something has to change. Open Source provides an equally structured process, but different than the one Microsoft utilizes. Open source – modular in its nature - is much more flexible and, being open, it's processes and code are much more amenable to scrutiny and improvement. Partly for this reason, Linux has a strong security record. Mr. Ballmer brings up the Forrester report titled "Is Linux More Secure than Windows?" He concludes that the study "highlighted that the four major Linux distributions have a higher incidence and severity of vulnerabilities, and are slower than Microsoft to provide security updates." Mr. Ballmer failed to mention that the study found Microsoft had the highest number of critical flaws. 67 percent of Windows flaws had been rated "critical", under the U.S. National Institute for Standards and Technology's ICAT project standard for high-severity vulnerabilities. This compared to 63 percent for (pre-Novell) SuSE Linux, 60 percent for MandrakeSoft, 57 percent for Debian and 56 percent for Red Hat. Note also that this study measures the time to fix a flaw from the time it is made public. In open source, this is immediate, so a fix can be generated quickly. Microsoft delays making the existence of a flaw known as long as possible, unless your company has signed a special non-disclosure agreement with them. The Forrester study does not take this differing public start time into account. This is like a golfer starting on a tee closer to the hole saying they are a better golfer because they have fewer strokes. The Yankee Group study that Mr. Ballmer referred to earlier in his message states "Overall, a 76 percent majority rated Linux and UNIX reliability comparable, while 70 percent of the respondents rated Windows Server 2003 reliability equal to Linux. However, Windows administrators complained about the amount of network administration time and manpower spent performing security and patch management functions in their environments. In addition, although Windows servers—particularly the newer Windows Server 2003—rarely crashed, the administrators often said installing a critical security patch comes with unplanned downtime. This is because they did not want to risk delay in applying the patch until off-peak hours or the weekend. Overall, security and patch management were clearly the biggest problems for corporate customers. In addition, from a customer’s standpoint, they are the most glaring Windows weaknesses. In this regard, only 12 percent of Windows 2000 customers said that the Microsoft platform was comparable to Linux. Security and patch management specific reliability improved somewhat for Windows Server 2003 —with 18 percent reporting that it is comparable to Linux reliability in terms of unnecessary reboots." Evans Data Corporation, in their Linux Development Survey dated Summer, 2004, shows:
  • Ninety two percent of survey respondents indicated that their Linux systems have never been infected with a virus.
  • Fewer than 7% said that they'd been the victims of three of more unauthorized intrusions.
  • Only 22% of Linux developers said that their systems had ever been invaded (of those, almost a quarter of cases (23%) involved unauthorized intrusion initiated by companies' employees, i.e. people having available accounts allowing to log in corporate Linux servers.)
A similar survey by Evans last spring found that nearly 60% of non-Linux developers admitted they'd been victimized by security breaches, and 32% had been hit three or more times.
  • Twenty five percent of developers believe that the Linux operating system has the best innate security
  • Nine of ten companies developing Linux claim that their systems have never been infected by a virus, while four of five companies assert that their systems haven't ever been down due to hacking.

5. IP ISSUES: On the subject of indemnification, Mr. Ballmer states that "it is rare for open source software to provide customers with any indemnification at all". If he were to check the slides he himself used at the Massachusetts Software Council address he gave on September 1, 2004, he would see a slide where both Microsoft and Novell are "checked" as offering indemnification, Novell referring to our Linux offering. Granted that same slide showed a "no check" for Novell regarding patents. Since that time Novell has made public its stance of using its patents to protect its open source offerings. See here.

6. SAVINGS FROM UNIX MIGRATIONS: On the subject of costs savings and UNIX migrations, Mr. Ballmer claims customers will save significantly by switching to Windows. But many of the savings realized by customers moving off UNIX will be on hardware costs as they move to x86 systems. UNIX skills and administration knowledge are more transferable to Linux than Windows. It would be unlikely that the resultant Windows environment would be less costly than an equivalent Linux one. The Yankee study quoted earlier by Mr. Ballmer states "Linux shows measurably improved TCO compared with UNIX and Windows in small firms, in organizations with customized vertical applications and in "greenfield" networking situations where there is no existing software infrastructure." In talking about Unix migrations, Mr. Ballmer highlights a survey purporting gains in performance by moving to Windows and suggests that Windows outperforms Linux in UNIX migration scenarios. We provide the following independent analysis of the performance capabilities of Windows 2003 vs. SUSE LINUX Enterprise Server 9: Flexbeta posted a Microsoft Windows 2003 vs. Novell SUSE Linux Enterprise 9 Comparison, dated 23 October 2004. Flexbeta states "that Novell's SLES9 is a very worthy contender to Microsoft's Win2k3 Server in a Windows environment. Not only does SLES9 perform better on the same hardware, but it costs less – possibly more than 1/10th the cost of a Microsoft solution." Flexbeta also states "Novell's SLES9 pretty much more than doubles the performance of Microsoft's Windows 2003 Server on the exact same hardware in both categories. This is very, very impressive, and shows the strengths of both Samba and the Linux kernel, as well as the attention to detail Novell/SUSE employees had when implementing the default settings. With this hardware Windows 2003 Server seems to max out on performance at approximately 30 Clients with a throughput of about 135Mbps, where SLES seems to max out on performance at approximately 60 Clients with a throughput of about 255Mbps. The response time is also about twice as fast on SLES9 than on Win2k3 on the same hardware. So, in theory, you can handle twice as many clients on the same hardware using SLES9 compared to using Windows 2003 Server." CONCLUSION
In his closing remarks Mr. Ballmer states that "it's pretty clear that the facts show that Windows provides a lower total cost of ownership than Linux; the number of security vulnerabilities is lower on Windows, and Windows responsiveness on security is better than Linux; and Microsoft provides uncapped IP indemnification of their products, while no such comprehensive offering is available for Linux or open source." The facts do not show this at all; read the complete reports on Microsoft's site, not just Microsoft's chosen sound bites. Given the increased adoption rates of Linux by customers, many of them also appear to disagree with Mr. Ballmer's negative assessment of Linux. So do the large number of ISVs who have already or are planning to port to Linux. Microsoft's most recent 10k presents another, perhaps more realistic, assessment of the prospect for Linux and Open Source software: We believe that Microsoft’s share of server units grew modestly in fiscal 2004, while Linux distributions rose slightly faster on an absolute basis. The increase in Linux distributions reflects some significant public announcements of support and adoption of open source software in both the server and desktop markets in the last year. To the extent open source software products gain increasing market acceptance, sales of our products may decline, which could result in a reduction in our revenue and operating margins.