Microsoft Office versus Open Office shootout

[Updated 9/15/2005 12:00PM: Added Open Office Beta2 results] When I started this blog questioning the wisdom of XML, it started a storm of comments and was followed up by fellow bloggers John Carroll and Joe McKendrick.  Since my response "Proof that XML is extremely bloated" appeared last week, I got a lot of talkback complaining that I was only testing Microsoft Excel and not Open Office Calc.  ZDNet reader Yagotta B. Kidding took it upon himself to do some testing with Open Office but he stopped testing when he reached a file half the size of my 16-sheet XML file, yet his results were even slower than my Excel results.  I requested that Kidding provide the results for the identical 16-sheet file, but he declined and wouldn't provide a specific reason.  This, of course, sparked my curiosity, so I did some additional research.

We even had some conspiracy theorists like figgle that went as far as saying "If I was a bit more paranoid I might think that Microsoft deliberately de-optimized their 'open format' file loaders."  The only problem with that theory is that Microsoft doesn't even support Open Office file formats, and there would be no reason to deliberately reduce the speed of its own XML document because Microsoft is committing to XML in the next release of Microsoft Office.  I guess figgle has never tried doing the same thing with Open Office Calc because he would really start breaking out the conspiracy theories.

Others like George Jay resorted to the knee jerk reaction that "Perhaps we should avoid using Windows because it too is bloated".  Not only is it not true, my Windows XP machine boots up in 30 seconds from a cold start to post login, and Microsoft Excel loads in under 2 seconds the first time without the office pre-loader cheat installed.  I can even run Windows XP and Office 2003 on a PC with 64-MBs of RAM.  This is less than some individual Java applications require and any modern Linux distribution takes nearly a full minute to boot on identical hardware using the default installation.  Jay responded by saying that bloat isn't defined by load times and insinuated that Microsoft only loads fast.  The problem with Jay's theory is that Microsoft Office loads fast, reads and writes data fast, and keeps a smaller memory foot print from start to finish.

I like free software as much as the next guy and I had never intended to do a Microsoft Office versus Open Office shootout because it really isn't a fair comparison when development budgets are taken in to account.  Open Office development is supported by donations while Microsoft Office is supported by a massive commercial software development budget.  Open Office is the cream of the crop as far as free software goes, but I couldn't stand idly by while so many people make patently false statements about Microsoft Windows and Office bloat.  I decided to do this shootout of Microsoft Excel versus Open Office Calc on my own desktop PC so that the hardware parameters are identical.  As with any good scientific tests, the methodology must be fully disclosed so that it is repeatable.  You can download this compressed 16-sheet MS-XML file and run the tests on your own.  For Open Office users, download this 16-sheet SXC compressed XML file.

Methodology:

• Test spreadsheet performance of Microsoft Excel and Open Office Calc on identical hardware with identical 16-sheet sample.
• Test various spreadsheet data file formats like Microsoft XLS, Microsoft XML, Microsoft Zipped XML, Open Office SXC compressed XML.
• Record memory utilization of Excel and Calc.
• Record file creation and file load times.
• Record file sizes.

Hardware test bed:

• Intel Pentium 4 3.4 GHz with Hyperthreading enabled
• FSB 800 with 1 GB DDR 400 MHz RAM
• NVIDIA Geforce 6800 OC
• Two SATA Seagate 160 GB 7200 RPM Hard Drive (test files non-OS drive)

Software test bed:

• Windows XP Professional with SP2 plus all critical updates
• Microsoft Office 2003 Professional full installation
• Open Office 1.1.4 Win32 stable custom default installation (Java disabled)
• Open Office 1.9.125 Win32 full installation

Conclusion:
XML documents are not only slow with Microsoft Excel, but even slower with Open Office compressed XML documents.  Even combining Excel XML creation and ZIP compression times, Excel was substantially faster than Open Office Calc.  Here is a break down of the results:

As nice as Open Office Calc is for a free application, it doesn't even come close to the performance and usability of Microsoft Office Excel.  There is no shame in losing to a commercially funded application, but no one should make false statements that Microsoft Office is bloated and inferior.  The only remaining question for businesses and homes is whether Excel is worth the money and most have apparently decided yes.

It is well known even by XML proponents that processing XML files are slow.  When a certain technology scales poorly and runs slow on the desktop, it also scales poorly and runs slow on a server.  When software is slowed down by more than an order of magnitude, it is clear that XML will always knock application performance back 10 years no matter how fast our hardware gets.  Most computer users would pay a huge premium for hardware that is 2 times faster, so why would they want software that was up to 98 times slower?  The next time your computer is crawling along, just blame the XML bloat monster.