Vista SP1 vs. XP SP2 - Benchmarked

Vista SP1 vs. XP SP2 - Benchmarked

Summary: Over the past few days I posted two sets of benchmarks comparing Windows Vista RTM with Vista SP1. These posts generated a lot of feedback, and from reading this feedback it's clear that what many people are really interested in is not the performance differences between Vista RTM and Vista SP1, but between Vista SP1 and XP SP2. How does Windows Vista SP1 compare to Windows XP SP2? Read on ...

SHARE:

Latest: Vista SP1 vs. XP SP2: Part Deux 

[UPDATE 02/18/08: Some people are concerned that in order to get an accurate measurement of boot time the system needs to be booted many times.  Let me assure you that the systems here under testing were rebooted more times than most normal PCs are over the course of several months.

Over the past few days I posted two sets of benchmarks comparing Windows Vista RTM with Vista SP1 (first post here, second here).  These posts generated a lot of feedback, and from reading this feedback it's clear that what many people are really interested in is not the performance differences between Vista RTM and Vista SP1, but between Vista SP1 and XP SP2.

Vista SP1 vs. XP SP2 - BenchmarkedA few days ago I posted in reply to several TalkBack comments that I wouldn't carry out these tests until XP SP3 is released.  There didn't seem any point.  That didn't satisfy the crowds who wanted to see Vista SP1 and XP SP2 go head to head.  So, to cut a long story short, the pestering continued and I eventually gave in.  So what follows are the fruits of nearly two whole days of work at the PC Doc HQ (the test was hampered by the death of a motherboard, something which rendered hours of work obsolete).

How does Windows Vista SP1 compare to Windows XP SP2?  Read on ...

The Test Rig -->

The Test Rig

The test ring that we'll be looking at here is one of the systems that I benched for the Vista RTM/SP1 tests.  It's my Pentium 950D system and is representative of the most "middle of the road" systems that I have.  Too slow a system and the results could be inconclusive, too powerful and that might skew the tests.

Here's the spec:

  • Pentium 950D processor (3.4GHz)
  • 4GB of RAM (being 32-bit, the OS recognizes a shade over 3GB)
  • 2 x 250GB Maxtor hard drive

What I did was simple.  I took out the boot drive and found two identical 250GB drives.  I fitted one, installed Vista Ultimate 32-bit, set up the system and got it up and running.  Then I took out that drive and replaced it with the second drive, installed Windows XP SP2 onto this one and got this OS up and running.  Both systems are fresh and have all the latest drivers installed.

Note:  We abandoned the idea of using dual-boot early on because of the effect this could have on disk performance.

No optimizations were carried out other than defragging the drive and making sure that all background tasks had finished.

The Tests -->

The Tests

The tests are the same tests I've carried out before, consisting of moving, compressing and extracting a set of files.  I had two sets of test files:

  • A single 664MB file
  • A folder containing 500 files in 10 folders, total file size 995MB

The tests that were carried out using both sets of files were:

  • Drive to drive copy
  • Add to compressed folder (using Windows built-in compression)
  • Extract files from compressed folder (using Windows built-in compression)
  • Push files to a network share over a 1Gb Ethernet connection (destination system running Vista SP1)
  • Pull files over a network share over a 1Gb Ethernet connection (source system running Vista SP1)
  • Push files to an external USB hard drive
  • Pull files from an external USB hard drive

I also measured the boot up time (I took this as the time from the boot loader to a usable desktop).

Each test was run four times, the poorest result was discarded and the average of the remaining three taken.

During the testing close attention was given to Microsoft's SP1 testing guidelines which were supplied to me prior to getting my hands on the SP1 code.

The Results - Tabulated -->

The Results - Tabulated

OK, with all that out of the way, here's the bit that you've been waiting for - the results.

The Results - Graphs -->

The Results - Graphs

For those of you who prefer to take in information visually, here's the data displayed in a series of graphs.

Conclusions and Caveats -->

Conclusions and Caveats

Note:  I didn't benchmark against XP SP3 because the EULA prevents the disclosure of benchmark results.  Also, since it's a beta, things can change between now and the final release.

So, onto conclusions.  Looking at the data there's only one conclusion that can be drawn - Windows XP SP2 is faster than Windows Vista SP1.  End of story.  Out of the fifteen tests carried out, XP SP2 beat Vista SP1 in eleven, Vista SP1 beat XP SP2 in two of the tests, and two of the tests resulted in a draw.

The best result for Vista SP1 was in the single file drive-to-drive copy, while the best result for XP SP2 was extracting multiple files from a compressed folder.  Given these results and taking into account the improvments that SP1 bought to Vista, if I was to go back and compare XP SP2 with Vista RTM, XP would have hammered Vista even harder.

Reminder:  These results apply to a single system and no optimizations were carried out to any of the systems.  Your mileage can, and probably will, vary.

I'll carry out some gaming benchmarks over the next few days.  Watch this space!

Thoughts?  Comments?

<< Home >>

Topics: Windows, Microsoft, Operating Systems, Software

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

296 comments
Log in or register to join the discussion
  • I/O reliability is more important than speed

    I/O reliability is more important than speed.
    Vista is more reliable than XP
    qmlscycrajg
    • Give me some proof.

      I have NO problems with XP I/O. I don't know about Vista but it can't get any better than my XP I/O.
      bjbrock
    • I don't believe this

      Since I find XP 99% reliable I don't believe Vista would be any better. I'll upgrade to Vista only with a gun to my head.
      tbcass
      • And for the comsumer, 99% might be ...

        ... good enough but in the corporate world, 99.99% might be more appropriate. I/O benchmarks only paint part of the pisture.
        M Wagner
    • Vista isn't more reliable at my house.

      My mew Dell continually stumbled with Vista reporting incompatibility, especially after coming out of sleep stage. The fact that Vista just ran without complaint and will probably run later isn't much comfort. I first replaced the hardware and then went back to XP. I plan to use XP to get the work done and Vista to get ready for when the other shoe drops.
      marabrook@...
      • Vista SP1 is supposed to address ...

        ... problems wiht Vista SLEEP mode but I never found that feature very reliable in XP either.
        M Wagner
    • i just dont have problems with xp pro, whats your deal

      i disagree.

      i just installed visual studio express 08 (runs great), play my games etc. and this box is 4 years old.

      if i have ever crashed ... i have found out it was because of bad software i installed or malware...

      its so rare (any problem)... that i would never switch if i didnt have to.

      Vista is just another certification. networking is networking, software is just software, and hardware is just hardware too. everythings just faster or slower and processes + or - bits more or less. capiche?
      pcguy777
    • How is Vista more reliable in IO?

      I've been using XP on several machines for several years and have never had an IO failure that was not caused by a specific hardware problem. I don't see how Vista could improve on this record.
      w_c_mead
    • False assertion, reliability; XP still wins.

      XP fiel storage is very reliable; the key is not about XP versus Vista, but about NTFS that provides most of the reliability, as well as the hardware quality.
      There's a point where XP still wins in front of Vista in terms of reliability: the stability of its implementation of the drivers model that have been experimented since very long now, and was tested before in Windows 2000. The filesystem driver and the hardware and BIOS drivers for Vista are completely new, because it implements a new security model, that is still largely unstable (the UAC) and with the GAC control now trying to control more tightly the hardware, and caused most drivers to be rewritten. Also the threading and memory model is completely changed, cuasing lots of troubles for criticla system drivers.

      I've seen Vista crashing regularly since the begining, and files getting corrupted, somtheing that never occurs now with XP, even when it crashes: you can still boot an an emergency CDROM to repair things and have your data safe. On the opposite, Vista is always rewriting many files and spreading them everywhere on disk, at a dramatic speed, and this makes data recovery very difficult to do safely after a crash.

      Notably, I've seen frequent needs to perform a full CHKDSK at boot time in Vista, because the USN journal of the boot NTFS partition was inconsistent; i've never seen any inconsistency in the USN journal of the boot HTFS partition in XP.

      Also Vista can crash at any time in my absence, simply when performing its own background management and "cleanup" tasks.

      I've not been able to run Vista without seeing it rebooting every week. XP can run for months without rebooting, except when installing some security updates, if automatic installation is checked. Personnally I prefer being informed before automatic installaion starts, so that I can see what problems may be detected and need some attention.

      Vista has too many down times compared to XP, and this adds to the unreliability of the system as a whole.

      But the point is that I/O reliability is efectively important than pure speed, but Vista still does this wrong compared to XP and Windows 2000 that was and is still rock solid.

      Let's separate the problem about security holes and needed patches for running in security when connecting the system to the net or to a LAN with Internet-connected hosts: Windows XP & 2000 still win, because most existing security tools and investigation tools still do not work in Vista dur to the new security model that just makes things more complex to use or create more holes if it is disabled, when 2000/XP security management was well understood since long and more in line with Unix security management with simpler concepts that are easier to debug after any kind of problem!

      Vista promisses to solve lots of problems but really does not help anyone trying to fix the situation. Even the System restoration is not stable, compared to XP/Windows 2000, and the fact that Vista takes sonsiderable time to boot (even if it present a logon screen faster, it does not mean that booting is finished, and that the desktop will come faster.

      Finally one thing to recommand on Vista: disable the desktop gadgets: they are considerably extending the time to boot, and take up considerable resources at boot time. There's an alternative for them if you like desktop gadgets: use Google desktop gadgets that are MUCH faster and take much less resources.
      PhilippeV
    • so reliable, it burned my hard drive twice

      After installing Vista my laptop's HD failed within 2 days (still under warranty) and the replacement failed within 3 months after that. Everybody I know who is using Vista has had trouble with their HD.
      jdubray
    • There is nothing in Vista that says it is more reliable in IO

      One, any significant improvement would likely ocur only in a change of protocol, which would mean both sides need to be capable of handling the protocol change. TCP/IP is still TCP/IP. Microsoft isn't a network company, and they have a long history of not making good choices when ever it came to netowrking. Remember, Bill Gates said he saw no commercial future in the internet, Windows did NOT support IP, MS created NetBeui, and developed WINS. Networking is something Microsoft has always done very badly.
      royalef
    • crap

      crap
      bonga1td@...
    • why

      why do people judge a 1year old sp1 to a 8 year old sp2? give vista a chance 98me judging
      thomaswort@...
  • Did you quit timing...

    when windows said it was done or when the hard drive actually quit?
    bjbrock
    • When Windows was done ...

      ... hard to measure anything else.
      Adrian Kingsley-Hughes
      • I wonder how long...

        each of the OS's are set to before they are forced to commit a dirty cache.
        bjbrock
        • Mark Russinovich talked about this; XPs dialog closes long before committal

          Vista's copy dialog goes away when the cache is committed, whereas when XP's copy dialog goes away the committal is still pending. So it's really a difference in perception.

          Mark Russinovich mentioned this in a blog posting (http://blogs.technet.com/markrussinovich/archive/2008/02/04/2826167.aspx):

          "Perhaps the biggest drawback of the algorithm, and the one that has caused many Vista users to complain, is that for copies involving a large group of files between 256KB and tens of MB in size, the [b]perceived[/b] performance of the copy can be significantly worse than on Windows XP. That?s because the previous algorithm?s use of cached file I/O lets Explorer finish writing destination files to memory and [b]dismiss the copy dialog long before[/b] the Cache Manager?s write-behind thread has actually committed the data to disk; with Vista?s non-cached implementation, Explorer is forced to wait for each write operation to complete before issuing more, and ultimately for all copied data to be on disk before indicating a copy?s completion."
          (emphasis mine)
          PB_z
          • Define "long before" though ...

            We thought about this and did some tests in a virtual machine. Did large copies and as soon as the OS said it was done copying files, did a forced power down of the VM. We expected data loss but in fact didn't see any.
            Adrian Kingsley-Hughes
          • about Russinovich

            I'm still intrigued about the Russinovich comment. There is a setting in Vista: device manager > (choose an individual drive) > properties > policies > enable advanced performance. Choosing to enable advanced performance comes with a warning about possible data loss should the system lose power and is therefore recommended only for systems on battery backup.
            Have you benched Vista with this enabled? Just curious. Is this a setting which turns off the feature Mark is talking about?

            Just wondering if maybe the slowness of Vista is due to reliability improvements after all. I think some serious effort should be expended to get to the bottom of this before we close the case.

            I am also wondering why these file copying times are being used to determine overall 'OS speed'. Should not other measures of speed be included? Not arguing - there may be a good reason why you are focusing on this. Just asking.

            How about games? People keep saying XP is so much faster in frame rates but this Firing Squad study seems to suggest otherwise. http://www.firingsquad.com/hardware/amd_nvidia_windows_vista_driver_performance_update/
            eggmanbubbagee@...
          • From Russinovich excerpt...

            it is clear that this is not [i]perception[/i] but a lousy implementation of the copy algorithm. :-)

            I wonder what was the reason of not using write-back cache?
            Solid Water