Vista SP1 vs. XP SP2 - Part Deux

Following the benchmarks I carried out last week, I decided that the PC Doc HQ lab rats needed to pull a few all-nighters and carry out some more benchmarking tests on Vista SP1 and XP SP2.  Is it possible to determine conclusively which is the faster, more responsive OS?

One of the main criticisms of my initial benchmark tests was that I overlooked the fact that, under the hood, Windows Vista SP1 works differently to XP SP2 in the file copy department.  The difference is down to caching and how Vista carried out uncached I/O while XP caches the files.  On top of that is the further complication that the file transfer progress bars are coded to work differently.  Vista's file copy dialog box goes away when the cache is committed, while under XP the copy dialog goes away while the committal is still pending.  In other words XP is coded to appear fast.  Mark Russinovich has the details:

Perhaps the biggest drawback of the algorithm, and the one that has caused many Vista users to complain, is that for copies involving a large group of files between 256KB and tens of MB in size, the perceived performance of the copy can be significantly worse than on Windows XP. That’s because the previous algorithm’s use of cached file I/O lets Explorer finish writing destination files to memory and dismiss the copy dialog long before the Cache Manager’s write-behind thread has actually committed the data to disk; with Vista’s non-cached implementation, Explorer is forced to wait for each write operation to complete before issuing more, and ultimately for all copied data to be on disk before indicating a copy’s completion. In Vista, Explorer also waits 12 seconds before making an estimate of the copy’s duration and the estimation algorithm is sensitive to fluctuations in the copy speed, both of which exacerbate user frustration with slower copies.

OK, so if this is the case XP has tricked us into believing that file copy is fast by sacrificing reliability for perceived performance.  In that case let's put copy speed on one side for a while and try to look at this problem from a different angle.  Let's consider responsiveness during large file transfers.  OK then, let's benchmark the system while it's performing file copy operations.

Next -->

The Test

This time we're going to combine synthetic benchmarks with real-world operations.  To put that in simpler terms, we're going to carry out a background file copy of multiple files (2,000 files in 40 folders, 3.8GB) while running a PassMark PerformanceTest 6.1 benchmark.

Note:  Note that this system is not representative of the kind of PCs around when XP SP2 was released.  Also, this system predates the release of Vista by a few months.

The test system is the same system detailed here.

Note:  With PassMark ratings, the higher the number, the better the score.

First, a baseline was established by benching the systems under no load.  We use our usual number crunching method of making four separate benchmark runs, remembering to reboot, defrag, and force the processing of idle tasks between each run.  We then examined the four results, discarded the lowest score and averaged the remaining results.

Note:  The only change that was made to Windows Vista SP1 for this test was to disable Windows Defender.  Testing indicates that the effect that Windows Defender has on file copy in Vista SP1 is negligible, but to satisfy the critics I chose to disable it for this test.

Using this method we ended up with the following PassMark ratings for the systems.

Average PassMark ratings:

• XP SP2: 509.0
• Vista SP1: 469.5

   

XP SP2 results on the left, Vista SP1 on the right.

Based on these results from the synthetic benchmark, XP SP2 gets a PassMark rating that's 8.4% greater than Vista SP1. 

Next -->

The Results

With a baseline established we carried out another set of benchmark runs while the file copy operation was running.  This is a pretty aggressive test of responsiveness but since sluggishness would result in a drop in the final PassMark rating (and the greater the sluggishness, the greater the drop in responsiveness), we decided to give this a try.

Eight runs later, the data was collected. 

Here are the PassMark ratings averages for both systems under copy load.

Average PassMark ratings, both systems under copy load:

• XP SP2: 490.1
• Vista SP1: 384.4

Based on these results, under copy load XP SP2 achieves a PassMark rating that's fully 27.5% better than Vista SP1. 

However, using the baseline that we gathered earlier we can see the effect that the file copy load has on the benchmark rating for OS: 

• Under load, XP SP2 achieves a PassMark PerformanceTest rating that's 3.7% less than the OS under no load.
• Under load, Vista SP1 achieves a PassMark PerformanceTest rating that's 18.1% less than the OS under no load.

However, oddly enough, Vista SP1 felt more responsive to user inputs such as opening applications and saving files while the tasks were being performed (we tried this out on separate runs). Problem is that it's darn hard to measure this end responsiveness without relying more on synthetic benchmarks.

However, let's see what we can do to clear things up ...

Next -->

More Tests, More Results

Looking at the overall results it was clear that what was pulling the scores down the most of both operating systems was the disk activity.  This gave us an idea.  We carried out another set of PassMark PerformanceTest runs, but this time rather than do a full run of tests this time the tests exclude all of the hard disk tests (therefore eliminating much of the effect of the copy operation on the benchmark). 

First, we established a baseline rating under no copy load with the disk test excluded.

Average PassMark rating, no copy load with all disk tests excluded:

• XP SP2: 450.5
• Vista SP1: 403.3

Based on these results, XP SP2 achieves a PassMark rating that's 11.7% better than Vista SP1. 

We then placed the system under the same file copy loads and ran the tests again.  Number crunching these results in the usual way gives us the following results.

Average PassMark rating, under copy load with all disk tests excluded:

• XP SP2: 389.3
• Vista SP1: 369.0

Number crunching these results, the lead that XP SP2 has over Vista SP1 has fallen to 5.5%. 

Rather than compare the two operating systems directly, let's compare these under load benchmark results to the no copy load with the disk test excluded.

The change from the baseline score when excluding PassMark disk test scores is as follows: 

• Under load, XP SP2 achieves a PassMark PerformanceTest rating that's 13.7% less than the OS under no load.
• Under load, Vista SP1 achieves a PassMark PerformanceTest rating that's 8.5% less than the OS under no load.

For those wanting to take a more detailed look at these test results, I've uploaded detailed outputs.

• XP SP2 - No load/No disk tests
• XP SP2 - Load/No disk tests
• Vista SP1 - No Load/No disk tests
• Vista SP1 - Load/No disk tests

Next -->

Conclusions

This is far more complicated than I'd hope that it would turn out to be, however the results are interesting.  Let me summarize the results here:

• Tested using PassMark PerformanceTest 6.1, XP SP2 consistently achieves a higher rating than Vista SP1.
• Under file copy load, XP SP2 consistently achieves a higher rating than Vista SP1.
• When running a partial PassMark PerformanceTest run (a run where all disk tests are excluded), XP SP2 again achieves a higher rating than Vista SP1 when under copy load and under no copy load.
• However, if you look at the effect that file copy has on a partial PassMark PerformanceTest run, we see that the file copy operation on Vista SP1 has less of a detrimental effect on the overall rating than under XP SP2 system.

So, what this long-winded series of tests shows is that heavy file copy operations has less of an effect on the overall responsiveness when running Vista SP1 than when running XP SP2 (on the test system, all things being equal). 

This benchmark, along with the one I posted last week, go to show how unsatisfying it can be to benchmark one OS against another.  Even when you're dealing with one system there are a huge number of factors to contend with.

Later in the week I hope to have a set of results that are far more conclusive and convincing - I'll be testing each operating system and seeing which can deliver the best frame rates in some of my favorite games.

Stay tuned!

Thoughts?

Be the first to read new posts - subscribe to the Hardware 2.0 RSS feed.

<< Home >>