Yesterday I noticed that Principled Technologies had released updated reports that examined the overall responsiveness of Windows Vista SP1 to Windows XP SP 2 for common business and home tasks. Given that I've examined the effect of SP1 on Vista in great detail, I feel that I should comment on the findings.
Principled Technologies drew two sets of conclusions from the tests. Here are the conclusions when comparing business scenarios:
- Overall, Windows Vista SP1 and Windows XP performed comparably on most test operations. Performance differences between the two operating systems were typically less than a half second. Significant differences of over a second occurred on only nine of 128 measures; Windows Vista SP1 led on eight of those.
- Windows Vista SP1 was noticeably more responsive after rebooting than Windows XP on several common business operations.
- Overall, Windows Vista SP1 was more responsive than Windows Vista on most comparisons. Performance differences between the two operating systems were typically less than a quarter second.
Here are the conclusions when comparing home scenarios:
- Overall, Windows Vista SP1 and Windows XP performed comparably on most test operations. Differences were typically less than a half second.
- Windows Vista SP1 was noticeably more responsive after rebooting than Windows XP on several common home operations.
- Overall, Windows Vista SP1 and Windows Vista performed comparably on most test operations, with differences typically less than a half second.
These test results are remarkably similar to those I came up with. For example, when I tested the responsiveness of Office 2007 using OfficeBench I concluded that there's really little difference between Vista RTM and Vista SP1. When Principled Technologies are talking about differences of halves and quarters of seconds, that's just another way of saying that there's really little difference.
Another point worth noting from the reports is that when Vista RTM and Vista SP1 are compared, there are hardly any real improvements to be seen. Given that SP1 really does speed up operations such as file copy, these don't really have much of an effect on real world usage. Again, this aligns with what I've seen during testing. Sure, if you look for the changes, they are there, but in normal use you don't see the effect that much at all.
And finally, something that bothers me. Both reports contain an appendix which covers an HP a1320y desktop system. According to the reports the results from this system were omitted because it "produced unacceptably high variability in its test results." However, a quick run through the scores would seem to indicate that XP beat Vista RTM and SP1 on many of the tests that were carried out. I'm far from satisfied with the reason for the exclusion and have approached Principled Technologies for a response.
Hey, I'm glad that it's not just my benchmarks that generate more questions than answers!
Bottom line of the reports: Nothing to see here, move along.