Is Vista's Windows Experience Index fatally flawed?

Is Vista's Windows Experience Index fatally flawed?

Summary: Is Vista's Windows Experience Index system rating tool fatally flawed?

SHARE:
TOPICS: Windows
12

Windows Vista has some complicated system requirements, especially when it comes to memory and graphics card performance.  In an attempt to try to simplify things for user and buyers, Microsoft has bundled a utility into Vista called Windows Experience Index.  The idea is that the Windows Experience Index examines key areas of the PC and assigns them a score between 1 and 5.9 and then uses this as a basis for a system rating (in fact, currently the lowest score is rounded down and used as the index).  But is Vista's Windows Experience Index fatally flawed?

The Windows Experience Index examines five key areas of the system:

  • Processor
  • Memory (RAM)
  • Primary hard disk
  • Graphics
  • Gaming graphics (memory)

Here is the score that my Vista system got when I initially installed Vista Beta 2, Build 5456:

Vista's Windows Experience Index

But here's the score it got today:

Vista's Windows Experience Index

In that time, nothing has changed on the system yet the score has dropped from a healthy 4 to an average 3.  Now I don't really care about this score, but if it's supposed to give me a base score for the system, surely the number should be static.  I can't find any reason why the score dropped, apart from the fact that the tool has changed the score that it gives to the system RAM.

OK, but that's not the only beef that I have with output generated by the Windows Experience Index. 

  • 32-bit Windows can only access 4GB of memory (in fact, 4GB is the total address space accessible under a 32-bit OS, so the amount of RAM that the system sees will be less than that), why is the score that my system, which has 4GB of RAM installed, not higher?  This seems to indicate that the scoring scale for 32-bit and 64-bit versions of Vista will be based on the same scale, which doesn't make sense given that both the RAM requirements and maximum system RAM for 32-bit Vista is different to 64-bit Vista.
  • Why does the Windows Experience Index only look at the capacity of the primary hard drive?  Surely the total hard drive capacity of the entire system should be considered.
  • Why are there two scores for graphics considered?  Why not consider the graphics card and memory as one unit since there isn't all that variation in graphics RAM?
  • No consideration is given to optical drives installed.
  • No consideration given to networking capabilities.

While I think that Vista's Windows Experience Index is a step in the right direction, I really can't see the number outputted being of much use to anyone.  Why?  Because it's too simplistic.  A single factor can skew the scoring such that the score in meaningless.  You could have a PC with a powerful CPU, bags of RAM and a monster graphics card, and put those in a system that has 250GB hard drive (not a particularly shabby amount of space) and get a poor index rating as a result. 

To be honest, I can't see the score that the Windows Experience Index generates being good for anything - it's certainly no good as an indication of what kind of hardware or software a system can run, and it's no good an as indicator as to whether a system can run Vista with the Aero Glass interface.  It's also so vague that it's useless as a buying guide.  Microsoft might release a tool at some point that allows users to enter different system configurations and get the Windows Experience Index score, but that information is no more useful.

But what about the sub-rating scores?  Surely they're useful?  Not really, unless hardware and software manufacturers actually start to use these numbers, which I don't see happening any time soon because it's easier and more accurate to use plain system requirements.  But what about games?  Will they adopt the scheme.  No way!  Games have some of the most complicated system requirements of all software sold, and they routinely list supported CPUs and graphics chipsets.  There's no way that they are going to start simplifying these requirement listings because it's the equivalent of shooting in the dark.

So what use is the Windows Experience Index?  Short of providing a tidbit of trivia about the system, it's meaningless.

Topic: Windows

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

12 comments
Log in or register to join the discussion
  • The WHY is somewhat logical..

    [b]Why does the Windows Experience Index only look at the capacity of the primary hard drive? Surely the total hard drive capacity of the entire system should be considered. [/b]

    I tend to agree, but if you think about it a bit, it makes sense from the OS's perspective.

    Where does Windows default as the location for ALL file saves? C:\Users\UserID\Documents or a subfolder under that.

    Windows (and programmers who write Windows apps) assume the lowest common denominator - the typical sub-average, technically challenged, slightly technophobic user who can barely run Word and type up a document. Where's he going to save it? The Default Location.

    A power user will know to change the document's location - and how to do it. But that's not your garden variety user.

    Case in point. Several months ago, a client of mine called me and asked me for help - his hard drive was out of space (mostly due to his kid dumping his iTunes collection on the C: drive). The computer is a Sony VAIO laptop that was about 4 yrs old. It has a 20 GB hard drive in it. Sony, in their infinite lack of common sense partitioned the hard drive into two partitions. C: - the root drive - was partitioned into an 8 GB partition while the E: drive was partitioned into the balance of the disk's space - about 11 odd GB.

    Sony's reasoning is that when you connect your Sony camcorder and set it to download your videos to it, the software transfering the video would be configured automagically to dump it to the empty, big partiton. It would also not cause issues with the OS partition. Of course, they're ASSUMING (and we all know what that gets you - makes an A$$ out of U and ME) that you would have a Sony camcorder and that you would download videos to it.

    I was able to catch the fact there was a second partition immediately - just by looking at Windows Explorer. But they never did check into that. The E: partition was as empty as the day it was formatted. 11 odd GB was going to waste because it was something outside of their realm of experience.

    The point is - while you, and I and most power users KNOW those other drives exist, most who fall short of that level of expertise won't even bother to look until their C: partition runs out of space.
    Wolfie2K3
    • Yeah, I buy that .... but

      A 250GB drive isn't all that shabby a drive ... considering that Vista system requirements aren't at that level (yet!).

      The scores just feel arbitrary to me ...
      Adrian Kingsley-Hughes
      • Still..

        Remember what the index measures isn't the disk's capacity but rather its transfer speed. A 250 GB disk will still get a low score if it has a low transfer speed.
        VMLM3
  • One More Thing..

    [b]Why are there two scores for graphics considered? Why not consider the graphics card and memory as one unit since there isn't all that variation in graphics RAM? [/b]

    Because graphics memory CAN be changed. Motherboards with built in video can have the amount dedicated to video changed, usually in the BIOS. Some stand alone video cards (AGP or PCIe) will allow you to upgrade the RAM on board as well.
    Wolfie2K3
    • But again ...

      Microsoft are just changing one number (say 256MB) into another number (4,7). Microsoft seem to be taking one set of values (CPU, RAM, etc), transforming this data into a set of arbitrary (and meaningless) numbers, and then using that to derive another number.

      I just don't get it!
      Adrian Kingsley-Hughes
  • Some of the questions are not so good

    Why look at only the root capacity: Windows NT variants have typically not been good at distributing their system functions across different physical drives. You can make it happen, but (at least on the workstation versions), it's not easy. So clearly, capacity of the [i]system[/i] drive is much more important than total drive capacity.

    Video and main system RAM do different things, and the amount of each has a different impact on system performance. Therefore, they [i]absolutely[/i] [b]should[/b] be considered separately.

    Finally, the presence of optical drives and network access are both irrelevant to system performance. They might have relevance to system [i]usability[/i], so it's arguable that they're relevant to "experience" understood very broadly. But that relevance depends largely on what you're going to use the system for. I haven't used the optical drive on my main system for six months. It could not work at all, and not only would it not matter, I wouldn't know it. Similarly, if you don't use the network all the time -- e.g., if you're into standalone gaming, or you only have dialup (wake up, dude, that's still a LOT of people) -- then networking hardware is completely irrelevant to your "experience."
    escoles@...
  • windows exsperience index

    when figuring a score for the hard drive and the memory it isn't the amount that you have because my system with 1 gb memory and windows installed on a 36 gb hard drive scored a 4.5 , memory being the lowest sub score a 4.5. windows measures the speed of the hard drive and the speed of the memory. my 36 gb raptor hard drive scored a 5.9, and as far as memory its not how much you have but how fast your memory is
    jpenrow
  • Get informed, please.

    You should probably get yourself informed before posting wild speculations. Not the SIZE of your RAM is measured, it is the SPEED.
    Look at this blog by the Windows Vista Team: https://209.34.241.68/windowsvista/pages/458117.aspx
    They describe in detail what is being rated and how. It may just be that your RAM timing is different before, thus being slower now. That explains the worse rating...
    veichtlbauer
  • Hmmm

    For someone who has devoted over a decade to technology your understanding of it is somewhat vague at best. As others have said the amount of Ram or HD space isn't the important factor here its more so how quick they are, its pointless having loads of Ram or HD space if its too slow to be any use, Microsoft have simplified things to make it easy for beginers as well as more experienced people. For me it seems that you are going out of your way to pick fault where there is none, it gives the impression that your bitter about your poor index score which is pretty childish.
    Skullet
  • Educateing the masses

    In recent years Microsoft has taken great steps in its efforts to accommodate all its end user?s, with what seems to be it?s never ending commitment in encompassing not only the minority of high end user?s, who are able to understand the intricacies of how hardware and software determine the capability of your system to accomplish tasks, but also the lower end user who typically might have used a computer to distract them self?s with a quick game of solitaire due to lack of knowledge and understanding, in this respect I believe Microsoft have done extremely well, giving a much more stable environment in which the user can search files, documents, folders, pictures, programs and the internet, all the time educating themselves without fear of, deleting files or changing settings, also giving the end users more knowledge on what task?s can be performed on a system as well as allowing them the opportunity to accomplish many of these tasks without having to buy any extra, expensive hardware or software.
    It is this drive to educate its end user I believe that Microsoft has created the windows experience index, at the moment it is vary basic but does tell the end user a little about their system without over extending them, it gives the end users some fundamental knowledge to grow from, instead of ?throwing? information at them, which means a lot to very few, they will soon begin to piece together how a computer works, runs, and operates. Eventually due to this process users of all levels will see how different peripherals can change the usage of their computer. Knowing that Microsoft have done this, I believe, with only one end result in mind, educating there users, and thus with there users more educated and involved they will branch out into more demanding fields and want more products, of which they can purchase from Microsoft.
    On the hole I think the windows experience index is accurate enough for most users to use as a simple benchmark, and at least it is one most people can be expected to understand. When the knowledge base of the end user?s becomes greater so will the sophistication of the windows Experience index, as i believe is true for all Microsoft?s products and software, but while there is still such a huge gap between the levels of users you cannot expect Microsoft to only cater for high end users. If they did, they wouldn?t get that ?wow? factor they want from the masses now would they
    Hunterthepunter
  • RE: Is Vista's Windows Experience Index fatally flawed?

    This is a poorly written and misinformed article. The windows experience index is there as an easy measure of what your system can or can't run. If there is a bottleneck in your system and you're trying to run an app or game, it can only go as fast as the slowest component. I.e - if you had a system with a ?1000 cpu but only used integrated graphics, then you're score would be low. And if you tried to run a game like crysis on that system it wouldn't stand a chance, no matter how much you'd spent on your cpu.

    At the end of the day, a good system should be balanced. Microsoft recognise this with the index. By setting it to the lowest performing components value, it ensures no nasty surprises for the end user.

    The differences in your systems performance could be
    due to different drivers. I had this issue when updating my graphics card driver - i actually got a lower score after it updated!

    Theres also 2 numbers for the graphics as its measuring different things. One is 2d desktop performance and one is 3d performance.

    And as others have said, the memory and hard drive numbers are based on speed. A bit of research could have saved you writing this article!
    virtualmark
  • next time do some research

    the hard drive score isn't based on capacity its based on speed, the ram score is the same way. size doesn't matter only how fast it will go. so yes you can have 4TB of storage at 7,500 RPM and get a average score and 64GB of ram wont matter if its the slowest ram you could find. although in that case the shear number of ram inputs would probably make up for the speed.<br><br>GPU's don't necessarily need to ratings, until you consider there are two types of GPU's those for gaming and those for rendering.<br><br>also your numbers varied because the rating system was still in a "released beta" state being actively tweaked. also drivers are a likely factor in the score diference
    c1c2c3c4c