Watching Windows Vista decay

Watching Windows Vista decay

Summary: By definition, beta software is buggy. So, how buggy is Windows Vista? According to Vista's built-in Reliability Monitor, one computer in my office has decayed from a perfect 10 to a 1.70 in just four weeks. Is this beta really that bad?

SHARE:
TOPICS: Windows
68

By definition, beta software is buggy. So, how buggy is Windows Vista? Microsoft has helpfully included a new tool for measuring stability and reliability in Windows Vista. It's called the Reliability Monitor tool, and you can find it in the Performance Diagnostic Console. (To run this tool, you can go the long way: Start, Control Panel, System and Maintenance, Performance Rating and Tools, Advanced Tools, Open Windows Diagnostic Console. Or you can just click Start, type Perfmon in the Search box, and click the Perfmon shortcut when it appears.)

I included screenshots of the Performance Diagnostic Console and the Reliability Monitor in my earlier post Vista Beta 2, up close and personal (the specific images are here and here). To switch to the Reliability Monitor, click its link in the left column of the Performance Diagnostic Console.

On one computer in my office, the Reliability Monitor paints a thoroughly depressing picture. See for yourself:

eb_reliability_monitor.png 

That line chart at the top represents four weeks' worth of ever-increasing instability, according to Vista's own tools. On May 24, after a clean install, the Stability Index was at 10; today, four weeks later, it’s at 1.70. On another computer in my office, the same picture is emerging. After a clean install on May 31, the index has slipped to 3.16 today.

Is Windows Vista really decaying right before my eyes? The short answer is, no. This is a crude measurement, to be sure, and it's misleading as well. The problems I’ve been experiencing (and which are logged in detail in the Reliability Monitor) are pretty much the same bugs,  in Windows and in application software, occurring repeatedly, which is what you expect from a beta. So the inference that the system is somehow getting much less stable over time may not be accurate. In other words, my system stability was never a 10, and it’s certainly not a 1.70 now.

This is at least the second place within Windows Vista where some product designer decided that creating an arbitrary index number would be valuable. (The other is the much-criticized System Performance Rating, which looks at the details of your PC's parts and reduces the calculation to an overall rating on a 1-to-5 scale.) I'm not sure what the user is supposed to take away from this number, though. If my Stability Index slips below 5, is it time to do a complete reinstall? Is it really fair to conclude that my overall system stability dropped from a perfect 10 to 8.17 because Explorer crashed twice on May 25, or that it then slid all the way down to 5.77 the next day because OneNote 2007 Beta 1 stopped working twice (and hasn't failed since)? In fact, a quick scan of the details shows that both systems are fairly reliable overall; it's that Office 2007 Beta that is really the unstable actor.

If I had left my computer running and had gone on vacation for two weeks, the Stability Index would have probably risen. As near as I can tell, the algorithm gives a tiny boost to the Index number for any day where there are no crashes or application failures.

Don't get me wrong - I think the Reliability Monitor is a good idea and a potentially useful tool. Its real value is its ability to track and organize system events, such as software and driver installations, and display them in a format that allows you to identify correlations between failures and other system configuration events. The day-by-day display at the bottom makes it easy to see when a series of unpleasant events started, and you don't have to be a master troubleshooter to look at the list of Software (Un)Installs to zero in on a potential cause.

But that silly line chart at the top is just plain misleading. Microsoft should ditch it and the accompanying index number.

Topic: Windows

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

68 comments
Log in or register to join the discussion
  • Grain of salt

    I don't doubt Vista is buggy, but how [i]reliable[/i] do you think the Reliability Monitor is?
    Taz_z
    • Just as buggy

      as Vista itself .
      Intellihence
  • I have to say one thing here.....

    I have been using Redhat Linux since 6.0 and have my RHCT in RHEL3 (paid for by work) and Fedora Core 5 has bugs in it, but it is the test bed for the next Redhat RHEL release.

    Get over it, ALL software has bugs, some never get fixed completely just patched over and then something else breaks.

    If all software was foolproof, then no one would have a job.
    Southern.Pride
    • Agreed, but...

      Fedora is a free download and is delibrately bleeding edge, so my expectations are less (I have it running on the family PC nonetheless). If I'm paying for an OS, however, my expectations are higher; doesn't need to be perfect, but there does need to be some assurance that things work substantially as advertised (if proprietory developers don't so some professional pride, what good are they?).
      John L. Ries
      • That is tough there

        for someone to think of all the different combinations of hardware and all to configure.
        This is where the vendor needs to work with the software (propietory) to get the stuff right the first time.

        Something tells me that it is a full circle of pass the buck just like at all workplaces, just enough to get it done.
        Southern.Pride
        • You make a great argument....

          ---That is tough there
          for someone to think of all the different combinations of hardware and all to configure---

          Which is a great argument for buying a Mac.
          tic swayback
          • Confusing lack-of-choice with safety

            [i]Which is a great argument for buying a Mac.[/i]

            Simply because company 'A' takes away all of your choices does not mean that company 'B', by giving you ultimate freedom, is offering something that is guarenteed to fail. While there is only one combination of hardware that works when you go Apple (the combination that Apple graces you with), there are hundreds of combinations that work just fine on the PC side. Granted, there are thousands of combinations that don't work fine so you certainly have to be more careful but there are enough Dells and HPs for those who don't know which combinations work well and which ones don't.

            To bring in the "tyred" car analogy (har har), a company could say:
            [i]We refuse to sell any car with more than 100hp because we don't want our customers hurting themselves.[/i]
            However, that in no way means that a 500hp car cannot be driven safely. The lack of choice is a way of [b]forcing[/b] safety onto customers but it is not the only way (or even the best way) for a customer to be safe.
            NonZealot
          • Interesting bit of history....

            Back in WWII the Panzer's and Tigers pretty much chewed
            through our Sherman's. In general they were recongized as the
            "superior" weapon system. One famous German tank
            commanders quote I killed 10 Sherman's today but there is
            always an 11th" Or somehting close to that effect. One of the
            problems for the Germans was that their tanks were very
            complicated and difficult to repair on the fly. However for we
            Americans the Sherman was relatively easy to repair and
            uncomplicated. In the end the German tanks had too many
            pieces parts.....:)

            Pagan jim
            Laff
          • Reminds me

            Reminds me of a quote from Star Trek IV I think.
            Scotty to Bones:
            something like this:
            "The more they over-think their plumbing the easier it is to stop up the drain."
            Loved that line.
            ajmeyercan
          • Your right.

            There have been a number of references to the exact ?complexity of design? issue of the German equipment in WWII. Not just the tanks, some of their big bore artillery pieces where the finest ever produced at the time and out shot every other artillery piece on the battlefield. Problem was maintenance was time consuming due to the vast number of pieces comprising the firing breach alone, and the breakdowns in battle became more frequent as the war went on as there were so many small pieces to break.

            Of course that became a deadly problem for the Germans when they eventually lacked the capacity to patch up or replace their equipment once they had a great deal of their infrastructure blown away. It takes a pretty sizable infrastructure for a country to support a complex and massive military.

            Windows, like any complex thing needs that patching up and maintenance, but unlike WWII Germany Microsoft still has the infrastructure to accomplish the feat, as a result we do not have people pitching out their XP disks after a few months because its gone buggy and there are no fixes available, but its no joke, the complexity vs. long term stability issue is a fact.

            I know the ?We Hate Windows? crowd doesn?t want to hear much about that because they recognize that if Windows complexity to achieve versatility really is the root of the majority of Windows problems, what they are actually seeing in an OS like XP is actually analogous to the ?Tiger Tank? of OS?s that is actually getting the maintenance it needs to remain a working and powerful ?Tiger Tank? of OS?s. That would ruin their argument that Windows is simply a bad and pointless design. I?ve always wondered why Microsoft would just decide to go with a ?bad and pointless? design, if it was in fact actually what they decided to do. I have never heard an explanation as to why they would do such a thing, although many Linux zealots say Microsoft could have created a better OS, but that assertion just brings us back full circle, even harder to the question; if it was really reasonable for them to do so then why didn?t they if they could have?

            I suspect a Tiger Tank of an OS was as good as anyone could have done, but without the support of a huge infrastructure, it could devolve into a problem, and that?s why Linux must rely on the generally K.I.S.S. (keep it simple stupid) school of design or suffer the fate of an over complex OS without sufficient support.
            Cayble
      • Then, it's a good thing...

        ...that nobody has paid a retail dime for Vista. Good article, but the TalkBack (as usual) begins to miss the target by the third of fourth post.

        It's a good idea to lower your expectations, John, until Vista is shipping to home addresses on new PC's.

        I'm having good luck with Vista so far minus some pretty annoying "features". (I think I'm becoming more irritated by the safeguards they've built into it then I would be if the system tipped over.)

        Oh, well...it's still early...
        GuyAlanDye
        • Vista is, of course, in beta

          My point was that the standards need to be higher for a commercial product (which Vista will be when it ships officially) than for free downloads like Fedora (which SP specifically mentioned). I agree that you're never going to get all the bugs out, but one has the right to expect higher quality in commercial software than in the free stuff; otherwise, what's the point in paying?

          Really, what the article has done is to demonstrate why Vista won't be shipping until sometime next year.
          John L. Ries
        • Message has been deleted.

          slack9999
    • Not true....

      If software was perfect then the time used to cover the currrent
      plauge of imperfections would be used to create more productivity
      and in other areas. A business who uses the software would have
      more time and money to well make more money still. We in the
      tech business would find something else to do. Perhaps work on
      the next generation of robots. Terra-form Mars. Develop cures
      rather than expensive and never ending treatments.

      Pagan jim
      Laff
      • But then again...

        One mans bug is another man's feature depending on how you look at it.
        Granted if the program crashes or the O/S crashes it's a bug plain and simple, but their are times when a program might allow certain actions to be performed that might not have been part of the original design plan. Wether or not that is a bug or not is a mater of your point of view.
        ajmeyercan
      • Message has been deleted.

        slack9999
  • Needs analysis

    Notice all the faiures are in Messenger and Internet Exporer... those are both USER mode applications not Kernel applications.

    Also notice the windows failures (Kernel) is at zero in your graph... looks to me like IE and Messenger aren't stable but Vista is...
    LinuxHippie
    • I couldn't show all the data...

      But you're essentially correct. There were no Windows kernel failures.

      In that screen, you're only seeing one day's worth of data. There were a total of 32 application failures in all on that machine. Of those, 18 were from Office 2007 apps (and 14 of those were from Outlook). There were a total of 7 failures in Explorer.exe, the Windows shell, 2 in Internet Explorer, and 2 in Windows Live Messenger beta.
      Ed Bott
      • further clarification

        The only one I would put towards Vista specifically is the explorer.exe ones, 7 isn't super bad for beta code.

        The others in Office, IE, Messenger, Outlook will be fixed by other groups when they release. You really have 4 beta user mode apps and 1 beta OS here, I don't think it is fair for you to lump all those into Vista failures.

        Personally I think your title and opinions are a little unfair here. Try to divide them out and judge each product on it's own merits and I think your article would definately be more interesting.
        LinuxHippie
        • Maybe I'm being too nuanced here...

          But what I'm reporting here is the impression I got from the Reliability Monitor when I first looked at it. It looks like it's decaying, although when I look at the data more carefully it seems to be a different story. From a PR point of view, Microsoft is shooting itself in the foot here by including this silly index number.

          Anyway, I'm not the one "lumping all those into Vista failures." That's what the Vista Reliability Monitor is doing. Poorly, in my opinion.
          Ed Bott