Five things every Windows beta tester should know

Five things every Windows beta tester should know

Summary: A few Windows 7 beta testers (and some high-profile pundits) are working themselves into a lather over the perceived shortage of feedback from Microsoft. I’ve read many of the complaints, public and private, and I think some beta testers need a refresher course in the basics of what it means to be involved in the development of a product as complex as Windows.

SHARE:

1. Things have changed. According to Thurrott, “The real problem here is that the feature set of Windows 7 was frozen well before the Beta release. So the feedback [Sinofsky] discusses throughout this post is 99 percent bug testing, really (and 1 percent, we hear your concerns but have a million reasons why we can't change a thing).”

And this is a problem? I don’t think it’s any accident that the two most troublesome releases in the history of Windows also had the longest beta cycles. I was running pre-beta builds of Windows 95 in 1993, nearly two years before it was released. The first alpha releases of Longhorn, which eventually became Windows Vista, were handed out at the PDC in late 2003, nearly three years before Vista shipped.

By the time a beta is released, the feature set should be pretty much frozen. That’s how you concentrate on things like quality and performance.

As for the complaint that Microsoft hasn’t listened to feedback and ignored its most loyal customers when developing the feature set for Windows 7, I say, “Give me a break.” Since November 2006, Microsoft has gotten an earful about its Windows design decisions. The feedback loop includes:

  • Every blog post, review, newsgroup posting, and rant about Vista ever published
  • Support calls to Microsoft and its partners
  • Requests from PC makers and software developers
  • Telemetry data (all those crash reports really do go somewhere)
  • Field research and usability testing
  • Interviews with opinion leaders, including Paul and me, who have given feedback to Microsoft in person and on the phone many times in recent years. You think they weren't taking notes?

That’s a helluva lot of feedback to take into account. There comes a point where more doesn’t mean better.

2. Windows design is a series of compromises. A lot of the complaints I’ve heard boil down to “Well, that’s not how would have designed that feature.” Right. When you’re building a product that is going to be used by hundreds of millions of people, you have to find some common denominators. And as I wrote last year, sometimes there is no right answer: you can bet that for any decision you make, some nontrivial number of people will think you’re a complete idiot, no matter which option you choose.

I also hear lots of feedback suggesting that Microsoft should never remove a feature and should always give its old-time users a way to preserve the procedures they learned five or 10 or even 20 years ago. Seriously, I’ve heard people argue that Windows 7 should include the old File Manager utility from Windows 3.1. Can you imagine how complicated, even bloated, Windows code would be if no feature was ever cut and you could choose from a dozen or so Classic interfaces going back to 1991? But that's the logical conclusion from that line of argument.

Things evolve. Old features disappear, and new ones are introduced. Deal with it. If you think there's a better way to implement a new feature than the one Microsoft chose, blog it. But it helps if you can make a rational case - remember, you're dealing with engineers. Simply saying "XYZ feature sucks" isn't likely to win hearts or minds.

3. Writing good bug reports is hard work. I sympathize with testers who complain that their bug report was closed as “Non-repro.” But that’s reality. If it was easy to reproduce, the bug would most likely have been caught in one of the many, many automated testing cycles that each Windows build goes through. The really tricky bugs are those that are triggered by unusual combinations of hardware and software under specific conditions.

In fact, if you talk to the developers who dig into those incoming bug reports from technical beta testers, as I’ve done, you’ll quickly learn that it’s a pretty low-yield process. Most are duplicates and the vast majority are just requests for new features or changes to existing ones. When they get closed as “won’t fix” or “by design,” it’s because someone already considered that request and decided for any of a thousand reasons (budget, compatibility, risk of regression, or conflicting data) that the feature is going to remain as it is.

4. One more build does not mean a better product. In fact, you could argue as I did last week that the work of building an “official” beta release slows down progress. Pytlovany, who has worked inside Microsoft, agrees:

Every new beta release is a distraction to developers. The time it takes to create a frozen version takes away from a developers imagination and productivity. […] The internal testing required before any public beta is a lot longer than you might think.

If you’ve identified a bug and it’s made it onto the must-fix list, it shouldn’t take multiple passes to fix it. Microsoft’s Charlie Owen, who works on the Media Center team, had some great advice for beta testers  in a blog post last week:

When the Windows 7 Release Candidate becomes available immediately download, install, test deeply and quickly provide actionable feedback.

Seriously: As the release candidate is downloading and with tenderness, kiss your spouse on the cheek and tell him or her you'll be back in a week or so. Then lock yourself in the home office and be relentless and unforgiving in your testing of the Windows 7 Release Candidate and provide feedback.

5. Shipping is a feature. There is no such thing as perfect software. If the developers of any complex software product like an OS waited for every bug report to be “fixed.” the software would literally never ship.

The one thing Microsoft can do going forward that it has not done well in the past is to incorporate feedback from the current test cycle into the next version. The best way for that to happen is for Microsoft to develop a consistent, professional process for planning and shipping new releases on a predictable schedule. In theory, features that don’t make it into this release have a legitimate shot at making it into Windows 8. As Charlie Owen put it, "You should consider the Windows 7 Release Candidate as your first and best opportunity to influence the next version of Windows."

Topics: Operating Systems, Microsoft, Software, Windows

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

92 comments
Log in or register to join the discussion
  • Great Write-up....

    I think you reflected what the vast majority of the testers feel.

    Even though I could b!tch and moan about not being a special 'tester' anymore w.r.t to Technet subscription which is essentially what some detractors are saying. Thurrott reflects that real well when he ridicules a beta tester's feedback to make a now empty accusation.

    Gnutella
  • Number 2 of your 5

    doesn't hold water.

    "When you?re building a product that is going to be used by hundreds of millions of people, you have to find some common denominators. And as I wrote last year, sometimes there is no right answer: you can bet that for any decision you make, some nontrivial number of people will think you?re a complete idiot, no matter which option you choose."

    This different slant on the argument from popularity doesn't wash. It is simply another apology for Microsoft.

    If the company wants people to understand their decision process it should speak about it.

    Although I am unimpressed by the 'Engineering Windows 7' blog, I see it as a step in the right direction.

    I also agree with this

    "If you?ve identified a bug and it?s made it onto the must-fix list, it shouldn?t take multiple passes to fix it."

    but since we agree, can you tell me why there are problems with the Windows Explorer that have NOT been fixed from Windows 3.0 to Windows Vista (haven't used an SP1 machine, but it was certainly there in the original release)?

    The exact problem? If I use Explorer to start renaming a number of files (different numbers depending on amount of memory installed) I will eventually reach a point where the Explorer crashes - sure, it will restart, but it still has crashed, a behavior that has been ingrained for quite some time, don't you think?
    chrome_slinky@...
    • Never seen that one

      By coincidence, I just renamed 8600+ files in a folder yesterday. Worked just fine on a system with 2GB RAM and Vista SP1.
      Ed Bott
    • IE Crashes

      My IE started crashing about 5 weeks ago, and I see you have some problems also from your post.
      I have deleted completely IE from my computer and use Firefox. However I find that it also hangs up and doesn't load after a short while.
      You have any solutions you could offer. :-)
      Thanks in advance. PS I don't know where to look for a solution.
      larrybeay@...
  • Huh

    I thought you only posted something when you had something to say!
    nizuse
    • There you are.

      Please reread Bott's blog from last week. I sent you a link to possibly fix your update problem.

      and I don't understand your need to criticize him so quickly and without saying anything concrete. It makes it seem that you just want to comment to hear the keys clicking.
      ( That is a modern version of my mother's " You talk just to hear your head rattle"). Not that you care but you are not likeable.
      sjbinaz
      • resend link

        I didn't get any link. Maybe my spam filter blocked it. Nevertheless I am very interested in any possible solution to the 'phase 3 of 3' update problem.
        nizuse
        • Not in Email

          Here:
          http://talkback.zdnet.com/5208-12354-0.html?forumID=1&threadID=61183&messageID=1128212
          It is in a reply to a post you made to ED's article last week. If you want me to email it, I would need the address.
          sjbinaz
          • Thanks

            I'll check it out. Much appreciated.
            nizuse
          • keep in touch

            If you would, please let me know. You may email me at my junk account platypusinaz@yahoo.com and we can trade email addys
            sjbinaz
          • aaah no success

            Unfortunately I had no success with the links.
            Actually I am hesitant to try any other solutions - my fear is that Vista will mess up somehow and I would then have to reformat and install Windows (and my SuSE dual boot) again. The phase3of3 thing is annoying, but bearable. A nasty side-effect could however be that new updates can't get through - not sure of that though. Sigh...
            nizuse
  • And a 6th...

    Good write up, Ed. Funny I was just talking to an MVP about this last night. Another thing that has changed about beta testing is the increasing importance of the "Dr. Watson" crash reports. While most beta testers can't write a decent bug report, having telemetry from millions of installs on real world machines is VERY valuable data that simply couldn't be simulated in a lab. The sophistication of the crash reports and their analysis, while it might not be as sexy as filing a showstopper bug report, provides information that just wasn't available years ago. It's why betas have changed from small tight knit groups of IT pros to the millions of testers you see in the Win7 beta today.
    Kip Kniskern
  • Windows 95: what a disaster

    Did you seriously mean to say that Win95 was one of the "two most troublesome releases in the history of Windows"? You couldn't have. That was a long beta that resulted in a revolutionary (relative to Windows) and phenomenal product. In that case, long beta equaled success. Maybe you meant WinMe (though I think its beta was relatively short)?

    No one's going to dispute your second choice, Vista, but it's a little unfair to count Longhorn in the beta timeline. It was scrapped, and they started all over. If you start the count there, it's not that long.

    BTW, W2K was another epic-length beta. It felt the longest for me, even though it probably wasn't, but that was another fine result.

    Bottom line: you might want to scrap that paragraph.
    rseiler
    • No, I meant Windows 95

      It was revolutionary in its impact, but it was a mess in the field. It wasn't truly usable until Windows 98 came along.

      Yes, W2K was also a long beta, but the result was pretty good.
      Ed Bott
      • Windows ME

        And here I thought Windows ME was a disaster of epic proportions.

        Or was it merely an answer in search of a question?
        Too Old For IT
    • Windows 95; immediate victory with a short term failure.

      Windows 95 was and I think still is known as the first operating system for the common person who could learn alot of it on their own. It was like a change in the trajectory of a wave, and it was a huge change that took hold so there is no reason to discount Windows 95 as anything less then groundbreaking on that issue alone.

      On the other hand, Windows 95 was the first OS to be caught flat footed by the ever exponentially expanding law of hardware development where 3 to 4 years after Windows 95 was released it was not a nice or easy task to get it inboard with a new high end system without some significant BSOD.

      Win98 was a slight improvement on the viable time line issue, but not buy a lot. By 2003 Win98 could run most hardware but it was not anything close to unusual to see people who pushed their Win98 systems hard with anything close to high end hardware got more then their fair share of BSOD and as such the XP holdouts rapidly bargain to sign on-board for XP by the 2002-2003 years span.

      Windows 95 broke huge ground but it exhibited problems with high end hardware being pushed to the limits far sooner then just about any Microsoft OS since. That doesn't make it a bad OS; it just makes it a ground breaking kick butt OS for its day that got caught by the short and curlies due to the high end development of hardware.
      Cayble
    • I don't think that's what he meant

      I think he meant that the testing process was a nightmare. These 2 major disasters eventually produced the 2 best OSs hat Microsoft ever released. (Win 98 was actually better than 95 but was essentially a service pack which you had to pay for).
      mark16_15@...
      • Isn't Win7 an SP for Vista?

        Win7 is the pr release to overshadow Vista's incompatable problems with existing software that we are expected to pay hundreds of dollars for.
        Don't get me wrong. I'm using Vista to post this, but the constant securety nag screens is maddening when your an only user.
        The article does hit the nail on the head with "this sucks" with no constructive critism.
        sykandtyed
  • RE: Five things every Windows beta tester should know

    Bill Pytlovany's right on the money... MANY beta testers DO suck. Evidence of that can be found on the Technet beta forums.

    But that's part of what you're going to get with an open beta program that will let just anyone get in on. You're going to find the guys with just enough computer knowledge to be considered "highly dangerous" mainly because they can get themselves into DEEP manure and can't get themselves out. Guess they're only interested in running the beta just for bragging rights or some such.

    Then there's the ones who didn't read the instructions - particularly the bit about NOT installing it on your primary work machine because it's a BETA and bad things can happen to your data especially without a good solid backup.

    This time around, we've even got a few TARDS showing up and posting crap just to start something. Stuff like the most alarming bugs popping up without any details as to the system the guy's got.

    There's a bunch of people who are whining about the "Classic" interface and are demanding it be put back. Gotta wonder just where these guys were when Win 95 came out...

    All in all, it seems to be going fairly well in spite of all of the distractions...
    Wolfie2K3
    • That makes me shake my head, too.

      Why the desire for the old Classic ?
      sjbinaz