Must Read: No niche for iPad: Why I returned mine

Topic: Operating Systems

Windows 7: Beta testers, not Microsoft, need more feedback

Summary: When I saw Windows 7 chief Steven Sinofsky had published another (epic-length) post, entitled "Feedback and Engineering Windows 7, I was encouraged. Until I read the post, that is.

Mary Jo Foley

By for All About Microsoft |

When I asked last week what Windows 7 testers wanted, the predominant answer was more information -- specifically more information about which bugs were being fixed (and not) as the product progressed through the pipeline.

When I saw Windows 7 chief Steven Sinofsky had published another (epic-length) post, entitled "Feedback and Engineering Windows 7, I was encouraged. Until I read the post, that is.

If you want more information on how Microsoft obtains feedback and debates that feedback internally, this Februrary 25 entry is your post. If you were hoping for more about how and when the team was going to relay that feedback to its testers, you won't be happy.

Some interesting factoids from the post:

  • During the first few week of the Windows 7 beta, more than 1 million people downloaded and installed it.
  • Microsoft has released a "reliability update" test release for Windows 7 testers with Internet Explorer 8. (My blogging colleague Zack Whittaker has more thoughts on that IE 8 update.)
  • "During a peak week in January we (the Windows dev team) were receiving one Send Feedback report every 15 seconds for an entire week, and to date we’ve received well over 500,000 of these reports."
  • Microsoft has "fixes in the pipeline for nearly 2,000 bugs in Windows code (not in third party drivers or applications) that caused crashes or hangs."

That's great. Microsoft is getting a lot of feedback about Windows 7. What kind of feedback are testers getting from the team in return? Very little. I get lots of e-mail from testers asking me whether Microsoft has fixed specific bugs that have been reported on various comment boards and Web sites. I have no idea, and neither do they.

The feedback loop cannot continue indefinitely. But would finding some way to relay to testers which bugs have been fixed and which haven't derail the development process?

Sinofsky doesn't address that issue specifically in the post. He repeats the new Microsoft mantra -- "Shipping is a feature." Specifically, in his words:

"(U)ltimately on the Windows development team we have to make a call as we’re seeing a lot of people are looking forward to us finishing Windows 7, which means we need to stop changing the product and ship it."

That's definitely true, but don't the testers spending hours, days and weeks providing you with feedback deserve more information back from Microsoft? Any thoughts about how Microsoft might provide more and better tester feedback with the least amount of time/effort?

Update: Speaking of Windows 7, Microsoft rolled out an update to Windows 7 yesterday. This wasn't one of the "placeholder" updates that it is delivering this week just for test purposes. The February 24 update was "an app compat refresh for Window 7 and is focused on older PC games," according to a company spokesperson.

Update No. 2 (February 26): Microsoft has begun posting details on some of the changes it is making to Windows 7 in response to tester feedback on the Engineering Windows 7 blog.

Topics: Operating Systems, CXO, Microsoft, Software, Windows

About

Mary Jo has covered the tech industry for more than 25 years for a variety of publications and Web sites, and is a frequent guest on radio, TV and podcasts, speaking about all things Microsoft-related. She is the author of Microsoft 2.0: How Microsoft plans to stay relevant in the post-Gates era (John Wiley & Sons, 2008).

Contact Disclosure

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

40 comments
Log in or register to join the discussion

  • Public Beta Testers are given a ....

    ... chance to try Windows 7 for no charge. They are entitled to nothing in return. Formal beta testers file bugs and receive feedback on those bugs and fixes. It is quite unmanageable to provide feedback to a million people and Microsoft never commited to doing so. If you don't like it, I suggest to format your hard drive and go back to what ever OS you were using before.
    ShadeTree
    Reply Vote

    • why testers test

      Hi. I think some testers download and test betas because they enjoy the process and like to be "in the know." But at least on the business side of the world, many are in betas because they are trying to get a jump on knowing which software they own will work and not work with a still-unfinished OS or browser. They do it for planning purposes. I'd argue -- as I bet many of them would, too -- that they would like more information on whether/when a bug will be fixed.

      Thanks. MJ
      Mary Jo Foley
      Reply Vote

      • They can take confidence in the fact that .....

        .... Microsoft gathers all the feedback and fixes what is causing issues for the majority of users on a priority basis. What they don't do is fix issues caused by 3rd party applications or drivers. That is up to the authors of those items. I can tell you on the driver side we, (OEMS), receive very detailed reports on which of our drivers are causing issues. We use that information to update them accordingly.

        The point of a public beta is to get user feedback not to provide it. While the users might desire that feedback it simply is not possible due to the high number of participants and costs involved.
        ShadeTree
        Reply Vote

      • You forget developers MJ

        And in my opinion they are the ones that need to test the most. If that had happened in the past maybe Vista would not have been such a flop.

        We seen there were needed changes in our products to work with Vista and during the beta testing we found "issues". (Many of the MS "Controls" would not work with Vista's security.) Trying to ask MS if something would be fixed by RTM was useless. You could have asked a rock and gotten more feed back. The end result, millions if users installed Vista only to find their favorite apps failed to work properly.

        One of MS's biggest issues on feedback is testers are left clueless if a problem they find is a one off or if its a problem for many. If MS would rate reported bugs by numbers of instances it would go a very long way in helping developers. If two guys in the world found a bug its not a huge issue. If 25% of the testers find the same bug then its a huge issue and developers need to pay attention to writing their code around it (or beg MS to fix it). If MS were up front and say, "we are not fixing this issue" before RTM then at least the dev folks know they need to code around it.

        The problem is, MS will NEVER admit a large perentage of testers are seeing problems. That is not good marketing.

        No_Ax_to_Grind
        Reply Vote

        • Total mis-statement of what occurred.

          Because application developers used bad practices and did not follow Microsoft advice their applications failed. This was not a Vista problem. Try writing your application with in the guidelines.
          ShadeTree
          Reply Vote

          • That old excuse would hold a lot more water...

            ...if only Microsoft followed it's own "best practices". However, there are two sets of rules, those for MS and those for third party developers.
            Breaking news! I've heard that Microsoft suddenly changed their policy, they've decided to become open and share their "best practices" with the rest of Windows developers! Oh no, wait, that would require them becoming open... never mind.
            914four
            Reply Vote

          • They do share their best practices.

            It starts with don't run every application as administrator. Haven't you ever been to a developers conference?
            ShadeTree
            Reply Vote

          • Several.

            But not in the past couple of years; and I learned from experience to take what I heard with a grain of salt.
            914four
            Reply Vote

          • Wrong, it was Mictosoft's

            controls that could not work properly under Vista. Specifically video overlays with Areo turned on.
            No_Ax_to_Grind
            Reply Vote

      • I certainly would like information on when a certain bug will be fixed

        "I stared, in horror, as chkdsk replaced all of my security descriptors on my 10, 000 files and replaced them with "default security id's"."
        http://www.eggheadcafe.com/software/aspnet/32455222/chkdsk-in-vista-32bit-sp.aspx


        Wait until it happens to you. Then you'll find out it's been over 10 years and MS has failed to fix this catastrophic bug.

        Microsoft! FIX THIS BUG!!!
        emcauley
        Reply Vote

        • got a KB article number ?

          or just the rantings of a single person, backed up by another than admits he might've screwed up his drive with paragon.

          MS can't provide a software fix to hardware failures or user created issues with low level hardware.

          It'd be like demanding MS fix the bug of allowing formatting of a drive and the subsequent lose of data.
          rtk
          Reply Vote

      • "...and to date we?ve received well over 500,000 of these reports.?

        Okay, so last I heard there were 94,000 MS employees world wide. I'm guessing that *maybe* 4000 to 5000 of them actually work on Windows. I don't have actual numbers, sorry, I'm pulling them out of the air. Consider them "statics" if that makes it easier for you to drink the koolaid.

        With bug fixes already done, plus bug fixes that still need to be made, testing that needs to be done, etc.

        Where would they get the TIME to reply back to half a million and increasing feedback submissions?

        Here's how I look at it. The feedback will com in the form of RC or Release.

        When you get it and try it, if the bug no longer exists, then hey, it got fixed, and maybe you (whichever hunreds of "you"s submitted the feedback about that particular issue) had a hand in pushing that particular bug into the "need to fix" category.

        After 14+ years in the software industry, I can tell you that the only "bug free" software that I know of is "Hello world." And that's only if the person typing in the code didn't make a mistake.
        PollyProteus
        Reply Vote

      • Beta Testers vs. Beta "Users"

        The people who downloaded the beta from where ever they found it (microsoft, torrents, etc.) are NOT beta testers. They are beta users.

        Beta users are not entitled to anything more than a free glimpse at whatever product they downloaded to use.

        Beta testers have an entirely different expectation of and from Microsoft and have access to tools and reporting instruments beta users don't.

        Including the beta users in the overly broad term "testers" does a disservice to the people who _are_ testers and really put the code through the wringer to find bugs, reliability issues, and potential crashes.

        I have been on many Windows and Office betas and I know the difference between testers and users.
        Confused by religion
        Reply Vote

    • That's an excellent idea!

      You heard it folks! Format your hard drives and stop testing MS's new operating system!

      I am sure Redmond would be more than just a bit concerned about that advice.
      emcauley
      Reply Vote

  • What's the point?

    What's the point giving feedback to beta testers? Downloading the beta does not grant them anything more than the right to test the software and report bugs. And having no feedback doesn't block them from continuing testing the software.
    Since Windows 7, they never announced a feature unless they are 100% confident they can ship it, I don't see why people think it's not going to be the same for bug fixes.
    gdfg@...
    Reply Vote

  • Boy, does Windows need testing...

    Here is a nice little tidbit about putting all your eggs in the Microsoft basket: http://www.defenceaviation.com/2009/02/french-fighter-planes-grounded-by.html

    It seems third world countries have added a new weapon to their arsenal, it is call Microsoft Windows. Man, I'm glad I'm a Mac user.
    gregory.dworak@...
    Reply Vote

    • Jumping to wrong conclusions ....

      .... seems to be an Olympic sport for Mac users. The worm in question attacks a vulnerability that Microsoft has already patched. The article in question indicts the French government more then it does Microsoft. Why weren't their systems patched? I wouldn't expect you to understand that. After all, you are just a Mac user!
      ShadeTree
      Reply Vote

      • Yea, just a lowly Mac user...

        Who never once had to deal with a Virus, Malware, Blue Screen of Death, Trojans and all those other nasties Windows user are confounded by each day. Yea, I'm just an ignorant hick Mac user who likes to use his computer for productive reasons instead of running virus scans and re-imaging my machine every six months. Ignorance is bliss indeed.
        gregory.dworak@...
        Reply Vote

        • Funny that!

          I use multiple computers running Windows and don't deal with Virus, Malware, BSOD or Trojans on a daily basis. I also use my computer for productive purposes. My virus scans run in background and I don't have to re-image my machines ever. So yes in using a generalization and describing problems that many users don't face you did demonstrate your ignorance of the subject.
          ShadeTree
          Reply Vote

          • You, either? You, too?

            [i]I use multiple computers running Windows and don't deal with Virus, Malware, BSOD or Trojans on a daily basis.[/i]

            Me, neither.

            [i]I also use my computer for productive purposes.[/i]

            Me, too.

            [i]Yea, I'm just an ignorant hick Mac user who likes to use his computer for productive reasons instead of running virus scans and re-imaging my machine every six months. [/i]

            Well, I'm no ignorant hick. I'm not a Mac user. I use my Windows computers for productive reasons. Like ShadeTree, my scans run in the background. And the only machines I ever image, are the ones that come in at work and are imaged to a particular dept.

            So, what was the problem he was having again?
            MGP2
            Reply Vote
CNET Join | Log In | Privacy | Cookies Close