Will the 'real' Windows 7 testers please stand up?

Will the 'real' Windows 7 testers please stand up?

Summary: There's a group of "elite" Windows 7 testers that few have talked about publicly, known as the "Test Pilots." What has their role been in the Windows 7 development process and how have they influenced the product?

SHARE:

After two-plus years of development, Windows 7 is almost done. Microsoft officials have been thanking (though not to some testers' liking -- more on that later) the millions who downloaded the various test builds and provided feedback for their help in building the product.

However, Microsoft's brass isn't thanking publicly an inner circle of about 40 or so individuals who actually were the "real" Windows 7 testers. This group of hand-picked individuals are the ones on whom the Windows team seemed to count for a lot of the feedback about the various test builds of Windows 7 as it made its way through the development pipeline.

These so-called Test Pilots saw more Windows 7 builds than the rest of the testing community, and they usually saw them earlier. Microsoft officials were especially attuned to their feedback when tweaking the product. In spite of the Windows client team's repeated insistence that "telemetry, telemetry, telemetry!" means Windows 7 is a customer-driven release, some believe that the Test Pilots and Microsoft's corporate Technology Adoption Program (TAP) testers were the only external entities that had much of any real impact on the final Windows 7 product.

(Update: For the record, the Windows client team insists that this perception is untrue and that testers' feedback made a difference in Windows 7. See the Engineering 7 blog posts here, here and here for more on Microsoft's defense of its testing policies.)

None of the Test Pilots I contacted was willing to be identified by name. Some refused to acknowledge that  they were Pilots or knew about the Pilot program, fearing the wrath of the Windows client management. (You can't blame them; they want to be part of the Windows 8 inner circle, too.)

Microsoft's decision to create this "elite" group has created considerable animosity among other Windows testers. While the majority of the unwashed masses of public testers don't assume their suggestions and bug finds will matter much in terms of Windows' final design, Microsoft's "technical beta testers" do.

Many of technical beta testers put a lot of time and energy into unearthing and documenting bugs and passing them on to Microsoft. And a number of them were angry that they were provided with only two official Windows 7 test builds from Microsoft over the past three years -- and that all they are getting for their work on Windows 7 is a belated "we gave you a chance to get Win 7 at 50 percent off" head-pat and no free copy of the final product.

Up until the start of the Windows 7 development cycle, the technical beta testers were treated as more integral to the creation of Windows. Testers felt like their feedback and guidance mattered and that they were kept in the loop more about what was coming when. Betas of Windows felt more like malleable builds that were subject to user-contributed changes -- unlike the case with Windows 7, which a number of testers felt was nearly done by the time it hit its pre-beta milestone builds, let alone the first beta.

In short, the Windows 7 beta testing process was now a lot like Microsoft Office's, with the first test build seeming more like a final version of the product than something that Microsoft actually expected to be open to tester feedback. (Given the number of former Office managers and execs who now are leading the Windows team, that change isn't too surprising.)

Was the Windows 7 testing experience any different/better for the Test Pilots?

One Test Pilot with whom I spoke said that while the pilots received new builds of Windows 7 every month or so, they weren't privy to much information, either.

[More from the Test Pilots] -->

"We were kept in the dark as to the new (Windows 7) UI just as everyone else was," said the pilot. "The alpha builds didn't have the 'superbar' or the other major UI changes and we were never given a build that showcased them until 7100."

The pilot noted that almost all of his suggestions back to the client team regarding suggested changes and fixes were turned down. But at least "they did say 'no,' instead of silence," the pilot said.

"Microsoft is very serious about feedback," another Test Pilot told me. "We (the pilots) represent a very wide range of customers" and have a lot to contribute regarding consumer and business scenarios for Windows 7. "We were the first people to test XP Mode," for example, the pilot said.

But to suggest the Test Pilots knew all that was going on with Windows 7 is an overstatement, this pilot said.

"We were supposed to get an early Windows 7 milestone build, but after a leak, Microsoft ended up giving it only to key IHVs (independent hardware vendors) and ISVs (independent software vendors)," the pilot recalled.

The pilots did have a sense of the overall timetable for Windows 7, this pilot said. When the beta build slipped a bit around the holidays (Microsoft ended up releasing it in January), the pilots knew it -- even though Microsoft could claim it hadn't, since company officials continually refused to provide any real beta or final ship targets for Windows 7 during most of its development.

How to incorporate user feedback is a balancing act for Microsoft, especially with a product as widely used and tested as Windows. Some say "too many cooks" with too many recipes is a big reason that Microsoft's PC-maker partners and software vendors were caught flat-footed when Vista finally was released to manufacturing. With Windows 7, Microsoft has attempted to lock down everything from the feature set to the schedule from a very early date. Maybe with Windows 8, Microsoft will find a happier medium?

What's your take on how Microsoft's testing process has/hasn't worked with Windows 7?

Topics: Windows, Microsoft, Operating Systems, Software

About

Mary Jo has covered the tech industry for 30 years for a variety of publications and Web sites, and is a frequent guest on radio, TV and podcasts, speaking about all things Microsoft-related. She is the author of Microsoft 2.0: How Microsoft plans to stay relevant in the post-Gates era (John Wiley & Sons, 2008).

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

159 comments
Log in or register to join the discussion
  • Reward for testing: Use old code until October

    Given the number of beta testers, and the fact most of them just provided telemetry data and were only really interested in getting a shiny new toy to play with (like most beta testers these days, sadly), I think it's fair enough that beta testers are not being given a free copy of Windows 7.

    What I think *isn't* right, though, is that the reward for testing Windows 7 beta and RC1 is to continue using RC1 until October, even though the RTM code will be complete in the next couple of weeks.

    Why thank people by forcing them to keep using old/broken code when new/fixed code is available? IMO it'd be better if Microsoft gave all beta testers access to the RTM with their licence keys expiring at the time of the retail release. What is there to lose from doing that?

    It's not only annoying for those testers: It's bloody annoying for software developers as well.

    We have people reporting issues in our software when run on Windows 7 beta, RC1 and various leaked builds. Others then report the issues aren't there in later (leaked) builds... It'd be nice if we didn't have to worry about which of the many Win 7 builds people are using (and the back & forth of finding out) when they report issues and we try to reproduce them.

    For both users and developers, it'd be nice not to have to investigate and resolve bugs which it turns out have already been fixed.
    LeoD
    • i agree

      windows could also use feedback from these testers from the release build to improve the rtm with an almost immediate update available when the pc connects to the internet.
      hansonjb
      • Never going to happen...

        Can youu even imagine the bandwidth it would
        take to push out a new OS over Windows Update?
        The beta didn't "support" build-to-build
        upgrades (although it was definitely possible)
        yet you expect MS to develop not just the OS
        build by build but also to push it out as a
        seperate "upgrade" process from build to build?

        Look, I'm a HUGE critic of MS (I prefer Apple
        on ALL accounts) however the complaints on here
        people are voicing just seem incredibly
        ignorant and whiny... It's like an
        overpriveledged child complaining that their
        parents only bought them a Lexus for their 16th
        BDay instead of a BMW...

        MS and Win7 is what it is so let it go already
        people...
        J242
        • It'd take as much bandwidth as the beta or RC. So what?

          MS had the bandwidth to give all those people the beta and RC releases. Giving them the RTM would be like having a third "beta" in terms of bandwidth.

          Nobody is asking for anything for free here. I explicitly said I didn't think most beta testers deserved a free copy of Windows 7 for the minimal effort they put in. I just think it is wrong to leave people -- and the developers who have to support the *other* software those people run -- using beta code when the final code is ready.

          As I said, they could give all those people the RTM with it set to expire on "beta" licence keys after the retail release date.

          It could even get them more sales sooner as those people would then have to buy a full licence in October or find their OS crippled sooner rather than later (given that the RC doesn't expire until much later).
          LeoD
          • Seriously?

            [i]I just think it is wrong to leave people -- and the developers who have to support the *other* software those people run -- using beta code when the final code is ready.[/i]

            Really? Have you not heard anything people have been telling you? What your doing by encouraging people to use beta software (especially an OS) is just wrong. You DON'T have to support beta and RC OS, and SHOULDN'T.

            If they want support, they need to be running RTM software. What kinda shim sham operation are you running?
            ShadowGIATL
          • Thousands of people are ALREADY running the beta

            They're not running the beta because of me.

            They are ALREADY running it.

            They are not going to stop running it because I tell them to.

            They are not going to stop running it to get me to look at a bug that, to them, is in another program.

            And, as I keep saying FFS, I *do* tell people I won't support them in some cases. My wish is that those people could install the RTM, once available, so that I did not have to tell them that and could support them.
            LeoD
          • Well... you can...

            when it is available. All they have to do is go buy it.
            ShadowGIATL
    • Isn't that why Microsoft limited the public ...

      ... beta to ONE build (7000) and RC to only ONE build (7100). To make it easier on developers?

      That being the case, Developers should only be concerned about reports involving RC (7100) at this point and then test those reports against RTM whenever they can get the code.

      Developers who do not have access to intermediate builds should not take reports using those builds into account. Period.
      M Wagner
      • Yes, but...

        That doesn't change the fact that users will still be on RC1 for months when RTM is ready and could have been given to them -- set to expire in October -- with no downside to doing that that I can see.

        MS did a pretty bad job of limiting the public to two builds, too. Okay, it's not their fault that so many builds leaked but it is their fault that the demand for leaked builds was created: If they had given people more up-to-date builds then there would be no demand for the leaks.

        It's anything but simple right now. People could be using RC1 because it's the latest version they can get. They could be using one of the various leaks they got hold of through dodgy means. Or they could have MSDN access and be using the RTM.

        Wouldn't it be a lot simpler for everyone to be using the RTM once the RTM is ready?

        What's the downside to that, if people are forced to pay for a retail copy after October?
        LeoD
        • Is it that serious?

          [i]That doesn't change the fact that users will still be on RC1 for months when RTM is ready and could have been given to them -- set to expire in October -- with no downside to doing that that I can see.[/i]

          Why are you worried that some people will use outdated RC software, when at this point the only code you should be developing for is RTM. No company is expected to support a RC version of an OS, when even Microsoft doesn't.

          It clearly states, and is general common sense, that beta and RC OS are not meant for production. Simply tell your clients that you're working on finallizing support for 7 RTM, and be done with it.
          ShadowGIATL
          • It isn't the end of the world, but it is still wrong.

            Why are MS thanking users who helped them by leaving them in the lurch with RC code when the RTM is available?

            And it doesn't matter if the OS versions are supported or not: We still get users reporting problems with our software when run on those OS versions. We do tell them that we don't (officially) support our software running on beta versions of Windows 7 but, unlike most companies it seems, we do actually care about fixing problems for our users and want to help them.

            The fact is that lots of people installed the beta on their main machines, even if they weren't meant to. It was obvious that this would happen.

            (I'm talking about home users here, not corporations who are probably still running Win2k or XP, change OS very slowly and would never install a beta OS in production.)

            I'd much rather say to those people, who are also using our product, that we'll look into their issues but only if they are using the latest Windows 7 build. I can't say that because most people don't have access to that build.

            So what should I say? "Sorry, I won't look at your problem until October unless you reinstall your OS?" We do say that in some cases, but I wish we didn't have to.
            LeoD
          • Yeaaaa.. well....

            [i]Why are MS thanking users who helped them by leaving them in the lurch with RC code when the RTM is available?[/i]

            Because they didn't tell you that you had to test it, and clearly stated that you would have to pay for RTM if you wanted to continue to use it? Just a guess... but most probably didn't read any of that EULA.

            [i]And it doesn't matter if the OS versions are supported or not: We still get users reporting problems with our software when run on those OS versions. We do tell them that we don't (officially) support our software running on beta versions of Windows 7 but, unlike most companies it seems, we do actually care about fixing problems for our users and want to help them.[/i]

            That is nice and all, but they knew the risk of using beta software when they installed it... and if not, then it's still their resonsibility. I'd tell them I'll help you the best I can, but until final testing is done on the RTM, there is no way to ensure good results.

            [i]I'd much rather say to those people, who are also using our product, that we'll look into their issues but only if they are using the latest Windows 7 build. I can't say that because most people don't have access to that build.[/i]

            Again, explain to them that as beta versions aren't officially supported that the best results may not be achieved until the RTM of Win 7 or they revert to the older OS.

            [i]So what should I say? "Sorry, I won't look at your problem until October unless you reinstall your OS?" We do say that in some cases, but I wish we didn't have to.[/i]

            So you would rather bleed for everyheart that choose to take those risks? Aren't you a nice guy. You'll never make it in the IT/developer world with a heart like that. Maybe EMT or something of that nature?

            I don't mean to be rude and mean, but it's the nature of the business. Users do dumb things, and we try to pad their fall as much as possible. But by doing that, aren't we really just creating bigger monsters? They think it's all ok cause IT will take care of it, when most times it could have been avoided completely.

            I'm not saying be mean about it, but sometimes you just have to draw a line, and say hey, you took a risk, and now it might be out of our hands.
            ShadowGIATL
          • Wouldn't have to be mean if the RTM was an option.

            Most of what you say is true, but it doesn't change the fact that the problems go away if the Win7 beta testers are given the RTM code -- set to expire in October -- now that it is ready.
            LeoD
          • It's just the way it is..

            [i]Most of what you say is true, but it doesn't change the fact that the problems go away if the Win7 beta testers are given the RTM code -- set to expire in October -- now that it is ready.[/i]

            But it's not ready to be delivered to consumers. They always release it to business and OEM first (as a means of RTM testing) to see if all is working as it should, and most importantly, to avoid the mass rush to activate it while they are working with those business and OEM's. This would case a huge strain if everything doesn't go well.

            It all has a reason. Just have to take everything into acount.

            All I'm saying is, don't take it all personally. Users are like AT&T, it's never their fault. :p
            ShadowGIATL
          • I agree

            why is it not likely that the updates are taking care of getting the RC to be like the RTM? Since it has been said that there is little difference between the two, I believe that my RC plus updates, which is good til 6/2010, will work fine until Oct 22.
            sjbinaz
        • Really?

          "MS did a pretty bad job of limiting the public
          to two builds, too. Okay, it's not their fault
          that so many builds leaked but it is their
          fault that the demand for leaked builds was
          created: If they had given people more up-to-
          date builds then there would be no demand for
          the leaks."

          Seriously? Back in the Vista beta fellow
          testers (Not me mind you, I couldn't care less
          at the time) were screaming bloody murder about
          not getting a new build 6 weeks after the
          previous one! When a build (I don't recall
          which one off hand) was announced and delayed
          for two weeks because of recall-class issues
          being found at the last minute (Or so we were
          told) people were foaming at the mouth! Also,
          even then with sometimes bi-weekly builds, the
          MOMENT a build was announced, youu could find a
          torrent for it from a non-connect source.

          My point is that the leaks have ALWAYS been
          there in the MS betas, it just wasn't as
          obvious during the Vista beta because they
          dropped more frequent builds to the testers...
          J242
          • Less/More doesn't mean None/All.

            Obviously there will always be people who want the absolute latest code, even if it comes from a dodgy source and probably has bugs that will render their system useless.

            I wasn't making an absolute statement.

            There will be a lot less of them if you give out more frequent updates.
            LeoD
          • Fact is...

            they don't have to release public beta builds at all. People should be glad they get included in this process at all, and understand that it is for testing purposes and NOT just so you can show off with a new OS before everyone else can.

            Showing off comes second of course... :p
            ShadowGIATL
    • Aren't most developers on MSDN?

      If so, they would have access to the final code
      on August 6th.

      I used to be on the so called inside 98 up
      through XP Home edition beta team. Getting
      discs regularly. If you want to stay in the
      program you had to be productive and submit
      bugs.

      The current system is a lot more convenient and
      you don't have to be hand picked. But reporting
      under this system is welcomed but not required.
      I like the no pressure method.

      The 40 or so Elite testers work VERY hard to
      have what they have and deserve full free
      copies. Us hobby testers get our pay in early
      access.

      Pony up for MSDN to get the final bits now.
      danilko1
      • I have MSDN access. My users do not.

        It's no problem for my own machines, but my access to MSDN doesn't allow me to give RTM copies to all my users who are still using Windows 7 RC1 or dodgy leaked builds and reporting problems which may only be reproducible on those builds (and may or may not be a problem in Windows itself rather than my software).

        And ignore developers like me for a moment: What kind of "thank you" is it to leave the normal people running broken beta/RC1 code (and RC1 is very much a beta given the changes since then) when the RTM code is ready? Or are those people supposed to go back to Vista for now, now that their usefulness is over?

        LeoD