Will the 'real' Windows 7 testers please stand up?

After two-plus years of development, Windows 7 is almost done. Microsoft officials have been thanking (though not to some testers' liking -- more on that later) the millions who downloaded the various test builds and provided feedback for their help in building the product.

However, Microsoft's brass isn't thanking publicly an inner circle of about 40 or so individuals who actually were the "real" Windows 7 testers. This group of hand-picked individuals are the ones on whom the Windows team seemed to count for a lot of the feedback about the various test builds of Windows 7 as it made its way through the development pipeline.

These so-called Test Pilots saw more Windows 7 builds than the rest of the testing community, and they usually saw them earlier. Microsoft officials were especially attuned to their feedback when tweaking the product. In spite of the Windows client team's repeated insistence that "telemetry, telemetry, telemetry!" means Windows 7 is a customer-driven release, some believe that the Test Pilots and Microsoft's corporate Technology Adoption Program (TAP) testers were the only external entities that had much of any real impact on the final Windows 7 product.

(Update: For the record, the Windows client team insists that this perception is untrue and that testers' feedback made a difference in Windows 7. See the Engineering 7 blog posts here, here and here for more on Microsoft's defense of its testing policies.)

None of the Test Pilots I contacted was willing to be identified by name. Some refused to acknowledge that  they were Pilots or knew about the Pilot program, fearing the wrath of the Windows client management. (You can't blame them; they want to be part of the Windows 8 inner circle, too.)

Microsoft's decision to create this "elite" group has created considerable animosity among other Windows testers. While the majority of the unwashed masses of public testers don't assume their suggestions and bug finds will matter much in terms of Windows' final design, Microsoft's "technical beta testers" do.

Many of technical beta testers put a lot of time and energy into unearthing and documenting bugs and passing them on to Microsoft. And a number of them were angry that they were provided with only two official Windows 7 test builds from Microsoft over the past three years -- and that all they are getting for their work on Windows 7 is a belated "we gave you a chance to get Win 7 at 50 percent off" head-pat and no free copy of the final product.

Up until the start of the Windows 7 development cycle, the technical beta testers were treated as more integral to the creation of Windows. Testers felt like their feedback and guidance mattered and that they were kept in the loop more about what was coming when. Betas of Windows felt more like malleable builds that were subject to user-contributed changes -- unlike the case with Windows 7, which a number of testers felt was nearly done by the time it hit its pre-beta milestone builds, let alone the first beta.

In short, the Windows 7 beta testing process was now a lot like Microsoft Office's, with the first test build seeming more like a final version of the product than something that Microsoft actually expected to be open to tester feedback. (Given the number of former Office managers and execs who now are leading the Windows team, that change isn't too surprising.)

Was the Windows 7 testing experience any different/better for the Test Pilots?

One Test Pilot with whom I spoke said that while the pilots received new builds of Windows 7 every month or so, they weren't privy to much information, either.

[More from the Test Pilots] -->

"We were kept in the dark as to the new (Windows 7) UI just as everyone else was," said the pilot. "The alpha builds didn't have the 'superbar' or the other major UI changes and we were never given a build that showcased them until 7100."

The pilot noted that almost all of his suggestions back to the client team regarding suggested changes and fixes were turned down. But at least "they did say 'no,' instead of silence," the pilot said.

"Microsoft is very serious about feedback," another Test Pilot told me. "We (the pilots) represent a very wide range of customers" and have a lot to contribute regarding consumer and business scenarios for Windows 7. "We were the first people to test XP Mode," for example, the pilot said.

But to suggest the Test Pilots knew all that was going on with Windows 7 is an overstatement, this pilot said.

"We were supposed to get an early Windows 7 milestone build, but after a leak, Microsoft ended up giving it only to key IHVs (independent hardware vendors) and ISVs (independent software vendors)," the pilot recalled.

The pilots did have a sense of the overall timetable for Windows 7, this pilot said. When the beta build slipped a bit around the holidays (Microsoft ended up releasing it in January), the pilots knew it -- even though Microsoft could claim it hadn't, since company officials continually refused to provide any real beta or final ship targets for Windows 7 during most of its development.

How to incorporate user feedback is a balancing act for Microsoft, especially with a product as widely used and tested as Windows. Some say "too many cooks" with too many recipes is a big reason that Microsoft's PC-maker partners and software vendors were caught flat-footed when Vista finally was released to manufacturing. With Windows 7, Microsoft has attempted to lock down everything from the feature set to the schedule from a very early date. Maybe with Windows 8, Microsoft will find a happier medium?

What's your take on how Microsoft's testing process has/hasn't worked with Windows 7?