What does it mean to be a Microsoft beta tester any more?

What does it mean to be a Microsoft beta tester any more?

Summary: There's a lot of unrest in the Windows Home Server 'Vail' community right now, with a number of enthusiasts and testers wondering aloud whether their feedback means anything any more. And they're not the only ones asking.

SHARE:

There's a lot of unrest in the Windows Home Server 'Vail' community right now, with a number of enthusiasts and testers wondering aloud whether their feedback means anything any more.

The Vail testers -- who are angry about Microsoft's decision to pull what many of them consider a compelling feature very late in the test cycle -- aren't alone in their questioning. I've heard/read the same lately from Microsoft Most Valuable Professionals (MVPs), Windows testers, Windows Live testers, Office testers and others. Corporate decisions to de-emphasize or cut programs like the the Windows Clubhouse or Windows Test Pilots -- all the while talking up how much "telemetry" data Microsoft takes into account when building its products -- has even some long-term Microsoft loyalists wondering why bother.

This isn't a new concern: In fact, it was the topic of much debate when Windows 7 and Windows Live Waves 3 and 4 were in the works. But with Vail nearing its final release -- and as we move closer to the start of public test programs for Windows 8 and Office 15/Windows Live Wave 5 -- I'm sure it's going to come up again.

I started thinking about this last week while attending a press/analyst event on the Microsoft campus last week about the future of productivity, where members of the Office team revisited the role of telemetry data in building new Microsoft products.

P.J. Hough, Corporate Vice President of Office Program Management, noted that the Office team is doing everything from running traditional focus groups. to following information workers every minute of their work days from 9 to 5, to analyzing the "Send a Smile" data and other in-product feedback, to build a substantial set of data on how people use (and try to use) Office. It sounds like Microsoft is planning to continue to do the same as it develops and ultimately tests Office 15.

Hough reiterated during the event the philosophy which has ruled Office -- and the post-Vista Windows world -- regarding betas. The Windows and Office teams these days consider "beta" to be nearly-cooked versions of their products. Hough said Microsoft doesn't want to experiment on its beta testers, but still value their feedback and take it into consideration in shaping the final builds of Office.

Many of the Microsoft beta testers I know expect and want to be guinea pigs, however. They don't want to get products in near-final form; they are willing to get them when they are still raw and far from finished.

Office execs respond by saying the way the testing process is designed is to allow increasingly larger pools of people to test beta products. They say that there are much smaller, select groups who get the code for things like Office and Windows earlier, and that Microsoft increasingly grows this pool as the products move toward their release-to-manufacturing (RTM) date.

Maybe it's the terminology that needs updating. Tech previews, betas, Release Candidates (RCs): That nomenclature used to stand for something that is different from how those terms are being used now by some teams. And the window of time between when Microsoft provides information and code to its more tech-savvy "insiders" and the public seems to be shrinking substantially.

I realize Microsoft products are used by millions (or more than a billion, in the case of Windows) and that it is impossible to take every piece of feedback into account. Extrapolating from data gathered from a variety of means is necessary.

But I'd argue not all users/testers are equal and shouldn't be treated as such. Isn't some tester feedback more credible and noteworthy than others'? All the telemetry data in the world means nothing if you discount some of your most loyal (and savvy) customers'/testers' input.

Can/should Microsoft remedy this situation? Anyone have thoughts to share that may open closed ears?

Topics: Collaboration, Microsoft, Operating Systems, Software, Windows

About

Mary Jo has covered the tech industry for 30 years for a variety of publications and Web sites, and is a frequent guest on radio, TV and podcasts, speaking about all things Microsoft-related. She is the author of Microsoft 2.0: How Microsoft plans to stay relevant in the post-Gates era (John Wiley & Sons, 2008).

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

48 comments
Log in or register to join the discussion
  • RE: What does it mean to be a Microsoft beta tester any more?

    It takes too long for Microsoft to release a product and we are all testers as we have to work with beta software-Win7 and copy and paste.
    dsherman@...
    • Maybe you just hit on something?

      @dsherman@...
      Maybe they were getting to many trolls "testing" their software pre-releases and sending back a bunch of FUD?

      Hmmm.
      John Zern
      • Really?

        [i]Maybe they were getting to many trolls "testing" their software pre-releases and sending back a bunch of FUD?[/i]

        Any way to substantiate this?

        Naw, of course not... lol...
        search & destroy
    • Title: What does it mean ................any more?

      @dsherman@...

      There is something wrong there somewhere. Maybe an English major can explain it.
      Economister
      • RE: What does it mean to be a Microsoft beta tester any more?

        @Economister : How about "What does it mean to be a Microsoft beta tester these days"? Or "... today"?
        levinson
    • Release timeframe....sweet spot?

      @dsherman@... So what would the sweet spot be? MS appears to have a 2-3 year update cycle these days. Back when Win 95/98/ME had major release updates every 13-18 months, professionals much like ourselves complained that the timeframe was too short. When Vista took over 5 years to release, everyone whined and complained about that. So.. I think the 3 year cycle is a "happy" medium. Just look at the quality improvements that MS has been able to achieve in their server and desktop products with this cycle? Office 2010, Win 7, and Server 2008R2?? All of which are great products that are each capturing market share....?!
      rock06r
    • RE: What does it mean to be a Microsoft beta tester any more?

      @dsherman@... <br>Hey guys, I have been a M!crosoft beta tester since 95 went out, and am proud to have helped in any way asked.<br>I admit there were some hits and misses as far as holes and possible exploits, but, Bills kids have worked hard to fix them.<br>Win7 has been a godsend for most IT people, myself included, by way of being organized and actually simpler to dig into and root those nasty little root kits and scareware drive-by spyware. I will admit that when thousands of people help out on the final rc candidates the input can be staggering....so even trolls input will get added to the pile.....<br>Now on the length of time, remember when Microsoft first put out Millennium? It came out way to fast with way to many bugs, maybe a bit more time in testing would have been needed?
      egcrocco
  • The importance of seeming

    "...some long-term Microsoft loyalists wondering why bother."<br><br>MS practices haven't changed, now they're exposed. Sorry fanboys, the emperor was never wearing those magic clothes.<br><br>Another windows product being hobbled. Just they'll need yet another SKU, and price point;-)
    Richard Flude
    • You've never been involved, but are qualified to assume?

      Wow, this site is rampant with trolls, waiting for [i]any[/i] story to twist and turn to their own satisfaction.

      Luckilly there are good people here without their own hidden agenda like your's.
      AllKnowingAllSeeing
      • Hardly a hidden agenda

        After a decade of exposing the uninformed MCSE posts it's time to enjoy the moment.<br><br>The fanboys believe a troll is anyone critical of MS. Suck it up.<br><br>What twisting was required? Was I off topic?
        Richard Flude
      • Look who's talking

        [i]Luckilly there are good people here without their own hidden agenda like your's.[/i]

        Got lies?
        search &amp; destroy
    • You did start....

      @Richard Flude ...by calling people "fanboys". That's certainly a sign of a troll. Other than nonsensical namecalling, what intelligent commentary do you have to offer? I think this is the part where you start rubbing your ring, holding it gently and whisper "mmmmyyy pprrreeeshious..."
      rock06r
  • RE: What does it mean to be a Microsoft beta tester any more?

    Hi Mary-Jo,
    I'd like to point out that a lot of the points raised above do not apply across the board. I do extensive beta testing for a (perhaps) less newsworthy Microsoft product and the consitent level of engagement that I see from that product team toward beta testers is huge. I sometimes get involved when a feature is in planning and have input at that early stage.

    The point being that not all product teams are the same. Generally I have observed that the teams building consumer facing products are the most secretive and I suspect that is because that's where all the external interest lies. In the boring old enterprise world we do fine, ta very much.
    jamiet
    • RE: What does it mean to be a Microsoft beta tester any more?

      @jamiet,
      I have to agree with you that the issue does not seem to be across the board. In my experience, enterprise products do take into account the feedback of the smaller pool of testers which are all bound by NDA. However, these are done long before public CTP, beta and RC programs.

      The anger specific to removing DE from "Vail" probably centers around how late in the process (during CTP) the decision was made. I suspect it was a factor of delivery schedule, cross product dependency ("Aurora") and opportunity cost. The opportunity cost of delaying the release of "Aurora" probably exceeded the projected loss of revenue from shipping "Vail" without DE. Additionally, the projected revenue for "Vail" probably didn't justify the costs associated with branching it from "Aurora" to include DE.

      This certainly isn't the first time Microsoft has disappointed customers by discontinuing products or features that didn't win a cost/benefit analysis. Just ask consumers that invested their time and money in the Microsoft Money PFM product or enterprises that did the same for the Planning module of Office PerformancePoint Server 2007.

      I enjoyed flying on planes before they were always oversold, when I felt less like a sardine, didn't have to wait in line for the bathroom, and didn't have to pay for meals in coach. Unfortunately, successful businesses have to cut costs to remain profitable.
      bizNtel
      • RE: What does it mean to be a Microsoft beta tester any more?

        @bizNtel The "Vail" case was particularly bad because not only was there a huge and unpopular change late in the process, but the most requested new feature for WHS (Media Center) had already been excluded. So both early and late feedback, each with a tremendous amount of public agreement, were ignored.
        ggreig
    • Good feedback

      And yes, consumer and enterprise product testing is definitely different (for now, at least).

      I have talked to some enterprise testers for products like Forefront/Stirling and Online services who have felt they are not clued in enough. There are also a lot of enterprise users of Windows and Office who often seem surprised about what's in the pipeline, making me wonder the extent to which their feedback is taken into account in early stages. MJ
      Mary Jo Foley
      • RE: What does it mean to be a Microsoft beta tester any more?

        @Mary Jo Foley Thankyou for taking up the baton and getting the voice heard of many former and current Windows MS Beta testers.

        Its my view that the consumerisation of IT and the pressures somewhat exerceted by Apple and its product methodology has led to the situation we find ourselves today.

        Apple has been sucessful with and has brought products to market with very little external testing. However, it is arguable (and I don't want to get into a fanboy/girl debate here) as to whether or not Mac OS has the same breadth and individuality as that of the Windows OS. Mac OS by its own makeup is quite self contained, descrete and properietary whereas Windows is on the other hand installable on many different types of PC, many different brands, etc. Its more diverse in this way, and so one could say that the testing and feedback loop should be different to that used for Mac OS.

        It does seem due to consumerisation and needs for being somewhat secretive about future plans (to avoid the coyping of those ideas or IP) MS has 'shutup shot' with regards to road maps or those future plans - for the Windows Enthusisast and IT Professional who makes there living on Windows based products and systesm and has to support them this is a great shame.

        Not everybody lives in the USA, has access to feeback panels, or can partcipate in a TAP program. However, a good many MS Tech Beta Testers did have a passion for MS software, and worked with there mutual customrs for deliverying MS solutions. Part of MS Beta testing was the buzz of testing, but also for the opporunity, if you don't mind the pun, to see whats coming next, so that you can "Be Whats next", that communication/feedback channel has been removed.

        Nobody reading this should doubt the passion, drive and committment that Wendy Stidmon had towards managing the beta process. She was the glue between the beta testers, Product Managers and developers alike. She brought about a great community spirit and the abilty to interact with MS Developer first hand - you know what, they almost came across as being human again!!! ;o)

        I work for a company and IT Application Delviery and Infraturcture is my career. Losing the abilty now to have an insight into what is down the road, how best to prepare, which skills to sharpen up on, and to provide timely, detailed feedback to product groups at Microsoft will be certainly missed.

        Sure there is now telemtary, but it can only tell you so much. Further a lot of the reason why people will bother to feel in a survey or resond to feedback is if they have the interaction to demonstrate that the feedback is actioned - and sorry Mr Sinofsky, a by-weekely blog post will not cut it on that. Its about communication - 2 way communication, and building a rapport - not clicking a few buttons and hoping your voice may be heard...

        Cheers

        Janson
        janson.ragon
  • Not sure what you're getting at

    I read the linked articles you provided an saw:
    1) a guy that saw no value in for himself, as the requirements are something he couldn't do anymore.
    2) the other article had the line: [i]However, judging by the lack of interest or activity associated with Windows Clubhouse, I?m doubtful that many care about the fate of the site[/i]
    3) A personal opinion about a someone who was terminated for reasons they this person doesn't know, but will add their own reason to.
    AllKnowingAllSeeing
  • But the 5 million Public Testers can't even provide feedback

    It would be nicer though if they could allow private testers to probably provide design change request that can be made during the products development. Not when the product is actually baked.

    Looking back on the Windows 7 Beta and RC, along with Office 2010 beta Public releases. For the Public, they were really evaluations, not actual testing and feedback programs. I remember participating in the Pre-release MS Answers Forum setup for Windows 7 and that was the only way persons could provide some form of feedback.

    Considering that this was way larger than a pool of smaller private testers which is a more intimate setting, it would be chaotic for Microsoft to get feedback using the Public beta. Microsoft would never have the time to sift through millions of persons responses and feedback.

    I will admit this though, comparing Windows 7 to previous versions of Windows releases, the reliability seems to turn out better. Thousands of persons privately tested Windows Vista and provided feedback, yet it still was panned, but I remember testers telling Microsoft the product was still too slow up to RC stages and copy and pasting is like molasses. Then again, Microsoft said they listened to us after the Vista debacle, that is why our PC's are 'simplified' with Windows 7.

    The way I look at beta testing, Microsoft is realizing its probably expensive to setup exclusive programs just for a small group when you have these different resources available to you such as the following people around, research and testing labs. I also suspect Microsoft has its own internal testers in addition to the 90,000 employees. I was very surprised at how finished Windows 7 build 7000 was, I could hardly find any issues with it. Back in the 90s when the pervasiveness of the Internet wasn't available, beta programs were a must. But today, anyone can be a beta tester. If you have a popular blog and post that you found a flaw in a Microsoft pre-release product and it gets attention from other sites like CNET, ZDNET, then Microsoft takes interest.

    So, to put a conclusion on Microsoft beta testing, I would say it has evolved. Its up to us I guess to evolve with it.
    Mr. Dee
    • Good points, but

      Do you think "evolving with it" means testers and those with some in-depth knowledge about the products actually have adequate and satisfactory ways to impact how the products evolve? Do you care if your feedback affects the next Windows or next WHS? MJ
      Mary Jo Foley