Microsoft: To ensure Windows 10 update quality, these are the tools we use
The Windows 10 version 1809 release didn't quite go as smoothly as Microsoft hoped, prompting lots of questions about how it tests the quality of its builds before release. To answer these questions, Microsoft has now revealed one of the key tools it's developed to measure the health of Windows updates.
Windows 10
These days Microsoft is heavily reliant on diagnostics data to ensure Windows 10 is humming along nicely across the world, and key to this ability is Release Quality View (RQV), an internal dashboard Microsoft uses to assess the ongoing health of millions of systems before and after updates roll out.
According to Jane Liles and Rob Mauceri from the Windows Data Science team, the company approaches every new release with the question: "Is this Windows update ready for customers?".
"This is a question we ask for every build and every update of Windows, and it's intended to confirm that automated and manual testing has occurred before we evaluate quality via diagnostic data and feedback-based metrics," they write in a blogpost.
So while Microsoft does depend heavily on diagnostic data, the pair of data scientists emphasize the internal checking by engineers that takes place even before Windows Insider testers receive a build.
First, a build that goes out must pass initial quality testing. Then Microsoft engineers who "aggressively self-host Windows" give it a thorough once-over for potential problems. Diagnostics data is involved at all stages.
"We look for stability and improved quality in the data generated from internal testing, and only then do we consider releasing the build to Windows Insiders, after which we review the data again, looking specifically for failures," explain Liles and Mauceri.
They're responsible for ensuring that Microsoft's metrics are "reliable, repeatable, precise, true and unbiased" before the big "Is Windows ready" question is broached.
SEE: How to build a successful developer career (free PDF)
The data science team is always aiming to ensure current metrics are higher than the quality levels of the previous release.
To understand what 'quality' looks like through data, the team counts the unique 'active' monthly devices — a number that Microsoft doesn't publish — and then they look at how the upgrade process went, as well as the "general health of the user experience".
Then they look at certain user scenarios that indicate success for Windows, which include "success rates for connecting to Wi-Fi, or opening a PDF file from Microsoft Edge, or logging in using Windows Hello".
Central to all this data-driven analysis is the RQV dashboard, which includes over 1,000 other measures. This dashboard is used to assess the customer experience while a build is still in the hands of engineers and Insiders, and after an update becomes generally available.
The dashboard is also critical to Windows managers' 'readiness sessions', where engineering teams review RQV measures and decide what bugs to fix.
"Starting with the lowest scoring problem areas, we run down the list of areas whose measures are proportionally farthest from their targets. The engineering owners for those areas are then called on to explain what is causing the problem, who is on point to resolve it, and when they expect the quality of that area, as represented by the measures, to be back within target."
RQV developers also built the capability to see whether a bug fix actually resolves the problem, which can be seen by a particular measure returning to a healthy range.
"Fixes that engineers check into future builds are tracked through the system, so reviewers can see when a fix will be delivered via a new build and can monitor impact as the build moves through its normal validation path: through automated quality gates, to self-hosted devices in our internal engineering 'rings', and to our Windows Insiders."
The company acknowledges there are gaps and specifically notes customer feedback as an area it is investing in "to help us identify gaps or inconsistencies in our diagnostic data-based measures".
It's also looking at how to "provide insight into the experiences you have on your actual devices". And of course the company is creating new machine-learning models for "earlier detection through text analytics".
This view of the trend for networking measures in internal and Windows Insider builds between January 25 and February 19, 2019, shows an improved score over the previous build.
Previous and related coverage
Windows 10 1809 bungle: We won't miss early problem reports again, says Microsoft
Microsoft makes changes to its Feedback Hub after failing to notice early reports flagging up data losses caused by the Windows 10 October 2108 Update.
After Windows 10's buggy patches, Microsoft talks up its 'high-quality' fixes
Microsoft promises its software updates will improve next year.
Worst Windows 10 version ever? Microsoft's terrible, horrible, no good, very bad October
Microsoft released the Windows 10 October 2018 Update on October 2, then pulled it days later. Now, with November fast approaching, the update has still not been re-released. Where's the problem? And will it happen again?
Microsoft security chief: IE is not a browser, so stop using it as your default
Internet Explorer is a 'compatibility solution' and should only be used selectively, warns Microsoft exec.
Windows 10 update is freezing games: Microsoft's fix? Uninstall it for now
Microsoft's latest update for Windows 10 users has been causing "massive lag spikes" in some games.
Here's the real reason Microsoft is already testing publicly next spring's Windows 10 release
When Microsoft began testing its Windows 10 20H1 release more than a year before it is expected to start rolling out, many company watchers wondered why. The answer may be more boring -- and a lot more complicated -- than you'd think.
Microsoft rolls out Google's Retpoline Spectre mitigation to Windows 10 users
KB4482887, released today, enables Google's Retpoline mitigation in the Windows 10 kernel (only for v1809 users).
Windows 10 previews grind to halt over GSOD bug that Microsoft can't fix
Microsoft has hit pause on part of its Windows 10 Insider preview program due to a green screen of death glitch that it can't fix.
All Intel chips open to new Spoiler non-Spectre attack: Don't expect a quick fix
Researchers say Intel won't be able to use a software mitigation to fully address the problem Spoiler exploits.
Microsoft: You really should bookmark this Windows 10 update history page
The Windows 10 version 1809 update history page alone has over one million page views and counting.
Windows 10: New study shows Home edition users are baffled by updates
How annoying are Windows 10's automatic updates? In a new study, a group of UK researchers report that users of Home edition experience unexpected restarts and inconsistent installation times, caused by inappropriate defaults and inadequate notice of pending updates.
Windows 10's new free Office app: Microsoft Store PWA now open to all
Office progressive web app, or PWA, replaces the pre-installed MyOffice app.
Chrome users: This Windows 10 Timeline extension has just landed from Microsoft
Chrome users can now get a view in Timeline of their web-browsing history thanks to an official Microsoft Chrome extension.
Microsoft security chief: IE is not a browser, so stop using it as your default
Internet Explorer is a 'compatibility solution' and should only be used selectively, warns Microsoft exec.
Microsoft just made Windows 10 updates a little easier to understand
Microsoft once again changes definitions of Windows 10 updates, but it should make life simpler.
Windows 10: Now you can get 1990s Windows File Manager from Microsoft Store
Microsoft publishes the Windows File Manager from the 1990s in its app store.
Windows Update problems: Fixed now but here's what went wrong, says Microsoft
Microsoft says Windows Update DNS outage is fixed and things should return to normal for all customers soon.
Windows 95 reborn: Windows 10, Linux, Mac get major nostalgia upgrade
Relive Windows 95 on a Mac, Linux or Windows 10 machine.
New Windows 10 updates: These are the bugs Microsoft just fixed
Microsoft releases new Windows 10 builds, says to contact your ISP if Windows Update is still broken.
How to turn features on and off in Microsoft Windows 10 from the Control Panel TechRepublic
Microsoft decided to conceal the traditional Control Panel, but you can still access it if you know how.
CES 2019: Everything we saw, from 8K TVs to amazing fake burgers CNET
The show opened with a bombshell from Apple, closed with a surprise from Samsung and had plenty of news in between.