Microsoft's One Engineering System brings Git Virtual File System to the masses

"One" -- as in "One Microsoft" -- is Microsoft's favorite number these days.

We've heard previously about One Strategy, One Windows, One Commercial Partner. There's also an initiative inside the company called One Engineering System (1ES) that's dedicated to unifying and improving Microsoft's own engineering processes and tools -- and, as much as possible, releasing these tools to customers.

Late last week, Microsoft officials announced availability of one of the tools created under 1ES. That tool, known as Git Virtual File System (GVFS), was developed to help Microsoft's own product teams to scale the Git client to repos of extremely large sizes.

Microsoft open sourced the client code for GVFS (which still relies ona pre-release file system driver), so its use isn't recommended for production at this point. However, the Windows team is using GVFS to help deal with the Windows codebase across all of its many variants (Windows OneCore, desktop, IoT, HoloLens, Xbox, etc.), which consists of 3.5 million files and is over 270 GB in size.

As part of 1ES, Microsoft chose to use Git for source control rather than its own Source Depot source control system or TFS and its Team Foundation Version Control system because some of the company's biggest teams like Windows and Office never went with these for their own use, explained Corporate Vice President Brian Harry in a February 3 blog post about 1ES and GVFS.

While Microsoft does use its own TFS for everything from work-item tracking, to version control, to build, it also uses other Microsoft-developed and third-party tools to create its own products and services.

Microsoft "put some interesting stakes in the ground" in creating 1ES, Harry said. As the cloud is the future, Microsoft made a bet on Team Services as the backbone.

"At this point, virtually every team at Microsoft has made this transition and all of our engineering work is being managed in Team Services," Harry said.

Team Services build service is the build-orchestration system of choice, and the Team Services Build management is the UI, he said. Microsoft also has built a new "make engine" that it is not yet shipping outside the company that supports "extremely high scale and fine grained caching, parallelization and incrementality." He said this make engine has resulted in some "multi-hour builds drop sometimes to minutes."

What Harry is describing here is CloudBuild. CloudBuild is Microsoft's distributed and caching build service (detailed in this Microsoft Research paper from 2016).

CloudBuild is meant to go beyond current build systems like Make, Maven, Graddle and MSBuild, which execute on a single machine, usually the developer's desktop or laptop.CloudBuild, instead, allows teams of developers to work in parallel on a codebase.

CloudBuild -- which Microsoft originally created to enable Bing's continuous delivery cadence -- was constructed on top of Microsoft's internal Autopilot cluster-management system. CloudBuild currently runs in several company datacenters and uses thousands of servers. According to Microsoft's research paper, roughly 20,000 builds are launched per day using CloudBuild.

Microsoft also created a new build language codenamed Concord, for the Windows team. Concord "guarantees reliable builds, no over-build, and allows for efficient distribution," according to an IEEE/ACM conference paper from 2016. "Adopting Concord has led to immense performance improvements, we have seen up to 100X speedup for Windows builds," Microsoft officials said. But it sounds as if Microsoft isn't 100 percent sure whether evolution (CloudBuild) or revolution (Concord) is the best strategy forward.

With 1ES, Microsoft's goal is "to use what we ship and ship what we use," in terms of tooling, Harry said. "It's not 100% and it's not always concurrent but it's the direction -- the default assumption -- unless there's a good reason to do something else."

Video: Microsoft wants AI to help, not replace humans