Cloud computing: Why Microsoft's open source data plan is a big step forward

Microsoft's new Project Zipline compression algorithm is fast enough to compress data while it's being written to an SSD or uploaded from an IoT device, and efficient enough to get up to 96 percent compression on the Microsoft internal workloads it was first developed for. 

The reason it's so fast and so efficient is that it uses a custom hardware accelerator to look for many times more patterns than compression algorithms can usually handle; data that matches any of those patterns gets replaced by a reference to the pattern, taking up much less space.

So as well as publishing the specification of the compression algorithm as part of its contribution to the Open Compute Project Foundation, Microsoft is also publishing the Verilog register-transfer level (RTL) files required to create the silicon that runs the algorithm.

It's planning to do the same with the next version of Project Cerberus, the hardware 'root of trust' specification that aims to protect against malware for firmware and avoid worries about whether the hardware you order has been tampered with before it reaches you. The first version has a separate controller that plugs in to the PCI bus on the server, but Microsoft wants to see the same protection move into silicon like the CPU and even memory and storage. And again, when the second Cerberus spec is contributed to OCP, it will include the RTL files so vendors can easily add it to their silicon designs.

That physical implementation is the tricky bit; it's the kind of thing that hardware suppliers are usually left to figure out on their own when they deliver systems built on open standards, because that's part of how they can compete with each other. But you want vendors to adopt your open standard because it's more useful to you the more people are using it.

SEE: 20 pro tips to make Windows 10 work the way you want (free PDF)    

If you want to move data from Azure to Adobe's marketing cloud or between SAP and Dynamics the way the three companies encourage customers to do under the Open Data Initiative, then it would make more sense to move it while it's compressed rather than expanding it. And doing that means both clouds and any servers you're using will need to understand Zipline.

Specifying silicon designs with Verilog and the RTL files that describe the circuitry that makes them up is hard. Programmers with those skills are few and far between; it's not a common skill, which is why it's hyperscale clouds rather than ordinary enterprises who take advantage of the flexibility of FPGAs to deliver hardware that's precisely designed to run a specific algorithm efficiently. By handing out the kind of implementation design implementation that would usually stay inside the company, Microsoft makes it far easier to build Project Zipline and Cerberus into products, so they're likely to get used.

Intel, AMD, Ampere, Arm, Marvell and SiFive are already developing CPUs that use Zipline; Broadcom, Fungible, Mellanox, Eideticom, NGD and Pure Storage are building it in to network cards and storage systems. And while Intel didn't say it will build the next version of Cerberus into its CPUs, inviting the Microsoft team behind Cerberus to come up on stage and explain it as part of their keynote at this week's OCP Summit was a pretty big sign.

This might be an unprecedented level of open contribution but it's very much in the spirit of OCP, which started as a way for the hyperscale cloud providers to make it easier for vendors to build the hardware they want to buy. 

That's now expanding a long way beyond servers and racks and the spinoffs are going to be useful to a lot more organizations, so it makes perfect sense for Microsoft to be quite this open.

PREVIOUS AND RELATED COVERAGE

MongoDB "open-source" Server Side Public License rejected

Red Hat won't use MongoDB in Red Hat Enterprise Linux or Fedora thanks to MongoDB's new Server Side Public License.

Microsoft is revamping its Office Home Use Program to feature discounted Office 365 subscriptions

Microsoft hasn't yet launched its updated Home Use Program for Office in the U.S. But once it does, it will be offering qualified business customers 30 percent off on Office 365 Home/Personal subscriptions for employee home use.

Microsoft's HoloLens 2: Why it's really all about the cloud

Microsoft at Mobile World Congress took the wraps off of its HoloLens 2 and it has some hardware upgrades that'll help enterprise usage and the front-line workers using it. But HoloLens for Microsoft is all about landing cloud subscriptions.

Microsoft cuts roughly 200 positions in its commercial sales business

The latest Microsoft layoffs follow a January reorganization in its Worldwide Commercial Business that was meant to further accelerate cloud growth and simplify the group's structure.

Azure keeps on stacking up the cloud features on-premises TechRepublic

New services and hardware add to Microsoft's on-premises cloud-consistent Azure Stack.

Microsoft shows off Project xCloud by playing Forza Horizon 4 on phone CNET

The demo comes a week before Google is expected to announce its streaming game service.