X
Tech

Facebook open sources Zstandard data compression algorithm, aims to replace technology behind Zip

Everyone has unzipped files via an algorithm called Deflate that has been used for data compression for 20 years. Now Facebook says it has something better called Zstandard.
Written by Larry Dignan, Contributor

Facebook is open sourcing a new compression algorithm called Zstandard that aims to replace the common technology behind the Zip file format.

The most common algorithm behind the Zip file format is called Deflate. Facebook, which is detailing its latest compression efforts at the @Scale conference, said Zstandard brings 5 percent to 10 percent more efficiency to general data compression and processing data 2x to 3x faster.

While Zstandard is notable, it may have some trouble replacing Deflate. Facebook notes in a blog post that Deflate and formats like Zip, gzip, and zlib have been around for about two decades. Those compression tools are built into nearly every computing device.

Facebook noted:

Today, the reigning data compression standard is Deflate, the core algorithm inside Zip, gzip, and zlib. For two decades, it has provided an impressive balance between speed and space and, as a result, is used in almost every modern electronic device (and, not coincidentally, used to transmit every byte of the very blog post you are reading). Over the years, other algorithms have offered either better compression or faster compression, but rarely both.

The bet here is that Zstandard 1.0 can combine the latest compression tools and then optimize from there.

zstnd.png

Like most code Facebook uses, Zstandard is designed to scale. Facebook said:

Zstandard is designed to excel in today's landscape and to scale into the future. Most algorithms have "levels" based on time/space trade-offs: The higher the level, the greater the compression achieved at a loss of compression speed. Zlib offers nine compression levels; Zstandard currently offers 22, which enables flexible, granular trade-offs between compression speed and ratios for future data. For example, we could use level 1 if speed is most important and level 22 if size is most important.

Beyond Zstandard, Facebook said it is deploying MyRocks, which is a next-gen MySQL storage engine, and open sourcing the code. Facebook said MyRocks is allowing it to use half the servers to host its MySQL databases.

The company also said that it has created new tools for 360-video stabilization. The technology combines 2D and 3D stabilization methods with a rotation of motion modifiers. The upshot is that the video stabilization Facebook created is reducing bitrate by 10 percent to 20 percent with the same video quality.

Editorial standards