Quobyte's TensorFlow plug-in speeds machine learning
Quobyte announced last week that its Data Center File System is the first distributed file system to offer a TensorFlow plug-in, providing increased throughput performance and linear scalability for ML-powered applications. They say it provides 30 percent faster throughput performance improvement for ML training workflows
Quobyte's Inc., whose major product is their scale-out Data Center File System (DCFS), is the first distributed file system to offer a TensorFlow plug-in. TensorFlow is Google's open-source ML software library for numerical computation and large-scale machine learning. It's used across industries such as autonomous vehicles, robotics, financial services, healthcare, government, aerospace, defense, and many others.
Bypass the system kernel
The plug-in allows TensorFlow applications to talk directly to Quobyte, bypassing the operating system kernel to significantly reduce kernel mode context switches and CPU usage. The resulting increased GPU utilization from the TensorFlow plug-in speeds up model training of ML workflows.
Because it entirely bypasses the kernel, the plug-in works with both current and older versions of Linux, providing a full range of flexible deployment options for use in ML. No application modifications are needed.
DCFS is a modern, multi-tenant, scale-out file system, like those used in cloud infrastructures. You can start with as few as 3 servers, and grow to thousands. Highly automated, customers can manage 100PB of data with only a couple of admins - a major savings over traditional storage systems.
The Storage Bits take
One of the growing trends in storage is placing computes within storage to avoid the overhead of large data transfers. As we transition to a digital civilization, data growth will continue to be exponential for decades to come. Private clouds will become the norm, not the exception.
Quobyte's TensorFlow plug-in is only the latest example of how we're going to manage the growing data deluge. But it's particularly timely, given the explosion of ML-enabled application development. If you need highly scalable storage, check them out.
Comments welcome!