LinkedIn on Thursday will announce a new open source project that aims to provide native support for running TensorFlow jobs on Hadoop.
TensorFlow is the Google-built open-source machine learning library that's become one of the most popular platforms for creating machine learning and deep learning applications. Hadoop is the open-source software framework for storing data and running applications on clusters of hardware. Getting these two systems to function together natively is often an obstacle for data scientists and engineers, according to LinkedIn.
In steps TonY, LinkedIn's framework to natively run TensorFlow on Hadoop. The core idea is to run TensorFlow jobs as reliably and flexibly as other first-class citizens on Hadoop including MapReduce and Spark, LinkedIn said.
"We wanted a flexible and sustainable way to bridge the gap between the analytic powers of distributed TensorFlow and the scaling powers of Hadoop," Jonathan Hung, a senior software engineer on the Hadoop development team at LinkedIn, wrote in a blog post.
"Similar to how MapReduce provides the engine for running Pig/Hive scripts on Hadoop, and Spark provides the engine for running scala code that uses Spark APIs, TonY aims to provide the same first-class support for running TensorFlow jobs on Hadoop by handling tasks such as resource negotiation and container environment setup," Hung explained.
In addition to supporting this baseline functionality, TonY also offers features that aim improve the experience of running large-scale training, including GPU scheduling, fine-grained resource requests, TensorBoard support, and fault tolerance.