Revolution Analytics targets R language, platform at growing need to handle 'big data' crunching challenges

Revolution Analytics is working to revolutionize big data analysis with better crunching tools and an updated platform that brings the open source R statistics language to some the the largest data sets.

The company is betting its new big data scalability platform will help R transition from a research and prototyping tool to a production-ready platform for such enterprise applications as quantitative finance and risk management, social media, bioinformatics, and telecommunications data analysis.

The latest version of Revolution R Enterprise comes complete with an add-on package called RevoScaleR, a framework for multi-core processing of large data sets. With RevoScaleR, Revolution Analytics targets some of the largest levels of capacity and performance for analyzing big data, they said.

“With RevoScaleR, we’ve focused on making analytical models not just scale to the big data sets, but run the analysis in a fraction of the time compared to traditional systems,” says David Smith, vice president of Community and Marketing at Revolution Analytics. “For example, the FAA publishes a data set that contains every commercial airline take off and landing between 1987 and 2008. That’s more than 13 gigabytes of data. By analyzing that data, we can figure out the likelihood of airline delays in one second.”

A rows-and-columns approach

One second to analyze 13 GB of data should turn some heads because it takes 300 seconds with traditional methods. Under the hood of RevoScaleR is rapid fire access to data. For example, the RevoScaleR uses an XDF file format, a new binary big data file format with an interface to the R language that offers high-speed access to arbitrary rows, blocks and columns of data.

We’ve taken that one step further to develop a system that accesses the database by rows and columns at the same time

“The new SQL movement was all about going from relational databases to a flat file on a disk that offers fast to access by columns. A lot of the technology that’s behind things like Twitter and Facebook take this approach,” Smith said. “We’ve taken that one step further to develop a system that accesses the database by rows and columns at the same time, which is really well-attuned to doing these statistical computations.”

RevoScaleR also relies on a collection of the most-common statistical algorithms optimized for big data, including high-performance implementations of summary statistics, linear regression, binomial logistic regression and crosstabs. Data reading and transformation tools let users interactively explore and prepare large data sets for analysis. And, extensibility lets expert R users develop and extend their own statistical algorithms.

Integrating Hadoop

Based on the open-source R technologies, Revolution R Enterprise accordingly plays well with other modern big data architectures. Revolution R Enterprise leverages sources such as Hadoop, NoSQL or key value databases, relational databases, and data warehouses. These products can be used to store, regularize, and do basic manipulation on very large data sets—while Revolution R Enterprise now provides advanced analytics.

“Together, Hadoop and R can store and analyze massive, complex data,” says Saptarshi Guha, developer of the popular RHIPE R package that integrates the Hadoop framework with R in an automatically distributed computing environment. “Employing the new capabilities of Revolution R Enterprise, we will be able to go even further and compute dig data regressions and more.”

The new RevoScaleR package will be delivered as part of Revolution R Enterprise 4.0, which will be available for 32-and 64-bit Microsoft Windows in the next 30 days. Support for Red Hat Enterprise Linux (RHEL 5) is planned for later this year.

BriefingsDirect contributor Jennifer LeClaire provided editorial assistance and research on this post. She can be reached at http://www.linkedin.com/in/jleclaire and http://www.jenniferleclaire.com.

You may also be interested in:

• Aster Data delivers 30 analytics packages and MapReduce functions for mainstream data analytics
• Greenplum pushes envelope with MapReuce and parallelism enhancements to its extreme-scale data offering
• Delivering data analytics through Workday SaaS ERP apps empowers business managers at actual decision points