This guest post comes to us from Greg Todd, chief technology officer at Revolution Analytics. Greg has been a technical architect supporting analytics and predictive management for more than 20 years—most recently completing a 15-year run at Accenture, where he was a managing director in the Digital Data & Analytics practice.
By Greg Todd
In the changing world of technology, information, and analytics, the ability to leverage the latest techniques and tools has historically fallen to a limited group of experts - the statisticians. Over the past few years, that role has evolved to become more one of management scientist, coupled with a business analyst, both of whom try to hypothesize and solve the ever increasing complexity of analysis of customer and market behaviors to gain competitive advantage.
Enter the tools
Now we are seeing the deployment of, and access to, new and unique vendor applications and platforms to help ease the complexity of analyzing large data sets for competitive advantage. Growing networks of predictive analytics communities have allowed for vendor applications to become more visual, easier to use, and geared more towards business analysts vs. the hard-to-find statisticians and management scientists of the world. Understanding that there is a gap in talent for statisticians and management scientists in general, these vendor applications are bridging the gap by allowing for easier-to-use predictive analytics capabilities to be made available to anyone, at any level of an organization.
For example, one of the most widely used predictive analytics platforms is Open Source 'R'. The community that leverages this language has grown to several million and this has become one of the most popular statistical modeling languages in universities across the globe. However, the open source version of this language has not been architected to handle big data sets and thus there are vendors delivering applications that can handle the scalability, performance, and visualization of statistical model created using open source R.
A layered approach
In preparation for the development of an analytical platform, one common trend is to focus on three specific categories of applications - the Consumption Layer, the Modeling Layer, and the Data Management Layer. Innovations in these three categories are filling in the white space from when nothing existed to now a growing tidal wave of vendor applications and solutions. Consider these layers independently and we find:
The Consumption Layer was once dominated by BI tools (those that focused on descriptive analytics or 'what happened?' analytics). Now we find a strong surge of vendor applications focused on predictive analytics ('what might/could happen?'), and even prescriptive analytics ('what do we do when it does happen?'). This is where there is tremendous diversity of applications, platforms, and products that help to visualize data.
The Modeling Layer has new entrants as well. We discussed the open source R community, but that has now been augmented by a number of new and necessary languages to help drive predictive analytics to new heights. Languages like Hive, Python, and Pig, as well as Weka, KNIME, and others are now thriving with their own user communities and strong supporters.
The Data Management Layer has gone through a massive shift from the enterprise data warehouse dominating the landscape to new data warehouse appliances and platforms like Hadoop which are helping to manage the onslaught of Big Data challenges across every organization. Data transformation, storage, security, and integration are at the forefront of every IT organization.
Ultimately, it is the convergence of the predictive analytics tools and platforms that are shaping the next generation of business leaders and competitive landscapes, and it's all exciting. The information age is truly upon us.