Open Source Business Intelligence in the real world

I came admittedly very late to the TV show 24 this holiday season, but it’s always fun to see how Hollywood interprets fundamental software practices to fit an entire afternoon’s work into 15 seconds of screen drama. Data mining mobile phone records can be done in a flash to stop a bomb trigger and real time geo-mapping systems work at lighting speed, but only in the movies or TV right?

'

In the real world, data mining and business intelligence (BI) have been extrapolating additional value from raw data for many years now, albeit at a more sedate pace for the most part.

This is all well and good, but as data sources grow and more data manipulation and management occurs, so to do the regulations and restrictions governing how this data is used.

Reporting recently in the Financial Time, Geoff Nairn pointed out that, “Information compliance and governance tasks are most onerous in the financial services and healthcare sectors, which are subject to strict data protection regulations.”

Not that these restrictions would stop the Counter Terrorism Unit (CTU) on 24. But for the rest of us in the UK at least, what has previously been described as a ‘cavalier’ attitude to data digestion and investigation has been reigned in by legislation including the Data Protection Act and corporate governance measures in the US such as Sarbanes-Oxley.

So is it possible to maintain an equilibrium between the restrictions put in place by corporate governance and the desire to use BI and data mining techniques to unlock new value from our data? Is it, for example, all about setting policies for access – and is that enough?

What are the options for developers constructing intensive data churning software systems that need to overcome the hurdles of compliance? Surprisingly perhaps, if you simply Google “BI” for business intelligence, the third result (at the time of writing) comes in US open-source vendor Pentaho.

This is a system being used in the UK by the NHS and is apparently currently enjoying more than 100,000 downloads per month, making it (says an independent report by Beyenetwork) the world’s most popular open source BI suite.

But are open-source development tools any kind of answer in this sensitive area of business intelligence? I would have thought that I might be more logically talking about BI from the post-acquisition engorged mega-vendors SAP, IBM, Microsoft and Oracle.

For its part, Pentaho insists that the company presents a number of advantages when compared to traditional proprietary BI products for embedded business intelligence including a more, “Modular, componentised architecture designed for embedding within applications.”

Ultimately, the company says that the fees that an ISV or OEM pays for proprietary BI software licenses inflate the price of the ISV’s product, or reduce the ISV’s margins and that Pentaho’s solutions provide embeddable BI at a lower cost to OEMs than proprietary alternatives. Interestingly, this all comes at the same time as Gartner’s prophecies for open source BI, which centre around the prediction that it will it will escalate by 500% over the next two years.

A necessary word of caution here though, San Francisco-based freelance writer and Certified Business Intelligence Professional Ted Cuzzillo wrote on open BI tools for eWeek last year, “Naturally, there's a downside to buying from independents. People with expertise with some of the less well-known tools can be harder to find. Even so, the advantage of an open system is clear. While supposedly ‘integrated’ platforms might serve for today, open systems serve today's needs and also tomorrow's - as they leave a path open for rapid and unpredictable evolution.”

I don’t know, what do you think? We’ve had a good eighteen months of honeymoon period now since the 2007 acquisitions of Cognos, Business Objects and Hyperion. So is it time for open source BI to create a new more enterprise-compliant niche for itself in this space? Either way, Jack Bauer’s happy if he can get the terrorist’s phone lines tapped when he wants to right?