Most modern large enterprise Business Intelligence (BI) tools are very robust and feature rich these days. Up until a few years ago BI users could blame vendors for most of their BI ills. This is getting harder and harder to do. Many of the BI tools, especially the ones reviewed in our latest BI Wave, are very function rich, robust, stable and scalable. However, while the tools have really improved for the better over the last 5, typical BI issues and challenges remain the same as when I first tackled them as a BI programmer over 25 years ago: silo'd implementations, incomplete data sets, dirty data, poor management and governance, heavy reliance on IT, and many more.We are right now in the middle of running a BI survey, exploring these and other BI issues. While the results are still pouring in, the preliminary findings are 100% supportive of the evidence we've collected qualitatively and anecdotally over the past few years:
- Not all data is available in BI applications
- Data is less than 100% trustworthy
- BI applications are somewhat difficult to learn, use and navigate
- Most of the reports and dashboards are developed by IT, not end users
At the risk of repeating myself, I will state again, that over 90% of these problems are not due to poor technology, but rather due to the lack of business ownership and governance, cross LOB and departmental coordination, agreement on metric definitions and priorities, and integrated master data and metadata processes and architectures. However, even though the remaining 20% of impending technology improvements is relatively small compared to the organizational and process challenges, these 20% potentially hold a huge promise to make BI tools, and applications much more powerful.
I am currently tracking over 20 impending significant, and potentially market disruptive, next generation BI technologies. Here's a little taste of the 5 technologies that are out there right now:
- 80% of most BI projects is about data integration. And 80% of those 80% (or over 60% of the total project effort) is about discovering where the data is in the first place. These 60% remain largely un-automated even in the most modern BI applications. Ah, not so fast. Take a look at www.exeros.com and www.sypherlink.com that can auto-discover a significant portion of your data universe and relationships between data elements (not based on metadata, like most profiling tools, but based on content), and at www.ids-sheer.com that can reverse engineer (auto discover) a process, simply based on data that supports the process.
- BI architectural and process "stacks" are very complex, sometimes involving over 40 components (data sourcing, extraction, integration, cleansing, modeling, OLAP, reporting, delivery, alerts, and many more). Even a slightest change at the source application or source database can potentially trigger dozens, sometimes hundreds of changes for all of the downstream components. Even though there are no technologies today that can automate such a change with a single click of a button, www.kalido.com automates at least part of that end-to-end BI lifecycle.
- BI applications do not exist in isolation - they are an integral part of what Forrester calls Information Workplace, which involves many other applications such as collaboration, portals, search, office applications and others. While many BI vendors provide seamless integration with office applications like spreadsheets and word processors, www.panoramasoftware.com delivers BI integration with email and instant messaging. And what application is more widely used and more mission critical in any enterprise than email?
- Training advanced or power users in navigating BI applications is one thing. Casual user empowerment is a whole different ball game. Even with the most intuitive point-and-click BI GUI, many of these casual users still have difficulty translating a business question they have in their minds into BI application facts and dimensions. Some of these users may not even have a question well formed in their minds. Guided search to the rescue! Many of the leading BI vendors such as IBM Cognos, Business Objects SAP, Oracle, Information Builders, and smaller vendors, like FAST Search (recently acquired by Microsoft), Endeca and Attivio, let these casual users start exploration with a search string. The application then presents the whole universe (existing reports and raw data) of information that fits the search string, and lets a user pick and choose what he/she wants or filter out unneeded and irrelevant information.
- And last, but not least: back to the power users. These folks typically suffer from two inherent flaws of any BI environment: pre-built, rigid data models (relational or multidimensional) that form the basis of any BI application, and heavy dependence on IT staff. In-memory BI applications, where data models can be constructed on the fly by any user (well, any knowledgeable and trained user) from www.qliktech.com and www.spotfire.com , and desktop based MOLAP cubes, such as PowerCube from IBM Cognos or Essbase from Oracle already address the first limitation (see my BI Workspaces research). And of course, spreadsheets are always the last, and the most popular, resort for power user self service. The latest Microsoft announcement takes it up a notch. Its Gemini tool (to be available for beta testing sometime in 2009 and general availability in 2010) will not only enable power users to build their own models and BI applications, but easily make them available to power users, almost completely taking IT out of the loop. In Gemini, the in-memory, on the fly modeling will be done via a familiar Excel interface. Once a new model and an application is built in Excel, a power user can then publish the application to Sharepoint, making it instantly available to casual users. Not only that, but the act of publishing the model to Sharepoint also creates a SQLServer Analysis Services cube, which can be instantaneously accessed by any other BI, even non Microsoft, tool.
Add to all that another 15 or so new approaches to traditional BI that many mainsteam and small vendors are beginning to roll out, or working hard back in their R&D labs. I don't have a crystal ball, but if I were a betting person, I would bet that BI applications will look and behave completely differently in 2 years. Who ever said that BI market is mature?