Battling business data overload

COMMENTARY--Business Intelligence (BI) has reached a new level of importance for decision-makers. Conditions are in a constant state of flux, so companies must quickly adapt to new patterns in customer behavior, pricing and operations, in order to get and stay ahead. The speed at which companies can access, parse, analyze and leverage complex information is critical. Yet companies are struggling and spending a fortune trying to quickly get information from massive amounts of data for important business decisions.

The rapid growth of corporate data and increasing demand for complex analysis pose a serious challenge to the existing business intelligence infrastructure. Built upon a patchwork of technology over time, the infrastructure often includes many types of general-purpose DBMS software, a partially implemented middleware strategy, a collection of different mid-tier SMP servers, assorted disk arrays, and a myriad of end-user applications that rely upon different database and communication standards. The entire system is held together by database and system administrators who are struggling to keep up with growing business user demands.

Current BI infrastructure
These BI architectures, formulated years ago to serve online transaction processing needs, weren't designed to handle analytical processing of terabytes of data. Take into account real time requirements, and the patchwork of systems built on general-purpose machines cannot keep up. With data estimated to double every nine months, the analytics bottleneck is only going to get worse.

Today's solutions are constantly moving large amounts of information around complex and inefficient systems of software, servers and storage. For a typical request, a server gets all the information--often entire tables--from a separate storage device. Not until the information is moved to the server does it process the request and figure out exactly which portions of that information it needs. As more and more users run larger and more complex queries, the entire system is strained--CPU power, fabric bandwidth, disk space, memory usage and other elements. And, while companies respond by spending lots of money to add memory, fast CPUs and endless database tuning, this doesn't address the underlying problem--it merely moves the bottleneck from one part of the BI platform to another.

Purpose-built appliances
A new generation of purpose-built appliances are exploding across a multitude of mature technologies, ranging from security to Web servers and streaming media. IDC Research predicts that server appliances, including high-end purpose-built servers, will exceed $31 billion annually by 2005.

A next-generation BI platform would combine massively parallel hardware, software and storage, directly focused on providing optimal response times and scalability at the terabyte level.

By being fully compatible with the existing infrastructure, a BI appliance can both rationalize and revitalize BI efforts to enable optimized access to information. The purpose-built BI appliance balances parallel processing, filters unnecessary data and moves intelligence closer to the source of the data, all while using low cost components. Adding intelligence and processing as close to the actual data as possible creates an efficient, streamlined flow of information. It gives businesses real-time access to information, allowing them to quickly and cost effectively respond to market needs.

As business and technical demands continue to grow and change in the new century, the new BI appliance will be designed from the outset to scale with data size, scope and performance needs. And it must do all of this at an affordable and predictable price. When evaluating purpose-built appliance technology there are several important criteria to consider:

Performance: The goal of any purpose-built appliance is to provide dramatic performance improvements. As a result, complex and ad-hoc queries on terabytes of information go from hours or days to minutes or seconds.

Effortless Scalability: From gigabytes to hundreds of terabytes of user data--BI appliances must scale with little response-time degradation.

High User Concurrency: The growth of information within today's computing environment demands that BI appliances be able to handle many requests simultaneously, serving hundreds of geographically dispersed users.

Compatibility: It's critical that any purpose-built appliance fit seamlessly within the organization's environment, leveraging the investments they've already made in technology.

Affordability: Perhaps the most pressing need for business intelligence appliances is the need for a reasonable total cost of ownership. Appliances that are purpose built for analytics must have lower up-front costs, but also must reduce the on-going maintenance costs of the solution.

Flexibility: You must build a flexible system that's designed from the outset to evolve and scale with data size, scope and performance needs.

In the new environment, BI appliances will drive growth for companies by changing the way they can do analytics. Queries that were once impossible will now be executed cost effectively in minutes. The evolution of next generation BI platforms can help companies make more informed, better decisions.

Jit Saxena is co-Founder and CEO of Netezza Corp. Previously, Saxena was founder, chairman and CEO of Applix Inc., a leading provider of analytical CRM software that he took public in 1994.