The rise of the column-based database

I always get the feeling that I need to tread carefully when talking about the relationship between developers and DBAs and that there might be some sense of hostility between the two based upon the perceived significance of their roles in the total build process. Correct me if I am wrong, but one beast can’t exist without the other and it seems to me that there ought to be more team connectivity at this level.

Thinking about this in relation to the job I am working on today, when you consider the fact that the structure and very nature of data is becoming more complex every day, the worth of a DBA should be becoming ever higher as a result shouldn’t it? I’ve interviewed a chap called Irfan Khan who happens to be chief technology officer with Sybase and his take on technical evangelism and the changing nature of data got me thinking. Let me explain…

Khan reckons that trends in Business Intelligence are driving the need for columns-based data stores and analytical tools to function with them. Exponential data stack growth and complex queries within it mean that analytics tools (which are becoming more mainstream) are having to cope with an ever smaller time window between transactional systems populating these “systems of record” and complex query analysis being performed across these.

The answer to this situation (if you buy the above argument) according to Khan is that, “Ultimately, customers are able to benefit from column-based stores since they enable real world problems to be solved; such as being able to analyse the humongous data explosion, permit scaling for increasing user populations and help expand the data mining and productivity capabilities of knowledge workers. Column-based stores are architected and optimised for complex ‘what-if’ analysis across terabytes of data, they are substantially more efficient than a conventional RDBMS in managing storage, memory and processing resources."

I think there’s a fairly watertight proposition there. No surprise at all of course to find that the company does indeed have a column-based offering in the form of Sybase IQ. But I’ve met Khan a few times and an hour’s lunch with him is like having your brain jammed with the complete thoughts of Einstein and Hawkings all in one go. Sorry Irfan, I mean that in a good way.