Big Data's ground floor consulting opportunity

If you're in the technology consulting world for very long, you quickly encounter the opposing force of horizontal competency and industry vertical specialty.  This isn't a Big Data issue, per se; it's a technology services issue.  Clients want consultants who know their business, while technologists usually prefer to concentrate on the technology.  Some technologists just don't care much about business domain expertise; others have a real passion for it, but don't want to limit themselves to one vertical.  Either way, the gravitas most technologists achieve and pursue is on the horizontal plane.

Early on in a technology's life cycle, the requirement of vertical expertise is trumped by demand for the raw technology skill.   This set of circumstances doesn't last forever, though.  As a technology matures, tech knowledge stops being the prize and favoritism falls on those who can apply the technology with industry savvy.  At that point, competition becomes tougher, barriers to entry higher and the larger technology and professional services firms tend to gain big leverage over boutique shops.

Currently Big Data is in its horizontal golden age.  The technology is significantly more powerful, in terms of speed and data volume tolerance, than the relational database technology that has preceded it. That drives demand.  Meanwhile, the open source Hadoop stack is rather enterprise un-friendly, which puts the customer's emphasis on competency rather than business domain knowledge.  Competent implementation is Job One; contextual knowledge can come later.

I recently spoke to a medium-sized, but growing, technology services firm based in the Boston metropolitan region.  They are relatively new to Hadoop and related technologies, but they've picked it up quickly and have established a competency in it that's already an important line of business for them.   Their Big Data practice augments, and is augmented by, the more conventional enterprise database and software development services that they offer.

As I spoke to this firm's leadership, they mentioned something casually that I found to be rather significant.  They explained how the Big Data project they did for a travel client provided them with expertise in a certain type of data analysis.  All by itself that was good stuff.  But the intriguing part came when they explained to me how they were able to apply that same technique on a different project, this time for a health care customer.  To me that's huge.

Intellectually, it's not that surprising that analysis techniques are portable between industries.  But from the business point of view, customer markets and consulting teams tend to be so stratified that this kind of industrial trancendence doesn't often occur.  20 years ago it did, but as client/server technology, followed by Web development, and then mobile app development, matured, this has become far less common.  Now, it seems to me, Big Data has created a new horizontal era.

For the Boston firm I chatted with, this allows them to move beyond their travel customer base and move into healthcare.  If it plays its cards right, this company will gain the vertical expertise that will become a prerequisite again at some point, but by then the company will be ready, and won't be locked out.  Viewed this way, Big Data has become an equalizer for smaller companies (not to mention new bloggers).  This shakes up the industry a bit.  It forces the larger, incumbent tech firms to step up their game or to step aside and make room for new competitors.

This phenomenon -- a disruption in its own right -- is good for the tech world, and good for its customers too.  Firms just need to bear in mind that it's also the trait of a relatively immature technology space, and the window of horizontal opportunity will certainly close at some point.