Big Data's ground floor consulting opportunity

Big Data's ground floor consulting opportunity

Summary: Big Data is in a golden age of horizontal opportunity, keeping the prerequisite of vertical market expertise at bay. This provides some early opportunities for tech services firms to gain industry specialist expertise. Big Data is a Big Equalizer.


If you're in the technology consulting world for very long, you quickly encounter the opposing force of horizontal competency and industry vertical specialty.  This isn't a Big Data issue, per se; it's a technology services issue.  Clients want consultants who know their business, while technologists usually prefer to concentrate on the technology.  Some technologists just don't care much about business domain expertise; others have a real passion for it, but don't want to limit themselves to one vertical.  Either way, the gravitas most technologists achieve and pursue is on the horizontal plane.

Early on in a technology's life cycle, the requirement of vertical expertise is trumped by demand for the raw technology skill.   This set of circumstances doesn't last forever, though.  As a technology matures, tech knowledge stops being the prize and favoritism falls on those who can apply the technology with industry savvy.  At that point, competition becomes tougher, barriers to entry higher and the larger technology and professional services firms tend to gain big leverage over boutique shops.

Currently Big Data is in its horizontal golden age.  The technology is significantly more powerful, in terms of speed and data volume tolerance, than the relational database technology that has preceded it. That drives demand.  Meanwhile, the open source Hadoop stack is rather enterprise un-friendly, which puts the customer's emphasis on competency rather than business domain knowledge.  Competent implementation is Job One; contextual knowledge can come later.

I recently spoke to a medium-sized, but growing, technology services firm based in the Boston metropolitan region.  They are relatively new to Hadoop and related technologies, but they've picked it up quickly and have established a competency in it that's already an important line of business for them.   Their Big Data practice augments, and is augmented by, the more conventional enterprise database and software development services that they offer.

As I spoke to this firm's leadership, they mentioned something casually that I found to be rather significant.  They explained how the Big Data project they did for a travel client provided them with expertise in a certain type of data analysis.  All by itself that was good stuff.  But the intriguing part came when they explained to me how they were able to apply that same technique on a different project, this time for a health care customer.  To me that's huge.

Intellectually, it's not that surprising that analysis techniques are portable between industries.  But from the business point of view, customer markets and consulting teams tend to be so stratified that this kind of industrial trancendence doesn't often occur.  20 years ago it did, but as client/server technology, followed by Web development, and then mobile app development, matured, this has become far less common.  Now, it seems to me, Big Data has created a new horizontal era.

For the Boston firm I chatted with, this allows them to move beyond their travel customer base and move into healthcare.  If it plays its cards right, this company will gain the vertical expertise that will become a prerequisite again at some point, but by then the company will be ready, and won't be locked out.  Viewed this way, Big Data has become an equalizer for smaller companies (not to mention new bloggers).  This shakes up the industry a bit.  It forces the larger, incumbent tech firms to step up their game or to step aside and make room for new competitors.

This phenomenon -- a disruption in its own right -- is good for the tech world, and good for its customers too.  Firms just need to bear in mind that it's also the trait of a relatively immature technology space, and the window of horizontal opportunity will certainly close at some point.

Topics: Enterprise Software, CXO, Outsourcing

Andrew Brust

About Andrew Brust

Andrew J. Brust has worked in the software industry for 25 years as a developer, consultant, entrepreneur and CTO, specializing in application development, databases and business intelligence technology.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • Logically absurd

    "The technology is significantly more powerful, in terms of speed and data volume tolerance, than the relational database technology that has preceded it."

    The relational model is a mathematical model for representing data. To say it is slow, or unable to handle big data volumes is a little bit like saying long division is slow and unable to handle large numbers of calculations because the only implementation you have is pencil and paper.

    The Big Data people really aren't very well informed about the very basics of data management.
  • Logically Extended - Not Absurd

    When looking at the Golden age of Big Data we are not apporaching or answering the questions that were asked of relational databases the same way. To say that "Big Data people" don't understand data management is extremely inaccurate. These quite often are the same people that were pushing the envolope of capaticy and performance tuning of relational systems only 12 months ago. While Andrew may not have explicitly said it we all know that Big Data is characterized by its ability to handle data with Velocity, Varitey, and Volume characteristics, that a pure mathmatical model in a relational engine can not be optimized for.

    Big Data logically extends our analysis capibilities.
    • In what way "logically extended"?

      Please explain clearly.

      In what way are "we are not apporaching or answering the questions that were asked of relational databases the same way"? What questions are "we" asking and how are we approaching asking these questions?

      Understanding capacity and performance tuning is not necessarily the same as understanding data management.

      If your model isn't mathematical how do you know you are getting correct answers out of it?
      • Scaling Logic

        @Jorwell, your point about relational databases being capable of good/great performance is of course correct. It's just that whatever the maximum perf is in that arena could always be extended through a distributed/parallel/grid/clustered setup. That's what Hadoop offers, and because you can just keep adding nodes, you can also keep adding scale.

        That said, Massively Parallel Processing (MPP) data warehouse appliances combine relational *and* distributed query, and I fully believe MPP is a Big Data technology too. In my post of March 2nd (at I focus on that very subject.

        You are teasing out one very important point: while Hadoop parallelizes across nodes very well it doesn't necessarily do that across CPU cores on a single node. And since some RDBMSes do, they could arguably perform better than Hadoop for certain bona fide Big Data queries. That's something I hope to write about soon.
      • Scaling logic


        With an RDBMS you are working purely at the logical level.

        Whether the processors are all in one box or in several different boxes is an implementation issue and should be completely invisible to users and programmers.

        I see no reason why an RDBMS shouldn't run in an implementation where the processors are in several different boxes rather than one.

        I don't see why anyone would want to abandon the huge advantages relational offers.
  • Nice business insights

    Thanks for a great article. The horizontal business opportunity idea had not occurred to me yet.
  • yes good for consultants

    I heard a very successful consultant say he loves when software companies release buggy, hard to use technologies. Means more consulting hours!
    Not saying any of the big data/NOSQL systems are buggy or hard to use but theres a whole lottof learning curve to understand why one companies implementation is better than the other..... Or how to deploy it to fit performance and cost requirements... Or whether something is really good performance but really bad operationally/fitting in to enterprise....etc etc. And thats just on the infrastructure concerns.

    theres a great video from David CHappell called "The Case for Custom Development"
    where he makes the following three points..

    Business strategy means being different from the competition
    Being different relies on strategic IT investments to support that differentiation
    Strategic IT investments are most often custom applications
    ( Any leading-edge organization today must be good at writing custom apps )

    you could make simlar arguement for organizations that learn how to start introducing these big data solutions now, when you can benefit from strategic differentiation... rather than later when big data becomes utility IT that must be implemented to keep up with your competitor who will be able to do data processing in faster more robust ways.
    • Mo' Bugs = Mo' Billable Hours?

      @Juan - Yes, that sentiment about buggy code being good for business sounds familiar!
  • Ground floor and going up...

    I heartily agree that we are living in the early days of a Big Data technology boom.

    I started working for a mid-western consulting company 1 year ago. At that time, management made the decision to throw their hats in with a leading data warehouse appliance manufacturer and develop a niche Big Data and Analytics practice. Since then, our company has been riding a wave of growth. There have been opportunities for new contracts, corporate preferred dealer partnerships, much increased industry exposure, and people around here are getting certifications like mad to establish themselves and our company as the premier leaders in this emerging technology.

    As a side note, I don't think that @jorwell should be so defensive about Relational vs Big Data architecture. One is not a replacement for the other, but they are complementary technologies and neither is going away anytime soon. Both have their strengths and limitations, and a consultant in today's BI marketplace needs to understand both so as to address customer requirements with the best solution possible.