Hadoop creator: Proprietary software platforms are finished

Hadoop creator: Proprietary software platforms are finished

Summary: Doug Cutting, Hadoop creator and Apache Software Foundation chairman, tells ZDNet why he believes the days of businesses using proprietary OS and software platforms are numbered.


The days of businesses relying on locked-down operating systems and platforms tightly controlled by commercial organisations are numbered, according to Hadoop creator Doug Cutting.

Proprietary software platforms put businesses at the mercy of the platform vendor, often locking them into accepting regular price hikes, Cutting told ZDNet.

Cutting's own baby, the open source platform Apache Hadoop, is fast becoming the standard platform for big data analytics, with tech giants such as EMC and IBM releasing their own Hadoop distributions and adoption by firms growing by 60 percent annually according to research by IDC.

Hadoop creator Doug Cutting. Photo: Tim Bray (http://en.wikipedia.org/wiki/User:TimBray) under licence (http://creativecommons.org/licenses/by-sa/3.0/)

The success of Hadoop has reinforced Cutting's opinion that firms and developers will shift away from proprietary towards open source platforms.

"I don't think people are going to want a platform technology to be proprietary ever again. I think we moved past that. Linux might have been the first to make that point and I think we're emphasising that here [with Hadoop]," said Cutting, now chief architect at Hadoop software and services company Cloudera.

Cutting believes that what will kill business demand for proprietary platforms is the threat of lock-in, where a customer is dependent on a vendor for products and services such that they cannot switch to another vendor without suffering substantial costs. These costs often stem from a lack of interoperability between a platform or application a business is using and a competing product they want to switch to.

Overcoming this lack of interoperability can require extensive work in transforming data formats, rewriting custom software around new APIs and other tasks.

"Users don't want to have someone who owns their fate. If you own the software that a business is dependent upon you can change the pricing arbitrarily and that's frightening to businesses," Cutting said.

Building platforms around open standards and open-source software not only reduces the threat of lock-in for businesses, he said, but frees developers to build an ecosystem more rapidly and at less expense than on a proprietary alternative.

"If you make that central point open source there isn't a choke hold of one company profiting from every transaction and taxing, it lets it grow more freely," he said.

It's these advantages of open-source platforms over cloud-source that leads Cutting to believe companies will turn against proprietary platforms. He sees the future of proprietary software in the enterprise as lying further up the software stack, not in the platforms that applications run upon, but in the applications running on the platform.

"I don't think we're done with proprietary software but people are thinking a more reasonable place for it is at the fringes rather than at the centre," he said.

"At higher levels there's less risk of lock-in. You're not locked into it in the same way, because all your business logic, all the applications are written in terms of the open source platform."

Cutting is also chairman of the Apache Software Foundation, a non-profit whose mission is to "provide software for the public good" and provides organisational, legal, and financial support for a broad range of over 140 open source software projects.

Topics: Big Data, Enterprise Software, Open Source


Nick Heath is chief reporter for TechRepublic UK. He writes about the technology that IT-decision makers need to know about, and the latest happenings in the European tech scene.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • Sorry Doug I think you are biased

    Open source has never been about vendor lock in or proprietary software, it's been about a free lunch. There is not enough money to be made out of providing services to open source software. Once an 800 pound gorrilla such Intel, EMC or IBM becomes interested in an open source product area they take it over, leaving the smaller companies to struggle.

    Whether it is open source or proprietary software there is always a 'lock-in'. If a company puts major investment of time and money in taking on a product it is very difficult to convince management to move to another product. For example (not saying a company would attempt this) but moving from Oracle to Hadoop would take a lot of time and money. Then the next great thing comes out called SuperBigData. It again takes time and money (oh and don't forget risk) to move from Hadoop to SuperBigData.

    So Dough, you are really biased because Cloudera is paying your wage.
    • Rubbish

      Reinventing the wheel every time you need a SW function is HUGELY expensive, while reusing code and collaborating can be VERY efficient. Being locked in to a single vendor's solution can also be VERY inefficient, especially when the SW does not meet all your needs and there is little you can do to solve that problem, except wait and hope that the vendor will address your particular needs, at a price.

      The pressures to drive costs down will make more and more SW an open source commodity and proprietary solutions will slowly fall by the wayside. It is simply inevitable. Business leaders are not stupid.
      • You know that's BULL.

        Every platform is a lock-in once you adopt it. There's no easy way to move your solution from one platform to another, especially when the underline platform is complex. Just b/c you are open does not eliminate that inherent problem. This Dong is another FOSS blowhard interested in self-promotion only.
        • To some extent, yes

          But when you have the source you can modify it. There are probably others out there with similar needs and you can collaborate on solutions. You can get involved in the direction of the SW development. You do not need to sit and wait/hope.

          And you do not get a bill every so often for the privilege of just using the SW.
          • When you "modify"

            you have now taken ownership of that source and YOU have to support it. You have "locked" yourself in to that variant of software. Yes, you have now created YOUR OWN Proprietary brand of that software. Hmm... seems as though we are right back where we started. :)
          • But you can give it away again

            So don't make it sound like an anchor around your neck. Somebody else can pick up where you just left off. Happens all the time.
          • here's a hint

            your org uses skype as a voip service (now acquired by a benevolent corporation in Redmond, WA)
            1. Okay, you find that the client app is buggy, lacks features for certain needs of your operation. You do what? Try other voip clients --- noway, the skype voip protocol won't work with those
            2. You'd like to connect to some other voip networks outsuide of skype, like sip clients. Noway --
            skype is a proprietary protocol incompatible with the open SIP voip protocol
            3. you'd want to use some other long distance call providers (like voipcheap.com), since the call-out skype rates are too high. Noway with the skype client, it doesn't support other voip protocols nor the SIP protocol
            4. a security vulnerability is actively exploited in the wild. Microsoft and skype are slow to respond. Weeks and even months pass whilst they are "finding a solution and thoroughly testing it" . You have some skilled staff that would be able to patch it more quickly, if the code were open.
            Conclusion: you're locked-in to the vendor of proprietary software.
          • Great!

            Now you have the same scenario(s) where you have an open source solution. You have the same situiation where the open source code is bugyy or not able to "play with others" appropriately (happens A LOT). Well, its Open source, so you ALTER the source which is the big #1 selling point of Open source. Now (after the alteration) you and ONLY YOU own this source code and YOU and ONLY YOU have to support this code, you will not get help from anyone else including the originators of the source code because you have now ALTERED it. You are now LOCKED INTO YOUR OWN PROPRIETARY CODE! An it simply gets worse and you sink deeper and deeper as you continue to make modifications to this source code.
          • Re: you and ONLY YOU own this source code

            Or, you could contribute the patch back to the community. After all, you benefited from patches contributed by the community, didn't you?

            Also, if you have the expertise to make a patch, that must mean you have the expertise to support it. I never heard of anybody who could write code but couldn't debug it.
          • Just because something is "open source"....

            ....don't assume that others have any interest in (1) patching your bugs, (2) spending their time fixing up exploits which they don't know about (3) making sure that patches are easily available and communicated to all endpoints, and (my personal favorite): (4) THAT ANYONE IS EVEN LOOKING FOR EXPLOITS TO PATCH IN THE FIRST PLACE. Wasn't this (zdnet) the very site of this article: "Linux trailed Windows in patching zero-days in 2012, report says"???
          • Thats the selling point

            For example, you spend $10,000 for M$ or Oracle software licensing to upgrade to a piece of software which you will have to spend another $10,000 in 5 years to upgrade again -- or you pay someone $1,000 to implement that feature you need in the FOSS, or fix the bug which no one wants to fix yet (ultimately all major bugs in FOSS do get fixed!), in the process, you paid a guy to do some work for you, you contributes back to the FOSS project you use, you contributed back to the community and implemented a feature that probably others will need, you garnered more support in the FOSS project you use and maybe even got a few more major contributors interested in working on the project as they see there is opportunities there.
            Darko Gavrilovic
          • zdnet

            >> Wasn't this (zdnet) the very site of this article: "Linux trailed Windows in patching zero-days in 2012, report says"???
            It also might have said that "The Sun revolves around the Earth ... at least, as how it appears to an observer sitting a balcony"
          • s/a balcony/on a balcony/

        • Not quite

          That's not really true. Its a hell of a lot easier and less time consuming to move from FOSS PHP/MySQL based CMS to another than it is to move from one version of SharePoint CMS to another.
          Darko Gavrilovic
      • Ok tell have many development jobs have been created.

        Lets say open source keeps rolling on and large companies take up the free beer and only pay for support (forget the small companies they won't pay), 'proprietary' software companies start going to the wall and loosing jobs. The next generation of kids going to University see no point in going into computing, no jobs. You suddenly find you have no technology industry 10 years down the line.

        Companies start to find that they are struggling to compete because the technology skills are no longer available to help them. You get where I am going.

        Guess what, China and India keep going from strength to strength.

        Now you might start saying what a load of rubbish, however people laughed at me 5 years ago when I told them that I cross-shred all of my bills etc. Now everybody is doing it.

        If anybody can prove without a shadow of doubt that open source will even keep the level of jobs there are today and maybe increase them in the future, then I will gladly support it.

        However I do not need a degree in Business and Economics to know that will not happen.
        • Have you tried open source?

          Actually, open source could be the greatest thing to happen to college graduates. Instead of concentrating the programmers in to a handful of big companies, they will be spread out across the corporate sector. It takes an engineer to configure, use, and maintain Linux. It's not user friendly, it doesn't just work, it's for engineers by engineers. The big difference will be instead of paying a huge software company a big markup for the software, and then employing people to make it work. Companies will simply employ people to modify open source as well as to make it work.
          • You know nothing

            I have been using open source for the last 15 years in the form of Linux and gcc. All Linux has been doing is catching up with Unix. Every company I have worked for only uses Redhat and that is due to the fact it is the only fully supported OS on Intel/AMD that 'proprietary' software companies will support in general.

            We pay Redhats 'subscription' for the privilege.

            You can dream but open source still will not pay your wages in the future.
        • Corrections

          a) There will *never* be no technology industry. It is here to stay. It is a part of how the word works now.

          b) The off-shoring, outsourcing, and globalization always comes back.

          c) Do not let current media and new trends influence what you take in university. 20 years ago when I was in school, I would guess that at least 50% of the jobs that exist today no one even had a clue they would exist back then and no career councilor had any inkling of it. So its safe to assume that in 10 to 20 years from now, there will be 50% of new types of jobs which no one has a cue will exist in the future. and no career councilor (with with 10 year old data) will have a clue about them.
          Darko Gavrilovic
        • IT gone already

          the sea change in IT has been coming into fruition for years now and it is on its last leg toward being fully implemented. You will see a small pool of top level engineers, both in infrastructure and development being employed by the top 10 IT firms globally. All hardware is becoming a throw away technology that relies entirely on a virtual world that can come to life on that piece of hardware from the cloud at any time. Once this plan is fully executed you could literally pick up a tablet that is just lying around and that does absolutely nothing, power it on and using fingerprint/retina/voice etc. it will recognize you and pull your profile from the cloud, from there you get to your services, desktops, etc instantly. When you are done simply log out and it is as if you were never even on the device...brick...waiting for the next user. IT gone...cloud, everything.
          • The "Cloud" has existed for over 20 years

            We've all just referred to the "cloud" as the Internet or Web in the past, but always drawn it as a cloud in any network diagrams. Now the marketing bozos are pushing "cloud", and everyone thinks using the Internet to sync user settings it some type of new technology. Our IT department has not downsized yet. I'll let them know to watch out though. And who maintains all the computers running on the cloud, configures them, etc. I'm not denying change is going on in the industry, I think you're just exaggerating - a lot.