The 80-20 problem

The 80-20 problem

Summary: When a closed source project gets most of the way there, its owner will redouble efforts to win market share. When an open source project gets most of the way there, its developer doesn't have a big incentive to make changes -- it works fine for them.

SHARE:
TOPICS: Open Source
20

Over at Dr. Dobb's Journal, Gregory Wilson recently brought up an interesting point.

A lot of open source projects get 80% done but then, instead of doing the hard work to make it right, developers go off somewhere else.

Wilson cites the example of Python Web application frameworks. CherryPy, Quixote, Twisted, WebWare, and others all work, but none of them are great. The documentation is poor or the user interfaces don't work well for folks unfamiliar with them.

Point taken. It's the difference between animals and plants in an ecosystem. Animals will work to gain domination, to control the territory. Plants will reproduce. One process is conscious, the other unconscious.

When a closed source project gets 80% done, its owner will redouble efforts to win market share. They will advertise heavily, work hard on enhancements, and try to take over. When an open source project gets most of the way there, its developer doesn't have a big incentive to make changes -- it works fine for them. They may work on bugs, or assume that other members of the community need to pull their load now. They may even go to work on something else.

This is human nature. The question is what can be done about it?

I don't have an answer. Maybe you do.

Topic: Open Source

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

20 comments
Log in or register to join the discussion
  • Problem or Feature?

    I see the point, but, surely, this also reflects the way that the two different approaches to creating software work?

    Open:
    Stakeholders are almost entirely developers and customers. The developers have the skills to learn what they are not told - and quickly find the support networks on-line for a far more in-depth help system than will ever be included under F1.

    In addition, most of the developers involved in the initial 80% program have a living to make and, hey, teaching people how to use the 80%, and how to customise the other 20% to be an exact fit to requirements makes very few rich - but it pays the bills.

    From that perspective you, Mr. Customer, have to find the right guy. But this is not hard for any of the big spenders on ICT. They know how to search, and how to lurk.

    Other than that, you get 100% fit from first implementation, top quality support, and a very low price.

    Closed:
    You spend ('invest' I'm not so comfortable with) in creating the 80%, then show it to lead customers who tell you about their 20% needs to make fit for purpose. You collate these and spend some more money trying to create a 20% that nearly fits all the real-life requirements of this small group.

    You launch the 'finished' product, and spend some more money on advertising, and teaching your sales people about workarounds, and how to answer all those dumb questions about the 20%. For real, as heard from software sales people:
    - "You don't need that feature"
    - "Yes, but our software works this way"
    - "You really need to review that process"
    etc.

    Reality check:
    Having said all that, the OS community doesn't do itself any favors by presenting the world with half-baked bread and no instructions on how long it needs in the oven, what temperature, etc.

    OS developers need to look at best in class projects and ask, are we doing enough?

    By the way, has anyone in the OS community qualified and listed the necessary features of a Good 80% project, a Good 100% project, and what constitutes Good OS support? I'm not aware of any.
    Stephen Wheeler
  • List them..

    There are many open source projects that are 100% or at least 90% and still growing. Or are you just talking about the ones you are interested in?
    Patrick Jones
  • finishing up

    As someone new to the Linux / open source scene,
    I have found exactly what you speak of. I finally
    got Debian software to work on my machines, the
    latest distro is great!! but alas there is little
    documentation for their latest, greatest machine,
    certainly not on their website. I did a great
    deal of searching and found several articles
    which were of immense help, and hundreds that did
    not apply, since they referred to old versions of
    the kernel. It is difficult, if not impossible to
    find help for recent issues, versions etc. I
    have become a better hacker for it, as I have no
    choice but to try,try,try again, mixing and
    matching man pages, articles, and lessons learned
    to finally achieve the answer. In many cases I
    have a serious inkling of what the problem is,
    but do not have the resources to find the answer.
    In all it is a time consuming labor of love, one
    that I doubt many technophobes could tackle, and
    the real reason, in my humble opinion, that open
    source is not taking off. Those who wrote
    articles for free on the internet, the gods of
    computing bless you!!! but motivations to be that
    nice (or motivated)are few.
    I saw an blog the other day about "selling the
    razors" and not the handle, and I think that this
    is where money could be made, and incentive
    created for developers to finish their work.
    If you just worked for seven months to create an
    application, it would be worth several thousands
    of dollars (figuring that the docs would be worth
    even .50 cents to 4 or 5 thousand users), which
    would go a long way to defray the cost of doing
    business (at least pay the electric bill, and for
    a case of beer for the launch party)
    Documentation in the form of books, often
    outdated by the time of publication, worthless
    after six months,is around $35.00 or more. A
    clearinghouse of information, current documents,
    tips, tricks etc. that charged a reasonable fee
    for information which would go to the writer,
    supported by advertising for the editorial staff
    would go a long way to filling the gap.
    Example:
    book on apache $65.00
    book on linux kernel $40.00
    book on debian distribution $35.00
    time spent trying to put it all together $1500.00
    VS
    online apache articles $10.00 (20 @ .50)
    online linux kernel articles $10.00
    latest debian instructions $20.00
    online asstd articles related to project $20.00
    time spent putting it together $200.00

    In the first example the developer got nothing
    and the installer lost money on the installation,
    since it was a one shot deal (only the publishers
    made anything for junk), in the second the
    various developers got $60.00 and the installer
    finished in reasonable time, encouraged his reuse
    of the OS, and since Linux is a moving target,
    guaranteed his return, since the information is
    constantly changing and he found the resources
    necessary to complete his mission that were
    relevant, timely, and accurate. Cheap is not
    necessarily free.
    I have been involved with computing since the
    70's as a hobby,(back when ZDNET was the computer
    shopper, a newspaper like rag,a 5.25" drive
    was an amazing luxury, and motorolla ruled the
    world with it's 6800 processor) and recently as
    an IT person for small networks.
    pesky_z
  • Incentive needed

    Open source is a hobby so the hobbyists only work on what interests them.

    You want to fix that problem, you need to create an incentive to work on the unpleasant stuff. That incentive is a paycheck. Somebody's going to have to pay somebody else to polish off that last 20%.
    baggins_z
    • Open Source is not always a hobby

      There are many people who make their living creating and working with open source tools today.

      If you had several "customers" who wanted the same set of improvements, they could pool their professional programming resources and "get 'r done."

      This, however, requires a bit of business imagination.
      DanaBlankenhorn
      • IBM often brags about billions in profits from Open Source

        IBM often brags about billions in profits from Open Source because of consulting revenues. The problem with that is the guy who contributed most to the original source code who did it for the love of coding probably didn't see too many of those dollars.

        While it is possible for guy who did the coding to sell his consulting services, how does he compete with IBM global services? He doesn't have 1/100th the reputation, sales, and marketing force of IBM global services. IBM global services can just take your GPL code and implement it however they want. Once in a while, they might throw you a few crumbs for you. The same goes for Redhat who basically provides ?the other 20%? and collect the $1500/year support contract for a copy of Linux. How much of that $1500 do the contributors of the original code see?
        george_ou
        • Good Point

          I also liked your other point on small customers of OS. I guess I just haven't seen enough of these kinds of customers for the problem to register - but they must find getting good help very tough sometimes.

          Thinking about IBM. Doesn't the GNU mean that they ought to be putting most of their 20% out there anyway?

          I have said, elsewhere on ZDNet, that I believe software is a service industry from beginning to end anyway - so maybe anything they write, even if it is only for one customer, should be sitting on a repository somewhere? Want to bet it isn't?

          Things that make you go Hmm.
          Stephen Wheeler
  • The 20% is the boring unglamorous part

    The ?20%? of the work actually takes 80% of the time. It?s the part of the job that no one will do unless they?re getting paid a decent salary for it. This lack of incentive is one of the fundamental problems with open source. No one is going to pick up the phone at night and help me if I never paid them a penny.
    george_ou
    • George has a point

      The answer lies (as so many things do these days) in business models.

      IBM uses the "service" business model for just this reason. They get paid, by the hour, for doing that 80% of the work, and the result at the end may be open source code they can re-use.

      IBM is scaled to do this. Many major consulting companies are also properly scaled. But most of those outfits (like EDS) are very closely tied to Microsoft.

      What Bruce Perens proposes to do at SourceLabs is pay people to do that 80% of the work, then deliver the result as open source.

      The question, in other words, is a good one. But sometimes there are good answers to even the best questions.

      I hate to end with a cliche but time will tell on this one.

      Good one, though, George. Thanks for stopping by.
      DanaBlankenhorn
    • How often does that 20% EVER get done?

      There is a serious lack of complete software in the world, be it open source, commercial, or internally developed. Things like good user documentation, help screens, consistent labeling, consistent look and feel, meaningful error messages, etc, etc never gets done.

      Why not? It doesn't pay. It's hard. A customer says: "The labels on all the buttons and fields is inconsistent!" The person in charge of the software says: "Ok, that's easy. Tell me what the labels should be and we'll fix it." Then the customer huffs and puffs and learns to live with inconsistent labeling.

      Go read some MS Office documentation. MS Office produces billions of dollars in profits. Is it's UI consistent? No. Is it's documentation complete? No. Does it sporadicly crash and lose your documents at inopportune times? Yes (vs. losing that email you never should sent ;).

      Software is still an American dominated industry. Americans aren't good at doing that 20% of the work. Just look at American cars...and what's happened to the American auto industry...
      Erik1234
      • RE: The 80-20 problem

        They will advertise heavily, work hard on enhancements, and try to take over. When an open source project gets most of the way there, its developer doesnt have a big incentive to make changes it works fine for them.<a href="http://tnxinvitationcode.wordpress.com/"><font color="LightGrey">tnx invitation code</font></a>
        zakkiromi
  • Intended users

    Most open source projects do not focus on the polish because the intended users are similar to the programmers. In other words, the programmers expect users to be at their skill and experience level. This is natural for a programmer, particularly for really good ones.

    Some of the best open-source projects started as tools written by the author just for their own use. They wrote it, so they did not need to polish the interface to make it usable.

    Consider DNS, e-mail, and web servers. BIND, Sendmail, and Apache. Each serves well over half of all internet traffic for their protocol. None are particularly "polished". The users (the admins of the servers) are expected to be skilled, and in fact can configure, diagnose, and monitor their applications with simple text interfaces and configuration files. There is no further interface needed for expert users. At this level, less is more... point and click interfaces actualy lower flexibility and can make essoteric configurations impossible.
    RestonTechAlec
    • yes and no

      unfortunately, many end users, even network
      administrators can't be counted on to embrace
      command line interfaces, and certainly not
      desktop, and home users. It seems that to be a
      part of the Open Source movement, you have to be
      a programmer, or have some degree in computers.
      I believe that
      even non-technical people can learn to use new
      tools if documentation is provided. Most folks
      can follow directions, they just don't have 4
      years to spare so they can set up a desktop to
      surf the web. So far it seems like no
      solution is the solution, job security seems
      paramount.
      Like I have said before, linux users seem to LIKE
      the closed doors, it may be open source, but it's
      like looking at a roadmap of the entire US at
      once, it's all there, just not much use to the
      average person. I just can't understand why they
      wonder how microsoft can do as well as they do.
      pesky_z
      • Linux

        I beg to differ. Setting up a Linux computer, from the ground up, is easier than Windows, provided you accept all the defaults and are using a modern, mainstream distrobution.

        It also takes less time, and in the end you not only have a fully functional system complete with a web browser, but a full office suite, graphics editing software, games, etc. etc. Almost all of this would be separately installed on Windows, with separate license keys, separate CD (or DVDs).

        One Linux DVD can do it all, with a bare minimum of user interaction.

        Of course, PCs with all the Windows software preinstalled are common. Ones with Linux preinstalled are not.
        Erik1234
        • easy set up

          I definitely agree that the initial install is a
          snap. After that the waters get murkier,
          especially if you want to install your favorite
          application, understand the filesystem,or have a
          printer or other piece of gear that is not
          supported by your install cd.
          pesky_z
  • Real understanding

    First let's understand that there is a wealth of open source applications. Too many to list! Much of these applications are very useful, unlike shareware which compairs to cheap impalse items at the Walmart check out. Second, Open source is a development in the making. It doesn't always have funds to help it along or programers working 50 hours a week.

    What's shocking, is the fact that out of this open development. We have gained some well known applications that are used and prefered for their stability. Let's keep in mind too that it has come this far and become this strong all on it's own. Without money and marketing. Even more, it wasn't intended to challange the main stream software market. Just a hobby, useful tools to make better tools and applications.

    With that said, look at all the action it's getting now. What's to be said about that? No one sold it to IBM but their marketing it. How far open source has come says alot! in such a short time, all on it's own. Who'd of thought we would be bloging open source?

    *grin.. I have worked in an open source environment a long time.. makes me smile to see all the hype about it today. BTW, there is some of the best documentation I have ever seen today. Well known app's have great documintation, user groups, Wikki's, and forums. I remember when I had to figure this stuff out on my own (didn't mind.. was a hobby I liked) very little help!

    *Grin just cracks me up lol..
    xstep
    • just one mre thing

      If an open source application is laking 20% our whatever. Grab the source code, add or edit the code to what you think is better. Recomplie it, and submit it. Please try and make it useful for everyone.
      xstep
  • Not entirely true

    I have to disagree. The most likely reason for the 80-20 problem lies in the fact that the "20" is the part most coders hate. i.e. Testing it, improving the small things that real-world users need, support etc. The problem today is that the open source development environment attracts some great programmers but not enough support-type engineers that could polish the library/product/whatever. Once the clever guys do the cool bits they're off looking for other cool bits to do. Which, in my opinion, explains the missing 20.
    charlie@...
    • I'm inclined to agree here

      Having done some coding in my studies, I can definitely attest to the loathsome nature of the testing process. Trying to track down every single bug in the program--including the ones that surface after another bug is "fixed"--leads to no small amount of frustration.

      The worst part of testing, of course, would be the "run-time" errors. It's bad enough when you have an error whose location you know, but whose "fix" winds up exposing other errors. But when you have an error in the code that doesn't get picked up during the compile, you often wind up getting completely wrong results, and you're forced to comb through the entire source to find what caused the "glitch" in the system.

      Of course, anything worth doing (even in such abstractions as code) is worth doing well. But the trouble involved in debugging would probably stretch even Job's patience to its limits, to wax poetic.
      Third of Five
  • Nothing has changed

    I spent 40 years in the electronics field doing design and development work. The 20 percent was the unglamourous work. I once had a boss that had a favorite saying that, the job isn't finished until the paperwork is done, here and in the bathroom.

    The fun part was the breadboarding, getting the first PC boards to work. Then you had to write the specifications and the test procedures for the factory. Then you had to critique the publication writers who usually didn't have a clue about what the box was really supposed to do and how it did it.
    wayne1932