CentOS: Getting Their S#!t Together is a Top Priority

CentOS: Getting Their S#!t Together is a Top Priority

Summary: The CentOS developers were ready to burn the place down if their AWOL project founder didn't show up and resolve several outstanding issues that put the popular Open Source project and Linux distribution at risk.


The CentOS developers were ready to burn the place down if their AWOL project founder didn't show up and resolve several outstanding issues that put the popular Open Source project and Linux distribution at risk.

For those of us in the "Inside Baseball' crowd in Open Source, a lot of armchair quarterbacking has been going on vis-a-vis the Open Letter that was published by the CentOS developers on July 30th which was addressed to Lance Davis, the founder and one of the lead developers of the CentOS project, one of the most-used free Linux distributions on the planet.

Click on the "Read the rest of this entry" link below for more.

As I wrote about in my round up of free Enterprise-class OSes, CentOS is a free source-level clone of Red Hat Enterprise Linux. Along with Scientific Linux, which is a separate distribution developed by Fermilab and CERN, these two distributions are the Linux of choice for environments wanting all the rock-solid stability of RHEL but don't want or need Red Hat's commercial technical support.

Also See: Frugal Friday, Interview with Troy Dawson of Scientific Linux

CentOS and Scientific Linux are built on the published GPLv2 source of Red Hat Enterprise Linux, are re-compiled without Red Hat's trademark logos, and use the community YUM package manager for updates rather than Red Hat Network, which is a commercial, fee-based support update service. CentOS in particular is used by a number of appliance and virtual appliance vendors, such as Kickfire and Schooner Information Technology in order to produce turnkey systems for security, storage, web applications and databases.

In short, the CentOS developers were brought to the brink of rebellion and walking away from the popular project because its founder, Lance Davis, had disappeared in 2008, ceased contribution to the project, and was holding the keys to the centos.org domain and the project's PayPal accounts. None of this was popular knowledge until the following letter was published on the CentOS project web site on July 31:

July 30, 2009 04:39 UTC

This is an Open Letter to Lance Davis from fellow CentOS Developers

It is regrettable that we are forced to send this letter but we are left with no other options. For some time now we have been attempting to resolve these problems:

You seem to have crawled into a hole ... and this is not acceptable.

You have long promised a statement of CentOS project funds; to this date this has not appeared.

You hold sole control of the centos.org domain with no deputy; this is not proper.

You have, it seems, sole 'Founders' rights in the IRC channels with no deputy ; this is not proper.

When I (Russ) try to call the phone numbers for UK Linux, and for you individually, I get a telco intercept 'Lines are temporarily busy' for the last two weeks. Finally yesterday, a voicemail in your voice picked up, and I left a message urgently requesting a reply. Karanbir also reports calling and leaving messages without your reply.

Please do not kill CentOS through your fear of shared management of the project.

Clearly the project dies if all the developers walk away.

Please contact me, or any other signer of this letter at once, to arrange for the required information to keep the project alive at the 'centos.org' domain.


Russ Herrold Ralph Angenendt Karanbir Singh Jim Perrin Donavan Nelson Tim Verhoeven Tru Huynh Johnny Hughes

Over this weekend on August 1, Lance Davis put his tail between his legs, made a rare appearance and surrendered the centos.org domain to the project, and the project made a statement that other administrative issues were in the process of being resolved:

The CentOS Development team had a routine meeting today with Lance Davis in attendance. During the meeting a majority of issues were resolved immediately and a working agreement was reached with deadlines for remaining unresolved issues. There should be no impact to any CentOS users going forward.

# CentOS is not dead or going away. The signers of the Open Letter are fully committed to continue the CentOS Project. Updates and new releases will continue.

# Most of the Issues have been resolved, there is an action plan with agreed upon dates for any outstanding issues.

# The CentOS Project now owns the CentOS.org and CentOS.info domains and there will be no disruption in services.

# We thank the people who have stepped forward and want to donate to the CentOS project. We ask that you hold off for now until issues surrounding our new donation policy are put into place.

# The CentOS Project is run completely by volunteers and we are aware that this requires a different management style. We have been and continue to work to prevent issues like these from occurring in the future. We will continue this effort in the future, look for some new policy information soon.

So everything is being resolved, CentOS is getting its act together, and we can all go about our business. Nothing to see here, move along. Move along. Right?


Frankly, I am absolutely amazed that such a high-profile Open Source project could continue in operation without addressing who was in control of its financials  as well as its Internet presence. This particular incident raises a whole bunch of questions as to how large Open Source projects should be managed and who should be made accountable.

For starters, there needs to be clear policies of who is going to administer what, and that there should never, ever be a situation where one person holding the keys to the domain or any key administrative and financial accounts can just walk away and leave the developers holding their you-know-whats in their hands. This behavior is absolutely abysmal and Davis should be lucky not to walk out of this with civil charges pressed against him.

CentOS, if it isn't already, should also immediately seek Not-For-Profit status in either the UK or the United States, where contributions made to it can be tax deductible. A formalization of the organization as a Not-For-Profit corporation (in the US, this is referred to as a 501(c)3 under the US tax codes) would require a number of bylaws as well as officers responsible for the welfare of the Corporation. This is exactly how the other two high profile community Linux distributions, Debian and Ubuntu operate in public, as do several of the BSD Unixes.

There is also the question of whether or not it makes sense for there to be two community-driven Red Hat clones. Scientific Linux, while under the stewardship of Fermilab and CERN, two very stable organizations, only has a small amount of developers working on it. It might behoove both Scientific and CentOS to join forces and pool their resources in producing a single "Mother Distribution" which both can be based on, and form a Not-for-Profit organization that can be made responsible for the non-technical administravia and financial aspects that will ensure their continuance.

Did the CentOS imbroglio with its developers shock you as well? Talk Back and Let Me Know.

Topics: Software Development, Browser, Linux, Open Source, Operating Systems, Software


Jason Perlow, Sr. Technology Editor at ZDNet, is a technologist with over two decades of experience integrating large heterogeneous multi-vendor computing environments in Fortune 500 companies. Jason is currently a Partner Technology Strategist with Microsoft Corp. His expressed views do not necessarily represent those of his employer.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • Mother distro...

    This is the biggest issue, IMHO, with the whole Linux environment. There are too many distro that each want to do things a little different from each other. This is a major reason for its lack of acceptance in the business environment. Now we see there are too many "sub-distro's".

    On the brighter side, this is a testament to it resiliency, the fact that Mother Hen can disappear and the project stay strong. What would happen to Windows if Ballmer disappeared for 6 months... wait, that might actually be good for Windows. Never mind.
    • ???

      What in the world does this have to do with multiple distros. If anything this minimizes the problem because if CentOS went under there are more than users can switch to. And CentOS is in the server realm...a realm where I haven't seen any lack of acceptance in the business environment.
      • No sh#t!

        -Linux programs are not cross compatible between version, with some rare exceptions (specially where you must compile the program). RPM, DEB and so on.
        -A Linux expert can manage without any problem X distro but haven't a clue about other distro, specially "finding" where is it some program or configuration. For example, the home user folder can vary in different kind of version.

      • Jason specifically asked...

        about two separate Red Hat clones. This goes back to the same thing that drives multiple distro's. Each community wanting to do something a little different than the next.

        Did you read his whole post?
    • Ballmer's absence?

      Quote: [i]What would happen to Windows if Ballmer disappeared for 6 months...[/i]

      You mean do something like release 20,000 lines of source code??? [b]I bet chairs flew out the window when Ballmer found out about that one!!!![/b]

      If I were a fly on that wall......
  • re: CentOS

    Seems a tad early to state they won't get their sh*t together.

    There was a real problem (i.e. Davis) but the crew that's left over are working to address the problems he created. There's not enough publicized facts to determine if they just put a bandaid on it, or if they are truly addressing the past problems.

    I think the safest bet is to wait and watch and see what happens.

    If they are actually solving the past problems, they will need to be WAY more transparent about how CentOS is setup and run.

    We'll see......
  • RE: CentOS: Getting Their S#!t Together is a Top Priority

    CENTOS is just a clone of WhiteBox Linux so even if they did go away it wouldn't be a big deal.

    Reading the CENTOS site it looks like most things are resolved and the rest are going to be fixed fairly soon.
    • Incorrect

      CentOS is the successor to White Box Linux. White Box ceased to be an active project in December of 2007.

  • You don't follow the mailing list do you Perlow?

    If you did, I think some of your fears might be reduced and you would see the discussion between Scientific and CentOS about why they haven't been one in the past and what can be done in the future.

    Boy, worst case (which can't happen now) would have been is that they couldn't get their domain back. How does that compare in severity to say OEM's putting a rootkit in so that users can find them if lost.

    You like to be alarmist to make people feel like they have to read your columns.

    I think you are badmouthing people who do a lot of quality work. Ok they are not perfect but what have you contributed?
    • I don't have to follow their mailing list

      If I can talk to Karanbir Singh directly. Who has been in contact with me the entire time and will be a guest on Frugal Friday in September.
      • LOL pwned!!!

        Hallowed are the Ori
  • Shocking, but no need to merge with Scientific Linux

    I hope the devs will regard this as a powerful incentive to organize themselves more professionally. The amateurism certainly shocked me. However, there's no need for a merger with Scientific Linux, I think.

    That has to do with the way these two distro's are assembled. As you said, they are basically Red Hat Enterprise Linux with other logo's and with another package manager. Plus, I should add, a more modern version of Firefox and of Open Office thrown in.

    Their technical base comes from elsewhere; what technical advantages would a merger have?
  • Non-profit status yes, Merger no

    It seems incredible that they don't have any formal structure like a corporation or a 501c3. Under the current organization someone, probably Lance Davis, is incurring personal tax liability for the monies that they have received. The project needs to file for non-profit status ASAP.

    Merging with SL is a really bad idea. Having two completely separate RHEL clones provides redundancy. Either one of these projects can disappear overnight, having a second project guarantees that their users have an alternative. CentOS has just had a near death experience, Whitebox Linux was killed by an act of God (hurricane Katrina). SL could be killed by the act of a bureaucrat. As a CentOS user I get a great deal of comfort knowing that SL is available if CentOS disappears.
    • 501c3 would handle both issues

      A single 501c3 could take care of both CentOS and SciLinux, whether the two merge or not. I think a merger makes sense because even if a bureaucrat kills off SL (very unlikely) there will be a larger pool of developers to work on ONE project rather than two. It is more likely that Troy or his co-developer at SL could get sick or laid off or something like that -- in which case, you would need other developers to pick up the slack. Two distros, both without formalized nonprofits taking care of them doesn't make sense.
      • Combining SL an CentOS creates a single point of failure

        When designing fault tolerant systems you need to avoid single points of failure. Two completely independent systems are more reliable than a single system with some redundant components.

        We've seen an example of the vulnerability of CentOS. SL is subject to different pressures. I'll give you several scenarios that could kill SL. 1) The IT groups at FermiLab and Cern decide they like another distro better, and that distro is already freely available (Ubuntu, Debian, Slack). 2) They get a deal from Redhat, i.e. they pay a fixed price for support and they get the right to use as many copies of RHEL as the want. I'm sure they both have large contracts with Redhat already so this is a likely scenario. 3) Political pressure, i.e. someone at the Whitehouse asks the DOE why they are spending taxpayer's money to compete with private enterprise. 4) The people maintaining SL lose interest, this is the same threat that hangs over CentOS. SL isn't core to the mission of Fermilab and Cern, it's merely convenient, so it could be left to die at anytime. Likewise CentOS is maintained by volunteers who could lose interest or be forced by personal financial pressures to stop participating. Two independent projects have a much smaller likelihood of failing at the same time then a single project would. If a failure of one project does occur there will be time for someone to start replacement project without any disruption to the user base of the failed project.

        • Listen to my podcast with SciLinux


          SciLinux has very, very, very limited funding, and FermiLab/CERN has no support contract with RedHat. Fermilab may have money to power their accelerators but they have little funds for software licenses, if you listen to Troy he explains the situation. They've been using Redhat since the free version (Red Hat Linux 9) and the fedora transition and been using SciLinux ever since, they have a huge investment in it.

          It would behoove Fermi/CERN/SciLinux and CentOS to set up a 501c3 or an equivalent in Europe to protect both projects, whether they merge the code or not, they both cooperate on a high level with patches and fixes already.
  • If there is a major problem, move to another distro...

    Perhaps I'm just naively involved with Linux, but if there is
    a major problem, just move to another distro. Though I
    will be flamed for it, there aren't that many differences
    between them. Ubuntu, Redhat and Opensuse all basically
    run the same core technologies.

    About a year ago, I made the decision to move my lab
    workstations away from Mac OS X to Dell running
    OpenSuse. And then six months ago decided to move to
    Ubuntu. The initial move from OS X to Linux was
    extremely painful. We went through the process of finding
    Linux equivalents of our tools and automating our
    processes. Then we had to adjust to the new environment.

    But the move to Ubuntu was near painless. Sure, you have
    to tweak the desktop, but the tweaking is easier to do than
    in OpenSuse. All the underlying technologies are actually
    the same.

    Now that we are over the hurdle, everyone is rather happy.
    My development people are ecstatic with the move since
    versions of nearly all their tools can be found in the
    community repositories. And end users aren't really
    complaining. They still have their MS Office (through a
    virtual machine running Windows, exactly like it was on
    Mac OS X) and Thunderbird can be configured so that it
    acts almost exactly like Mail.app, iCal and Address book.
    Rob Oakes
    • Aint that easy

      Large organizations with hundreds of servers and workstations can't just "Move" to another distribution that easily. Especially if you have highly customized in-house apps like CERN/Fermilab does. For them to move to Debian or Ubuntu would be a huge hassle and require a significant level of effort to port it all over.
    • Yeah, right, easier said than done... in large scale.

      Yeah, right, easier said than done... in large scale. Try to change the distro of more than 100 desktops or tens of servers... with minimal downtime.

      I rely on CentOS to run too many servers... moving to another distro is not a cheap/easy option. An I guess many other sysadmins are in the same situation.



  • RE: CentOS: Getting Their S#!t Together is a Top Priority

    As long as its based on linux it will never get its act together. That is one of the main problems with using linux, too many people involved, no real structure. So anyone can take the code, modify it, do whatever they want, then someone else comes along and duplicates it and now you have wasted man time and a mish mash of spaghetti code.
    Loverock Davidson