Tech Broiler

Jason Perlow and Scott Raymond

CentOS: Getting Their S#!t Together is a Top Priority

By | August 2, 2009, 12:57pm PDT

Summary: The CentOS developers were ready to burn the place down if their AWOL project founder didn’t show up and resolve several outstanding issues that put the popular Open Source project and Linux distribution at risk. For those of us in the “Inside Baseball’ crowd in Open Source, a lot of armchair quarterbacking has been going on [...]

The CentOS developers were ready to burn the place down if their AWOL project founder didn’t show up and resolve several outstanding issues that put the popular Open Source project and Linux distribution at risk.

For those of us in the “Inside Baseball’ crowd in Open Source, a lot of armchair quarterbacking has been going on vis-a-vis the Open Letter that was published by the CentOS developers on July 30th which was addressed to Lance Davis, the founder and one of the lead developers of the CentOS project, one of the most-used free Linux distributions on the planet.

Click on the “Read the rest of this entry” link below for more.

As I wrote about in my round up of free Enterprise-class OSes, CentOS is a free source-level clone of Red Hat Enterprise Linux. Along with Scientific Linux, which is a separate distribution developed by Fermilab and CERN, these two distributions are the Linux of choice for environments wanting all the rock-solid stability of RHEL but don’t want or need Red Hat’s commercial technical support.

Also See: Frugal Friday, Interview with Troy Dawson of Scientific Linux

CentOS and Scientific Linux are built on the published GPLv2 source of Red Hat Enterprise Linux, are re-compiled without Red Hat’s trademark logos, and use the community YUM package manager for updates rather than Red Hat Network, which is a commercial, fee-based support update service. CentOS in particular is used by a number of appliance and virtual appliance vendors, such as Kickfire and Schooner Information Technology in order to produce turnkey systems for security, storage, web applications and databases.

In short, the CentOS developers were brought to the brink of rebellion and walking away from the popular project because its founder, Lance Davis, had disappeared in 2008, ceased contribution to the project, and was holding the keys to the centos.org domain and the project’s PayPal accounts. None of this was popular knowledge until the following letter was published on the CentOS project web site on July 31:

July 30, 2009 04:39 UTC

This is an Open Letter to Lance Davis from fellow CentOS Developers

It is regrettable that we are forced to send this letter but we are left with no other options. For some time now we have been attempting to resolve these problems:

You seem to have crawled into a hole … and this is not acceptable.

You have long promised a statement of CentOS project funds; to this date this has not appeared.

You hold sole control of the centos.org domain with no deputy; this is not proper.

You have, it seems, sole ‘Founders’ rights in the IRC channels with no deputy ; this is not proper.

When I (Russ) try to call the phone numbers for UK Linux, and for you individually, I get a telco intercept ‘Lines are temporarily busy’ for the last two weeks. Finally yesterday, a voicemail in your voice picked up, and I left a message urgently requesting a reply. Karanbir also reports calling and leaving messages without your reply.

Please do not kill CentOS through your fear of shared management of the project.

Clearly the project dies if all the developers walk away.

Please contact me, or any other signer of this letter at once, to arrange for the required information to keep the project alive at the ‘centos.org’ domain.

Sincerely,

Russ Herrold
Ralph Angenendt
Karanbir Singh
Jim Perrin
Donavan Nelson
Tim Verhoeven
Tru Huynh
Johnny Hughes

Over this weekend on August 1, Lance Davis put his tail between his legs, made a rare appearance and surrendered the centos.org domain to the project, and the project made a statement that other administrative issues were in the process of being resolved:

The CentOS Development team had a routine meeting today with Lance Davis in attendance. During the meeting a majority of issues were resolved immediately and a working agreement was reached with deadlines for remaining unresolved issues. There should be no impact to any CentOS users going forward.

# CentOS is not dead or going away. The signers of the Open Letter are fully committed to continue the CentOS Project. Updates and new releases will continue.

# Most of the Issues have been resolved, there is an action plan with agreed upon dates for any outstanding issues.

# The CentOS Project now owns the CentOS.org and CentOS.info domains and there will be no disruption in services.

# We thank the people who have stepped forward and want to donate to the CentOS project. We ask that you hold off for now until issues surrounding our new donation policy are put into place.

# The CentOS Project is run completely by volunteers and we are aware that this requires a different management style. We have been and continue to work to prevent issues like these from occurring in the future. We will continue this effort in the future, look for some new policy information soon.

So everything is being resolved, CentOS is getting its act together, and we can all go about our business. Nothing to see here, move along. Move along. Right?

Wrong.

Frankly, I am absolutely amazed that such a high-profile Open Source project could continue in operation without addressing who was in control of its financials  as well as its Internet presence. This particular incident raises a whole bunch of questions as to how large Open Source projects should be managed and who should be made accountable.

For starters, there needs to be clear policies of who is going to administer what, and that there should never, ever be a situation where one person holding the keys to the domain or any key administrative and financial accounts can just walk away and leave the developers holding their you-know-whats in their hands. This behavior is absolutely abysmal and Davis should be lucky not to walk out of this with civil charges pressed against him.

CentOS, if it isn’t already, should also immediately seek Not-For-Profit status in either the UK or the United States, where contributions made to it can be tax deductible. A formalization of the organization as a Not-For-Profit corporation (in the US, this is referred to as a 501(c)3 under the US tax codes) would require a number of bylaws as well as officers responsible for the welfare of the Corporation. This is exactly how the other two high profile community Linux distributions, Debian and Ubuntu operate in public, as do several of the BSD Unixes.

There is also the question of whether or not it makes sense for there to be two community-driven Red Hat clones. Scientific Linux, while under the stewardship of Fermilab and CERN, two very stable organizations, only has a small amount of developers working on it. It might behoove both Scientific and CentOS to join forces and pool their resources in producing a single “Mother Distribution” which both can be based on, and form a Not-for-Profit organization that can be made responsible for the non-technical administravia and financial aspects that will ensure their continuance.

Did the CentOS imbroglio with its developers shock you as well? Talk Back and Let Me Know.

Kick off your day with ZDNet's daily e-mail newsletter. It's the freshest tech news and opinion, served hot. Get it.

Topics

Jason Perlow, Sr. Technology Editor at ZDNet, is a technologist with over two decades of experience integrating large heterogeneous multi-vendor computing environments in Fortune 500 companies.

Disclosure

Jason Perlow

My Full-Time Employer is IBM. I write as a freelancer for ZDNet.

Disclaimer: The postings and opinions on this blog are my own and don't necessarily represent IBM's positions, strategies or opinions.

I own no investments or direct financial instruments in the companies I write about.

Biography

Jason Perlow

Jason Perlow, Sr. Technology Editor at ZDNet is a technologist with over two decades of experience with integrating large heterogeneous multi-vendor computing environments in Fortune 500 companies. A long-time computer enthusiast starting the age of 13 with his first Apple ][ personal computer, he began his freelance writing career starting at ZD Sm@rt Reseller in 1996 and has since authored numerous guest columns for ZDNet Enterprise and Ziff-Davis Internet. Jason was previously Senior Technology Editor for Linux Magazine, where he wrote about Open Source issues from 1999 to 2008.

In his spare time, Jason is an avid amateur chef and food writer, where his work reviewing New Jersey restaurants has appeared in The New York Times. He is also the founder of the popular food web site eGullet and blogs about restaurants and cooking at OffTheBroiler.com.

Related Discussions on TechRepublic

Did you know you can take part in these discussions with your ZDNet membership?
25
Comments

Join the conversation!

Just In

Strength in Fragmentation
Jeff Dickey Updated - 9th Aug 2009
There is a school of thought, historically deeply embedded within the
Linux "heavy thinker" crowd, that would argue that fragmentation is a
strength rather than a weakness.

1. If you're trying to solve a problem that's similar but not identical to
one already solved, take the existing solution, fork it, and adapt your
fork to your specific needs. The prevailing culture, combined with
popular understanding of the GPL v2, argues that you should publicly
release your changes so that the next guy can fork from what you
just did. "Standing on the shoulders of giants" and all that.

2. The old saw about "open source means never having to say 'it's
dead, Jim'" has enough truth to be relevant. If you're a large outfit like
CERN or Fermilab, you can take undead code (like Red Hat 9) and roll
your own internal support. If you're a smaller company, you almost
certainly are not the only one to have built around a no-longer-
supported-to-your-liking distro; the intersection of free software with
the open market ensures that somebody will pick up your old distro,
see that there's enough users to build a support business around, and
hang out their shingle on the mailing lists and Web fora.

I saw this sort of thing firsthand here in Singapore recently; a friend of
mine works in a small company that have numerous home-grown
apps running on top of Stampede Linux. (Remember that?) These
systems are running on a network that is completely internal to their
organization (no access from the raw Internet); the apps should
(and in prototype, are) portable to other distros, but they've got things
working to their liking as is with regular incremental improvements to
the app stack. These (so far) haven't required changes to the
underlying system stack beyond what their team is able to do for
themselves (and, since there's no external distribution of the as-built
system, GPL disclosures don't apply - they see this as a competitive
advantage). They're dealing with a system that is as close to absolutely
known as several years of refinement can make it; why change in the
absence of compelling business reasons?

I strongly suspect that these "consistent improvement, not continual
drastic change" shops are more common than generally realized -
precisely because they're submarines - self-contained, not visible to
the casual observer, and going quietly about their business without
the need (or desire) to make any noise while doing so.

There's also the point that, if you argue there's too much
fragmentation in Linux as it is, what would be "sufficient"
consolidation? Isn't that kind of thinking how business got into the
Windows monoculture to begin with... and hasn't that worked out
splendidly for all involved, with airtight security, rock-solid stability,
and customers firmly in control of their own business processes?

Right. I can hear you laughing from here.
0 Votes
+ -
Mother distro...
bjbrock 2nd Aug 2009
This is the biggest issue, IMHO, with the whole Linux environment. There are too many distro that each want to do things a little different from each other. This is a major reason for its lack of acceptance in the business environment. Now we see there are too many "sub-distro's".

On the brighter side, this is a testament to it resiliency, the fact that Mother Hen can disappear and the project stay strong. What would happen to Windows if Ballmer disappeared for 6 months... wait, that might actually be good for Windows. Never mind.
0 Votes
+ -
???
storm14k 2nd Aug 2009
What in the world does this have to do with multiple distros. If anything this minimizes the problem because if CentOS went under there are more than users can switch to. And CentOS is in the server realm...a realm where I haven't seen any lack of acceptance in the business environment.
0 Votes
+ -
No sh#t!
magallanes 3rd Aug 2009
-Linux programs are not cross compatible between version, with some rare exceptions (specially where you must compile the program). RPM, DEB and so on.
-A Linux expert can manage without any problem X distro but haven't a clue about other distro, specially "finding" where is it some program or configuration. For example, the home user folder can vary in different kind of version.


0 Votes
+ -
Jason specifically asked...
bjbrock 3rd Aug 2009
about two separate Red Hat clones. This goes back to the same thing that drives multiple distro's. Each community wanting to do something a little different than the next.

Did you read his whole post?
0 Votes
+ -
Ballmer's absence?
fatman65535 3rd Aug 2009
Quote: What would happen to Windows if Ballmer disappeared for 6 months...

You mean do something like release 20,000 lines of source code??? I bet chairs flew out the window when Ballmer found out about that one!!!!

If I were a fly on that wall......
0 Votes
+ -
re: CentOS
vonskippy 2nd Aug 2009
Seems a tad early to state they won't get their sh*t together.

There was a real problem (i.e. Davis) but the crew that's left over are working to address the problems he created. There's not enough publicized facts to determine if they just put a bandaid on it, or if they are truly addressing the past problems.

I think the safest bet is to wait and watch and see what happens.

If they are actually solving the past problems, they will need to be WAY more transparent about how CentOS is setup and run.

We'll see......
CENTOS is just a clone of WhiteBox Linux so even if they did go away it wouldn't be a big deal.

Reading the CENTOS site it looks like most things are resolved and the rest are going to be fixed fairly soon.
0 Votes
+ -
Contributr
Incorrect
jperlow 3rd Aug 2009
CentOS is the successor to White Box Linux. White Box ceased to be an active project in December of 2007.

http://www.whiteboxlinux.org/
0 Votes
+ -
If you did, I think some of your fears might be reduced and you would see the discussion between Scientific and CentOS about why they haven't been one in the past and what can be done in the future.

Boy, worst case (which can't happen now) would have been is that they couldn't get their domain back. How does that compare in severity to say OEM's putting a rootkit in so that users can find them if lost.

You like to be alarmist to make people feel like they have to read your columns.

I think you are badmouthing people who do a lot of quality work. Ok they are not perfect but what have you contributed?
0 Votes
+ -
Contributr
I don't have to follow their mailing list
jperlow Updated - 2nd Aug 2009
If I can talk to Karanbir Singh directly. Who has been in contact with me the entire time and will be a guest on Frugal Friday in September.
0 Votes
+ -
LOL pwned!!!
Hallowed are the Ori 3rd Aug 2009
Nice.
0 Votes
+ -
I hope the devs will regard this as a powerful incentive to organize themselves more professionally. The amateurism certainly shocked me. However, there's no need for a merger with Scientific Linux, I think.

That has to do with the way these two distro's are assembled. As you said, they are basically Red Hat Enterprise Linux with other logo's and with another package manager. Plus, I should add, a more modern version of Firefox and of Open Office thrown in.

Their technical base comes from elsewhere; what technical advantages would a merger have?
0 Votes
+ -
Non-profit status yes, Merger no
bjrosen@... 3rd Aug 2009
It seems incredible that they don't have any formal structure like a corporation or a 501c3. Under the current organization someone, probably Lance Davis, is incurring personal tax liability for the monies that they have received. The project needs to file for non-profit status ASAP.

Merging with SL is a really bad idea. Having two completely separate RHEL clones provides redundancy. Either one of these projects can disappear overnight, having a second project guarantees that their users have an alternative. CentOS has just had a near death experience, Whitebox Linux was killed by an act of God (hurricane Katrina). SL could be killed by the act of a bureaucrat. As a CentOS user I get a great deal of comfort knowing that SL is available if CentOS disappears.
0 Votes
+ -
Contributr
501c3 would handle both issues
jperlow 3rd Aug 2009
A single 501c3 could take care of both CentOS and SciLinux, whether the two merge or not. I think a merger makes sense because even if a bureaucrat kills off SL (very unlikely) there will be a larger pool of developers to work on ONE project rather than two. It is more likely that Troy or his co-developer at SL could get sick or laid off or something like that -- in which case, you would need other developers to pick up the slack. Two distros, both without formalized nonprofits taking care of them doesn't make sense.
When designing fault tolerant systems you need to avoid single points of failure. Two completely independent systems are more reliable than a single system with some redundant components.

We've seen an example of the vulnerability of CentOS. SL is subject to different pressures. I'll give you several scenarios that could kill SL. 1) The IT groups at FermiLab and Cern decide they like another distro better, and that distro is already freely available (Ubuntu, Debian, Slack). 2) They get a deal from Redhat, i.e. they pay a fixed price for support and they get the right to use as many copies of RHEL as the want. I'm sure they both have large contracts with Redhat already so this is a likely scenario. 3) Political pressure, i.e. someone at the Whitehouse asks the DOE why they are spending taxpayer's money to compete with private enterprise. 4) The people maintaining SL lose interest, this is the same threat that hangs over CentOS. SL isn't core to the mission of Fermilab and Cern, it's merely convenient, so it could be left to die at anytime. Likewise CentOS is maintained by volunteers who could lose interest or be forced by personal financial pressures to stop participating. Two independent projects have a much smaller likelihood of failing at the same time then a single project would. If a failure of one project does occur there will be time for someone to start replacement project without any disruption to the user base of the failed project.



0 Votes
+ -
Contributr
Listen to my podcast with SciLinux
jperlow Updated - 3rd Aug 2009
http://blogs.zdnet.com/perlow/?p=10669

SciLinux has very, very, very limited funding, and FermiLab/CERN has no support contract with RedHat. Fermilab may have money to power their accelerators but they have little funds for software licenses, if you listen to Troy he explains the situation. They've been using Redhat since the free version (Red Hat Linux 9) and the fedora transition and been using SciLinux ever since, they have a huge investment in it.

It would behoove Fermi/CERN/SciLinux and CentOS to set up a 501c3 or an equivalent in Europe to protect both projects, whether they merge the code or not, they both cooperate on a high level with patches and fixes already.
Perhaps I'm just naively involved with Linux, but if there is
a major problem, just move to another distro. Though I
will be flamed for it, there aren't that many differences
between them. Ubuntu, Redhat and Opensuse all basically
run the same core technologies.

About a year ago, I made the decision to move my lab
workstations away from Mac OS X to Dell running
OpenSuse. And then six months ago decided to move to
Ubuntu. The initial move from OS X to Linux was
extremely painful. We went through the process of finding
Linux equivalents of our tools and automating our
processes. Then we had to adjust to the new environment.

But the move to Ubuntu was near painless. Sure, you have
to tweak the desktop, but the tweaking is easier to do than
in OpenSuse. All the underlying technologies are actually
the same.

Now that we are over the hurdle, everyone is rather happy.
My development people are ecstatic with the move since
versions of nearly all their tools can be found in the
community repositories. And end users aren't really
complaining. They still have their MS Office (through a
virtual machine running Windows, exactly like it was on
Mac OS X) and Thunderbird can be configured so that it
acts almost exactly like Mail.app, iCal and Address book.
0 Votes
+ -
Contributr
Aint that easy
jperlow 3rd Aug 2009
Large organizations with hundreds of servers and workstations can't just "Move" to another distribution that easily. Especially if you have highly customized in-house apps like CERN/Fermilab does. For them to move to Debian or Ubuntu would be a huge hassle and require a significant level of effort to port it all over.

Yeah, right, easier said than done... in large scale. Try to change the distro of more than 100 desktops or tens of servers... with minimal downtime.

I rely on CentOS to run too many servers... moving to another distro is not a cheap/easy option. An I guess many other sysadmins are in the same situation.


Regards,

MV

0 Votes
+ -
As long as its based on linux it will never get its act together. That is one of the main problems with using linux, too many people involved, no real structure. So anyone can take the code, modify it, do whatever they want, then someone else comes along and duplicates it and now you have wasted man time and a mish mash of spaghetti code.
There are too many Linux distros anyway, it fragments the market way too much, so I say let CENTOS die and just join one of the other distros efforts instead of starting yet another one!
I agree, why not combine instead of fragment. I look at all the distros and its almost a joke the way this could happen to an OS trying to be something worth getting.
"Frankly, I am absolutely amazed that such a high-
profile Open Source project could continue in
operation without addressing who was in control of
its financials as well as its Internet presence."

Ha! I'm not. This has been my biggest issue with
Open Source over the years. You hook in to a good
product with strong developer involvement and then
all of a sudden it forks and changes what you liked
in the first place or dies is miserable and fast
death. Time and time again this happens to Open
Source projects which makes Open Source most often
unreliable at best, period.
People talk about CentOS being for companies who don't want Red Hat support. I think this is very misleading, and hides a more significant "gap" in "people doing the right thing".

Red Hat provides much more than just support under their subscription. They certify software and hardware to work properly. Yes, CentOS does not get those certifications, but they do get the "draft" and users choose CentOS because it is likely to work well since it is based on Red Hat's work. They can use Red Hat Documentation, since the operation of the software should be similar. They rely on Red Hat to make the patches and then get them from CentOS.

Don't get me wrong, I am not Anti-CentOS, but I think the users of CentOS who say they just don't want the support are avoiding the issue. They just don't want to pay for the value that Red Hat provides, shame on them!!
0 Votes
+ -
Strength in Fragmentation
Jeff Dickey Updated - 9th Aug 2009
There is a school of thought, historically deeply embedded within the
Linux "heavy thinker" crowd, that would argue that fragmentation is a
strength rather than a weakness.

1. If you're trying to solve a problem that's similar but not identical to
one already solved, take the existing solution, fork it, and adapt your
fork to your specific needs. The prevailing culture, combined with
popular understanding of the GPL v2, argues that you should publicly
release your changes so that the next guy can fork from what you
just did. "Standing on the shoulders of giants" and all that.

2. The old saw about "open source means never having to say 'it's
dead, Jim'" has enough truth to be relevant. If you're a large outfit like
CERN or Fermilab, you can take undead code (like Red Hat 9) and roll
your own internal support. If you're a smaller company, you almost
certainly are not the only one to have built around a no-longer-
supported-to-your-liking distro; the intersection of free software with
the open market ensures that somebody will pick up your old distro,
see that there's enough users to build a support business around, and
hang out their shingle on the mailing lists and Web fora.

I saw this sort of thing firsthand here in Singapore recently; a friend of
mine works in a small company that have numerous home-grown
apps running on top of Stampede Linux. (Remember that?) These
systems are running on a network that is completely internal to their
organization (no access from the raw Internet); the apps should
(and in prototype, are) portable to other distros, but they've got things
working to their liking as is with regular incremental improvements to
the app stack. These (so far) haven't required changes to the
underlying system stack beyond what their team is able to do for
themselves (and, since there's no external distribution of the as-built
system, GPL disclosures don't apply - they see this as a competitive
advantage). They're dealing with a system that is as close to absolutely
known as several years of refinement can make it; why change in the
absence of compelling business reasons?

I strongly suspect that these "consistent improvement, not continual
drastic change" shops are more common than generally realized -
precisely because they're submarines - self-contained, not visible to
the casual observer, and going quietly about their business without
the need (or desire) to make any noise while doing so.

There's also the point that, if you argue there's too much
fragmentation in Linux as it is, what would be "sufficient"
consolidation? Isn't that kind of thinking how business got into the
Windows monoculture to begin with... and hasn't that worked out
splendidly for all involved, with airtight security, rock-solid stability,
and customers firmly in control of their own business processes?

Right. I can hear you laughing from here.

Join the conversation!

Formatting +
BB Codes - Note: HTML is not supported in forums
  • [b] Bold [/b]
  • [i] Italic [/i]
  • [u] Underline [/u]
  • [s] Strikethrough [/s]
  • [q] "Quote" [/q]
  • [ol][*] 1. Ordered List [/ol]
  • [ul][*] · Unordered List [/ul]
  • [pre] Preformat [/pre]
  • [quote] "Blockquote" [/quote]
ie8 fix

The best of ZDNet, delivered

ZDNet Newsletters

Get the best of ZDNet delivered straight to your inbox

Facebook Activity

White Papers, Webcasts, & Resources
ie8 fix