A question of leadership

Old habits die hard, it seems.

My very first ZDNet column was about the pain being experienced by users upgrading to Red Hat 5.0, about a year and a half ago.

Version 5 featured a major upgrade of Linux's libc, which is easily thethird most important and complex software component of a Linux system (afterthe kernel and "C" language compiler).

(And, yes, it's still a "Linux system," despite RichardStallman's continuing moans to the contrary. His recently-stated points are nothingnew, and neither are theexplanations about why the term "GNU/Linux" is just plain dumb.)

The upgrade of libc in Red Hat 5.0 was a major overhaul of the library,going to a GNU-developed codebase more commonly referred to asglibc. Most importantly at the time, the upgrade brokecompatibility with previous editions, meaning that software developed onthat new version of Red Hat wouldn't run on earlier versions of Red Hat. But it also wouldn't run on other Linux distributions that hadn't yet taken the plunge into glibc. The result was a mess that lastedsome months until the other distributions caught up.

I fear that with the recent release of Red Hat 7 we may be back inthis mess again.

Lucky seven?
The Red Hat 7 package is certainly impressive, and comes with the firstnon-WordPerfect keyboard template I've seen in a long time. But what'smost significant -- for the entire user and developer communities -- isRed Hat 7's default use of new versions of the RPM package manager, thegcc C compiler, and the aforementioned glibc.

The compiler is so new that it can't be used to compile the kernel on RedHat 7 boxes -- you use a different (older) compiler, renamed kgcc,to do that. The use of this new compiler, and the new RPM(version 4) and new glibc (2.1.92), means that:

• Software compiled against the libc used by Red Hat 7 won't work on anyother current Linux distribution, or on earlier versions of Red Hat.
• Software packaged on a Red Hat 7 system can't be installed on any earlier versions of Red Hat, or on other distributions that haven'tupgraded to RPM release 4.
• Software compiled with the gcc compiler now default on Red Hat 7will be incompatible with code produced by the current "standard"version of gcc used by most other distributions and by older versions ofRed Hat.

Past and future at odds
On the surface, this is fair game. According to Erik Troan, vice president ofproduct development at Red Hat, the company was caught between the need toinnovate, meet deadlines, and provide a stable platform. "We were in a bind,"Troan said. "We plan our releases well in advance, and we've deliveredreleases at regular intervals for many years." He added that a numberof new glibc features, such as support for 16-bit languages and multiplehardware platforms, justified breaking some compatibility. "With RedHat 7, we decided it was the righttime to hurt forward compatibility in order to move to the nextgeneration."

Fair enough. The company has gone to admirable pains to ensure that backwardscompatibility (the ability to run older software and packages oncurrent releases of Red Hat) has been maintained.Still, the Linux community takes Microsoft to task for its breaking offorwards compatibility in file formats as a way to strong-arm users toupgrade, and this situation is similar in some ways. The Red Hat upgradeisn't done primarily as a money maker as a Microsoft Office one is -- allthis new Red Hat software is, after all, open source. Still there arecertainly common frustrations. Users of perfectly good Linux systems willhave to upgrade to get new Red Hat packages, unless they want tore-compile and re-assemble the code themselves.

Troan believes that Red Hat can use its position of leadershipto help drive users into using more-advanced software.In the case of the move to glibc in Red Hat 5, Red Hat used a versionof glibc (2.0.6) that wasn't a fully-cooked production release."We didn't feel that the software would ever stabilize unless mainstreamdistributions used glibc and the benefits glibc 2.0 provided," hesaid.

Following Red Hat's lead, the other distributions eventually caught up,but the transitional period was especially painful for non-Red Hatusers and software developers wanting to create software installable onall Linux systems.

Unfortunately, I suspect that the same mistakes are about to berepeated.

Version (out of) control
Red Hat 7's version of glibc hasn't been cleared for production use by theglibc maintainers at the GNU project. The GNU web page onglibc insists its most current stable release is 2.1.2 -- older thanthe version already running in Red Hat's 6.2 package.

The fact that the GNU page doesn't appear to have been updated in a yearmakes one wonder whether it's the definitive source of glibc progressreports and stability information. Release information for glibc is allover the place. The GNUFTP site doesn't go past 2.1.91. Debian's "stable" version is 2.1.3and its "unstable" one is at 2.1.94. And the freshmeatlisting for glibc says 2.1.3 for production version and nothing for thedevelopment version.

In other words, glibc's maintainers haven't expressed confidence thatanything newer than 2.1.3 will be stable, and you can't even findanything newer than 2.1.91 on the official glibc website. Yet somethingnewer than both is exactly what's in today's shipping release of Red Hat.

The base code for 2.1.92 (and newer) releases of glibc are currentlyfound only atthe website ofCygnus, the influential development shop that's now part of Red Hat.Coincidentally (or maybe not), the main developer of glibc, Ulrich Drepper, has acygnus.com email address. Drepper's glibc FAQ istotally different from the one on the GNUsite that says the most recent stable version of glibc is 2.0.5c (!).To make matters worse, non-techie documentation on glibc is abysmal.While every patch and pre-patch for the kernel is well documented atsites such asKernel Notes, to see what'schanged in glibc you actually need to obtain and dissect its package,examining a file called "ChangeLog." (And good luck comprehending theChangeLog if you're not a programmer.)

The situation with the gcc C compiler is almost as strange. The version inRed Hat is 2.96 which, according the to the GNU maintainers, does not exist.Indeed, the GNU gcc maintainers -- far more up-to-date than their glibccounterparts -- recentlyissued awarning about the gcc used in Red Hat 7.

Leadership or gamesmanship?
The inevitable question is, then: who's in control of glibc and gcc?The GNU Project or Cygnus? TheRed Hat page on glibcstill says it's GNU. Yet the GNU website information is ancient andits release recommendations are ignored.

And RPM, the package management system Red Hat invented that's widely usedon other distributions, is also not immune from such games. While Red Hat7 boasts the new (and also not forwards-compatible) release 4 of RPM, the(Red Hat maintained) RPM community sitesays the newest release is still only 3.0.4. RPM 4.0 is hidden away on theRPM website and there's no announcement that it's available, let alone readyfor production use. It'll take weeks, maybe months, for the otherRPM-using distributions to return to the same level.

Maybe this is also what Red Hat considers leadership at work, filling avoid left where GNU appears to be dropping the ball on its maintenance ofglibc, making its own non-standard changes to gcc, and silentlysynchronizing RPM development with that of its own distribution. Thefallout experienced by users of other distributions, or by softwaredevelopers wanting to use Red Hat to develop distribution-neutralproducts, is apparently irrelevant to the company.

.Such attitude is fair game for a competitive player in a marketplace ofLinux vendors. However, I submit that Red Hat's actions, which needlessly throw partsof the community into disarray for a significant period oftime, work against the company's long-term goals of portraying Linux as anattractive environment and a cohesive community. Furthermore, theinevitable scramble to catch up with Red Hat's new versions, or at least becompatible with them, diverts community resources and energy from the coretask of making Linux -- all Linux, not just not just Red Hat's -- as goodas it can be.

The cynical, including those who are wary of such "leadership,"might wonder whether Red Hat is leveraging its ownership of Cygnus andits control of RPM development to forever have a leg up on other distributionson such core components as gcc, glibc and RPM. They might ask whether suchactions indicate a desire by Red Hat to enhance its stature at theexpense of other players in the Linux community, consequences be damned.It's not a critical problem, so long as all the code in question is opensource. But it's still a serious nuisance that most certainly exacts atoll on the community at large, and it has the potential to cause realharm to newcomers not aware of these games.

In any case, there's got to be a better way to evolve the Linux world intobetter technologies than to use development-stage releases in productiondistributions. I vividly remember the problems that happened last time,and I hope the community will get through this transition with less griefthis time around.

Errata

In last week's column, posted October 4, I incorrectly made anunjustified analysis based on the value of Apple stock. This commentdid not consider a split of Apple stock that occurred in June 2000. Thecolumn was modified to remove this reference, and I believe theother points made in the column are still valid given the IDC report andother circumstances. However, I am truly sorry for the oversight and anyinconvenience or upset this may have caused.

At the very least, perhaps it might be best to stick with the common wisdomthat I've heard from many places: wait to deploy Red Hat until it comesout with 7.1. If we're very lucky, all this mess will have sorted itselfout by then.

Are you willing to upgrade to Red Hat 7, complete with all itscompatibility issues? Tell Evan in the TalkBack below or in the ZDNet Linux Forum. Or write to Evandirectly at evan@starnix.com.