Tech
The joys and perils of open-source life
The author of an important development tool for Linux programmers, writes about his project (the LTT) and complex GPL copyright issues.
In light of recent events on the tracing tools side of Linux, some reflections came to mind which I would like to share with the Linux community. It is my hope that my perspective and experience as a maintainer of an open-source project called the Linux Trace Toolkit (LTT) and as a consultant in the open-source field may be helpful to others in understanding how the open-source movement is evolving. After all, this is what I specialize in: helping people understand complex systems.
The origins of LTT
I started developing the Linux Trace Toolkit in the beginning of 1999 as part of my research at the Ecole Polytechnique de Montreal ("Poly"). I had one purpose: to provide a way for users to understand the dynamic behavior of a complex operating system -- in this case, Linux. From the start, I was convinced that numbers and lists of items were insufficient to described such systems. Instead, what was needed was a way to visually illustrate the system's behavior.
As the old adage says, a picture is worth a thousand words. Many ideas came to mind, but right from the start the one that seemed to make the most sense was a control-graph similar to the one currently displayed by LTT. Hence, I started out with a very clear idea of what I expected of this tool once it was running.
LTT takes form
Thus began a six month period of experiment after experiment, trying to probe Linux's behavior. With some guidance from Professor Michel Dagenais at Poly, I began to make rapid progress. By March, I had a working prototype that enabled me to trace timer interrupts, buffer these events in a data log, retrieve them using a daemon, and print them off-line.
By May, I had perfected the technique while adding to it more events, all the while becoming more and more familiar with the sources ("use the source, Luke!") and continually gaining a better understanding of the flow of control within the kernel. By then, the entries were marked "Linux trace utility" in my log book. Meanwhile, the event format had changed many times over, as trace logs were getting very large and I was looking for ways to minimize their size.
By the end of May, I was quite satisfied with the quantity of events I could retrieve; so I started working on a graphical front end to it all. By mid-June, the whole interface was built, and I was quite satisfied with how the graphs displayed and how the information was laid out.
From there on, I spent a month debugging things and testing the tool as much as I could in various situations. I remember spending a lot of time on the routines that manage the display of the graphs. While it may be obvious from reading a list of events to find out where the system is, whether in a process or in the kernel, it is quite another story to try to write down the rules that govern this behavior in a programming language.
First LTT release -- and first reactions
Finally, the first LTT release came out on July 22, 1999 (with intent to come out on the 21st). After that, I was off for some fresh air.
The first reaction to come in was from Peter Wurmsdobler: "It seems to be . . . a well appropriate tool to . . . give an understanding for the operating system . . ." This was closely followed by "It was with great interest that we recently read about your Linux Trace Toolkit, as it provides some good capability that was not previously available for Linux," from David Beal, of Zentropix (later acquired by Lineo). Later, came "Looks very interesting, particularly for debugging interactions among groups of daemons and/or the kernel," from Werner Almesberger (LILO maintainer and kernel contributor).
From the feedback I was receiving, it was clear that my initial purpose was fulfilled. LTT was providing a unique capability enabling users to understand the behavior of a complex operating system. It was now time to sharpen the knife.
In the following months, advice and support came in from various sources.
Jacques Gelinas, author of Linuxconf, suggested I use macros for the kernel instrumentation, as it would enable conditional compilation of trace statements in the kernel. Jean-Hughes Deschenes helped in refining the graphical interface by providing a toolbar, event icons, selectable events, and other very useful visual enhancements. Andi Kleen suggested some better ways to go up the call stack when retrieving system call address origins.
In parallel, I pushed other things. First, I completed instrumentation of the kernel. Then, I did a complete rewrite of the event database browsing engine. (It used to read the whole trace into custom structures and then browse through those traces to display trace information. Now it maps the trace into memory and reads as it is.)
In effect, this was almost a complete rewrite of the visualization tool, because major portions had to be modified. The event structures changed all the while. Later, I added keyboard support to the graphical tool, since browsing through the large traces armed only with a mouse proved to be a challenging affair.
Gaining altitude
By April 2000, things were starting to take a whole new turn. I did a first presentation about LTT at the Linux Expo in Montreal in front of a very interested crowd, and had a presentation scheduled for the refereed track of the Usenix Annual Technical Conference for June in San Diego. Meanwhile, I had started receiving requests from various Linux vendors wanting to add features to LTT. At that point, I had to quickly decide on the strategy I would adopt in receiving help, either technical or financial, from commercial entities.
Just at that time, I was fortunate to run into Richard Stallman at Linux Expo, and Richard was kind enough to spend some time with me discussing the issues of open-source, free software, copyrights, the GPL, etc. These discussions were extremely valuable in giving me access to Richard's 16 years of GPL experience -- a source of wisdom which it would have been foolish to overlook. Richard gave me a breakdown of the do's and don'ts of managing GPL source with outside contributors, the reasons for these to exist, and the possible consequences of not taking the right precautions. It was serious food for thought!
A sane protocol
Following these discussions and many reflections, I arrived at what I consider a "sane protocol" for managing outside contributions: any source code contributed to LTT is owned (copyrighted) by it's author, regardless of whether the author is an employee of a corporation or is sponsored by a corporation.
What does this mean? Well, it simply means that if John Doe works for company X and he writes code which is meant to be included in LTT, then the copyright is attributed to John Doe and NOT company X. Which does not mean company X cannot receive credit for having helped in developing LTT.
The reason for this is very simple. Whereas company X might be well intentioned at the time of the contribution, I, as the project maintainer, have no guaranty as to future intent. Therefore, I would do a great disservice to LTT and to the open-source community at large by allowing the company to own the copyright of the contribution.
Moreover, take the situation where John Doe leaves company X and starts working for company Y. Does he loose control of the source he actually wrote and that he actually understands better than anyone else? Again, as a project maintainer, this is just a risk I cannot afford to take.
I put my money where my mouth is. Specifically, you will find that all of the code I wrote in LTT is copyrighted in my personal name and not in the name of my company (Opersys Inc.), although I could have easily done otherwise.
Needless to say, this protocol has raised eyebrows and has met opposition. Nonetheless, I believe that in the long run it makes the most sense. And recent events have reinforced this belief. Let me explain.
With the rise of popularity of Linux in both mainstream and embedded computing, many companies have moved to capitalize (financially) on Linux. First, let me be clear: there is nothing inherently wrong with doing this.
In the beginning, the corporate messages tended to be: "It's free, and we offer packaging and support." As time went by, this evolved into variants like: "It's free, but we've added some very cool features to it." Again, such an attitude is not inherently wrong; there can be good reasons for taking that approach.
The problem, however, comes when a partisan philosophy, burdened by corporate agendas, starts to wind it's way into the very projects that made the free software market emerge. That's when things start to become very slippery. Project management then becomes a difficult compromise between accepting "cool features" and ensuring that the project remains on course and free from corporate agendas. In this regard, my motto is "don't shoot yourself in the foot."
A satanic belief in the Devil
Having said that, I tend to agree with Messadie when he says: "It is my belief that it is profoundly satanic to believe in the Devil."
What do I mean by that? Simply that there are no bad guys and no good guys. That, by the way, is why I didn't call this editorial "Misleading announcements about Linux tracing tools." Reality often tends to be complex -- be it fortunately or unfortunately.
After all, every company in this market wants to survive. Need I remind anyone of the SEC fillings of some Linux companies which explicitly state that there are no known open-source companies that make a profit? To some, this means acquiring legitimacy by promoting open-source projects; to others, it means differentiating their company's offerings by "adding value."
Whatever it might mean to the parties involved, let no one forget how the Linux market got where it is today, and the impact that free software is having on the computer industry and the world. While we would all like to make a living out of this, don't forget the fundamental significance of the source code being freed (and this is where the "raison d'être" of the GPL comes in) is that it becomes the basis of further development as were Greek theorems, as was the railway system, and as is the Internet.
From this perspective, it is easy to see that closed code is, most likely, dead code.
My goal with the LTT, as tiny as it may seem against such a grand panorama, is to provide tools that assist both the novice and the expert to gain a better understanding of the overall interactions between and within the systems being analyzed. Make no mistake, tracing tools have been around before, and more are coming. But LTT is the only one that has been developed under the GPL.
As described above, LTT has progressed at a phenomenal rate, in a very short time -- AND with lots of outside help. It has been said that LTT has already surpassed many available tracing tools. This is confirmed by the large number of Fortune 500 companies that currently use LTT to develop Linux based applications. In fact, many say they are planning to use Linux in their systems precisely because such a tracing tool exists for Linux.
Since last April, RTAI tracing has been added to LTT thanks to funding by Lineo. This is a big step forward, as it enables real-time Linux developers to debug their systems using LTT. Lately, PowerPC support came in from several directions. I had been working on a port with funding from Lineo when I received a patch from Bao Gang Liu of Agilent China, followed closely by code from Andy Lowe of MontaVista. Just today, I received word from Maxwell Patten of Nortel Networks that he too has been working on a PowerPC port of LTT. And it isn't over yet . . . !
In the future, LTT will aim to be available for additional architectures that Linux is running on. Also, there are requests for LTT to be run on additional open-source operating systems, such as BSD and HURD. Along with these system-level issues, there are numerous planned tool enhancements and flexibility additions.
Feel free to contribute to this project, either by providing feedback, expertise, or source code. Most important of all, try to keep an eye on the big picture. Try to see how each piece fits with the others, and how they all interact together. After all, that's what LTT is all about.
About the author: Karim Yaghmour is the author and maintainer of the Linux Trace Toolkit. He is the founder of Opersys Inc., which provides expertise on the Linux kernel and it's real-time derivatives.
Related stories:
Will the real "first PPC LTT" please stand up?
First Linux Trace Toolkit for PowerPC
Event trace tool doesn't require instrumented libraries
Linux Trace Toolkit supports real-time Linux
Talkback! Do you have questions or comments on this article? talkback here
The origins of LTT
I started developing the Linux Trace Toolkit in the beginning of 1999 as part of my research at the Ecole Polytechnique de Montreal ("Poly"). I had one purpose: to provide a way for users to understand the dynamic behavior of a complex operating system -- in this case, Linux. From the start, I was convinced that numbers and lists of items were insufficient to described such systems. Instead, what was needed was a way to visually illustrate the system's behavior.
As the old adage says, a picture is worth a thousand words. Many ideas came to mind, but right from the start the one that seemed to make the most sense was a control-graph similar to the one currently displayed by LTT. Hence, I started out with a very clear idea of what I expected of this tool once it was running.
LTT takes form
Thus began a six month period of experiment after experiment, trying to probe Linux's behavior. With some guidance from Professor Michel Dagenais at Poly, I began to make rapid progress. By March, I had a working prototype that enabled me to trace timer interrupts, buffer these events in a data log, retrieve them using a daemon, and print them off-line.
By May, I had perfected the technique while adding to it more events, all the while becoming more and more familiar with the sources ("use the source, Luke!") and continually gaining a better understanding of the flow of control within the kernel. By then, the entries were marked "Linux trace utility" in my log book. Meanwhile, the event format had changed many times over, as trace logs were getting very large and I was looking for ways to minimize their size.
By the end of May, I was quite satisfied with the quantity of events I could retrieve; so I started working on a graphical front end to it all. By mid-June, the whole interface was built, and I was quite satisfied with how the graphs displayed and how the information was laid out.
From there on, I spent a month debugging things and testing the tool as much as I could in various situations. I remember spending a lot of time on the routines that manage the display of the graphs. While it may be obvious from reading a list of events to find out where the system is, whether in a process or in the kernel, it is quite another story to try to write down the rules that govern this behavior in a programming language.
First LTT release -- and first reactions
Finally, the first LTT release came out on July 22, 1999 (with intent to come out on the 21st). After that, I was off for some fresh air.
The first reaction to come in was from Peter Wurmsdobler: "It seems to be . . . a well appropriate tool to . . . give an understanding for the operating system . . ." This was closely followed by "It was with great interest that we recently read about your Linux Trace Toolkit, as it provides some good capability that was not previously available for Linux," from David Beal, of Zentropix (later acquired by Lineo). Later, came "Looks very interesting, particularly for debugging interactions among groups of daemons and/or the kernel," from Werner Almesberger (LILO maintainer and kernel contributor).
From the feedback I was receiving, it was clear that my initial purpose was fulfilled. LTT was providing a unique capability enabling users to understand the behavior of a complex operating system. It was now time to sharpen the knife.
In the following months, advice and support came in from various sources.
Jacques Gelinas, author of Linuxconf, suggested I use macros for the kernel instrumentation, as it would enable conditional compilation of trace statements in the kernel. Jean-Hughes Deschenes helped in refining the graphical interface by providing a toolbar, event icons, selectable events, and other very useful visual enhancements. Andi Kleen suggested some better ways to go up the call stack when retrieving system call address origins.
In parallel, I pushed other things. First, I completed instrumentation of the kernel. Then, I did a complete rewrite of the event database browsing engine. (It used to read the whole trace into custom structures and then browse through those traces to display trace information. Now it maps the trace into memory and reads as it is.)
In effect, this was almost a complete rewrite of the visualization tool, because major portions had to be modified. The event structures changed all the while. Later, I added keyboard support to the graphical tool, since browsing through the large traces armed only with a mouse proved to be a challenging affair.
Gaining altitude
By April 2000, things were starting to take a whole new turn. I did a first presentation about LTT at the Linux Expo in Montreal in front of a very interested crowd, and had a presentation scheduled for the refereed track of the Usenix Annual Technical Conference for June in San Diego. Meanwhile, I had started receiving requests from various Linux vendors wanting to add features to LTT. At that point, I had to quickly decide on the strategy I would adopt in receiving help, either technical or financial, from commercial entities.
Just at that time, I was fortunate to run into Richard Stallman at Linux Expo, and Richard was kind enough to spend some time with me discussing the issues of open-source, free software, copyrights, the GPL, etc. These discussions were extremely valuable in giving me access to Richard's 16 years of GPL experience -- a source of wisdom which it would have been foolish to overlook. Richard gave me a breakdown of the do's and don'ts of managing GPL source with outside contributors, the reasons for these to exist, and the possible consequences of not taking the right precautions. It was serious food for thought!
A sane protocol
Following these discussions and many reflections, I arrived at what I consider a "sane protocol" for managing outside contributions: any source code contributed to LTT is owned (copyrighted) by it's author, regardless of whether the author is an employee of a corporation or is sponsored by a corporation.
What does this mean? Well, it simply means that if John Doe works for company X and he writes code which is meant to be included in LTT, then the copyright is attributed to John Doe and NOT company X. Which does not mean company X cannot receive credit for having helped in developing LTT.
The reason for this is very simple. Whereas company X might be well intentioned at the time of the contribution, I, as the project maintainer, have no guaranty as to future intent. Therefore, I would do a great disservice to LTT and to the open-source community at large by allowing the company to own the copyright of the contribution.
Moreover, take the situation where John Doe leaves company X and starts working for company Y. Does he loose control of the source he actually wrote and that he actually understands better than anyone else? Again, as a project maintainer, this is just a risk I cannot afford to take.
I put my money where my mouth is. Specifically, you will find that all of the code I wrote in LTT is copyrighted in my personal name and not in the name of my company (Opersys Inc.), although I could have easily done otherwise.
Needless to say, this protocol has raised eyebrows and has met opposition. Nonetheless, I believe that in the long run it makes the most sense. And recent events have reinforced this belief. Let me explain.
With the rise of popularity of Linux in both mainstream and embedded computing, many companies have moved to capitalize (financially) on Linux. First, let me be clear: there is nothing inherently wrong with doing this.
In the beginning, the corporate messages tended to be: "It's free, and we offer packaging and support." As time went by, this evolved into variants like: "It's free, but we've added some very cool features to it." Again, such an attitude is not inherently wrong; there can be good reasons for taking that approach.
The problem, however, comes when a partisan philosophy, burdened by corporate agendas, starts to wind it's way into the very projects that made the free software market emerge. That's when things start to become very slippery. Project management then becomes a difficult compromise between accepting "cool features" and ensuring that the project remains on course and free from corporate agendas. In this regard, my motto is "don't shoot yourself in the foot."
A satanic belief in the Devil
Having said that, I tend to agree with Messadie when he says: "It is my belief that it is profoundly satanic to believe in the Devil."
What do I mean by that? Simply that there are no bad guys and no good guys. That, by the way, is why I didn't call this editorial "Misleading announcements about Linux tracing tools." Reality often tends to be complex -- be it fortunately or unfortunately.
After all, every company in this market wants to survive. Need I remind anyone of the SEC fillings of some Linux companies which explicitly state that there are no known open-source companies that make a profit? To some, this means acquiring legitimacy by promoting open-source projects; to others, it means differentiating their company's offerings by "adding value."
Whatever it might mean to the parties involved, let no one forget how the Linux market got where it is today, and the impact that free software is having on the computer industry and the world. While we would all like to make a living out of this, don't forget the fundamental significance of the source code being freed (and this is where the "raison d'être" of the GPL comes in) is that it becomes the basis of further development as were Greek theorems, as was the railway system, and as is the Internet.
From this perspective, it is easy to see that closed code is, most likely, dead code.
My goal with the LTT, as tiny as it may seem against such a grand panorama, is to provide tools that assist both the novice and the expert to gain a better understanding of the overall interactions between and within the systems being analyzed. Make no mistake, tracing tools have been around before, and more are coming. But LTT is the only one that has been developed under the GPL.
As described above, LTT has progressed at a phenomenal rate, in a very short time -- AND with lots of outside help. It has been said that LTT has already surpassed many available tracing tools. This is confirmed by the large number of Fortune 500 companies that currently use LTT to develop Linux based applications. In fact, many say they are planning to use Linux in their systems precisely because such a tracing tool exists for Linux.
Since last April, RTAI tracing has been added to LTT thanks to funding by Lineo. This is a big step forward, as it enables real-time Linux developers to debug their systems using LTT. Lately, PowerPC support came in from several directions. I had been working on a port with funding from Lineo when I received a patch from Bao Gang Liu of Agilent China, followed closely by code from Andy Lowe of MontaVista. Just today, I received word from Maxwell Patten of Nortel Networks that he too has been working on a PowerPC port of LTT. And it isn't over yet . . . !
In the future, LTT will aim to be available for additional architectures that Linux is running on. Also, there are requests for LTT to be run on additional open-source operating systems, such as BSD and HURD. Along with these system-level issues, there are numerous planned tool enhancements and flexibility additions.
Feel free to contribute to this project, either by providing feedback, expertise, or source code. Most important of all, try to keep an eye on the big picture. Try to see how each piece fits with the others, and how they all interact together. After all, that's what LTT is all about.
About the author: Karim Yaghmour is the author and maintainer of the Linux Trace Toolkit. He is the founder of Opersys Inc., which provides expertise on the Linux kernel and it's real-time derivatives.
Related stories:
Will the real "first PPC LTT" please stand up?
First Linux Trace Toolkit for PowerPC
Event trace tool doesn't require instrumented libraries
Linux Trace Toolkit supports real-time Linux
Talkback! Do you have questions or comments on this article? talkback here