OpenDocument Format community steadfast despite theatrics of now impotent 'Foundation'

When in mid-October 2007, the OpenDocument Foundation (ODf, yes, that's a little "f" that's not to be confused with the OASIS- and 400-member strong OpenDocument Alliance-backed big F-ODF: the OpenDocument Format) announced that the World Wide Web Consoritum (W3C)-backed Common Document Format (CDF) was the heir-apparent to what it believed was a dead-on-arrival OpenDocument Format, many confused the ODf to be one in the same with the ODF and the latter to have one foot in the grave. Given the striking resemblance between the names and acronyms of the Foundation and the Format, that mistaken obituary was an easy one for casual observers to write. Especially given the way Microsoft, the company whose Office empire is probably more threatened by ODF than most people realize, capitalized on the confusion by spreading its own FUD on the story.

But that and other FUD couldn't be further from the truth. Based on dozens of interviews that I've conducted over the last few weeks, the OpenDocument Foundation, whose three principals are Sam Hiser, Gary Edwards, and a legal eagle who goes by the nickname "Marbux," went out on a very thin limb where no one else -- not the vendors behind ODF, not OASIS (the consortium that hosts the technical committee responsible for the standard's development), and not the World Wide Web Consortium (chaperone to the Common Document Format [CDF] standard) -- was willing to join them.

Not only does it appear as though they were on a thin limb with their opinions that ODF should be buried and that CDF should take its place, they crawled out even further when they publicly disclosed that the W3C and IBM shared those opinions as well. Any statements corroborating the ODf's position from either organization, particularly IBM given the millions of dollars it has invested and continues to invest in ODF, could very well have cast a dark shadow on the productivity document standard that just recently earned its stripes as an international standard from the International Organisation of Standardisation (ISO). It's an honor that Microsoft's competing Office Open XML (OOXML) has so far been denied (but it is up for reconsideration next year).

Citing specific interactions (conversations, emails, etc.) with the W3C's lead contact for CDF Doug Schepers and Doug Heintzman, director of strategy for IBM's Lotus Division (where IBM's collaboration technologies are developed), Edwards claims that both organizations were supportive of his and Hiser's belief that, at the expense of ODF, CDF should be the strategic target for anyone seeking to store their documents in a file format that was universal, open, and that provided a clear transition path from formats that predispose or lock customers into certain applications like those (formats, applications) from Microsoft.

It is true that Edwards and Hiser interacted with both the W3C and IBM. Unfortunately for them however, this is where Edwards' and Hiser's recollections of those interactions varies wildly from those of Schepers (W3C) and Heintzman (IBM).

One thing that's important to keep in mind about how standards are set (and how decisions are made in technical committees at consortia like the W3C [CDF] and OASIS[ODF]) is that the process often involves vociferous debate among those involved. To the extent that many of the participants who contribute to technical committee meetings are also employees of vendors with some interest in the standards associated with those committees, part of their roles in the process is to represent those interests. Since not all vendors' interests are aligned, disagreement and debate comes with the territory. They're to be expected. But so too is a willingness to compromise. At some point, in the name of progress, everyone who participates in the standards setting process knows they may have to give-in on certain issues that may be of import to their employers.

Representing the OpenDocument Foundation, Edwards and Hiser were both participants in the Open Document Format technical committee work at OASIS and respected ones at that. But somewhere along the line, their beliefs regarding ODF and CDF could not be reconciled with the positions of the other committee members. Pretty much everybody I spoke to agreed that this was one of those disagreements that happens in the standards setting process where someone wasn't going to get their way. It happens. It's a part of the process. But what happened next is not nearly as common. Claiming that the OpenDocument Format wasn't nearly as "open" as its supporters claimed it to be, the ODf walked off in a huff.

If IBM or Sun, two of the OpenDocument's Format's biggest supporters walked away in such a "huff," it probably would have meant the end of the OpenDocument Format. But in the bigger picture of the OpenDocument Format, between its backers at both OASIS and in the OpenDocument Alliance, the OpenDocument Foundation's irreconcilable differences with the rest of community were just that: irreconcilable differences that lacked any potence to affect the momentum or direction of the Open Document Format. Unfortunately for the OpenDocument Format community, the ODf's "huff" was a molehill that became a mountain when, in addition to the ODf<>ODF naming confusion, Edwards and Hiser not only became very vocal about their convictions (convictions that are voluminously documented in easy to find passages around the Web), they cited the W3C and IBM as having tacitly endorsed those convictions.

This is where Schepers (W3C) and Heintzman (IBM) as well as others in both organizations feel as though Edwards and Hiser are grossly misrepresenting the content of their interactions. According to W3C spokesperson Janet Daly, when Schepers first heard of the Foundation's interest in CDF, he did what the W3C often does -- he reached out to the Foundation with an invitation to further the conversation. According to Daly, "Any time it looks like a third party may be doing interesting work with one of our recommendations (that's W3C-speak for "standards"), it's not unusual for us to want to learn more." But this is where the W3C's account of that "conversation" and Edwards' account differ. Whereas the W3C viewed the "conversation" as par for the course outreach, Edwards' e-mails to me describe the ODf's interactions with the W3C as more of a relationship that had to be kept secret from OASIS. Wrote Edwards to me via e-mail:

....When the Andy Updegrove published his article (W3C's Chris Lilley: CDF Not Suitable for Use as an Office Format Can't Replace ODF), a member of our team sent a copy of earlier eMail exchanges with our W3C contacts to Updegrove arguing that Andy's article mis-characterized both our relationship with the W3C and, the work we were doing with CDF and WICD. All of which is true.

There were however a couple of problems with this action. For one thing, we were not authorized by our W3C contacts to share these discussions with anyone, let alone the lawyer for OASIS who had already declared a hostility to anything the Foundation might do....

....I hope you can understand our reluctance at this point to discuss this issue in detail or provide evidence certain to compromise the positions of innocent and sincere bystanders.

The implication of Edwards' note is that the conversations with the W3C had matured far beyond a level of basic outreach and involved a relationship that saw merit in the Foundation's thinking about CDF as a better strategic format for universal document interoperability than ODF.

The W3C however has a different version of its interactions with the Foundation. The reference to Andy Updegrove's interview with the W3C's Chris Lilley (who is also intimately familiar with CDF) is significant. In that interview, Lilley flatly rejected the idea that CDF should be the target in the world's search for an open, universal file format for productivity applications:

So we were in a meeting when these articles about the Foundation and CDF started to appear, and we were really puzzled. CDF isn't anything like ODF at all – it's an "interoperability agreement," mainly focused on two other specifications - XHTML and SVG. You'd need to use another W3C specification, called Web Interactive Compound Document (WICD, pronounced "wicked"), for exporting, and even then you could only view, and not edit the output.

The one thing I'd really want your readers to know is that CDF (even together with WICD) was not created to be, and isn't suitable for use, as an office format.

In a subsequent e-mail to me, Sam Hiser argued that the Foundation's words had been twisted and that it never suggested that CDF would take the place of ODF. However, in both e-mails to me and posts to the Web, Hiser and Edwards have made it clear that the day that ODF-supporter and Massachusetts CIO Louis Gutierrez resigned was the day that ODF died, in their estimation. In his e-mail to me, Hiser wrote:

It's unfortunate you're pointing to the Updegrove|Lilley statements. They are as confusing as can be...Right about now Andy's bloated corpse may be floating down [Boston's] Charles [River] and Chris [Lilley] is doing his best to shade for his W3C colleagues his 180-degree incorrect statements.

On November 10th, in a public thread on the OpenDocument Fellowship's Web site, Edwards wrote:

Chris Lilley's comments are in direct opposition to those we received a week ago from Doug Shepers, the head of the CDF Workgroup. doug however asked that we not publicise his comments until Sir Timothy has had a chance to weigh in.

In my interviews, not only does the W3C reject the reference to W3C director Sir Tim Berners-Lee as a fabrication of the facts and stand behind Chris Lilley's statements 100 percent, the W3C also remains emphatic that its conversations with the Foundation were never more than cursory in level. In fact, at one point when technical information was e-mailed to Schepers, Schepers purposefully ignored it.

Why?

It didn't take many patent infringement lawsuits for the standards consortia industry to wisen up. To prevent patents from inadvertently becoming part of a standard (thereby entitling the patent holder to royalties), standards consortia now require a full intellectual property (IP) rights disclosure before anyone will even look at some potentially patented or copyrighted technical material.

If Schepers or any other W3C staffer laid eyes on such technical material prior to such disclosures being made and cleared by the W3C's general counsel (Danny Weitzner) and, later, some W3C standard coincidentally ended up with a similar technology in it, the IP holder to that technical material could claim that the W3C saw its patented technology and willfully infringed on it.

As said earlier, the W3C isn't the only organization claiming that Edwards and Hiser have misrepresented certain conversations and exchanges. Via e-mail on November 19th, Edwards wrote to me:

On November 8th, 2007, Sam and I conferenced with IBM's Doug Heintzman. Doug laid out IBM's "grand strategy", as well as expressed his concerns that CDF might be damaged by the ODf communities public hostility to the Foundation.

The IBM "grand strategy" turns out to be CDF+ (using multiple profile variations) to connect ODf desktops to the IBM Cloud of web platform Notes Hub, SaaS, HaaS, SOA and Web 2.0 collaboration services. The key is connecting desktop Lotus Symphony ODf documents to the IBM Cloud using fluid ODf <> CDF+ conversions.

We found this strategy to be very cool and very resonant with our own thinking. Like IBM we were also very concerned about the Exchange/SharePoint juggernaut. Unlike IBM though, we think the marketplace is unable and unwilling to tolerate the disruptive costs of replacing MSOffice with ODf alternatives.

Back to the aforelinked thread on the OpenDocument Fellowship's site, Edwards wrote:

[Sun and IBM] have been arguing for years that CDF is the way to go. Today IBM admitted to us that ODF is a transitional field format only. They know that CDF is the future, and have spent enormous resources positioning WebSphere, Lotus Notes, and the Eclipse Community in that direction. ODF is a transitional desktop play only.

If what Edwards was saying was true about CDF being the linchpin to IBM's "grand strategy," then the OpenDocument Format's strategic viability could easily be called into question. One day earlier, under the heading Document Format FUD: A Guide for the Perplexed, OASIS ODF TC co-chair and IBM staffer Rob Weir blogged his own thoughts on what had been publicly said and wrote:

So, does IBM then oppose CDF in favor of ODF? .....No. IBM supports both the development of ODF and CDF and has a leadership role in both working groups. These are two good standards for two different things.

Since Weir was obviously prepared to speak on IBM's behalf about its roles in ODF and CDF, I asked him about the "grand strategy" that Edwards suggested was a part of IBM's master plan. That's when Weir suggested I talk to Heintzman, but not without writing:

Lotus Symphony supports ODF as its native file format.It doesn't support CDF and we have no plans to support CDF in Symphony.IBM participates in the W3C's CDF activity, as we participate and support many W3C activities.But we don't see CDF and ODF as operating in the same space.They are two different markups for two different purposes.

Not completely satisfied by this answer, I followed his advice and went to Heintzman. Weir spoke of the ODF support in Lotus Symphony. But what wasn't clear was whether ODF was strategic in IBM's thinking about Web-based documents, or whether CDF might have a role there instead. In his e-mails, Edwards has insisted that, whereas Microsoft's Office Open XML is designed to support documents on the Web as well it supports documents on the desktop, that ODF isn't nearly as robust. In order to loosen Microsoft's grip, argues Edwards, CDF will have to play a role.

But in a telephone interview, Heintzman, who remains confounded by Edwards' version of their conversation, flatly denied that CDF has a role in some IBM master strategy:

I sat down with them, said we're all on the same side. Our interests are aligned and I gave them some advice on how to move the industry in the right direction. I told them a bit about our emerging strategy, beyond Office, the importance of semantic layers and compound documents where we see things evolving to. But they dramatically misinterpreted and misrepresented what we said....

...ODF has a level of sophistication for rendering office documents that CDF doesn't pretend to. We know what CDF is, we chaired the [W3C's] CDF committee at one point. But they are apples and oranges....

...Gary and company had a technical disagreement with the OASIS ODF Technical Committee. Other members voted that their approach was not reasonable. It's a normal part of a healthy standards process....If Gary and company want to move on to greener pastures and invest in other parts of industry, then more power to them. But to position ODF as transitional is ridiculous. He got two things confused.

Heintzman went on to describe IBM's vision whereby a compound document architecture like CDF serves as a container for something that he hesitates to even refer to as a document. In his eyes, the container is more like a mashup of information coming from all sorts of different sources many of which are deeply intertwined with business process. One bit of that information could come from a range in a spreadsheet that itself is formatted in ODF, but that only a small portion of which is relevant to the information being mashed together in that container. Other bits of information could come from database queries, business processes, sources of video, audio, etc. Where something like CDF could be relevant, says Heintzman, is in the semantic layers that live on top of the content. For example, security: who has the access rights to edit the spreadsheet data or change the database query. Said Heintzman:

In our vision, the containers (documents) become much more intimately ingrained into a business process: You're not creating a document, but rather completing a mortgage application. The container will have bits and pieces of feeds, videos, and information coming from some back-end Siebel or SAP system and multiple people will interact in real time with different parts of that container. There will be a content layer and specialized sub-editors being able to manipulate the information in that layer -- one of which could be an ODF-based editor to the extent the content is coming from an office document.

There will be a semantic layer for the semantics of authorship and approval and it may include content semantics so that the container can be tagged manually or automaticaly through pattern recognition and fingerprinting. This would enable searching and discovery in ways not doable before. A lot of this goes beyond what office document editors can do today, but that doesn't mean that [the role of office document editors] changes.

What's CDF's role in this? It's really hard to say at this point whether it will be CDF, Xforms, or something else that becomes the meta-container. These are young days.

Heintzman also made it clear that nothing he knows of in the container space he referred to offers the kind of fidelity that a office document format like ODF offers and therefore, IBM remains fully committed to the OpenDocument Format indefinitely. When I asked Heintzman to size up that commitment in terms of dollars, he said it was hard to put a finger on it. But between the programmers, researchers, consultants, lobbyists, and lawyers who are heads down on ODF, or have just part of their time allocated to it, the expenditure is easily north of $5 million per year and the company has no plans of slowing down any time soon.

So where are we?

Well, for starters, any insinuation that the OpenDocument Format is dead, hurt, or even scratched is just pure FUD. The people behind the OpenDocument Foundation have clearly participated in the OpenDocument Format's evolution in a meaningful way that has earned the respect of their contemporaries. But to say that the loss of their involvement or their changing opinions about the long term viability of the OpenDocument Format are in any way a reflection of the direction that the OpenDocument Format is taking would be mispeaking. The Foundation's closure and departure from OASIS can't even be characterized as a splintering of ODF the way Unix splintered (or the way Red Hat and Novell's treatment of Linux could be considered a splintering). In the big picture, it's a non-event.

As for the differences over what was said, I don't want to say anyone is a liar. I wasn't in the room or party to the relevant threads. So all I have to go on is what everyone on both sides of the debate is telling me. I can repeat that here (which I've done) and leave the decision as to which one of the three following things is true to you: (1) Hiser and Edwards are accurately representing their interactions with the W3C and IBM and the people they communicated with like IBM's Heintzman and the W3C's Schepers are part of a well-organized conspiracy to discredit them, (2) Hiser and Edwards are purposefully misrepresenting the content of their communications, (3) it's all a big mix-up -- an honest misunderstanding.

Where there's more information to be had to help you make that decision, the Web is very liberally sprinkled with opinion over the matter. A lot of it comes from Gary Edwards and Sam Hiser who, at the very least, feel very strongly about the validity of their approach to document portability and interoperability. In everything they've written to me and on the Web, they argue passionately about why they really believe CDF is strategically the world's best option as a strategic universal document format. I've purposely left most of the technical arguments out. At this point - they're opinions. Today's not the day to vet the merits of the different approaches. I promise to come back to that at some point (perhaps in 2008). Today, I just wanted to get to the bottom of the he said/she said and, to the extent that more facts are needed, my sense is that additional information will come to light once I press the publish button. That is after all the beauty of the blogosphere. The mob usually finds the truth, eventually. Someone will be vindicated.

Finally, in doing my homework for this story, I thought I'd draw attention to one glimpse of the future that I found to be fascinating.

At the beginning of this post, I pointed to a blog written by Microsoft's Director of Corporate Standards Jason Matusow who was clearly seizing the opportunity created by the ODf<>ODF controversy to undermine the OpenDocument Format Camp. Don't fault him. To the extent that the ODF community loves to take shots at Microsoft, he has a job too and he's doing it well as evidenced by headlines like Formats, Formats, and more Formats....some say there should be only...except the other one...and that one...and the new one...and...<sigh>. Matusow is a sharp guy and I have a lot of respect for him, especially after my last podcast interview with him.

But towards the end of his post, there was something he wrote that caught my eye:

All of this seems to make the point stronger than ever that when you are speaking about document formats, you are really speaking about an adjunct technology to the applications which are the real "solutions" in this discussion.

After re-reading that statement several times, it dawned on me how very different the thinking appears to be at Microsoft versus IBM. Matusow sees the application as the real solution. True to form, Microsoft is a very application centric solution provider. But in my discussions with IBM's Heintzman, he told me:

I can have multiple people interacting with a container -- some through a tiny Web browser gadget that can get cells from an ODF spreadsheet and others who are specialized people with different tools for affecting different parts of the document.

In fact, in everything Heintzman told me, IBM's strategy sounded more information-, business process-, and people-centric than it sounded application centric. I was vaguely reminded of a blog I posted last year about knowledge/information centricity vs. document centricity. While it speaks nothing of the implementations (Microsoft's solutions could easily turn out to be more knowledge-centric than IBM's), I think IBM is using the right language and Microsoft would be well-served by de-emphasizing "the applications." Long term, applications as we know them are dead and the functionality we associate with applications today will simply be a part of the data we're interacting with tomorrow.