When is a standard not a standard?

Some of the most common formats in computing today veer away from the standards process, but are considered "de facto standards" nonetheless. Should we be worried?
Written by David Becker, Contributor
Technology users are pondering that conundrum more and more as the industry moves to technology standards. Most are administered by a standards body, such as the World Wide Web Consortium, to ensure interoperability between disparate systems.
But some of the most common formats in computing veer away from the standards process in one way or another, creating a nebulous area of "de facto standards" that mix democratic ideals with corporate concerns.
As a result, these technologies are testing long-held assumptions about the standards process.
Some of the most common examples -- such as Microsoft Office and Adobe document formats, Flash Web animations and RSS (really simple syndication) blogging -- map out a changing battleground between proprietary and open publishing models. The ideals of open software and readily interoperable standards are pitted against the drawbacks -- slow-moving standards bodies and a feared design-by-committee mentality -- and the advantages of proprietary software, which include speed and control.
Often, though, software makers find out they don't have to choose -- software that never went to any standards organisation might still, by dint of being first or most popular, become the industry's preferred application by default.
"There are lots of de facto standards in use, and a lot of them work fine," said Sun Microsystems software guru Tim Bray, co-inventor of XML (extensible markup language), one of the most widely used standards in modern computing.
Bray cites Perl, the ubiquitous Web programming language, as an example of a successful de facto standard. "Perl is defined by one implementation, and that's not been a problem."
One of the biggest obstacles to companies entering formal standardisation processes is the fear of getting bogged down in bureaucracy, said Stephen O'Grady, an analyst with research firm RedMonk. "The primary complaint we hear time and time again from vendors about working with standards bodies is the speed," he said. "Developing anything by committee takes time. To get everyone on the same page takes time."
The Microsoft default
For years, one of the most ubiquitous and potentially troublesome of de facto standards has been the document formats used by Office, Microsoft's widespread productivity package. The .doc format for Word documents, .xls for spreadsheets and other formats were and remain proprietary to Microsoft, meaning non-Microsoft applications may have trouble opening the files, or may not display them as intended.
Closed file formats that may not work with other applications have been a significant part of Microsoft's domination of the productivity software market, said Gordon Haff, an analyst for research firm Illuminata.
"The concerns about compatibility ... certainly are a major reason that keep people on Microsoft Office," he said.
Microsoft took major steps toward addressing compatibility concerns with the arrival of Office 2003, which includes options to save documents based on XML, an open standard widely recognised by competing applications. Microsoft took the next step by publishing the proprietary XML dialects, or "schemas," used by the main Office applications under a royalty-free plan intended to boost support among outside software publishers.
Publishing the schemas was an effective way to address growing concerns among governments and other customers about interoperability and archival issues surrounding documents, said David Kaefer, director of Microsoft's intellectual property and licensing group.
The European Union praised the schema publication in a recent report and suggested Microsoft take a few more steps toward ensuring interoperability and document integrity, including submitting the Office XML schemas to a recognized standards body.
Influential open-source developer Bruce Perens agreed that supervision by a standards body would be a significant step toward achieving true standards.
"The key thing about open standards is that they're fair, impartial and available for everybody to implement," he said. "The Microsoft offerings fall short of that in a number of ways. It would be nice if we could have a level playing field."
Kaefer argued that submission to a standards body wouldn't offer any practical advantage over Microsoft's commitment to publish the schemas royalty-free, and it could substantially impede innovation surrounding the formats.
"We don't see a lot of upside to a standardisation process here," he said. "For technologies that are pretty mature, you're always worried about, 'Are we going to be limited in how we can innovate?'"
RedMonk's O'Grady said the partial step of publishing the XML schemas could be enough to protect Microsoft from regulatory pressure to further open the formats, particularly the more familiar .doc and .xls extensions, as Microsoft faces ongoing scrutiny from the European Union.
The Adobe way
On the other end of the "de facto" spectrum is Adobe's PDF (portable document format), widely used for electronic distribution of documents. Adobe still owns the specification, which originated in the early 1990s and was considered for submission to a standards body. But it's so freely published that hundreds of non-Adobe tools for generating PDF documents are on the market, including components built into Apple Computer's Macintosh operating system and the Openoffice.org productivity package.
Even standards purists such as Perens and Bray say there'd be little to gain from turning PDF over to a standards body at this point.
"PDF is an example of a proprietary standard that has achieved so much inertia in the marketplace that it's hard to see where standardising it would benefit anyone much," Bray said.
Melonie Warfel, director of worldwide standards for Adobe, said Adobe participates in numerous standards bodies, including specialist groups creating open-standard extensions of PDF for archival and advertising uses. But the main PDF specification remains under Adobe's control so it can be quickly adapted to meet new needs, such as the bar-code capability recently added.
"If you start developing a standard through a standards body, it takes forever," Warfel said. "My concern is we wouldn't be able to keep up with technological changes."
But Adobe achieves most of the goals of open standards by publishing the PDF specification under liberal terms that allow other software makers to use it as they please, Warfel said.
Flash frozen
The standards picture gets a little fuzzier for Macromedia and its widespread Flash animation format. Macromedia began freely publishing the Flash specification in the late 1990s to encourage widespread adoption, a successful tactic that has resulted in the Flash client being installed on more than 95 percent of all Internet-connected PCs.
But Macromedia has declined to submit the specification to a standards body. And the market for Flash authoring tools, while spawning dozens of applications for creating little bits of Flash content, is still dominated by Macromedia applications.
"Comparing Flash to PDF, it's clear Adobe has done a better job of opening that format and fostering a community around the format," O'Grady said. "The thing with Flash that's a little different ... is that it's really just now starting to become more of a business tool. People are maturing in the use of the format."
Bray said he has "mixed feelings about Flash. It's clearly been useful on the Web, but it does bother me that it's owned by Macromedia, and they don't make much money on it. Are they going to have the energy and motivation to take care of it in the long term?"
Such worries have helped fuel a slowly building backlash against Flash and a groundswell in favour of SVG (scalable vector graphics), an emerging graphics standard endorsed by World Wide Web Consortium (W3C), the main body behind Web standards such as HTML.
"To some extent, I think Macromedia may have missed the boat," Perens said. "Eventually, browsers will all have SVG plug-ins, and at that point, Flash starts to become irrelevant."
David Mendels, general manager of Macromedia's tools and platforms group, said the company has supported open standards where it makes sense, including wide-ranging support for formats such as XML, SOAP -- even SVG. With Flash, the company was able to achieve open-standards goals without going through a standards body, he said.
"We don't think there's one approach that applies to the whole stack of things we have here," Mendels said. 'We made a commitment in the 1990s that we weren't going to keep (Flash) a narrow, quirky thing that doesn't interoperate with other people's products, and we've followed through on that. I think we've been very successful in making Flash a widely supported open standard."
"Some people have very strong religious views that everything should be open-source or everything should go through a standards body," he continued. "I don't have views like that ... I want to address what people are telling us are the real problems, through whatever vehicle makes sense."
Blog slog
One of the newest levels in standards limbo has been created by the booming popularity of Web logs. RSS, the specification that allows efficient posting and browsing of blogs ranging from news alerts to personal musings, was developed by pioneering software developer Dave Winer and is now managed by a Harvard Law School project, which distributes it under a liberal "creative commons" license.
Winer said he's investigated submitting RSS to one of several standards bodies but worries that the "really simple" part of RSS might be comprised by overeager developers.
"The problem is that a number of these organisations don't want to just take something and ratify it -- they want to create something and ratify it," he said.
Winer's reluctance to submit RSS to a standards body and insistence on freezing the specification have attracted growing criticism from some blogging interests, who'd like a more tricked-out blogging format. The result has been growing support from Google and others for the "Atom" format under development and intended for submissions to a standards body.
Bray said it's wrong to cast Atom as a direct rival. "The view that there's some sort of war between RSS and Atom is silly," he said. "RSS is widely deployed and it will continue to do a fine job of meeting a lot of people's needs. For new applications, Atom is going to be a sweet spot for some applications where people want more strictly structured data."
Still, it would help to have RSS established as an open standard, Bray said. "I don't think it would have been smart to put RSS through a standards process before now," he said. "But I think we're mature enough to write down the rules now."
The debate over RSS illustrates the classic debate over standardisation. What's to gain by subjecting already popular technologies to a long, sometimes painful standards process? Ultimately, the protection that standards provide to consumers that one company's products will work with another's.
"At the end of the day," Bray said, "the best way to ensure interoperation between technologies is by adhering to an open standards process. By default, I think standardisation should be the process. If a new technology doesn't want to be standardised, it should prove why."
O'Grady said there are valid objections to be made about how standards bodies work. But there are few substitutions. "We hear a lot of complaints about standards bodies, but I've yet to hear someone come up with a better way to do it.
"It ultimately ends up hurting everybody when you don't have a uniform standard," he said.
Editorial standards