Democracy and standards

Democracy and standards

Summary: David Berlind thinks Massachusetts' decision to reject Office XML as an approved document format was a shining example of democracy in action. I think it's a glaring example of power politics and the manner in which interest groups skew government procurements in costly ways.

TOPICS: Government

David Berlind recently completed an interesting in-depth analysis of the events that led to Massachusetts decision to jettison Office XML from its approved list of document standards. Assuming the decision stands, this will affect a lot of people besides workers at government agencies.  Contractors that work for the state or regular citizens required to send digital documents to government offices will have to hew the line drawn by state technology authorities.

Berlind closed his analysis with the following paragraph:

Not only has Massachusetts developed a modern and virtually unassailable test (particularly for its requirements since virtually every part of the test can be connected to the state's need for sovereignty), it has fully leveraged the democratizing forces of government and technology to arrive at an informed decision that serves the best interests of its public.

Let it be said that I fully respect Berlind's opinion, his journalistic skills, and his integrity. I have long been one of his "fans," and read almost everything he publishes on ZDNet. That doesn't mean that I always agree with him, and today is one of those times that I don't. Over the course of this week, therefore, I'll outline why I disagree with some of the conclusions presented in his analysis.  I'll start today with his claim that the process demonstrated the power of democracy - and by implication, government - in the creation of software standards.

Let's step back a moment to remember that the central issue in this discussion is one of control over GOVERNMENT procurements.  When is the last time you looked to GOVERNMENT procurements and thought "if only I could spend my own money as efficiently as government spends its tax revenue?"

Governments have a very hard time efficiently allocating resources because governments aren't using the same procurement rules that businesses and inviduals use. Businesses and individuals are spending their own money. They worry about making that money go as far as possible.

Costs matter to businesses and individuals in ways they don't to governments. If you ever had any doubt about that, remember that government is also famous for purchasing $10,000 toilet seats or building a bridge in Alaska that goes nowhere.

Governments spend money "democratically," if "democratically" is read to mean spending it in a way that results in the least bad press or satisfies the interests of well-organized groups. Governments tend to respond to the loudest voice. That's why lobbying is such an important activity, and is also the reason political interest groups often have power out of proportion to their numbers.

Ask yourself why teachers unions in California can tie the Sacramento Legislature in such knots even though high deficits and poor test results prove that California is spending its money badly, or why farmers that represent tiny fractions of working populations in rich countries have convinced governments to pour huge sums of money into a vanishingly small sector? The loudest voice gets heard first, and few voices are as loud as California teachers unions or farm lobbies in Western countries.

The same principle applies in Massachusetts' battle over a technology standard.

Microsoft's Office XML specification was once on Massachusetts’ approved list (and I'll talk more about the details in future installments). That all changed, however, after a period of public comment, as explained on page 3 of David's article...

As pointed out earlier, Massachusetts Information Technology Division general counsel Linda Hamel noted that the state received a "firestorm of commentary" in response to the early 2005 MA ETRM draft that listed Microsoft's file formats as the supported "open" standard. According to various sources, the response was so opposed to the inclusion of a Microsoft format in what was otherwise an open initiative that it caused Massachusetts officials to take a deep breath and reconsider its test for openness.

Receiving bagfulls of email in response to a proposal to include a Microsoft technology on an approved list of document formats shouldn't surprise anyone that has participated in debates between proponents of open source and proprietary software. Open source advocates, a group who are are VERY well organized and have VERY loud voices, did the same thing to Unisys over it's LZW patent, to Kollar-Kotelly in comments relating to her proposed remedy to the Microsoft antitrust case, to the EU Commission in its battle over software patents, or on any other front in the war declared by open source on proprietary software. It really isn't anything unexpected.

What such feedback definitely does NOT do, however, is put Massachusetts in the position of a group flush with feedback similar to what developers get during the software usability testing process. That was the opinion of Dan Bricklin, co-inventor of the electronic spreadsheet. A few moments in the feedback section on Slashdot for an article that says anything remotely nice about Microsoft should disabuse anyone of that notion.

Of course, it wasn't just the open source movement that ensured an outcome that didn't include a Microsoft-backed technology. Another group which can be counted upon to oppose initiatives backed by Microsoft are competitors.  In other words, it doesn't surprise me in the least when the following happened:

Page 3: The meeting came on the heels of ODF 1.0's ratification by the Organization for the Advancement of Structured Information Standards (OASIS) and, according to Johnson, the tone of that meeting was that XML was the way to go and that virtually everyone in the room but Microsoft was behind ODF.

That meeting was attended by Sun, IBM, Adobe, and many others, groups in direct competition with Microsoft. They also backed a "standard" that accounts for a small fraction of the market. They have nothing to lose in doing so, simply because they have already lost in the wider market for Office products (PDF is popular as Word formats?). How better to undo that outcome than to back a standard that is used by very few?  Even if ODF doesn't acquire critical mass, it never hurts to throw confusion into a market already consolidated around a competitor's product.

A group of Microsoft's opponents backing non-Microsoft technology is like a council meeting with an audience filled with partisans of the losing party. Should it surprise anyone when that audience loudly opposes policies backed by the winning party? The resulting arguments aren't a promotional poster for the power of democracy, but the power of interest group politics.

Microsoft is a large company, and can certainly speak loudly. Every Microsoft press release is scrutinized by armies of pundits, and even if those press releases are spun in a negative direction, their voice is at least heard. That doesn't mean, however, that Microsoft can shout louder than the assembled voices of its opponents, nor alter the tone of an industry that views hatred of Microsoft as an entry requirement into the technology intelligentsia.

Berlind thinks the Massachusetts government has made a brilliant decision, in spite of the fact that it backed a technology that is mostly a no-show in a much larger democratic process (the global marketplace) that has standardized on Microsoft Office and its associated technologies. He thinks it reveals the power of democracy, even though that "democratic power" is the same force that creates laws banning interstate wine shipments (in order to protect local liquor wholesalers and tax revenues), or spends huge amounts of money to buy product from non-competitive local companies (the likely explanation for stories of $10,000 toilet seats).

In my opinion, governments should endeavor as much as possible to match the choices made by the larger marketplace, simply because that marketplace is likely to be fairer, and less skewed, than council meetings or government-assembled standardization committees can ever hope to be. The best way to do that is to let individual agencies make decisions based on needs. Given that government is supposed to be a FACILITATOR and not a TREND-SETTER, agencies that are effective will tend to use technology that its customers - the voting public - have chosen to use in their business lives.

If ODF swept the business world and became a new, de facto standard for digital documents, I would have zero difficulties accepting the new world order.  Where I have a problem is when special interests attempt to shoehorn a new standard onto an unwilling marketplace by playing the game of power politics.  

Government is both absolutely critical to the proper functioning of a market economy, as well as its biggest threat.  Massachusetts' attempt to redesign the shape of the marketplace by using its giant procurement club falls into the latter category. 

More tomorrow (or sometime this week).

Topic: Government

John Carroll

About John Carroll

John Carroll has delivered his opinion on ZDNet since the last millennium. Since May 2008, he is no longer a Microsoft employee. He is currently working at a unified messaging-related startup.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion

    The man who champions "the market shall decide" is upset because the market decided against MS for once?!?

    Really, John, let us be honest here. The requirment was simple. A completely open file format for storing government generated information. The reason for this requirement was that they wanted to retain ownership of the generated information and allow for easier dissemination of said information.

    That was the [b]base-line requirement[/b], any issues about usability, market-share, maturity of format, etc. where second to that requirement. If you can't clear the first hurdle all the others are immaterial.
    Robert Crocker
    • Where was the market in this?

      Certainly not in meetings stacked with people backing a file format used by under 1% of businesses.

      As for a "completely open file format," stay tuned for my posts which will follow later this week. I talk about that at length. Just some advance hints, though, it is okay to insist that a specification be open and available to others. Office XML IS open in all the ways that Massachusetts wants it to be, at least as open as PDF...a format that WAS included on the list.

      But like I said, more on that later this week.
      John Carroll
      • Don't be a tease, John

        Just post the URL for the full specification for MSXML (whichever version MS was proposing Massachusetts use, since there seems to be some difference of opinion on the matter.)

        For extra credit, point us at the compliance suite. You know, the objective test that can be used to determine if a proposal meets the requirement? That's a key part of any serious development specification and an even greater part of any RFP -- it's so handy when sorting out the "qualified bidders."
        Yagotta B. Kidding
        • I will tease...

          ...because I'm not playing out the full discussion today.

          Here's the link, though...

          John Carroll
          • What's that?!? License you say?

            From your link:
            Microsoft may have patents and/or patent applications that are necessary for you to license in order to [b]make, sell, or distribute software programs that read or write files that comply with the Microsoft specifications[/b] for the Office Schemas.

            Wow, that's an interesting definition of open you have there John.
            Robert Crocker
          • Rob...

            you haven't read David's long article on this subject, as one of the points he makes is that Microsoft grants you the right to use ANY patent, even those unrelated to Office XML, to implement a solution that reads and writes according to the format.

            In other words, it's an open-ended royalty free grant. That's pretty open.

            There are limits, and David also discusses those. I'll discuss them, too...later this week.
            John Carroll
          • Correction

            [i]Microsoft grants you the right to use ANY patent, even those unrelated to Office XML, to implement a solution that reads and writes according to the format.[/i]

            Not so. Reads, not writes, and the license is not transferrable -- which means that every end-user has to execute a personal license with Microsoft, assuming that Microsoft continues to make the offer and assuming that the software continues comply with Microsoft's (undefined) terms for full rendering of the format.
            Yagotta B. Kidding
          • John...

            So how do you parse this particular portion of the license?
            Notwithstanding the foregoing, "Necessary Claims" [b]do not include[/b] any claims: (i) that would require a payment of royalties by Microsoft to unaffiliated third parties; (ii) covering any Enabling Technologies that may be necessary to make or use any product incorporating a Licensed Implementation, or [b](iii) covering the reading or writing of files other than those complying with the requirements of the specifications for the Office Schemas.[/b]

            One possible reading is that you couldn't create an Office ML to ODF converter (depends on how you read that "or" I suppose).

            The other more important point is that this license ONLY covers Office 2003's XML format. There is no guarantee whatsoever that the next version would be licensed under such "generous" terms.
            Robert Crocker
          • Wait, it gets worse

            Also from the license:
            By way of clarification of the foregoing, given the unique role of government institutions, end users will not violate this license by merely reading government documents that constitute files that comply with the Microsoft specifications for the Office Schemas, or by using [b](solely for the purpose of reading such files)[/b] any software that enables them to do so.

            Why solely for the purpose of reading? In that case they might as well just generate PDF's and be done with it since they're not licensed to manipulate them.
            Robert Crocker
          • Ummm -- John?

            I know about the Borg implants and all that, but that's a EULA -- not a file specification. I realize that the distinction is not recognized in some circles.
            Yagotta B. Kidding
          • Yagotta...

   know, those little blue underlined things are called LINKS. And, if you want to see the full license, why don't you CLICK ON THEM.

            Granted, you'll need to install the Office SDK, but that's all part of my evil little plan.
            John Carroll
          • In other words

            As with everything else at Microsoft, there is no formal specification -- the only authoritative definition of the file is Microsoft (closed) code.

            Thank you for confirming that point.
            Yagotta B. Kidding
      • Massachusetts was the market

        They defined a need and they filled it.

        MS's Office XML is [b]NOT[/b] open. They have attached a license to it claiming that there are potential patent licenses needed in order to use the format. They then leverage this claim to limit what you are allowed to do with their published format.
        Robert Crocker
        • Quibbles

          [i]They then leverage this claim to limit what you are allowed to do with their published format.[/i]

          Yeah, such as write to the files.

          Oh, and it's not published. As John confirmed, the "official" definition is an installable binary SDK from Microsoft.
          Yagotta B. Kidding
      • So did Berlind misunderstand the Office 12 format license?

        He seemed to explain it pretty clearly that it was not as open as MA wanted. It doesn't allow developers to take a portion of it and use it in their products, as ODF and PDF do (though Adobe has a stipulation about calling a modified format "PDF"), and best I can tell the Office 12 format is not available for technical peer-review and joint stewardship (ie. submitted to an independent standards committee), as they also wanted.

        I think these sorts of requirements are the kinds of things bureaucrats would think up. One of the goals, apparently, of the Council was to promote software competition. They think by adopting this standard that this will happen. I don't doubt it opens up opportunities for those who didn't have them before, but so does breaking a bunch of windows in housing developments.

        I liked something that Jonathan Zuck of ACT said at the Sept. 16 meeting. He asked that they "take a step back in terms of what problem you are SOLVING. In other words, people's ability to get to documents from the state, because that's the problem that you stated. And how is it that you're not CREATING a bigger problem than the one that you're solving?"

        The thing with any standard, is that companies in particular will want to add extensions to it. It's the competitive impulse to differentiate. What this means is a bunch of incompatible formats floating around. I have no doubt that with ODF, compliant applications will at least be able to read a subset of the features of any ODF document, but is that going to be satisfactory in most cases?

        MS Office's ubiquity means, if nothing else, that at least most people can read each other's Office documents consistently and with full fidelity.
        Mark Miller
        • Fidelity

          [i]MS Office's ubiquity means, if nothing else, that at least most people can read each other's Office documents consistently and with full fidelity.[/i]

          I'll pass that along to my friendly neighborhood tech writer who, working in a 100% MSOffice shop, spends a good bit of her time dealing with the obscenities that MSOffice does to its own files.

          I will [b]not[/b] mention version changes, since that tends to send her postal.
          Yagotta B. Kidding
          • I've sent out hundreds of them

            I've sent out resumes in MS Office format hundreds of times over the last several years (they ask for hardly anything else--though they sometimes like plain text), as have my friends. Not one thinks that an employer or hiring agent won't be able to read it. As a precaution, I typically send it out in an older format, like Word 2000, just in case. But to date I haven't had a problem with it, at least that I know of...

            In any case, I haven't heard of any recruiters asking for PDF or ODF format resumes, and neither have I heard from fellow job seekers that they needed to use a different format, due to incompatibility problems. We've talked to HR folks and they haven't mentioned it as a concern either. I don't care about the format. Frankly I think it's great that I can apply for a job electronically. 15 years ago I would've had to have printed it out on a laser printer on nice stock paper, and sent it in a nice envelope with postage.
            Mark Miller
          • Also imported documents from the Mac

            I managed a newsletter for my mom for a few years. We got many documents from contributors and people advertising their services, usually in Word format. With a couple contributors I had to import Word documents from Mac users. That was a technical challenge, but only because the Mac e-mailer encoded binary attachments differently than my Windows e-mail client did, and they had to adjust a setting to send it to me in the right encoding format.

            We were able to import them all into the newsletter (in MS Publisher) with little problem. The only problem we ran into typically was someone using a font we didn't have, but that is going to occur with any document editor you use. Many of the fonts out there are copyrighted and you have to buy them to use them. The one time I did have a real problem with an ad, that I can remember, was when an advertiser had one done up for her, and was saved in PhotoShop format. I couldn't import it directly.
            Mark Miller
        • The license in question

          was for MSWord2K3 -- not the fabled Office 12 format, which has yet to be publicly disclosed.
          Yagotta B. Kidding
        • Perhaps you'd like to try

          to propose a revision to the Massachusetts ETRM that would replace ODF with MSXML?

          So far, no takers on that one.
          Yagotta B. Kidding