Sympathy, and thanks, for Harry

Reading through the programmer's notes released as part of the Hadley leak had me squirming in sympathy with the writer(s). Data that's supposed to say one thing saying another, data with no referents, measures or provenance; bad code with magical jumps to right answers hard coded right in? been there - seen all that. What bothered me, however, was how this mess got so far out of hand: one mistake piled on another, one compromise after another, and another - why didn't this guy hang his employers out to dry years ago?

One of the files made public by the Hadley leaker was "HARRY_READ_ME.txt", apparently produced by one or more people (collectively known as harry) who worked there in data analysis and programming.

(If you don't already know enough about this, watch this video spoof for fun, and read this summary by Charles Martin for the facts.)

The harry files are mainly about his struggles with data, with interspersed bits about the difficulty of making the analysis of that data fit what the activists were saying about it. Thus a summary offered by Watt's Up With That contributor Michael Alexis strikes me as bitingly accurate:

Simplified code for CRU to use (from an average programmer with no climatology expertise):

For intYear = 1400 to 2009;
floatGlobalMeanTemerature = floatGlobalMeanTemperature + WHATEVER_THE_HELL_YOU_WANT_IT_TO_BE;

Print "Holy Crap! We're all going to die!"

There are several things that are utterly appalling about the harry files - starting with harry's failure to blow the whistle on these guys much earlier.

I understand, of course why he didn't; and why, if you're in a similar position, you might be afraid to - but it's a sucker bet that harry will end up wearing this as his bosses claim they didn't know he was using the data they published to reconstruct the data they cited in those publications.

On the other it's hard not to feel harry's pain - God knows I've been there; several times.

I understand exactly what he was up against with respect to the data - back in the early eighties one of my clients bought about 300,000 sets of regulatory agency records pertaining to producing, dry, and capped oil or gas wells in western Canada - records nominally going back to the 1920s. Unfortunately, the basis for locating a well had changed several times for each regulator; key measures (barrels, gallons, presure per cubic unit, reporting periods, and various boundaries) had all changed several times over the period; companies had come and gone; each new tax, royalty, or regulatory regime affecting then active wells had led the record keepers to adapt their then current records to the fiscal needs of the period; the electronic files incorporated several -unmarked and undifferentiated- generations of coding logic; and, a general wink nudge relationship between the industry and its record keepers had some companies paying royalties on wells that didn't exist, not paying for some that did, and moving some across regulatory boundaries for reporting purposes.

Since all of of the regulators involved regularly published retrospective summary statistics with no obvious basis in fact, our inability to tie records to record producers, data to reality, or data to published summaries was broadly similar to harry's -with the big difference being that nobody, then or since, has tried to ram through a wildly destructive political agenda on corrupted well data.

Much later, another client did try to perpertrate a fraud on the public using data I was responsible for: specifically, "our" Finance VP made a public market announcement of the company's glowing success in implementing activity based costing - and gently reminding the president that we hadn't actually picked the software for this yet got this stopped, but also cost me a job I was enjoying.

The latter outcome was, I'd guess, the kind of denouement "Harry" didn't want to face - and I feel his pain, but it's exactly this kind of getting along by going along that lets a little latitude with the numbers grow into the kind of massive abuse of public trust committed by the activists at Hadley.

The closest direct parallel I know of, however, is one I wasn't personally involved in: the situation at Worldcom in the years preceeding its collapse. What I'm told happened was that the information systems people simply "lost the plot" as one acquisition after another overwhelmed their ability to manage the processes producing the information the executive relied on for their public reporting - thus enabling a number of them to pyramid an increasing preference for guesses they liked over the ones they didn't into a major corporate fraud.

Eventually their CEO, Bernie Ebers, went to jail for presiding (whether he knew it or not) over a fraud on the public - and that's a precedent people like Gore, Jonas, and Hansen should be worrying about right about now because there are other Harrys out there, and every single one of them has to be thinking about doing the right thing before people like Chris Horner force their employers to throw them to the wolves.

And that's the bottom line other harrys should see for themselves here too: yes, amputation hurts; but it can also save your life - so don't be harry: if your bosses are lying to the public, their shareholders, or even just their bosses: be absolutely sure you're right; and if you are, start by speaking quietly to a senior executive or board member in your organization, but carry a big whistle and be prepared to blow it as long and as loud you can.