The dog ate my Census, says ABS

Everything is obvious in hindsight, everyone is an armchair expert, and schadenfreude is an ugly emotion. Reading the Australian Bureau of Statistics (ABS) submission to the Senate inquiry into the 2016 Census is sheer pleasure for all these reasons.

The ABS should be praised for one thing at least.

On census night, the website had already fallen under the barrage of what ZDNet understands were relatively modest distributed denial of service (DDoS) attacks, when monitoring detected an "unusual spike in outbound traffic". That's when they pulled the plug and called in the Australian Signals Directorate (ASD).

Good call. If in doubt, protect the data. But that's about the only noble act in the entire 119-page submission.

The document's main thrust is that the ABS is a trusted organisation that knows how to keep things secret, and that all the problems are IBM's fault. Or the media's fault. Or the public's fault.

The full submission has been pulled from the Senate inquiry website because -- irony alert -- it contained commercial-in-confidence material that should've been redacted. That's probably down to the Senate, not the ABS, but either way it looks bad. Needless to say, someone has posted the unredacted report [PDF] online.

The confidential section -- page 61 onwards if you're following along at home -- explains how the ABS issued a request for tender (RFT) in July 2014, and IBM submitted its tender documents in August 2014.

"The contract required that IBM develop, deliver, implement and host the online Census system in accordance with the SOR [Statement of Requirements] and its response to the RFT, including DDoS mitigation. The contract included specific service level agreement requirements for availability of the Census form overall, with specific targets for Census night (98 percent of time during the peak four hour window) and fault resolution times on Census night (maximum 30 minutes)," the ABS wrote.

"The IBM response also outlined measures to ensure that it would be 'highly resistant to web application security attacks', including DDoS attacks."

So the ABS threw IBM under a bus, and continued to do so.

"During 2016 the ABS had sought and received various assurances from IBM about operational preparedness and resilience to DDoS attacks. In July 2016 the ABS arranged for a meeting with ASD and IBM to receive briefings from ASD on cyber threats and incident response support. The potential for DDoS attacks was discussed, as were general mitigations for a range of threats. ABS does not believe that any new areas of concern were raised, nor were there any suggestions of potential mitigations or additional preparations that were not pursued," the ABS wrote.

"The ABS did not independently test the DDoS protections that IBM was contracted to put in place, as it considered that it had received reasonable assurances from IBM."

I reckon it'd be fair to summarise the conversation like this:

ABS: Will it be good on the night?

IBM: She'll be right.

ABS: You sure?

IBM: Yeah.

This isn't to say there was no risk assessment at all, as the ABS explained.

"The Census Program engaged a recognised Australian and International Census expert to undertake a full Census Program Risk review. The Census Program Board considered the risk review at its June 2016 meeting. One of the program risks was 'Security breach -- online attack' (which included, but was not limited to, DDoS attack). For this risk, the inherent risk rating was 'extreme', the control effectiveness rating was 'good' and the residual risk rating was 'moderate'."

Is reducing the residual risk rating to 'moderate' rather than, say, 'negligible' enough, I wonder?

There's also the question of how the risk management plan was developed.

"The IBM contract required the delivery of a risk management plan for the online Census," the ABS wrote.

Oh.

Not everything that went wrong was IBM's fault, though.

The ABS had decided that its strategy for coping with a high volume of telephone enquiries would be to ask people to call back later. "Call blocking," the ABS called it.

"Media strategies and operations were designed to try to spread calls in order to minimise peak periods," the ABS wrote.

"Early on, both the PFRS [Paper Form Request System] and the CIS [Census Inquiry Service] experienced demand that exceeded planned forecasts. ABS believes this was primarily caused by a number of factors. Unexpected and unprompted media and social media focus on potential of Census fines creating a degree of public fear (as noted previously, Census approach is to not mention fines before the Census night); faster than expected postage of approach letters; and the effectiveness of the Census advertising campaign in drawing the population's attention to the Census. Furthermore, with the unavailability of the online Census, on Census night, significant call numbers were received."

So it's the media's fault for mentioning the fines, Australia Post's fault for delivering letters promptly, the advertising's fault for being effective, and the public's fault for getting scared, or phoning when the website fell over.

Oh, and we can blame the Australia Taxation Office (ATO), who ran the CIS, and contractor Stratum for the PFRS.

But my favourite bit of blame-passing starts on page 83: more blame for the media and other commentators. Here's just part of it.

"The ABS was rarely approached for its perspective on privacy related stories. Some journalists did claim to try to get the ABS view from searching the ABS website, others did not do that (a set of Frequently Asked Questions published on the ABS website are provided at Appendix 12). Other journalists appeared to seek their information from the Australian Privacy Foundation [APF] which had published a webpage titled, The problems with the 2016 Census. The ABS met with the Australian Privacy Foundation about these matters and they continued to reflect their views irrespective of ABS explanations of facts and process. Appendix 13 outlines the claims made on the Australian Privacy Foundation webpage and the ABS response to these claims.

"The ABS is aware of a number of instances where the community benefit of what the ABS was proposing to deliver from the 2016 Census was not able to get a reasonable representation in the media, through editing of material and difficulties some expert researchers had in having their views published."

I can't comment on the ABS' discussions with the APF, although it's certainly their right to be unpersuaded by the "explanations of facts and process".

I can say, however, that some journalists and other writers who did approach the ABS with specific questions were either fobbed off to the FAQ, or given generic responses which didn't answer their questions either, or ignored.

For my own part, I asked the ABS to comment on why it had dropped its claim to hold an Australian National Audit Office (ANAO) "cyber secure zone" rating. I was promised an answer. I didn't get one.

If all this isn't enough for you, technology analyst Justin Warren has posted a deeper dissection of the ABS submission.

The key message here is someting that the information security community has been hammering for years. You can outsource the work, but you can't outsource the responsibility for the risk.

"At no time was the ABS offered or advised of additional DDoS protections that could be put into place. Additionally, no suggestion was made to the ABS that the DDoS protections that were planned were inadequate," the ABS wrote.

Sure, but did you look? Did you even know to look?

"The dog ate my Census," the ABS seems to be saying. That's just not good enough.

ZDNet understands the government is genuinely angry about the Census failure, and wants to put some heads on spikes. I think I know where they should start.