Census 2016: A case study in the confluence of failure

With continuous cuts to its funding, alongside an arrogant, dispassionate attitude towards the community and a systems provider that couldn't get the simple things right, it's little wonder the 2016 Australian Census turned into an absolute debacle.

As the official inquiries and reviews into what happened last Tuesday night are set to play out in the coming weeks and months, few players in this game are likely to exit it without blame.

"It's up to the government to accept responsibility," said opposition frontbencher Anthony Albanese last week, wheeling out a well-worn attack on the government, but it held a kernel of truth.

"It's the government that have made cuts to ABS staff and the ABS budget. It's the government that failed to explain the changes that were happening prior to Census night. It's the government who said it was all going well."

In February last year, the Australian Bureau of Statistics (ABS) thought about moving to a 10-year Census cycle, such were its IT woes and need to save money after "efficiency dividends" were imposed on it by governments of both stripes.

Former chief statistician Brian Pink warned in the 2013 ABS annual report that ageing infrastructure and reduced budgets from the government had the potential to "seriously compromise" the agency's sustainability. It also certainly did not help that Pink's role as chief statistician was left vacant for over a year.

As detailed in Fairfax Media over the weekend, the ABS said in February this year that it was not able to meet the Census timetable and budget, and consequently took the decision to retain names and addresses for four years to cross-link and squeeze monetary value out of the data the agency was collecting.

The ABS was attempting to cast itself as the government's data integration agency.

"We continue to collect names, as we have done in previous Censuses, in order to produce accurate population numbers and better estimate Indigenous life expectancy," current ABS chief statistician David Kalisch said in July. "In 2016, I've decided to keep names and addresses for longer. This is for statistical purposes only, and will increase the value of Census data.

"Names and addresses will be permanently destroyed by August 2020."

For some in the community, the increased retention of personal data was the independent statistics agency using the tools available to it to enrich its data and provide better service to government. For others, it was the greatest and most widespread privacy intrusion in living memory.

But rather than stepping in to explain its decision and hose down privacy and security concerns, the ABS let it fester, and hid behind a thinly veiled arrogance that it knew best and everything would work out fine. At the same time, the Australian government would install three different overseeing ministers in the space of the year leading up to the contentious Census survey, and do less than the ABS to address concerns.

By the time Tuesday, August 9, rolled around, politicians were openly boycotting the Census, and it looked like the lack of communication from the ABS would poison the data it desperately wanted.

Had everything gone off without a hitch, the ABS might have got away with it, and privacy concerns about the amount and nature of data collected by governments have left the national conversation, but it did not.

At this stage, an overabundance of technical incompetence soon matched the exposure levels previously exhibited in the institutional and communication areas -- the "confluence of events" was ready to turn the Census website into a dumpster fire.

Technical incompetence was already in the system; it wasn't something new. The fact that in 2016, a multimillion-dollar national survey cannot handle the name Zoë without some sort of compromise is a disgrace.

In the washup of the Census site collapsing on Tuesday, Special Adviser to the Prime Minister on Cyber Security Alastair MacGibbon pointed out the damningly obvious.

"There are no absolutes in cybersecurity; anyone that gives you absolutes is not fully understanding the cybersecurity game," he said.

Whether on social media, or briefing the minister responsible, the ABS had no qualms whatsoever with dealing in absolutes and boasting of its security posture.

The true failure of Tuesday night, should reports be true, was the ABS being spooked enough by a series of small denial-of-service attacks to rip the Census site down due to seeing data leaving the datacentre and heading overseas. That the outbound data is said to be logs and other analytical information, not personal data, tops off a series of failures by the agency and its contractor, IBM.

An angry prime minster fronted the media on Friday, promising that heads will roll over the debacle.

"The fact is that the service provider for the Australian Bureau of Statistics, IBM, did not put in place sufficient measures to deal with an entirely predictable circumstance: Denial-of-service attacks," Malcolm Turnbull said. "That's a fact. So there was a failure in provision."

As the recriminations over the failures begin, and a Senate inquiry looks likely to happen along with the MacGibbon probe, by no means is the quest for privacy in any better shape than it was before.

The ABS has been reduced to a laughing stock, but governments are still cooking up schemes to cross-link more data than ever.

Malcolm Turnbull's agile agency of innovation, the Digital Transformation Office (DTO), is set to build a federated identity hub for federal government that could potentially work with state governments and banks.

For all the furore and theories that appeared on social media before the Census, decrying the establishment of an ABS-created national identity scheme -- something it seems the Labor party is completely onboard with -- the true identity scheme is currently being developed by the DTO.

At the same time, in New South Wales, the state government is undertaking a data analytics project in South Sydney to determine who lives where and with whom. By feeding in data such as utility connections and disconnections, and rental bonds, the NSW Data Analytics Centre (DAC) wants to get down to an update interval of 30 minutes.

If you are worried about the ABS holding personal data for four years, then the DTO and the DAC, mandatory telecommunications data retention, and the other silos within government should have the alarm bells ringing loudly and often.

Most concerning, even though the ABS has shown its lack of technical nous in the recent past, it is still rated higher than many other agencies in being able to deal with external threats.

It is a damning assessment of those departments and organisations rated lower, but no less than what experience has shown with the state of government IT in this country.

Whether it is the IBM-led billion-dollar disaster with the Queensland Health payroll system, or the ongoing efforts to replace the 30-year-old welfare payments platform, or the attempts at online services such as MyGov or e-health records, no IT failure has been this public for a long time.

If any good is to come from the Census meltdown, it will be because the opportunity is taken to have a look at the decrepit legacy IT systems used across all forms of government, and to consider the amount of personal data that is used and collected within the public sector, not just at Census time.

ZDNet Monday Morning Opener

The Monday Morning Opener is our opening salvo for the week in tech. As a global site, this editorial publishes on Monday at 8:00am AEST in Sydney, Australia, which is 6:00pm Eastern Time on Sunday in the US. It is written by a member of ZDNet's global editorial board, which is comprised of our lead editors across Asia, Australia, Europe, and the US.

Previously on Monday Morning Opener:

• The dirtiest little secret about big data: Jobs
• Windows 10, Android, iOS, and the billion user question
• With AWS on deck, cloud rivals eye margin improvements, platform plays
• Why Trump and Brexit are an indictment of the tech industry and its worldview
• The internet as battleground: Where do we go from here?
• Apple winning the enterprise security race, Samsung makes push
• Tech after Brexit: Where do we go from here?