Leaked paper reveals Australia's obsessive metadata secrecy

Leaked paper reveals Australia's obsessive metadata secrecy

Summary: The Australian government has been discussing a detailed data-retention wish list with internet service providers for more than four years — with citizens kept in the dark.


Last Friday, the Australian Attorney-General's Department sent internet service providers (ISPs) a confidential discussion paper — subsequently leaked to Fairfax Media — that attempts to clarify exactly what metadata they'll be required to store under the government's proposed mandatory data-retention scheme. The detailed requirements are presumably designed to feed into the "statutory specification" of metadata that will be included in legislation to be introduced to parliament in coming weeks.

Until now, the only official government description of metadata we'd seen — apart from that breathtakingly confused TV performance by Australia's favourite Attorney-General Senator George Brandis QC — was the hilariously inadequate one-pager (PDF) that the Attorney-General's Department (AGD) tabled in Senate Estimates on October 15, 2012, after much prodding by Greens Senator Scott Ludlam.

You might therefore think that the description of the government's metadata needs in Friday's document was a recent development.

You'd be wrong.

A confidential document obtained by ZDNet shows that even more detailed descriptions of the government's data-collection ambitions had been discussed with ISPs as far back as early 2010.

The document, Carrier-Carriage Service Provider Data Set Consultation Paper version 1.0 (PDF), is a 16-page PDF file created on March 9, 2010, at 14:49. Its core sections are similar in structure to the nine-page document obtained by Fairfax Media this week, with the addition of tables of "sample data to further illustrate the expected type of data to be retained for each specific retention requirement from the data set", discussion questions for industry to answer, and an introductory background section rather than an executive summary.

The 2010 version of the document was quite specific about the data to be collected. For mobile calls, for example, the data would include the IMSI and IMEI of both the calling party's and called party's devices, whereas the current version simply specifies the "identifier(s)" of the devices. This is in line with the government's intention to make the legislation technology neutral.

References to web-browser sessions and file transfers that were in the 2010 version have vanished, too, in line with such ideas being dropped as the data-retention debate has evolved.

The document made clear that it was a "proposed data set" and a "basis for dialogue".

Industry was asked to comment on eight questions:

  1. Which elements of the proposed data set are presently retained? For those retained, how long are they retained for, and for what reason?
  2. How much storage space is required to store data currently being retained?
  3. Is the majority of your network equipment ETSI LI compliant?
  4. Which requirements of the proposed data set are presently not retained?
  5. Are there major technological changes required to retain any of the requirements of the proposed data set? If so, what are they?
  6. Should data retained under this regime be available to the C-CSP for commercial purposes?
  7. Should a mandatory data-retention regime apply to all telecommunications industry participants?
  8. Are there significant issues associated with a 12- to 18-month lead time for full implementation of a data regime? If so, what are they?

This document had previously been released under Freedom of Information in 2010 — but with 90 percent of the content blacked out, including the entire description of data to be collected. Disclosure of the document uncensored "could be misleading to the public and cause confusion and premature and unnecessary debate," wrote a legal officer in the AGD's FoI and Privacy Section, Claudia Hernandez, at the time. "As the matters are not settled, and proposed recommendations may not necessarily be adopted, release of such documents would not make a valuable contribution to public debate."

It's those redactions that make this leaked, unredacted version most interesting. The description of the data set to be collected, yes, but also certain glossary entries, the question of whether ISPs would be able to make commercial use of the data, the proposed time frame — even the rather obvious points that metadata "can be used to reveal associations between members of criminal organisations" and "that new requirements be introduced to ensure that the telecommunications data currently retained continues to be available for law enforcement and national security purposes".

Even the headings "A. Data Set" and "B. Data Set Explanatory Statements" were redacted, as well as the part of the glossary entry for BRAS, a Broadband Remote Access Server that said it can "provide unique identifiers such as IP address to subscribers" — which is really just a straightforward description of what a BRAS does.

Surely there comes a point where secrecy goes beyond protecting law enforcement and intelligence methods, and beyond preventing "premature and unnecessary debate" — assuming such prevention is a desirable thing in a democracy — and becomes obsessive.

Topics: Security, Government AU, Privacy


Stilgherrian is a freelance journalist, commentator and podcaster interested in big-picture internet issues, especially security, cybercrime and hoovering up bulldust.

He studied computing science and linguistics before a wide-ranging media career and a stint at running an IT business. He can write iptables firewall rules, set a rabbit trap, clear a jam in an IBM model 026 card punch and mix a mean whiskey sour.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • It's a nonsense !

    Until such time as the AG &/or his bureaucrats stop redacting what they are intending or trying to get from the data they want retained, let's stop messing about with the nonsense.

    I for one, want to know what my democratic government is intending. They are elected BY the people FOR the people, which seems to me to be something they all consistently forget.
    • By the people for the LOBY

      Big Business needs our Data for our protection.
  • Lazy spies

    Welcome to the world of lazy spies.. The spies simply want to use big data to find some criminal targets. Setting up a data retention scheme puts in place the mechanismi to feed that big datamart. Rest assured that over time the definition of "metadata" will be gradually changed to inlcude just about every data element flying between ISPS and customers. Then the spy agencies can get lazy and rely on catching the crims from the data stream. Then, when the eventually realise they are not catching anything, except a bunch of innocent internet users it will be too late. Nothing replaces traditional methods of intelligence. The investigation into the NSA spying showed, it was proven that all the data gathered did not result in more or better intelligence on criminals or terrorists. This will simply be waste of taxpayers money and the users of the internet will carry the cost through their subscription fees. It's sad it has come to this.
  • Emo

    Give the government an inch an they will take a yard. They will never tell us the whole story behind their metadata collection policies and will when questioned on it will simply say they cannot discuss "operational matters." If we don't speak up now and get the whole story of what they are doing, we are giving them an open check to do whatever they like.
    It's crucial to have the right checks and balances in place so that whatever metadata they do collect cannot be abused and judging by the way the AFP have just revealed sensitive information in relation to metadata in a criminal investigation, it doesn't paint a very good picture as to how safe this information will be and who will have access to it. This will be a massive invasion of privacy as we are fast becoming a police state.
  • Scumbags

    Why is it that each successive government is more corrupt, evil and/or incompetent than the previous one?