Mandatory metadata retention needs to go ahead, the Australian Security Intelligence Organisation (ASIO) told Senate Estimates on Tuesday night, even though almost all of the collected data will never be accessed by law-enforcement or intelligence agencies.
"We will not, and never will, use the majority of this data, but unless the data set is complete, then it would be an incomplete exercise when we are going in on a lead," ASIO director-general Duncan Lewis said. "You wouldn't know what you had missed out on and what you hadn't."
Under questioning from Senator David Leyonhjelm, Lewis conceded that VPN usage would weaken the completeness of the data retained, but said it is important nevertheless.
"I understand the technological issue, but for the data to be as complete as possible, it's necessarily for us to fully understand networks -- and it is largely about networking," Lewis said.
"Who is talking to who is a very important part of investigation. For us to have what I would describe as the best understanding of a breadth of a network, then you would want to make sure that you had at least the best dataset that you could.
"95 percent, 99 percent, whatever the figure is, which will be of no particular relevance to the investigation."
Lewis' comments that most of the data will be unused come in contrast to comments made by his predecessor, David Irvine, who said that metadata retention is not "some great mass surveillance exercise" when the legislation was first introduced to parliament.
"It is very, very carefully targeted against those people who give us good reason to suspect that they may be engaged in activities which are a threat to national security and the lives of Australians," Irvine said at the time.
Not only is metadata retention famously like an envelope, Lewis also said the storage of metadata is similar to driver's licence and car registry databases.
"In very much the same way that we have driver's licences, I mean, should we only have driver's licences for bad drivers? You can't predict these sort of things in advance, so we do hold databases in our community -- of driver's licences, car registration, and so forth -- of people who are just going about their normal business," he said.
"But from time to time, that data set is inevitably useful to police officers who might be looking for a stolen vehicle, or whatever. The analogy is quite good around that."
Over the weekend, Australian Federal Police (AFP) assistant commissioner Tim Morris said that an individual's right to anonymity should play second fiddle to the AFP's right to access metadata without a warrant if that anonymity leads to unlawful activity.
"The AFP understands the importance of individual privacy, and supports this as a fundamental right in Australia," said Morris. "However, the AFP cannot support the right -- if you like -- to anonymity, especially when it becomes related to unlawful activity."
Lewis also backed warrantless access to metadata, and said that needing a warrant for access would slow down investigations.
"To have a warrant system would be unwieldy for all sorts of practical reasons, but obviously systems are not there for our convenience," Lewis said. "Our reaction time, our responsiveness, our agility as an organisation would be diminished -- diminished to a point that would have a significant impact on the way we do business."