Australia should have a mandatory metadata-retention scheme that covers all telecommunications providers, an advisory report from the Joint Parliamentary Committee on Intelligence and Security on the Telecommunications Act has recommended.
Among its 38 recommendations, the committee backed the establishment of a two-year period for metadata retention; the government making a "substantial contribution" to the costs of creating the regime; and the data set to be retained and agencies able to access the metadata being set out in legislation.
PricewaterhouseCoopers analysis showed upfront capital costs of the retention scheme would be between AU$188.8 million and AU$319.1 million, with small service providers to receive a relatively larger share of funding to handle the costs than larger providers.
"The committee accepts that it may not be in the public interest for government to fully fund the costs of implementing data retention in all cases," the report said.
"The committee expects that national security and law enforcement agencies will continue to contribute to the operational costs associated with accessing data under the scheme under the existing 'no profit, no loss' arrangements."
Although the committee recommended curbing the declaration powers of the attorney-general in relation to what falls under the definition of metadata, under the recommendations, the attorney-general would still be able to unilaterally make changes to what is retained and who could access the data in "emergency circumstances". Any such declarations made by the attorney-general would expire after 40 sitting days of parliament, with amendments to legislation to be brought before parliament within the 40-day window.
"The set of telecommunications data that service providers will be required to retain is central to the operation of the proposed data retention regime," the report said.
"It is critical that industry and the Australian public are assured that the data set proposed comprises that which is necessary and proportionate, and that safeguards are in place to monitor any future proposals to amend the data set."
"Currently the Committee does not see a situation where emergency changes to the dataset may be required."
After much lobbying, the Australian Securities and Investments Commission and Australian Competition and Consumer Commission will be able to access retained data, along with criminal law-enforcement agencies.
The committee recommended that civil litigants be prohibited from accessing data held as part of the metadata regime; however, the committee also recommended the creation of a regulation-making power for exceptions to this prohibition.
Individuals would have a right to access their own metadata under the scheme, with telcos being able to charge for the cost of servicing such requests.
Telcos would have to encrypt any metadata retained, with the committee recommending the creation of encryption standards regulation. Other "robust security measures" would be authorised in limited circumstances where encryption was technically difficult on existing systems.
In the event of a security breach of a metadata store, the telco would be compelled to provide notification of such an event.
"The Committee considers that a mandatory data breach notification scheme would provide a strong incentive for service providers to implement robust security measures to protect data retained under the data retention regime," the report said.
The committee recommended breach-notification be in place by the end of 2015, prior to the start of the retention scheme.
The thorny issue of accessing the metadata of journalists to identify a source was kicked down the road, with the committee recommending a report back to parliament on the issue within three months.
The legislation is due to return to the House of Representatives on Tuesday, with passage assured after Labor backed down earlier on Friday and said it would pass the legislation.
Updated at 5.15pm AEDT: Article previously said the retention scheme would begin in 2015; this is the date for the breach-notification scheme.