Audit Office wants innovation agenda evaluation strategy finalised
After probing the effectiveness of the design process and monitoring arrangements for Australia's AU$1.1 billion National Innovation and Science Agenda, the Australian National Audit Office (ANAO) has asked the government to finalise its evaluation strategy.
In Design and Monitoring of the National Innovation and Science Agenda, ANAO made two recommendations, with the first calling on the Department of the Prime Minister and Cabinet (PM&C) and the Department of Industry, Innovation and Science to review and update their policy development guidance and training materials.
ANAO wants the updated literature to be "fit for purpose" for the range of activities undertaken, including cross-entity taskforces; clearly articulate an acceptable standard of analysis and evidence; and include mechanisms to provide assurance that the guidelines are consistently applied.
The report [PDF] also asks that the Department of Industry finalise the evaluation strategy for the National Innovation and Science Agenda, and establish formal monitoring arrangements with relevant entities, so that the results of evaluation activities can be used to inform advice to government on future measures and the continuation of existing measures.
"An evaluation framework was developed but not in a timely or fully effective manner," ANAO wrote in its report.
"Limited advice was provided to government during the design process about the specific impacts of the agenda, and how or when they were to be measured. While evaluation arrangements were progressively developed post-announcement, there were delays and issues associated with the identification of suitable performance measures and data sources."
The billion-dollar innovation and science agenda was announced in December 2015 by Prime Minister Malcolm Turnbull and covers more than 20 measures centred on his "ideas boom" rhetoric.
The innovation agenda has four principles: Building the science culture and capital; strengthening collaboration; encouraging science and innovation talent; and the government leading as an exemplar.
According to ANAO, implementation and evaluation readiness of the strategy might have been improved if clearer descriptions of each measure's objectives, particularly outcomes and outputs, had been developed as part of the agenda.
"This would have enabled earlier development of evaluation material and in some case provided greater clarity on deliverables," it said.
Interested in the design process and monitoring arrangements of the agenda, ANAO said it has mostly been effective.
"The design process for the National Innovation and Science Agenda allowed the government to make decisions within short timeframes, and the monitoring arrangements have, in most respects, been effective," the ANAO report explains.
"The quality of advice to government could have been improved by a better articulation of the evidence base and likely impacts of the proposals, including the likely net benefits of the overall AU$1.1 billion in proposed expenditure."
ANAO said the design process for the NISA was timely in supporting a government decision-making process; however, aside from some sector-level material and some guidance on the development of policy available within PM&C and the Department of Industry, ANAO said it is not evident how this material was applied to the work of PM&C's innovation taskforce or to the input provided by entities.
"The ANAO observed variability in the quality of the advice provided," it said.
"However, much of the advice was general in nature and did not present quantitative or in-depth analysis of problems, expected impacts, or how outcomes would be measured."
Offering advice to the government, ANAO said accountable authorities should implement a framework that supports the development of quality policy advice to government and clear accountability for the provision of that advice.
"Good practice is to provide advice on implementation risks, and prepare an implementation plan in the design phase; or if that is not feasible, early in the post-announcement period, prior to actual implementation," the report explained.
The audit office was also happy that the oversight bodies provided regular and generally clear advice to government on the status of measures and the risks of not meeting milestones or announced timeframes.
The audit cost approximately AU$319,000.