Science Excellence vs Research Impact: Can Researchers Achieve Both?

“A novel experiment” is how former chief science advisor Sir Peter Gluckman described his brainchildren, New Zealand’s National Science Challenges, to Nature in 2013.

The 11 National Science Challenges (NSCs) are transdisciplinary, highly collaborative, mission-led programmes designed to tackle New Zealand’s biggest science-based challenges.

Publishing results in academic journals is not the endpoint for NSC-funded research. Researchers are asked to target outcomes – such as the adoption of new tools or practices – that have a positive impact on our environment, society or economy.

The 11-NSC ‘experiment’ is at its halfway point, and it seems results are promising enough that much of New Zealand’s publicly funded research will increasingly be required to demonstrate its benefit to society, with MBIE’s October 2019 position paper clearly outlining its plans and expectations.

Is the Experiment Working for Researchers?

New research from the Our Land and Water National Science Challenge’s Collaboration Lab builds on the Outcome Spaces Framework – developed by Dena Fam and colleagues at the Institute of Sustainable Futures, University of Technology Sydney – to analyse the experience of researchers in four NSCs doing mission-led research.

The paper’s lead author Ronlyn Duncan, a senior researcher at Manaaki Whenua Landcare Research, says the research highlights the challenges of doing research for impact, the additional time and resources required for quality transdisciplinary and stakeholder collaboration, and the influence of different theories of change on potential impact.

In a workshop setting, NSC researchers were asked to estimate where their NSCs were dedicating resources to achieve what outcomes, and how this differed between what was intended and what happened.

Workshop participants identified several barriers and risks in doing mission-led research, and tensions in achieving science excellence alongside impact within a system that has a narrow definition of excellence.

The research found that NSCs have been predominantly working in two outcomes spaces: knowledge stocks and knowledge flows, but with greater attention on knowledge stocks such as journal articles.

The paper argues that New Zealand’s science system has been built on a knowledge deficit theory of change, which assumes situations and practices will change through the eventual diffusion of academic outputs. It is increasingly recognised, however, this has low potential for impact, but the long-standing legacy is that publication in high-ranking academic journals remains the measure of success across the science system.

“Researchers in the Challenges are struggling with the competing demands of producing academic papers that demonstrate science excellence and doing the work needed to bring about wider impact,” says Ronlyn.

Several workshop participants reported being directed to prioritise academic outputs above requirements for co-design and transdisciplinary collaboration.

“I think it is a novel experiment in many ways, in terms of how best a small country must do science” – Sir Peter Gluckman

“This research highlights that the NSCs provide a really useful experiment in doing research for impact,” says Ronlyn. “However, they are required to operate within an institutional system that rewards publications, with little recognition for what co-designed impact-focused research actually requires.”

“Involving stakeholders is a continuous, time-consuming process. We also need to better support transdisciplinarity, and recognise the skills, resources, money and time it requires to achieve impact.”

“Measuring impact is important, but it doesn’t recognise the extent to which the system needs to change. Budgets and timeframes need to expand alongside expectations.

How the Outcome Spaces Framework Can Help

Ronlyn says that using the Outcome Spaces Framework (with the New Zealand innovations) has a number of benefits. “The framework can be used to kick-off research, see how it’s tracking, or as an evaluation tool. It encourages researchers, research partners and stakeholders to ask – how can we have impact beyond the project, and where should the resources all these parties bring be dedicated to do that?”

The Outcome Spaces Framework can help researchers:

• identify a bigger range of stakeholders and their desired outcomes,
• identify a wider range of potential outcomes for a research bid,
• reveal implicit theories of change,
• prioritise what’s most important (is it getting papers published or spending resources on knowledge flows?),
• see the limits of the project,
• assess impact potential, especially by comparing intended, actual and desired outcomes,
• think beyond the journal paper to what is done with it, and
• think beyond the project boundary about how to make a change into the future.

___

For more information:

• Assessing research impact potential: using the transdisciplinary Outcome Spaces Framework with New Zealand’s National Science Challenges (Kōtuitui: New Zealand Journal of Social Sciences Online, January 2020)
• How to apply the outcome spaces framework in practice (Mitchell C, Fam D 2020. Outcome spaces framework. Swiss Academies of Arts and Sciences: td-net toolbox for co-producing knowledge)
• The Collaboration Lab
• The Impact of Research: Position paper (MBIE, October 2019)