What does it take to create a responsible research funding culture in science?
Research funders are central to ongoing debates about better ways to govern the science system. Even more so, their powerful position as the principal actors distributing funds to researchers has drawn particular attention to the logics that govern critical funding practices. Such practices include identifying and supporting ‘best quality’ research, defining and operationalising research priorities in the light of global societal challenges, and responding to movements and transformative practices such as open science and citizen science.
The dynamic nature of the current funding environment is intensified by the numerous reform initiatives in train and the diverse actors engaging with defining, debating, and agreeing on issues around (responsible) research assessment, and the criteria that determine who and what is funded. In the first half of 2022, the European Commission, in collaboration with Science Europe (representing 34 European funders) and the European University Association, launched a report and a process for an agreement on reforming research assessment. These efforts have been channeled into the establishment of a Coalition of Advancing Research Assessment (CoARA), which will be set up before the end of 2022. It follows a lengthy process of engagement among numerous actors in the research system. A set of recommendations on research assessment was developed by Science Europe based on consultation with its members and beyond. The Research on Research Institute published results of a survey and best-practice consultations about the changing role of funders in responsible research assessment. These initiatives of course build on prior community-driven efforts to improve researchers’ assessment, including DORA, the Hong Kong Principles, and the Leiden Manifesto.
It was in this light that SUPER MoRRI invited research funders, research performing organisations, and policymakers from various European countries to come together to respond to a central question: what does it take to create a responsible research funding culture in science?
This event was scaffolded by thinking developed through the work carried out in the SUPER MoRRI project that asked the question; how do research funders exert ‘responsibility pressure’? Building on the participation of 55 public and private funders in this work, our Third Annual Event was used as an opportunity to reflect on findings, explore issues and identify trends with actors affected by and involved in the designing and transformation of research funding. In what follows, we describe two observations from this event. These observations relate to how, together, these actors address a research funding reality in need of change and their motivations in doing so.
In a prior briefing to the invited speakers, a strong emphasis was placed on critical reflection on ongoing practices of research funding organisations: what are the implicit assumptions in grant application processes? What knowledge or contribution is (not) included in current research assessment practices? Quickly, these questions led to an open conversation about the potent and invisible categorisations of everyday funding practices; allowing us to render them visible for scrutiny during our event.
Quickly, dominant categories of research funding, such as the standard practice of an academic CV with long publication lists or aggregate metrics (e.g. JIF or H-index) in the context of research(er) assessment, and their sociotechnical formations, became issues for deliberation. By valuing a publication-driven assessment of research(ers) and luring evaluators to using these metrics or standard academic CVs as ‘shortcuts’ or proxies for ‘quality’, these categories can become antithetical to fair and transparent research assessments. When embedded in funding assessment processes, categories such as the journal impact factor (JIF) do not only hold considerable power over researchers, but also perform what should be deemed worthy (by evaluators). In turn, researchers too can succumb to the pervasive power of these instruments, designing research processes in relation to their potential productive yield of published articles rather than considering other criteria for scientific quality such as societal relevance or epistemic originality.
Fostering experimentation
Both researchers and funders alike agreed that allowing for different forms of experimentation is a necessary prerequisite for finding alternatives to practices that are deemed questionable for fostering a responsible funding culture. Research (funding) organisations should therefore, as Sean Sapcariu from the Luxembourg National Research Fund stressed, invest the time and resources needed to deliberate and tinker with new approaches and share learning from case studies or experiments. Uncertainty about which elements of research assessment approaches are valuable to retain, albeit possibly in a reformed format, and which should be changed entirely, permeated these deliberations.
Circulating ideas include moving from aggregated to contextual metrics in research evaluation, or to emphasizing the rewarding of teams rather than individuals. Rewarding a wider scope of scientific contributions in order to recognise collective work – especially with regards to early career researchers, was a concern that Marta Agostinho from EU-Life highlighted strongly.
James Wilsdon from the RoRI Institute underlined the Swiss National Science Foundation’s experimentation with randomisation of funding allocation and evaluation as an alternative approach to decision-making. The funder has used such lotteries as tie-breaking instruments when it came to selecting between two applications of similar quality in order to eliminate decision-bias.
Another idea that assumed a prominent role was narrative CVs in the discussions around fair and transparent researcher assessment – an approach that has been put into practice by NWO, the Dutch Research Council.
As became very evident, there is no lack of ideas to address the challenges that research funders (and the research system more generally) face. The question remains: what can different actors – and research funders especially – do to take concerted steps to foster more responsible research funding cultures?
From experimentation to responsible funding practice
Stephen Curry from the San Francisco Declaration on Research Assessment presented two clear recommendations. First, he stressed the need for research funders to develop “credibly holistic approaches to research(er) assessment” by adopting narrative CVs, rewarding team-building and leadership, and by taking a hard line on bullying and harassment. Also, he encouraged researchers to exploit their position in order to incentivise and support open science practices by, for example, recognising the value of preprints in funding applications and supporting data sharing. The second recommendation was to build trust in assessment procedures. This could be done by opening up the processes and providing a picture of how and what information is used in the assessment of applications, and by bringing diverse perspectives from non-academic users of research into evaluation panels.
Whilst these recommendations speak directly to research funders, it was noted that researchers are not powerless to drive reform. For instance, the newly launched initiative Publish Your Reviews allows researchers to exert demands of more open assessment processes towards publishers in the context of peer review.
SUPER MoRRI’s own work with research funders showed that a majority do have multiple responsibility-related policies, many of which are included in wider strategies. Other funders aspire to moving in responsible directions, within the constraints of their own institutional autonomy. Societal stakeholders were found to be present in a majority of RFO’s formal advisory bodies. Many funders are including non-academic stakeholders in assessment processes and some are providing guidance or even training on key aspects of responsible assessment such as gender bias. Further results from SUPER MoRRI’s work with funders about their efforts to cultivate a responsible funding culture will emerge throughout the remainder of the project.
But to conclude for now, we would like to reiterate Stephen Curry and Sean Sapcariu’s point that knowledge sharing, such as disseminating best-practice case studies, and embracing further collective learning opportunities is integral to not only fostering experimentation, but also to advancing more responsible funding practices and cultures in the medium- and longer-term.
- October 18, 2022
