Responsibility in research evaluation practices:
lessons from a global discussion
SUPER MoRRI is a European Union-funded project endeavouring to develop a monitoring and evaluation system for Responsible Research and Innovation (RRI). One of the project’s core concerns is the inclusion of responsibility into evaluation practices, and that was the central theme of an international event organised by the project team throughout April 2021. Even though the current pandemic has created the need for the event to take place in a virtual setting, the digital alternative led to an experience that would not be possible otherwise. Instead of the full-day seminar originally planned, four webinars were promoted across different continents, exploring responsible evaluation from a global perspective.
Enabling mutual learning over RRI and RRI-like activities beyond Europe, the first three webinars were focused on the Americas, Asia/Pacific and Africa/Middle East. Each one of these consisted of a plenary session with three country-based panel statements about responsible research evaluation practices from a local perspective. Those presentations contributed to inspire subsequent discussions in small breakout sessions with invited participants from research funding organisations, research performing organisations and policy-makers.
The Americas session had Brazil as the initial focus, counting with presentations from André Brasil (a SUPER MoRRI team member also affiliated with the Brazilian Agency for Support and Evaluation of Graduate Education – CAPES), Laerte Ferreira (from the board of directors of the Brazilian Forum of Pro-Rectors for Research and Graduate Studies – FOPROP), and Odir Dellagostin (president of the Brazilian Confederation of State Funding Agencies – CONFAP). The Asia/Pacific session was centred around China and the presentations from Yang Yun (National Centre for Science and Technology Evaluation), Lin Zhang (Wuhan University) and Junpeng Juan (Chinese Academy of Sciences). Finally, the Africa/Middle East session counted with presenters from South Africa: Nelius Boshoff (Stellenbosch University), Therina Theron (president of the South African Research and Innovation Management Association – SALIMA) and Rocky Skeef (executive director for Reviews and Evaluations at the National Research Foundation of South Africa).
You can find more information about the panels and presentations, including the slides used in the three distinct sessions, following the appropriate links below:
The three regional sessions, combined, recorded nearly 100 participants from thirty different countries, such as Australia, Austria, Botswana, Colombia, Germany, Ghana, Japan, Kenya, India, Iran, Mexico, Netherlands, Rwanda, Spain, and the USA. Among the benefits of the broad representation was not only the exchange of experiences from distinct countries but also the inception of a stakeholders network that may influence research and innovation through responsible evaluation practices.
From the discussions in those meetings, a reflection webinar was organised to bring ideas together with the SUPER MoRRI team and its ecosystem of partners, country correspondents and connected projects. The debate was also enriched with the contribution of representatives from DORA (Anna Hatch), ENRESSH (Michael Ochsner), OECD (Fernando Galindo Rueda) and UNESCO (April Tash), who brought their own perspectives on responsible evaluation practices not only to those who attended the regional webinars but also to many other participants who joined the final discussion. Information on the program and presenters at the reflection webinar can be found here:
Different realities, converging perspectives
From the presentations and fruitful discussions taking place during SUPER MoRRI’s annual event, we can clearly see significant differences between the social, economical, political and geographical characteristics of participant countries around the world, something also reflected in the development stages of their science systems. Despite that, we noticed that recurrent themes appear when responsible evaluation is concerned. A qualitative study is being performed over the rich exchange of experiences throughout the webinar series, but we can already list a few of these converging topics.
Move away from metrics, but beware of subjectivity
The use of indicators that universalise information seems inappropriate across the board. When we speak about a system of responsible practices, no country wants to use or produce information that is damaging for cultures or that encourages horse races between regions to satisfy indicators. Despite that, several countries seem to have a valid concern regarding subjectivity in evaluation. This is particularly true for places where professionalized research management and evaluation practices are still in their early stages of development. For them, metrics are insufficient to take into account the complexity of national knowledge production systems, but they are still in demand to justify the distribution of limited funds for science.
Change takes time
Most countries participating in the SUPER MoRRI annual event agreed that, when it comes to responsible practices in research, it is not particularly difficult to change policy; the problem is changing the culture. While new legislation and regulations can impose change, they must be aligned with people’s motivations towards responsibility, and the transition needs to be incremental. Abrupt course corrections might be able to impose compliance but, as officials from different continents clearly state, excessive pressure often leads to research integrity problems.
Change is expensive
Implementing responsible practices of evaluation in research requires a considerable investment in terms of financial and human resources. Funding is a very important part of this equation especially when it comes to training research managers and high-level evaluators for science systems across the globe. The concept of evidence-based policy applied to the scientific environment only makes sense if people are qualified to produce and consume the evidence, and there are substantial costs of both time and money to make that happen.
Distinct realities, different purposes
Any movement that proposes more standardized evaluation systems, in special those based on internationally established indicators, is seen with fear where responsibility is concerned. Many evaluation models are based on auditing or checkbox-ticking approaches that rarely generate positive impacts on local science. Evaluation is only responsible if it introduces more active engagement of researchers and stakeholders in the evaluation system, so they can add perspective to indicators and promote valuation of locally relevant research in parallel to the internationalized output expected from scientists. In that sense, the distinct reality of each country must be taken into account to determine what it means to be responsible in that environment.
These are, of course, only initial observations from the discussions in the SUPER MoRRI annual event, but each of them will benefit from deeper investigations that you will be able to find in future blog posts. Keep tuned!
- June 1, 2021
