One size doesn’t fit all: Evaluating SwafS projects with questionnaires

One size doesn’t fit all
Evaluating SwafS projects with questionnaires

Questionnaires often seem simple and straightforward means of monitoring and assessment to grasp ongoing changes and engage with people. How complex this evaluation practice turns out to be in practice!

Tjitske Holtrop from the SUPER MoRRI project talked about these issues with representatives from two SwafS projects: Nhien Nguyen and Eran Vigoda-Gadot from SeeRRI and Alessandro Deserti, Francesca Rizzo and Felicitas Schmittinger from SISCODE.

What topics does your project deal with?

Nhien (SeeRRI)

SeeRRI is about co-creating sustainable regional innovation ecosystems together with a variety of regional actors. We are developing a framework and a set of core principles, using the RRI approach, which territories, regions, and other social enterprises can use as guidance in this co-creation process.

We are working with quadruple helix stakeholders in three specific territories in Norway, Spain, and Austria to figure out how best to integrate RRI principles into the smart specialization policies of the regions and how to engage citizens in a democratic process. At the same time, all 12 partners in SeeRRI are also committed to implementing RRI within their own organizations. We want to achieve transnational learning that ultimately encourages wider societal adoption of RRI.

Alessandro (SISCODE)

There are similarities with SISCODE. SISCODE explores the different cultural, organizational, institutional, and regulatory conditions under which co-creation flourishes. This means we consider how diversities among stakeholders favour co-creation, what kind of challenges arise and what kind of co-created solutions produce impact.

We want to understand the conditions for fruitful co-creation and incorporate them into STI policy-making, thereby facilitating organizational and institutional change. We ran 10 pilot co-creation experiments across Europe to explore the potential of bringing together citizens, stakeholders and policymakers.

How did questionnaires play a role in your projects?

Eran (SeeRRI)

We used questionnaires to evaluate the activities carried out in the three pilot territories, along with an assessment of their impact at the EU level. The results inform the previously mentioned guidelines we are developing suggesting conditions and governance structures for R&I ecosystems and a roadmap tool identifying RRI openings in the Smart Specialization process.

We gathered data through two different surveys as well as informal evaluations, including workshop participants’ impressions, subjective observations, and structured face-to-face interviews. We believe that mixing qualitative and quantitative data provides us with a more complete and holistic evaluation that can be transformed and adopted in future projects.

We designed two surveys. The first was to evaluate the direct impact of the SeeRRI project on the organizations in the consortium to be filled out by the project partners. The second survey evaluated the activities initiated in the SeeRRI project with stakeholders from outside the consortium. The surveys were complementary and provide valuable information on the integration of RRI principles in practice.

Francesca (SISCODE)

The set-up of SISCODE’s questionnaire is somewhat like the first one mentioned by Nhien: It was created to assess the 10 pilot experimentations conducted within the project. Likewise, it focused less on direct outputs of co-creation, but on broader project outcomes in terms of changes within the organizations and their practices.

To track these changes over a longer timeframe, the same questionnaire was submitted three times throughout the experimentation. Furthermore, it was also supposed to trigger self-reflection and awareness of the issues raised in the questionnaire.

What kind of issues did you encounter with these questionnaires and how did you solve them?

Felicitas (SISCODE)

At first, the overall responsiveness to the questionnaire was quite low and respondents found the project’s dynamics too complex to be really evaluated. We recognized this and decided to change the scope from judging the success of the single pilots to more general insights about co-creation and RRI. Highlighting the benefits of using the questionnaire as a self-assessment and reflection tool helped to get more stakeholders on board.

Nhien

The first year’s questionnaire was focused on the MoRRI indicators addressing the five RRI keys (gender, ethics, open access, science education, public engagement). In the second year, SeeRRI engaged with the territorial level more, which brought out the RRI process dimensions (anticipative, reflexive, inclusive, responsive) and made the nationally oriented MoRRI indicators less suitable. As a result, we revised our evaluation approach and focused more on qualitative tools, maximizing the benefit of contextual and open feedback information. The adaptation of the scope of the questionnaire is similar to what happened in SISCODE.

What did you learn from working with these questionnaires? 

Alessandro (SISCODE)

Like SeeRRI, dealing with these questionnaires taught us how to be flexible and adaptive. Gathering relevant and effective data requires a skillfully designed survey, attention for the collection process, and the continuous mobilization and motivation of the respondents. Even though questionnaires can yield big amounts of data and do not require face-to-face meetings, we could have used a more responsive method for the smaller samples we dealt with in SISCODE. Interviews would have allowed us to zoom in on interesting points and immediately avoid misunderstandings. Even if our questionnaire did not aid comparison and quantitative evaluation (which I believe is difficult for RRI, because of its context-dependency and uniqueness) we got a lot of precious insights and information from it.

Nhien

In SeeRRI we also believe that contextualization is very important in any evaluation practice. We learned that the first step is to understand the cultural and environmental context of the cases and stakeholders involved. What are the regional specificities, and what would success look like for the region? Second, a co-creative dialogue between the evaluation team and territorial partners is crucial. This should be an ongoing and mutual learning process to ensure the evaluation captures what it needs to capture. Finally, we think an evaluation of RRI should follow the RRI principles of being reflexive and adaptive. Evaluations should be subject to revisions and adjustments depending on context and strategy while not forgetting the underlying causal narratives.

Thanks, Alessandro, Francesca, Felicitas, Nhien and Eran, for these interesting reflections. Your projects are as interesting as they are complex. An important lesson to take from your cases seems to be the need to “RRI-ify” the evaluation methods with which we assess RRI projects. The nuances of co-creation, stakeholder participation or transnational learning cannot be grasped solely by focusing on long and complex surveys, or short and sweet open questions, checking boxes, using alternate responses, or scaling the feedback. If our territorial projects aspire to mainstream responsible research and innovation, we should start by making our own research and innovation responsible. As RRI in context encounters enormous diversity our projects require careful translation and engagement in order to achieve local ownership. Flexibility, adaptiveness, and a constant evaluative conversation between project and territorial partners are key, not just towards the end, but all throughout the project. This confirms SUPER MoRRI’s ‘credible contextualization’ principle.