Evaluation of Research IG rev-002

Introduction

The evolution of the evaluation of research from being mostly bibliographic index-based is now recognized to be indispensable to enable Open Science. In the same way that Open Science promotes the open sharing of FAIR data and other research outputs, evaluators must also value and consider these outputs as part of the wider research evaluation framework. The RDA is an appropriate forum for addressing the evolving requirements of research evaluation as they relate to Open Science practices. It is proposed to create an RDA Interest Group “Evaluation of Research” to organise the discussion on the subject in RDA, in particular as it relates to data, and also in relation to RDA activities on research software. 

The topic is perfectly aligned with the RDA mission of “building the social and technical bridges to enable open sharing and re-use of data” since the evolution of research evaluation is an essential social bridge to enable open sharing and re-use of data.

RDA’s reach across the international community, including research organisations and funders, can help to facilitate and align a global discussion of research evaluation, drawing on data experts from a range of domains. Research software experts are also active in the RDA and their involvement will be sought. Liaison will be sought with the Research Software Alliance, with which RDA has established a strong collaboration.

It is important for the RDA to identify its role and value in this topic, which is increasingly prominent with more research assessment reform initiatives being established at the local, organisational, national, regional and international levels. In addition, the Interest Group (IG) will be a value-added contribution to the organisations, including research organisations and universities, funders and policy makers who use the RDA as an international, neutral forum: it will enable them to discuss their proposals on the evolution of research evaluation with the community, whose comments will help improve these proposals and make them more relevant and applicable. It will also be a high value-added contribution to the community of RDA data experts, who will benefit from better recognition of their activities. The IG will bring their voice to the attention of the organisations, funders and policy makers who participate in the RDA. 

User scenario(s) or use case(s) the IG wishes to address

The necessary evolution of the evaluation of research has been discussed internationally for years. One can of course cite DORA, the Declaration on Research assessment, which is a decade old in 2023, and the Leiden Manifesto for Research Metrics (Hicks et al, 2015). Research evaluation (research assessment) was also recently a major topic of the Paris Open Science European Conference (OSEC 2022) organised in the context of the French Presidency of the European Union with a strong European and international participation, including speakers from the European Commission, South America, UNESCO and the United States. 

A successful Birds of a Feather (BoF) session (web page, collaborative notes) held on 20 June 2022 during the nineteenth RDA Plenary Meeting demonstrated that there is significant interest in  organising an activity on this topic in the RDA, under the form of an Interest Group. The community interest was confirmed during the very well attended BoF held during the twentieth Plenary Meeting (session web page, collaborative notes).

Objectives

The Paris Call on Research Assessment was presented during the OSEC 2022 Conference. It “calls for the creation of a coalition of research funding organisations, research performing organisations, and assessment authorities, willing and committed to reform the current research assessment system along commonly agreed objectives, principles and actions (such as mutual learning, shared documentation and commonly agreed monitoring effort). The success of such a coalition will be deeply connected to its capacity to propose concrete implementation processes and to its capacity to associate and involve researchers at all levels.” The IG will be supporting these capacities. 

The Agreement on Reforming Research Assessment, based on 10 commitments establishing a common direction for research assessment reform while respecting organisations’ autonomy, was published on 20 July 2022. Organisation from across the world were invited to sign it, and the Coalition for Advancing Research Assessment (CoARA) was created, and its Steering Board elected, during a Constitutive Meeting gathering the organisations which had signed the Agreement held on 1 December 2022. The RDA actively contributed to the CoARA implementation guidelines, has signed, and joined the Coalition. Liaising with CoARA, which shares RDA global reach and attachment to the principle of openness of research, will be an objective of the IG. 

Other relevant research evaluation/assessment initiatives globally include DORA, GRC research assessment working group, Higher Education Leadership Initiative for Open Scholarship (HELIOS), The Latin American Forum on Research Assessment (FOLEC-CLACSO), UKRI Future research assessment program, the Dutch initiative “Room for everyone’s talent”, HuMetrics HSS. Reforming research assessment is also one of the 20 priority actions under the European Research Area policy agenda. There are also projects undergoing activities in the domain, e.g. EOSC.

IG initial topics for discussion:

• Mapping of the RDA activities relevant to research evaluation/assessment;

• Gathering of links to relevant information, identifying relevant initiatives and determining how to engage with the initiatives and organisations that are most important to the group’s objectives and outcomes, in particular CoARA;

• Liaison with the developments linked to CoARA and other initiatives identified as relevant and important;

• Liaison with the relevant RDA Bodies, Interest Groups (IG) and Working Groups (WG);

• Identifying additional topics and liaisons to establish;

• Act as an incubator for relevant RDA activities to be developed, in particular new WGs;

The longer term goals/objectives of the IG are to focus its efforts and support assessment of the group’s impact and achievements in the future.

Participation

The activity will involve all the members of the RDA community (individual members, organisations, regions, funders) interested in the topic. Inside the RDA, liaisons have thus to be established with the Organisational Advisory Board (OAB), Regional Advisory Board (RAB), and the Funders’ Forum, as well as with many of the most relevant RDA IGs and WGs. Outside the RDA liaisons should be established with the international initiatives on research evaluation/ assessment identified above. One of the standing items of the IG activity will be to identify other initiatives with which it would be useful to liaise.

The most relevant RDA Groups identified at this stage are :

• Sharing Rewards and Credit (SHARC) IG

• Data Policy Standardisation and Implementation IG

• Data Repository Attributes WG (DRAWG)

• Early Career and Engagement IG (ECEIG)

• Engaging Researchers with Data IG

• Research Funders and Stakeholders on Open Research and Data Management Policies and Practices IG

• Software Source Code IG

• Data Usage Metrics WG

This list will be completed using the mapping activity. 

Identified IG members who are also co-chairs or members of the OAB, RAB and Funders’ Forum will ensure the liaison with RDA organisations and regions, Amy Nurnberger, Francoise Genova and Matthew Lucas respectively. Anne Cambon-Thomsen, Iain Hrynaszkiewicz, Kathleen Schaerer, Elli Papadopoulou, Andrea Medina-Smith, Jeremy Geelen and Tom Honeyman will ensure liaison between the IG and respectively the SHARC IG, Data Policy Standardisation and Implementation, DRAWG,  ECEIG, Engaging Researchers with Data IG, Research Funders and Stakeholders IG, and Software Source Code IG. These Groups contributed to the P20 BoF session. We still have to contact the Data Usage Metrics WG.

Outcomes

At this initial stage, expected outcomes are the following:

• A document on RDA Value for Evaluation of Research built from the results of the mapping activities, to be published on the RDA web site with the other similar documents

• Networking of the most relevant RDA Groups on the subject

• Established liaison with CoARA, and other relevant initiatives 

• Define the most relevant contributions a community-led activity from the RDA can bring to the landscape and organise the IG work accordingly, including the possibility to launching a multi-stakeholder forum 

• Analyse gaps between the principles and practice of research evaluation to assemble best practices;

• Assessing the interest of providing a document gathering relevant links and eventually stories, for instance to disseminate information about good practices. A draft template of such a document was tested during P19 BoF, it is available here. This document would have to be updated continuously, eventually with a version published each year. The objectives of such a document and the feasibility of establishing it will be discussed by the IG at an early stage. If it is decided to prepare such a document, its initial version could be a topic for a Working Group under the Evaluation of Research IG.  

Mechanism

As a starting point, the IG will have four co-chairs:

• Emma Crott, Australian Research Data Commons (ARDC), Australia
• Francoise Genova, Strasbourg Astronomical Observatory, France
• Devika Madalli, Indian Statistical Institute, India
• Amy Nurnberger, Massachusetts Institute of Technology (MIT), USA

A Stakeholder Forum will gather the liaisons identified with RDA Groups and other relevant initiatives.

The IG co-chairs will aim to meet regularly, on a monthly basis at the beginning of the IG life, and more frequently when needed. 

The Stakeholder Forum will regularly be informed of and consulted about the IG activities, by email and if useful through specific meetings.

IG meetings will be organised at least quarterly, during and between RDA Plenaries. Additional IG meetings will be organised if a topic to be discussed more urgently with the community is identified. 

Each task will have at least one identified person in charge.

Timeline

This timeline is indicative and will be revised depending on the results of the first IG activities.

Month 1: Initiate the mapping exercise

Month 3:

Liaison established with CoARA

Webinar organised with the Early Career and Engagement IG including a discussion of the IG objectives and a presentation of the Harbingers survey, a global questionnaire survey of the scholarly communication attitudes and behaviours of early career researchers.

Month 6:

Decision on the Evaluation of Research – Stories and Links document

Liaison established with the relevant RDA Groups

Month 9: Document on RDA Value for Evaluation of Research

IG20Evaluation20of20Research20-20Charter2020230502_0.pdf