Queen Margaret University Institutional Statement on Responsible Research Assessment
Background
QMU applies equitable, fair and transparent mechanisms in research assessment and the principles in this statement underpin our submission to the Research Excellence Framework (REF 2029) and our REF 2029 Code of Practice.
Introduction
Responsible research assessment is an umbrella term for approaches to assessment which incentivise, reflect, and reward the plural characteristics of high-quality research, in support of diverse and inclusive research cultures.
Responsible research assessment includes responsible research metrics which refer to the thoughtful and ethical use of quantitative indicators when assessing research quality, impact, and performance. Rather than relying on simplistic measures like journal impact factors or citation counts alone, responsible metrics aim to support fair, transparent, and context-sensitive evaluation.
QMU adheres to the principles that responsible research assessment:
- is primarily based on qualitative judgement, recognising the various roles and career pathways in research
- is supported by quantitative indicators, promoting diversity, equality and inclusion by adhering to transparent criteria and methods
- recognises the full range of activities and outputs that contribute to the quality and impact of research
- values integrity, rigour and openness in the conduct and communication of research
Both qualitative and quantitative indicators are used by QMU to assess individual and institutional performance and we acknowledge the limitations of using either approach in isolation. QMU is committed to the fair and transparent use of quantitative indicators in the external measurements of our reputation, as measured by league tables and funding agencies.
QMU does not use single, non-normalised metrics in research assessment. It only uses indicators that are transparent and contextualised with citation practices within the relevant discipline.
This statement provides a set of principles outlining good practice. These principles reinforce the key role of peer review and support an inclusive and transparent process to research assessment, respectful of researchers and of the plurality of research.
Sector Guidance and Best Practice
It outlines QMU’s commitment to:
- Becoming a signatory to the San Francisco Declaration on Research Assessment (DORA): DORA is a worldwide initiative, with recommendations covering all scholarly disciplines, which recognises the need to improve the ways in which the outputs of scholarly research are evaluated and seeks to develop and promote best practice. In particular, DORA seeks to address the practice of using journal impact factors as a proxy for quality.
- Adopting the principles of the Leiden Manifesto: The Leiden Manifesto brings together accepted but disparate principles of good practice in research evaluation. The manifesto represents the “distillation of best practice in metrics-based research assessment so that researchers can hold evaluators to account and evaluators can hold their indicators to account”.
- Implementing the recommendations of the Forum for Responsible Research Metrics (FFRM) and the principles of the Metric Tide Report: Further frameworks have emerged since the publication of DORA (2013) on the use of quantitative research indicators in the assessment and management of research and QMU is committed to the recommendations of the FFRM Report, the Metric Tide (2015) and the principles of the Leiden Manifesto (2015).
- To the Scottish Funding Council (SFC) Responsible Research Assessment: Collaborative Training for Scotland’s Universities.
Institutional Commitments
- QMU is committed to considering research outputs within the context of their research environment, disciplinary norms, and the research objectives of the researcher.
- QMU works with the sector to explore, develop and share best practice in relation to the responsible use of research metrics and new approaches to evaluating research.
- QMU researchers uphold the highest level of research integrity, including acknowledging the contributions of others and citing original research.
Guiding Principles
Implement assessment procedures that only use quantitative indicators in parallel with peer review. Indicators may be used in a variety of processes but indicators will not supplant peer review of both research outputs and their environment.
Diversity should be recognised and accounted for
- Research assessment approaches should align to the plurality of research and recognise that indicators will not serve all disciplines equally.
- The diverse research objectives of individual researchers and of Research Centres should be taken into account.
- Recognise and value research that informs policy and practice – celebrate the value of research outputs in public engagement, impact creation and dissemination to users.
- Acknowledge varying publication and citation practices across disciplines and that quantitative metrics work better for some forms of research output than others.
- Recognise that the timeframes involved will vary according to disciplines and outputs.
- Best practice is to apply quantitative indicators responsibly and normalise use by discipline.
- Many citation tools and quantitative indicators are inherently skewed to English language outputs. QMU values the international dimension of much of its research and encourages publication in the appropriate language for the research user.
- Recognise areas of potential bias and aim to address them: such a consideration applies to the most widely used citation databases and their alignment with QMU disciplines and output types and gender bias in citation practices.
Processes should be open and transparent
- Openly declare when quantitative indicators are used and promote researcher awareness, understanding and scrutiny of assessment methodologies, ensuring that processes are transparent and documented.
- Aim for a balance between simplicity and accuracy in the use of quantitative indicators.
- Ensure research assessment expectations are transparent, fair and consistent by setting expectations in advance through the annual QMU performance enhancement review process and Academic Workload models.
- Encourage researchers to understand and challenge the indicators used in relation to their outputs.
- Roll out new processes to ensure that all researchers are registered for an ORCID ID to ensure consistent and reliable attribution of work.
Misplaced concreteness and false precision should be avoided
- Use metrics only where their strengths, weaknesses and limitations are understood and where placing undue significance on quantitative differences could be avoided.
- Include caveats responsibly in research assessment data and reports.
- Undertake regular reassessment of any indicators used.
The systemic effects of assessment and indicators should be recognised
- Anticipate and mitigate any unintended effects established by using indicators by encouraging researcher feedback and review.
Assessment of individual research outputs
- These should be assessed primarily by expert peer review of the output, for example using the REF approach to assessment based on originality, significance and rigour.
- Citation counts should only be used if interpreted in the light of disciplinary norms and with an understanding of the factors which affect citation counts, including paper, journal and author related factors. For example, an article in an English-language journal, written by several authors in an international collaboration is likely to be cited more often than an article written by a single author in a journal published in a language other than English.
- Article level metrics are more appropriate than journal level metrics in the assessment of individual outputs and can inform peer review, but all indicators must be normalised to the discipline.
- Journal impact factors will not be used as an indicator of the quality of the output.
Assessing a researcher's body of work
- This should be assessed by expert peer judgement of the researcher's portfolio and their personal research objectives.
- Criteria used for academic recruitment, promotion and review should be founded in expert judgement reflecting the academic quality of outputs and the wider impact of the work.
- The publication and citation practices within the subject area should be taken into account.
- The use of eResearch as the source of output data for research assessment and management is recommended as researchers will be able to check and maintain their outputs and the source of this data will be transparent.
- Research quality indicators are affected by equality and diversity factors; career stage and discipline and these should be accounted for in interpreting indicators.
- Recognise that researchers undertake a wide range of activities, not all of which can be easily measured or benchmarked, e.g. Practice as Research.
Research Income, Doctoral Supervision and Doctoral Degrees Awarded
- Research income secured, diversity of income sources, doctoral candidates supervised and doctoral degrees awarded are QMU research KPIs and metrics used by REF and in league tables.
- Application of such metrics at individual or unit level should be normalised in line with discipline variations and career stage.
Recruitment, Performance Management and Promotion
- When utilising indicators to inform decisions on recruitment, performance management and promotion, provide advanced specification on criteria used for selection.
- When assessing the performance of individuals, consideration should be given to as wide a view of their expertise, experience, activities and influence as possible.
- Make assessments based on research content rather than quantitative indicators.
QMU supports its researchers to challenge research practices that rely inappropriately on quantitative indicators, and in fostering best practice that focuses on the value and influence of a broad range of research outputs.
Staff training programmes on the responsible use of research indicators are provided as part of general development opportunities. It is important that QMU researchers are empowered with the necessary understanding and knowledge of process, in order to use such research indicators appropriately. Beyond REF we endeavour to improve our research culture by embracing sector developments such as Narrative CVs, and Research Integrity, as we continue to support the full range of contributions researchers make including skills and experiences.
Where practice is found to contravene the principles of DORA or the Leiden Manifesto please contact RKEDU Email Address. We are committed to providing a route for researchers to support the implementation of this statement, report poor practice and advise us of priority areas for review.
RKEDU
Updated version of October 2019 statement March 2022
