Queen Margaret University Institutional Statement on the Use of Metrics in Research Assessment
Background
QMU applies fair and transparent mechanisms in research assessment and the principles in this statement underpinned our submission to the Research Excellence Framework (REF 2021) and our REF 2021 Code of Practice.
The joint UK Funding Bodies issued REF 2021 Guidance for standardising quantitative indicators in the environment statement in REF 2021. This provided advice and examples on the inclusion of additional quantitative indicators in the environment, based on the work of the Forum for Responsible Research Metrics, and incorporating advice from the REF 2021 main and subpanels.
Introduction
Both qualitative and quantitative indicators are used by QMU to assess individual and institutional performance and we acknowledged the limitations of using either approach in isolation.
QMU also acknowledges the need for the fair and transparent use of quantitative indicators in the external measurements of our reputation, as measured by league tables and funding agencies.
QMU does not use single, non-normalised metrics in research assessment. It only uses indicators that are transparent and contextualised with citation practices within the relevant discipline.
Purpose
This statement is a guide to responsible research assessment. It provides a set of principles outlining good practice. These principles reinforce the key role of peer review and support an inclusive and transparent process to research assessment, respectful of researchers and of the plurality of research.
It outlines QMU’s commitment to:
- Becoming a signatory to the San Francisco Declaration on Research Assessment (DORA) to underpin its commitment to the responsible use of research metrics.
- Adopting the principles of the Leiden Manifesto.
- Implementing the recommendations of the Forum for Responsible Research Metrics (FFRM) and the principles of the Metric Tide Report
QMU is committed to considering research outputs within the context of the Unit, Centre, Division and University research environment and the research objectives of the researcher.
QMU works with the sector to explore, develop and share best practice in relation to the responsible use of research metrics and new approaches to evaluating research.
QMU researchers uphold the highest level of research integrity, including acknowledging the contributions of others and citing original research.
Background
DORA is a worldwide initiative, with recommendations covering all scholarly disciplines, which recognises the need to improve the ways in which the outputs of scholarly research are evaluated and seeks to develop and promote best practice. In particular DORA seeks to address the practice of using journal impact factors as a proxy for quality.
Further frameworks have emerged since the publication of DORA (2013) on the use of quantitative research indicators in the assessment and management of research and QMU is committed to the recommendations of the FFRM Report, the Metric Tide (2015) and the principles of the Leiden Manifesto (2015).
In 2017 the FFRM partnership between HEFCE (now Research England), Research Councils UK, the Wellcome Trust, Universities UK and JISC, was created to develop a series of initiatives to support the responsible use of metrics in the HE sector and research organisations across the UK. Although the FFRM focuses on the recommendations made in the Metric Tide Report it supports the principles set out in both Leiden and DORA.
The Leiden Manifesto brings together accepted but disparate principles of good practice in research evaluation. The manifesto represents the “distillation of best practice in metrics-based research assessment so that researchers can hold evaluators to account and evaluators can hold their indicators to account”.
Both of these initiatives have come from groups of researchers who share a concern with the way the outputs from research are evaluated by funders, academic publishers, institutions and other parties. While appreciating the value of quantitative research metrics, they are concerned about their inappropriate use in decision making, for example around the allocation of funds and in academic career progression.
What are quantitative indicators?
Citation-based metrics
Many of the quantitative indicators used in research assessment are citation-based bibliometric indicators such as citation counts; journal impact factors; and the h-index. These are derived from the data found in Web of Science, Scopus, or in Google Scholar. These metrics are used in many commonly used sources, such as on publisher's journal web sites and in QMU reports derived from eResearch. It is important that all staff involved in research, and not just those directly involved in the assessment of research, have an understanding of the underpinnings, benefits and limitation of these indicators and their responsible use.
Altmetrics
Alternative metrics ('altmetrics') are a relatively new kind of indicator which provide information about attention to research outputs in social media such as Twitter and also information about captures, shares and number of views and downloads. There are still many uncertainties about these developing metrics, including about their reliability. The FFRM therefore recommends that altmetrics should not be used in REF style evaluations of outputs although there may be some value in their use in assessment of impact.
Quantitative evaluation could support qualitative, expert assessment
Guiding Principles
Implement assessment procedures that only use quantitative indicators in parallel with peer review. Indicators may be used in a variety of processes but indicators will not supplant peer review of both research outputs and their environment.
Diversity should be recognised and accounted for
- Research assessment approaches should align to the plurality of research and recognise that indicators will not serve all disciplines equally.
- The diverse research objectives of individual researchers and of Research Centres should be taken into account.
- Recognise and value research that informs policy and practice – celebrate the value of research outputs in public engagement, impact creation and dissemination to users.
- Acknowledge varying publication and citation practices across disciplines and that quantitative metrics work better for some forms of research output than others.
- Recognise that the timeframes involved will vary according to disciplines and outputs.
- Best practice is to apply quantitative indicators responsibly and normalise use by discipline.
- Many citation tools and quantitative indicators are inherently skewed to English language outputs. QMU values international dimension of much research and courages publication in the appropriate language for the research user.
- Recognise areas of potential bias and aim to address them: such a consideration applies to the most widely used citation databases and their alignment with QMU disciplines and output types and gender bias in citation practices.
Processes should be open and transparent
- Openly declare when quantitative indicators are used and promote researcher awareness, understanding and scrutiny of assessment methodologies, ensuring that processes are transparent and documented.
- Aim for a balance between simplicity and accuracy in the use of quantitative indicators.
- Ensure research assessment expectations are transparent, fair and consistent by setting expectations in advance through the QMU PER process.
- Encourage researchers to understand and challenge the indicators used in relation to their outputs.
- Roll out new processes to ensure that all researchers are registered for an ORCID ID to ensure consistent, reliable attribution of work.
Misplaced concreteness and false precision should be avoided
- Use metrics only where their strengths, weaknesses and limitations are understood and where placing undue significance on quantitative differences could be avoided.
- Include caveats responsibly in research assessment data and reports.
- Undertake regular reassessment of any indicators used.
The systemic effects of assessment and indicators should be recognised
- Anticipate and mitigate any unintended effects established by using indicators by encouraging researcher feedback and review.
Application of Quantitative Indicators in Research Assessment
Assessment of individual research outputs
- These should be assessed primarily by expert peer review of the output, for example using the REF approach to assessment based on originality, significance and rigour.
- Citation counts should only be used if interpreted in the light of disciplinary norms and with an understanding of the factors which affect citation counts, including paper, journal and author related factors. For example, an article in an English-language journal, written by several authors in an international collaboration is likely to be cited more often than an article written by a single author in a journal published in a language other than English.
- Article level metrics are more appropriate than journal level metrics in the assessment of individual outputs and can inform peer review, but all indicators must be normalised to the discipline.
- Journal impact factors will not be used as an indicator of the quality of the output.
Assessing a researcher's body of work
- This should be assessed by expert peer judgement of the researcher's portfolio and their personal research objectives.
- Criteria used for academic recruitment, promotion and review should be founded in expert judgement reflecting the academic quality of outputs and the wider impact of the work.
- The publication and citation practices within the subject area should be taken into account.
- The use of eResearch as the source of output data for research assessment and management is recommended as researchers will be able to check and maintain their outputs and the source of this data will be transparent.
- Research quality indicators are affected by equality and diversity factors; career stage and discipline and these should be accounted for in interpreting indicators.
- Recognise that researchers undertake a wide range of activities, not all of which can be easily measured or benchmarked, e.g. Practice as Research
Research Income, Doctoral Supervision and Doctoral Degrees Awarded
- Research income secured, diversity of income sources, doctoral candidates supervised and doctoral degrees awarded are QMU research KPIs and metrics used by REF and in league tables.
- Application of such metrics at individual or unit level should be normalised in line with discipline variations and career stage.
Recruitment, Performance Management and Promotion
- When utilising indicators to inform decisions on recruitment, performance management and promotion, provide advanced specification on criteria used for selection.
- When assessing the performance of individuals, consideration should be given to as wide a view of their expertise, experience, activities and influence as possible.
- Make assessments based on research content rather than quantitative indicators.
Implementing the statement
QMU supports its researchers in challenging research practices that rely inappropriately on quantitative indicators and in teaching best practice that focuses on the value and influence of a broad range of research outputs.
A staff training programme will be introduced on the responsible use of research indicators. It is important that QMU researchers are empowered, both with the necessary understanding and knowledge of process to use research indicators appropriately.
Beyond REF 2021 we will endeavour to improve our research culture by embracing sector developments around the principles of narrative CVs as we continue to support the full range of contributions researchers make including skills and experiences.
Where practice is found to contravene the principles of DORA or the Leiden Manifesto please contact RKEDU Email Address We are committed to providing a route for researchers to support the implementation of this statement, report poor practice and advise us of priority areas for review.
RKEDU
Updated version of October 2019 statement March 2022
