Research assessment and the responsible use of metrics

Responsible Use of Metrics

The San Francisco Declaration on Research Assessment (DORA), of which the University of Dundee is a signatory, requires that organisations embrace the principles which underpin the Responsible Use of Metrics.

What we mean by the responsible metrics and why it is important

Metrics are a series of measures used to quantify, assess, and monitor published research. Traditional bibliometrics count citations based on individual authors, articles, and journals. Alternative metrics, commonly referred to as altmetrics measure attention through social media and news platforms and are a useful way to assess impact and reach beyond academia.

Metrics have played an important role in academia for many years, being used to assess individual researchers, the research they produce and, in the case of university impact rankings, even the institutions themselves. However, there are concerns that the metrics used are flawed.

In 2015 the findings of the Independent Review of the Role of Metrics in Research Assessment and Management were published in the report, The Metric Tide. The report proposes a framework and a series of recommendations designed to ensure that metrics are used responsibly and as part of a suite of assessment measures, never in isolation.

James Wilsden who chaired the review notes that, “The metric tide is certainly rising. Unlike King Canute, we have the agency and opportunity – and in this report, a serious body of evidence – to influence how it washes through higher education and research.”

The report identifies 5 key factors which should be considered with regards to responsible metrics:

• Robustness: basing metrics on the best possible data in terms of accuracy and scope
• Humility: recognising that quantitative evaluation should support – but not supplant – qualitative, expert assessment
• Transparency: keeping data collection and analytical processes open and transparent, so that those being evaluated can test and verify the results
• Diversity: accounting for variation by field, and using a variety of indicators to support diversity across the research system
• Reflexivity: recognising systemic and potential effects of indicators and updating them in response.

Summary

Responsible metrics requires the academic community to consider a range of tools when evaluating research to include both quantitative and qualitative methods. We should make the most of the expertise of peer-review and not rely solely upon citation metrics as a measure of quality. If used to compliment each other, this approach should underpin a transparent process which, crucially, can be applied to all disciplines.

As a community we should embrace open research practices that lead to greater transparency in research, applying, where possible, persistent identifiers and standards to enable accurate attribution to all types of research outputs and outcomes.

We should regularly review the measures we do use, whether quantitative or qualitative, to ensure that they are fit for purpose and are adapting to the changes, challenges, and diversity of the global research environment.