Research Metrics

Bibliometrics provide helpful quantitative measures of citation impact but do not provide a complete picture of research impact on their own. 

There is increasing recognition worldwide of the importance of responsible use of bibliometrics in research assessment. Several frameworks have emerged to assist with the move to the responsible use of research metrics, amongst them ‘The Metric Tide’, the ‘Leiden Manifesto for Research Metrics’, and most notably ‘The San Francisco Declaration on Research Assessment’ (DORA). 

Therefore, it is essential that we use bibliometrics responsibly. Some things to keep in mind are:

• Citation patterns can vary widely between research fields, so we need to compare like with like
• Don’t just use one citation tool 

Remember:

• Coverage varies in content, depth, and discipline. No citation tool is comprehensive as they don’t index all publications or research areas.
• Some disciplines don’t publish as much in journals
• Data needs to be looked at in context. Use a variety of metrics, data & other qualitative information where appropriate.
• A citation does not have to be positive.

Several researchers and organizations have proposed standards and recommendations for best practices. Specific recommendations vary, but two principles are consistent:

1. Evaluate and give the most weight to research quality, then consider quantitative indicators; and
2. Use more than one indicator and note their sources.

San Francisco Declaration on Research Assessment (DORA) - recommendations for funding agencies, institutions, publishers, researchers and organizations that supply metrics along three themes:

1. eliminate the use of journal-based metrics, such as Journal Impact Factors, in funding, appointment, and promotion considerations;
2. assess research on its own merits rather than based on the journal in which the research is published; and
3. capitalize on the opportunities provided by online publication (such as relaxing unnecessary limits on the number of words, figures, and references in articles and exploring new indicators of significance and impact).

The Leiden Manifesto for Research Metrics - 10 principles to guide research evaluation outlined in this video:

The Metric Tide: Final Report with Executive Summary by the Independent Review of the Role of Research Metrics in Research Measurement and Assessment, including these dimensions of responsible metrics (p. X):

1. Robustness: basing metrics on the best possible data in terms of accuracy and scope;

2. Humility: recognising that quantitative evaluation should support – but not supplant – qualitative, expert assessment;

3. Transparency: keeping data collection and analytical processes open and transparent so that those being evaluated can test and verify the results; 

4. Diversity: accounting for variation by field and using a range of indicators to reflect and support a plurality of research and researcher career paths across the system; 

5. Reflexivity: recognising and anticipating indicators' systemic and potential effects and updating them in response.

The DORA organization has released a guide for research institutions, "SPACE, a rubric for analyzing institutional conditions and progress indicators." The rubric covers:

• Standards for Scholarship
• Process Mechanics and Policies
• Accountability
• Culture within Organizations
• Evaluative and Iterative Feedback

at foundation, expansion and scaling levels. This recognizes that a researcher's affiliated organisation's culture, policies and practices can drive choices about what metrics are chosen and how they are used. 

The Metrics Toolkit has definitions, scope, appropriate use cases, limitations, inappropriate use cases, transparency and more about indicators for authors, books, book chapters, datasets, journals, journal articles and software/code/scripts.