Indicators for research performance evaluation: an overview

Bibliometric indicators to evaluate scientific activity

Bibliometric indicators have been devised to quantify scientific production and to try to evaluate its impact in the community. In general, bibliometric indicators can be classified according to whether the unit of analysis is the author (individual or group) or journal. The most widely used indicators for authors are those that measure an individual author's production, such as the Crown index or the h-index and its derivatives (e-index, h5-index, and the absolute or Ab-index, among others). The bibliometric indicators devised to try to evaluate journal quality are associated with Journal Citation Reports (e.g., impact factor, field-weighted citation impact, Eigenfactor, and article influence) or with Scopus (Scimago Journal Rank (SJR), source normalized impact per paper (SNIP), and CiteScore). This article describes the main bibliometric indicators, explains how they are calculated, and discusses their advantages and limitations.

Evaluation of the Implementation Research Institute: Trainees' Publications and Grant Productivity

With growth in the field of dissemination and implementation (D&I) research, there has been growth in capacity building, with many training opportunities. As such, it is important to continue to evaluate D&I research training programs. This paper reports the results of an evaluation of the Implementation Research Institute (IRI), a R25 funded by the National Institute of Mental Health with additional funding by the Department of Veterans Affairs (VA). The fourth cohort also had a supplement from the National Institute on Drug Abuse. Using bibliometrics data, we report on a quasi-experimental retrospective cohort study assessing whether the rates of scholarly productivity in D&I science of IRI fellows (those who applied and were accepted to the training) were greater than those who applied but were not accepted to IRI. Our findings show that Selected Applicants' odds of publishing in implementation science were higher for earlier alumni, starting at 12% 1 year out and increasing to 94% for those who were 4 years out from starting training. Chances for Non-Selected Applicants remained relatively stable, starting at 47% at 1 year and going to 33% at 4 years since their application, a pattern that was stable even after controlling for demographic characteristics. These results support the hypothesis that IRI is increasing the D&I research productivity of those selected to the program, and that our fellows are advancing the field of D&I compared to those investigators not selected to our institute. Our finding also indicates the importance of a 2-year training.

A bibliometric analysis for global research trends on ectomycorrhizae over the past thirty years

Purpose

Based on the significance of ectomycorrhizae (ECM) and increased publication activity on this subject, it was decided to carry out a bibliometric analysis of scientific outputs in this area. The purpose of this study is to reveal the research trends of scientific outputs on ECM for the past 30 years and provide a potential guide for future research.

Design/methodology/approach

A method of bibliometric analysis was performed, based on the online version of the Science Citation Index Expanded, Web of Science, from 1986 to 2017. The authors evaluated the publication types, languages, source countries, journals, the patterns of publication outputs, most-cited articles, most-productive authors, institutional distributions, subject categories, high-frequency keywords and keywords plus and high-frequency terms in the title and abstract of ectomycorrhizal research. Keywords, keywords plus and high-frequency terms in the title and abstract were analyzed via VOSviewer to illustrate the extent of co-occurrence. This study further describes the recent research priority or hotspots and reveals the research trends.

Findings

From 1986 to 2017, the publication output on ECM showed a rising trend; the number of articles has rapidly increased after 2003. Based on co-occurrence analysis for keywords, keywords plus and terms in the title and abstract, “ectomycorrhizal fungi” is the most popular keyword and keywords plus; “concentration” is the most high-frequency terms in the title and abstracts. Plant biology, mycology and ecology are the hotspots in the ectomycorrhizal research. Ectomycorrhizal taxonomy, the molecular mechanisms of ectomycorrhizal symbioses and the common mycorrhizal networks are the future direction.

Originality/value

A bibliometric analysis has been carried out to analyze the trends of ECM research with 30 years. This study provides a potential guide for future research related to ectomycorrhizae.

Change of perspective: bibliometrics from the point of view of cited references-a literature overview on approaches to the evaluation of cited references in bibliometrics

Citation analyses normally investigate the number of citations of publications (e.g. by people, institutions or journals) where the information on times cited from the bibliographic databases (such as Scopus or Web of Science) is evaluated. But in recent years, a series of works have also been published which have undertaken a change of perspective and are based on the evaluation of the cited references. The cited references are the works cited in the publications which are used to calculate the times cited. Since these evaluations have led to important insights into science and into scientometric indicators, this paper presents an overview of methods based on cited references, and examples of some empirical results from studies are presented. Thus, the investigation of references allows general statements to be made on the precision of citation analyses, and offers alternatives for the normalization of citation numbers in the framework of research evaluation using citation impact. Via the analysis of references, the historical roots of research areas or the works of decisive importance in an area can be determined. References allow quantitative statements on the interdisciplinarity of research units and the overall growth of science. The use of a selection for the analysis of references from the publications of specific research areas enables the possibility of measuring citation impact target-oriented (i.e. limited to these areas). As some empirical studies have shown, the identification of publications with a high creative content seems possible via the analysis of the cited references. The possibilities presented here for cited reference analysis indicate the great potential of the data source. We assume that there are additional possibilities for its application in scientometrics.

Development and use of a research productivity assessment tool for clinicians in low-resource settings in the Pacific Islands: a Delphi study

Background

Research performance assessments have proliferated, but research indicators for use amongst clinicians in poorly resourced countries have been ill-defined. The aims of the present paper were to determine a set of indicators as determined by clinician participants from the Pacific Islands and a panel of research experts for use in the performance assessment of clinicians.

Methods

Two focus group discussions, one for nurses and one for doctors, were used to obtain the views of 28 Pacific Island clinicians of the BRRACAP Study about what the research indicators should be. A modified Delphi survey was used to obtain a consensus amongst 19 research experts, with Pacific Island research experience, as to what the indicators should be and then to rank these in terms of importance. A survey of the participants obtained data on the research tasks/actions performed 20 months after the initial research workshop. A resultant tool comprising of 21 indicators was used to assess the performance of 18 Pacific participants.

Results

The Pacific Island clinicians determined that research was important and that performance should be measured. They identified research indicators that could be used in their settings and ranked their importance using a points system. The panel of experts identified implementation of research findings, collaborations and actual change in practice as more important, with bibliometric measurements low down in the scale. Although only 64 % of the 28 BRRACAP Study participants returned the questionnaire, 39 % of those performed more than half of the 21 indicators used. Of the 18 Pacific clinicians assessed, 7 (39 %) performed 10 or more tasks.

Conclusions

A research performance assessment tool was developed using process and output indicators identified by Pacific clinicians and a panel of research experts. The tool, which placed emphasis on process and outputs that were not bibliometric based, proved useful in assessing the performance of Pacific clinicians working in a low-resource setting.

Electronic supplementary material

The online version of this article (doi:10.1186/s12961-016-0077-4) contains supplementary material, which is available to authorized users.

Measuring the outcome of biomedical research: a systematic literature review

Background

There is an increasing need to evaluate the production and impact of medical research produced by institutions. Many indicators exist, yet we do not have enough information about their relevance. The objective of this systematic review was (1) to identify all the indicators that could be used to measure the output and outcome of medical research carried out in institutions and (2) enlist their methodology, use, positive and negative points.

Methodology

We have searched 3 databases (Pubmed, Scopus, Web of Science) using the following keywords: [Research outcome* OR research output* OR bibliometric* OR scientometric* OR scientific production] AND [indicator* OR index* OR evaluation OR metrics]. We included articles presenting, discussing or evaluating indicators measuring the scientific production of an institution. The search was conducted by two independent authors using a standardised data extraction form. For each indicator we extracted its definition, calculation, its rationale and its positive and negative points. In order to reduce bias, data extraction and analysis was performed by two independent authors.

Findings

We included 76 articles. A total of 57 indicators were identified. We have classified those indicators into 6 categories: 9 indicators of research activity, 24 indicators of scientific production and impact, 5 indicators of collaboration, 7 indicators of industrial production, 4 indicators of dissemination, 8 indicators of health service impact. The most widely discussed and described is the h-index with 31 articles discussing it.

Discussion

The majority of indicators found are bibliometric indicators of scientific production and impact. Several indicators have been developed to improve the h-index. This indicator has also inspired the creation of two indicators to measure industrial production and collaboration. Several articles propose indicators measuring research impact without detailing a methodology for calculating them. Many bibliometric indicators identified have been created but have not been used or further discussed.