Assessing Individual Intellectual Output in Scientific Research: Mexico's National System for Evaluating Scholars Performance in the Humanities and the Behavioral Sciences

Evaluation of the national research foundation-rated researchers’ output at a South African university

Purpose

The purpose of this paper is to establish the research impact of the National Research Foundation (NRF)-rated researchers’ output at the North-West University (NWU), South Africa, from 2006 to 2017.

Design/methodology/approach

The study used bibliometrics and altmetrics methods to determine the production of research outputs and the impact of NWU’s NRF-rated researchers’ publications. Various tools including Google Scholar (GS), Web of Science (WoS), Scopus, ResearchGate (RG) and Mendeley were used to collect data. The citations in the three bibliographic databases were used as proxy for academic impact, while reads and readerships in RG and Mendeley were used to determine societal impact of the researchers. The Statistical Package for the Social Sciences (SPSS) was used to test the relationship between citations in the three bibliographic databases and reads and readerships in RG and Mendeley.

Findings

The main findings were that the majority of NWU’s NRF-rated researchers’ publications emanated from GS, followed by Scopus and then WoS. GS output also had more citations. There were 6,026 research outputs in RG which were read for 676,919 times and 5,850 in Mendeley with 142,621 readerships. Correlations between RG and all three bibliographic databases’ citations were scant. Strong relationships between the three bibliographic databases’ citations and Mendeley readerships were found.

Practical implications

Academic librarians who interact with researchers who would like to predict future academic impact of their documents can be advised to consider Mendeley readerships with some level of confidence compared to RG reads. These results point to the importance of constant self-evaluation by researchers to ensure that they have balanced profiles across the three main bibliographic databases that are also considered for ratings. These results point to the relevancy of GS to evaluate research beyond the academy.

Social implications

The fact that researchers are contributing research that seeks to improve the general welfare of the population (beyond the academy) is a positive sign as society look up to researchers and research to solve their socio-economic problems. Social media play an important role as they serve as indicators that indicators point to wider research impacts and wider access by many different groups of people including the members of society at large. They point to research that is accessible to not only researchers and those who have access to their research but also the society at large.

Originality/value

Although the practice of rating researchers is common in different research ecosystems, the researchers could not find any evidence of studies conducted using a combination of bibliometrics and altmetrics to asses rated researchers’ output. This study covers and compares social impact based on data obtained from two academic social media sites and three main bibliographic databases (GS, Scopus and WoS).

AUTHORSHIP MALPRACTICES IN DEVELOPING COUNTRIES

Although the variety of research malpractices in low- and middle-income countries (LMIC) is similar to those documented in developed nations, authorship misuse and related issues appear to be more prevalent in the former. This article focuses on some authorship-related topics in LMIC, namely authorship disputes, excessive co-authorship and monetary incentives, authorship issues in student-mentor relationships, and authorship patterns in North-South collaborations. Universities and official bodies in LMIC must supervise the integrity of the whole research process, provide instructions on responsible research and authorship, set up the required postgraduate training programs, foster good role models in authorship, and designate an ombudsperson to advice and respond to complaints of researchers, especially students and junior faculty. Lastly, a recent framework that combines descriptive and normative elements and then uses proper metaphors to achieve an ethical definition of authorship useful in international settings is highlighted.

Comprehensive Researcher Achievement Model (CRAM): a framework for measuring researcher achievement, impact and influence derived from a systematic literature review of metrics and models

OBJECTIVES

Effective researcher assessment is key to decisions about funding allocations, promotion and tenure. We aimed to identify what is known about methods for assessing researcher achievements, leading to a new composite assessment model.

DESIGN

We systematically reviewed the literature via the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols framework.

DATA SOURCES

All Web of Science databases (including Core Collection, MEDLINE and BIOSIS Citation Index) to the end of 2017. ELIGIBILITY CRITERIA: (1) English language, (2) published in the last 10 years (2007-2017), (3) full text was available and (4) the article discussed an approach to the assessment of an individual researcher's achievements.

DATA EXTRACTION AND SYNTHESIS

Articles were allocated among four pairs of reviewers for screening, with each pair randomly assigned 5% of their allocation to review concurrently against inclusion criteria. Inter-rater reliability was assessed using Cohen's Kappa (ĸ). The ĸ statistic showed agreement ranging from moderate to almost perfect (0.4848-0.9039). Following screening, selected articles underwent full-text review and bias was assessed.

RESULTS

Four hundred and seventy-eight articles were included in the final review. Established approaches developed prior to our inclusion period (eg, citations and outputs, h-index and journal impact factor) remained dominant in the literature and in practice. New bibliometric methods and models emerged in the last 10 years including: measures based on PageRank algorithms or 'altmetric' data, methods to apply peer judgement and techniques to assign values to publication quantity and quality. Each assessment method tended to prioritise certain aspects of achievement over others.

CONCLUSIONS

All metrics and models focus on an element or elements at the expense of others. A new composite design, the Comprehensive Researcher Achievement Model (CRAM), is presented, which supersedes past anachronistic models. The CRAM is modifiable to a range of applications.