What do the scientists think about the impact factor?

Bibliometrics in Press. Representations and uses of bibliometric indicators in the Italian daily newspapers

Scholars in science and technology studies and bibliometricians are increasingly revealing the performative nature of bibliometric indicators. Far from being neutral technical measures, indicators such as the Impact Factor and the h-index are deeply transforming the social and epistemic structures of contemporary science. At the same time, scholars have highlighted how bibliometric indicators are endowed with social meanings that go beyond their purely technical definitions. These social representations of bibliometric indicators are constructed and negotiated between different groups of actors within several arenas. This study aims to investigate how bibliometric indicators are used in a context, which, so far, has not yet been covered by researchers, that of daily newspapers. By a content analysis of a corpus of 583 articles that appeared in four major Italian newspapers between 1990 and 2020, we chronicle the main functions that bibliometrics and bibliometric indicators played in the Italian press. Our material shows, among other things, that the public discourse developed in newspapers creates a favorable environment for bibliometrics-centered science policies, that bibliometric indicators contribute to the social construction of scientific facts in the press, especially in science news related to medicine, and that professional bibliometric expertise struggles to be represented in newspapers and hence reach the general public.

Philosophers’ appraisals of bibliometric indicators and their use in evaluation: from recognition to knee-jerk rejection

The knowledge and stance of researchers regarding bibliometric indicators is a field of study that has gained weight in recent decades. In this paper we address this issue for the little explored areas of philosophy and ethics, and applied to a context, in this case Spain, where bibliometric indicators are widely used in evaluation processes. The study combines data from a self-administered questionnaire completed by 201 researchers and from 14 in-depth interviews with researchers selected according to their affiliation, professional category, gender and area of knowledge. The survey data suggest that researchers do not consider bibliometric indicators a preferred criterion of quality, while there is a fairly high self-perception of awareness of a number of indicators. The qualitative data points to a generalised perception of a certain rejection of the specific use of indicators, with four main positions being observed: (1) disqualification of the logic of metrics, (2) scepticism about the possibility of assessing quality with quantitative methods, (3) complaints about the incorporation of methods that are considered to belong to other disciplines, and (4) criticism of the consequences that this generates in the discipline of philosophy.

Russian University Teachers’ Perceptions of Scientometrics

This article is devoted to the attitude of Russian university teachers toward scientometrics and its indicators, which have been imposed on them by university administrations and the state since 2012. In addition to substantiating the problem’s urgency, the article contains a brief critical outline of the main scientometric parameters and their application in practice in Russia. To evaluate this, 283 people from leading universities in Russia (included in Program 5-100) were questioned. As the study showed, faculties of Russian universities understand the specifics of scientometrics, relate to it relatively positively, and over the past years have been able to adapt to the new requirements of the administration regarding implementing scientometric tasks and standards. The higher the position and scholarly qualification of a respondent, the more complete the knowledge about scientometrics. Scholars in the humanities know and relate comparatively better to scientometrics than representatives of technical and general scientific specialties.

Do researchers know what the h-index is? And how do they estimate its importance?

The h-index is a widely used scientometric indicator on the researcher level working with a simple combination of publication and citation counts. In this article, we pursue two goals, namely the collection of empirical data about researchers’ personal estimations of the importance of the h-index for themselves as well as for their academic disciplines, and on the researchers’ concrete knowledge on the h-index and the way of its calculation. We worked with an online survey (including a knowledge test on the calculation of the h-index), which was finished by 1081 German university professors. We distinguished between the results for all participants, and, additionally, the results by gender, generation, and field of knowledge. We found a clear binary division between the academic knowledge fields: For the sciences and medicine the h-index is important for the researchers themselves and for their disciplines, while for the humanities and social sciences, economics, and law the h-index is considerably less important. Two fifths of the professors do not know details on the h-index or wrongly deem to know what the h-index is and failed our test. The researchers’ knowledge on the h-index is much smaller in the academic branches of the humanities and the social sciences. As the h-index is important for many researchers and as not all researchers are very knowledgeable about this author-specific indicator, it seems to be necessary to make researchers more aware of scholarly metrics literacy.

Past, current, and future perspectives on eco-tourism: a bibliometric review between 2001 and 2018

With increasing attentions on climate change, solid wastes, over-tourism, and improved environmental awareness worldwide, eco-tourism has been widely promoted worldwide. This requires that governmental agencies at different levels should prepare appropriate policies to facilitate eco-tourism and local communities should take necessary actions to preserve their natural resources, protect their environment, and support sustainable tourism. Academically, literatures on eco-tourism have increased faster with an annual growth rate of 10-30% during the recent years. Under such a circumstance, it is important to conduct a comprehensive review so that research progress on eco-tourism can be summarized and future research directions can be identified. Based on 1771 publications published during 2001-2018, a systematic method combining bibliometric analysis and network analysis is applied in this study to uncover the dynamic trends, academic collaboration, and research hotspots related with eco-tourism. Results show that the total number of relevant publications has gradually increased. Key journals include Journal of Hospitality and Tourism Management, Annals of Tourism Research, Conservation Biology, and Biological Conservation. Authors from USA have the most publications and international co-authorships, while the most influential institution is the Chinese Academy of Science. Moreover, research keywords have been identified, including eco-tourism, management, biodiversity, national park, sustainability, and sustainable tourism. Research findings of this study provide valuable insights to further improve eco-tourism research so that this emerging research field can be proactively fostered.

Hobson’s choice: the effects of research evaluation on academics’ writing practices in England

Purpose

The purpose of this paper is to examine the influence of research evaluation policies and their interpretation on academics’ writing practices in three different higher education institutions and across three different disciplines. Specifically, the paper discusses how England’s national research excellence framework (REF) and institutional responses to it shape the decisions academics make about their writing.

Design/methodology/approach

In total, 49 academics at three English universities were interviewed. The academics were from one Science, Technology, Engineering and Mathematics discipline (mathematics), one humanities discipline (history) and one applied discipline (marketing). Repeated semi-structured interviews focussed on different aspects of academics’ writing practices. Heads of departments and administrative staff were also interviewed. Data were coded using the qualitative data analysis software, ATLAS.ti.

Findings

Academics’ ability to succeed in their career was closely tied to their ability to meet quantitative and qualitative targets driven by research evaluation systems, but these were predicated on an unrealistic understanding of knowledge creation. Research evaluation systems limited the epistemic choices available to academics, partly because they pushed academics’ writing towards genres and publication venues that conflicted with disciplinary traditions and partly because they were evenly distributed across institutions and age groups.

Originality/value

This work fills a gap in the literature by offering empirical and qualitative findings on the effects of research evaluation systems in context. It is also one of the only papers to focus on the ways in which individuals’ academic writing practices in particular are shaped by such systems.

How to reduce the number of rating scale items without predictability loss?

Rating scales are used to elicit data about qualitative entities (e.g., research collaboration). This study presents an innovative method for reducing the number of rating scale items without the predictability loss. The “area under the receiver operator curve method” (AUC ROC) is used. The presented method has reduced the number of rating scale items (variables) to 28.57% (from 21 to 6) making over 70% of collected data unnecessary. Results have been verified by two methods of analysis: Graded Response Model (GRM) and Confirmatory Factor Analysis (CFA). GRM revealed that the new method differentiates observations of high and middle scores. CFA proved that the reliability of the rating scale has not deteriorated by the scale item reduction. Both statistical analysis evidenced usefulness of the AUC ROC reduction method.

Factors associated with citation rate of randomised controlled trials in physiotherapy

Background

Despite the use of citation rate as a measure of quality of research is strongly criticized and debated, it remain a widely used method to evaluate performances of researchers, articles and journals. The aim of this study was to test which factors are associated with citation rate of Randomised Controlled Trials (RCTs) published on the physiotherapy field.

Methods

All RCTs abstracted in the Physiotherapy Evidence Database (PEDro), indexed in Scopus database and published in 2008 were included. PEDro score, language of publication, indexing in PubMed database, type of access to articles, subdiscipline, the number of authors, the country where the study was performed, the type of institution where the study was conducted and the number of centres involved in the study (multicentric vs single-centre). and the 2013 5-year impact factor of the publishing journals were considered as independent variables. Citation rate until December 2013 was extracted from Scopus database and used as dependent variable.

Results

Six hundred and nineteen RCTs, published in 283 journals, were included and analysed. The 5-year impact factor was the strongest variable associated with the citation rate and explained approximately 50 % of the variance, and the number of authors explained an additional small part (about 1 %) of variability. The other variables were excluded from the model.

Conclusions

The study highlights that 5-year Impact Factor, not accessibility (language of publication, indexing in PubMed database or the type of access to articles) or reported quality (PEDro score), is a strong predictor of the number of citations for RCTs in the physiotherapy field. Our findings support the increasingly widespread idea that citation analysis does not reflect the scientific merit of the cited work, at least in terms of reported quality.

The results of this study need to be confirmed with a publication window larger than one year.

Accounting for Impact? The Journal Impact Factor and the Making of Biomedical Research in the Netherlands

The range and types of performance metrics has recently proliferated in academic settings, with bibliometric indicators being particularly visible examples. One field that has traditionally been hospitable towards such indicators is biomedicine. Here the relative merits of bibliometrics are widely discussed, with debates often portraying them as heroes or villains. Despite a plethora of controversies, one of the most widely used indicators in this field is said to be the Journal Impact Factor (JIF). In this article we argue that much of the current debates around researchers' uses of the JIF in biomedicine can be classed as 'folk theories': explanatory accounts told among a community that seldom (if ever) get systematically checked. Such accounts rarely disclose how knowledge production itself becomes more-or-less consolidated around the JIF. Using ethnographic materials from different research sites in Dutch University Medical Centers, this article sheds new empirical and theoretical light on how performance metrics variously shape biomedical research on the 'shop floor.' Our detailed analysis underscores a need for further research into the constitutive effects of evaluative metrics.