What makes databases tick: Logics that underpin national databases for research output in Europe

The increasing use of research metrics in the monitoring and evaluation of research has led to the expansion of research information infrastructure, as indicated by the rising number of national databases for research output and research information systems. At the same time, debates on requirements of national databases for research output are typically guided by simplistic assumptions about what an infrastructure is and how it shapes our understanding of different phenomena. We present an alternative framing of databases wherein infrastructure and context are seen as mutually constitutive. Drawing upon qualitative data on 12 national databases in Europe, we investigate the role that databases for research output play in research metrics and consequently in our understanding of research activities. Specifically, we bring to the surface database features that are typically left implicit and yet are crucial for understanding these databases in context. A key outcome of this study is an identification of two database logics—New Public Management and Enlightenment—that underpin database designs and the organization of database work. These logics do not merely guide practical database work (e.g. by determining the features of highest priority), but through the outcomes of that work they both enact and distribute a particular conception of what research is. In this article, we show how the two logics manifest and how they are related to each other. Situating these findings in the landscape of developments in research information infrastructure, we discuss implications for research evaluation and the understanding of research in general.

Research performance and scholarly communication profile of competitive research funding: the case of Academy of Finland

The Academy of Finland (AKA), Finland’s major public research funding agency, uses a Web of Science (WoS) based bibliometric indicator to assess the performance of research it has funded. We use an alternative methodology to compare (1) the research performance and (2) the scholarly communication profile of AKA-funded research to the Finnish universities’ entire output across the major fields of arts and sciences. Our data consists of 142,742 publications (years 2015–2018) registered in the national information service, which integrates Current Research Information System (CRIS) data of 13 Finnish universities. Research performance is analyzed using the Finnish community-curated expert-based rating of publication channels (so-called JUFO). Our results show that compared to the Finnish universities’ entire output a larger share of AKA-funded research is published in leading JUFO rated journals and book publishers. JUFO and WoS-based indicators produced consonant results regarding the performance of AKA-funded research. Analysis of publication profiles shows that AKA-funded research is more focused than the universities’ output on using peer-reviewed publications, articles published in journals, English language, foreign publishers and open access publishing. We conclude that the CRIS-based publication data can support multidimensional assessments of research performance and scholarly communication profiles, potentially also in other countries and institutions. CRIS development and maintenance require multi-stakeholder commitment, resources and incentives to ensure data quality and coverage. To fully recognize diverse open science practices and to enable international comparisons, CRISs need further development and integration as data sources.

Citation patterns between impact-factor and questionable journals

One of the most fundamental issues in academia today is understanding the differences between legitimate and questionable publishing. While decision-makers and managers consider journals indexed in popular citation indexes such as Web of Science or Scopus as legitimate, they use two lists of questionable journals (Beall’s and Cabell’s), one of which has not been updated for a few years, to identify the so-called predatory journals. The main aim of our study is to reveal the contribution of the journals accepted as legitimate by the authorities to the visibility of questionable journals. For this purpose, 65 questionable journals from social sciences and 2338 Web-of-Science-indexed journals that cited these questionable journals were examined in-depth in terms of index coverages, subject categories, impact factors and self-citation patterns. We have analysed 3234 unique cited papers from questionable journals and 5964 unique citing papers (6750 citations of cited papers) from Web of Science journals. We found that 13% of the questionable papers were cited by WoS journals and 37% of the citations were from impact-factor journals. The findings show that neither the impact factor of citing journals nor the size of cited journals is a good predictor of the number of citations to the questionable journals.

Towards complexity-sensitive book metrics for scholarly monographs in national databases for research output

Purpose

This study investigates an approach to book metrics for research evaluation that takes into account the complexity of scholarly monographs. This approach is based on work sets – unique scholarly works and their within-work related bibliographic entities – for scholarly monographs in national databases for research output.

Design/methodology/approach

This study examines bibliographic records on scholarly monographs acquired from four European databases (VABB in Flanders, Belgium; CROSBI in Croatia; CRISTIN in Norway; COBISS in Slovenia). Following a data enrichment process using metadata from OCLC WorldCat and Amazon Goodreads, the authors identify work sets and the corresponding ISBNs. Next, on the basis of the number of ISBNs per work set and the presence in WorldCat, they design a typology of scholarly monographs: Globally visible single-expression works, Globally visible multi-expression works, Miscellaneous and Globally invisible works.

Findings

The findings show that the concept “work set” and the proposed typology can aid the identification of influential scholarly monographs in the social sciences and humanities (i.e. the Globally visible multi-expression works).

Practical implications

In light of the findings, the authors outline requirements for the bibliographic control of scholarly monographs in national databases for research output that facilitate the use of the approach proposed here.

Originality/value

The authors use insights from library and information science (LIS) to construct complexity-sensitive book metrics. In doing so, the authors, on the one hand, propose a solution to a problem in research evaluation and, on the other hand, bring to attention the need for a dialogue between LIS and neighbouring communities that work with bibliographic data.

Journal article publishing in the social sciences and humanities: A comparison of Web of Science coverage for five European countries

This study compares publication pattern dynamics in the social sciences and humanities in five European countries. Three are Central and Eastern European countries that share a similar cultural and political heritage (the Czech Republic, Slovakia, and Poland). The other two are Flanders (Belgium) and Norway, representing Western Europe and the Nordics, respectively. We analysed 449,409 publications from 2013–2016 and found that, despite persisting differences between the two groups of countries across all disciplines, publication patterns in the Central and Eastern European countries are becoming more similar to those in their Western and Nordic counterparts. Articles from the Central and Eastern European countries are increasingly published in journals indexed in Web of Science and also in journals with the highest citation impacts. There are, however, clear differences between social science and humanities disciplines, which need to be considered in research evaluation and science policy.

Tracing the context in disciplinary classifications: A bibliometric pairwise comparison of five classifications of journals in the social sciences and humanities

Despite the centrality of disciplinary classifications in bibliometric analyses, it is not well known how the choice of disciplinary classification influences bibliometric representations of research in the social sciences and humanities (SSH). This is especially crucial when using data from national databases. Therefore, we examine the differences in the disciplinary profile of an article along with the absolute and relative number of articles across disciplines using five disciplinary classifications for journals. We use data on journal articles (2006–2015) from the national bibliographic databases VABB-SHW in Flanders (Belgium) and Cristin in Norway. Our study is based on pairwise comparisons of the local classifications used in these databases, the Web of Science subject categories, the Science-Metrix, and the ERIH PLUS journal classifications. For comparability, all classifications are mapped to the OECD Fields of Research and Development classification. The findings show that the choice of disciplinary classification can lead to over- or underestimation of the absolute number of publications per discipline. In contrast, if the focus is on the relative numbers, the choice of classification has practically no influence. These findings facilitate an informed choice of a disciplinary classification for journals in SSH when using data from national databases.

Representation of Slovak Research Information (A Case Study)

In the light of the increasing importance of the societal impact of research, this article attempts to address the question as to how social sciences and humanities (SSH) research outputs from 2019 are represented in Slovak research portfolios in comparison with those of the EU-28 and the world. The data used for the analysis originate from the R&D SK CRIS and bibliographic Central Register of Publication Activities (CREPČ) national databases, and WoS Core Collection/InCites. The research data were appropriate for the analysis at the time they were structured, on the national level; of high quality and consistency; and covering as many components as possible and in mutual relations. The data resources should enable the research outputs to be assigned to research categories. The analysis prompts the conclusion that social sciences and humanities research outputs in Slovakia in 2019 are appropriately represented and in general show an increasing trend. This can be documented by the proportion represented by the SSH research projects and other entities involved in the overall Slovak research outputs, and even the higher ratio of SSH research publications in comparison with the EU-28 and the world. Recommendations of a technical character include research data management, data quality, and the integration of individual systems and available analytical tools.

Open access at the national level: A comprehensive analysis of publications by Finnish researchers

Open access (OA) has mostly been studied by relying on publication data from selective international databases, notably Web of Science (WoS) and Scopus. The aim of our study is to show that it is possible to achieve a national estimate of the number and share of OA based on institutional publication data providing a comprehensive coverage of the peer-reviewed outputs across fields, publication types, and languages. Our data consists of 48,177 journal, conference, and book publications from 14 Finnish universities in 2016–2017, including information about OA status, as self-reported by researchers and validated by data-collection personnel through their Current Research Information System (CRIS). We investigate the WoS, Scopus, and DOI coverage, as well as the share of OA outputs between different fields, publication types, languages, OA mechanisms (gold, hybrid, and green), and OA information sources (DOAJ, Bielefeld list, and Sherpa/Romeo). We also estimate the role of the largest international commercial publishers compared to the not-for-profit Finnish national publishers of journals and books. We conclude that institutional data, integrated at national and international level, provides one of the building blocks of a large-scale data infrastructure needed for comprehensive assessment and monitoring of OA across countries, for example at the European level.

Multilingual publishing in the social sciences and humanities: A seven-country European study

We investigate the state of multilingualism across the social sciences and humanities (SSH) using a comprehensive data set of research outputs from seven European countries (Czech Republic, Denmark, Finland, Flanders [Belgium], Norway, Poland, and Slovenia). Although English tends to be the dominant language of science, SSH researchers often produce culturally and societally relevant work in their local languages. We collected and analyzed a set of 164,218 peer-reviewed journal articles (produced by 51,063 researchers from 2013 to 2015) and found that multilingualism is prevalent despite geographical location and field. Among the researchers who published at least three journal articles during this time period, over one-third from the various countries had written their work in at least two languages. The highest share of researchers who published in only one language were from Flanders (80.9%), whereas the lowest shares were from Slovenia (57.2%) and Poland (59.3%). Our findings show that multilingual publishing is an ongoing practice in many SSH research fields regardless of geographical location, political situation, and/or historical heritage. Here we argue that research is international, but multilingual publishing keeps locally relevant research alive with the added potential for creating impact.

Professional standards in bibliometric research evaluation? A meta-evaluation of European assessment practice 2005-2019

Despite growing demand for practicable methods of research evaluation, the use of bibliometric indicators remains controversial. This paper examines performance assessment practice in Europe-first, identifying the most commonly used bibliometric methods and, second, identifying the actors who have defined wide-spread practices. The framework of this investigation is Abbott's theory of professions, and I argue that indicator-based research assessment constitutes a potential jurisdiction for both individual experts and expert organizations. This investigation was conducted using a search methodology that yielded 138 evaluation studies from 21 EU countries, covering the period 2005 to 2019. Structured content analysis revealed the following findings: (1) Bibliometric research assessment is most frequently performed in the Nordic countries, the Netherlands, Italy, and the United Kingdom. (2) The Web of Science (WoS) is the dominant database used for public research assessment in Europe. (3) Expert organizations invest in the improvement of WoS citation data, and set technical standards with regards to data quality. (4) Citation impact is most frequently assessed with reference to international scientific fields. (5) The WoS classification of science fields retained its function as a de facto reference standard for research performance assessment. A detailed comparison of assessment practices between five dedicated organizations and other individual bibliometric experts suggests that corporate ownership and limited access to the most widely used citation databases have had a restraining effect on the development and diffusion of professional bibliometric methods during this period.

Methodological considerations for identifying questionable publishing in a national context: The case of Swedish Higher Education Institutions

The overall scope of this study is an attempt at a methodological framework for matching publication lists at the national level against a combined set of blacklists for questionable publishing. Using the total verified set of publications from Swedish Higher Education Institutions (HEI) as a case, we examined the number, distribution, and proportion of publishing in questionable journals at the national level. Journal publication data was extracted from the national SwePub database and matched against three curated blacklists of questionable publishing. For the period 2012–2017, we identified 1,743 published papers in blacklisted journals, equal to an average of 0.5–0.9% of the total publications from Swedish HEIs. There was high variability between different HEI categories, with more established universities at the lower end of the spectrum, while university colleges and new universities had a much higher proportion (∼2%). There was a general decreasing trend during the study period (ρ = 0.83) for all categories of HEIs. The study presents a methodology to identify questionable publishing in academia that could be applied to other countries with similar infrastructure. Thus, it could serve as a starting point for the development of a general framework for cross-national quantitative estimation of questionable publishing.

Researchers publishing monographs are more productive and more local-oriented

In this study, we investigate what share of researchers publish monographs across fields, gender and seniority. We acquired data from the Polish current research information system (POL-on) containing metadata of about 1,031,141 peer-reviewed publications from 67,415 Polish researchers, including 30,185 monographs from 2013 to 2016. The data are aggregated at the researcher level, which allows us to shed new light on publication patterns in all fields. We show that scholars who publish monographs also publish journal articles at the same time. This pattern is observed in all dimensions, i.e. research fields, gender and seniority. However, substantial differences between the fields are observed. Moreover, we show that researchers who publish monographs are also more productive in terms of the number of publications than researchers who did not publish any monographs. This result is independent of the publication counting method, i.e. fractional or whole counting. At the same time, scholars who publish monographs are more local-oriented in terms of the publication channels they choose.

Identifying publications in questionable journals in the context of performance-based research funding

In this article we discuss the five yearly screenings for publications in questionable journals which have been carried out in the context of the performance-based research funding model in Flanders, Belgium. The Flemish funding model expanded from 2010 onwards, with a comprehensive bibliographic database for research output in the social sciences and humanities. Along with an overview of the procedures followed during the screenings for articles in questionable journals submitted for inclusion in this database, we present a bibliographic analysis of the publications identified. First, we show how the yearly number of publications in questionable journals has evolved over the period 2003–2016. Second, we present a disciplinary classification of the identified journals. In the third part of the results section, three authorship characteristics are discussed: multi-authorship, the seniority–or experience level–of authors in general and of the first author in particular, and the relation of the disciplinary scope of the journal (cognitive classification) with the departmental affiliation of the authors (organizational classification). Our results regarding yearly rates of publications in questionable journals indicate that awareness of the risks of questionable journals does not lead to a turn away from open access in general. The number of publications in open access journals rises every year, while the number of publications in questionable journals decreases from 2012 onwards. We find further that both early career and more senior researchers publish in questionable journals. We show that the average proportion of senior authors contributing to publications in questionable journals is somewhat higher than that for publications in open access journals. In addition, this paper yields insight into the extent to which publications in questionable journals pose a threat to the public and political legitimacy of a performance-based research funding system of a western European region. We include concrete suggestions for those tasked with maintaining bibliographic databases and screening for publications in questionable journals.

Money, morale, and motivation: a study of the Output-Based Research Support Scheme in University College Dublin

Adapted from the Norwegian model, University College Dublin has implemented the Output-Based Research Support Scheme (OBRSS) to stimulate research performance. Semi-structured interviews were conducted to understand the perception of the OBRSS after two years of implementation, its implications on research and publication practices, and the responses to monetary reward as an incentive. This study shows that the effects of the OBRSS on publication practices are intertwined with intrinsic and instrumental values. More importantly and unexpectedly, the study reveals the norms and values concerning academic integrity and research culture, the importance of intrinsic motivation in research and scholarship, as well as morale issues in academic work environment. The findings are consistent with studies of self-determination theory that an incentive scheme can be highly effective if it conveys clear goals and values with a vision of enhanced intrinsic motivation; however, a scheme can be demoralizing when it is perceived as a controlling mechanism.

The Flemish Performance-based Research Funding System: A Unique Variant of the Norwegian Model

The BOF-key is the performance-based research funding system that is used in Flanders, Belgium. In this paper we describe the historical background of the system, its current design and organization, as well as its effects on the Flemish higher education landscape. The BOF-key in its current form relies on three bibliometric parameters: publications in Web of Science, citations in Web of Science, and publications in a comprehensive regional database for SSH publications. Taken together, the BOF-key forms a unique variant of the Norwegian model: while the system to a large extent relies on a commercial database, it avoids the problem of inadequate coverage of the SSH. Because the bibliometric parameters of the BOF-key are reused in other funding allocation schemes, their overall importance to the Flemish universities is substantial.

Applications of, and Experiences with, the Norwegian Model in Finland

The purpose of this article is to describe the development, components and properties of a publication indicator that the Ministry of Education and Culture in Finland uses for allocating direct core funding annually to universities. Since 2013, 13% of the core funding has been allocated on basis of publication indicator that, like the Norwegian model, is based on comprehensive national level publication data that is currently provided by the VIRTA publication information service. In 2015, the publication indicator was complemented with other components of the Norwegian model, namely, quality-weighted publication counts based on national Publication Forum authority list of the publication channels with ratings established by experts in the field. The funding model allocates around 1.6 billion euros annually to universities with the publication indicator annually distributing over 200 million euros. Besides the funding model, the indicator provides comparable data for monitoring the research performance of Finnish universities, fields and subunits. The indicator may also be used in the universities’ local funding models and research management systems, sometimes even at individual level evaluation. Positive and negative effects of the indicator have been extensively discussed and speculated. Since 2011, the Finnish universities’ productivity appears to have increased in terms of both quantity and quality of publications.

How arbitrary are the weights assigned to books in performance-based research funding? An empirical assessment of the weight and size of monographs in Flanders

Purpose

The purpose of this paper is to present an empirical assessment of the weight assigned to monographs in the publication indicator of the performance-based research funding system (PRFS) in Flanders, Belgium. By relating publication weight to publication size the authors offer an alternative perspective on the production of scholars who publish monographs. This perspective on weights is linked to the aggregation level at which PRFS indicators are used: the national/regional one as opposed to the local one. In Flanders as elsewhere the publication indicator designed for funding distribution between universities has sometimes trickled down to institutions, their faculties and departments.

Design/methodology/approach

As an alternative indicator of scholarly production the authors propose the median number of pages of a publication type. Measuring the size of publications allows to compare the weight ratio between monographs and journal articles in the publication indicator to their size ratio in the VABB-SHW database. The authors compare two levels, one of four universities and one of 16 disciplines.

Findings

Median publication size differences between disciplines are much larger than those between universities. This indicates that an increase of monographs’ weight in the publication indicator would hardly affect funding distribution at the regional level. Disciplines with a relatively large share of monographs, however, would contribute more to the publication indicator. Hence an increase of monographs’ weight might provide a better balance between fields and between publication types.

Originality/value

This paper presents a thought experiment regarding the weight assigned to different publication types in the publication indicator of the Flemish PRFS: what would happen if this weight were replaced by the median number of pages of a publication type? In doing so, we highlight that such weighting schemes play an important role in finding a balance between fields of research. The sizeable differences between weight and size ratios offer a new and critical perspective on the weighting schemes currently used in PRFS, also in other countries.

Scholarly book publishers’ ratings and lists in Finland and Spain

Purpose

The purpose of this paper is twofold: first, to compare the lists of publishers in SPI (Spain) and the lists of VIRTA (Finland), in order to determine some of the potential uses of a merged list, such as complementing each other; and, second, to assess the effects of cross-field variability in the SPI rankings on the potential uses identified in the previous objective.

Design/methodology/approach

VIRTA and SPI lists were matched and compared in terms of level and number of submissions (VIRTA) and prestige (SPI).

Findings

There is a set of international publishers common to both information systems, but most publishers are nationally oriented. This type of publisher is still highly relevant for scholars. Consequently, a merge of national lists would provide useful information for all stakeholders involved in terms of grounding information for the rating of foreign, non-international publishers. Nevertheless, several issues should be considered in an eventual merging process, such as the decisions related to the use of field-specific rankings or general rankings.

Practical implications

If merged, ratings ought to be kept separately. Ratings of national publishers can be imputed in other systems’ evaluation process, thus making the merging process potentially useful.

Originality/value

This research explores obstacles and opportunities for merging scholarly publishers’ lists from an empirical perspective. It provides groundwork for future efforts toward supra-national combinations of publishers’ lists.