Advancing science or advancing careers? Researchers' opinions on success indicators

Mismatch in perceptions of the quality of supervision and research data management as an area of concern: Results from a university-wide survey of the research integrity culture at a Belgian university

Researchers of KU Leuven, a large Belgian university, were invited to complete a bespoke questionnaire assessing their attitudes toward research integrity and the local research culture, with specific emphasis on the supervision of junior researchers. A total of 7,353 invitations were sent via e-mail and 1,866 responses were collected (25.3% response rate), of which 1,723 responses are reported upon here. Some of the findings are relevant to the broader research community. Whereas supervisors evaluated their supervision of junior researchers almost unanimously as positive, fewer supervisees evaluated it as such. Data management emerged as an area of concern, both in terms of reviewing raw data and of data storage. More female than male professors emphasized open communication and supported their supervisees' professional development and personal well-being. At the same time, fewer female professors felt safe to speak up than male professors. Finally, researchers who obtained their master's degree outside Europe evaluated their supervision and KU Leuven's research culture more positively than researchers with a master's degree from KU Leuven. The results of the survey were fed back to the university's board and several bodies and served as input to update the university's research policy. Faculties and departments received a detailed report.

Fostering open science and responsible research practices: A pre-post study

Background

Educational initiatives could foster the adoption of open science (OS) and responsible research practices (RRPs). This single group pre-post study evaluated the impact of an educational intervention on increasing the adherence, knowledge and perceptions about adopting OS practices and RRPs among graduate researchers at a Brazilian University.

Methods

Graduate students from a southern Brazilian university were invited to participate in a course addressing OS and RRPs. The intervention was an online interactive course on OS and RRPs. The number of OS outputs, including Open Science Framework (OSF) accounts, study registrations, protocols, analysis plans, data sets, preprints, and the number of projects published by each participant were collected before and after the intervention. Additionally, a self-administered online questionnaire was applied before and after the intervention to evaluate participants' perceptions on RRPs, OS practices and on the current researchers' evaluation system.

Results

Eighty-four students finished the course and 80 agreed to participate in the study. The number of OSF accounts increased from 7 to 78 after the course, and the number of projects increased from 7 to 10, six months after the intervention. No registrations, protocols, analysis plans, data sets, or preprints were found after 6 and 12 months, respectively. The participants' perceptions of the current research evaluation system and on the OS practices and RRPs changed positively with the intervention. Also, the intention to adopt practices like registration, protocol and preprint publications has noticeably increased after the course.

Conclusions

The number of participants' OSF outputs showed little or no improvement after the intervention. The most important impact difference could be identified in terms of the participants' perceptions and intentions to adhere to such practices in the future.

Regional and institutional trends in assessment for academic promotion

The assessment of research performance is widely seen as a vital tool in upholding the highest standards of quality, with selection and competition believed to drive progress. Academic institutions need to take critical decisions on hiring and promotion, while facing external pressure by also being subject to research assessment1-4. Here we present an outlook on research assessment for career progression with specific focus on promotion to full professorship, based on 314 policies from 190 academic institutions and 218 policies from 58 government agencies, covering 32 countries in the Global North and 89 countries in the Global South. We investigated how frequently various promotion criteria are mentioned and carried out a statistical analysis to infer commonalities and differences across policies. Although quantitative methods of assessment remain popular, in agreement with what is found in more geographically restricted studies5-9, they are not omnipresent. We find differences between the Global North and the Global South as well as between institutional and national policies, but less so between disciplines. A preference for bibliometric indicators is more marked in upper-middle-income countries. Although we see some variation, many promotion policies are based on the assumption of specific career paths that become normative rather than embracing diversity. In turn, this restricts opportunities for researchers. These results challenge current practice and have strategic implications for researchers, research managers and national governments.

Perceptions of research integrity and open science practices: a survey of Brazilian dental researchers

The objective of this study was to evaluate Brazilian dental researchers' perceptions of research integrity and open science practices, as well as their perceptions of the way researchers are evaluated for promotion, hiring, and receiving grants. In a self-administered online survey, the respondents were presented with 3 questions on researcher evaluation in Brazil. Additionally, for 25 academic activities or characteristics, researchers rated their perceived importance for a) career advancement, b) science advancement, c) personal satisfaction, and d) social impact. The questionnaire was sent to a total of 2,179 dental researchers working in graduate programs in dentistry in Brazil. Multilevel regressions were performed to statistically confirm the differences between the predefined subgroups. Three hundred and fifty-five (16%) researchers completed the survey. Most respondents (96.1%) considered the current evaluation system to be flawed and indicated the need for improvement. Non-traditional activities were considered more important than traditional ones for science advancement (p < 0.01), and social impact (p < 0.01), whereas traditional activities were perceived to be more important only for career advancement (p < 0.01). Although Brazilian dental researchers recognize the value of open science and research integrity practices for science advancement and impact on society, they perceive that the current evaluation system emphasizes traditional activities, such as publishing many papers in well-recognized journals as criteria for advancing their careers.

How to combine rules and commitment in fostering research integrity?

Research integrity (RI) is crucial for trustworthy research. Rules are important in setting RI standards and improving research practice, but they can lead to increased bureaucracy; without commensurate commitment amongst researchers toward RI, they are unlikely to improve research practices. In this paper, we explore how to combine rules and commitment in fostering RI. Research institutions can govern RI using markets (using incentives), bureaucracies (using rules), and network processes (through commitment and agreements). Based on Habermas' Theory of Communicative Action, we argue that network processes, as part of the lifeworld, can legitimize systems - that is, market or bureaucratic governance modes. This can regulate and support RI practices in an efficient way. Systems can also become dominant and repress consensus processes. Fostering RI requires a balance between network, market and bureaucratic governance modes. We analyze the institutional response to a serious RI case to illustrate how network processes can be combined with bureaucratic rules. Specifically, we analyze how the Science Committee established at Tilburg University in 2012 has navigated different governance modes, resulting in a normatively grounded and efficient approach to fostering RI. Based on this case, we formulate recommendations to research institutions on how to combine rules and commitment.

The SCOPE framework - implementing ideals of responsible research assessment

Background

Research and researchers are heavily evaluated, and over the past decade it has become widely acknowledged that the consequences of evaluating the research enterprise and particularly individual researchers are considerable. This has resulted in the publishing of several guidelines and principles to support moving towards more responsible research assessment (RRA). To ensure that research evaluation is meaningful, responsible, and effective the International Network of Research Management Societies (INORMS) Research Evaluation Group created the SCOPE framework enabling evaluators to deliver on existing principles of RRA. SCOPE bridges the gap between principles and their implementation by providing a structured five-stage framework by which evaluations can be designed and implemented, as well as evaluated.

Methods

SCOPE is a step-by-step process designed to help plan, design, and conduct research evaluations as well as check effectiveness of existing evaluations. In this article, four case studies are presented to show how SCOPE has been used in practice to provide value-based research evaluation.

Results

This article situates SCOPE within the international work towards more meaningful and robust research evaluation practices and shows through the four case studies how it can be used by different organisations to develop evaluations at different levels of granularity and in different settings.

Conclusions

The article demonstrates that the SCOPE framework is rooted firmly in the existing literature. In addition, it is argued that it does not simply translate existing principles of RRA into practice, but provides additional considerations not always addressed in existing RRA principles and practices thus playing a specific role in the delivery of RRA. Furthermore, the use cases show the value of SCOPE across a range of settings, including different institutional types, sizes, and missions.

Early-career factors largely determine the future impact of prominent researchers: evidence across eight scientific fields

Can we help predict the future impact of researchers using early-career factors? We analyze early-career factors of the world's 100 most prominent researchers across 8 scientific fields and identify four key drivers in researchers' initial career: working at a top 25 ranked university, publishing a paper in a top 5 ranked journal, publishing most papers in top quartile (high-impact) journals and co-authoring with other prominent researchers in their field. We find that over 95% of prominent researchers across multiple fields had at least one of these four features in the first 5 years of their career. We find that the most prominent scientists who had an early career advantage in terms of citations and h-index are more likely to have had all four features, and that this advantage persists throughout their career after 10, 15 and 20 years. Our findings show that these few early-career factors help predict researchers' impact later in their careers. Our research thus points to the need to enhance fairness and career mobility among scientists who have not had a jump start early on.

Valuation regimes in academia: Researchers’ attitudes towards their diversity of activities and academic performance

Evaluation systems have been long criticized for abusing and misusing bibliometric indicators. This has created a culture by which academics are constantly exposing their daily work to the standards they are expected to perform. In this study, we investigate whether researchers’ own values and expectations are in line with the expectations of the evaluation system. We conduct a multiple case study of five departments in two Dutch universities to examine how they balance between their own valuation regimes and the evaluation schemes. For this, we combine curriculum analysis with a series of semi-structured interviews. We propose a model to study the diversity of academic activities and apply it to the multiple case study to understand how such diversity is shaped by discipline and career stage. We conclude that the observed misalignment is not only resulting from an abuse of metrics but also by a lack of tools to evaluate performance in a contextualized and adaptable way.

DORA-compliant measures of research quality and impact to assess the performance of researchers in biomedical institutions: Review of published research, international best practice and Delphi survey

OBJECTIVE

The San Francisco Declaration on Research Assessment (DORA) advocates for assessing biomedical research quality and impact, yet academic organizations continue to employ traditional measures such as Journal Impact Factor. We aimed to identify and prioritize measures for assessing research quality and impact.

METHODS

We conducted a review of published and grey literature to identify measures of research quality and impact, which we included in an online survey. We assembled a panel of researchers and research leaders, and conducted a two-round Delphi survey to prioritize measures rated as high (rated 6 or 7 by ≥ 80% of respondents) or moderate (rated 6 or 7 by ≥ 50% of respondents) importance.

RESULTS

We identified 50 measures organized in 8 domains: relevance of the research program, challenges to research program, or productivity, team/open science, funding, innovations, publications, other dissemination, and impact. Rating of measures by 44 panelists (60%) in Round One and 24 (55%) in Round Two of a Delphi survey resulted in consensus on the high importance of 5 measures: research advances existing knowledge, research plan is innovative, an independent body of research (or fundamental role) supported by peer-reviewed research funding, research outputs relevant to discipline, and quality of the content of publications. Five measures achieved consensus on moderate importance: challenges to research productivity, potential to improve health or healthcare, team science, collaboration, and recognition by professional societies or academic bodies. There was high congruence between researchers and research leaders across disciplines.

CONCLUSIONS

Our work contributes to the field by identifying 10 DORA-compliant measures of research quality and impact, a more comprehensive and explicit set of measures than prior efforts. Research is needed to identify strategies to overcome barriers of use of DORA-compliant measures, and to "de-implement" traditional measures that do not uphold DORA principles yet are still in use.

The failure of success: four lessons learned in five years of research on research integrity and research assessments

In the past 5 years, we captured the perspectives from a broad array of research stakeholders to better understand the impact that current approaches to success and research assessment may have on the integrity and the quality of research. Here, we translate our findings in four actions that are urgently needed to foster better research. First, we need to address core research structures to overcome systemic problems of the research enterprise; second, we must realign research assessments to value elements that advance and strengthen science; third, we need to remodel, diversify, and secure research careers; and finally, we need to unite and coordinate efforts for change.

Prevalence of responsible research practices among academics in The Netherlands

Background: Traditionally, research integrity studies have focused on research misbehaviors and their explanations. Over time, attention has shifted towards preventing questionable research practices and promoting responsible ones. However, data on the prevalence of responsible research practices, especially open methods, open codes and open data and their underlying associative factors, remains scarce. Methods: We conducted a web-based anonymized questionnaire, targeting all academic researchers working at or affiliated to a university or university medical center in The Netherlands, to investigate the prevalence and potential explanatory factors of 11 responsible research practices. Results: A total of 6,813 academics completed the survey, the results of which show that prevalence of responsible practices differs substantially across disciplines and ranks, with 99 percent avoiding plagiarism in their work but less than 50 percent pre-registering a research protocol. Arts and humanities scholars as well as PhD candidates and junior researchers engaged less often in responsible research practices. Publication pressure negatively affected responsible practices, while mentoring, scientific norms subscription and funding pressure stimulated them. Conclusions: Understanding the prevalence of responsible research practices across disciplines and ranks, as well as their associated explanatory factors, can help to systematically address disciplinary- and academic rank-specific obstacles, and thereby facilitate responsible conduct of research.

Prevalence of responsible research practices among academics in The Netherlands

Background: Traditionally, research integrity studies have focused on research misbehaviors and their explanations. Over time, attention has shifted towards preventing questionable research practices and promoting responsible ones. However, data on the prevalence of responsible research practices, especially open methods, open codes and open data and their underlying associative factors, remains scarce. Methods: We conducted a web-based anonymized questionnaire, targeting all academic researchers working at or affiliated to a university or university medical center in The Netherlands, to investigate the prevalence and potential explanatory factors of 11 responsible research practices. Results: A total of 6,813 academics completed the survey, the results of which show that prevalence of responsible practices differs substantially across disciplines and ranks, with 99 percent avoiding plagiarism in their work but less than 50 percent pre-registering a research protocol. Arts and humanities scholars as well as PhD candidates and junior researchers engaged less often in responsible research practices. Publication pressure negatively affected responsible practices, while mentoring, scientific norms subscription and funding pressure stimulated them. Conclusions: Understanding the prevalence of responsible research practices across disciplines and ranks, as well as their associated explanatory factors, can help to systematically address disciplinary- and academic rank-specific obstacles, and thereby facilitate responsible conduct of research.

Prevalence of questionable research practices, research misconduct and their potential explanatory factors: A survey among academic researchers in The Netherlands

Prevalence of research misconduct, questionable research practices (QRPs) and their associations with a range of explanatory factors has not been studied sufficiently among academic researchers. The National Survey on Research Integrity targeted all disciplinary fields and academic ranks in the Netherlands. It included questions about engagement in fabrication, falsification and 11 QRPs over the previous three years, and 12 explanatory factor scales. We ensured strict identity protection and used the randomized response method for questions on research misconduct. 6,813 respondents completed the survey. Prevalence of fabrication was 4.3% (95% CI: 2.9, 5.7) and of falsification 4.2% (95% CI: 2.8, 5.6). Prevalence of QRPs ranged from 0.6% (95% CI: 0.5, 0.9) to 17.5% (95% CI: 16.4, 18.7) with 51.3% (95% CI: 50.1, 52.5) of respondents engaging frequently in at least one QRP. Being a PhD candidate or junior researcher increased the odds of frequently engaging in at least one QRP, as did being male. Scientific norm subscription (odds ratio (OR) 0.79; 95% CI: 0.63, 1.00) and perceived likelihood of detection by reviewers (OR 0.62, 95% CI: 0.44, 0.88) were associated with engaging in less research misconduct. Publication pressure was associated with more often engaging in one or more QRPs frequently (OR 1.22, 95% CI: 1.14, 1.30). We found higher prevalence of misconduct than earlier surveys. Our results suggest that greater emphasis on scientific norm subscription, strengthening reviewers in their role as gatekeepers of research quality and curbing the "publish or perish" incentive system promotes research integrity.

Speeding up to keep up: exploring the use of AI in the research process

There is a long history of the science of intelligent machines and its potential to provide scientific insights have been debated since the dawn of AI. In particular, there is renewed interest in the role of AI in research and research policy as an enabler of new methods, processes, management and evaluation which is still relatively under-explored. This empirical paper explores interviews with leading scholars on the potential impact of AI on research practice and culture through deductive, thematic analysis to show the issues affecting academics and universities today. Our interviewees identify positive and negative consequences for research and researchers with respect to collective and individual use. AI is perceived as helpful with respect to information gathering and other narrow tasks, and in support of impact and interdisciplinarity. However, using AI as a way of ‘speeding up—to keep up’ with bureaucratic and metricised processes, may proliferate negative aspects of academic culture in that the expansion of AI in research should assist and not replace human creativity. Research into the future role of AI in the research process needs to go further to address these challenges, and ask fundamental questions about how AI might assist in providing new tools able to question the values and principles driving institutions and research processes. We argue that to do this an explicit movement of meta-research on the role of AI in research should consider the effects for research and researcher creativity. Anticipatory approaches and engagement of diverse and critical voices at policy level and across disciplines should also be considered.

Creating SPACE to evolve academic assessment

Universities and research institutions have to assess individuals when making decisions about hiring, promotion and tenure, but there are concerns that such assessments are overly reliant on metrics and proxy measures of research quality that overlook important factors such as academic rigor, data sharing and mentoring. These concerns have led to calls for universities and institutions to reform the methods they use to assess research and researchers. Here we present a new tool called SPACE that has been designed to help universities and institutions implement such reforms. The tool focuses on five core capabilities and can be used by universities and institutions at all stages of reform process.