Exploring the Gray Area: Similarities and Differences in Questionable Research Practices (QRPs) Across Main Areas of Research

Is something rotten in the state of Denmark? Cross-national evidence for widespread involvement but not systematic use of questionable research practices across all fields of research

Questionable research practices (QRP) are believed to be widespread, but empirical assessments are generally restricted to a few types of practices. Furthermore, conceptual confusion is rife with use and prevalence of QRPs often being confused as the same quantity. We present the hitherto most comprehensive study examining QRPs across scholarly fields and knowledge production modes. We survey perception, use, prevalence and predictors of QRPs among 3,402 researchers in Denmark and 1,307 in the UK, USA, Croatia and Austria. Results reveal remarkably similar response patterns among Danish and international respondents (τ = 0.85). Self-reported use indicates whether respondents have used a QRP in recent publications. 9 out of 10 respondents admitted using at least one QRP. Median use is three out of nine QRP items. Self-reported prevalence reflects the frequency of use. On average, prevalence rates were roughly three times lower compared to self-reported use. Findings indicated that the perceived social acceptability of QRPs influenced self-report patterns. Results suggest that most researchers use different types of QRPs within a restricted time period. The prevalence estimates, however, do not suggest outright systematic use of specific QRPs. Perceived pressure was the strongest systemic predictor for prevalence. Conversely, more local attention to research cultures and academic age was negatively related to prevalence. Finally, the personality traits conscientiousness and, to a lesser degree, agreeableness were also inversely associated with self-reported prevalence. Findings suggest that explanations for engagement with QRPs are not only attributable to systemic factors, as hitherto suggested, but a complicated mixture of experience, systemic and individual factors, and motivated reasoning.

Commentary: Expanding the vision of Registered Reports for qualitative mental health research: A response and extension to 'Misaligned incentives in mental health research - the case for Registered Reports', Baldwin (2023)

There is a growing exploration of how Registered Reports can benefit individual researchers and wider research fields as part of a wider shift towards open research principles and practices. In 'Misaligned incentives in mental health research - the case for Registered Reports', Baldwin examines this in the context of mental health research, arguing that Registered Reports (RRs) can be a valuable solution to misaligned incentive structures in the field. However, this original piece was generally inclined towards how such incentives and the use of RRs can play out in the context of quantitative research. Such reflection is valuable, but to examine the case for RRs in mental health research as a field, we must also explore such practices within the context of qualitative research. In this commentary, we therefore expand and reframe this discussion to make the case for RRs in qualitative mental health research. We explore the place for qualitative research in the mental health research field and examine possibilities for how RRs fit within principles and practices in such methods. We discuss the various benefits and challenges of RRs in qualitative research, reflecting on our experiences as authors and reviewers of qualitative RRs and exploring how research infrastructure can facilitate engagement with this publishing approach.

Navigating the Science System: Research Integrity and Academic Survival Strategies

Research Integrity (RI) is high on the agenda of both institutions and science policy. The European Union as well as national ministries of science have launched ambitious initiatives to combat misconduct and breaches of research integrity. Often, such initiatives entail attempts to regulate scientific behavior through guidelines that institutions and academic communities can use to more easily identify and deal with cases of misconduct. Rather than framing misconduct as a result of an information deficit, we instead conceptualize Questionable Research Practices (QRPs) as attempts by researchers to reconcile epistemic and social forms of uncertainty in knowledge production. Drawing on previous literature, we define epistemic uncertainty as the inherent intellectual unpredictability of scientific inquiry, while social uncertainty arises from the human-made conditions for scientific work. Our core argument-developed on the basis of 30 focus group interviews with researchers across different fields and European countries-is that breaches of research integrity can be understood as attempts to loosen overly tight coupling between the two forms of uncertainty. Our analytical approach is not meant to relativize or excuse misconduct, but rather to offer a more fine-grained perspective on what exactly it is that researchers want to accomplish by engaging in it. Based on the analysis, we conclude by proposing some concrete ways in which institutions and academic communities could try to reconcile epistemic and social uncertainties on a more collective level, thereby reducing incentives for researchers to engage in misconduct.

Mismatch in perceptions of the quality of supervision and research data management as an area of concern: Results from a university-wide survey of the research integrity culture at a Belgian university

Researchers of KU Leuven, a large Belgian university, were invited to complete a bespoke questionnaire assessing their attitudes toward research integrity and the local research culture, with specific emphasis on the supervision of junior researchers. A total of 7,353 invitations were sent via e-mail and 1,866 responses were collected (25.3% response rate), of which 1,723 responses are reported upon here. Some of the findings are relevant to the broader research community. Whereas supervisors evaluated their supervision of junior researchers almost unanimously as positive, fewer supervisees evaluated it as such. Data management emerged as an area of concern, both in terms of reviewing raw data and of data storage. More female than male professors emphasized open communication and supported their supervisees' professional development and personal well-being. At the same time, fewer female professors felt safe to speak up than male professors. Finally, researchers who obtained their master's degree outside Europe evaluated their supervision and KU Leuven's research culture more positively than researchers with a master's degree from KU Leuven. The results of the survey were fed back to the university's board and several bodies and served as input to update the university's research policy. Faculties and departments received a detailed report.

How Competition for Funding Impacts Scientific Practice: Building Pre-fab Houses but no Cathedrals

In the research integrity literature, funding plays two different roles: it is thought to elevate questionable research practices (QRPs) due to perverse incentives, and it is a potential actor to incentivize research integrity standards. Recent studies, asking funders, have emphasized the importance of the latter. However, the perspective of active researchers on the impact of competitive research funding on science has not been explored yet. Here, I address this issue by conducting a series of group sessions with researchers in two different countries with different degrees of competition for funding, from three scientific fields (medical sciences, natural sciences, humanities), and in two different career stages (permanent versus temporary employment). Researchers across all groups experienced that competition for funding shapes science, with many unintended negative consequences. Intriguingly, these consequences had little to do with the type of QRPs typically being presented in the research integrity literature. Instead, the researchers pointed out that funding could result in predictable, fashionable, short-sighted, and overpromising science. This was seen as highly problematic: scientists experienced that the 'projectification' of science makes it more and more difficult to do any science of real importance: plunging into the unknown or addressing big issues that need a long-term horizon to mature. They also problematized unintended negative effects from collaboration and strategizing. I suggest it may be time to move away from a focus on QRPs in connection with funding, and rather address the real problems. Such a shift may then call for entirely different types of policy actions.

Interdisciplinary lessons and recommendations for the evaluation of replicability in behavioral sciences

As the scientific community becomes aware of low replicability rates in the extant literature, peer-reviewed journals have begun implementing initiatives with the goal of improving replicability. Such initiatives center around various rules to which authors must adhere to demonstrate their engagement in best practices. Preliminary evidence in the psychological science literature demonstrates a degree of efficacy in these initiatives. With such efficacy in place, it would be advantageous for other fields of behavioral sciences to adopt similar measures. This letter provides a discussion on lessons learned from psychological science while similarly addressing the unique challenges of other sciences to adopt measures that would be most appropriate for their field. We offer broad considerations for peer-reviewed journals in their implementation of specific policies and recommend that governing bodies of science prioritize the funding of research that addresses these measures.

Factors influencing the promotion and implementation of research integrity in research performing and research funding organizations: A scoping review

Promoting and implementing research integrity is considered the joint responsibility and effort of multiple stakeholders in the research community. We conducted a scoping review and analyzed 236 research articles and gray literature publications from biomedical sciences, social sciences, natural sciences (including engineering), and humanities that dealt with the factors that may positively or negatively impact the promotion and implementation of research integrity. Critical appraisal of evidence was performed for studies describing interventions aimed at research integrity promotion in order to provide insight into the effectiveness of these interventions. The results of this scoping review provide a comprehensive taxonomy of factors with positive or negative impact and their relatedness to individual researchers, research performing and funding organizations, and the system of science. Moreover, the results show that efforts for fostering and promoting research integrity should be implemented at all three levels (researcher, institution, system) simultaneously to deliver greater adherence and implementation of research integrity practices. Although various educational interventions aiming at research integrity promotion exist, we were not able to conclude on the effectiveness of explored interventions due to the methodological quality issues in the studies.

Questionable research practices in competitive grant funding: A survey

There has been a surge of interest in research integrity over the last decade, with a wide range of studies investigating the prevalence of questionable research practices (QRPs). However, nearly all these studies focus on research design, data collection and analysis, and hardly any empirical research has been done on the occurrence of QRPs in the context of research funding. To fill this gap, we conducted a cross-sectional pre-registered survey of applicants, reviewers and panel members from the Research Foundation-Flanders (FWO), one of the main funding agencies in Belgium. We developed a bespoke survey and further refined it through feedback from experienced researchers and a pilot study. We asked how often respondents had engaged in a series of QRPs over the last ten years. A total of 1748 emails were sent, inviting recipients to participate in the survey, complemented by featuring the survey in the FWO newsletter. This resulted in 704 complete responses. Our results indicate that such QRPs are remarkably prevalent. Of the 496 participants who answered both the applicant and reviewer track, more than 60% responded that they engaged regularly in at least one of such practices, and around 40% indicated that they engaged at least occasionally in half of the QRPs queried. Only 12% reported not to have engaged in any of the QRPs. Contrary to our hypotheses, male respondents did not self-report to engage in the QRPs more often than female respondents, nor was there an association between the prevalence of QRPs and self-reported success rate in grant funding. Furthermore, half of the respondents indicated that they doubted the reliability of the grant peer review process more often than not. These results suggest that preventive action is needed, and provide new reasons to reconsider the practice of allocating research money through grant peer review.

Fifty years of research on questionable research practises in science: quantitative analysis of co-citation patterns

Questionable research practises (QRPs) have been the focus of the scientific community amid greater scrutiny and evidence highlighting issues with replicability across many fields of science. To capture the most impactful publications and the main thematic domains in the literature on QRPs, this study uses a document co-citation analysis. The analysis was conducted on a sample of 341 documents that covered the past 50 years of research in QRPs. Nine major thematic clusters emerged. Statistical reporting and statistical power emerged as key areas of research, where systemic-level factors in how research is conducted are consistently raised as the precipitating factors for QRPs. There is also an encouraging shift in the focus of research into open science practises designed to address engagement in QRPs. Such a shift is indicative of the growing momentum of the open science movement, and more research can be conducted on how these practises are employed on the ground and how their uptake by researchers can be further promoted. However, the results suggest that, while pre-registration and registered reports receive the most research interest, less attention has been paid to other open science practises (e.g. data sharing).

Perception and reaction of Nanyang Technological University (NTU) researchers to different forms of research integrity education modality

BACKGROUND

Research and academic institutions use various delivery channels to deliver Research Integrity (RI) education in their communities. Yet there is no consensus on the best delivery method and the effectiveness of these channels in inculcating a positive RI culture varies across institutions. Hence, this study aimed to understand the preferences of the research community in Nanyang Technological University (NTU), Singapore.

METHODS

An online survey was conducted on NTU research community to understand their experience with, and preference for each RI education mode offered in NTU. The RI education modes surveyed in the general ranking question are Data Management Plan (DMP) workshops, Epigeum e-Learning, Compass e-newsletter (email), and NTU policy on Research Integrity and Responsible Conduct of Research. There were 242 responses, comprising 50% research students, 32.2% research staff and 17.8% faculty members. Non-parametric statistical techniques were used to analyse preferences across different RI education modes and within sub-groups (i.e., fields, age, native language, roles in research community).

RESULTS

More than 92% of respondents subscribed to the importance of RI education, but with different preferences for education modes. With respect to RI education in NTU, Compass e-newsletters were ranked the lowest (p < 0.05). Most felt that they were too wordy and unengaging, making it difficult to absorb information. Similarly, Epigeum e-Learning (p < 0.05) and 'policy' (p < 0.05) were found to be too lengthy in presentation. The compulsory NTU RI education modes (Epigeum e-learning and 'policy') enjoyed higher participation rates of 70-80% compared with 32-37% for the self-regulated modes (DMP workshop and e-newsletter). This suggests that regulatory mechanisms are still necessary to promote participation in RI education, and thus, core RI education content should be made compulsory in research/academic institutions. Although Epigeum is a compulsory course, some may not have participated in the programme due to technical issues or they might have forgotten to participate in the programme within the permissible timeframe. For all four RI education modes in NTU, the lack of awareness was among the top cited reasons for not participating.

CONCLUSIONS

Most NTU researchers perceived RI education positively although they may have reservations for some approaches. Conversely, e-Learning is favored over all the other modes except for the mode of Policy. Findings from this study are useful for improving the design of RI education strategies to be more appealing to the research community by enhancing user experience in terms of user-friendliness, relevance to specialisation, providing concise information and better presentation of materials For institutions with similar modes of RI education as NTU, these results may be relevant in improving participation rates and presentation of RI education modes, such as the use of infographics and more concise information.

Psychology Education Reform and Quality Cultivation of College Music Major From the Perspective of Entrepreneurship Education

With the continuous development of the times, music education in primary and secondary schools is constantly innovating in terms of teaching concepts, teaching methods, and implementation methods. The reform of basic music education has aroused the reflection of music teacher education in colleges and universities, and the talent training model of music teachers has become a hot issue that has been widely concerned in the new era. To explore the educational significance and influencing factors of entrepreneurship education in college music education and examine the mediating role of self-efficacy and gender adjustment in entrepreneurial education, based on the analysis of music education and entrepreneurial talent training in colleges and universities, a questionnaire survey, statistics, and data processing were adopted in this work. In this work, a path model was established based on the mediating effect and the moderating effect, and the equation model was tested. The results show that music education has a positive correlation with entrepreneurial self-efficacy and a negative correlation with entrepreneurial intention, which has not been proved to have a certain effect on entrepreneurial intention. Entrepreneurial self-efficacy has a positive correlation with entrepreneurial intention, and the mediating effect between music professional education and entrepreneurial intention has also been proved. Since there are gender differences and certain stereotypes that cannot be ignored in music education learning, gender adjustment also has a certain influence on entrepreneurial self-efficacy based on music professional education, thereby having a moderating effect on entrepreneurial intentions. Entrepreneurship education can improve the comprehensive competitiveness of students. As a result, entrepreneurship education can improve the innovation and entrepreneurship education model and the music education system in colleges and universities by combining with the relationship between entrepreneurial efficacy and intention, and it contributes to the overall cultivation of talent in colleges and universities.

Heterogeneity estimates in a biased world

Meta-analyses typically quantify heterogeneity of results, thus providing information about the consistency of the investigated effect across studies. Numerous heterogeneity estimators have been devised. Past evaluations of their performance typically presumed lack of bias in the set of studies being meta-analysed, which is often unrealistic. The present study used computer simulations to evaluate five heterogeneity estimators under a range of research conditions broadly representative of meta-analyses in psychology, with the aim to assess the impact of biases in sets of primary studies on estimates of both mean effect size and heterogeneity in meta-analyses of continuous outcome measures. To this end, six orthogonal design factors were manipulated: Strength of publication bias; 1-tailed vs. 2-tailed publication bias; prevalence of p-hacking; true heterogeneity of the effect studied; true average size of the studied effect; and number of studies per meta-analysis. Our results showed that biases in sets of primary studies caused much greater problems for the estimation of effect size than for the estimation of heterogeneity. For the latter, estimation bias remained small or moderate under most circumstances. Effect size estimations remained virtually unaffected by the choice of heterogeneity estimator. For heterogeneity estimates, however, relevant differences emerged. For unbiased primary studies, the REML estimator and (to a lesser extent) the Paule-Mandel performed well in terms of bias and variance. In biased sets of primary studies however, the Paule-Mandel estimator performed poorly, whereas the DerSimonian-Laird estimator and (to a slightly lesser extent) the REML estimator performed well. The complexity of results notwithstanding, we suggest that the REML estimator remains a good choice for meta-analyses of continuous outcome measures across varied circumstances.