Questionable research practices in student final theses - Prevalence, attitudes, and the role of the supervisor's perceived attitudes

Is something rotten in the state of Denmark? Cross-national evidence for widespread involvement but not systematic use of questionable research practices across all fields of research

Questionable research practices (QRP) are believed to be widespread, but empirical assessments are generally restricted to a few types of practices. Furthermore, conceptual confusion is rife with use and prevalence of QRPs often being confused as the same quantity. We present the hitherto most comprehensive study examining QRPs across scholarly fields and knowledge production modes. We survey perception, use, prevalence and predictors of QRPs among 3,402 researchers in Denmark and 1,307 in the UK, USA, Croatia and Austria. Results reveal remarkably similar response patterns among Danish and international respondents (τ = 0.85). Self-reported use indicates whether respondents have used a QRP in recent publications. 9 out of 10 respondents admitted using at least one QRP. Median use is three out of nine QRP items. Self-reported prevalence reflects the frequency of use. On average, prevalence rates were roughly three times lower compared to self-reported use. Findings indicated that the perceived social acceptability of QRPs influenced self-report patterns. Results suggest that most researchers use different types of QRPs within a restricted time period. The prevalence estimates, however, do not suggest outright systematic use of specific QRPs. Perceived pressure was the strongest systemic predictor for prevalence. Conversely, more local attention to research cultures and academic age was negatively related to prevalence. Finally, the personality traits conscientiousness and, to a lesser degree, agreeableness were also inversely associated with self-reported prevalence. Findings suggest that explanations for engagement with QRPs are not only attributable to systemic factors, as hitherto suggested, but a complicated mixture of experience, systemic and individual factors, and motivated reasoning.

Superb supervision: A pilot study on training supervisors to convey responsible research practices onto their PhD candidates

One way to strengthen research integrity, is through supervision. According to previous research, a supervisor should be well-versed in responsible research practices (RRPs) and possess the necessary interpersonal skills to convey RRPs. We developed a 3-day pilot training for PhD supervisors that combined RRPs and interpersonal skills. Our aim was to assess: perceptions regarding supervision skills (before and after the pilot) and participants' views on combining RRPs and interpersonal skills. Before and after the pilot, we sent the Research Supervision Quality Evaluation survey to the participating PhD supervisors and their PhD candidates. The pilot was concluded with a focus group where participants deliberated over the combination of training in interpersonal skills and RRPs and whether such training should become compulsory. Both supervisors and PhD candidates were more positive about the supervisor's interpersonal skills and the ability to foster RRPs after the training. Participants were enthusiastic about the training's dual focus but believed that making the training compulsory would be undesirable. The results highlight the potential of RRPs training for supervisors. However, caution is warranted, as the results regard a small sample of volunteering supervisors, underscoring the need for larger programs to foster responsible supervision that are rigorously evaluated.

Biomedical doctoral students' research practices when facing dilemmas: two vignette-based randomized control trials

Our aim was to describe the research practices of doctoral students facing a dilemma to research integrity and to assess the impact of inappropriate research environments, i.e. exposure to (a) a post-doctoral researcher who committed a Detrimental Research Practice (DRP) in a similar situation and (b) a supervisor who did not oppose the DRP. We conducted two 2-arm, parallel-group randomized controlled trials. We created 10 vignettes describing a realistic dilemma with two alternative courses of action (good practice versus DRP). 630 PhD students were randomized through an online system to a vignette (a) with (n = 151) or without (n = 164) exposure to a post-doctoral researcher; (b) with (n = 155) or without (n = 160) exposure to a supervisor. The primary outcome was a score from - 5 to + 5, where positive scores indicated the choice of DRP and negative scores indicated good practice. Overall, 37% of unexposed participants chose to commit DRP with important variation across vignettes (minimum 10%; maximum 66%). The mean difference [95%CI] was 0.17 [- 0.65 to 0.99;], p = 0.65 when exposed to the post-doctoral researcher, and 0.79 [- 0.38; 1.94], p = 0.16, when exposed to the supervisor. In conclusion, we did not find evidence of an impact of postdoctoral researchers and supervisors on student research practices.Trial registration: NCT04263805, NCT04263506 (registration date 11 February 2020).

The Mentor's Role in Fostering Research Integrity Standards Among New Generations of Researchers: A Review of Empirical Studies

Promoting research integrity practices among doctoral candidates and early career researchers is important for creating a stable and healthy research environment. In addition to teaching specific technical skills and knowledge, research supervisors and mentors inevitably convey research practices, both directly and indirectly. We conducted a scoping review to summarise the role of mentors in fostering research integrity practices, mentors' responsibilities and the role that institutions have in supporting good mentorship. We searched five different databases and included studies that used an empirical methodology. After searching, a total of 1199 articles were retrieved, of which 24 were eligible for analysis. After snowballing, a total of 35 empirical articles were selected. The review discusses various themes such as the importance of good mentorship, poor mentorship practices, virtues and qualities of mentors, responsibilities and activities of mentors, group mentoring and responsibilities of the institution in supporting good mentorship. This review demonstrates the importance of mentors instilling responsible research practices and attitudes, and promoting research integrity among their mentees. Mentors are responsible for providing explicit guidance and for acting as good role models. The review highlights how poor mentorship can have a bad impact on the research climate. In addition, the review highlights the important influence that institutions can have in supporting mentorship.

Teaching open and reproducible scholarship: a critical review of the evidence base for current pedagogical methods and their outcomes

In recent years, the scientific community has called for improvements in the credibility, robustness and reproducibility of research, characterized by increased interest and promotion of open and transparent research practices. While progress has been positive, there is a lack of consideration about how this approach can be embedded into undergraduate and postgraduate research training. Specifically, a critical overview of the literature which investigates how integrating open and reproducible science may influence student outcomes is needed. In this paper, we provide the first critical review of literature surrounding the integration of open and reproducible scholarship into teaching and learning and its associated outcomes in students. Our review highlighted how embedding open and reproducible scholarship appears to be associated with (i) students' scientific literacies (i.e. students' understanding of open research, consumption of science and the development of transferable skills); (ii) student engagement (i.e. motivation and engagement with learning, collaboration and engagement in open research) and (iii) students' attitudes towards science (i.e. trust in science and confidence in research findings). However, our review also identified a need for more robust and rigorous methods within pedagogical research, including more interventional and experimental evaluations of teaching practice. We discuss implications for teaching and learning scholarship.

Medical research: what to expect in a student-supervisor relationship

BACKGROUND

A medical research supervisor is of crucial importance to the undergraduate student enrolled in a research methodology course. A solid relationship between the two is vital to the success of the research project and the overall well-being of the student. The structure of the relationship between a student and a research supervisor is seldom discussed in the context of undergraduate medical research. This study evaluates students' expectations of their research contributions and their supervisors' roles.

METHODS

This was an observational study in a large health university in Saudi Arabia. A total of 320 medical students enrolled in a two-year medical research program completed an online survey, of a previously validated instrument, that is, Role Perceptions Rating Scale. Demographic questions such as the current level in the research program (junior or senior) were added.

RESULTS

The results showed that most students expected the responsibility to be equally shared between the supervisor and student during the development and execution of the research project. Additionally, students expected the research supervisor to be responsible for the research themes and contents, ensuring access to facilities, and assisting in the actual writing of the final research manuscript. Furthermore, the results indicated differences in expectations between junior and senior students.

CONCLUSION

This study demonstrates that medical students expect their research supervisors to support them to a significant extent. Understanding medical students' expectations in a supervisor-student relationship is essential to successful research and collaboration. The evidence gathered in this study has practical implications for educational institutes to base their research training program on these insights. Providing clarity on the expectations and responsibilities of those participating in the research program is crucial, as this would, in turn likely advance the output of the research program and encourage clinicians to join the program as research supervisors.

Questionable Research Practices, Low Statistical Power, and Other Obstacles to Replicability: Why Preclinical Neuroscience Research Would Benefit from Registered Reports

Replicability, the degree to which a previous scientific finding can be repeated in a distinct set of data, has been considered an integral component of institutionalized scientific practice since its inception several hundred years ago. In the past decade, large-scale replication studies have demonstrated that replicability is far from favorable, across multiple scientific fields. Here, I evaluate this literature and describe contributing factors including the prevalence of questionable research practices (QRPs), misunderstanding of p-values, and low statistical power. I subsequently discuss how these issues manifest specifically in preclinical neuroscience research. I conclude that these problems are multifaceted and difficult to solve, relying on the actions of early and late career researchers, funding sources, academic publishers, and others. I assert that any viable solution to the problem of substandard replicability must include changing academic incentives, with adoption of registered reports being the most immediately impactful and pragmatic strategy. For animal research in particular, comprehensive reporting guidelines that document potential sources of sensitivity for experimental outcomes is an essential addition.

Transparent, Open, and Reproducible Prevention Science

The field of prevention science aims to understand societal problems, identify effective interventions, and translate scientific evidence into policy and practice. There is growing interest among prevention scientists in the potential for transparency, openness, and reproducibility to facilitate this mission by providing opportunities to align scientific practice with scientific ideals, accelerate scientific discovery, and broaden access to scientific knowledge. The overarching goal of this manuscript is to serve as a primer introducing and providing an overview of open science for prevention researchers. In this paper, we discuss factors motivating interest in transparency and reproducibility, research practices associated with open science, and stakeholders engaged in and impacted by open science reform efforts. In addition, we discuss how and why different types of prevention research could incorporate open science practices, as well as ways that prevention science tools and methods could be leveraged to advance the wider open science movement. To promote further discussion, we conclude with potential reservations and challenges for the field of prevention science to address as it transitions to greater transparency, openness, and reproducibility. Throughout, we identify activities that aim to strengthen the reliability and efficiency of prevention science, facilitate access to its products and outputs, and promote collaborative and inclusive participation in research activities. By embracing principles of transparency, openness, and reproducibility, prevention science can better achieve its mission to advance evidence-based solutions to promote individual and collective well-being.

Replikationskrise, p-hacking und Open Science

Zusammenfassung. In den letzten Jahren gab es innerhalb der Psychologie eine intensive Auseinandersetzung mit den Auswirkungen der Replikationskrise sowie dem hieraus entstandenen Diskurs über die Weiterentwicklung der Disziplin. Als ein Grund für die mangelnde Replizierbarkeit psychologischer Forschung wurde die Verwendung fragwürdiger Forschungspraktiken (eng. QRPs) identifiziert. Während es umfangreiche Untersuchungen zur Prävalenz von QRPs unter Wissenschaftler*innen gibt, ist bisher wenig über die Verbreitung dieser Praktiken unter Studierenden bekannt. Mit der hier vorgestellten Arbeit wurde erstmals eine größere Befragung unter 1397 Psychologie-Studierenden im deutschsprachigen Raum durchgeführt, um die Verbreitung von QRPs in studentischen Projekten sowie den aktuellen Stand der akademischen Lehre in Bezug auf die Replikationskrise und Open Science zu erheben. Die gemeinsame Betrachtung der Lehre und des Einsatzes fragwürdiger Forschungspraktiken versprechen Aufschluss darüber, wie die psychologische Lehre mit dem empirischen Vorgehen der Studierenden zusammenhängt. Die Ergebnisse zeigen, dass QRPs auch in studentischen Projekten vorkommen, wobei große Unterschiede in der Verbreitung einzelner QRPs bestehen. Auch zwischen den verschiedenen Projekttypen zeigten sich Unterschiede, so war die Anwendung von QRPs in Experimentalpraktika am stärksten und in Masterarbeiten am schwächsten ausgeprägt. Unsere Daten weisen insgesamt darauf hin, dass die selbstberichtete Verbreitung von QRPs über den Studienverlauf abnimmt. Zudem scheint ein Großteil der Studierenden bereits mit der Thematik der Replikationskrise in der Lehre in Berührung gekommen zu sein. Deren Behandlung findet größtenteils in der Methodenlehre und weniger in inhaltlich spezialisierten Lehrveranstaltungen statt. Wir geben abschließend Impulse zur Weiterentwicklung der psychologischen Lehre, in denen die Prinzipien der Offenheit, Transparenz und Kollaboration beim Hervorbringen inhaltlich robuster Forschung bereits während des Studiums im Vordergrund stehen.

Gray (Literature) Matters: Evidence of Selective Hypothesis Reporting in Social Psychological Research

Selective reporting practices (SRPs)—adding, dropping, or altering study elements when preparing reports for publication—are thought to increase false positives in scientific research. Yet analyses of SRPs have been limited to self-reports or analyses of pre-registered and published studies. To assess SRPs in social psychological research more broadly, we compared doctoral dissertations defended between 1999 and 2017 with the publications based on those dissertations. Selective reporting occurred in nearly 50% of studies. Fully supported dissertation hypotheses were 3 times more likely to be published than unsupported hypotheses, while unsupported hypotheses were nearly 4 times more likely to be dropped from publications. Few hypotheses were found to be altered or added post hoc. Dissertation studies with fewer supported hypotheses were more likely to remove participants or measures from publications. Selective hypothesis reporting and dropped measures significantly predicted greater hypothesis support in published studies, supporting concerns that SRPs may increase Type 1 error risk.

Research practices and statistical reporting quality in 250 economic psychology master's theses: a meta-research investigation

The replicability of research findings has recently been disputed across multiple scientific disciplines. In constructive reaction, the research culture in psychology is facing fundamental changes, but investigations of research practices that led to these improvements have almost exclusively focused on academic researchers. By contrast, we investigated the statistical reporting quality and selected indicators of questionable research practices (QRPs) in psychology students' master's theses. In a total of 250 theses, we investigated utilization and magnitude of standardized effect sizes, along with statistical power, the consistency and completeness of reported results, and possible indications of p-hacking and further testing. Effect sizes were reported for 36% of focal tests (median r = 0.19), and only a single formal power analysis was reported for sample size determination (median observed power 1 - β = 0.67). Statcheck revealed inconsistent p-values in 18% of cases, while 2% led to decision errors. There were no clear indications of p-hacking or further testing. We discuss our findings in the light of promoting open science standards in teaching and student supervision.

Use of P-values and the terms "significant", "non-significant" and "suggestive" in Abstracts in the European Annals of Otorhinolaryngology, Head & Neck Diseases

OBJECTIVES

To evaluate the use of P-values and the terms "significant", "non-significant" and "suggestive" in Abstracts in the European Annals of Otorhinolaryngology, Head & Neck Diseases.

MATERIALS AND METHODS

Consecutive articles accepted for publication during the period January 2016 - February 2019 were systematically reviewed. Main goal: descriptive analysis of the citation of P-values and use of the terms "significant", "non-significant" and "suggestive" in Abstracts. Secondary goal: analytic study of: (i) correlations between citation of a P-value and the main characteristics of authors and topics; and (ii) misuse of the terms "significant", "non-significant" and "suggestive" with respect to cited P-values, and correlations with author and topic characteristics.

RESULTS

In all, 91 articles were included. P-values and the terms "significant", "non-significant" and "suggestive" were cited in 35.1%, 41.7%, 10.9% and 0% of Abstracts, respectively. Citing a P-value did not significantly correlate with author or topic characteristics. There were discrepancies between the terms "non-significant", "significant" and "suggestive" and P-values given in the body of the article in 57.1% of Abstracts, with 30.7% overestimation and 25.2% underestimation of results, without significant correlation with author or topic characteristics.

CONCLUSION

Authors, editors and reviewers must pay particular attention to the spin resulting from inappropriate use of the terms "significant", "non-significant" and "suggestive" in Abstracts of articles submitted to the European Annals of Otorhinolaryngology, Head & Neck Diseases, to improve the rigor, quality and value of the scientific message delivered to the reader.