Good scientific practice in EEG and MEG research: Progress and perspectives

Good scientific practice (GSP) refers to both explicit and implicit rules, recommendations, and guidelines that help scientists to produce work that is of the highest quality at any given time, and to efficiently share that work with the community for further scrutiny or utilization. For experimental research using magneto- and electroencephalography (MEEG), GSP includes specific standards and guidelines for technical competence, which are periodically updated and adapted to new findings. However, GSP also needs to be regularly revisited in a broader light. At the LiveMEEG 2020 conference, a reflection on GSP was fostered that included explicitly documented guidelines and technical advances, but also emphasized intangible GSP: a general awareness of personal, organizational, and societal realities and how they can influence MEEG research. This article provides an extensive report on most of the LiveMEEG contributions and new literature, with the additional aim to synthesize ongoing cultural changes in GSP. It first covers GSP with respect to cognitive biases and logical fallacies, pre-registration as a tool to avoid those and other early pitfalls, and a number of resources to enable collaborative and reproducible research as a general approach to minimize misconceptions. Second, it covers GSP with respect to data acquisition, analysis, reporting, and sharing, including new tools and frameworks to support collaborative work. Finally, GSP is considered in light of ethical implications of MEEG research and the resulting responsibility that scientists have to engage with societal challenges. Considering among other things the benefits of peer review and open access at all stages, the need to coordinate larger international projects, the complexity of MEEG subject matter, and today's prioritization of fairness, privacy, and the environment, we find that current GSP tends to favor collective and cooperative work, for both scientific and for societal reasons.

Common Pitfalls and Recommendations for Grand Challenges in Medical Artificial Intelligence

With the impact of artificial intelligence (AI) algorithms on medical research on the rise, the importance of competitions for comparative validation of algorithms, so-called challenges, has been steadily increasing, to a point at which challenges can be considered major drivers of research, particularly in the biomedical image analysis domain. Given their importance, high quality, transparency, and interpretability of challenges is essential for good scientific practice and meaningful validation of AI algorithms, for instance towards clinical translation. This mini-review presents several issues related to the design, execution, and interpretation of challenges in the biomedical domain and provides best-practice recommendations. PATIENT SUMMARY: This paper presents recommendations on how to reliably compare the usefulness of new artificial intelligence methods for analysis of medical images.

BIAS: Transparent reporting of biomedical image analysis challenges

The number of biomedical image analysis challenges organized per year is steadily increasing. These international competitions have the purpose of benchmarking algorithms on common data sets, typically to identify the best method for a given problem. Recent research, however, revealed that common practice related to challenge reporting does not allow for adequate interpretation and reproducibility of results. To address the discrepancy between the impact of challenges and the quality (control), the Biomedical Image Analysis ChallengeS (BIAS) initiative developed a set of recommendations for the reporting of challenges. The BIAS statement aims to improve the transparency of the reporting of a biomedical image analysis challenge regardless of field of application, image modality or task category assessed. This article describes how the BIAS statement was developed and presents a checklist which authors of biomedical image analysis challenges are encouraged to include in their submission when giving a paper on a challenge into review. The purpose of the checklist is to standardize and facilitate the review process and raise interpretability and reproducibility of challenge results by making relevant information explicit.

Good Scientific Practice: Developing a Curriculum for Medical Students in Germany

German medical schools have not yet sufficiently introduced students to the field of good scientific practice (GSP). In order to prevent scientific misconduct and to foster scientific integrity, courses on GSP must be an integral part of the curriculum of medical students. Based on a review of the literature, teaching units and materials for two courses on GSP were developed and tested in a pilot course. The pilot course was accompanied by a pre-post evaluation that assessed students' knowledge and attitudes towards scientific integrity and scientific misconduct. A syllabus was designed that comprised the following six topics: theoretical foundations of GSP; scientific publishing; empirical data; scientific supervision and teamwork; clinical research; personal interests. The comparison pre versus post-intervention yielded statistically significant changes in regard to the participants' knowledge and attitude toward all forms of scientific misconduct treated in the course. As the majority of participants was not familiar with the fundamental regulations or guidelines of GSP, it seems crucial to train students in actively applying such norms to real-world conflicts. Students' unfamiliarity with the fundamentals of GSP can be linked to the fact that many students have already experienced forms of scientific misconduct. Thus, GSP syllabi should be closely adjusted to a student's realm of experience. All in all, courses on GSP can be seen as a potential means to increase the number of young scholars.

Result dissemination from clinical trials conducted at German university medical centers was delayed and incomplete

OBJECTIVES

Timely and comprehensive reporting of clinical trial results builds the backbone of evidence-based medicine and responsible research. The proportion of timely disseminated trial results can inform alternative national and international benchmarking of university medical centers (UMCs).

STUDY DESIGN AND SETTING

For all German UMCs, we tracked all registered trials completed between 2009 and 2013. The results and an interactive website benchmark German UMCs regarding their performance in result dissemination.

RESULTS

We identified and tracked 2,132 clinical trials. For 1,509 trials, one of the German UMCs took the academic lead. Of these 1,509 "lead trials," 39% published their results (mostly via journal publications) in a timely manner (<24 months after completion). More than 6 years after study completion, 26% of all eligible lead trials still had not disseminated results.

CONCLUSION

Despite substantial attention from many stakeholders to the topic, there is still a strong delay or even absence of result dissemination for many trials. German UMCs have several opportunities to improve this situation. Further research should evaluate whether and how a transparent benchmarking of UMC performance in result dissemination helps to increase value and reduce waste in medical research.

[European ethics committees in transition: challenges of new requirements]

Guided by the example of ethics committees in medicine, the ethical review system of research projects in Europe is increasingly expanding to other disciplines such as psychology, the social sciences, and engineering, especially when human subjects are directly involved or personal data are collected. However, ethical reviews are not uniformly regulated in most European countries.With the new EU Regulation No. 536/2014, which is expected to come into force in 2020, it is intended to standardize the work of medical research ethics committees for clinical trials. Clear deadlines and the use of a central electronic EU portal are also given in order to accelerate the approval process for economic reasons. Member states are left free in terms of the scope of the ethical review, but this would run counter to unification and could lead some to choose a test model that excludes the evaluation of the methodology and risks of a study and thus contributes less to the protection of the participants.The European Research Ethics Committees have been working together since 2005 within the European Network of Research Ethics Committees (EUREC). For a discussion on the impact and implementation of the new EU Regulation and on the development of the EU portal, EUREC has provided a forum for a qualified debate.In this period of change, it is not clear yet which direction the ethics committees will develop. Professional management can have a positive impact on the quality of the ethics review. However, the marginalization inherent in the EU Regulation is counteracting the increasing evaluation of nonmedical research projects and adequate protection of subjects.

[Evaluation of methodological quality in published RCTs on cataract surgery : Pilot study on the degree of adherence to CONSORT statement requirements and their qualitative validity]

BACKGROUND

The CONSORT statement can be considered as a guideline to ensure transparency in the reporting of randomized clinical trials (RCT), in addition to specific author instructions and requirements of journals. It provides a total of 25 criteria and 12 additional subcriteria on methodological and regulatorical determinants of clinical trials. The availability of the CONSORT recommendations, however, does not necessarily imply adherence to their obligations and correct realisation of the latter from a methodological perspective, so that even in ophthalmology a lack of transparency in trial reporting cannot be excluded.

OBJECTIVE

The question was whether a consistent consideration of the CONSORT checklist criteria by authors actually implied transparent reporting of underlying study results.

MATERIALS AND METHODS

This pilot study was based on a random sample of six published RCTs on cataract surgery extracted from an existing trial publication register. Compliance with each of the 25 CONSORT criteria and its 12 subcriteria and the content accuracy of the latter were independently assessed by two parallel raters for the six trial publications.

RESULTS

The median compliance with the 37 CONSORT criteria and subcriteria was 62% [min-max 48-81%]; the median fraction of their correct implementation was 47% [min-max 34-69%].

CONCLUSIONS

Promotion of transparent reporting by means of the CONSORT statement appears to be problematic in implementation. There is a discrepancy between information as required by CONSORT and the content accuracy of its actual presentation. Thus, in particular, reviewers of clinical trial publications should not only check for the presence of data to be provided according to CONSORT, but also verify the meaningfulness in the respective context, at least on a random basis.