Reporting bias in imaging: higher accuracy is linked to faster publication

Assessment of Standards for Reporting of Diagnostic Accuracy (STARD) 2015 guideline adherence in medical imaging diagnostic accuracy studies published in 2023

BACKGROUND

The Standards for Reporting of Diagnostic Accuracy (STARD) 2015 guideline facilitates evaluation of key aspects of diagnostic test accuracy (DTA) studies and their findings, including the risk of bias and applicability of findings.

OBJECTIVE

To evaluate the completeness of reporting in medical imaging DTA research in a sample of studies published in 2023.

METHODS

A systematic search of Medline, Embase, and the Cochrane Library was performed to identify medical imaging DTA studies published between January and June 2023 that assessed one or more index imaging tests compared to a reference standard and reported test performance using relevant outcome measures. Completeness of reporting amongst the included studies was assessed using the 30-item STARD-2015 guideline. Multiple linear regression was subsequently performed to identify study characteristics associated with more complete reporting.

RESULTS

A total of 116 studies were included in our analysis with a median journal impact factor of 2.7 (range 0.9-19.7). The mean number of items reported was 17.5/30 (58%, SD 2.2). Items that were infrequently reported (reported in less than 33% of included studies) included items 9 ('whether participants formed a consecutive, random or convenience series'), 13.2 ('whether clinical information and index test results were available to the assessors of the reference standard'), 15 ('how indeterminate index test or reference standard results were handled'), 16 ('how missing data on the index test and reference standard were handled'), 22.1 ('time interval between the index test and the reference standard'), 22.2 ('clinical interventions between the index test and the reference standard') and 29 ('where the full study protocol can be accessed'). Adherence was significantly higher in journals with a higher than median journal impact factor (18.1/30 vs 16.8/30 items reported; P < .001).

CONCLUSION

The completeness of reporting in medical imaging DTA research is moderate and remains relatively static in absolute terms compared to a previous evaluation of studies published in 2016 performed by Hong and colleagues, acknowledging differences in sample study characteristics limit direct comparison. Potential strategies to support more complete reporting in medical imaging DTA research include mandating adherence to the STARD guideline in journal instructions to authors, requiring completed STARD checklists to be submitted alongside all DTA study manuscripts, and integrating quality of reporting assessment as a routine component of the peer review process.

Time to publication for results of clinical trials

BACKGROUND

Researchers conducting trials have a responsibility to publish the results of their work in a peer-reviewed journal, and failure to do so may introduce bias that affects the accuracy of available evidence. Moreover, failure to publish results constitutes research waste.

OBJECTIVES

To systematically review research reports that followed clinical trials from their inception and their investigated publication rates and time to publication. We also aimed to assess whether certain factors influenced publication and time to publication.

SEARCH METHODS

We identified studies by searching MEDLINE, Embase, Epistemonikos, the Cochrane Methodology Register (CMR) and the database of the US Agency for Healthcare Research and Quality (AHRQ), from inception to 23 August 2023. We also checked reference lists of relevant studies and contacted experts in the field for any additional studies.

SELECTION CRITERIA

Studies were eligible if they tracked the publication of a cohort of clinical trials and contained analyses of any aspect of the publication rate or time to publication of these trials.

DATA COLLECTION AND ANALYSIS

Two review authors performed data extraction independently. We extracted data on the prevalence of publication and the time from the trial start date or completion date to publication. We also extracted data from the clinical trials included in the research reports, including country of the study's first author, area of health care, means by which the publication status of these trials were sought and the risk of bias in the trials.

MAIN RESULTS

A total of 204 research reports tracking 165,135 trials met the inclusion criteria. Just over half (53%) of these trials were published in full. The median time to publication was approximately 4.8 years from the enrolment of the first trial participant and 2.1 years from the trial completion date. Trials with positive results (i.e. statistically significant results favouring the experimental arm) were more likely to be published than those with negative or null results (OR 2.69, 95% CI 2.02 to 3.60; 19 studies), and they were published in a shorter time (adjusted HR 1.92, 95% CI 1.51 to 2.45; 4 studies). On average, trials with positive results took 2 years to publish, whereas trials with negative or null results took 2.6 years. Large trials were more likely to be published than smaller ones (adjusted OR 1.92, 95% CI 1.33 to 2.77; 11 studies), and they were published in a shorter time (adjusted HR 1.41, 95% CI 1.18 to 1.68; 7 studies). Multicentre trials were more likely to be published than single-centre trials (adjusted OR 1.20, 95% CI 1.03 to 1.40; 2 studies). We found no difference between multicentre and single-centre trials in time to publication. Trials funded by non-industry sources (e.g.governments or universities) were more likely to be published than trials funded by industry (e.g. pharmaceutical companies or for-profit organisations) (adjusted OR 2.13, 95% CI 1.82 to 2.49; 14 studies); they were also published in a shorter time (adjusted HR 1.46, 95% CI 1.15 to 1.86; 7 studies).

AUTHORS' CONCLUSIONS

Our updated review shows that trial publication is poor, with only half of all trials that are conducted being published. Factors that may make publication more likely and lead to faster publication are positive results, large sample size and being funded by non-industry sources. Differences in publication rates result in publication bias and time-lag bias that may influence findings and therefore ultimately affect treatment decisions. Systematic review authors should consider the possibility of time-lag bias when conducting a systematic review, especially when updating their review.

FUNDING

This Cochrane review had no dedicated funding.

REGISTRATION

This review combines and updates two earlier Cochrane reviews. The two protocols and previous versions of the two updated reviews are available via 10.1002/14651858.MR000006 and 10.1002/14651858.MR000006.pub3 and 10.1002/14651858.MR000011 and 10.1002/14651858.MR000011.pub2.

Is There Evidence of P-Hacking in Imaging Research?

BACKGROUND

P-hacking, the tendency to run selective analyses until they become significant, is prevalent in many scientific disciplines.

PURPOSE

This study aims to assess if p-hacking exists in imaging research.

METHODS

Protocol, data, and code available here https://osf.io/xz9ku/?view_only=a9f7c2d841684cb7a3616f567db273fa. We searched imaging journals Ovid MEDLINE from 1972 to 2021. Text mining using Python script was used to collect metadata: journal, publication year, title, abstract, and P-values from abstracts. One P-value was randomly sampled per abstract. We assessed for evidence of p-hacking using a p-curve, by evaluating for a concentration of P-values just below .05. We conducted a one-tailed binomial test (α = .05 level of significance) to assess whether there were more P-values falling in the upper range (e.g., .045 < P < .05) than in the lower range (e.g., .04 < P < .045). To assess variation in results introduced by our random sampling of a single P-value per abstract, we repeated the random sampling process 1000 times and pooled results across the samples. Analysis was done (divided into 10-year periods) to determine if p-hacking practices evolved over time.

RESULTS

Our search of 136 journals identified 967,981 abstracts. Text mining identified 293,687 P-values, and a total of 4105 randomly sampled P-values were included in the p-hacking analysis. The number of journals and abstracts that were included in the analysis as a fraction and percentage of the total number was, respectively, 108/136 (80%) and 4105/967,981 (.4%). P-values did not concentrate just under .05; in fact, there were more P-values falling in the lower range (e.g., .04 < P < .045) than falling just below .05 (e.g., .045 < P < .05), indicating lack of evidence for p-hacking. Time trend analysis did not identify p-hacking in any of the five 10-year periods.

CONCLUSION

We did not identify evidence of p-hacking in abstracts published in over 100 imaging journals since 1972. These analyses cannot detect all forms of p-hacking, and other forms of bias may exist in imaging research such as publication bias and selective outcome reporting.

Orthopaedic Surgeons Should Consider Online and E-publication Resources for the Most Current Evidence-Based Medicine Following the COVID-19 Pandemic

Purpose

To compare the time to publication of accepted manuscripts and content in orthopaedic sports medicine journals during the first 2 years of the COVID-19 pandemic.

Methods

A convenience sample of articles published in January, May, and September during the years 2019-2021 was taken from Arthroscopy, American Journal of Sports Medicine (AJSM), and Knee Surgery, Sports Traumatology, Arthroscopy (KSSTA). The duration between the aspects of the article publication process was compared between journals and years.

Results

Overall, 826 journal articles were included. Arthroscopy demonstrated no significant differences in the time from manuscript submission to journal publication from 2019 to 2021, a significant decrease in time from acceptance to e-Pub (140 vs 74 vs 16 days; P < .001), but an increase from e-Pub to journal publication (23 vs 74 vs 130 days; P < .001). In AJSM, there was an overall increase in time from submission to journal publication significant between 2019 and 2021 (P = .05) and 2020 and 2021 (P = .001). KSSTA demonstrated the longest timelines in 2020. There was a trend toward a greater number of systematic reviews and meta-analyses.

Conclusion

Changes in various aspects of the time to publication and journal content occurred in orthopaedic sports medicine journals in the years surrounding the peak of the COVID-19 pandemic in 2020. Although it is not possible to know whether these delays are caused by journal or author-related factors, orthopaedic surgeons should be aware of the possible delay in time to publication and consider online and e-publication resources for the most current evidence-based medicine, while journals may take this information into account to consider ways of improving the publication process and when determining journal content.

Clinical Relevance

It is important to understand the impact the COVID-19 pandemic had on the publications which orthopaedic sports medicine surgeons rely on for clinical knowledge and the practice of evidence-based medicine.

Sonoelastography for Testicular Tumor Identification: A Systematic Review and Meta-Analysis of Diagnostic Test Accuracy

The objective of this review was to summarize the applications of sonoelastography in testicular tumor identification and inquire about their test performances. Two authors independently searched English journal articles and full conference papers from CINAHL, Embase, IEEE Xplore®, PubMed, Scopus, and Web of Science from inception and organized them into a PIRO (patient, index test, reference test, outcome) framework. Eleven studies (n = 11) were eligible for data synthesis, nine of which (n = 9) utilized strain elastography and two (n = 2) employed shear-wave elastography. Meta-analyses were performed on the distinction between neoplasm (tumor) and non-neoplasm (non-tumor) from four study arms and between malignancy and benignity from seven study arms. The pooled sensitivity of classifying malignancy and benignity was 86.0% (95%CI, 79.7% to 90.6%). There was substantial heterogeneity in the classification of neoplasm and non-neoplasm and in the specificity of classifying malignancy and benignity, which could not be addressed by the subgroup analysis of sonoelastography techniques. Heterogeneity might be associated with the high risk of bias and applicability concern, including a wide spectrum of testicular pathologies and verification bias in the reference tests. Key technical obstacles in the index test were manual compression in strain elastography, qualitative observation of non-standardized color codes, and locating the Regions of Interest (ROI), in addition to decisions in feature extractions. Future research may focus on multiparametric sonoelastography using deep learning models and ensemble learning. A decision model on the benefits-risks of surgical exploration (reference test) could also be developed to direct the test-and-treat strategy for testicular tumors.

Has the STARD statement improved the quality of reporting of diagnostic accuracy studies published in European Radiology?

OBJECTIVES

To investigate whether encouraging authors to follow the Standards for Reporting Diagnostic Accuracy (STARD) guidelines improves the quality of reporting of diagnostic accuracy studies.

METHODS

In mid-2017, European Radiology started encouraging its authors to follow the STARD guidelines. Our MEDLINE search identified 114 diagnostic accuracy studies published in European Radiology in 2015 and 2019. The quality of reporting was evaluated by two independent reviewers using the revised STARD statement. Item 11 was excluded because a meaningful decision about adherence was not possible. Student's t test for independent samples was used to analyze differences in the mean number of reported STARD items between studies published in 2015 and in 2019. In addition, we calculated differences related to the study design, data collection, and citation rate.

RESULTS

The mean total number of reported STARD items for all 114 diagnostic accuracy studies analyzed was 15.9 ± 2.6 (54.8%) of 29 items (range 9.5-22.5). The quality of reporting of diagnostic accuracy studies was significantly better in 2019 (mean ± standard deviation (SD), 16.3 ± 2.7) than in 2015 (mean ± SD, 15.1 ± 2.3; p < 0.02). No significant differences in the reported STARD items were identified in relation to study design (p = 0.13), data collection (p = 0.87), and citation rate (p = 0.09).

CONCLUSION

The quality of reporting of diagnostic accuracy studies according to the STARD statement was moderate with a slight improvement since European Radiology started to recommend its authors to follow the STARD guidelines.

KEY POINTS

• The quality of reporting of diagnostic accuracy studies was moderate with a mean total number of reported STARD items of 15.9 ± 2.6. • The adherence to STARD was significantly better in 2019 than in 2015 (16.3 ± 2.7 vs. 15.1 ± 2.3; p = 0.016). • No significant differences in the reported STARD items were identified in relation to study design (p = 0.13), data collection (p = 0.87), and citation rate (p = 0.09).

Preferential reporting of significant p-values in radiology journal abstracts

AIM

To assess the prevalence of publication bias in the radiology literature, data-mining techniques were used to extract p-values in abstracts published in key radiology journals over the past 20 years.

MATERIALS AND METHODS

A total of 34,699 abstracts published in Radiology, Investigative Radiology, European Radiology, American Journal of Roentgenology, and American Journal of Neuroradiology published between January 2000 and December 2019 were included in the analysis. Automated text mining using regular expressions was used to mine abstracts for p-values.

RESULTS

The text mining algorithm detected 43,489 p-values, the majority (82.4%) of which were reported as "significant", i.e., p<0.05. There has also been an increased propensity to report more p-values over time. The distribution of p-values showed a step change at the conventional significance threshold of 0.05. The odds ratio of a "significant" p-value being reported in the abstract compared to the full text was calculated to be 2.52 (95% confidence interval 1.78-3.58; p<0.001). Taken together, these results provide strong evidence for selective reporting of significant p-values in abstracts.

CONCLUSION

Statistically significant p-values are preferentially reported in radiology journal abstracts.

Tweeting Bias in Diagnostic Test Accuracy Research: Does Title or Conclusion Positivity Influence Dissemination?

PURPOSE

To examine if tweeting bias exists within imaging literature by determining if diagnostic test accuracy (DTA) studies with positive titles or conclusions are tweeted more than non-positive studies.

METHODS

DTA studies published between October 2011 to April 2016 were included. Positivity of titles and conclusions were assessed independently and in duplicate, with disagreements resolved by consensus. A negative binomial regression analysis controlling for confounding variables was performed to assess the relationship between title or conclusion positivity and tweets an article received in the 100 days post-publication.

RESULTS

354 DTA studies were included. Twenty-four (7%) titles and 300 (85%) conclusions were positive (or positive with qualifier); 1 (0.3%) title and 23 (7%) conclusions were negative; and 329 (93%) titles and 26 (7%) conclusions were neutral. Studies with positive, negative, and neutral titles received a mean of 0.38, 0.00, and 0.45 tweets per study; while those with positive, negative, and neutral conclusions received a mean of 0.44, 0.61, and 0.38 tweets per study. Regression coefficients were -0.05 (SE 0.46) for positive relative to non-positive titles, and -0.09 (SE 0.31) for positive relative to non-positive conclusions. The positivity of the title (P = 0.91) or conclusion (P = 0.76) was not significantly associated with the number of tweets an article received.

CONCLUSIONS

The positivity of the title or conclusion for DTA studies does not influence the amount of tweets it receives suggesting that tweet bias is not present among imaging diagnostic accuracy studies. Study protocol available at https://osf.io/hdk2m/.

Preferred reporting items for journal and conference abstracts of systematic reviews and meta-analyses of diagnostic test accuracy studies (PRISMA-DTA for Abstracts): checklist, explanation, and elaboration

For many users of the biomedical literature, abstracts may be the only source of information about a study. Hence, abstracts should allow readers to evaluate the objectives, key design features, and main results of the study. Several evaluations have shown deficiencies in the reporting of journal and conference abstracts across study designs and research fields, including systematic reviews of diagnostic test accuracy studies. Incomplete reporting compromises the value of research to key stakeholders. The authors of this article have developed a 12 item checklist of preferred reporting items for journal and conference abstracts of systematic reviews and meta-analyses of diagnostic test accuracy studies (PRISMA-DTA for Abstracts). This article presents the checklist, examples of complete reporting, and explanations for each item of PRISMA-DTA for Abstracts.

Searching practices and inclusion of unpublished studies in systematic reviews of diagnostic accuracy

Introduction

Many diagnostic accuracy studies are never reported in full in a peer‐reviewed journal. Searching for unpublished studies may avoid bias due to selective publication, enrich the power of systematic reviews, and thereby help to reduce research waste. We assessed searching practices among recent systematic reviews of diagnostic accuracy.

Methods

We extracted data from 100 non‐Cochrane systematic reviews of diagnostic accuracy indexed in MEDLINE and published between October 2017 and January 2018 and from all 100 Cochrane systematic reviews of diagnostic accuracy published by December 2018, irrespective of whether meta‐analysis had been performed.

Results

Non‐Cochrane and Cochrane reviews searched a median of 4 (IQR 3‐5) and 6 (IQR 5‐9) databases, respectively; most often MEDLINE/PubMed (n = 100 and n = 100) and EMBASE (n = 81 and n = 100). Additional efforts to identify studies beyond searching bibliographic databases were performed in 76 and 98 reviews, most often through screening reference lists (n = 71 and n = 96), review/guideline articles (n = 18 and n = 52), or citing articles (n = 3 and n = 42). Specific sources of unpublished studies were searched in 22 and 68 reviews, for example, conference proceedings (n = 4 and n = 18), databases only containing conference abstracts (n = 2 and n = 33), or trial registries (n = 12 and n = 39). At least one unpublished study was included in 17 and 23 reviews. Overall, 39 of 2082 studies (1.9%) included in non‐Cochrane reviews were unpublished, and 64 of 2780 studies (2.3%) in Cochrane reviews, most often conference abstracts (97/103).

Conclusion

Searching practices vary considerably across systematic reviews of diagnostic accuracy. Unpublished studies are a minimal fraction of the evidence included in recent reviews.

Publication bias may exist among prognostic accuracy studies of middle cerebral artery Doppler ultrasound

OBJECTIVES

The objective of this study was to assess if there is evidence of publication bias in prognostic accuracy studies of middle cerebral artery (MCA) or cerebroplacental ratio (CPR) for adverse perinatal outcome.

STUDY DESIGN AND SETTING

We queried PubMed, EMBASE, the Cochrane Library, and ClinicalTrials.gov and searched abstract books of five perinatal conferences (1989-2017). We included prognostic accuracy studies on MCA and/or CPR. Highest reported accuracy estimates, sample size, study design, and conclusion positivity were extracted and compared.

RESULTS

We included 127 full-text articles and 51 conference abstracts, 29 of which had not been reported as full-text article. In conference abstracts not reported in full, median negative predictive value was significantly lower compared to full-text articles (0.79 [interquartile range 0.67-0.97] vs. 0.95 [0.89-0.99]; P < 0.001). No significant difference was identified for positive predictive value (0.62 vs. 0.59; P = 0.827), sensitivity (0.67 vs. 0.71; P = 0.159), and specificity (0.86 vs. 0.86; P = 0.632). Study design differed significantly as well (P = 0.030), with fewer prospective studies in conference abstracts not reported in full compared to full-text articles (28% vs. 54%). We found no significant differences in sample size or conclusion positivity.

CONCLUSION

Possibly, a publication bias in previously published meta-analyses of MCA and CPR has led to overly generous estimates of prognostic performance.

Publication Bias: Association of Diagnostic Accuracy in Radiology Conference Abstracts with Full-Text Publication

Background Recent investigations have identified a faster time to publication for imaging studies with higher diagnostic test accuracy (DTA), but it is unknown whether such studies are more likely to be published. A higher probability of full-text publication for studies with higher DTA could have negative consequences on clinical decision making and patient care. Purpose To evaluate the proportion of imaging diagnostic accuracy studies presented as conference abstracts that reach full-text publication and to identify whether there is an association between diagnostic accuracy and full-text publication in peer-reviewed journals within 5 years after abstract submission. Materials and Methods Diagnostic accuracy research abstracts presented at the Radiological Society of North America (RSNA) Annual Meeting in 2011 and 2012 were evaluated between September 1, 2017, and January 11, 2018. Sensitivity and specificity from the abstracts were used to calculate the Youden index (sensitivity + specificity-1); additional abstract characteristics were extracted. To identify full-text publications within 5 years after abstract submission, PubMed and Google Scholar were searched, and authors were contacted. Logistic regression analysis was used to assess for associations between higher diagnostic accuracy and full-text publication. Results A total of 7970 abstracts were evaluated, and 405 were included. Of these, 288 (71%) reached full-text publication within 5 years after abstract submission. Logistic regression analysis accounting for several confounding variables failed to show an association between reported Youden index in the conference abstract and probability of full-text publication (odds ratio, 1.01; 95% confidence interval: 0.99, 1.02; P = .21). Conclusion More than a quarter of abstracts presented at the RSNA Annual Meeting do not reach full-text publication in peer-reviewed journals. The magnitude of reported diagnostic accuracy was not associated with full-text publication, which is consistent with results of diagnostic accuracy studies in other medical specialties. © RSNA, 2019 Online supplemental material is available for this article. See also the editorial by Fielding in this issue.

The Changing Landscape of Journalology in Medicine

In the early 1970s, when Seminars in Nuclear Medicine started publication, little was known about the quality of reporting in biomedical journals. Senior scholars were invited to become scientific editors of journals based on their research credibility and stature. Their knowledge of journalology (publication science) was not assessed. Similarly, while the use of peer review was gaining momentum, there was limited guidance on the tasks and expectations of peer reviewing. Almost 50 years later, the evidence base regarding the quality of reporting is vast. This paper highlights some of this evidence including that relevant to imaging and nuclear medicine research. In biomedical publications, there is a crisis in reproducibility; high prevalence rates of reporting biases, such as selective outcome reporting; spin; low registration rates of research protocols; and endemic poor reporting of research across biomedicine. These issues and some more immediate solutions are also discussed in the paper. The use of reporting guidelines has been shown to be associated with better reporting of clinical trials and other research articles. The use of audit and feedback tools is likely to provide an important gauge about the functions of biomedical journals. Finally, the push to better equip scientific editors and peer reviewers is taking a more concerted effort.