Reporting of imaging diagnostic accuracy studies with focus on MRI subgroup: Adherence to STARD 2015

Cross-Sectional Evaluation of Open Science Practices at Imaging Journals: A Meta-Research Study

Objective: To evaluate open science policies of imaging journals, and compliance to these policies in published articles. Methods: From imaging journals listed we extracted open science policy details: protocol registration, reporting guidelines, funding, ethics and conflicts of interest (COI), data sharing, and open access publishing. The 10 most recently published studies from each journal were assessed to determine adherence to these policies. We calculated the proportion of open science policies into an Open Science Score (OSS) for all journals and articles. We evaluated relationships between OSS and journal/article level variables. Results: 82 journals/820 articles were included. The OSS of journals and articles was 58.3% and 31.8%, respectively. Of the journals, 65.9% had registration and 78.1% had reporting guideline policies. 79.3% of journals were members of COPE, 81.7% had plagiarism policies, 100% required disclosure of funding, and 97.6% required disclosure of COI and ethics approval. 81.7% had data sharing policies and 15.9% were fully open access. 7.8% of articles had a registered protocol, 8.4% followed a reporting guideline, 77.4% disclosed funding, 88.7% disclosed COI, and 85.6% reported ethics approval. 12.3% of articles shared their data. 51% of articles were available through open access or as a preprint. OSS was higher for journal with DOAJ membership (80% vs 54.2%; P < .0001). Impact factor was not correlated with journal OSS. Knowledge synthesis articles has a higher OSS scores (44.5%) than prospective/retrospective studies (32.6%, 30.0%, P < .0001). Conclusion: Imaging journals endorsed just over half of open science practices considered; however, the application of these practices at the article level was lower.

Publications on the diagnostic accuracy of dermatopathology tests: A cross-sectional quality analysis

BACKGROUND

Ancillary diagnostic tests are frequent in dermatopathology practice. Publications on their accuracy influence their utilization. The transparency and completeness of these publications are unknown.

METHODS

We performed a cross-sectional study on diagnostic accuracy studies in dermatopathology published between 2020 and 2022 for compliance with Standards for Reporting of Diagnostic Accuracy Studies (STARD) and the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2).

RESULTS

14.67 ± 3.02 STARD items were reported in 62 publications (range, 9.5-23.5 out of the recommended total of 30). More items were reported in high-impact factor journals (16.01 vs. 13.32, p = 0.0002) and journals that endorsed STARD in their author instructions (17.22 vs. 14.11, p = 0.0039). Less than 10% of publications reported quantifiable hypotheses, sample size calculations, flow diagrams, or study registrations. The risk of bias by our analysis of QUADAS-2 criteria was high or uncertain for index test interpretation (36/62, 58%) and patient selection (44/62, 71%).

CONCLUSIONS

Publications on dermatopathology tests are exploratory studies without prespecified hypotheses or study designs. They do not meet the criteria for transparent reporting. We suggest that medical journal leadership should consider updating their instructions with more explicit guidance on recommended manuscript elements.

Best Practice for MRI Diagnostic Accuracy Research With Lessons and Examples from the LI-RADS Individual Participant Data Group

Medical imaging diagnostic test accuracy research is strengthened by adhering to best practices for study design, data collection, data documentation, and study reporting. In this review, key elements of such research are discussed, and specific recommendations provided for optimizing diagnostic accuracy study execution to improve uniformity, minimize common sources of bias and avoid potential pitfalls. Examples are provided regarding study methodology and data collection practices based on insights gained by the liver imaging reporting and data system (LI-RADS) individual participant data group, who have evaluated raw data from numerous MRI diagnostic accuracy studies for risk of bias and data integrity. The goal of this review is to outline strategies for investigators to improve research practices, and to help reviewers and readers better contextualize a study's findings while understanding its limitations. LEVEL OF EVIDENCE: 5 TECHNICAL EFFICACY: Stage 3.

Applications of artificial intelligence in magnetic resonance imaging of primary pediatric cancers: a scoping review and CLAIM score assessment

PURPOSES

To review the uses of AI for magnetic resonance (MR) imaging assessment of primary pediatric cancer and identify common literature topics and knowledge gaps. To assess the adherence of the existing literature to the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) guidelines.

MATERIALS AND METHODS

A scoping literature search using MEDLINE, EMBASE and Cochrane databases was performed, including studies of > 10 subjects with a mean age of < 21 years. Relevant data were summarized into three categories based on AI application: detection, characterization, treatment and monitoring. Readers independently scored each study using CLAIM guidelines, and inter-rater reproducibility was assessed using intraclass correlation coefficients.

RESULTS

Twenty-one studies were included. The most common AI application for pediatric cancer MR imaging was pediatric tumor diagnosis and detection (13/21 [62%] studies). The most commonly studied tumor was posterior fossa tumors (14 [67%] studies). Knowledge gaps included a lack of research in AI-driven tumor staging (0/21 [0%] studies), imaging genomics (1/21 [5%] studies), and tumor segmentation (2/21 [10%] studies). Adherence to CLAIM guidelines was moderate in primary studies, with an average (range) of 55% (34%-73%) CLAIM items reported. Adherence has improved over time based on publication year.

CONCLUSION

The literature surrounding AI applications of MR imaging in pediatric cancers is limited. The existing literature shows moderate adherence to CLAIM guidelines, suggesting that better adherence is required for future studies.

Systematic review of adherence to the standards for reporting of diagnostic accuracy studies (STARD) 2015 reporting guideline in cerebral aneurysm imaging diagnostic accuracy studies

BACKGROUND

Diagnostic neuroimaging plays an essential role in guiding clinical decision-making in the management of patients with cerebral aneurysms. Imaging technologies for investigating cerebral aneurysms constantly evolve, and clinicians rely on the published literature to remain up to date. Reporting guidelines have been developed to standardise and strengthen the reporting of clinical evidence. Therefore, it is essential that radiological diagnostic accuracy studies adhere to such guidelines to ensure completeness of reporting. Incomplete reporting hampers the reader's ability to detect bias, determine generalisability of study results or replicate investigation parameters, detracting from the credibility and reliability of studies.

OBJECTIVE

The purpose of this systematic review was to evaluate adherence to the Standards for Reporting of Diagnostic Accuracy Studies (STARD) 2015 reporting guideline amongst imaging diagnostic accuracy studies for cerebral aneurysms.

METHODS

A systematic search for cerebral aneurysm imaging diagnostic accuracy studies was conducted. Journals were cross examined against the STARD 2015 checklist and their compliance with item numbers was recorded.

RESULTS

The search yielded 66 articles. The mean number of STARD items reported was 24.2 ± 2.7 (71.2% ± 7.9%), with a range of 19 to 30 out of a maximum number of 34 items.

CONCLUSION

Taken together, these results indicate that adherence to the STARD 2015 guideline in cerebral aneurysm imaging diagnostic accuracy studies was moderate. Measures to improve compliance include mandating STARD 2015 adherence in instructions to authors issued by journals.

Prevalence of malignant or possibly malignant renal masses among homogeneous low-attenuation masses that are too small to characterize at computed tomography

BACKGROUND

Homogeneous low-attenuation renal masses that are too small to characterize (tstc) are considered clinically insignificant; however, based primarily on expert opinion.

OBJECTIVE

To determine the prevalence of malignant or possibly malignant masses among homogeneous low-attenuation renal masses that are tstc.

MATERIALS AND METHODS

This retrospective cross-sectional study evaluated 75 patients with 104 tstc who underwent renal CT and MRI between Jan 2016 and Jul 2022. Low-attenuation renal masses measuring < 1 cm in size were identified and, independently evaluated by two blinded radiologists measuring attenuation (Hounsfield Units, HU) at non-contrast enhanced CT (NECT) and nephrographic phase contrast-enhanced (CE)-CT when possible. Reference standard for benign cyst was MRI and for other renal masses was pathology or MRI showing enhancement.

RESULTS

Average tstc size was 6 ± 2 (range 2-10) mm. Considering only incidental tstc (CT performed for another reason), 100% (98/98, 95%CI 96-100%) tstc were benign. Overall, considering both incidental and tstc referred for further characterization, there were 94% (98/104; 95% Confidence Intervals [CIs] 88-98%) benign cysts and 6% (6/104; 95%CI 2-12%) other masses (1 Bosniak 2F cystic mass, 2 probable renal cell carcinoma (RCC), three metastases). Pseudoenhancement, attenuation change > 10 HU or > 20 HU, was present in 29% (15/59) and 12% (7/59) benign cysts. All six other masses enhanced by > 20 HU. CECT threshold of ≤ 30 HU correctly classified 62% of benign cysts (61/98). All six other masses measured > 30 HU at CECT.

CONCLUSION

The prevalence of malignant or possibly malignant renal masses among homogeneous low-attenuation too small to characterize masses among incidental tstc masses is near zero. Attenuation measurements misclassify a substantial proportion of these cysts, likely due to their small size.

Is There Evidence of P-Hacking in Imaging Research?

BACKGROUND

P-hacking, the tendency to run selective analyses until they become significant, is prevalent in many scientific disciplines.

PURPOSE

This study aims to assess if p-hacking exists in imaging research.

METHODS

Protocol, data, and code available here https://osf.io/xz9ku/?view_only=a9f7c2d841684cb7a3616f567db273fa. We searched imaging journals Ovid MEDLINE from 1972 to 2021. Text mining using Python script was used to collect metadata: journal, publication year, title, abstract, and P-values from abstracts. One P-value was randomly sampled per abstract. We assessed for evidence of p-hacking using a p-curve, by evaluating for a concentration of P-values just below .05. We conducted a one-tailed binomial test (α = .05 level of significance) to assess whether there were more P-values falling in the upper range (e.g., .045 < P < .05) than in the lower range (e.g., .04 < P < .045). To assess variation in results introduced by our random sampling of a single P-value per abstract, we repeated the random sampling process 1000 times and pooled results across the samples. Analysis was done (divided into 10-year periods) to determine if p-hacking practices evolved over time.

RESULTS

Our search of 136 journals identified 967,981 abstracts. Text mining identified 293,687 P-values, and a total of 4105 randomly sampled P-values were included in the p-hacking analysis. The number of journals and abstracts that were included in the analysis as a fraction and percentage of the total number was, respectively, 108/136 (80%) and 4105/967,981 (.4%). P-values did not concentrate just under .05; in fact, there were more P-values falling in the lower range (e.g., .04 < P < .045) than falling just below .05 (e.g., .045 < P < .05), indicating lack of evidence for p-hacking. Time trend analysis did not identify p-hacking in any of the five 10-year periods.

CONCLUSION

We did not identify evidence of p-hacking in abstracts published in over 100 imaging journals since 1972. These analyses cannot detect all forms of p-hacking, and other forms of bias may exist in imaging research such as publication bias and selective outcome reporting.

Has the quality of reporting improved since it became mandatory to use the Standards for Reporting Diagnostic Accuracy?

OBJECTIVES

To investigate whether making the Standards for Reporting Diagnostic Accuracy (STARD) mandatory by the leading journal 'Radiology' in 2016 improved the quality of reporting of diagnostic accuracy studies.

METHODS

A validated search term was used to identify diagnostic accuracy studies published in Radiology in 2015 and 2019. STARD adherence was assessed by two independent reviewers. Each item was scored as yes (1 point) if adequately reported or as no (0 points) if not. The total STARD score per article was calculated. Wilcoxon-Mann-Whitney tests were used to evaluate differences of the total STARD scores between 2015 and 2019. In addition, the total STARD score was compared between studies stratified by study design, citation rate, and data collection.

RESULTS

The median number of reported STARD items for the total of 66 diagnostic accuracy studies from 2015 to 2019 was 18.5 (interquartile range [IQR] 17.5-20.0) of 29. Adherence to the STARD checklist significantly improved the STARD score from a median of 18.0 (IQR 15.5-19.5) in 2015 to a median of 19.5 (IQR 18.5-21.5) in 2019 (p < 0.001). No significant differences were found between studies stratified by mode of data collection (prospective vs. retrospective studies, p = 0.68), study design (cohort vs. case-control studies, p = 0.81), and citation rate (two groups divided by median split [< 0.56 citations/month vs. ≥ 0.56 citations/month], p = 0.54).

CONCLUSIONS

Making use of the STARD checklist mandatory significantly increased the adherence with reporting standards for diagnostic accuracy studies and should be considered by editors and publishers for widespread implementation.

CRITICAL RELEVANCE STATEMENT

Editors may consider making reporting guidelines mandatory to improve the scientific quality.

How to Critically Appraise and Interpret Systematic Reviews and Meta-Analyses of Diagnostic Accuracy: A User Guide

Systematic reviews of diagnostic accuracy studies can provide the best available evidence to inform decisions regarding the use of a diagnostic test. In this guide, the authors provide a practical approach for clinicians to appraise diagnostic accuracy systematic reviews and apply their results to patient care. The first step is to identify an appropriate systematic review with a research question matching the clinical scenario. The user should evaluate the rigor of the review methods to evaluate its credibility (Did the review use clearly defined eligibility criteria, a comprehensive search strategy, structured data collection, risk of bias and applicability appraisal, and appropriate meta-analysis methods?). If the review is credible, the next step is to decide whether the diagnostic performance is adequate for clinical use (Do sensitivity and specificity estimates exceed the threshold that makes them useful in clinical practice? Are these estimates sufficiently precise? Is variability in the estimates of diagnostic accuracy across studies explained?). Diagnostic accuracy systematic reviews that are judged to be credible and provide diagnostic accuracy estimates with sufficient certainty and relevance are the most useful to inform patient care. This review discusses comparative, noncomparative, and emerging approaches to systematic reviews of diagnostic accuracy using a clinical scenario and examples based on recent publications.

Overinterpretation of findings in diagnostic accuracy studies of infectious diseases

OBJECTIVES

To assess the prevalence of overly positive interpretation - also called 'spin'- of results in diagnostic accuracy studies of infectious diseases and to identify suggestions for improvement.

METHODS

A PubMed search was performed to identify diagnostic accuracy studies of infectious diseases published between January and March 2019. Each article was assessed by two authors independently to identify study characteristics and forms of actual and potential overinterpretation. 'Actual overinterpretation' was defined as conclusions that were not based on the study aims or conclusions that were more favourable than was justified by the study findings. There are other practices that may result in the overinterpretation of study findings and these have been described as 'potential overinterpretation'.

RESULTS

The final analysis included 120 studies. Favourable or promising recommendations were made in the main text of 101 (84%) of the included studies. Evidence of actual overinterpretation (spin) was found in 30 (25%) articles, with 22 studies reporting a conclusion that did not match the study aims and 56 studies with a more positive conclusion in the abstract than in the main text. All analysed studies exhibited at least one form of potential overinterpretation, with was most commonly a lack of sample size calculation (n = 109, 91%) and not reporting a null hypothesis (n = 115, 96%).

CONCLUSIONS

Evidence of overinterpretation of results was found in a third of the included studies. We propose possible interventions to prevent overly positive interpretation of results in diagnostic accuracy studies.

Has the STARD statement improved the quality of reporting of diagnostic accuracy studies published in European Radiology?

OBJECTIVES

To investigate whether encouraging authors to follow the Standards for Reporting Diagnostic Accuracy (STARD) guidelines improves the quality of reporting of diagnostic accuracy studies.

METHODS

In mid-2017, European Radiology started encouraging its authors to follow the STARD guidelines. Our MEDLINE search identified 114 diagnostic accuracy studies published in European Radiology in 2015 and 2019. The quality of reporting was evaluated by two independent reviewers using the revised STARD statement. Item 11 was excluded because a meaningful decision about adherence was not possible. Student's t test for independent samples was used to analyze differences in the mean number of reported STARD items between studies published in 2015 and in 2019. In addition, we calculated differences related to the study design, data collection, and citation rate.

RESULTS

The mean total number of reported STARD items for all 114 diagnostic accuracy studies analyzed was 15.9 ± 2.6 (54.8%) of 29 items (range 9.5-22.5). The quality of reporting of diagnostic accuracy studies was significantly better in 2019 (mean ± standard deviation (SD), 16.3 ± 2.7) than in 2015 (mean ± SD, 15.1 ± 2.3; p < 0.02). No significant differences in the reported STARD items were identified in relation to study design (p = 0.13), data collection (p = 0.87), and citation rate (p = 0.09).

CONCLUSION

The quality of reporting of diagnostic accuracy studies according to the STARD statement was moderate with a slight improvement since European Radiology started to recommend its authors to follow the STARD guidelines.

KEY POINTS

• The quality of reporting of diagnostic accuracy studies was moderate with a mean total number of reported STARD items of 15.9 ± 2.6. • The adherence to STARD was significantly better in 2019 than in 2015 (16.3 ± 2.7 vs. 15.1 ± 2.3; p = 0.016). • No significant differences in the reported STARD items were identified in relation to study design (p = 0.13), data collection (p = 0.87), and citation rate (p = 0.09).

Systematic review identifies deficiencies in reporting of diagnostic test accuracy among clinical decision support systems

OBJECTIVES

This systematic review assesses the reporting quality and risk of bias in studies evaluating the diagnostic test accuracy (DTA) of clinical decision support systems (CDSS).

STUDY DESIGN AND SETTING

The Cochrane Library, PubMed/MEDLINE, Scopus, and Web of Science were searched for studies, published between January 1, 2016 and May 31, 2021, evaluating the DTA of CDSS for human patients. Articles using a patient's self-diagnosis, assessing disease severity, focusing on treatment/follow-up, or comparing pre-post CDSS implementation periods were excluded. All eligible studies were assessed for reporting quality using STARD 2015 and for risk of bias using QUADAS-2. Item ratings were presented using heat maps. This study was reported as per PRISMA-DTA.

RESULTS

In total, 158 of 2,820 screened articles were included in the analysis. The studies were heterogeneous in terms of study characteristics, reporting quality, risk of biases, and applicability concerns with few highly rated studies. Mostly the overall quality was deficient for items addressing the domains 'methodology,' 'results,' and 'other information'.

CONCLUSION

Our analysis revealed shortcomings in critical domains of reporting quality and risk of bias, indicating the need for additional guidance and training in an interdisciplinary scientific field with mixed biostatistical expertise.

Portable stroke detection devices: a systematic scoping review of prehospital applications

BACKGROUND

The worldwide burden of stroke remains high, with increasing time-to-treatment correlated with worse outcomes. Yet stroke subtype determination, most importantly between stroke/non-stroke and ischemic/hemorrhagic stroke, is not confirmed until hospital CT diagnosis, resulting in suboptimal prehospital triage and delayed treatment. In this study, we survey portable, non-invasive diagnostic technologies that could streamline triage by making this initial determination of stroke type, thereby reducing time-to-treatment.

METHODS

Following PRISMA guidelines, we performed a scoping review of portable stroke diagnostic devices. The search was executed in PubMed and Scopus, and all studies testing technology for the detection of stroke or intracranial hemorrhage were eligible for inclusion. Extracted data included type of technology, location, feasibility, time to results, and diagnostic accuracy.

RESULTS

After a screening of 296 studies, 16 papers were selected for inclusion. Studied devices utilized various types of diagnostic technology, including near-infrared spectroscopy (6), ultrasound (4), electroencephalography (4), microwave technology (1), and volumetric impedance spectroscopy (1). Three devices were tested prior to hospital arrival, 6 were tested in the emergency department, and 7 were tested in unspecified hospital settings. Median measurement time was 3 minutes (IQR: 3 minutes to 5.6 minutes). Several technologies showed high diagnostic accuracy in severe stroke and intracranial hematoma detection.

CONCLUSION

Numerous emerging portable technologies have been reported to detect and stratify stroke to potentially improve prehospital triage. However, the majority of these current technologies are still in development and utilize a variety of accuracy metrics, making inter-technology comparisons difficult. Standardizing evaluation of diagnostic accuracy may be helpful in further optimizing portable stroke detection technology for clinical use.

Thoracic imaging tests for the diagnosis of COVID-19

BACKGROUND

Our March 2021 edition of this review showed thoracic imaging computed tomography (CT) to be sensitive and moderately specific in diagnosing COVID-19 pneumonia. This new edition is an update of the review.

OBJECTIVES

Our objectives were to evaluate the diagnostic accuracy of thoracic imaging in people with suspected COVID-19; assess the rate of positive imaging in people who had an initial reverse transcriptase polymerase chain reaction (RT-PCR) negative result and a positive RT-PCR result on follow-up; and evaluate the accuracy of thoracic imaging for screening COVID-19 in asymptomatic individuals. The secondary objective was to assess threshold effects of index test positivity on accuracy.

SEARCH METHODS

We searched the COVID-19 Living Evidence Database from the University of Bern, the Cochrane COVID-19 Study Register, The Stephen B. Thacker CDC Library, and repositories of COVID-19 publications through to 17 February 2021. We did not apply any language restrictions.

SELECTION CRITERIA

We included diagnostic accuracy studies of all designs, except for case-control, that recruited participants of any age group suspected to have COVID-19. Studies had to assess chest CT, chest X-ray, or ultrasound of the lungs for the diagnosis of COVID-19, use a reference standard that included RT-PCR, and report estimates of test accuracy or provide data from which we could compute estimates. We excluded studies that used imaging as part of the reference standard and studies that excluded participants with normal index test results.

DATA COLLECTION AND ANALYSIS

The review authors independently and in duplicate screened articles, extracted data and assessed risk of bias and applicability concerns using QUADAS-2. We presented sensitivity and specificity per study on paired forest plots, and summarized pooled estimates in tables. We used a bivariate meta-analysis model where appropriate.

MAIN RESULTS

We included 98 studies in this review. Of these, 94 were included for evaluating the diagnostic accuracy of thoracic imaging in the evaluation of people with suspected COVID-19. Eight studies were included for assessing the rate of positive imaging in individuals with initial RT-PCR negative results and positive RT-PCR results on follow-up, and 10 studies were included for evaluating the accuracy of thoracic imaging for imagining asymptomatic individuals. For all 98 included studies, risk of bias was high or unclear in 52 (53%) studies with respect to participant selection, in 64 (65%) studies with respect to reference standard, in 46 (47%) studies with respect to index test, and in 48 (49%) studies with respect to flow and timing. Concerns about the applicability of the evidence to: participants were high or unclear in eight (8%) studies; index test were high or unclear in seven (7%) studies; and reference standard were high or unclear in seven (7%) studies. Imaging in people with suspected COVID-19 We included 94 studies. Eighty-seven studies evaluated one imaging modality, and seven studies evaluated two imaging modalities. All studies used RT-PCR alone or in combination with other criteria (for example, clinical signs and symptoms, positive contacts) as the reference standard for the diagnosis of COVID-19. For chest CT (69 studies, 28285 participants, 14,342 (51%) cases), sensitivities ranged from 45% to 100%, and specificities from 10% to 99%. The pooled sensitivity of chest CT was 86.9% (95% confidence interval (CI) 83.6 to 89.6), and pooled specificity was 78.3% (95% CI 73.7 to 82.3). Definition for index test positivity was a source of heterogeneity for sensitivity, but not specificity. Reference standard was not a source of heterogeneity. For chest X-ray (17 studies, 8529 participants, 5303 (62%) cases), the sensitivity ranged from 44% to 94% and specificity from 24 to 93%. The pooled sensitivity of chest X-ray was 73.1% (95% CI 64. to -80.5), and pooled specificity was 73.3% (95% CI 61.9 to 82.2). Definition for index test positivity was not found to be a source of heterogeneity. Definition for index test positivity and reference standard were not found to be sources of heterogeneity. For ultrasound of the lungs (15 studies, 2410 participants, 1158 (48%) cases), the sensitivity ranged from 73% to 94% and the specificity ranged from 21% to 98%. The pooled sensitivity of ultrasound was 88.9% (95% CI 84.9 to 92.0), and the pooled specificity was 72.2% (95% CI 58.8 to 82.5). Definition for index test positivity and reference standard were not found to be sources of heterogeneity. Indirect comparisons of modalities evaluated across all 94 studies indicated that chest CT and ultrasound gave higher sensitivity estimates than X-ray (P = 0.0003 and P = 0.001, respectively). Chest CT and ultrasound gave similar sensitivities (P=0.42). All modalities had similar specificities (CT versus X-ray P = 0.36; CT versus ultrasound P = 0.32; X-ray versus ultrasound P = 0.89). Imaging in PCR-negative people who subsequently became positive For rate of positive imaging in individuals with initial RT-PCR negative results, we included 8 studies (7 CT, 1 ultrasound) with a total of 198 participants suspected of having COVID-19, all of whom had a final diagnosis of COVID-19. Most studies (7/8) evaluated CT. Of 177 participants with initially negative RT-PCR who had positive RT-PCR results on follow-up testing, 75.8% (95% CI 45.3 to 92.2) had positive CT findings. Imaging in asymptomatic PCR-positive people For imaging asymptomatic individuals, we included 10 studies (7 CT, 1 X-ray, 2 ultrasound) with a total of 3548 asymptomatic participants, of whom 364 (10%) had a final diagnosis of COVID-19. For chest CT (7 studies, 3134 participants, 315 (10%) cases), the pooled sensitivity was 55.7% (95% CI 35.4 to 74.3) and the pooled specificity was 91.1% (95% CI 82.6 to 95.7).

AUTHORS' CONCLUSIONS

Chest CT and ultrasound of the lungs are sensitive and moderately specific in diagnosing COVID-19. Chest X-ray is moderately sensitive and moderately specific in diagnosing COVID-19. Thus, chest CT and ultrasound may have more utility for ruling out COVID-19 than for differentiating SARS-CoV-2 infection from other causes of respiratory illness. The uncertainty resulting from high or unclear risk of bias and the heterogeneity of included studies limit our ability to confidently draw conclusions based on our results.

Reporting completeness in abstracts of systematic reviews of diagnostic test accuracy studies in cardiovascular diseases is suboptimal

OBJECTIVE

Journal abstracts are crucial for the identification and initial assessment of content of studies. We evaluated whether authors in the field of cardiovascular diseases (CVDs) reported Diagnostic Test Accuracy Systematic Reviews (DTA SRs) abstracts adequately, as defined by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)-DTA guidelines.

METHODS

SRs of DTA studies in CVDs published in general and specialized medical journals were identified in a MEDLINE search between 2010-2020. Adherence to 12 PRISMA-DTA for abstracts items was assessed independently by two reviewers and compared by journal's type. Moreover, the association of reporting completeness with different characteristics was investigated.

RESULTS

We included 72 abstracts. Studies published in general medical journals had higher mean reporting score than those in specialized journals (6.2 vs 5.3 out of 12 items; mean difference: 0.88; 95% confidence interval: 0.21, 1.55). PRISMA-DTA adherence was higher in journals that adopted this guideline and in articles with structured abstracts. However, number of participants analysed, funding and registration were the least-reported items in the identified abstracts.

CONCLUSION

The reporting of abstracts of DTA reports in CVDs is suboptimal according to PRISMA-DTA guidelines. Abstract reporting could be improved with the use of higher word count limits and the adoption of PRISMA-DTA guidelines especially in specialized journals.

Association of Accuracy, Conclusions, and Reporting Completeness With Acceptance by Radiology Conferences and Journals

BACKGROUND

Preferential publication of studies with positive findings can lead to overestimation of diagnostic test accuracy (i.e. publication bias). Understanding the contribution of the editorial process to publication bias could inform interventions to optimize the evidence guiding clinical decisions.

PURPOSE/HYPOTHESIS

To evaluate whether accuracy estimates, abstract conclusion positivity, and completeness of abstract reporting are associated with acceptance to radiology conferences and journals.

STUDY TYPE

Meta-research.

POPULATION

Abstracts submitted to radiology conferences (European Society of Gastrointestinal and Abdominal Radiology (ESGAR) and International Society for Magnetic Resonance in Medicine (ISMRM)) from 2008 to 2018 and manuscripts submitted to radiology journals (Radiology, Journal of Magnetic Resonance Imaging [JMRI]) from 2017 to 2018. Primary clinical studies evaluating sensitivity and specificity of a diagnostic imaging test in humans with available editorial decisions were included.

ASSESSMENT

Primary variables (Youden's index [YI > 0.8 vs. <0.8], abstract conclusion positivity [positive vs. neutral/negative], number of reported items on the Standards for Reporting of Diagnostic Accuracy Studies [STARD] for Abstract guideline) and confounding variables (prospective vs. retrospective/unreported, sample size, study duration, interobserver agreement assessment, subspecialty, modality) were extracted.

STATISTICAL TESTS

Multivariable logistic regression to obtain adjusted odds ratio (OR) as a measure of the association between the primary variables and acceptance by radiology conferences and journals; 95% confidence intervals (CIs) and P-values were obtained; the threshold for statistical significance was P < 0.05.

RESULTS

A total of 1000 conference abstracts (500 ESGAR and 500 ISMRM) and 1000 journal manuscripts (505 Radiology and 495 JMRI) were included. Conference abstract acceptance was not significantly associated with YI (adjusted OR = 0.97 for YI > 0.8; CI = 0.70-1.35), conclusion positivity (OR = 1.21 for positive conclusions; CI = 0.75-1.90) or STARD for Abstracts adherence (OR = 0.96 per unit increase in reported items; CI = 0.82-1.18). Manuscripts with positive abstract conclusions were less likely to be accepted by radiology journals (OR = 0.45; CI = 0.24-0.86), while YI (OR = 0.85; CI = 0.56-1.29) and STARD for Abstracts adherence (OR = 1.06; CI = 0.87-1.30) showed no significant association. Positive conclusions were present in 86.7% of submitted conference abstracts and 90.2% of journal manuscripts.

DATA CONCLUSION

Diagnostic test accuracy studies with positive findings were not preferentially accepted by the evaluated radiology conferences or journals.

EVIDENCE LEVEL

3 TECHNICAL EFFICACY: Stage 2.

Reporting of Artificial Intelligence Diagnostic Accuracy Studies in Pathology Abstracts: Compliance with STARD for Abstracts Guidelines

Artificial intelligence (AI) research is transforming the range tools and technologies available to pathologists, leading to potentially faster, personalized and more accurate diagnoses for patients. However, to see the use of tools for patient benefit and achieve this safely, the implementation of any algorithm must be underpinned by high quality evidence from research that is understandable, replicable, usable and inclusive of details needed for critical appraisal of potential bias. Evidence suggests that reporting guidelines can improve the completeness of reporting of research, especially with good awareness of guidelines. The quality of evidence provided by abstracts alone is profoundly important, as they influence the decision of a researcher to read a paper, attend a conference presentation or include a study in a systematic review.

AI abstracts at two international pathology conferences were assessed to establish completeness of reporting against the STARD for Abstracts criteria. This reporting guideline is for abstracts of diagnostic accuracy studies and includes a checklist of 11 essential items required to accomplish satisfactory reporting of such an investigation. A total of 3488 abstracts were screened from the United States & Canadian Academy of Pathology annual meeting 2019 and the 31st European Congress of Pathology (ESP Congress). Of these, 51 AI diagnostic accuracy abstracts were identified and assessed against the STARD for Abstracts criteria for completeness of reporting. Completeness of reporting was suboptimal for the 11 essential criteria, a mean of 5.8 (SD 1.5) items were detailed per abstract. Inclusion was variable across the different checklist items, with all abstracts including study objectives and no abstracts including a registration number or registry. Greater use and awareness of the STARD for Abstracts criteria could improve completeness of reporting and further consideration is needed for areas where AI studies are vulnerable to bias.

Adherence to the Standards for Reporting of Diagnostic Accuracy Studies (STARD): a survey of four journals in laboratory medicine

Background

The Standards for Reporting of Diagnostic Accuracy Studies (STARD) statement has been updated in 2015. Many diagnostic test accuracy (DTA) studies have been published in medical laboratory journals, but their adherence to the updated STARD statement remains unknown.

Methods

We searched the PubMed database to verify studies published in 4 laboratory journals, including Clinical Chemistry, Clinical Chemistry and Laboratory Medicine, Clinica Chimica Acta, and Clinical Biochemistry, in 2019. DTA studies were identified and their adherence to the STARD statement was assessed.

Results

A total of 45 studies were included in this analysis. Overall, 18 out of 34 STARD items were reported. The items (adherence rate) of sample size estimation (4%), adverse events (9%), protocol (9%), registration (16%), missing value (22%), indeterminate results (18%), and cross-tabulation (22%) were the most frequently unreported items.

Conclusions

Adherence to the STARD statement in DTA articles published in laboratory medicine seems as yet unsatisfactory. Our study emphasizes the necessity to improve the reporting quality of DTA studies published in medical laboratory journals.

Blinding practices during acute point-of-care ultrasound research: the BLIND-US meta-research study

OBJECTIVES

Point-of-care ultrasound (POCUS) diagnostic accuracy research has significant variation in blinding practices. This study characterises the blinding practices during acute POCUS research to determine whether research methodology adequately reflects POCUS use in routine clinical practice.

DESIGN, SETTINGS AND PARTICIPANTS

A search for POCUS diagnostic accuracy studies published in Emergency Medicine, Anaesthesia and Critical Care journals from January 2016 to January 2020 was performed. Studies were included if they were primary diagnostic accuracy studies. The study year, journal impact factor, population, hospital area, body region, study design, blinding of the POCUS interpreter to clinical information, whether the person performing the POCUS scan was the same person interpreting the scan, and whether the study reported incremental diagnostic yield were extracted in duplicate by two authors. Descriptive statistics were provided and prespecified subgroup analysis was performed.

MAIN OUTCOME MEASURES

The primary outcome was the number of studies that blinded the POCUS interpreter to at least some part of the clinical information. Secondary outcomes included whether the person performing the POCUS scan was the same person interpreting it and whether the study reported incremental diagnostic yield.

RESULTS

520 abstracts were screened with 97 studies included. The POCUS interpreter was blinded to clinical information in 37 studies (38.1%), not blinded in 34 studies (35.1%) and not reported in 26 studies (26.8%). The POCUS interpreter was the same person obtaining the images in 72 studies (74.2%), different in 14 studies (14.4%) and not reported in 11 studies (11.3%). Only four studies (4.1%) reported incremental diagnostic yield for POCUS. Inter-rater reliability was moderate (k=0.64). Subgroup analysis based on impact factor, body region, hospital area, patient population and study design did not show significant differences after completing pairwise comparisons.

CONCLUSIONS

Although blinding the POCUS interpreter to clinical information may be done in a perceived attempt to limit bias, this may result in accuracy estimates that do not reflect routine clinical practice. Similarly, having a different clinician perform and interpret the POCUS scan significantly limits generalisability to practice as it does not truly reflect 'point-of-care' ultrasound at all. Reporting incremental diagnostic yield from implementing POCUS into a diagnostic pathway better reflects the value of POCUS; however, this methodology was infrequently used.

TRIAL REGISTRATION NUMBER

The study protocol was registered on Open Science Framework (https://osf.io/h5fe7/).

Tweeting Bias in Diagnostic Test Accuracy Research: Does Title or Conclusion Positivity Influence Dissemination?

PURPOSE

To examine if tweeting bias exists within imaging literature by determining if diagnostic test accuracy (DTA) studies with positive titles or conclusions are tweeted more than non-positive studies.

METHODS

DTA studies published between October 2011 to April 2016 were included. Positivity of titles and conclusions were assessed independently and in duplicate, with disagreements resolved by consensus. A negative binomial regression analysis controlling for confounding variables was performed to assess the relationship between title or conclusion positivity and tweets an article received in the 100 days post-publication.

RESULTS

354 DTA studies were included. Twenty-four (7%) titles and 300 (85%) conclusions were positive (or positive with qualifier); 1 (0.3%) title and 23 (7%) conclusions were negative; and 329 (93%) titles and 26 (7%) conclusions were neutral. Studies with positive, negative, and neutral titles received a mean of 0.38, 0.00, and 0.45 tweets per study; while those with positive, negative, and neutral conclusions received a mean of 0.44, 0.61, and 0.38 tweets per study. Regression coefficients were -0.05 (SE 0.46) for positive relative to non-positive titles, and -0.09 (SE 0.31) for positive relative to non-positive conclusions. The positivity of the title (P = 0.91) or conclusion (P = 0.76) was not significantly associated with the number of tweets an article received.

CONCLUSIONS

The positivity of the title or conclusion for DTA studies does not influence the amount of tweets it receives suggesting that tweet bias is not present among imaging diagnostic accuracy studies. Study protocol available at https://osf.io/hdk2m/.

Completeness of reporting for systematic reviews of point-of-care ultrasound: a meta-research study

OBJECTIVES

Systematic reviews are often considered among the highest quality of evidence. Completely reported systematic reviews, however, are required so readers can assess for generalisability of the research to practice and risk of bias. The objective of this study was to assess the completeness of reporting for systematic reviews assessing the diagnostic accuracy of point-of-care ultrasound (POCUS) using the Preferred Reporting Items for Systematic Reviews and Meta-analyses for Diagnostic Test Accuracy (PRISMA-DTA) checklist that was published in 2018.

DESIGN AND SETTING

In this meta-research study, MEDLINE, EMBASE and Cochrane Library databases were searched, with no date restriction, on March 1st, 2020 for systematic reviews assessing the diagnostic accuracy of POCUS. Adherence to PRISMA-DTA for the main text and abstract was scored independently and in duplicate using a modified checklist. Prespecified subgroup analyses were performed.

MAIN OUTCOME MEASURES

The primary outcome was the mean PRISMA-DTA checklist adherence for the full-text and abstract.

RESULTS

A total of 71 studies published from 2008 to 2020 met the inclusion criteria. The overall adherence for the full-text was moderate: 19.8 out of 26.0 items (76%) and for the abstract was 7.0 out of 11.0 items (64%). Although many items in the PRISMA-DTA checklist were frequently reported, several were r infrequently reported (<33% of studies), including item 5 (protocol registration), item D2 (minimally acceptable test accuracy) and item 14 (variability in target condition, index test and reference standards). Subgroup analyses showed a higher PRISMA-DTA mean adherence (SD) for high impact journals (20.9 (2.52) vs 18.9 (1.95); p<0.001), studies including supplemental materials (20.6 (2.48) vs 18.9 (2.28); p=0.004), studies citing adherence to PRISMA reporting guidelines (20.4 (1.95) vs 19.0 (3.00); p=0.038) and studies published in journals endorsing PRISMA guidelines (20.2 (2.47) vs 18.6 (2.37); p=0.025). There was variable adherence based on journal of publication (p=0.006), but not for study population (adult vs paediatric vs mixed) (p=0.62), year of publication (p=0.94), body region (p=0.78) or country (p=0.40). There was no variability in abstract adherence based on whether the abstract was structured with subheadings or not (p=0.25). A Spearman's correlation found moderate correlation between higher word counts and abstractand full-text adherence (R=0.45, p<0.001 and R=0.38, p=0.001), respectively.

CONCLUSIONS

Overall, the reporting of POCUS diagnostic accuracy systematic reviews and meta-analyses was moderate. We identified deficits in several key areas including the preregistration of systematic reviews in an online repository, handling of multiple definitions of target conditions, index tests and reference standards and specifying minimally acceptable test accuracy. Prospective registration of reviews and detailed reporting as per PRISMA-DTA during the research process could improve reporting completeness. At an editorial level, word count and supplemental material limitations may impede reporting completeness, whereas endorsement of reporting guidelines on journal websites could improve reporting.

Thoracic imaging tests for the diagnosis of COVID-19

BACKGROUND

The respiratory illness caused by SARS-CoV-2 infection continues to present diagnostic challenges. Our 2020 edition of this review showed thoracic (chest) imaging to be sensitive and moderately specific in the diagnosis of coronavirus disease 2019 (COVID-19). In this update, we include new relevant studies, and have removed studies with case-control designs, and those not intended to be diagnostic test accuracy studies.

OBJECTIVES

To evaluate the diagnostic accuracy of thoracic imaging (computed tomography (CT), X-ray and ultrasound) in people with suspected COVID-19.

SEARCH METHODS

We searched the COVID-19 Living Evidence Database from the University of Bern, the Cochrane COVID-19 Study Register, The Stephen B. Thacker CDC Library, and repositories of COVID-19 publications through to 30 September 2020. We did not apply any language restrictions.

SELECTION CRITERIA

We included studies of all designs, except for case-control, that recruited participants of any age group suspected to have COVID-19 and that reported estimates of test accuracy or provided data from which we could compute estimates.

DATA COLLECTION AND ANALYSIS

The review authors independently and in duplicate screened articles, extracted data and assessed risk of bias and applicability concerns using the QUADAS-2 domain-list. We presented the results of estimated sensitivity and specificity using paired forest plots, and we summarised pooled estimates in tables. We used a bivariate meta-analysis model where appropriate. We presented the uncertainty of accuracy estimates using 95% confidence intervals (CIs).

MAIN RESULTS

We included 51 studies with 19,775 participants suspected of having COVID-19, of whom 10,155 (51%) had a final diagnosis of COVID-19. Forty-seven studies evaluated one imaging modality each, and four studies evaluated two imaging modalities each. All studies used RT-PCR as the reference standard for the diagnosis of COVID-19, with 47 studies using only RT-PCR and four studies using a combination of RT-PCR and other criteria (such as clinical signs, imaging tests, positive contacts, and follow-up phone calls) as the reference standard. Studies were conducted in Europe (33), Asia (13), North America (3) and South America (2); including only adults (26), all ages (21), children only (1), adults over 70 years (1), and unclear (2); in inpatients (2), outpatients (32), and setting unclear (17). Risk of bias was high or unclear in thirty-two (63%) studies with respect to participant selection, 40 (78%) studies with respect to reference standard, 30 (59%) studies with respect to index test, and 24 (47%) studies with respect to participant flow. For chest CT (41 studies, 16,133 participants, 8110 (50%) cases), the sensitivity ranged from 56.3% to 100%, and specificity ranged from 25.4% to 97.4%. The pooled sensitivity of chest CT was 87.9% (95% CI 84.6 to 90.6) and the pooled specificity was 80.0% (95% CI 74.9 to 84.3). There was no statistical evidence indicating that reference standard conduct and definition for index test positivity were sources of heterogeneity for CT studies. Nine chest CT studies (2807 participants, 1139 (41%) cases) used the COVID-19 Reporting and Data System (CO-RADS) scoring system, which has five thresholds to define index test positivity. At a CO-RADS threshold of 5 (7 studies), the sensitivity ranged from 41.5% to 77.9% and the pooled sensitivity was 67.0% (95% CI 56.4 to 76.2); the specificity ranged from 83.5% to 96.2%; and the pooled specificity was 91.3% (95% CI 87.6 to 94.0). At a CO-RADS threshold of 4 (7 studies), the sensitivity ranged from 56.3% to 92.9% and the pooled sensitivity was 83.5% (95% CI 74.4 to 89.7); the specificity ranged from 77.2% to 90.4% and the pooled specificity was 83.6% (95% CI 80.5 to 86.4). For chest X-ray (9 studies, 3694 participants, 2111 (57%) cases) the sensitivity ranged from 51.9% to 94.4% and specificity ranged from 40.4% to 88.9%. The pooled sensitivity of chest X-ray was 80.6% (95% CI 69.1 to 88.6) and the pooled specificity was 71.5% (95% CI 59.8 to 80.8). For ultrasound of the lungs (5 studies, 446 participants, 211 (47%) cases) the sensitivity ranged from 68.2% to 96.8% and specificity ranged from 21.3% to 78.9%. The pooled sensitivity of ultrasound was 86.4% (95% CI 72.7 to 93.9) and the pooled specificity was 54.6% (95% CI 35.3 to 72.6). Based on an indirect comparison using all included studies, chest CT had a higher specificity than ultrasound. For indirect comparisons of chest CT and chest X-ray, or chest X-ray and ultrasound, the data did not show differences in specificity or sensitivity.

AUTHORS' CONCLUSIONS

Our findings indicate that chest CT is sensitive and moderately specific for the diagnosis of COVID-19. Chest X-ray is moderately sensitive and moderately specific for the diagnosis of COVID-19. Ultrasound is sensitive but not specific for the diagnosis of COVID-19. Thus, chest CT and ultrasound may have more utility for excluding COVID-19 than for differentiating SARS-CoV-2 infection from other causes of respiratory illness. Future diagnostic accuracy studies should pre-define positive imaging findings, include direct comparisons of the various modalities of interest in the same participant population, and implement improved reporting practices.

Accuracy of liver and spleen stiffness on magnetic resonance elastography for detecting portal hypertension: a systematic review and meta-analysis

INTRODUCTION

The purpose of this systematic review and meta-analysis was to evaluate the diagnostic accuracy of liver and spleen stiffness on magnetic resonance elastography (MRE) for detecting clinically significant portal hypertension.

METHODS

A systematic review of MEDLINE, EMBASE, Scopus, the Cochrane Library, and the Grey Literature through to 15 August 2019 was performed. Original articles with >10 patients evaluating liver and/or spleen stiffness on MRE using a reference standard of portal hypertension defined as intractable ascites, esophageal varices, encephalopathy and/or death were included in analysis. Patient, clinical, MRI, and diagnostic performance was independently acquired by two reviewers. Meta-analysis was performed using a bivariate mixed-effects regression model.

RESULTS

Fourteen studies were included with 12 studies evaluating liver stiffness and eight studies evaluating spleen stiffness. The pooled and weighted sensitivity, specificity, and area under the curve (AUC) values for liver stiffness on MRE were 83% [95% confidence interval (CI) 72-90%], 80% (95% CI 70-88%), and 88% (95% CI 85-91%), respectively. The pooled and weighted sensitivity, specificity, and AUC values for spleen stiffness on MRE were 79% (95% CI 61-90%), 90% (95% CI 80-95%), and 92% (95% CI 89-94%), respectively. The liver and spleen stiffness sensitivity and specificity values were comparable when evaluating for esophageal varices only at of 80% (95% CI 66-89%) and 76% (95% CI 62-86%) for liver stiffness, and 75% (95% CI 52-90%) and 89% (95% CI 70-96%) for spleen stiffness.

DISCUSSION

Liver and spleen stiffness on MRE can serve as a supplemental noninvasive assessment tools for detecting clinically significant portal hypertension. Spleen stiffness may be more specific and accurate than liver stiffness for detecting portal hypertension.

Barriers to reporting guideline adherence in point-of-care ultrasound research: a cross-sectional survey of authors and journal editors

OBJECTIVE

Although the literature supporting the use of point-of-care ultrasound (POCUS) continues to grow, incomplete reporting of primary diagnostic accuracy studies has previously been identified as a barrier to translating research into practice and to performing unbiased systematic reviews. This study assesses POCUS investigator and journal editor attitudes towards barriers to adhering to the Standards for Reporting of Diagnostic Accuracy Studies (STARD) 2015 guidelines.

DESIGN, SETTING, PARTICIPANTS

Two separate surveys using a 5-point Likert scale were sent to POCUS study investigators and journal editors to assess for knowledge, attitude and behavioural barriers to the complete reporting of POCUS research. Respondents were identified based on a previous study assessing STARD 2015 adherence for POCUS studies published in emergency medicine, anaesthesia and critical care journals. Responses were anonymously linked to STARD 2015 adherence data from the previous study. Written responses were thematically grouped into the following categories: knowledge, attitude and behavioural barriers to quality reporting, or other. Likert response items are reported as median with IQRs.

MAIN OUTCOME MEASURES

The primary outcome was the median Likert score for the investigator and editor surveys assessing knowledge, attitude and behavioural beliefs about barriers to adhering to the STARD 2015 guidelines.

RESULTS

The investigator survey response rate was 18/69 (26%) and the editor response rate was 5/21 (24%). Most investigator respondents were emergency medicine practitioners (13/21, 62%). Two-thirds of investigators were aware of the STARD 2015 guidelines (12/18, 67%) and overall agreed that incomplete reporting limits generalisability and the ability to detect risk of bias (median 4 (4, 5)). Investigators felt that the STARD 2015 guidelines were useful, easy to find and easy to use (median 4 (4, 4.25); median 4 (4, 4.25) and median 4 (3, 4), respectively). There was a shared opinion held by investigators and editors that the peer review process be primarily responsible for ensuring complete research reporting (median 4 (3, 4) and median 4 (3.75, 4), respectively). Three of 18 authors (17%) felt that the English publication language of STARD 2015 was a barrier to adherence.

CONCLUSIONS

Although investigators and editors recognise the importance of completely reported research, reporting quality is still a core issue for POCUS research. The shared opinion held by investigators and editors that the peer review process be primarily responsible for reporting quality is potentially problematic; we view completely reported research as an integral part of the research process that investigators are responsible for, with the peer review process serving as another additional layer of quality control. Endorsement of reporting guidelines by journals, auditing reporting guideline adherence during the peer review process and translation of STARD 2015 guidelines into additional languages may improve reporting completeness for the acute POCUS literature.

TRIAL REGISTRATION NUMBER

Open Science Framework Registry (https://osf.io/5pzxs/).

Thoracic imaging tests for the diagnosis of COVID-19

BACKGROUND

The respiratory illness caused by SARS-CoV-2 infection continues to present diagnostic challenges. Early research showed thoracic (chest) imaging to be sensitive but not specific in the diagnosis of coronavirus disease 2019 (COVID-19). However, this is a rapidly developing field and these findings need to be re-evaluated in the light of new research. This is the first update of this 'living systematic review'. This update focuses on people suspected of having COVID-19 and excludes studies with only confirmed COVID-19 participants.

OBJECTIVES

To evaluate the diagnostic accuracy of thoracic imaging (computed tomography (CT), X-ray and ultrasound) in people with suspected COVID-19.

SEARCH METHODS

We searched the COVID-19 Living Evidence Database from the University of Bern, the Cochrane COVID-19 Study Register, The Stephen B. Thacker CDC Library, and repositories of COVID-19 publications through to 22 June 2020. We did not apply any language restrictions.

SELECTION CRITERIA

We included studies of all designs that recruited participants of any age group suspected to have COVID-19, and which reported estimates of test accuracy, or provided data from which estimates could be computed. When studies used a variety of reference standards, we retained the classification of participants as COVID-19 positive or negative as used in the study.

DATA COLLECTION AND ANALYSIS

We screened studies, extracted data, and assessed the risk of bias and applicability concerns using the QUADAS-2 domain-list independently, in duplicate. We categorised included studies into three groups based on classification of index test results: studies that reported specific criteria for index test positivity (group 1); studies that did not report specific criteria, but had the test reader(s) explicitly classify the imaging test result as either COVID-19 positive or negative (group 2); and studies that reported an overview of index test findings, without explicitly classifying the imaging test as either COVID-19 positive or negative (group 3). We presented the results of estimated sensitivity and specificity using paired forest plots, and summarised in tables. We used a bivariate meta-analysis model where appropriate. We presented uncertainty of the accuracy estimates using 95% confidence intervals (CIs).

MAIN RESULTS

We included 34 studies: 30 were cross-sectional studies with 8491 participants suspected of COVID-19, of which 4575 (54%) had a final diagnosis of COVID-19; four were case-control studies with 848 cases and controls in total, of which 464 (55%) had a final diagnosis of COVID-19. Chest CT was evaluated in 31 studies (8014 participants, 4224 (53%) cases), chest X-ray in three studies (1243 participants, 784 (63%) cases), and ultrasound of the lungs in one study (100 participants, 31 (31%) cases). Twenty-six per cent (9/34) of all studies were available only as preprints. Nineteen studies were conducted in Asia, 10 in Europe, four in North America and one in Australia. Sixteen studies included only adults, 15 studies included both adults and children and one included only children. Two studies did not report the ages of participants. Twenty-four studies included inpatients, four studies included outpatients, while the remaining six studies were conducted in unclear settings. The majority of included studies had a high or unclear risk of bias with respect to participant selection, index test, reference standard, and participant flow. For chest CT in suspected COVID-19 participants (31 studies, 8014 participants, 4224 (53%) cases) the sensitivity ranged from 57.4% to 100%, and specificity ranged from 0% to 96.0%. The pooled sensitivity of chest CT in suspected COVID-19 participants was 89.9% (95% CI 85.7 to 92.9) and the pooled specificity was 61.1% (95% CI 42.3 to 77.1). Sensitivity analyses showed that when the studies from China were excluded, the studies from other countries demonstrated higher specificity compared to the overall included studies. When studies that did not classify index tests as positive or negative for COVID-19 (group 3) were excluded, the remaining studies (groups 1 and 2) demonstrated higher specificity compared to the overall included studies. Sensitivity analyses limited to cross-sectional studies, or studies where at least two reverse transcriptase polymerase chain reaction (RT-PCR) tests were conducted if the first was negative, did not substantively alter the accuracy estimates. We did not identify publication status as a source of heterogeneity. For chest X-ray in suspected COVID-19 participants (3 studies, 1243 participants, 784 (63%) cases) the sensitivity ranged from 56.9% to 89.0% and specificity from 11.1% to 88.9%. The sensitivity and specificity of ultrasound of the lungs in suspected COVID-19 participants (1 study, 100 participants, 31 (31%) cases) were 96.8% and 62.3%, respectively. We could not perform a meta-analysis for chest X-ray or ultrasound due to the limited number of included studies.

AUTHORS' CONCLUSIONS

Our findings indicate that chest CT is sensitive and moderately specific for the diagnosis of COVID-19 in suspected patients, meaning that CT may have limited capability in differentiating SARS-CoV-2 infection from other causes of respiratory illness. However, we are limited in our confidence in these results due to the poor study quality and the heterogeneity of included studies. Because of limited data, accuracy estimates of chest X-ray and ultrasound of the lungs for the diagnosis of suspected COVID-19 cases should be carefully interpreted. Future diagnostic accuracy studies should pre-define positive imaging findings, include direct comparisons of the various modalities of interest on the same participant population, and implement improved reporting practices. Planned updates of this review will aim to: increase precision around the accuracy estimates for chest CT (ideally with low risk of bias studies); obtain further data to inform accuracy of chest X-rays and ultrasound; and obtain data to further fulfil secondary objectives (e.g. 'threshold' effects, comparing accuracy estimates across different imaging modalities) to inform the utility of imaging along different diagnostic pathways.

Compliance With Standards for STARD 2015 Reporting Recommendations in Pathology

OBJECTIVES

Lack of experimental reproducibility has led to growing interest in guidelines to enhance completeness and transparency in research reporting. This retrospective survey sought to determine compliance with Standards for Reporting of Diagnostic Accuracy Studies (STARD) 2015 statement in the recent pathology scientific literature.

METHODS

Two raters independently scored 171 pathology diagnostic accuracy studies for compliance with 34 STARD items and subcomponents. Overall adherence was calculated as a proportion after excluding nonapplicable items.

RESULTS

After excluding nonapplicable items, there was 50% overall adherence to STARD reporting recommendations. In total, 15.44 ± 3.59 items were reported per article (range, 4-28 out of maximum possible of 34). There was substantial heterogeneity in individual item reporting, with greater than 75% reporting in eight of 34 items and less than 25% reporting in 11 of 34 items. Less than 10% of articles reported hypotheses, subgroup analyses for confounding, sample size calculations, subject flow diagrams, study registrations, and links to full study protocols. Significantly more items were reported in articles from journals that endorsed STARD (16.14 vs 14.84, P = .0175).

CONCLUSIONS

These findings demonstrate incomplete reporting of essential items in pathology diagnostic accuracy studies. More vigorous enforcement of reporting checklists might improve adherence to minimum reporting standards.

Reporting Quality of Diagnostic Accuracy Studies in Laboratory Medicine: Adherence to Standards for Reporting of Diagnostic Accuracy Studies (STARD) 2015

Background

Poor reporting quality in diagnostic accuracy studies hampers an adequate judgment of the validity of the study. The Standards for Reporting of Diagnostic Accuracy Studies (STARD) statement was published to improve the reporting quality of diagnostic accuracy studies. This study aimed to evaluate the adherence of diagnostic accuracy studies published in Annals of Laboratory Medicine (ALM) to STARD 2015 and to identify directions for improvement in the reporting quality of these studies.

Methods

Two independent authors assessed articles published in ALM between 2012–2018 for compliance with 30 STARD 2015 checklist items to identify all eligible diagnostic accuracy studies published during this period. We included 66 diagnostic accuracy studies. A total of the fulfilled STARD items were calculated, and adherence was analyzed on an individual-item basis.

Results

The overall mean±SD number of STARD items reported for the included studies was 11.2±2.7. Only five (7.6%) studies adhered to more than 50% of the 30 items. No study satisfied more than 80% of the items. Large variability in adherence to reporting standards was detected across items, ranging from 0% to 100%.

Conclusions

Adherence to STARD 2015 is suboptimal among diagnostic accuracy studies published in ALM. Our study emphasizes the necessity of adherence to STARD to improve the reporting quality of future diagnostic accuracy studies to be published in ALM.

Overinterpretation of Research Findings: Evaluation of “Spin” in Systematic Reviews of Diagnostic Accuracy Studies in High–Impact Factor Journals

Background

To compare the frequency of “spin” in systematic reviews of diagnostic accuracy studies in high-impact journals with the frequency a previously assessed series of reviews.

Methods

Medline was searched from January 2010 to January 2019. Systematic reviews of diagnostic accuracy studies were included if they reported a meta-analysis and were published in a journal with an impact factor &gt;5. Two investigators independently scored each included systematic review for positivity of conclusions and for actual and potential overinterpretation practices.

Results

Of 137 included systematic reviews, actual overinterpretation was present in ≥1 form in the abstract in 63 (46%) and in the full-text report in 52 (38%); 108 (79%) contained a form of potential overinterpretation. Compared with the previously assessed series (reviews published 2015–2016), reviews in this series were less likely to contain ≥1 form of actual overinterpretation in the abstract and full-text report or ≥1 form of potential overinterpretation (P &lt; 0.001 for all comparisons). The significance of these comparisons did not persist for actual overinterpretation in sensitivity analysis in which Cochrane systematic reviews were removed. Reviews published in the Cochrane Database of Systematic Reviews were less likely to contain actual overinterpretation in the abstract or the full-text report than reviews in other high-impact journals (P &lt; 0.001 for both comparisons).

Conclusions

Reviews of diagnostic accuracy studies in high-impact journals are less likely to contain overinterpretation or spin. This difference is largely due to the reviews published in the Cochrane Database of Systematic Reviews, which contain spin less often than reviews published in other high-impact journals.

Adherence to the Standards for Reporting of Diagnostic Accuracy (STARD) 2015 Guidelines in Acute Point-of-Care Ultrasound Research

IMPORTANCE

Incomplete reporting of diagnostic accuracy research impairs assessment of risk of bias and limits generalizability. Point-of-care ultrasound has become an important diagnostic tool for acute care physicians, but studies assessing its use are of varying methodological quality.

OBJECTIVE

To assess adherence to the Standards for Reporting of Diagnostic Accuracy (STARD) 2015 guidelines in the literature on acute care point-of-care ultrasound.

EVIDENCE REVIEW

MEDLINE was searched to identify diagnostic accuracy studies assessing point-of-care ultrasound published in critical care, emergency medicine, or anesthesia journals from 2016 to 2019. Studies were evaluated for adherence to the STARD 2015 guidelines, with the following variables analyzed: journal, country, STARD citation, STARD-adopting journal, impact factor, patient population, use of supplemental material, and body region. Data analysis was performed in November 2019.

FINDINGS

Seventy-four studies were included in this systematic review for assessment. Overall adherence to STARD was moderate, with 66% (mean [SD], 19.7 [2.9] of 30 items) of STARD items reported. Items pertaining to imaging specifications, patient population, and readers of the index test were frequently reported (>66% of studies). Items pertaining to blinding of readers to clinical data and to the index or reference standard, analysis of heterogeneity, indeterminate and missing data, and time intervals between index and reference test were either moderately (33%-66%) or infrequently (<33%) reported. Studies in STARD-adopting journals (mean [SD], 20.5 [2.9] items in adopting journals vs 18.6 [2.3] items in nonadopting journals; P = .002) and studies citing STARD (mean [SD], 21.3 [0.9] items in citing studies vs 19.5 [2.9] items in nonciting studies; P = .01) reported more items. Variation by country and journal of publication were identified. No differences in STARD adherence were identified by body region imaged (mean [SD], abdominal, 20.0 [2.5] items; head and neck, 17.8 [1.6] items; musculoskeletal, 19.2 [3.1] items; thoracic, 20.2 [2.8] items; and other or procedural, 19.8 [2.7] items; P = .29), study design (mean [SD], prospective, 19.7 [2.9] items; retrospective, 19.7 [1.8] items; P > .99), patient population (mean [SD], pediatric, 20.0 [3.1] items; adult, 20.2 [2.7] items; mixed, 17.9 [1.9] items; P = .09), use of supplementary materials (mean [SD], yes, 19.2 [3.0] items; no, 19.7 [2.8] items; P = .91), or journal impact factor (mean [SD], higher impact factor, 20.3 [3.1] items; lower impact factor, 19.1 [2.4] items; P = .08).

CONCLUSIONS AND RELEVANCE

Overall, the literature on acute care point-of-care ultrasound showed moderate adherence to the STARD 2015 guidelines, with more complete reporting found in studies citing STARD and those published in STARD-adopting journals. This study has established a current baseline for reporting; however, future studies are required to understand barriers to complete reporting and to develop strategies to mitigate them.

Percutaneous Image-Guided Biopsy of the Spleen: Experience at a Single Tertiary Care Center

PURPOSE

The purpose of this study is to assess the complication rate of percutaneous image-guided biopsy of the spleen at our institution and to evaluate for variables associated with complication rate.

METHODS

This is a Research Ethics Board approved retrospective study of consecutive patients who underwent image-guided biopsy of the spleen at our institution from January 2010 to November 2019. Complications, imaging findings, and pathologic diagnosis were reviewed. Complications (major and minor) were classified per Society of Interventional Radiology Guidelines, and complication rate was calculated. Logistic regression was applied to determine factors associated with complications. Diagnostic yield was calculated.

RESULTS

In all, 55 patients (28 female) underwent splenic biopsy using ultrasound guidance. The most common indication was possible lymphoma in 41 (71.7%) patients followed by query metastasis 18 (31.5%) patients. Core biopsies (18 g/20 g) were done in 53 (92%) cases, and fine-needle aspiration (22 g) was performed in 4 (8%). The median number of samples collected was 4 (range: 2-9). The results were diagnostic in 54 cases (94.7%, 95% confidence interval [CI]: 88.7-100.0). There were 12 (21%, 95% CI: 10.1-31.9) patients with minor complications and 2 (3.5%, 95% CI: 0.0-8.4) with major complications (2 splenic bleeds requiring embolization, no splenectomy, or deaths). No variables (needle size, lesion size, and number of passes) were associated with complication rate.

CONCLUSION

Percutaneous image-guided biopsy of the spleen at a single tertiary care institution demonstrates major complication rate comparable to that in the literature with no variables associated with complication rate; there were no cases of splenectomy or death.

Medical Diagnostic Tests: A Review of Test Anatomy, Phases, and Statistical Treatment of Data

Diagnostic tests are approaches used in clinical practice to identify with high accuracy the disease of a particular patient and thus to provide early and proper treatment. Reporting high-quality results of diagnostic tests, for both basic and advanced methods, is solely the responsibility of the authors. Despite the existence of recommendation and standards regarding the content or format of statistical aspects, the quality of what and how the statistic is reported when a diagnostic test is assessed varied from excellent to very poor. This article briefly reviews the steps in the evaluation of a diagnostic test from the anatomy, to the role in clinical practice, and to the statistical methods used to show their performances. The statistical approaches are linked with the phase, clinical question, and objective and are accompanied by examples. More details are provided for phase I and II studies while the statistical treatment of phase III and IV is just briefly presented. Several free online resources useful in the calculation of some statistics are also given.

Publication Bias: Association of Diagnostic Accuracy in Radiology Conference Abstracts with Full-Text Publication

Background Recent investigations have identified a faster time to publication for imaging studies with higher diagnostic test accuracy (DTA), but it is unknown whether such studies are more likely to be published. A higher probability of full-text publication for studies with higher DTA could have negative consequences on clinical decision making and patient care. Purpose To evaluate the proportion of imaging diagnostic accuracy studies presented as conference abstracts that reach full-text publication and to identify whether there is an association between diagnostic accuracy and full-text publication in peer-reviewed journals within 5 years after abstract submission. Materials and Methods Diagnostic accuracy research abstracts presented at the Radiological Society of North America (RSNA) Annual Meeting in 2011 and 2012 were evaluated between September 1, 2017, and January 11, 2018. Sensitivity and specificity from the abstracts were used to calculate the Youden index (sensitivity + specificity-1); additional abstract characteristics were extracted. To identify full-text publications within 5 years after abstract submission, PubMed and Google Scholar were searched, and authors were contacted. Logistic regression analysis was used to assess for associations between higher diagnostic accuracy and full-text publication. Results A total of 7970 abstracts were evaluated, and 405 were included. Of these, 288 (71%) reached full-text publication within 5 years after abstract submission. Logistic regression analysis accounting for several confounding variables failed to show an association between reported Youden index in the conference abstract and probability of full-text publication (odds ratio, 1.01; 95% confidence interval: 0.99, 1.02; P = .21). Conclusion More than a quarter of abstracts presented at the RSNA Annual Meeting do not reach full-text publication in peer-reviewed journals. The magnitude of reported diagnostic accuracy was not associated with full-text publication, which is consistent with results of diagnostic accuracy studies in other medical specialties. © RSNA, 2019 Online supplemental material is available for this article. See also the editorial by Fielding in this issue.

The Changing Landscape of Journalology in Medicine

In the early 1970s, when Seminars in Nuclear Medicine started publication, little was known about the quality of reporting in biomedical journals. Senior scholars were invited to become scientific editors of journals based on their research credibility and stature. Their knowledge of journalology (publication science) was not assessed. Similarly, while the use of peer review was gaining momentum, there was limited guidance on the tasks and expectations of peer reviewing. Almost 50 years later, the evidence base regarding the quality of reporting is vast. This paper highlights some of this evidence including that relevant to imaging and nuclear medicine research. In biomedical publications, there is a crisis in reproducibility; high prevalence rates of reporting biases, such as selective outcome reporting; spin; low registration rates of research protocols; and endemic poor reporting of research across biomedicine. These issues and some more immediate solutions are also discussed in the paper. The use of reporting guidelines has been shown to be associated with better reporting of clinical trials and other research articles. The use of audit and feedback tools is likely to provide an important gauge about the functions of biomedical journals. Finally, the push to better equip scientific editors and peer reviewers is taking a more concerted effort.

Accuracy of the Liver Imaging Reporting and Data System in Computed Tomography and Magnetic Resonance Image Analysis of Hepatocellular Carcinoma or Overall Malignancy-A Systematic Review

BACKGROUND & AIMS

The Liver Imaging Reporting and Data System (LI-RADS) categorizes observations from imaging analyses of high-risk patients based on the level of suspicion for hepatocellular carcinoma (HCC) and overall malignancy. The categories range from definitely benign (LR-1) to definitely HCC (LR-5), malignancy (LR-M), or tumor in vein (LR-TIV) based on findings from computed tomography or magnetic resonance imaging. However, the actual percentage of HCC and overall malignancy within each LI-RADS category is not known. We performed a systematic review to determine the percentage of observations in each LI-RADS category for computed tomography and magnetic resonance imaging that are HCCs or malignancies.

METHODS

We searched the MEDLINE, Embase, Cochrane CENTRAL, and Scopus databases from 2014 through 2018 for studies that reported the percentage of observations in each LI-RADS v2014 and v2017 category that were confirmed as HCCs or other malignancies based on pathology, follow-up imaging analyses, or response to treatment (reference standard). Data were assessed on a per-observation basis. Random-effects models were used to determine the pooled percentages of HCC and overall malignancy for each LI-RADS category. Differences between categories were compared by analysis of variance of logit-transformed percentage of HCC and overall malignancy. Risk of bias and concerns about applicability were assessed with the Quality Assessment of Diagnostic Accuracy Studies 2 tool.

RESULTS

Of 454 studies identified, 17 (all retrospective studies) were included in the final analysis, consisting of 2760 patients, 3556 observations, and 2482 HCCs. The pooled percentages of observations confirmed as HCC and overall malignancy, respectively, were 94% (95% confidence interval [CI] 92%-96%) and 97% (95% CI 95%-99%) for LR-5, 74% (95% CI 67%-80%) and 80% (95% CI 75%-85%) for LR-4, 38% (95% CI 31%-45%) and 40% (95% CI 31%-50%) for LR-3, 13% (95% CI 8%-22%) and 14% (95% CI 9%-21%) for LR-2, 79% (95% CI 63%-89%) and 92% (95% CI 77%-98%) for LR-TIV, and 36% (95% CI 26%-48%) and 93% (95% CI 87%-97%) for LR-M. No malignancies were found in the LR-1 group. The percentage of HCCs and overall malignancies confirmed differed significantly among LR groups 2-5 (P < .00001). Patient selection was the most frequent factor that affected bias risk, because of verification bias and case-control study design.

CONCLUSIONS

In a systematic review, we found that increasing LI-RADS categories contained increasing percentages of HCCs and overall malignancy based on reference standard confirmation. Of observations categorized as LR-M, 93% were malignancies and 36% were confirmed as HCCs. The percentage of HCCs found in the LR-2 and LR-3 categories indicate the need for a more active management strategy than currently recommended. Prospective studies are needed to validate these findings. PROSPERO number CRD42018087441.

Completeness of Reporting of Systematic Reviews of Diagnostic Test Accuracy Based on the PRISMA-DTA Reporting Guideline

BACKGROUND

We evaluated the completeness of reporting of diagnostic test accuracy (DTA) systematic reviews using the recently developed Preferred Reporting Items for Systematic Reviews and MetaAnalyses (PRISMA)-DTA guidelines.

METHODS

MEDLINE® was searched for DTA systematic reviews published October 2017 to January 2018. The search time span was modulated to reach the desired sample size of 100 systematic reviews. Reporting on a per-item basis using PRISMA-DTA was evaluated.

RESULTS

One hundred reviews were included. Mean reported items were 18.6 of 26 (71%; SD = 1.9) for PRISMA-DTA and 5.5 of 11 (50%; SD = 1.2) for PRISMA-DTA for abstracts. Items in the results were frequently reported. Items related to protocol registration, characteristics of included studies, results synthesis, and definitions used in data extraction were infrequently reported. Infrequently reported items from PRISMA-DTA for abstracts included funding information, strengths and limitations, characteristics of included studies, and assessment of applicability. Reporting completeness was higher in higher impact factor journals (18.9 vs 18.1 items; P = 0.04), studies that cited PRISMA (18.9 vs 17.7 items; P = 0.003), or used supplementary material (19.1 vs 18.0 items; P = 0.004). Variability in reporting was associated with author country (P = 0.04) but not journal (P = 0.6), abstract word count limitations (P = 0.9), PRISMA adoption (P = 0.2), structured abstracts (P = 0.2), study design (P = 0.8), subspecialty area (P = 0.09), or index test (P = 0.5). Abstracts with a higher word count were more informative (R = 0.4; P &lt; 0.001). No association with word counts was observed for full-text reports (R = −0.03; P = 0.06).

CONCLUSIONS

Recently published reports of DTA systematic reviews are not fully informative when evaluated against the PRISMA-DTA guidelines. These results should guide knowledge translation strategies, including journal level (e.g., PRISMA-DTA adoption, increased abstract word count, and use of supplementary material) and author level (PRISMA-DTA citation awareness) strategies.

Prospective comparative diagnostic accuracy evaluation of dynamic contrast-enhanced (DCE) vs. dynamic susceptibility contrast (DSC) MR perfusion in differentiating tumor recurrence from radiation necrosis in treated high-grade gliomas

BACKGROUND

The appearance of a new enhancing lesion after surgery and chemoradiation for high-grade glioma (HGG) presents a common diagnostic dilemma. Histopathological analysis remains the reference standard in this situation.

PURPOSE

To prospectively compare the diagnostic accuracy of dynamic contrast-enhanced (DCE) vs. dynamic susceptibility contrast (DSC) in differentiating tumor recurrence (TR) from radiation necrosis (RN).

STUDY TYPE

Prospective diagnostic accuracy study.

POPULATION

In all, 98 consecutive treated HGG patients with new enhancing lesion. We excluded 32 patients due to inadequate follow-up or technical limitation.

FIELD STRENGTH/SEQUENCE

3 T DCE and DSC MR.

ASSESSMENT

Histogram and hot-spot analysis of cerebral blood volume (CBV), corrected CBV, K^trans , area under the curve (AUC), and plasma volume (Vp). The reference standard of TR and/or RN was determined by histopathology in 43 surgically resected lesions or by clinical/imaging follow-up in the rest.

STATISTICAL TESTS

Mann-Whitney U-tests, receiver operating characteristic (ROC) curve, and logistic regression analysis.

RESULTS

A total of 68 lesions were included. There were 37 TR, 28 RN, and three lesions with equal proportions of TR and RN. TR had significantly higher CBV, corrected CBV, CBV ratio, corrected CBV ratio, AUC ratio, and Vp ratio (P < 0.05) than RN on hot-spot analysis. CBV had the highest diagnostic accuracy (AUROC 0.71). On histogram analysis, TR had higher CBV and corrected CBV maximal value compared with RN (P = 0.006, AUROC = 0.70). Only CBV on hot-spot analysis remained significant after correction for multiple comparison, with no significant improvement in diagnostic accuracy when using a combination of parameters (AUROC 0.71 vs. 0.76, P = 0.24).

DATA CONCLUSION

DSC-derived CBV is the most accurate perfusion parameter in differentiating TR and RN. DSC and DCE-derived parameters reflecting the blood volume in an enhancing lesion are more accurate than the DCE-derived parameter K^trans . Clinical practice may be best guided by blood volume measurements, rather than permeability assessment for differentiation of TR from RN.

LEVEL OF EVIDENCE

1 Technical Efficacy Stage: 4 J. Magn. Reson. Imaging 2019;50:573-582.

Diagnostic accuracy research in glaucoma is still incompletely reported: An application of Standards for Reporting of Diagnostic Accuracy Studies (STARD) 2015

Background

Research has shown a modest adherence of diagnostic test accuracy (DTA) studies in glaucoma to the Standards for Reporting of Diagnostic Accuracy Studies (STARD). We have applied the updated 30-item STARD 2015 checklist to a set of studies included in a Cochrane DTA systematic review of imaging tools for diagnosing manifest glaucoma.

Methods

Three pairs of reviewers, including one senior reviewer who assessed all studies, independently checked the adherence of each study to STARD 2015. Adherence was analyzed on an individual-item basis. Logistic regression was used to evaluate the effect of publication year and impact factor on adherence.

Results

We included 106 DTA studies, published between 2003–2014 in journals with a median impact factor of 2.6. Overall adherence was 54.1% for 3,286 individual rating across 31 items, with a mean of 16.8 (SD: 3.1; range 8–23) items per study. Large variability in adherence to reporting standards was detected across individual STARD 2015 items, ranging from 0 to 100%. Nine items (1: identification as diagnostic accuracy study in title/abstract; 6: eligibility criteria; 10: index test (a) and reference standard (b) definition; 12: cut-off definitions for index test (a) and reference standard (b); 14: estimation of diagnostic accuracy measures; 21a: severity spectrum of diseased; 23: cross-tabulation of the index and reference standard results) were adequately reported in more than 90% of the studies. Conversely, 10 items (3: scientific and clinical background of the index test; 11: rationale for the reference standard; 13b: blinding of index test results; 17: analyses of variability; 18; sample size calculation; 19: study flow diagram; 20: baseline characteristics of participants; 28: registration number and registry; 29: availability of study protocol; 30: sources of funding) were adequately reported in less than 30% of the studies. Only four items showed a statistically significant improvement over time: missing data (16), baseline characteristics of participants (20), estimates of diagnostic accuracy (24) and sources of funding (30).

Conclusions

Adherence to STARD 2015 among DTA studies in glaucoma research is incomplete, and only modestly increasing over time.

Selection and Reporting of Statistical Methods to Assess Reliability of a Diagnostic Test: Conformity to Recommended Methods in a Peer-Reviewed Journal

Objective

To evaluate the frequency and adequacy of statistical analyses in a general radiology journal when reporting a reliability analysis for a diagnostic test.

Materials and Methods

Sixty-three studies of diagnostic test accuracy (DTA) and 36 studies reporting reliability analyses published in the Korean Journal of Radiology between 2012 and 2016 were analyzed. Studies were judged using the methodological guidelines of the Radiological Society of North America-Quantitative Imaging Biomarkers Alliance (RSNA-QIBA), and COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) initiative. DTA studies were evaluated by nine editorial board members of the journal. Reliability studies were evaluated by study reviewers experienced with reliability analysis.

Results

Thirty-one (49.2%) of the 63 DTA studies did not include a reliability analysis when deemed necessary. Among the 36 reliability studies, proper statistical methods were used in all (5/5) studies dealing with dichotomous/nominal data, 46.7% (7/15) of studies dealing with ordinal data, and 95.2% (20/21) of studies dealing with continuous data. Statistical methods were described in sufficient detail regarding weighted kappa in 28.6% (2/7) of studies and regarding the model and assumptions of intraclass correlation coefficient in 35.3% (6/17) and 29.4% (5/17) of studies, respectively. Reliability parameters were used as if they were agreement parameters in 23.1% (3/13) of studies. Reproducibility and repeatability were used incorrectly in 20% (3/15) of studies.

Conclusion

Greater attention to the importance of reporting reliability, thorough description of the related statistical methods, efforts not to neglect agreement parameters, and better use of relevant terminology is necessary.