The challenge of clinical radiology research

Enhanced deep learning model enables accurate alignment measurement across diverse institutional imaging protocols

BACKGROUND

Achieving consistent accuracy in radiographic measurements across different equipment and protocols is challenging. This study evaluates an advanced deep learning (DL) model, building upon a precursor, for its proficiency in generating uniform and precise alignment measurements in full-leg radiographs irrespective of institutional imaging differences.

METHODS

The enhanced DL model was trained on over 10,000 radiographs. Utilizing a segmented approach, it separately identified and evaluated regions of interest (ROIs) for the hip, knee, and ankle, subsequently integrating these regions. For external validation, 300 datasets from three distinct institutes with varied imaging protocols and equipment were employed. The study measured seven radiologic parameters: hip-knee-ankle angle, lateral distal femoral angle, medial proximal tibial angle, joint line convergence angle, weight-bearing line ratio, joint line obliquity angle, and lateral distal tibial angle. Measurements by the model were compared with an orthopedic specialist's evaluations using inter-observer and intra-observer intraclass correlation coefficients (ICCs). Additionally, the absolute error percentage in alignment measurements was assessed, and the processing duration for radiograph evaluation was recorded.

RESULTS

The DL model exhibited excellent performance, achieving an inter-observer ICC between 0.936 and 0.997, on par with an orthopedic specialist, and an intra-observer ICC of 1.000. The model's consistency was robust across different institutional imaging protocols. Its accuracy was particularly notable in measuring the hip-knee-ankle angle, with no instances of absolute error exceeding 1.5 degrees. The enhanced model significantly improved processing speed, reducing the time by 30-fold from an initial 10-11 s to 300 ms.

CONCLUSIONS

The enhanced DL model demonstrated its ability for accurate, rapid alignment measurements in full-leg radiographs, regardless of protocol variations, signifying its potential for broad clinical and research applicability.

Radiology artificial intelligence: a systematic review and evaluation of methods (RAISE)

OBJECTIVE

There has been a large amount of research in the field of artificial intelligence (AI) as applied to clinical radiology. However, these studies vary in design and quality and systematic reviews of the entire field are lacking.This systematic review aimed to identify all papers that used deep learning in radiology to survey the literature and to evaluate their methods. We aimed to identify the key questions being addressed in the literature and to identify the most effective methods employed.

METHODS

We followed the PRISMA guidelines and performed a systematic review of studies of AI in radiology published from 2015 to 2019. Our published protocol was prospectively registered.

RESULTS

Our search yielded 11,083 results. Seven hundred sixty-seven full texts were reviewed, and 535 articles were included. Ninety-eight percent were retrospective cohort studies. The median number of patients included was 460. Most studies involved MRI (37%). Neuroradiology was the most common subspecialty. Eighty-eight percent used supervised learning. The majority of studies undertook a segmentation task (39%). Performance comparison was with a state-of-the-art model in 37%. The most used established architecture was UNet (14%). The median performance for the most utilised evaluation metrics was Dice of 0.89 (range .49-.99), AUC of 0.903 (range 1.00-0.61) and Accuracy of 89.4 (range 70.2-100). Of the 77 studies that externally validated their results and allowed for direct comparison, performance on average decreased by 6% at external validation (range increase of 4% to decrease 44%).

CONCLUSION

This systematic review has surveyed the major advances in AI as applied to clinical radiology.

KEY POINTS

• While there are many papers reporting expert-level results by using deep learning in radiology, most apply only a narrow range of techniques to a narrow selection of use cases. • The literature is dominated by retrospective cohort studies with limited external validation with high potential for bias. • The recent advent of AI extensions to systematic reporting guidelines and prospective trial registration along with a focus on external validation and explanations show potential for translation of the hype surrounding AI from code to clinic.

External Validation of Deep Learning Algorithms for Radiologic Diagnosis: A Systematic Review

Purpose

To assess generalizability of published deep learning (DL) algorithms for radiologic diagnosis.

Materials and Methods

In this systematic review, the PubMed database was searched for peer-reviewed studies of DL algorithms for image-based radiologic diagnosis that included external validation, published from January 1, 2015, through April 1, 2021. Studies using nonimaging features or incorporating non-DL methods for feature extraction or classification were excluded. Two reviewers independently evaluated studies for inclusion, and any discrepancies were resolved by consensus. Internal and external performance measures and pertinent study characteristics were extracted, and relationships among these data were examined using nonparametric statistics.

Results

Eighty-three studies reporting 86 algorithms were included. The vast majority (70 of 86, 81%) reported at least some decrease in external performance compared with internal performance, with nearly half (42 of 86, 49%) reporting at least a modest decrease (≥0.05 on the unit scale) and nearly a quarter (21 of 86, 24%) reporting a substantial decrease (≥0.10 on the unit scale). No study characteristics were found to be associated with the difference between internal and external performance.

Conclusion

Among published external validation studies of DL algorithms for image-based radiologic diagnosis, the vast majority demonstrated diminished algorithm performance on the external dataset, with some reporting a substantial performance decrease.Keywords: Meta-Analysis, Computer Applications-Detection/Diagnosis, Neural Networks, Computer Applications-General (Informatics), Epidemiology, Technology Assessment, Diagnosis, Informatics Supplemental material is available for this article. © RSNA, 2022.

Exploratory data analysis for pre and post 24/7/365 attending radiologist coverage support in an emergency department: fundamentals of data science

OBJECTIVE

To present a detailed exploratory data analysis for critically investigating the patterns in medical doctor (MD) to disposition time, pre and post 24/7/365 attending radiologist coverage, for patients presenting to an emergency department (ED).

MATERIALS AND METHODS

The process involved presenting several modeling techniques. To share an understanding of concepts and techniques, we used proportions, medians, and means, Mann-Whitney U test, Kaplan-Meier's (KM) survival analysis, linear and log-linear regression, log-ranked test, Cox proportional hazards model, Weibull parametric survival models and tertile analysis. Retrospective chart review was conducted to obtain a data set which was used to determine the trends in MD to disposition time. Data comprised of patients who had visited the emergency department (ED) during two distinct time periods and whose imaging studies were read by an attending emergency and trauma radiologist.

RESULTS

Median provided more insight into the data as compared with the mean. The Mann-Whitney U test was appropriate to evaluate MD to disposition time, but provided limited information. The Kaplan-Meier (KM) was able to offer more insight into the data since it did not assume an underlying model and that is the reason why it was appropriate. However, KM had limited ability to handle measured confounders and was unable to describe the magnitude of difference between curves. The Cox proportional hazards semi-parametric model or some other parametric model such as the Weibull could handle multiple measured confounders and described the magnitude of difference between two (survival) groups in the data set. However, both methods assumed underlying models that may not apply to the data set such as the one used in this study. Linear regression was unlikely to be appropriate due to the shape of survival time distributions, but log transforming the outcome could address the distribution issue. Nearly all the results of the KM subgroup analyses were consistent with the results of the log-transformed linear regression subgroup analyses and the interpretation of the results was the same for both.

CONCLUSION

Different statistical procedures may be applied to conduct exploratory subgroup analysis for a data set from a pre and post 24/7/365 attending coverage model. This could guide potential areas of further research to compare trends in MD to disposition time in ED. Pattern analysis provides evidence for various stakeholders to rethink the discourse about trends in MD to disposition time, pre and post 24/7/365 attending coverage. Graphical Illustration: The role of Emergency and Trauma Radiology in an Emergency Department.

[Critical reading of analytical observational studies]

Analytical observational studies provide very important information about real-life clinical practice and the natural history of diseases and can suggest causality. Furthermore, they are very common in scientific journals. The aim of this article is to review the main concepts necessary for the critical reading of articles about radiological studies with observational designs. It reviews the characteristics that case-control and cohort studies must have to ensure high quality. It explains a method of critical reading that involves checking the attributes that should be evaluated in each type of article using a structured list of specific questions. It underlines the main characteristics that confer credibility and confidence on the article evaluated. Readers are provided with tools for the critical analysis of the observational studies published in scientific journals.

[Introduction to critical reading of articles: study design and biases]

The critical evaluation of an article enables professionals to make good use of the new information and therefore has direct repercussions for the benefit of our patients. Before undertaking a detailed critical reading of the chosen article, we need to consider whether the study used the most appropriate design for the question it aimed to answer (i.e., whether the level of evidence is adequate). To do this, we need to know how to classify studies in function of their design (descriptive or analytical; prospective or retrospective; cross-sectional or longitudinal) as well as their correlation with the levels of evidence. In critical reading it is also important to know the main systematic errors or biases that can affect a study. Biases can appear in any phase of a study; they can affect the sample, the development of the study, or the measurement of the results.

Bias in Research Studies

Bias is a form of systematic error that can affect scientific investigations and distort the measurement process. A biased study loses validity in relation to the degree of the bias. While some study designs are more prone to bias, its presence is universal. It is difficult or even impossible to completely eliminate bias. In the process of attempting to do so, new bias may be introduced or a study may be rendered less generalizable. Therefore, the goals are to minimize bias and for both investigators and readers to comprehend its residual effects, limiting misinterpretation and misuse of data. Numerous forms of bias have been described, and the terminology can be confusing, overlapping, and specific to a medical specialty. Much of the terminology is drawn from the epidemiology literature and may not be common parlance for radiologists. In this review, various types of bias are discussed, with emphasis on the radiology literature, and common study designs in which bias occurs are presented.

Critically assessing the radiology literature

To practice evidence-based radiology, knowledge of how to critically assess the literature is vital. This article outlines how to evaluate the radiology literature by asking these questions: Is it true?, Is it relevant?, and Is it sufficient? Clinical examples are used to explain several important causes of bias, as well as to clarify how these biases can affect the relevance of a particular study to a given clinical situation. Finally, discussion centers on the strength of evidence for both positive and negative findings in a research study.

Technology Assessment for Radiologists

Health technology assessment is the systematic and quantitative evaluation of the safety, efficacy, and cost of health care interventions. This article outlines aspects of technology assessment of diagnostic imaging. First, it presents a conceptual framework of a hierarchy of levels of efficacy that should guide thinking about imaging test evaluation. In particular, the framework shows how the question answered by most evaluations of imaging tests, "How well does this test distinguish disease from the nondiseased state?" relates to the fundamental questions for all health technology assessment, "How much does this intervention improve the health of people?" and "What is the cost of that improvement?" Second, it describes decision analysis and cost-effectiveness analysis, which are quantitative modeling techniques usually used to answer the two core questions for imaging. Third, it outlines design and operational considerations that are vital if researchers who are conducting an experimental study are to make a quality contribution to technology assessment, either directly through their findings or as an input into decision analyses. Finally, it includes a separate discussion of screening--that is, the application of diagnostic tests to nonsymptomatic populations--because the requirements for good screening tests are different from those for diagnostic tests of symptomatic patients and because the appropriate evaluation methods also differ.