Clinical relevance of the ROC and free-response paradigms for comparing imaging system efficacies

Familiarity, confidence and preference of artificial intelligence feedback and prompts by Australian breast cancer screening readers

Objectives This study explored the familiarity, perceptions and confidence of Australian radiology clinicians involved in reading screening mammograms, regarding artificial intelligence (AI) applications in breast cancer detection. Methods Sixty-five radiologists, breast physicians and radiology trainees participated in an online survey that consisted of 23 multiple choice questions asking about their experience and familiarity with AI products. Furthermore, the survey asked about their confidence in using AI outputs and their preference for AI modes applied in a breast screening context. Participants' responses to questions were compared using Pearson's χ 2 test. Bonferroni-adjusted significance tests were used for pairwise comparisons. Results Fifty-five percent of respondents had experience with AI in their workplaces, with automatic density measurement powered by machine learning being the most familiar AI product (69.4%). The top AI outputs with the highest ranks of perceived confidence were 'Displaying suspicious areas on mammograms with the percentage of cancer possibility' (67.8%) and 'Automatic mammogram classification (normal, benign, cancer, uncertain)' (64.6%). Radiology and breast physicians preferred using AI as second-reader mode (75.4% saying 'somewhat happy' to 'extremely happy') over triage (47.7%), pre-screening and first-reader modes (both with 26.2%) (P  < 0.001). Conclusion The majority of screen readers expressed increased confidence in utilising AI for highlighting suspicious areas on mammograms and for automatically classifying mammograms. They considered AI as an optimal second-reader mode being the most ideal use in a screening program. The findings provide valuable insights into the familiarities and expectations of radiologists and breast clinicians for the AI products that can enhance the effectiveness of the breast cancer screening programs, benefitting both healthcare professionals and patients alike.

Radiologist Self-training: a Study of Cancer Detection when Reading Mammograms at Work Clinics or Workshops

Provision of online and remote specialist education and general continued professional education in medicine is a growing field. For radiology specifically, the ability to access web-based platforms that house high resolution medical images, and the high fidelity of simulated activities is increasingly growing due to positive changes in technology. This study investigates the differences in providing a self-directed specialist radiology education system in two modes: at clinics and in-person workshops. 335 Australian radiologists completed 562 readings of mammogram test sets through the web-based interactive BREAST platform with 325 at conference workshops and 237 at their workplaces. They engaged with test sets with each comprising of 60 mammogram cases (20 cancer and 40 normal). Radiologists marked the location of any cancers and had their performance measured via 5 metrics of diagnostic accuracy. Results show that the location of engagement with BREAST did not yield any significant difference in the performances of all radiologists and the same radiologists between two reading modes (P > 0.05). Radiologists who read screening mammograms for BreastScreen Australia performed better when they completed the test sets at designated workshops (P < 0.05), as was also the case for radiologists who read > 100 cases per week (P < 0.05). In contrast, radiologists who read less mammograms frequently recorded better performances in specificity and JAFROC at clinics (P < 0.05). Findings show that remotely accessed online education for specialised training and core skills building in radiology can provide a similar learning opportunity for breast radiologists when compared to on-site dedicated workshops at scientific meetings. For readers with high volumes of mammograms, a workshop setting may provide a superior experience while clinic setting is more helpful to less experienced readers.

EXTRACOLONIC FINDINGS-IDENTIFICATION AT LOW-DOSE CTC

In contrast to optical colonoscopy, computed tomography colonography (CTC) has the ability to reveal pathology outside of the colon. While identification of colorectal lesions at CTC requires only limited radiation dose, the detection of abnormalities in extracolonic soft tissue requires more radiation. The purpose of this study was to investigate the influence of ultra-low-dose (ULD) CTC on the detection and characterisation of extracolonic findings. In a prospective study 49 patients with colorectal symptoms were examined with CTC adding a ULD series (mean effective dose 0.9 ± 0.4 mSv) to the normal unenhanced standard dose (SD) series (mean effective dose 3.6 ± 1.2 mSv). Five radiologists individually and blindly evaluated the ULD, followed by evaluation of the SD after ≥9 weeks (median 35 weeks). A ViewDEX-based examination protocol was used, including a confidence scale and a graded assessment of need for follow-up according to the CTC Reporting and Data System (C-RADS E0-E4). The reference findings comprised the combined information from CTC (ULD, SD and contrast-enhanced CTC series) and a 4-year radiological and clinical follow-up. For the overall detection of reference findings (E2-E4) we found a statistically significant difference in favour of SD. This, however, was not the case when looking at classification of possibly important/important reference findings (E3-E4). Our results suggest that CTC with ULD (0.9 mSv) is comparable to SD (3.6 mSv) for identification of clinically relevant extracolonic pathology, but there is a large inter-observer variability.

Improving radiologist's ability in identifying particular abnormal lesions on mammograms through training test set with immediate feedback

It has been shown that there are differences in diagnostic accuracy of cancer detection on mammograms, from below 50% in developing countries to over 80% in developed world. One previous study reported that radiologists from a population in Asia displayed a low mammographic cancer detection of 48% compared with over 80% in developed countries, and more importantly, that most lesions missed by these radiologists were spiculated masses or stellate lesions. The aim of this study was to explore the performance of radiologists after undertaking a training test set which had been designed to improve the capability in detecting a specific type of cancers on mammograms. Twenty-five radiologists read two sets of 60 mammograms in a standardized mammogram reading room. The first test set focused on stellate or spiculated masses. When radiologists completed the first set, the system displayed immediate feedback to the readers comparing their performances in each case with the truth of cancer cases and cancer types so that the readers could identify individual-based errors. Later radiologists were asked to read the second set of mammograms which contained different types of cancers including stellate/spiculated masses, asymmetric density, calcification, discrete mass and architectural distortion. Case sensitivity, lesion sensitivity, specificity, receiver operating characteristics (ROC) and Jackknife alternative free-response receiver operating characteristics (JAFROC) were calculated for each participant and their diagnostic accuracy was compared between two sessions. Results showed significant improvement among radiologists in case sensitivity (+ 11.4%; P < 0.05), lesion sensitivity (+ 18.7%; P < 0.01) and JAFROC (+ 11%; P < 0.01) in the second set compared with the first set. The increase in diagnostic accuracy was also recorded in the detection of stellate/spiculated mass (+ 20.6%; P < 0.05). This indicated that the performance of radiologists in detecting malignant lesions on mammograms can be improved if an appropriate training intervention is applied after the readers' weakness and strength are identified.

Evaluation of an automatic method for detection of defects in linear and curvilinear ultrasound transducers

PURPOSE

The high incidence of defective ultrasound transducers in clinical practice has been shown in several studies. Recently, a novel method using only stored images for automatic detection of defective transducers was presented. The method makes it possible to remotely monitor many transducers at the same time and send a notification when a defective transducer is found. The purpose of the present study was to evaluate the novel method and assess how well it performs when compared to an established method as reference.

METHODS

To evaluate the novel method, in-air images were collected from 81 transducers in radiologic departments in nine hospitals. Two observers assessed the in-air images and marked the defects. Receiver operating characteristic (ROC)- and alternative free response receiver operating characteristic (AFROC)-curves and their figures of merit (FOM) were calculated for the novel method, using marked defects in the in-air images as reference truth.

RESULTS

The area under the ROC curve was 0.88 (SD 0.06), and the AFROC FOM was 0.71 (SE 0.07).

CONCLUSION

The result shows that the novel method has a good agreement with the in-air method for detecting defects in ultrasound systems. This indicates that the novel method could be a complement to the normal quality control for early, and automatic detection of defects.

Impact of time of day on radiology image interpretations

AIM

To examine the impact of the time of day on radiologists' mammography reading performance.

MATERIALS AND METHODS

Retrospective mammographic reading assessment data were collected from the BreastScreen Reader Assessment Strategy database and included timestamps of the readings and reader-specific demographic data of 197 radiologists. The radiologists performed the readings in a workshop setting with test case sets enriched with malignancies (one-third of cases were malignant). The collected data were evaluated with an analysis of covariance to determine whether time of day influenced radiologists' specificity, lesion sensitivity or the jackknife alternative free-response receiver operating characteristic (JAFROC).

RESULTS

After adjusting for radiologist experience and fellowship, specificity varied significantly by time of day (p=0.027), but there was no evidence of any significant impact on lesion sensitivity (p=0.441) or JAFROC (p=0.120). The collected data demonstrated that specificity during the late morning (10.00-12.00) was 71.7%; this was significantly lower than in the early morning (08.00-10.00) and mid-afternoon (14.00-16.00), which were 82.74% (p=0.003) and 81.39% (p=0.031), respectively. Specificity during the late afternoon (16.00-18.00) was 73.95%; this was significantly lower than in the early morning (08.00-10.00) and mid-afternoon (14.00-16.00), which were 82.74% (p=0.003) and 81.39% (p=0.031), respectively.

CONCLUSION

The results indicated that the time of day may influence radiologists' performance, specifically their ability to identify normal images correctly.

Mammographic detection of breast cancer in a non-screening country

OBJECTIVE:

To compare the diagnostic accuracy between radiologists' from a country with and without breast cancer screening.

METHODS:

All participating radiologists gave informed consent. A test-set involving 60 mammographic cases (20 cancer and 40 non-cancer) were read by 11 radiologists from a non-screening (NS) country during a workshop in July 2016. 52 radiologists from a screening country read the same test-set at the Royal Australian and New Zealand College of Radiologists' meetings in July 2015. The screening radiologists were classified into two groups: those with less than or equal to 5 years of experience; those with more than 5 years of experience, and each group was compared to the group of NS radiologists. A Kruskal-Wallis test followed by post-hoc multiple comparisons test were used to compare measures of diagnostic accuracy among the reader groups.

RESULTS:

The diagnostic accuracy of the NS radiologists was significantly lower in terms of sensitivity [mean = 54.0; 95% confidence interval (CI) (40.0-67.0)], location sensitivity [mean = 26.0; 95% CI (16.0-37.0)], receive roperating characteristic area under curve [mean = 73.0; 95% CI (66.5-81.0)] and Jackknifefree-response receiver operating characteristics figure-of-merit [mean = 45.0; 95% CI (40.0-50.0)] when compared with the less and more experienced screening radiologists, whilst no difference in specificity [mean = 75.0; 95% CI (70.0- 81.0)] was found. No significant differences in all measured diagnostic accuracy were found between the two groups of screening radiologists.

CONCLUSION:

The mammographic performance of a group of radiologists from a country without screening program was suboptimal compared with radiologists from Australia.

ADVANCES IN KNOWLEDGE:

Identifying mammographic performance in developing countries is required to optimize breast cancer diagnosis.

Vessel segmentation and microaneurysm detection using discriminative dictionary learning and sparse representation

Diabetic retinopathy (DR) is a major cause of visual impairment, and the analysis of retinal image can assist patients to take action earlier when it is more likely to be effective. The accurate segmentation of blood vessels in the retinal image can diagnose DR directly. In this paper, a novel scheme for blood vessel segmentation based on discriminative dictionary learning (DDL) and sparse representation has been proposed. The proposed system yields a strong representation which contains the semantic concept of the image. To extract blood vessel, two separate dictionaries, for vessel and non-vessel, capable of providing reconstructive and discriminative information of the retinal image are learned. In the test step, an unseen retinal image is divided into overlapping patches and classified to vessel and non-vessel patches. Then, a voting scheme is applied to generate the binary vessel map. The proposed vessel segmentation method can achieve the accuracy of 95% and a sensitivity of 75% in the same range of specificity 97% on two public datasets. The results show that the proposed method can achieve comparable results to existing methods and decrease false positive vessels in abnormal retinal images with pathological regions. Microaneurysm (MA) is the earliest sign of DR that appears as a small red dot on the surface of the retina. Despite several attempts to develop automated MA detection systems, it is still a challenging problem. In this paper, a method for MA detection, which is similar to our vessel segmentation approach, is proposed. In our method, a candidate detection algorithm based on the Morlet wavelet is applied to identify all possible MA candidates. In the next step, two discriminative dictionaries with the ability to distinguish MA from non-MA object are learned. These dictionaries are then used to classify the detected candidate objects. The evaluations indicate that the proposed MA detection method achieves higher average sensitivity about 2-15%, compared to existing methods.

Applications of ROC analysis in medical research: recent developments and future directions

With the growing focus on comparative effectiveness research and personalized medicine, receiver-operating characteristic analysis can continue to play an important role in health care decision making. Specific applications of receiver-operating characteristic analysis include predictive model assessment and validation, biomarker diagnostics, responder analysis in patient-reported outcomes, and comparison of alternative treatment options. The authors present a survey of the potential applications of the method and briefly review several relevant extensions. Given the level of attention paid to biomarker validation, personalized medicine and comparative effectiveness research, it is highly likely that the receiver-operating characteristic analysis will remain an important visual and analytic tool for medical research and evidence-based medicine in the foreseeable future.

Breast imaging: understanding how accuracy is measured when lesions are the unit of analysis

Medical imaging tests of breast cancer patients can be used to detect and provide information on the location of multiple malignant lesions within a patient. Within this context, it is often the case that one needs to evaluate the accuracy of an imaging test for finding the multiple lesions in a patient rather than simply detecting that a patient has disease. A natural way to approach this task is to estimate the accuracy of the test using a lesion-level analysis. Sensitivity, specificity, and receiver operating characteristic (ROC) curves are analytic measures that are frequently used to quantify the accuracy of medical tests. When the test or radiologist must first locate the lesions, however, it is not possible to directly estimate the specificity or an ROC curve keeping the individual lesions as the unit of analysis. The goal of this study is to demonstrate to clinicians conducting or reviewing studies evaluating breast imaging tests what measures of accuracy can and cannot be calculated in different types of studies and to describe in detail the difficulty with calculating specificity and ROC curves in a lesion-level analysis.