Do Reader Characteristics Affect Diagnostic Efficacy in Screening Mammography? A Systematic Review

To examine reader characteristics associated with diagnostic efficacy in the interpretation of screening mammograms. A systematic search of the literature was conducted using databases such as Cochrane, Scopus, Medline, Embase, Web of Science, and PubMed. Search terms were combined with "AND" or "OR" and included: "Radiologist's characteristics AND performance"; "radiologist experience AND screening mammography"; "annual volume read AND diagnostic efficacy"; "screening mammography performance OR diagnostic efficacy". Studies were included if they assessed reader performance in screening mammography interpretation, breast readers, used a reference standard to assess the performance, and were published in the English language. Twenty-eight studies were reviewed. Increasing reader's age was associated with lower false positive rates. No association was found between gender and performance. Half of the studies showed no association between years of reading mammograms and performance. Most studies showed that high reading volume was more likely to be associated with increased sensitivity, cancer detection rates (CDR), lower recall rate, and lower false positive rates. Inconsistent associations were found between fellowship training in breast imaging and reader performance. Specialization in breast imaging was associated with better CDR, sensitivity, and specificity. Limited studies were available to establish the association between performance and factors such as time spent in breast imaging (n = 2), screening focus (n = 1), formal rotation in mammography (n = 1), owner of practice (n = 1), and practice type (n = 1). No individual characteristics is associated with versatility in diagnostic efficacy, albeit reading volume and specialization in breast imaging appear to be associated with with increased sensitivity and CDR without significantly affecting other performance metrics.

Digital mammographic interpretation by UK radiographer mammographers: A JAFROC analysis of observer performance

INTRODUCTION

Radiologists utilise mammography test sets to bench mark their performance against recognised standards. Using a validated test set, this study compares the performance of radiographer readers against previous test results for radiologists.

METHODS

Under similar test conditions radiographer readers were given an established test set of 60 mammograms and tasked to identify breast cancer, they were measured against their ability to identify, locate and give a confidence level for cancer being present on a standard set of mammographic images. The results were then compared to previously published results for radiologists for similar or the same test sets.

RESULTS

The 10 radiographer readers demonstrated similar results to radiologists and for lesion sensitivity were the highest scoring group. The study group score a sensitivity of 83; a specificity of 69.3 and lesion sensitivity of 74.8 with ROC and JAFROC scores of 0.86 and 0.74 respectively.

CONCLUSION

Under test conditions radiographers are able to identify and accurately locate breast cancer in a range of complex mammographic backgrounds.

IMPLICATIONS FOR PRACTICE

The study was performed under experimental conditions with results comparable to breast radiologists under similar conditions, translation of these findings into clinical practice will help address access and capacity issues in the timely identification and diagnosis of breast cancer.

Receiver Operating Characteristic (ROC) Analysis of Image Search-and-Localize Tasks

RATIONALE AND OBJECTIVES

Receiver operating characteristic (ROC) analysis for the common image search-and-localize task, in which readers search an image for lesion or lesions not knowing a priori any exists, has been studied for over four decades. However, a satisfactory solution seems elusive.

MATERIALS AND METHODS

We show that the ROC curve predictive of clinical outcomes where readers are penalized appropriately for not correctly localizing known lesions cannot be obtained because it is a missing data problem. Further, this ROC curve is between the case-based ROC curve where readers are not penalized and the lesion-based ROC curve where penalty applies. Moreover, the lesion-based ROC curve is the LROC curve proposed by Starr et al. We show maximum-likelihood (ML) estimation of the LROC curve, validation of this procedure with Monte Carlo simulations, and its application to reader ROC datasets.

RESULTS

Monte Carlo simulations validated ML estimation of area under the LROC curve (AUC) and its variance. Example applications showed that ML estimate of LROC curve fits experimental datasets.

CONCLUSION

The ROC curve predictive of clinical performance cannot be estimated from reader ROC data alone because it is a missing data problem, and is between the case-based ROC curve where readers are not penalized for not correctly identifying known lesions and the lesion-based ROC curve where penalty applies. The lesion-based ROC curve is the LROC curve proposed by Starr et al. and can be estimated via ML estimation.

Evaluation of Pseudoreader Study Designs to Estimate Observer Performance Results as an Alternative to Fully Crossed, Multireader, Multicase Studies

RATIONALE AND OBJECTIVES

To examine the ability of a pseudoreader study design to estimate the observer performance obtained using a traditional fully crossed, multireader, multicase (MRMC) study.

MATERIALS AND METHODS

A 10-reader MRMC study with 20 computed tomography datasets was designed to measure observer performance on four novel noise reduction methods. This study served as the foundation for the empirical evaluation of three different pseudoreader designs, each of which used a similar bootstrap approach for generating 2000 realizations from the fully crossed study. Our three approaches to generating a pseudoreader varied in the degree to which reader performance was matched and integrated into the pseudoreader design. One randomly selected simulation was selected as a "mock study" to represent a hypothetical, prospective implementation of the design.

RESULTS

Using the traditional fully crossed design, figures of merit) (95% CIs) for the four noise reductions methods were 68.2 (55.5-81.0), 69.6 (58.4-80.8), 70.8 (60.2-81.4), and 70.9 (60.4-81.3), respectively. When radiologists' performances on the fourth noise reduction method were used to pair readers during the mock study, there was strong agreement in the estimated figures of merits with estimates using the pseudoreader design being within ±3% of the fully crossed design.

CONCLUSION

Fully crossed MRMC studies require significant investment in resources and time, often resulting in delayed implementation or minimal human testing before dissemination. The pseudoreader approach accelerates study conduct by combining readers judiciously and was found to provide comparable results to the traditional fully crossed design by making strong assumptions about exchangeability of the readers.

Chest X-ray Interpretation by Radiographers Is Not Inferior to Radiologists: A Multireader, Multicase Comparison Using JAFROC (Jack-knife Alternative Free-response Receiver Operating Characteristics) Analysis

RATIONALE AND OBJECTIVES

Chest X-rays (CXR) are one of the most frequently requested imaging examinations and are fundamental to many patient pathways. The aim of this study was to investigate the diagnostic accuracy of CXR interpretation by reporting radiographers (technologists).

METHODS

A cohort of consultant radiologists (n = 10) and reporting radiographers (technologists; n = 11) interpreted a bank (n = 106) of adult CXRs that contained a range of pathologies. Jack-knife alternate free-response receiver operating characteristic (JAFROC) methodology was used to determine the performance of the observers (JAFROC v4.2). A noninferiority approach was used, with a predefined margin of clinical insignificance of 10% of average consultant radiologist diagnostic accuracy.

RESULTS

The diagnostic accuracy of the reporting radiographers (figure of merit = 0.828, 95% confidence interval 0.808-0.847) was noninferior to the consultant radiologists (figure of merit = 0.788, 95% confidence interval 0.766-0.811), P < .0001.

CONCLUSIONS

With appropriate postgraduate education, reporting radiographers are able to interpret CXRs at a level comparable to consultant radiologists.

Missed Lung Cancer

The chest radiograph is one of the most commonly used imaging studies and is the modality of choice for initial evaluation of many common clinical scenarios. Over the last two decades, chest computed tomography has been increasingly used for a wide variety of indications, including respiratory illnesses, trauma, oncologic staging, and more recently lung cancer screening. Diagnostic radiologists should be familiar with the common causes of missed lung cancers on imaging studies in order to avoid detection and interpretation errors. Failure to detect these lesions can potentially have serious implications for both patients as well as the interpreting radiologist.

Acid tolerance properties of dental biofilms in vivo

BACKGROUND

The ecological plaque hypothesis explains caries development as the result of the enrichment of acid tolerant bacteria in dental biofilms in response to prolonged periods of low pH. Acid production by an acid tolerant microflora causes demineralisation of tooth enamel and thus, individuals with a greater proportion of acid tolerant bacteria would be expected to be more prone to caries development. Biofilm acid tolerance could therefore be a possible biomarker for caries prediction. However, little is known about the stability of biofilm acid tolerance over time in vivo or the distribution throughout the oral cavity. Therefore the aim of this study was to assess intra-individual differences in biofilm acid-tolerance between different tooth surfaces and inter-individual variation as well as stability of acid tolerance over time.

RESULTS

The majority of the adolescents showed low scores for biofilm acid tolerance. In 14 of 20 individuals no differences were seen between the three tooth sites examined. In the remaining six, acid-tolerance at the premolar site differed from one of the other sites. At 51 of 60 tooth sites, acid-tolerance at baseline was unchanged after 1 month. However, acid tolerance values changed over a 1-year period in 50% of the individuals.

CONCLUSIONS

Biofilm acid tolerance showed short-term stability and low variation between different sites in the same individual suggesting that the acid tolerance could be a promising biological biomarker candidate for caries prediction. Further evaluation is however needed and prospective clinical trials are called for to evaluate the diagnostic accuracy.

Diagnostic Accuracy and Visual Search Efficiency: Single 8 MP vs. Dual 5 MP Displays

This study compared a single 8 MP vs. dual 5 MP displays for diagnostic accuracy, reading time, number of times the readers zoomed/panned images, and visual search. Six radiologists viewed 60 mammographic cases, once on each display. A sub-set of 15 cases was viewed in a secondary study using eye-tracking. For viewing time, there was significant difference (F = 13.901, p = 0.0002), with 8 MP taking less time (62.04 vs. 68.99 s). There was no significant difference (F = 0.254, p = 0.6145) in zoom/pan use (1.94 vs. 1.89). Total number of fixations was significantly (F = 4.073, p = 0.0466) lower with 8 MP (134.47 vs. 154.29). Number of times readers scanned between images was significantly fewer (F = 10.305, p = 0.0018) with 8 MP (6.83 vs. 8.22). Time to first fixate lesion did not differ (F = 0.126, p = 0.7240). It did not take any longer to detect the lesion as a function of the display configuration. Total time spent on lesion did not differ (F = 0.097, p = 0.7567) (8.59 vs. 8.39). Overall, the single 8 MP display yielded the same diagnostic accuracy as the dual 5 MP displays. The lower resolution did not appear to influence the readers' ability to detect and view the lesion details, as the eye-position study showed no differences in time to first fixate or total time on the lesions. Nor did the lower resolution result in significant differences in the amount of zooming and panning that the readers did while viewing the cases.

Reporting instructions significantly impact false positive rates when reading chest radiographs

OBJECTIVES

To determine the impact of specific reporting tasks on the performance of radiologists when reading chest radiographs.

METHODS

Ten experienced radiologists read a set of 40 postero-anterior (PA) chest radiographs: 21 nodule free and 19 with a proven solitary nodule. There were two reporting conditions: an unframed task (UFT) to report any abnormality and a framed task (FT) reporting only lung nodule/s. Jackknife free-response operating characteristic (JAFROC) figure of merit (FOM), specificity, location sensitivity and number of true positive (TP), false positive (FP), true negative (TN) and false negative (FN) decisions were used for analysis.

RESULTS

JAFROC FOM for tasks showed a significant reduction in performance for framed tasks (P = 0.006) and an associated decrease in specificity (P = 0.011) but no alteration to the location sensitivity score. There was a significant increase in number of FP decisions made during framed versus unframed tasks for nodule-containing (P = 0.005) and nodule-free (P = 0.011) chest radiographs. No significant differences in TP were recorded.

CONCLUSIONS

Radiologists report more FP decisions when given specific reporting instructions to search for nodules on chest radiographs. The relevance of clinical history supplied to radiologists is called into question and may induce a negative effect.

KEY POINTS

• Framed reporting tasks increases false positive rates when searching for pulmonary nodules • False positive results were observed in both nodule-containing and nodule-free cases • Radiologist's decision-making may be influenced by clinical history in thoracic imaging.

Expert witness blinding strategies to mitigate bias in radiology malpractice cases: a comprehensive review of the literature

Like all physicians, radiologists in the United States are subject to frequent and costly medical malpractice claims. Legal scholars and physicians concur that the US civil justice system is neither precise nor accurate in determining whether malpractice has truly occurred in cases in which claims are made. Sometimes, this inaccuracy is driven by biases inherent in medical expert-witness opinions. For example, expert-witness testimony involving "missed" radiology findings can be negatively affected by several cognitive biases, such as contextual bias, hindsight bias, and outcome bias. Biases inherent in the US legal system, such as selection bias, compensation bias, and affiliation bias, also play important roles. Fortunately, many of these biases can be significantly mitigated or eliminated through the use of appropriate blinding techniques. This paper reviews the major works on expert-witness blinding in the legal scholarship and the radiology professional literature. Its purpose is to acquaint the reader with the evidence that unblinded expert-witness testimony is tainted by multiple sources of bias and to examine proposed strategies for addressing these biases through blinding.

Spectrum of diagnostic errors in radiology

Diagnostic errors are important in all branches of medicine because they are an indication of poor patient care. Since the early 1970s, physicians have been subjected to an increasing number of medical malpractice claims. Radiology is one of the specialties most liable to claims of medical negligence. Most often, a plaintiff's complaint against a radiologist will focus on a failure to diagnose. The etiology of radiological error is multi-factorial. Errors fall into recurrent patterns. Errors arise from poor technique, failures of perception, lack of knowledge and misjudgments. The work of diagnostic radiology consists of the complete detection of all abnormalities in an imaging examination and their accurate diagnosis. Every radiologist should understand the sources of error in diagnostic radiology as well as the elements of negligence that form the basis of malpractice litigation. Error traps need to be uncovered and highlighted, in order to prevent repetition of the same mistakes. This article focuses on the spectrum of diagnostic errors in radiology, including a classification of the errors, and stresses the malpractice issues in mammography, chest radiology and obstetric sonography. Missed fractures in emergency and communication issues between radiologists and physicians are also discussed.