Educational interventions to improve screening mammography interpretation: a randomized controlled trial

Does the availability of prior mammograms improve radiologists' observer performance?-a scoping review

Objective

The objective of this review was to examine the impact of previous mammogram availability on radiologists' performance from screening populations and experimental studies.

Materials and Methods

A search of the literature was conducted using five databases: MEDLINE, PubMed, Web of Science, ScienceDirect, and CINAHL as well as Google and reference lists of articles. Keywords were combined with "AND" or "OR" or "WITH" and included "prior mammograms, diagnostic performance, initial images, diagnostic efficacy, subsequent images, previous imaging, and radiologist's performance". Studies that assessed the impact of previous mammogram availability on radiologists' performance were reviewed. The Standard for Reporting Diagnostic Accuracy guidelines was used to critically appraise individual sources of evidence.

Results

A total of 15 articles were reviewed. The sample of mammogram cases used across these studies varied from 36 to 1,208,051. Prior mammograms did not affect sensitivity [with priors: 62-86% (mean = 73.3%); without priors: 69.4-87.4% (mean = 75.8%)] and cancer detection rate, but increased specificity [with priors: 72-96% (mean = 87.5%); without priors: 63-87% (mean = 80.5%)] and reduced false-positive rates [with priors: 3.7 to 36% (mean = 19.9%); without priors 13.3-49% (mean = 31.4%)], recall rates [with priors: 3.8-57% (mean = 26.6%); without priors: [4.9%-67.5% (mean = 37.9%)], and abnormal interpretation rate decreased by 4% with priors. Evidence for the associations between the availability of prior mammograms and positive-predictive value, area under the curve (AUC) from the receiver operating characteristic curve (ROC) and localisation ROC AUC, and positive-predictive value of recall is limited and unclear.

Conclusion

Availability of prior mammograms reduces recall rates, false-positive rates, abnormal interpretation rates, and increases specificity without affecting sensitivity and cancer detection rate.

Experiences of Women Who Refuse Recall for Further Investigation of Abnormal Screening Mammography: A Qualitative Study

Breast cancer has the highest incidence among all cancers for women in Taiwan. The current screening policy in Taiwan provides biennial mammogram tests for all women aged 45 to 69 years. A recommendation for further investigation is sent via post to women with a BI-RADS result of 0. The proportion of women who followed-up with a recall request for further investigation after an abnormal mammogram has been below 92.5% in recent years. Therefore, we aimed to explore the experiences of these women who refused recall for further investigation despite an abnormal mammogram. Purposive sampling was conducted on 13 women who refused recall for further examination of abnormal screening mammograms. Data collection included inductive, in-depth interviews or telephone interviews. A content analysis was applied. Three themes were identified: (1) negative screening experiences, (2) struggling with ’to go or not to go’, and (3) rationalizing without a follow-up examination. The first theme included three subthemes: (1) pain of examination, (2) the inconvenience of medical treatment; waiting, and (3) dissatisfaction with having to pay for further examination. The second major theme included three subthemes: (1) perceiving one’s susceptibility to breast cancer as very low, (2) questioning the accuracy of the results, and (3) procrastinating with a “head-in-the-sand” mentality. The third major theme included two subthemes: (1) fatalism and (2) paying attention to self-cultivation. In conclusion, the findings provide important information to healthcare providers involved in case management related to the actual living experiences of women with abnormal screening mammogram results and the additional education required to raise breast cancer awareness in the general public to achieve overall caring goals.

Incorporating Peer Learning Into Your Breast Imaging Practice

Traditional score-based peer review has come under scrutiny in recent years, as studies have demonstrated it to be generally ineffective at improving quality. Many practices and programs are transitioning to a peer learning model to replace or supplement traditional peer review. Peer learning differs from traditional score-based peer review in that the emphasis is on sharing learning opportunities and creating an environment that fosters discussion of errors in a nonpunitive forum with the goal of improved patient care. Creating a just culture is central to fostering successful peer learning. In a just culture, mistakes can be discussed without shame or fear of retribution and the focus is on systems improvement rather than individual blame. Peer learning, as it pertains to breast imaging, can occur in many forms and venues. Examples of the various formats in which peer learning can occur include through individual colleague interaction, as well as divisional, multidisciplinary, department-wide, and virtual conferences, and with the assistance of artificial intelligence. Incorporating peer learning into the practice of breast imaging aims to reduce delayed diagnoses of breast cancer and optimize patient care.

Pooling decisions decreases variation in response bias and accuracy

Decision makers in contexts as diverse as medical, judicial, and political decision making are known to differ substantially in response bias and accuracy, and these differences are a major factor undermining the reliability and fairness of the respective decision systems. Using theoretical modeling and empirical testing across five domains, we show that collective systems based on pooling decisions robustly overcome this important but as of now unresolved problem of experts' heterogeneity. In breast and skin cancer diagnostics and fingerprint analysis, we find that pooling the decisions of five experts reduces the variation in sensitivity among decision makers by 52%, 54%, and 41%, respectively. Similar reductions are achieved for specificity and response bias, and in other domains. Thus, although outcomes in individual decision systems are highly variable and at the mercy of individual decision makers, collective systems based on pooling decrease this variation, thereby promoting reliability, fairness, and possibly even trust.

•

Professional decision makers typically differ in their response bias and accuracy

•

Such differences undermine the reliability and fairness of decision systems

•

Pooling decisions reduces such variation in response bias and accuracy

•

This occurred in cancer diagnostics, fingerprint analysis, and forecasting

Speech-language therapists' perceptions of an eLearning program to support training in videofluoroscopic swallow studies

BACKGROUND

Speech-language therapists (SLTs) seek a range of educational opportunities for training in adult videofluoroscopic swallow studies (VFSS). However, variable training methods and/or unequal access to training can influence VFSS practice.

AIMS

To document current SLT needs and barriers to VFSS training and to determine if a new beginner-level VFSS eLearning program would assist to meet their training needs. The program incorporated multimedia modules on preparing, conducting, interpreting and reporting VFSS.

METHODS & PROCEDURES

SLTs with limited experience in adult VFSS completed surveys relating to VFSS training experience and barriers, and perceived changes in knowledge, skills and confidence on core VFSS module topics pre- (n = 36) and post- (n = 32) eLearning training.

OUTCOMES & RESULTS

Inconsistent access to VFSS training opportunities and time-related work pressures were reported as the greatest training barriers. SLTs viewed the eLearning program as a suitable option for VFSS training. Post-training, participants perceived they gained confidence, as well as improved knowledge and skills in all VFSS aspects along with generalised benefits for dysphagia management. SLTs indicated that key benefits of the eLearning program were its comprehensive content and self-directed learning with multimedia tools, which afforded theoretical and practical learning opportunities.

CONCLUSIONS & IMPLICATIONS

The eLearning program offered SLTs free access to beginner-level adult VFSS training, meeting many identified training needs and providing a foundation from which to develop further practical knowledge and skills within a VFSS clinic setting. What this paper adds What is already known on the subject SLTs demonstrate variable knowledge and skill in conducting and interpreting VFSS, which can impact dysphagia diagnosis and management. While access to VFSS training can be challenging, the barriers to training for SLTs have not been clearly documented. Research has confirmed that eLearning can be used effectively in healthcare education, and in some aspects of VFSS training; however, it is yet to be applied to address the broad range of VFSS training needs. What this paper adds to existing knowledge This study describes the SLT reported barriers to VFSS training which include limited access to formal and practical training, workload-related time pressures and the complexity of learning the VFSS skill set. The findings highlight that an eLearning program, was an accepted mode of learning for VFSS training. SLTs reported the online program met their learning needs by improving access to training, the multimedia program features supported their understanding of complex anatomical and physiological concepts, and training frameworks assisted their clinical reasoning and VFSS interpretation. What are the potential or actual clinical implications of this work? eLearning can assist in overcoming many VFSS training barriers identified by SLTs and the multimedia aspects of eLearning can effectively support VFSS beginner-level education to complement and expedite in-clinic practical training. Given that VFSS results inform decisions regarding commencement and progression of oral intake and swallow rehabilitation, enhanced VFSS training has the potential to positively influence dysphagia outcomes and quality of life.

Missed Breast Cancer: Effects of Subconscious Bias and Lesion Characteristics

Medical errors are a substantial cause of morbidity and mortality and the third leading cause of death in the United States. Errors resulting in missed breast cancer are the most common reason for medical malpractice lawsuits against all physicians. Missed breast cancers are breast malignancies that are detectable at retrospective review of a previously obtained mammogram that was prospectively reported as showing negative, benign, or probably benign findings. Investigators in prior studies have found that up to 35% of both interval cancers and screen-detected cancers could be classified as missed. As such, in conjunction with having awareness of the most common misleading appearances of breast cancer, it is important to understand the cognitive processes and unconscious biases that can impact image interpretation, thereby helping to decrease the number of missed breast cancers. The various cognitive processes that lead to unconscious bias in breast imaging, such as satisfaction of search, inattention blindness, hindsight, anchoring, premature closing, and satisfaction of reporting, are outlined in this pictorial review of missed breast cancers. In addition, strategies for reducing the rates of these missed cancers are highlighted. The most commonly missed and misinterpreted lesions, including stable lesions, benign-appearing masses, one-view findings, developing asymmetries, subtle calcifications, and architectural distortion, also are reviewed. This information will help illustrate why and how breast cancers are missed and aid in the development of appropriate minimization strategies in breast imaging. ^©RSNA, 2020.

The roles of clinical audit and test sets in promoting the quality of breast screening: a scoping review

Breast screening programmes enhance the probability of early breast cancer detection in many countries worldwide; however, the success of these efforts is highly dependent on the ability of breast screen readers to detect abnormalities in the screened population, which has low prevalence. Therefore, this task can be challenging. Clinical audit is a key quality assurance measure that aims to keep the screen reading performance within acceptable standards. Auditing, nonetheless, is a lengthy process, and its accuracy is dependent on available clinical data, which often can be limited. Mammographic standardised test sets are a different screen reading evaluation approach that provides participants with instant feedback based on a simulated environment. Although a test set provides unique evaluative qualities, its ability to represent clinical performance is debated. This article describes the distinctive roles of clinical audit and test sets in measuring and improving the quality of breast screening and highlights the relationship between test sets and clinical performance.

Deep learning can be used to train naïve, nonprofessional observers to detect diagnostic visual patterns of certain cancers in mammograms: a proof-of-principle study

The scientific, clinical, and pedagogical significance of devising methodologies to train nonprofessional subjects to recognize diagnostic visual patterns in medical images has been broadly recognized. However, systematic approaches to doing so remain poorly established. Using mammography as an exemplar case, we use a series of experiments to demonstrate that deep learning (DL) techniques can, in principle, be used to train naïve subjects to reliably detect certain diagnostic visual patterns of cancer in medical images. In the main experiment, subjects were required to learn to detect statistical visual patterns diagnostic of cancer in mammograms using only the mammograms and feedback provided following the subjects’ response. We found not only that the subjects learned to perform the task at statistically significant levels, but also that their eye movements related to image scrutiny changed in a learning-dependent fashion. Two additional, smaller exploratory experiments suggested that allowing subjects to re-examine the mammogram in light of various items of diagnostic information may help further improve DL of the diagnostic patterns. Finally, a fourth small, exploratory experiment suggested that the image information learned was similar across subjects. Together, these results prove the principle that DL methodologies can be used to train nonprofessional subjects to reliably perform those aspects of medical image perception tasks that depend on visual pattern recognition expertise.

How to detect high-performing individuals and groups: Decision similarity predicts accuracy

Distinguishing between high- and low-performing individuals and groups is of prime importance in a wide range of high-stakes contexts. While this is straightforward when accurate records of past performance exist, these records are unavailable in most real-world contexts. Focusing on the class of binary decision problems, we use a combined theoretical and empirical approach to develop and test a approach to this important problem. First, we use a general mathematical argument and numerical simulations to show that the similarity of an individual's decisions to others is a powerful predictor of that individual's decision accuracy. Second, testing this prediction with several large datasets on breast and skin cancer diagnostics, geopolitical forecasting, and a general knowledge task, we find that decision similarity robustly permits the identification of high-performing individuals and groups. Our findings offer a simple, yet broadly applicable, heuristic for improving real-world decision-making systems.

Utility of Supplemental Training to Improve Radiologist Performance in Breast Cancer Screening: A Literature Review

PURPOSE

The authors evaluate whether supplemental training for radiologists improves their breast screening performance and how this is measured.

METHODS

A systematic search was conducted in PubMed on August 3, 2017. Articles were included if they described supplemental training for radiologists reading mammograms to improve their breast screening performance and at least one outcome measure was reported. Study quality was assessed using the Medical Education Research Study Quality Instrument.

RESULTS

Of 2,199 identified articles, 18 were included, of which 17 showed improvement on at least one of the outcome measures, for at least one training activity or subgroup. Two measurement approaches were found. For the first approach, measuring performance on test sets, sensitivity, and specificity were the most reported outcomes (8 of 11 studies). Recall rate is the most reported outcome (6 of 7 studies) for the second approach, which measures performance in actual screening practice. The studies were mainly of moderate quality (Medical Education Research Study Quality Instrument score 11.7 ± 1.7), caused by small sample sizes and the lack of a control group.

CONCLUSIONS

Supplemental training helps radiologists improve their screening performance, despite the mainly moderate quality of the studies. There is a need for better designed studies. Future studies should focus on performance in actual screening practice and should look for methods to isolate the training effect. If test sets are used, focus should be on knowledge about correlation between performance on test sets and actual screening practice.

Correlation Between Screening Mammography Interpretive Performance on a Test Set and Performance in Clinical Practice

RATIONALE AND OBJECTIVES

Evidence is inconsistent about whether radiologists' interpretive performance on a screening mammography test set reflects their performance in clinical practice. This study aimed to estimate the correlation between test set and clinical performance and determine if the correlation is influenced by cancer prevalence or lesion difficulty in the test set.

MATERIALS AND METHODS

This institutional review board-approved study randomized 83 radiologists from six Breast Cancer Surveillance Consortium registries to assess one of four test sets of 109 screening mammograms each; 48 radiologists completed a fifth test set of 110 mammograms 2 years later. Test sets differed in number of cancer cases and difficulty of lesion detection. Test set sensitivity and specificity were estimated using woman-level and breast-level recall with cancer status and expert opinion as gold standards. Clinical performance was estimated using women-level recall with cancer status as the gold standard. Spearman rank correlations between test set and clinical performance with 95% confidence intervals (CI) were estimated.

RESULTS

For test sets with fewer cancers (N = 15) that were more difficult to detect, correlations were weak to moderate for sensitivity (woman level = 0.46, 95% CI = 0.16, 0.69; breast level = 0.35, 95% CI = 0.03, 0.61) and weak for specificity (0.24, 95% CI = 0.01, 0.45) relative to expert recall. Correlations for test sets with more cancers (N = 30) were close to 0 and not statistically significant.

CONCLUSIONS

Correlations between screening performance on a test set and performance in clinical practice are not strong. Test set performance more accurately reflects performance in clinical practice if cancer prevalence is low and lesions are challenging to detect.

Boosting medical diagnostics by pooling independent judgments

Collective intelligence refers to the ability of groups to outperform individual decision makers when solving complex cognitive problems. Despite its potential to revolutionize decision making in a wide range of domains, including medical, economic, and political decision making, at present, little is known about the conditions underlying collective intelligence in real-world contexts. We here focus on two key areas of medical diagnostics, breast and skin cancer detection. Using a simulation study that draws on large real-world datasets, involving more than 140 doctors making more than 20,000 diagnoses, we investigate when combining the independent judgments of multiple doctors outperforms the best doctor in a group. We find that similarity in diagnostic accuracy is a key condition for collective intelligence: Aggregating the independent judgments of doctors outperforms the best doctor in a group whenever the diagnostic accuracy of doctors is relatively similar, but not when doctors' diagnostic accuracy differs too much. This intriguingly simple result is highly robust and holds across different group sizes, performance levels of the best doctor, and collective intelligence rules. The enabling role of similarity, in turn, is explained by its systematic effects on the number of correct and incorrect decisions of the best doctor that are overruled by the collective. By identifying a key factor underlying collective intelligence in two important real-world contexts, our findings pave the way for innovative and more effective approaches to complex real-world decision making, and to the scientific analyses of those approaches.

Audit feedback on reading performance of screening mammograms: An international comparison

Objective

Providing feedback to mammography radiologists and facilities may improve interpretive performance. We conducted a web-based survey to investigate how and why such feedback is undertaken and used in mammographic screening programmes.

Methods

The survey was sent to representatives in 30 International Cancer Screening Network member countries where mammographic screening is offered.

Results

Seventeen programmes in 14 countries responded to the survey. Audit feedback was aimed at readers in 14 programmes, and facilities in 12 programmes. Monitoring quality assurance was the most common purpose of audit feedback. Screening volume, recall rate, and rate of screen-detected cancers were typically reported performance measures. Audit reports were commonly provided annually, but more frequently when target guidelines were not reached.

Conclusion

The purpose, target audience, performance measures included, form and frequency of the audit feedback varied amongst mammographic screening programmes. These variations may provide a basis for those developing and improving such programmes.

Acquisition and maintenance of medical expertise: a perspective from the expert-performance approach with deliberate practice

As a part of a special collection in this issue of Academic Medicine, which is focused on mastery learning in medical education, this Perspective describes how the expert-performance approach with deliberate practice is consistent with many characteristics of mastery learning. Importantly, this Perspective also explains how the expert-performance approach provides a very different perspective on the acquisition of skill. Whereas traditional education with mastery learning focuses on having students attain an adequate level of performance that is based on goals set by the existing curricula, the expert-performance approach takes an empirical approach and first identifies the final goal of training-namely, reproducibly superior objective performance (superior patient outcomes) for individuals in particular medical specialties. Analyzing this superior complex performance reveals three types of mental representations that permit expert performers to plan, execute, and monitor their own performance. By reviewing research on medical performance and education, the author describes evidence for these representations and their development within the expert-performance framework. He uses the research to generate suggestions for improved training of medical students and professionals. Two strategies-designing learning environments with libraries of cases and creating opportunities for individualized teacher-guided training-should enable motivated individuals to acquire a full set of refined mental representations. Providing the right resources to support the expert-performance approach will allow such individuals to become self-regulated learners-that is, members of the medical community who have the tools to improve their own and their team members' performances throughout their entire professional careers.

Collective intelligence meets medical decision-making: the collective outperforms the best radiologist

While collective intelligence (CI) is a powerful approach to increase decision accuracy, few attempts have been made to unlock its potential in medical decision-making. Here we investigated the performance of three well-known collective intelligence rules (“majority”, “quorum”, and “weighted quorum”) when applied to mammography screening. For any particular mammogram, these rules aggregate the independent assessments of multiple radiologists into a single decision (recall the patient for additional workup or not). We found that, compared to single radiologists, any of these CI-rules both increases true positives (i.e., recalls of patients with cancer) and decreases false positives (i.e., recalls of patients without cancer), thereby overcoming one of the fundamental limitations to decision accuracy that individual radiologists face. Importantly, we find that all CI-rules systematically outperform even the best-performing individual radiologist in the respective group. Our findings demonstrate that CI can be employed to improve mammography screening; similarly, CI may have the potential to improve medical decision-making in a much wider range of contexts, including many areas of diagnostic imaging and, more generally, diagnostic decisions that are based on the subjective interpretation of evidence.

Effect of radiologists' diagnostic work-up volume on interpretive performance

PURPOSE

To examine radiologists' screening performance in relation to the number of diagnostic work-ups performed after abnormal findings are discovered at screening mammography by the same radiologist or by different radiologists.

MATERIALS AND METHODS

In an institutional review board-approved HIPAA-compliant study, the authors linked 651 671 screening mammograms interpreted from 2002 to 2006 by 96 radiologists in the Breast Cancer Surveillance Consortium to cancer registries (standard of reference) to evaluate the performance of screening mammography (sensitivity, false-positive rate [ FPR false-positive rate ], and cancer detection rate [ CDR cancer detection rate ]). Logistic regression was used to assess the association between the volume of recalled screening mammograms ("own" mammograms, where the radiologist who interpreted the diagnostic image was the same radiologist who had interpreted the screening image, and "any" mammograms, where the radiologist who interpreted the diagnostic image may or may not have been the radiologist who interpreted the screening image) and screening performance and whether the association between total annual volume and performance differed according to the volume of diagnostic work-up.

RESULTS

Annually, 38% of radiologists performed the diagnostic work-up for 25 or fewer of their own recalled screening mammograms, 24% performed the work-up for 0-50, and 39% performed the work-up for more than 50. For the work-up of recalled screening mammograms from any radiologist, 24% of radiologists performed the work-up for 0-50 mammograms, 32% performed the work-up for 51-125, and 44% performed the work-up for more than 125. With increasing numbers of radiologist work-ups for their own recalled mammograms, the sensitivity (P = .039), FPR false-positive rate (P = .004), and CDR cancer detection rate (P < .001) of screening mammography increased, yielding a stepped increase in women recalled per cancer detected from 17.4 for 25 or fewer mammograms to 24.6 for more than 50 mammograms. Increases in work-ups for any radiologist yielded significant increases in FPR false-positive rate (P = .011) and CDR cancer detection rate (P = .001) and a nonsignificant increase in sensitivity (P = .15). Radiologists with a lower annual volume of any work-ups had consistently lower FPR false-positive rate , sensitivity, and CDR cancer detection rate at all annual interpretive volumes.

CONCLUSION

These findings support the hypothesis that radiologists may improve their screening performance by performing the diagnostic work-up for their own recalled screening mammograms and directly receiving feedback afforded by means of the outcomes associated with their initial decision to recall. Arranging for radiologists to work up a minimum number of their own recalled cases could improve screening performance but would need systems to facilitate this workflow.

Feasibility and acceptability of conducting a randomized clinical trial designed to improve interpretation of screening mammography

PURPOSE

To describe recruitment, enrollment, and participation in a study of US radiologists invited to participate in a randomized controlled trial of two continuing medical education (CME) interventions designed to improve interpretation of screening mammography.

METHODS

We collected recruitment, consent, and intervention-completion information as part of a large study involving radiologists in California, Oregon, Washington, New Mexico, New Hampshire, North Carolina, and Vermont. Consenting radiologists were randomized to receive either a 1-day live, expert-led educational session; to receive a self-paced DVD with similar content; or to a control group (delayed intervention). The impact of the interventions was assessed using a preintervention-postintervention test set design. All activities were institutional review board approved and HIPAA compliant.

RESULTS

Of 403 eligible radiologists, 151 of 403 (37.5%) consented to participate in the trial and 119 of 151 (78.8%) completed the preintervention test set, leaving 119 available for randomization to one of the two intervention groups or to controls. Female radiologists were more likely than male radiologists to consent to and complete the study (P = .03). Consenting radiologists who completed all study activities were more likely to have been interpreting mammography for 10 years or less compared to radiologists who consented and did not complete all study activities or did not consent at all. The live intervention group was more likely to report their intent to change their clinical practice as a result of the intervention compared to those who received the DVD (50% versus 17.6%, P = .02). The majority of participants in both interventions groups felt the interventions were a useful way to receive CME mammography credits.

CONCLUSIONS

Community radiologists found interactive interventions designed to improve interpretative mammography performance acceptable and useful for clinical practice. This suggests CME credits for radiologists should, in part, be for examining practice skills.