The use of 'Think-Out-Loud' methodology in the development of teaching materials for abbreviated breast Magnetic Resonance Imaging scan (FAST MRI) interpretation, and a comparison of the learning experience of two reader cohorts

Are the content and usability of a new direct observation tool adequate for assessing competency in delivering person-centred care: a think-aloud study with patients and healthcare professionals in Sweden

OBJECTIVE

To evaluate the content and usability of a new direct observation tool for assessing competency in delivering person-centred care based on the Gothenburg Centre for Person-Centred Care (gPCC) framework.

DESIGN

This is a qualitative study using think-aloud techniques and retrospective probing interviews and analyzed using deductive content analysis.

SETTING

Sessions were conducted remotely via Zoom with participants in their homes or offices.

PARTICIPANTS

11 participants with lengthy experience of receiving, delivering and/or implementing gPCC were recruited using purposeful sampling and selected to represent a broad variety of stakeholders and potential end-users.

RESULTS

Participants generally considered the content of the four main domains of the tool, that is, person-centred care activities, clinician manner, clinician skills and person-centred care goals, to be comprehensive and relevant for assessing person-centred care in general and gPCC in particular. Some participants pointed to the need to expand person-centred care activities to better reflect the emphasis on eliciting patient resources/capabilities and psychosocial needs in the gPCC framework. Think-aloud analyses revealed some usability issues primarily regarding difficulties or uncertainties in understanding several words and in using the rating scale. Probing interviews indicated that these problems could be mitigated by improving written instructions regarding response options and by replacing some words. Participants generally were satisfied with the layout and structure of the tool, but some suggested enlarging font size and text spacing to improve readability.

CONCLUSION

The tool appears to satisfactorily cover major person-centred care activities outlined in the gPCC framework. The inclusion of content concerning clinician manner and skills was seen as a relevant embellishment of the framework and as contributing to a more comprehensive assessment of clinician performance in the delivery of person-centred care. A revised version addressing observed content and usability issues will be tested for inter-rater and intra-rater reliability and for feasibility of use in healthcare education and quality improvement efforts.

Rapid Design of a Student-Centred App for Musculoskeletal Clinical Skills: An Example of a Theoretically Informed Approach to Developing Apps for Learning

Background and need for innovation

The process to design mobile apps for learning are infrequently reported and focus more on evaluation than process. This lack of clear process for health professional education mobile apps may explain the lack of quality mobile apps to support medical student learning.

Goal of innovation

The goal of this project was to develop a student informed ready for production wireframe model of a minimally viable mobile app to support learning of musculoskeletal (MSK) clinical skills.

Steps taken for development and implementation of innovation

The Information Systems Research (ISR) framework and Design Thinking were combined for the mobile app design. The process followed the cycles and modes of the combined framework to; systematically review available apps, use a focus group to identify attributes of the app valued by students, define the initial plan for the mobile app, develop an app prototype, and test and refine it with students.

Outcomes of innovation

The student focus group data had five themes: 1) interactive usability, 2) environment, 3) clear and concise layout, 4) anatomy and pathology, 5) cultural safety and 'red flags'. The prototyping of the app went through three cycles of student review and improvement to produce a final design ready for app development.

Critical reflection on our process

We used a student-centred approach guided by design frameworks to design a minimally viable product mobile app to support learning of MSK clinical skills in ten weeks with a small team. The framework supported nonlinear, iterative, rapid prototyping. Student data converged and diverged with the MSK teaching methods literature. Of note our students requested cultural safety learning in the app design, suggesting mobile apps could support cultural safety learning.

Optimising the diagnostic accuracy of First post-contrAst SubtracTed breast MRI (FAST MRI) through interpretation-training: a multicentre e-learning study, mapping the learning curve of NHS Breast Screening Programme (NHSBSP) mammogram readers using an enriched dataset

BACKGROUND

Abbreviated breast MRI (FAST MRI) is being introduced into clinical practice to screen women with mammographically dense breasts or with a personal history of breast cancer. This study aimed to optimise diagnostic accuracy through the adaptation of interpretation-training.

METHODS

A FAST MRI interpretation-training programme (short presentations and guided hands-on workstation teaching) was adapted to provide additional training during the assessment task (interpretation of an enriched dataset of 125 FAST MRI scans) by giving readers feedback about the true outcome of each scan immediately after each scan was interpreted (formative assessment). Reader interaction with the FAST MRI scans used developed software (RiViewer) that recorded reader opinions and reading times for each scan. The training programme was additionally adapted for remote e-learning delivery.

STUDY DESIGN

Prospective, blinded interpretation of an enriched dataset by multiple readers.

RESULTS

43 mammogram readers completed the training, 22 who interpreted breast MRI in their clinical role (Group 1) and 21 who did not (Group 2). Overall sensitivity was 83% (95%CI 81-84%; 1994/2408), specificity 94% (95%CI 93-94%; 7806/8338), readers' agreement with the true outcome kappa = 0.75 (95%CI 0.74-0.77) and diagnostic odds ratio = 70.67 (95%CI 61.59-81.09). Group 1 readers showed similar sensitivity (84%) to Group 2 (82% p = 0.14), but slightly higher specificity (94% v. 93%, p = 0.001). Concordance with the ground truth increased significantly with the number of FAST MRI scans read through the formative assessment task (p = 0.002) but by differing amounts depending on whether or not a reader had previously attended FAST MRI training (interaction p = 0.02). Concordance with the ground truth was significantly associated with reading batch size (p = 0.02), tending to worsen when more than 50 scans were read per batch. Group 1 took a median of 56 seconds (range 8-47,466) to interpret each FAST MRI scan compared with 78 (14-22,830, p < 0.0001) for Group 2.

CONCLUSIONS

Provision of immediate feedback to mammogram readers during the assessment test set reading task increased specificity for FAST MRI interpretation and achieved high diagnostic accuracy. Optimal reading-batch size for FAST MRI was 50 reads per batch. Trial registration (25/09/2019): ISRCTN16624917.

Evaluating the effectiveness of abbreviated breast MRI (abMRI) interpretation training for mammogram readers: a multi-centre study assessing diagnostic performance, using an enriched dataset

BACKGROUND

Abbreviated breast MRI (abMRI) is being introduced in breast screening trials and clinical practice, particularly for women with dense breasts. Upscaling abMRI provision requires the workforce of mammogram readers to learn to effectively interpret abMRI. The purpose of this study was to examine the diagnostic accuracy of mammogram readers to interpret abMRI after a single day of standardised small-group training and to compare diagnostic performance of mammogram readers experienced in full-protocol breast MRI (fpMRI) interpretation (Group 1) with that of those without fpMRI interpretation experience (Group 2).

METHODS

Mammogram readers were recruited from six NHS Breast Screening Programme sites. Small-group hands-on workstation training was provided, with subsequent prospective, independent, blinded interpretation of an enriched dataset with known outcome. A simplified form of abMRI (first post-contrast subtracted images (FAST MRI), displayed as maximum-intensity projection (MIP) and subtracted slice stack) was used. Per-breast and per-lesion diagnostic accuracy analysis was undertaken, with comparison across groups, and double-reading simulation of a consecutive screening subset.

RESULTS

37 readers (Group 1: 17, Group 2: 20) completed the reading task of 125 scans (250 breasts) (total = 9250 reads). Overall sensitivity was 86% (95% confidence interval (CI) 84-87%; 1776/2072) and specificity 86% (95%CI 85-86%; 6140/7178). Group 1 showed significantly higher sensitivity (843/952; 89%; 95%CI 86-91%) and higher specificity (2957/3298; 90%; 95%CI 89-91%) than Group 2 (sensitivity = 83%; 95%CI 81-85% (933/1120) p < 0.0001; specificity = 82%; 95%CI 81-83% (3183/3880) p < 0.0001). Inter-reader agreement was higher for Group 1 (kappa = 0.73; 95%CI 0.68-0.79) than for Group 2 (kappa = 0.51; 95%CI 0.45-0.56). Specificity improved for Group 2, from the first 55 cases (81%) to the remaining 70 (83%) (p = 0.02) but not for Group 1 (90-89% p = 0.44), whereas sensitivity remained consistent for both Group 1 (88-89%) and Group 2 (83-84%).

CONCLUSIONS

Single-day abMRI interpretation training for mammogram readers achieved an overall diagnostic performance within benchmarks published for fpMRI but was insufficient for diagnostic accuracy of mammogram readers new to breast MRI to match that of experienced fpMRI readers. Novice MRI reader performance improved during the reading task, suggesting that additional training could further narrow this performance gap.

Small lesion classification on abbreviated breast MRI: training can improve diagnostic performance and inter-reader agreement

OBJECTIVE

To determine whether the diagnostic performance and inter-reader agreement for small lesion classification on abbreviated breast MRI (AB-MRI) can be improved by training, and can achieve the level of full diagnostic protocol MRI (FDP-MRI).

METHODS

This retrospective study enrolled 1165 breast lesions (≤ 2 cm; 409 malignant and 756 benign) from 1165 MRI examinations for reading test. Twelve radiologists were assigned into a trained group and a non-trained group. They interpreted each AB-MRI twice, which was extracted from FDP-MRI. After the first read, the trained group received a structured training for AB-MRI interpretation while the non-trained group did not. FDP-MRIs were interpreted by the trained group after the second read. BI-RADS category for each lesion was compared to the standard of reference (histopathological examination or follow-up) to calculate diagnostic accuracy. Inter-reader agreement was assessed using multirater k analysis. Diagnostic accuracy and inter-reader agreement were compared between the trained and non-trained groups, between the first and second reads, and between AB-MRI and FDP-MRI.

RESULTS

After training, the diagnostic accuracy of AB-MRI increased from 77.6 to 84.4%, and inter-reader agreement improved from 0.410 to 0.579 (both p < 0.001), which were higher than those of the non-trained group (accuracy, 84.4% vs 78.0%; weighted k, 0.579 vs 0.461; both p < 0.001). The post-training accuracy and inter-reader agreement of AB-MRI were lower than those of FDP-MRI (accuracy, 84.4% vs 92.8%; weighted k, 0.579 vs 0.602; both p < 0.001).

CONCLUSIONS

Training can improve the diagnostic performance and inter-reader agreement for small lesion classification on AB-MRI; however, it remains inferior to those of FDP-MRI.

KEY POINTS

• Training can improve the diagnostic performance for small breast lesions on AB-MRI. • Training can reduce inter-observer variation for breast lesion classification on AB-MRI, especially among junior radiologists. • The post-training diagnostic performance and inter-reader agreement of AB-MRI remained inferior to those of FDP-MRI.

The potential utility of abbreviated breast MRI (FAST MRI) as a tool for breast cancer screening: a systematic review and meta-analysis

AIM

To synthesise evidence comparing abbreviated breast magnetic resonance imaging (abMRI) to full-protocol MRI (fpMRI) in breast cancer screening.

MATERIALS AND METHODS

A systematic search was undertaken in multiple databases. Cohort studies without enrichment, presenting accuracy data of abMRI in screening, for any level of risk (population, moderate, high risk) were included. Level of evidence was assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE). Meta-analyses (bivariate random effects model) were performed for abMRI, with fpMRI and histology from fpMRI-positive cases as reference standard, and with follow-up to symptomatic detection added to the fpMRI. The review also covers evidence comparing abMRI with mammographic techniques.

RESULTS

The title and abstract review retrieved 23 articles. Five studies (six articles) were included (2,763 women, 3,251 screening rounds). GRADE assessment of the evidence was very low because the reference standard was interpreted with knowledge of the index test and biopsy was not obtained for all abMRI positives. The overall sensitivity for abMRI, with fpMRI (and histology for fpMRI positives) as reference standard, was 94.8% (95% confidence interval [CI] 85.5-98.2) and specificity as 94.6% (95% CI: 91.5-96.6). Three studies (1,450 women, 1,613 screening rounds) presented follow-up data, enabling comparison between abMRI and fpMRI. Sensitivities and specificities for abMRI did not differ significantly from those for fpMRI (p=0.83 and p=0.37, respectively).

CONCLUSION

A very low level of evidence suggests abMRI could be accurate for breast cancer screening. Research is required, with follow-up to interval cancer, to determine the effect its use could have on clinical outcome.

Can mammogram readers swiftly and effectively learn to interpret first post-contrast acquisition subtracted (FAST) MRI, a type of abbreviated breast MRI?: a single centre data-interpretation study

OBJECTIVES

To assess whether NHS breast screening programme (NHSBSP) mammogram readers could effectively interpret first post-contrast acquisition subtracted (FAST) MRI, for intended use in screening for breast cancer.

METHODS

Eight NHSBSP mammogram readers from a single centre (four who also read breast MRI (Group 1) and four who do not (Group 2)) were given structured FAST MRI reader training (median 4 h: 32 min). They then prospectively interpreted 125 FAST MRIs (250 breasts: 194 normal and 56 cancer) comprising a consecutive series of screening MRIs enriched with additional cancer cases from 2015, providing 2000 interpretations. Readers were blinded to other readers' opinions and to clinical information. Categorisation followed the NHSBSP MRI reporting categorisation, with categories 4 and 5 considered indicative of cancer. Diagnostic accuracy (reference standard: histology or 2 years' follow-up) and agreement between readers were determined.

RESULTS

The accuracy achieved by Group 2 (847/1000 (85%; 95% confidence interval (CI) 82-87%)) was 5% less than that of Group 1 (898/1000 (90%; 95% CI 88-92)). Good inter-reader agreement was seen between both Group 1 readers (κ = 0.66; 95% CI 0.61-0.71) and Group 2 readers (κ = 0.63; 95% CI 0.58-0.68). The median time taken to interpret each FAST MRI was Group 1: 34 s (range 3-351) and Group 2: 77 s (range 11-321).

CONCLUSION

Brief structured training enabled multiprofessional mammogram readers to achieve similar accuracy at FAST MRI interpretation to consultant radiologists experienced at breast MRI interpretation.

ADVANCES IN KNOWLEDGE

FAST MRI could be feasible from a training-the-workforce perspective for screening within NHSBSP.