Using a tailored web-based intervention to set goals to reduce unnecessary recall

Improvements in appropriate ambulatory antibiotic prescribing using a bundled antibiotic stewardship intervention in general pediatrics practices

OBJECTIVE

To improve appropriate antibiotic prescribing for children in Tennessee.

DESIGN

We performed a before-and-after intervention study with 3 comparison periods: period 1 (P1, baseline) May 2018-September 2019; period 2 (P2, intervention before the COVID-19 pandemic) November 11, 2019-March 20, 2020; and period 3 (P3, intervention during the coronavirus disease 2019 [COVID-19] pandemic) March 21, 2020-November 10, 2020. We additionally surveyed participating providers to assess acceptance of the intervention.

SETTING

Community general pediatrics practices.

PARTICIPANTS

In total, 81 general pediatricians, family medicine physicians, and nurse practitioners in 5 general pediatrics practices participated in this study.

INTERVENTIONS

Each practice identified a practice and operations champion for the project. Practices chose 2-4 implementation strategies previously shown to be effective at reducing outpatient antibiotic use to implement in their practice throughout the study intervention period. Study personnel also held quarterly meetings with all providers to review deidentified peer comparison feedback both across practices enrolled in the study and at the provider level within each practice.

RESULTS

We detected improvements in guideline-concordant antibiotic use in the pre-COVID-19 intervention period, and they were sustained in the study period during the pandemic (P3): otitis media (P1 72.14% vs P2 81.42% vs P3 86.11%), group A streptococcal pharyngitis (P1 66.13% vs P2 81.56% vs P3 80.44%), pneumonia (P1 70.6% vs P2 76.2% vs P3 100%), sinusitis (P1 76.2% vs P2 83.78% vs P3 82.86%), skin and soft-tissue infections (P1 97.18% vs P2 100% vs P3 100%).

CONCLUSIONS

Bundled implementation strategies led to significant increases in guideline-concordant antibiotic prescribing for all diagnoses. Survey results demonstrate that the bundled implementation strategies were well-accepted by providers.

OUP accepted manuscript

Objectives

(1) Systematically review the literature on computerized audit and feedback (e-A&F) systems in healthcare. (2) Compare features of current systems against e-A&F best practices. (3) Generate hypotheses on how e-A&F systems may impact patient care and outcomes.

Methods

We searched MEDLINE (Ovid), EMBASE (Ovid), and CINAHL (Ebsco) databases to December 31, 2020. Two reviewers independently performed selection, extraction, and quality appraisal (Mixed Methods Appraisal Tool). System features were compared with 18 best practices derived from Clinical Performance Feedback Intervention Theory. We then used realist concepts to generate hypotheses on mechanisms of e-A&F impact. Results are reported in accordance with the PRISMA statement.

Results

Our search yielded 4301 unique articles. We included 88 studies evaluating 65 e-A&F systems, spanning a diverse range of clinical areas, including medical, surgical, general practice, etc. Systems adopted a median of 8 best practices (interquartile range 6–10), with 32 systems providing near real-time feedback data and 20 systems incorporating action planning. High-confidence hypotheses suggested that favorable e-A&F systems prompted specific actions, particularly enabled by timely and role-specific feedback (including patient lists and individual performance data) and embedded action plans, in order to improve system usage, care quality, and patient outcomes.

Conclusions

e-A&F systems continue to be developed for many clinical applications. Yet, several systems still lack basic features recommended by best practice, such as timely feedback and action planning. Systems should focus on actionability, by providing real-time data for feedback that is specific to user roles, with embedded action plans.

Protocol Registration

PROSPERO CRD42016048695.

Utility of Supplemental Training to Improve Radiologist Performance in Breast Cancer Screening: A Literature Review

PURPOSE

The authors evaluate whether supplemental training for radiologists improves their breast screening performance and how this is measured.

METHODS

A systematic search was conducted in PubMed on August 3, 2017. Articles were included if they described supplemental training for radiologists reading mammograms to improve their breast screening performance and at least one outcome measure was reported. Study quality was assessed using the Medical Education Research Study Quality Instrument.

RESULTS

Of 2,199 identified articles, 18 were included, of which 17 showed improvement on at least one of the outcome measures, for at least one training activity or subgroup. Two measurement approaches were found. For the first approach, measuring performance on test sets, sensitivity, and specificity were the most reported outcomes (8 of 11 studies). Recall rate is the most reported outcome (6 of 7 studies) for the second approach, which measures performance in actual screening practice. The studies were mainly of moderate quality (Medical Education Research Study Quality Instrument score 11.7 ± 1.7), caused by small sample sizes and the lack of a control group.

CONCLUSIONS

Supplemental training helps radiologists improve their screening performance, despite the mainly moderate quality of the studies. There is a need for better designed studies. Future studies should focus on performance in actual screening practice and should look for methods to isolate the training effect. If test sets are used, focus should be on knowledge about correlation between performance on test sets and actual screening practice.

E-learning for health professionals

BACKGROUND

The use of e-learning, defined as any educational intervention mediated electronically via the Internet, has steadily increased among health professionals worldwide. Several studies have attempted to measure the effects of e-learning in medical practice, which has often been associated with large positive effects when compared to no intervention and with small positive effects when compared with traditional learning (without access to e-learning). However, results are not conclusive.

OBJECTIVES

To assess the effects of e-learning programmes versus traditional learning in licensed health professionals for improving patient outcomes or health professionals' behaviours, skills and knowledge.

SEARCH METHODS

We searched CENTRAL, MEDLINE, Embase, five other databases and three trial registers up to July 2016, without any restrictions based on language or status of publication. We examined the reference lists of the included studies and other relevant reviews. If necessary, we contacted the study authors to collect additional information on studies.

SELECTION CRITERIA

Randomised trials assessing the effectiveness of e-learning versus traditional learning for health professionals. We excluded non-randomised trials and trials involving undergraduate health professionals.

DATA COLLECTION AND ANALYSIS

Two authors independently selected studies, extracted data and assessed risk of bias. We graded the certainty of evidence for each outcome using the GRADE approach and standardised the outcome effects using relative risks (risk ratio (RR) or odds ratio (OR)) or standardised mean difference (SMD) when possible.

MAIN RESULTS

We included 16 randomised trials involving 5679 licensed health professionals (4759 mixed health professionals, 587 nurses, 300 doctors and 33 childcare health consultants).When compared with traditional learning at 12-month follow-up, low-certainty evidence suggests that e-learning may make little or no difference for the following patient outcomes: the proportion of patients with low-density lipoprotein (LDL) cholesterol of less than 100 mg/dL (adjusted difference 4.0%, 95% confidence interval (CI) -0.3 to 7.9, N = 6399 patients, 1 study) and the proportion with glycated haemoglobin level of less than 8% (adjusted difference 4.6%, 95% CI -1.5 to 9.8, 3114 patients, 1 study). At 3- to 12-month follow-up, low-certainty evidence indicates that e-learning may make little or no difference on the following behaviours in health professionals: screening for dyslipidaemia (OR 0.90, 95% CI 0.77 to 1.06, 6027 patients, 2 studies) and treatment for dyslipidaemia (OR 1.15, 95% CI 0.89 to 1.48, 5491 patients, 2 studies). It is uncertain whether e-learning improves or reduces health professionals' skills (2912 health professionals; 6 studies; very low-certainty evidence), and it may make little or no difference in health professionals' knowledge (3236 participants; 11 studies; low-certainty evidence).Due to the paucity of studies and data, we were unable to explore differences in effects across different subgroups. Owing to poor reporting, we were unable to collect sufficient information to complete a meaningful 'Risk of bias' assessment for most of the quality criteria. We evaluated the risk of bias as unclear for most studies, but we classified the largest trial as being at low risk of bias. Missing data represented a potential source of bias in several studies.

AUTHORS' CONCLUSIONS

When compared to traditional learning, e-learning may make little or no difference in patient outcomes or health professionals' behaviours, skills or knowledge. Even if e-learning could be more successful than traditional learning in particular medical education settings, general claims of it as inherently more effective than traditional learning may be misleading.

A systematic review of electronic audit and feedback: intervention effectiveness and use of behaviour change theory

BACKGROUND

Audit and feedback is a common intervention for supporting clinical behaviour change. Increasingly, health data are available in electronic format. Yet, little is known regarding if and how electronic audit and feedback (e-A&F) improves quality of care in practice.

OBJECTIVE

The study aimed to assess the effectiveness of e-A&F interventions in a primary care and hospital context and to identify theoretical mechanisms of behaviour change underlying these interventions.

METHODS

In August 2016, we searched five electronic databases, including MEDLINE and EMBASE via Ovid, and the Cochrane Central Register of Controlled Trials for published randomised controlled trials. We included studies that evaluated e-A&F interventions, defined as a summary of clinical performance delivered through an interactive computer interface to healthcare providers. Data on feedback characteristics, underlying theoretical domains, effect size and risk of bias were extracted by two independent review authors, who determined the domains within the Theoretical Domains Framework (TDF). We performed a meta-analysis of e-A&F effectiveness, and a narrative analysis of the nature and patterns of TDF domains and potential links with the intervention effect.

RESULTS

We included seven studies comprising of 81,700 patients being cared for by 329 healthcare professionals/primary care facilities. Given the extremely high heterogeneity of the e-A&F interventions and five studies having a medium or high risk of bias, the average effect was deemed unreliable. Only two studies explicitly used theory to guide intervention design. The most frequent theoretical domains targeted by the e-A&F interventions included 'knowledge', 'social influences', 'goals' and 'behaviour regulation', with each intervention targeting a combination of at least three. None of the interventions addressed the domains 'social/professional role and identity' or 'emotion'. Analyses identified the number of different domains coded in control arm to have the biggest role in heterogeneity in e-A&F effect size.

CONCLUSIONS

Given the high heterogeneity of identified studies, the effects of e-A&F were found to be highly variable. Additionally, e-A&F interventions tend to implicitly target only a fraction of known theoretical domains, even after omitting domains presumed not to be linked to e-A&F. Also, little evaluation of comparative effectiveness across trial arms was conducted. Future research should seek to further unpack the theoretical domains essential for effective e-A&F in order to better support strategic individual and team goals.

Effect of radiologists' diagnostic work-up volume on interpretive performance

PURPOSE

To examine radiologists' screening performance in relation to the number of diagnostic work-ups performed after abnormal findings are discovered at screening mammography by the same radiologist or by different radiologists.

MATERIALS AND METHODS

In an institutional review board-approved HIPAA-compliant study, the authors linked 651 671 screening mammograms interpreted from 2002 to 2006 by 96 radiologists in the Breast Cancer Surveillance Consortium to cancer registries (standard of reference) to evaluate the performance of screening mammography (sensitivity, false-positive rate [ FPR false-positive rate ], and cancer detection rate [ CDR cancer detection rate ]). Logistic regression was used to assess the association between the volume of recalled screening mammograms ("own" mammograms, where the radiologist who interpreted the diagnostic image was the same radiologist who had interpreted the screening image, and "any" mammograms, where the radiologist who interpreted the diagnostic image may or may not have been the radiologist who interpreted the screening image) and screening performance and whether the association between total annual volume and performance differed according to the volume of diagnostic work-up.

RESULTS

Annually, 38% of radiologists performed the diagnostic work-up for 25 or fewer of their own recalled screening mammograms, 24% performed the work-up for 0-50, and 39% performed the work-up for more than 50. For the work-up of recalled screening mammograms from any radiologist, 24% of radiologists performed the work-up for 0-50 mammograms, 32% performed the work-up for 51-125, and 44% performed the work-up for more than 125. With increasing numbers of radiologist work-ups for their own recalled mammograms, the sensitivity (P = .039), FPR false-positive rate (P = .004), and CDR cancer detection rate (P < .001) of screening mammography increased, yielding a stepped increase in women recalled per cancer detected from 17.4 for 25 or fewer mammograms to 24.6 for more than 50 mammograms. Increases in work-ups for any radiologist yielded significant increases in FPR false-positive rate (P = .011) and CDR cancer detection rate (P = .001) and a nonsignificant increase in sensitivity (P = .15). Radiologists with a lower annual volume of any work-ups had consistently lower FPR false-positive rate , sensitivity, and CDR cancer detection rate at all annual interpretive volumes.

CONCLUSION

These findings support the hypothesis that radiologists may improve their screening performance by performing the diagnostic work-up for their own recalled screening mammograms and directly receiving feedback afforded by means of the outcomes associated with their initial decision to recall. Arranging for radiologists to work up a minimum number of their own recalled cases could improve screening performance but would need systems to facilitate this workflow.

Impact of an educational intervention designed to reduce unnecessary recall during screening mammography

RATIONALE AND OBJECTIVES

The aim of this study was to describe the impact of a tailored Web-based educational program designed to reduce excessive screening mammography recall.

MATERIALS AND METHODS

Radiologists enrolled in one of four mammography registries in the United States were invited to take part and were randomly assigned to receive the intervention or to serve as controls. The controls were offered the intervention at the end of the study, and data collection included an assessment of their clinical practice as well. The intervention provided each radiologist with individual audit data for his or her sensitivity, specificity, recall rate, positive predictive value, and cancer detection rate compared to national benchmarks and peer comparisons for the same measures; profiled breast cancer risk in each radiologist's respective patient populations to illustrate how low breast cancer risk is in population-based settings; and evaluated the possible impact of medical malpractice concerns on recall rates. Participants' recall rates from actual practice were evaluated for three time periods: the 9 months before the intervention was delivered to the intervention group (baseline period), the 9 months between the intervention and control groups (T1), and the 9 months after completion of the intervention by the controls (T2). Logistic regression models examining the probability that a mammogram was recalled included indication of intervention versus control and time period (baseline, T1, and T2). Interactions between the groups and time period were also included to determine if the association between time period and the probability of a positive result differed across groups.

RESULTS

Thirty-one radiologists who completed the continuing medical education intervention were included in the adjusted model comparing radiologists in the intervention group (n = 22) to radiologists who completed the intervention in the control group (n = 9). At T1, the intervention group had 12% higher odds of positive mammographic results compared to the controls, after controlling for baseline (odds ratio, 1.12; 95% confidence interval, 1.00-1.27; P = .0569). At T2, a similar association was found, but it was not statistically significant (odds ratio, 1.10; 95% confidence interval, 0.96 to 1.25). No associations were found among radiologists in the control group when comparing those who completed the continuing medical education intervention (n = 9) to those who did not (n = 10). In addition, no associations were found between time period and recall rate among radiologists who set realistic goals.

CONCLUSIONS

This study resulted in a null effect, which may indicate that a single 1-hour intervention is not adequate to change excessive recall among radiologists who undertook the intervention being tested.

Are radiologists' goals for mammography accuracy consistent with published recommendations?

RATIONALE AND OBJECTIVES

Mammography quality assurance programs have been in place for more than a decade. We studied radiologists' self-reported performance goals for accuracy in screening mammography and compared them to published recommendations.

MATERIALS AND METHODS

A mailed survey of radiologists at mammography registries in seven states within the Breast Cancer Surveillance Consortium (BCSC) assessed radiologists' performance goals for interpreting screening mammograms. Self-reported goals were compared to published American College of Radiology (ACR) recommended desirable ranges for recall rate, false-positive rate, positive predictive value of biopsy recommendation (PPV2), and cancer detection rate. Radiologists' goals for interpretive accuracy within desirable range were evaluated for associations with their demographic characteristics, clinical experience, and receipt of audit reports.

RESULTS

The survey response rate was 71% (257 of 364 radiologists). The percentage of radiologists reporting goals within desirable ranges was 79% for recall rate, 22% for false-positive rate, 39% for PPV2, and 61% for cancer detection rate. The range of reported goals was 0%-100% for false-positive rate and PPV2. Primary academic affiliation, receiving more hours of breast imaging continuing medical education, and receiving audit reports at least annually were associated with desirable PPV2 goals. Radiologists reporting desirable cancer detection rate goals were more likely to have interpreted mammograms for 10 or more years, and >1000 mammograms per year.

CONCLUSION

Many radiologists report goals for their accuracy when interpreting screening mammograms that fall outside of published desirable benchmarks, particularly for false-positive rate and PPV2, indicating an opportunity for education.