Testing feedback message framing and comparators to address prescribing of high-risk medications in nursing homes: protocol for a pragmatic, factorial, cluster-randomized trial

Engagement is a necessary condition to test audit and feedback design features: results of a pragmatic, factorial, cluster-randomized trial with an embedded process evaluation

BACKGROUND

While audit & feedback (A&F) is an effective implementation intervention, the design elements which maximize effectiveness are unclear. Partnering with a healthcare quality advisory organization already delivering feedback, we conducted a pragmatic, 2 × 2 factorial, cluster-randomized trial to test the impact of variations in two factors: (A) the benchmark used for comparison and (B) information framing. An embedded process evaluation explored hypothesized mechanisms of effect.

METHODS

Eligible physicians worked in nursing homes in Ontario, Canada, and had voluntarily signed up to receive the report. Groups of nursing homes sharing physicians were randomized to (A) physicians' individual prescribing rates compared to top-performing peers (the top quartile) or the provincial median and (B) risk-framed information (reporting the number of patients prescribed high-risk medication) or benefit-framed information (reporting the number of patients not prescribed). We hypothesized that the top quartile comparator and risk-framing would lead to greater practice improvements. The primary outcome was the mean number of central nervous system-active medications per resident per month. Primary analyses compared the four arms at 6 months post-intervention. Factorial analyses were secondary. The process evaluation comprised a follow-up questionnaire and semi-structured interviews.

RESULTS

Two hundred sixty-seven physicians (152 clusters) were randomized: 67 to arm 1 (median benchmark, benefit framing), 65 to arm 2 (top quartile benchmark, benefit framing), 75 to arm 3 (median benchmark, risk framing), and 60 to arm 4 (top quartile benchmark, risk framing). There were no significant differences in the primary outcome across arms or for each factor. However, engagement was low (27-31% of physicians across arms downloaded the report). The process evaluation indicated that both factors minimally impacted the proposed mechanisms. However, risk-framed feedback was perceived as more actionable and more compatible with current workflows, whilst a higher target might encourage behaviour change when physicians identified with the comparator.

CONCLUSIONS

Risk framing and a top quartile comparator have the potential to achieve change. Further work to establish the strategies most likely to enhance A&F engagement, particularly with physicians who may be most likely to benefit from feedback, is required to support meaningfully addressing intricate research questions concerning the design of A&F.

TRIAL REGISTRATION

ClinicalTrials.gov, NCT02979964 . Registered 29 November 2016.

Conceptual tensions and practical trade-offs in tailoring implementation interventions

Tailored interventions have been shown to be effective and tailoring is a popular process with intuitive appeal for researchers and practitioners. However, the concept and process are ill-defined in implementation science. Descriptions of how tailoring has been applied in practice are often absent or insufficient in detail. This lack of transparency makes it difficult to synthesize and replicate efforts. It also hides the trade-offs for researchers and practitioners that are inherent in the process. In this article we juxtapose the growing prominence of tailoring with four key questions surrounding the process. Specifically, we ask: (1) what constitutes tailoring and when does it begin and end?; (2) how is it expected to work?; (3) who and what does the tailoring process involve?; and (4) how should tailoring be evaluated? We discuss these questions as a call to action for better reporting and further research to bring clarity, consistency, and coherence to tailoring, a key process in implementation science.

Audit feedback interventions to address high-risk prescriptions in long-term care homes: a costing study and return on investment analysis

Background

Audit and feedback is a common implementation strategy, but few studies describe its costs. ‘MyPractice’ is a province-wide audit and feedback initiative to improve prescribing in nursing homes. This study sought to estimate the costs of ‘MyPractice’ and assess whether the financial benefit of ‘MyPractice’ offsets those costs.

Methods

We conducted a costing study from the perspective of the Ontario government. Total cost of ‘MyPractice’ was calculated as the sum of the costs of producing and disseminating the reports (covering three report releases) which were obtained from Ontario Health staff interviews and document reviews. Return on investment (ROI) was calculated as the ratio of net cost-savings and the intervention cost. Cost savings were based on the effectiveness of ‘MyPractice’ derived from a published cohort study. Cost-savings attributable to ‘MyPractice’ were estimated from the changes in the rates of antipsychotics over time between physicians who signed up and viewed the reports and those who did not sign up to the reports.

Results

Total intervention costs were C$223,691 (C$838 per physician and C$74,564 per release). Costs incurred during the development phase accounted for 74% of the total cost (C$166,117), while implementation costs for three report releases were responsible for 26% of the total costs (C$57,575). The ROI for every C$1 spent on the ‘MyPractice’ intervention was 1.02 (95% CI 0.51, 1.93) for three report releases.

Conclusion

‘MyPractice’ report offers a good return on investment and the value for money could improve with greater number of report releases.

Supplementary Information

The online version contains supplementary material available at 10.1186/s43058-021-00225-7.

Understanding the staff behaviours that promote quality for older people living in long term care facilities: A realist review

BACKGROUND

Little is known about how the workforce influences quality in long term care facilities for older people. Staff numbers are important but do not fully explain this relationship.

OBJECTIVES

To develop theoretical explanations for the relationship between long-term care facility staffing and quality of care as experienced by residents.

DESIGN

A realist evidence synthesis to understand staff behaviours that promote quality of care for older people living in long-term care facilities.

SETTING

Long-term residential care facilities PARTICIPANTS: Long-term care facility staff, residents, and relatives METHODS: The realist review, (i) was co-developed with stakeholders to determine initial programme theories, (ii) systematically searched the evidence to test and develop theoretical propositions, and (iii) validated and refined emergent theory with stakeholder groups.

RESULTS

66 research papers were included in the review. Three key findings explain the relationship between staffing and quality: (i) quality is influenced by staff behaviours; (ii) behaviours are contingent on relationships nurtured by long-term care facility environment and culture; and (iii) leadership has an important influence on how organisational resources (sufficient staff effectively deployed, with the knowledge, expertise and skills required to meet residents' needs) are used to generate and sustain quality-promoting relationships. Six theoretical propositions explain these findings.

CONCLUSION

Leaders (at all levels) through their role-modelling behaviours can use organisational resources to endorse and encourage relationships (at all levels) between staff, residents, co-workers and family (relationship centred care) that constitute learning opportunities for staff, and encourage quality as experienced by residents and families.

A multiple-behaviour investigation of goal prioritisation in physicians receiving audit and feedback to address high-risk prescribing in nursing homes

Background

As part of their professional role, healthcare providers enact multiple competing goal-directed behaviours in time-constrained environments. Better understanding healthcare providers’ motivation to engage in the pursuit of particular goals may help inform the development of implementation interventions. We investigated healthcare providers’ pursuit of multiple goals as part of a trial evaluating the effectiveness of an audit and feedback intervention in supporting appropriate adjustment of high-risk medication prescribing by physicians working in nursing homes. Our objectives were to determine whether goal priority and constructs from Social Cognitive Theory (self-efficacy, outcome expectations, and descriptive norms) predicted intention to adjust prescribing of multiple high-risk medications and to investigate how physicians in nursing homes prioritise their goals related to high-risk medication prescribing.

Methods

Physicians in Ontario, Canada, who signed up for and accessed the audit and feedback report were invited to complete a questionnaire assessing goal priority, self-efficacy, outcome expectations, descriptive norms, and intention in relation to the three targeted behaviours (adjusting prescribing of antipsychotics, benzodiazepines, and antidepressants) and a control behaviour (adjusting statin prescribing). We conducted multiple linear regression analyses to identify predictors of intention. We also conducted semi-structured qualitative interviews to investigate how physicians in nursing homes prioritise their goals in relation to appropriately adjusting prescribing of the medications included in the report: analysis was informed by the framework analysis method.

Results

Thirty-three of 89 (37%) physicians completed the questionnaire. Goal priority was the only significant predictor of intention for each medication type; the greater a priority it was for physicians to appropriately adjust their prescribing, the stronger was their intention to do so. Across five interviews, physicians reported prioritising adjustment of antipsychotic prescribing specifically. This was influenced by negative media coverage of antipsychotic prescribing in nursing homes, the provincial government’s mandate to address antipsychotic prescribing, and by the deprescribing initiatives or best practice routines in place in their nursing homes.

Conclusions

Goal priority predicted nursing home physicians’ intention to adjust prescribing. Targeting goal priority through implementation interventions therefore has the potential to influence behaviour via increased motivation. Implementation intervention developers should consider the external factors that may drive physicians’ prioritization.

Statistical Considerations for Embedded Pragmatic Clinical Trials in People Living with Dementia

There is overwhelming need for nonpharmacological interventions to improve the health and well-being of people living with dementia (PLWD). The National Institute on Aging Imbedded Pragmatic Alzheimer's Disease (AD) and AD-Related Dementias Clinical Trials (IMPACT) Collaboratory supports clinical trials of such interventions embedded in healthcare systems. The embedded pragmatic clinical trial (ePCT) is ideally suited to testing the effectiveness of complex interventions in vulnerable populations at the point of care. These trials, however, are complex to conduct and interpret, and face challenges in efficiency (i.e., statistical power) and reproducibility. In addition, trials conducted among PLWD present specific statistical challenges, including difficulty in outcomes ascertainment from PLWD, necessitating reliance on reports by caregivers, and heterogeneity in measurements across different settings or populations. These and other challenges undercut the reliability of measurement, the feasibility of capturing outcomes using pragmatic designs, and the ability to validly estimate interventions' effectiveness in real-world settings. To address these challenges, the IMPACT Collaboratory has convened a Design and Statistics Core, the goals of which are: to support the design and conduct of ePCTs directed toward PLWD and their caregivers; to develop guidance for conducting embedded trials in this population; and to educate quantitative and clinical scientists in the design, conduct, and analysis of these trials. In this article, we discuss some of the contemporary methodological challenges in this area and develop a set of research priorities the Design and Statistics Core will undertake to meet these goals. J Am Geriatr Soc 68:S68-S73, 2020.

Audit and feedback to improve laboratory test and transfusion ordering in critical care: a systematic review

BACKGROUND

Laboratory tests and transfusions are sometimes ordered inappropriately, particularly in the critical care setting, which sees frequent use of both. Audit and Feedback (A&F) is a potentially useful intervention for modifying healthcare provider behaviors, but its application to the complex, team-based environment of critical care is not well understood. We conducted a systematic review of the literature on A&F interventions for improving test or transfusion ordering in the critical care setting.

METHODS

Five databases, two registries, and the bibliographies of relevant articles were searched. We included critical care studies that assessed the use of A&F targeting healthcare provider behaviors, alone or in combination with other interventions to improve test and transfusion ordering, as compared to historical practice, no intervention, or another healthcare behaviour change intervention. Studies were included only if they reported laboratory test or transfusion orders, or the appropriateness of orders, as outcomes. There were no restrictions based on study design, date of publication, or follow-up time. Intervention characteristics and absolute differences in outcomes were summarized. The quality of individual studies was assessed using a modified version of the Effective Practice and Organisation of Care Cochrane Review Group's criteria.

RESULTS

We identified 16 studies, including 13 uncontrolled before-after studies, one randomized controlled trial, one controlled before-after study, and one controlled clinical trial (quasi-experimental). These studies described 17 interventions, mostly (88%) multifaceted interventions with an A&F component. Feedback was most often provided in a written format only (41%), more than once (53%), and most often only provided data aggregated to the group-level (41%). Most studies saw a change in the hypothesized direction, but not all studies provided statistical analyses to formally test improvement. Overall study quality was low, with studies often lacking a concurrent control group.

CONCLUSIONS

Our review summarizes characteristics of A&F interventions implemented in the critical care context, points to some mechanisms by which A&F might be made more effective in this setting, and provides an overview of how the appropriateness of orders was reported. Our findings suggest that A&F can be effective in the context of critical care; however, further research is required to characterize approaches that optimize the effectiveness in this setting alongside more rigorous evaluation methods.

TRIAL REGISTRATION

PROSPERO CRD42016051941.

Effectiveness of confidential reports to physicians on their prescribing of antipsychotic medications in nursing homes

Background

Antipsychotic medication use in nursing homes is associated with potential for harms. In Ontario, Canada, an agency of the provincial government offers nursing home physicians quarterly audit and feedback on their antipsychotic prescribing. We compared the characteristics of physicians who did and did not engage with the intervention, and assessed early changes in prescribing.

Methods

This population-level, retrospective cohort study used linked administrative databases to track prescribing practices in nursing homes pre-intervention (baseline), immediately post-initiative (3 months), and at follow-up (6 months). Exposure variables identified whether a physician signed up to participate (or not) or viewed the feedback following sign up (or not). Differences in the proportion of days that residents received antipsychotic medications at 6 months compared to baseline by exposure(s) were assessed using a linear mixed effects regression analysis to adjust for a range of resident, physician, and nursing home factors. Benzodiazepine and statin prescribing were assessed as a balance and tracer measures, respectively.

Results

Of 944 eligible physicians, 210 (22.3%) signed up to recieve the feedback report and 132 (13.9%) viewed their feedback. Physicians who signed up for feedback were more likely to have graduated from a Canadian medical school, work in urban nursing homes, and care for a larger number of residents. The clinical and functional characteristics of residents were similar across physician exposure groups. At 6 months, antipsychotic prescribing had decreased in all exposure groups. Those who viewed their feedback report had a signicantly greater reduction in antipsychotic prescribing than those who did not sign up (0.94% patient-days exposed; 95% CI 0.35 to 1.54%, p = 0.002). Trends in prescribing patterns across exposure groups for benzodiazepines and statins were not statistically significant.

Interpretation

Almost a quarter of eligible physicians engaged early in a voluntary audit and feedback intervention related to antipsychotic prescribing in nursing homes. Those who viewed their feedback achieved a small but statistically significant change in prescribing, equivalent to approximately 14,000 fewer days that nursing home residents received antipsychotic medications over 6 months. This study adds to the literature regarding the role of audit and feedback interventions to improve quality of care.

Clinical performance comparators in audit and feedback: a review of theory and evidence

BACKGROUND

Audit and feedback (A&F) is a common quality improvement strategy with highly variable effects on patient care. It is unclear how A&F effectiveness can be maximised. Since the core mechanism of action of A&F depends on drawing attention to a discrepancy between actual and desired performance, we aimed to understand current and best practices in the choice of performance comparator.

METHODS

We described current choices for performance comparators by conducting a secondary review of randomised trials of A&F interventions and identifying the associated mechanisms that might have implications for effective A&F by reviewing theories and empirical studies from a recent qualitative evidence synthesis.

RESULTS

We found across 146 trials that feedback recipients' performance was most frequently compared against the performance of others (benchmarks; 60.3%). Other comparators included recipients' own performance over time (trends; 9.6%) and target standards (explicit targets; 11.0%), and 13% of trials used a combination of these options. In studies featuring benchmarks, 42% compared against mean performance. Eight (5.5%) trials provided a rationale for using a specific comparator. We distilled mechanisms of each comparator from 12 behavioural theories, 5 randomised trials, and 42 qualitative A&F studies.

CONCLUSION

Clinical performance comparators in published literature were poorly informed by theory and did not explicitly account for mechanisms reported in qualitative studies. Based on our review, we argue that there is considerable opportunity to improve the design of performance comparators by (1) providing tailored comparisons rather than benchmarking everyone against the mean, (2) limiting the amount of comparators being displayed while providing more comparative information upon request to balance the feedback's credibility and actionability, (3) providing performance trends but not trends alone, and (4) encouraging feedback recipients to set personal, explicit targets guided by relevant information.

Reinvigorating stagnant science: implementation laboratories and a meta-laboratory to efficiently advance the science of audit and feedback

Audit and feedback (A&F) is a commonly used quality improvement (QI) approach. A Cochrane review indicates that A&F is generally effective and leads to modest improvements in professional practice but with considerable variation in the observed effects. While we have some understanding of factors that enhance the effects of A&F, further research needs to explore when A&F is most likely to be effective and how to optimise it. To do this, we need to move away from two-arm trials of A&F compared with control in favour of head-to-head trials of different ways of providing A&F. This paper describes implementation laboratories involving collaborations between healthcare organisations providing A&F at scale, and researchers, to embed head-to-head trials into routine QI programmes. This can improve effectiveness while producing generalisable knowledge about how to optimise A&F. We also describe an international meta-laboratory that aims to maximise cross-laboratory learning and facilitate coordination of A&F research.

Comparing the application of two theoretical frameworks to describe determinants of adverse medical device event reporting: secondary analysis of qualitative interview data

BACKGROUND

Post-market surveillance of medical devices is reliant on physician reporting of adverse medical device events (AMDEs). Few studies have examined factors that influence whether and how physicians report AMDEs, an essential step in the development of behaviour change interventions. This study was a secondary analysis comparing application of the Theoretical Domains Framework (TDF) and the Tailored Implementation for Chronic Diseases (TICD) framework to identify potential behaviour change interventions that correspond to determinants of AMDE reporting.

METHODS

A previous study involving qualitative interviews with Canadian physicians that implant medical devices identified themes reflecting AMDE reporting determinants. In this secondary analysis, themes that emerged from the primary analysis were independently mapped to the TDF and TICD. Determinants and corresponding intervention options arising from both frameworks (and both mappers) were compared.

RESULTS

Both theoretical frameworks were useful for identifying interventions corresponding to behavioural determinants of AMDE reporting. Information or education strategies that provide evidence about AMDEs, and audit and feedback of AMDE data were identified as interventions to target the theme of physician beliefs; improving information systems, and reminder cues, prompts and awards were identified as interventions to address determinants arising from the organization or systems themes; and modifying financial/non-financial incentives and sharing data on outcomes associated with AMDEs were identified as interventions to target device market themes. Numerous operational challenges were encountered in the application of both frameworks including a lack of clarity about how directly relevant to themes the domains/determinants should be, how many domains/determinants to select, if and how to resolve discrepancies across multiple mappers, and how to choose interventions from among the large number associated with selected domains/determinants.

CONCLUSIONS

Given discrepancies in mapping themes to determinants/domains and the resulting interventions offered by the two frameworks, uncertainty remains about how to choose interventions that best match behavioural determinants in a given context. Further research is needed to provide more nuanced guidance on the application of TDF and TICD for a broader audience, which is likely to increase the utility and uptake of these frameworks in practice.