Methodological pluralism for better evaluations of complex interventions: lessons from evaluating an innovation platform in Australia

Complex interventions, such as innovation platforms, pose challenges for evaluators. A variety of methodological approaches are often required to build a more complete and comprehensive understanding of how complex interventions work. In this paper, we outline and critically appraise a methodologically pluralist evaluation of an innovation platform to strengthen primary care for Aboriginal and Torres Strait Islander Australians. In doing so, we aim to identify lessons learned from the approach taken and add to existing literature on implementing evaluations in complex settings, such as innovation platforms. The pluralist design used four evaluation approaches-developmental evaluation, principles-focused evaluation, network analysis, and framework analysis-with differing strengths and challenges. Taken together, the multiple evaluation approaches yielded a detailed description and nuanced understanding of the formation, functioning and outcomes of the innovation platform that would be difficult to achieve with any single evaluation method. While a methodologically pluralist design may place additional pressure on logistical and analytic resources available, it enables a deeper understanding of the mechanisms that underlie complex interventions.

Findings and lessons learnt implementing a cardiovascular disease quality improvement program in Australian primary care: a mixed method evaluation

Background

There are discrepancies between evidence-based guidelines for screening and management of cardiovascular disease (CVD) and implementation in Australian general practice. Quality-improvement (QI) initiatives aim to reduce these gaps. This study evaluated a QI program (QPulse) that focussed on CVD assessment and management.

Methods

This mixed-methods study explored the implementation of guidelines and adoption of a QI program with a CVD risk-reduction intervention in 34 general practices. CVD screening and management were measured pre- and post-intervention. Qualitative analyses examined participants’ Plan-Do-Study-Act (PDSA) goals and in-depth interviews with practice stakeholders focussed on barriers and enablers to the program and were analysed thematically using Normalisation Process Theory (NPT).

Results

Pre- and post-intervention data were available from 15 practices (n = 19,562 and n = 20,249, respectively) and in-depth interviews from seven practices. At baseline, 45.0% of patients had their BMI measured and 15.6% had their waist circumference recorded in the past 2 years and blood pressure, lipids and smoking status were measured in 72.5, 61.5 and 65.3% of patients, respectively. Most high-risk patients (57.5%) were not prescribed risk-reducing medications. After the intervention there were no changes in the documentation and prevalence of risk factors, attainment of BP and lipid targets or prescription of CVD risk-reducing medications. However, there was variation in performance across practices with some showing isolated improvements, such as recording waist circumference (0.7-32.2% pre-intervention to 18.5-69.8% post-intervention), BMI and smoking assessment. Challenges to the program included: lack of time, need for technical support, a perceived lack of value for quality improvement work, difficulty disseminating knowledge across the practice team, tensions between the team and clinical staff and a part-time workforce.

Conclusion

The barriers associated with this QI program was considerable in Australian GP practices. Findings highlighted they were not able to effectively operationalise the intervention due to numerous factors, ranging from lack of internal capacity and leadership to competing demands and insufficient external support.

Trial registration

Australian New Zealand Clinical Trials Reference Number (ACTRN12615000108516), registered 06/02/2015.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-021-07310-6.

P5319Evidence-practice gaps in the screening and management of cardiovascular risk factors in the Australian General Practice population

Background

Cardiovascular disease (CVD) is a major cause of death and disability in Australia affecting 1 in 6 of the Australian population. Studies a decade ago showed large evidence-practice gaps in the screening and management of CVD risk in Australian General Practice. A new risk-based screening and management guideline was launched in 2012.

Purpose

This study aimed to update the evidence to evaluate appropriate screening for, and management of, cardiovascular risk factors in Australian General Practice and explore practice and patient level predictors for appropriate screening and management.

Methods

Cross-sectional de-identified data from GP electronic health records were extracted for patients >18 years having attended at least once in the last 6 months and 3 times in the last 2 years (i.e. active patients). Practice-level data were also collected manually. The statistical cohort included Aboriginal and Torres Strait Islander people 35+ years and all others 45+ years, or any individual classified as “high CVD risk” regardless of age. High risk was defined as having either established CVD, pre-defined clinically high risk conditions or a calculated 5-year risk >15% using a Framingham based risk calculator. Appropriate screening was defined as having recorded/updated all essential risk factors for measurement of CVD risk within recommended time frames. Appropriate management was defined as: ≥1 BP lowering drug and a statin for people at high risk without CVD and the addition of an antiplatelet or anticoagulant agent for people with established CVD.

Results

Data were available on 110686 patients from 98 General Practices of which 55% were female, 1.4% of Aboriginal or Torres Strait Islander background, 14% current or ex-smoker and 15% with Diabetes. Forty-nine percent had complete and up to date screening information. Twenty-six percent were classified as high risk of which 11% had established CVD. Fifty-one per cent of those with established CVD were on appropriate treatment, vs 38% of those at high risk but without CVD. A greater proportion of males received appropriate screening (51.5% vs 47.5%). Females were less likely to receive recommended therapy (44.2% vs 55.1%) for secondary prevention but more likely for primary prevention (42% vs 35.5%). For those on BP lowering therapy, only 37% of those with CVD were reaching their target BP compared to 54% of those at high risk without established disease. 56% of those with CVD on lipid lowering therapy were reaching their targets compared to 45% of those at high risk without CVD.

Conclusion

Despite availability of a national guideline, gaps remain large for the management of CVD in Australian General Practice. Female primary prevention patients appear to receive better screening and treatment than their male counterparts, but this is reversed when they have established disease. Analysis of patient and practice level predictors for these gaps is currently underway.

Acknowledgement/Funding

National Health and Medical Research Council

P5307A consumer-direct digital health intervention for cardiovascular risk management in primary care: the Consumer Navigation of Electronic Cardiovascular Tools (CONNECT) randomised controlled trial

Background

Digital health interventions linked to electronic health records offer patients an innovative approach to support cardiovascular disease (CVD) risk management.

Purpose

Test the effectiveness of a consumer-directed digital health strategy on medication adherence and cardiovascular health outcomes in patients presenting to primary care who are at moderate to high risk of CVD.

Methods

Randomised controlled trial (RCT) with 12 month follow-up. Intervention group received an interactive digital health strategy for CVD management. The intervention was integrated with patient's electronic health record to enable automated population of risk factor and medication data (Figure). Control group received usual care. Primary outcome was the proportion of patients taking guideline-recommended BP and cholesterol-lowering medications on at least 80% of days (administrative data). Secondary outcomes included mean SBP, LDL-cholesterol, BMI and proportion meeting risk targets for physical activity, achieving smoking cessation, quality of life (EQ5D) and ehealth literacy (eHEALS).

Results

Intervention (n=486) and control (n=448) groups were well matched at baseline (Table). Mean age of participants was 67±8.1 years, 77% were male, 41% had existing CVD. At 12 months, there was no significant difference between the groups for medication days covered or mean cholesterol and BP (Table). However, there were significantly more patients in the intervention than control group who were physically active and had high ehealth literacy (Table).

Table 1. Primary and Secondary Outcomes Baseline 12 months Outcome Control (n=448) Intervention (n=486) Control (n=431) Intervention (n=460) p value* Taking BP and cholesterol medication on ≥80% of days, % 29.7 28.9 29.9 32.8 0.485 SBP (mmHg), mean±SD 139.0±16.6 137.3±15.9 136.3±16.1 136.4±17.6 0.921 LDL (mmol/L), mean±SD 2.6±1.0 2.6±1.0 2.5±0.9 2.4±1.0 0.240 BMI (kg/m2), mean±SD 29.7±5.1 29.9±5.7 29.4±5.0 29.7±5.7 0.508 Proportion achieving BP and LDL target, % 11.2 12.3 11.7 16.5 0.065 Physically active, % 84.0 85.4 79.7 87.0 0.016 Current smoker, % 12.9 13.0 12.0 8.1 0.087 eHEALS, mean±SD 27.0±6.4 27.0±6.4 26.4±7.5 28.3±6.3 0.002 *Comparing intervention and control at 12 month follow-up.

Conclusions

The integrated digital health strategy did not significantly impact on days of medication covered however, patients in the intervention had higher physical activity and ehealth literacy. The RCT highlights the importance of conducting robust research if we are to understand the potential value of digital health interventions.

Acknowledgement/Funding

This work was supported by a Project Grant from the National Health and Medical Research Council of Australia

A randomised controlled trial of a consumer-focused e-health strategy for cardiovascular risk management in primary care: the Consumer Navigation of Electronic Cardiovascular Tools (CONNECT) study protocol

INTRODUCTION

Fewer than half of all people at highest risk of a cardiovascular event are receiving and adhering to best practice recommendations to lower their risk. In this project, we examine the role of an e-health-assisted consumer-focused strategy as a means of overcoming these gaps between evidence and practice. Consumer Navigation of Electronic Cardiovascular Tools (CONNECT) aims to test whether a consumer-focused e-health strategy provided to Aboriginal and Torres Strait Islander and non-indigenous adults, recruited through primary care, at moderate-to-high risk of a cardiovascular disease event will improve risk factor control when compared with usual care.

METHODS AND ANALYSIS

Randomised controlled trial of 2000 participants with an average of 18 months of follow-up to evaluate the effectiveness of an integrated consumer-directed e-health portal on cardiovascular risk compared with usual care in patients with cardiovascular disease or who are at moderate-to-high cardiovascular disease risk. The trial will be augmented by formal economic and process evaluations to assess acceptability, equity and cost-effectiveness of the intervention. The intervention group will participate in a consumer-directed e-health strategy for cardiovascular risk management. The programme is electronically integrated with the primary care provider's software and will include interactive smart phone and Internet platforms. The primary outcome is a composite endpoint of the proportion of people meeting the Australian guideline-recommended blood pressure (BP) and cholesterol targets. Secondary outcomes include change in mean BP and fasting cholesterol levels, proportion meeting BP and cholesterol targets separately, self-efficacy, health literacy, self-reported point prevalence abstinence in smoking, body mass index and waist circumference, self-reported physical activity and self-reported medication adherence.

ETHICS AND DISSEMINATION

Primary ethics approval was received from the University of Sydney Human Research Ethics Committee and the Aboriginal Health and Medical Research Council. Results will be disseminated via the usual scientific forums including peer-reviewed publications and presentations at international conferences

CLINICAL TRIALS REGISTRATION NUMBER

ACTRN12613000715774.

Overtopping of solitary waves and solitary bores on a plane beach

The overtopping of solitary waves and bores present major hazards during the initial phase of tsunami inundation and storm surges. This paper presents new laboratory data on overtopping events by both solitary waves and solitary bores. Existing empirical overtopping scaling laws are found to be deficient for these wave forms. Two distinct scaling regimes are instead identified. For solitary waves, the overtopping rates scale linearly with the deficit in run-up freeboard. The volume flux in the incident solitary wave is also an important parameter, and a weak dependence on the nonlinearity of the waves ( H / d ) is observed. For solitary bores, the overtopping cannot be scaled uniquely, because the fluid momentum behind the incident bore front is independent of the bore height, but it is in close agreement with recent solutions of the nonlinear shallow water equations. The maximum overtopping rate for the solitary waves is shown to be the lower bound of the overtopping rate for the solitary bores with the same deficit in freeboard. Thus, for a given run-up, the solitary bores induce greater overtopping rates than the solitary waves when the relative freeboard is small.

Polyhydroxyalkanoate (PHA) biosynthesis from structurally unrelated carbon sources by a newly characterized Bacillus spp

A newly acquired polyhydroxyalkanoate (PHA) producing Bacillus spp. was identified to be a strain of Bacillus cereus using a range of microbiological and molecular techniques. This strain, named B. cereus SPV, was found to be capable of using a wide range of carbon sources including glucose, fructose, sucrose, various fatty acids and gluconate for the production of PHAs, an advantage for the commercial production of the polymers. The media used for the polymer production was novel in the context of the genus Bacillus. The PHA, once produced, was found to remain at a constant maximal concentration, without any degradation, a great advantage for the commercial production of the PHAs. This particular strain of Bacillus spp. was able to synthesize various PHAs with 3-hydroxybutyrate (3HB), 3-hydroxyvalerate (3HV) and 4-hydroxybutyrate (4HB)-like monomer units from structurally unrelated carbon sources such as fructose, sucrose and gluconate. This is the first report of the incorporation of a 4HB related monomer containing PHA by the genus Bacillus and from structurally unrelated carbon sources. The PHAs isolated had molecular weights ranging between (0.4 and 0.8) x 10(6) and low polydispersity index values (M(W)/M(N)) ranging from 2.6 to 3.4.

Pulmonary hypertension and filariasis

Images