Effects of Community-Wide Teen Pregnancy Prevention Initiatives on Local Teen Birth Rates in the United States: A Synthetic Control Approach

The impact of community-wide teen pregnancy prevention initiatives (CWIs) on local U.S. birth rates among adolescents aged 15 to 19 years was examined using synthetic control methodology within a quasi-experimental design. CWIs were implemented in 10 U.S. communities from 2010 to 2015. Each initiative implemented evidence-based teen pregnancy prevention interventions at local organizations and enhanced best practices in adolescent reproductive health care at local health centers, while engaging diverse community sectors. The synthetic control method was used to estimate the impact of each CWI on overall and race- and ethnicity-specific teen births relative to rates in synthetic control communities. Additionally, we estimated the overall effect of CWIs across communities by pooling results from the 10 synthetic control case studies using the mean percentile rank. Pooled data across all 10 communities indicated an estimated average of 6.6 fewer births per 1000 teens per year overall during the initiative relative to each community's synthetic control (p = .001). By race and ethnicity, there were an estimated average of 6.4 fewer births per 1000 teens per year among Black teens (p = .03), 10.7 fewer births among Hispanic teens (p = .03), and 4.2 fewer births (p = .10) among White teens. Results from individual communities indicated an intervention effect on overall and/or race/ethnicity-specific teen birth rates in five communities. This study demonstrates the value of synthetic control methods in evaluating community-level outcomes of programmatic efforts. Findings indicate the CWIs had a positive impact on teen birth rates and have the potential to address racial and ethnic disparities in those rates.

Grand rounds in methodology: designing for integration in mixed methods research

Mixed methods research is a popular approach used to understand persistent and complex problems related to quality and safety, such as reasons why interventions are not implemented as intended or explaining differential outcomes. However, the quality and rigour of mixed methods research proposals and publications often miss opportunities for integration, which is the core of mixed methods. Achieving integration remains challenging, and failing to integrate reduces the benefits of a mixed methods approach. Therefore, the purpose of this article is to guide quality and safety researchers in planning and designing a mixed methods study that facilitates integration. We highlight how meaningful integration in mixed methods research can be achieved by centring integration at the following levels: research question, design, methods, results and reporting and interpretation levels. A holistic view of integration through all these levels will enable researchers to provide better answers to complex problems and thereby contribute to improvement of safety and quality of care.

Virtual quality improvement collaborative with primary care practices during COVID-19: a case study within a clinically integrated network

INTRODUCTION

Quality improvement collaboratives (QICs) are a common approach to facilitate practice change and improve care delivery. Attention to QIC implementation processes and outcomes can inform best practices for designing and delivering collaborative content. In partnership with a clinically integrated network, we evaluated implementation outcomes for a virtual QIC with independent primary care practices delivered during COVID-19.

METHODS

We conducted a longitudinal case study evaluation of a virtual QIC in which practices participated in bimonthly online meetings and monthly tailored QI coaching sessions from July 2020 to June 2021. Implementation outcomes included: (1) level of engagement (meeting attendance and poll questions), (2) QI capacity (assessments completed by QI coaches), (3) use of QI tools (plan-do-check-act (PDCA) cycles started and completed) and (4) participant perceptions of acceptability (interviews and surveys).

RESULTS

Seven clinics from five primary care practices participated in the virtual QIC. Of the seven sites, five were community health centres, three were in rural counties and clinic size ranged from 1 to 7 physicians. For engagement, all practices had at least one member attend all online QIC meetings and most (9/11 (82%)) poll respondents reported meeting with their QI coach at least once per month. For QI capacity, practice-level scores showed improvements in foundational, intermediate and advanced QI work. For QI tools used, 26 PDCA cycles were initiated with 9 completed. Most (10/11 (91%)) survey respondents were satisfied with their virtual QIC experience. Twelve interviews revealed additional themes such as challenges in obtaining real-time data and working with multiple electronic medical record systems.

DISCUSSION

A virtual QIC conducted with independent primary care practices during COVID-19 resulted in high participation and satisfaction. QI capacity and use of QI tools increased over 1 year. These implementation outcomes suggest that virtual QICs may be an attractive alternative to engage independent practices in QI work.

Development and validation of the Overall Fidelity Enactment Scale for Complex Interventions (OFES-CI)

BACKGROUND

In many quality improvement (QI) and other complex interventions, assessing the fidelity with which participants 'enact' intervention activities (ie, implement them as intended) is underexplored. Adapting the evaluative approach used in objective structured clinical examinations, we aimed to develop and validate a practical approach to assessing fidelity enactment-the Overall Fidelity Enactment Scale for Complex Interventions (OFES-CI).

METHODS

We developed the OFES-CI to evaluate enactment of the SCOPE QI intervention, which teaches nursing home teams to use plan-do-study-act (PDSA) cycles. The OFES-CI was piloted and revised early in SCOPE with good inter-rater reliability, so we proceeded with a single rater. An intraclass correlation coefficient (ICC) was used to assess inter-rater reliability. For 27 SCOPE teams, we used ICC to compare two methods for assessing fidelity enactment: (1) OFES-CI ratings provided by one of five trained experts who observed structured 6 min PDSA progress presentations made at the end of SCOPE, (2) average rating of two coders' deductive content analysis of qualitative process evaluation data collected during the final 3 months of SCOPE (our gold standard).

RESULTS

Using Cicchetti's classification, inter-rater reliability between two coders who derived the gold standard enactment score was 'excellent' (ICC=0.93, 95% CI=0.85 to 0.97). Inter-rater reliability between the OFES-CI and the gold standard was good (ICC=0.71, 95% CI=0.46 to 0.86), after removing one team where open-text comments were discrepant with the rating. Rater feedback suggests the OFES-CI has strong face validity and positive implementation qualities (acceptability, easy to use, low training requirements).

CONCLUSIONS

The OFES-CI provides a promising novel approach for assessing fidelity enactment in QI and other complex interventions. It demonstrates good reliability against our gold standard assessment approach and addresses the practicality problem in fidelity assessment by virtue of its suitable implementation qualities. Steps for adapting the OFES-CI to other complex interventions are offered.

Why it is hard to use PROMs and PREMs in routine health and care

Patient-reported outcome measures (PROMs) and patient-reported experience measures (PREMs) show the results of healthcare activities as rated by patients and others. Patients or their proxies record feedback using questionnaires. These can enhance quality for all and tailored care for individuals. This paper describes obstacles that inhibit widespread use of PROMs and PREMs and some potential solutions.Implementation is a prerequisite for any innovation to succeed. Health and care services are complex and people need to be engaged at every level. Most people are cautious about proven innovations such as PROMs and PREMs but champions and leaders can help them engage. The NASSS framework (reasons for Non-adoption, Abandonment and failure to Scale up, Spread or Sustain digital health innovations) helps indicate that implementation is complex why it may be resisted.The Plan-Do-Study-Act (PDSA) approach aids implementation and helps ensure that everyone knows who should do what, when, where, how and why. Noise is an under-appreciated problem, especially when tracking patients over time such as before and after treatment. Interoperability of PROMs and PREMs with electronic health records should use Fast Health Interoperability Resources and internationally accepted coding schemes such as SNOMED CT and LOINC.Most projects need multiple measures to meet the needs of everyone involved. Measure selection should focus on their relevance, ease of use, and response rates.If these problems are avoided or mitigated, PROMs and PREMs can help deliver better patient outcomes, patient experience, staff satisfaction and health equity.

Effect of implementing a heart failure admission care bundle on hospital readmission and mortality rates: interrupted time series study

This study aimed to evaluate the impact of developing and implementing a care bundle intervention to improve care for patients with acute heart failure admitted to a large London hospital. The intervention comprised three elements, targeted within 24 hours of admission: N-terminal pro-B-type natriuretic peptide (NT-proBNP) test, transthoracic Doppler two-dimensional echocardiography and specialist review by cardiology team. The SHIFT-Evidence approach to quality improvement was used. During implementation, July 2015-July 2017, 1169 patients received the intervention. An interrupted time series design was used to evaluate impact on patient outcomes, including 15 618 admissions for 8951 patients. Mixed-effects multiple Poisson and log-linear regression models were fitted for count and continuous outcomes, respectively. Effect sizes are slope change ratios pre-intervention and post-intervention. The intervention was associated with reductions in emergency readmissions between 7 and 90 days (0.98, 95% CI 0.97 to 1.00), although not readmissions between 0 and 7 days post-discharge. Improvements were seen in in-hospital mortality (0.96, 95% CI 0.95 to 0.98), and there was no change in trend for hospital length of stay. Care process changes were also evaluated. Compliance with NT-proBNP testing was already high in 2014/2015 (162 of 163, 99.4%) and decreased slightly, with increased numbers audited, to 2016/2017 (1082 of 1101, 98.2%). Over this period, rates of echocardiography (84.7-98.9%) and specialist input (51.6-90.4%) improved. Care quality and outcomes can be improved for patients with acute heart failure using a care bundle approach. A systematic approach to quality improvement, and robust evaluation design, can be beneficial in supporting successful improvement and learning.

Identifying value in healthcare transformation initiatives: an evaluation of an approach to benefits realisation

Benefits realisation management (BRM) aims to facilitate the process of identifying, measuring and tracking desired benefits derived from a project. Improvement methodology frameworks often describe BRM as integral to identifying and measuring value derived from transformation initiatives within the National Health Service and beyond. Despite this, reporting of benefits realisation plans and methodological approaches to identifying and measuring benefits remains surprisingly scarce.This project aimed to pilot and evaluate the application of a purpose-designed benefits mapping template with seven newly funded transformation projects across three hospitals in the UK. The scope of the template was to identify key project benefits and metrics associated with the project initiatives. Plan-do-study-act (PDSA) cycles were used to capture the approach and utilisation of the template by project teams. These methods also enabled critical review of the template as an enabler to identifying relevant benefits and project metrics.Stakeholder engagement with the templates was variable. This was attributed to clinical pressures induced by the second wave of COVID-19 in the UK. Despite this, teams were able to produce completed templates outlining a number of wide-ranging benefits. Themes of benefits drawn from the maps include patient experience, patient outcomes, staff experience, access to care and efficiency. Qualitative feedback from teams included the reported value of a structured template to help recognise all the potential benefits associated with each project initiative. The PDSA cycles highlighted the template as an early step in BRM. Further components to this process are recommended to include consensus of the key metrics to be measured, a tool that summarises the reporting details of those metrics, and an effective means to collate reported metrics overtime.

Quantitative analysis validation for sclerotherapy treatment of lower limb telangiectasias

BACKGROUND

The evaluation of sclerotherapy efficacy for lower limb telangiectasias, which is the standard treatment for such condition, is commonly assisted by scores based on before and after pictures. This method is marked by its subjectivity, which impairs the precision of studies on the subject, making it unfeasible to evaluate and compare different interventions. We hypothesize that a quantitative method for evaluating the effectiveness of sclerotherapy for lower limb telangiectasias may present more reproducible results. Reliable measurement methods and new technologies may become part of the clinical practice in the near future.

METHODS

Before and after treatment photographs were analyzed using a quantitative method and compared with a validated qualitative method based on improvement scores. Reliability analysis of the methods was performed, applying the intraclass correlation coefficient (ICC) and kappa coefficient with quadratic weights (Fleiss Cohen), for analysis of inter-examiner and intra-examiner agreement in both evaluation methods. Convergent validity was evaluated by applying the Spearman test. To assess the applicability of the quantitative scale, the Mann-Whitney test was used.

RESULTS

A better agreement between examiners is shown for the quantitative scale, with a mean kappa of .3986 (.251-.511) for qualitative analysis and a mean kappa of .788 (.655-.918) for quantitative analysis (P < .001 for all examiners). Convergent validity was achieved by correlation coefficients of .572 to .905 (P < .001). The quantitative scale results obtained between the specialists with different degrees of experience did not show statistical difference (seniors: 0.71 [-0.48/1.00] × juniors: 0.73 [-0.34/1.00]; P = .221).

CONCLUSIONS

Convergent validity between both analyses has been achieved, but quantitative analysis has been shown to be more reliable and can be applied by professionals of any degree of experience. The validation of quantitative analysis is a major milestone for the development of new technology and automated, reliable, applications.

Barriers to delirium screening and management during hospital admission: a qualitative analysis of inpatient nursing perspectives

BACKGROUND

Delirium in hospitalized patients is a major public health issue, yet delirium is often unrecognized and missed during inpatient admission. The objective of this study was to identify barriers to delirium screening, identification, and management from a nursing perspective on inpatient, acute care units.

METHODS

This was a pre-implementation, diagnostic evaluation study to determine current practice patterns and potential barriers to optimizing delirium care at a major university hospital. A qualitative approach was used, which included focus groups of inpatient nurses working on major medical and surgical acute care units. Focus groups were conducted until signs of thematic saturation were present, and data were analyzed via inductive thematic analysis, without predetermined theories or structures. A consensus approach was utilized for transcript coding, and final themes were generated after multiple reviews of initial themes against transcript datasets.

RESULTS

Focus group sessions (n = 3) were held with 18 nurses across two major inpatient units. Nurses reported several barriers to successful delirium screening and management. Specific challenges included difficulty with using delirium screening tools, an organizational culture not conducive to delirium prevention, and competing clinical priorities. Proposed solutions were also discussed, including decision-support systems with automated pager alerts and associated delirium order sets, which may help improve delirium care coordination and standardization.

CONCLUSION

At a major university hospital, nurses affirm the difficulty experienced with delirium screening and identification, particularly due to screening tool challenges, cultural barriers, and clinical workload. These impediments may serve as targets for a future implementation trial to improve delirium screening and management.

Complex implementation mechanisms in primary care: do physicians' beliefs about the effectiveness of innovation play a mediating role? Applying a realist inquiry and structural equation modeling approach in a formative evaluation study

BACKGROUND

The adoption of digital health technologies can improve the quality of care for polypharmacy patients, if the underlying complex implementation mechanisms are better understood. Context effects play a critical role in relation to implementation mechanisms. In primary care research, evidence on the effects of context in the adoption of digital innovation for polypharmacy management is lacking.

STUDY AIM

This study aims to identify contextual factors relevant to physician behavior and how they might mediate the adoption process.

METHODS

The physicians who participated in this formative evaluation study (n = 218) were part of the intervention group in a cluster-randomized controlled trial (AdAM). The intervention group implemented a digital innovation for clinical decision making in polypharmacy. A three-step methodological approach was used: (1) a realist inquiry approach, which involves the description of a context-mechanism-outcome configuration for the primary care setting; (2) a belief elicitation approach, which involves qualitative content analysis and the development of a quantitative latent contextualized scale; and (3) a mediation analysis using structural equation modeling (SEM) based on quantitative survey data from physicians to assess the mediating role of the contextualized scale (n = 179).

RESULTS

The key dimensions of a (1) context-mechanism-outcome model were mapped and refined. A (2) latent construct of the physicians' innovation beliefs related to the effectiveness of polypharmacy management practices was identified. Innovation beliefs play a (3) mediating role between the organizational readiness to implement change (p < 0.01) and the desired behavioral intent of physicians to adopt digital innovation (p < 0.01; R² = 0.645). Our contextualized model estimated significant mediation, with a relative size of 38% for the mediation effect. Overall, the model demonstrated good fit indices (CFI = 0.985, RMSEA = 0.034).

CONCLUSION

Physician adoption is directly affected by the readiness of primary care organizations for the implementation of change. In addition, the mediation analysis revealed that this relationship is indirectly influenced by primary care physicians' beliefs regarding the effectiveness of digital innovation. Both individual physician beliefs and practice organizational capacity could be equally prioritized in developing implementation strategies. The methodological approach used is suitable for the evaluation of complex implementation mechanisms. It has been proven to be an advantageous approach for formative evaluation.

TRIAL REGISTRATION

NCT03430336 . First registration: 12/02/2018.

CLINICALTRIALS

gov.

Using grounded theory to develop a theory of change for a violence intervention program

Hospital-based violence intervention programs (HVIP) are critical to interrupting the cycle of violence. These interventions are considered "complex" in that they have many mechanisms of change and related outcomes. Few HVIPs clearly identify the underlying mechanisms of intervention and explicitly link those with key outcomes however, limiting the field's ability to know what works best and for whom. To develop a program theory of change for these "complex interventions," a non-linear, robust methodology that is grounded in the lived experience of those delivering and receiving services is needed. To aid researchers, evaluators, students, and program developers, we describe the use of Grounded Theory as a methodology to enhance the development of complex interventions, illuminating a non-linear approach that engages key stakeholders. To illustrate application, we describe a case example of The Antifragility Initiative, a HVIP in Cleveland, Ohio. The development of the program theory of change was conducted in four phases: (1) review of existing program documents, (2) semi-structured interviews with program developers (n = 6), (3) a focus group with program stakeholders (n = 8), and (4) interviews with caregivers and youth (n = 8). Each phase informed the next and culminated in a theoretical narrative and visual model of the Antifragility Initiative. Together, the theoretical narrative and visual model identify the underlying mechanisms that can promote change by the program.

Patient involvement in quality improvement: a survey comparing naturalistic and reflective approaches

BACKGROUND

This study investigates reflective and naturalistic approaches to patient involvement in quality improvement. The reflective approach, using, for example, interviews, provides insights into patient needs and demands to support an established improvement agenda. The naturalistic approach, for example, observations, is used to discover practical problems and opportunities that professionals are currently unaware of.

METHODS

We assessed the use of naturalistic and reflective approaches in quality improvement to see whether they differed in their impact on patient needs, financial improvements and improved patient flows. Four possible combinations were used as a starting point: restrictive (low reflective-low naturalistic), in situ (low reflective-high naturalistic), retrospective (high reflective-low naturalistic) and blended (high reflective-high naturalistic). Data were collected through an online cross-sectional survey using a web-based survey tool. The original sample was based on a list of 472 participants enrolled in courses on improvement science in three Swedish regions. The response rate was 34%. Descriptives and ANOVA (Analysis of Variance) in SPSS V.23 were used for the statistical analysis.

RESULTS

The sample consisted of 16 projects characterised as restrictive, 61 as retrospective and 63 as blended. No projects were characterised as in situ. There was a significant effect of patient involvement approaches on patient flows and patient needs at the p<0.05 level (patient flows, (F(2, 128)=5.198, p=0.007) and patient needs (F(2, 127)=13.228, p=0.000)). No significant effect was found for financial results.

CONCLUSIONS

Moving beyond restrictive patient involvement is important to meet new patient needs and improve patient flows. This can be done either by increasing the use of a reflective approach or by increasing the use of both reflective and naturalistic approaches. A blended approach with high levels of both is likely to produce better results in addressing new patient needs and improving patient flows.

Developing a tool to measure enactment of complex quality improvement interventions in healthcare

Quality improvement (QI) projects are common in healthcare settings and often involve interdisciplinary teams working together towards a common goal. Many interventions and programmes have been introduced through research to convey QI skills and knowledge to healthcare workers, however, a few studies have attempted to differentiate between what individuals 'learn' or 'know' versus their capacity to apply their learnings in complex healthcare settings. Understanding and differentiating between delivery, receipt, and enactment of QI skills and knowledge is important because while enactment alone does not guarantee desired QI outcomes, it might be reasonably assumed that 'better enactment' is likely to lead to better outcomes. This paper describes the development, application and validation of a tool to measure enactment of core QI skills and knowledge of a complex QI intervention in a healthcare setting. Based on the Institute for Healthcare Improvement's Model for Improvement, existing QI assessment tools, literature on enactment fidelity and our research protocols, 10 indicators related to core QI skills and knowledge were determined. Definitions and assessment criteria were tested and refined in five iterative cycles. Qualitative data from four QI teams in long-term care homes were used to test and validate the tool. The final measurement tool contains 10 QI indicators and a five-point scale. Inter-rater reliability ranged from good to excellent. Usability and acceptability among raters were considered high. This measurement tool assists in identifying strengths and weaknesses of a QI team and allows for targeted feedback on core QI components. The indicators developed in our tool and the approach to tool development may be useful in other health related contexts where similar data are collected.

Implementing health system improvement: resources and strategies for interprofessional teams

Health system improvement (HSI) is focused on systematic changes to organisational processes and practices to improve the efficient delivery of safe care and quality outcomes. Guidelines that specify how interprofessional teams conduct HSI and knowledge translation are needed. We address this urgent requirement providing health professional teams with resources and strategies to investigate, analyse and implement system-level improvements. HSI encompasses similar, yet different, inter-related activities across a continuum. The continuum spans three categories of activities, such as quality improvement, health management research and translational health management research. A HSI decision making guide and checklist, comprising six-steps, is presented that can be used to select and plan projects. This resource comprises six interconnected steps including, defining the activity, project outcome, aim, use of evidence, appropriate methodology and implementation plan. Each step has been developed focusing on an objective, actions and resources. HSI activities provide a foundation for interprofessional collaboration, allowing multiple professions to create, share and disseminate knowledge for improved healthcare. When planned and executed well, HSI projects assist clinical and corporate staff to make evidence-informed decisions and directions for the benefit of the service, organisation and sector.

Benefits of electronic charts in intensive care and during a world health pandemic: advantages of the technology age

AIMS AND OBJECTIVES

This study sets out to describe benefits from the implementation of electronic observation charting in intensive care units (ICU). This was an extension to the existing hospital wide digital health system. We evaluated error reduction, time-savings and the costs associated with conversion from paper to digital records. The world health emergency of COVID-19 placed extraordinary strain on ICU and staff opinion was evaluated to test how well the electronic system performed.

METHODS

A clinically led project group working directly with programmers developed an electronic patient record for intensive care. Data error rates, time to add data and to make calculations were studied before and after the introduction of electronic charts. User feedback was sought pre and post go-live (during the COVID-19 pandemic) and financial implications were calculated by the hospital finance teams.

RESULTS

Error rates equating to 219 000/year were avoided by conversion to electronic charts. Time saved was the equivalent of a nursing shift each day. Recurrent cost savings per year were estimated to be £257k. Staff were overwhelmingly positive about electronic charts in ICU, even during a health pandemic and despite redeployment into intensive care where they were using the electronic charts for the first time.

DISCUSSION

Electronic ICU charts have been successfully introduced into our institution with benefits in terms of patient safety through error reduction and improved care through release of nursing time. Costs have been reduced. Staff feel supported by the digital system and report it to be helpful even during redeployment and in the unfamiliar environment of intensive care.

Grand rounds in methodology: four critical decision points in statistical process control evaluations of quality improvement initiatives

Quality improvement (QI) projects often employ statistical process control (SPC) charts to monitor process or outcome measures as part of ongoing feedback, to inform successive Plan-Do-Study-Act cycles and refine the intervention (formative evaluation). SPC charts can also be used to draw inferences on effectiveness and generalisability of improvement efforts (summative evaluation), but only if appropriately designed and meeting specific methodological requirements for generalisability. Inadequate design decreases the validity of results, which not only reduces the chance of publication but could also result in patient harm and wasted resources if incorrect conclusions are drawn. This paper aims to bring together much of what has been written in various tutorials, to suggest a process for using SPC in QI projects. We highlight four critical decision points that are often missed, how these are inter-related and how they affect the inferences that can be drawn regarding effectiveness of the intervention: (1) the need for a stable baseline to enable drawing inferences on effectiveness; (2) choice of outcome measures to assess effectiveness, safety and intervention fidelity; (3) design features to improve the quality of QI projects; (4) choice of SPC analysis aligned with the type of outcome, and reporting on the potential influence of other interventions or secular trends.These decision points should be explicitly reported for readers to interpret and judge the results, and can be seen as supplementing the Standards for Quality Improvement Reporting Excellence guidelines. Thinking in advance about both formative and summative evaluation will inform more deliberate choices and strengthen the evidence produced by QI projects.

Response of UK community hospitals to the COVID-19 pandemic: an appreciative inquiry

INTRODUCTION

The 500 community hospitals in the UK provide a range of services to their communities. The response of these small, mainly rural, hospitals to the COVID-19 pandemic has not yet been examined and so this study sought to address this gap.

METHOD

Appreciative inquiry was used to understand staff perspectives of how community hospitals responded to the COVID-19 (SARS-CoV-2) pandemic. A total of 20 organisations participated, representing 168 (34%) community hospitals in the UK. Qualitative interviews were conducted, with a total of 85 staff members, using an online video platform. 30 case studies were developed from these interviews.

RESULTS

Staff described positive changes that were made in the context of the fear and uncertainty experienced in the pandemic. Quality improvements were reported in a wide range of services and models of care such as the use of the inpatient beds, and the access and management of urgent care services. Rapid changes were made in the way that services were managed, such as communications and leadership. Programmes of accelerated training were offered for existing and redeployed staff. Attention to staff health and well-being was a feature and there were a variety of innovations designed to support patients and their families. The impact of the changes was viewed as strengthening of integrated working between staff and sectors, the ability to rapidly innovate and improve quality, and the scope to use local decision-making to make changes.

CONCLUSION

Staff of community hospitals described innovative and rapid quality improvements in their community hospitals in response to the pandemic. The case studies illustrated the features of community hospitals, showing that they can be resilient, flexible, responsive, creative, compassionate and integrated. The case studies of quality improvements are being used to encourage sharing and learning across community hospitals and beyond.

Approach to systematically examine the usefulness of quality measures in practice: Minnesota's nursing home quality indicators and scoring approach

BACKGROUND

Healthcare quality measurement systems, which use aggregated patient-level quality measures to assess organisational performance, have been introduced widely. Yet, their usefulness in practice has received scant attention. Using Minnesota nursing home quality indicators (QIs) as a case example, we demonstrate an approach for systematically evaluating QIs in practice based on: (a) parsimony and relevance, (b) usability in discriminating between facilities, (c) actionability and (d) construct validity.

METHODS

We analysed 19 risk-adjusted, facility-level QIs over the 2012-2019 period. Parsimony and relevance of QIs were evaluated using scatter plots, Pearson correlations, literature review and expert opinions. Discrimination between facilities was assessed by examining facility QI distributions and the impact of the distributions on scoring. Actionability of QIs was assessed through QI trends over time. Construct validity was assessed through exploratory factor analysis of domain structure for grouping the QIs.

RESULTS

Correlation analysis and qualitative assessment led to redefining one QI, adding one improvement-focused QI, and combining two highly correlated QIs to improve parsimony and clinical relevance. Ten of the QIs displayed normal distributions which discriminated well between the best and worst performers. The other nine QIs displayed poor discrimination; they had skewed distributions with ceiling or floor effects. We recommended scoring approaches tailored to these distributions. One QI displaying substantial improvement over time was recommended for retirement (physical restraint use). Based on factor analysis, we grouped the 18 final QIs into four domains: incontinence (4 QIs), physical functioning (4 QIs), psychosocial care (4 QIs) and care for specific conditions (6 QIs).

CONCLUSION

We demonstrated a systematic approach for evaluating QIs in practice by arriving at parsimonious and relevant QIs, tailored scoring to different QI distributions and a meaningful domain structure. This approach could be applied in evaluating quality measures in other health or long-term care settings.

Quantitative analysis of soil sustainability after applying stabilizing amendments in long-term Cd-contaminated paddy soils

Considering the biomagnification in food chains, cadmium (Cd) contamination in paddy fields has become concerning. The remediation of soil cadmium by the addition of amendments is a common agricultural practice. However, it remains ambiguous whether amendment use decreases soil environmental quality (SEQ) and sustainability. In this study, five compound amendments with different pH were utilized in long-term Cd-contaminated paddy soils. The SEQ of all treatments was quantitatively assessed according to a comprehensive evaluation mathematical model (Criteria Importance Through Inter-criteria Correlation (CRITIC)-Technique for Order Preference by Similarity to Ideal Solution (TOPSIS)), and the indicators involved in microbial functional gene (MFG) abundance, soil physicochemical and microbiological properties (CMP) and soil microbial function (N-related enzyme and transformation rate, N-ET) were measured. The results show that the SQE and remediation effect (expressed by the decrease in available Cd (A_Cd), %) in our treatments were alkaline > natural > acidic except for D alkaline treatment. The significant contradiction between soil SQE and remediation effect in D treatment attribute to its dose effects, which inhibiting microbial nitrogen assimilation and dissimilation and therefore counteracts the promoting effect of the decrease in A_Cd. Based on this discussion, three alkaline amendments (A, B and D) with similar effective remediation effect were employed in four other Cd-contaminated soils. Results indicated that both negative effect (D treatment) and promoting effect (A and B treatment) existed in the next 3 years.

Estimates of ACO savings in the presence of provider and beneficiary selection

BACKGROUND

Medicare's accountable care organizations (ACOs)-designed to improve quality and lower spending-were associated with growing savings in previous studies. However, savings estimates may be biased by beneficiary sorting among providers based on healthcare needs and by providers opting into the program based on anticipated gains.

METHODS

Using Medicare administrative claims (2009-2014), we compared annual spending changes after provider organizations joined ACOs to changes in non-ACOs (controls). To address provider selection, using novel data to identify non-ACO organizations, we restricted controls to comparably large provider organizations. To address beneficiary selection, we (a) estimated within-organization (including non-ACO comparison organizations) spending changes, (b) estimated within-beneficiary spending changes, (c) incorporated beneficiaries without qualifying healthcare expenses, and (d) used a fixed beneficiary ACO assignment using the pre-ACO period.

RESULTS

Each year, 19% of Medicare beneficiaries switched provider organizations. Spending was higher for switchers than stayers ($3163, p < .001) and grew more the next year ($2004; p < .001). Starting from a baseline regression modeled on previous ACO evaluations, estimated savings varied widely as we sequentially introduced methods to address selection. Combining methods, however, generated more stable estimated ACO savings of $46 (p = .022), averaged across cohorts.

CONCLUSIONS

When implementing a comprehensive suite of methods to adjust for provider and beneficiary selection, we estimated ACO savings that grew over time. Our estimates are in line with, but smaller than, previous estimates in the literature. Implementing piecemeal adjustments produced misleading results.

IMPLICATIONS

Our results confirm the importance of selection for savings estimates and for provider organizations managing costs and quality. Attribution rules that consider multiple years may help mitigate the impact of beneficiary churn for providers and payers. Implementing payment reform by randomizing early participants, or implementing fully across selected markets, may better serve efforts to evaluate and improve payment models.

LEVEL OF EVIDENCE

Level 3.

[Theory of change: a tool for participatory planning and evaluation in community health]

There is a growing interest in finding innovative approaches to the evaluation of complex interventions in public health. This methodological note presents the theory of change as a useful tool to facilitate both the planning and the evaluation of community health promotion interventions. A theory of change is a visual representation of how a programme or intervention works, in which its different components can be identified, together with how they relate to each other: what its objectives are, its expected results and through which actions they are expected to be achieved. It can be an applied research tool, developed using participatory approaches together with all relevant stakeholders, who can identify and jointly decide what to evaluate and how to.

Study of the adsorption mechanism on the surface of a ceramic nanomaterial for gaseous Hg(II) removal

Stable Hg(II)-containing flue gas has been successfully simulated by the plasma oxidation of Hg(0), and an effective solution for Hg(0) mercury fumes was obtained by combining the plasma with a ceramic nanomaterial. Characterization tests showed that the ceramic nanomaterial was mainly composed of silicon dioxide (SiO₂) with other minor constituents, including potassium mica (KAl₃Si₃O₁₁), iron magnesium silicate (Fe_0.24Mg_0.76SiO₃) and dolomite (CaMg(CO₃)₂). The nanomaterial had many tube bank structures inside with diameters of approximately 8-10 nm. The maximum sorption capacity of Hg(II) was 5156 μg/g, and the nanomaterial can be regenerated at least five times. During the adsorption, chemical adsorption first occurred between Hg(II) and sulfydryl moieties, but these were quickly exhausted, and Hg(II) was then removed by surface complexation and wrapped into Fe moieties. The pseudo-first-order kinetic model and the Langmuir equation had the best fitting results for the kinetics and isotherms of adsorption. This work suggests that the ceramic nanomaterial can be used as an effective and recyclable adsorbent in the removal of gaseous Hg(II).

Evaluation of a Tennessee statewide initiative to reduce early elective deliveries using quasi-experimental methods

BACKGROUND

Concerted quality improvement (QI) efforts have been taken to discourage the practice of early elective deliveries (EEDs), but few studies have robustly examined the impact of directed QI interventions in reducing EED practices. Using quasi-experimental methods, we sought to evaluate the impact of a statewide QI intervention to reduce the practice of EEDs.

METHODS

Retrospective cohort study of vital records data (2007 to 2013) for all singleton births occurring ≥36 weeks in 66 Tennessee hospitals grouped into three QI cohorts. We used interrupted-time series to estimate the effect of the QI intervention on the likelihood of an EED birth statewide, and by hospital cohort. We compared the distribution of hospital EED percentages pre- and post-intervention. Lastly, we used multivariable logistic regression to estimate the effect of QI interventions on maternal and infant outcomes.

RESULTS

Implementation of the QI intervention was associated with significant declines in likelihood of EEDs immediately following the intervention (odds ratio, OR = 0.72; p < 0.001), but these results varied by hospital cohort. Hospital risk-adjusted EED percentages ranged from 1.6-13.6% in the pre-intervention period, which significantly declined to 2.2-9.6% in the post-intervention period (p < 0.001). The QI intervention was also associated with significant reductions in operative vaginal delivery and perineal laceration, and immediate infant ventilation, but increased NICU admissions.

CONCLUSIONS

A statewide QI intervention to reduce EEDs was associated with modest but significant declines in EEDs beyond concurrent and national trends, and showed mixed results in related infant and maternal outcomes.

Pathways to use of communication campaigns' evaluation findings within international organizations

This article presents a study on the pathways and processes regarding the use of evaluation findings of communication campaigns from two international organizations, the Office of the High Commissioner for Human Rights (OHCHR) and the International Committee of the Red Cross (ICRC). Several years after the completion of the campaigns and their evaluations, our research identified 28 instances of use and six instances of non-use of the evaluation results, of which the large majority being surprising in nature. Results showed that evaluation use facilitated formal and informal changes at the individual and the organizational level; and, this pattern occurred in a predominantly non-linear fashion, interconnected and overlapping, while gradually decreasing in time and space. Evaluation use was mostly unpredictable, which reflected how meanings are constructed by staff members, as they adjusted and interpreted the findings in opportunistic ways.

Identifying quality improvement targets to facilitate colorectal cancer screening completion

The colorectal cancer (CRC) screening process involves multiple interfaces (communication exchanges and transfers of responsibility for specific actions) among primary care and gastroenterology providers, laboratory, and administrative staff. After a retrospective electronic health record (EHR) analysis discovered substantial clinic variation and low CRC screening prevalence overall in an urban, integrated safety-net system, we launched a qualitative analysis to identify potential quality improvement targets to enhance fecal immunochemical test (FIT) completion, the system's preferred screening modality. Here, we report examination of organization-, clinic-, and provider-level interfaces over a three-year period (December 2011–October 2014).

We deployed in parallel 3 qualitative data collection methods: (1) structured observation (90+ hours, 10 sites); (2) document analysis (n > 100); and (3) semi-structured interviews (n = 41) and conducted iterative thematic analysis in which findings from each method cross-informed subsequent data collection. Thematic analysis was guided by a conceptual model and applied deductive and inductive codes.

There was substantial variation in protocols for distributing and returning FIT kits both within and across clinics. Providers, clinic and laboratory staff had differing access to important data about FIT results based on clinical information system used and this affected results reporting. Communication and coordination during electronic referrals for diagnostic colonoscopy was suboptimal particularly for co-morbid patients needing anesthesia clearance.

Our multi-level approach elucidated organizational deficiencies not evident by quantitative analysis alone. Findings indicate potential quality improvement intervention targets including: (1) best-practices implementation across clinics; (2) detailed communication to providers about FIT results; and (3) creation of EHR alerts to resolve pending colonoscopy referrals before they expire.

•

Multi-level qualitative approach identified challenges to 3 clinical processes

•

Variation in fecal immunochemical testing (FIT) kit distribution and return

•

Incomplete transfer of key FIT result data across clinical information systems

•

Suboptimal communication and coordination during colonoscopy referrals

Impact of motor task execution on an individual's ability to mirror forearm positions

This work is motivated by our goal of determining why individuals with stroke are impaired when locating their arms in space. We assessed the ability of individuals without neurological impairments to mirror their forearms during various motor tasks so that we could identify baseline performance in an unimpaired population. Nine right-hand dominant participants without neurological impairments mirrored forearm positions bi-directionally (i.e., right forearm mirrors left forearm, vice versa) for three motor tasks (i.e., passive, passive/active, and active) and two position identification modes (i.e., mirroring to a position stored in working memory versus concurrently felt by the opposite arm). During each trial, the participant's reference forearm moved to a flexion ([Formula: see text]) or extension ([Formula: see text]) position, and then, their opposite forearm mirrored the position of their reference forearm. The main finding across all tested conditions is that participants mirrored forearm positions with an average magnitude of error [Formula: see text]. When controlling their forearms' movements (active motor task), participants mirrored forearm positions more accurately by up to, on average, [Formula: see text] at the flexion location than at the extension location. Moreover, participants mirrored forearm positions more accurately by up to, on average, [Formula: see text] when their forearms were moved for them rather than when they controlled their forearms' movements. Task directionality and position identification mode did not significantly affect participant arm mirroring accuracy. These findings are relevant for interpreting in future work the reason why impairments occur, on similar tasks, in individuals with altered motor commands, working memory, and arm impedance, e.g., post-stroke hemiparesis.

Representativeness of Scholander chamber measurements as a function of time after leaf removal

The objectives of the study were to determine the maximum tolerable period between leaf removal from a plant and the determination of leaf water potential (Ψ _l ), and to evaluate different ways of packing leaves after their removal from a plant (LR). Two experiments (Exp) were conducted in a randomized complete block design with four replicates, using bean, maize, and soybean crops. The predawn leaf water potential (Ψ _PD ) (Exp1) and the minimum leaf water potential (Ψ _min ) (Exp1 and Exp2) were determined. The maximum tolerable period between LR varied among species, ranging from 80 to 90 min for bean and soybean, and from 40 to 60 min for maize. A box with low interior air temperature and capable of blocking the effect of solar radiation reduced variation in leaf water potential. These results indicate that the determination of the Ψ _l in bean, soybean, and maize crops does not need to be carried out immediately after leaf removal.

Practical Approaches to Evaluating Progress and Outcomes in Community-Wide Teen Pregnancy Prevention Initiatives

This paper presents an overview of the key evaluation components for a set of community-wide teen pregnancy prevention initiatives. We first describe the performance measures selected to assess progress toward meeting short-term objectives on the reach and quality of implementation of evidence-based teen pregnancy prevention interventions and adolescent reproductive health services. Next, we describe an evaluation that will compare teen birth rates in intervention communities relative to synthetic control communities. Synthetic controls are developed via a data-driven technique that constructs control communities by combining information from a pool of communities that are similar to the intervention community. Finally, we share lessons learned thus far in the evaluation of the project, with a focus on those lessons that may be valuable for local communities evaluating efforts to reduce teen pregnancy.

What can a participatory approach to evaluation contribute to the field of integrated care?

Better integration of care within the health sector and between health and social care is seen in many countries as an essential way of addressing the enduring problems of dwindling resources, changing demographics and unacceptable variation in quality of care. Current research evidence about the effectiveness of integration efforts supports neither the enthusiasm of those promoting and designing integrated care programmes nor the growing efforts of practitioners attempting to integrate care on the ground. In this paper we present a methodological approach, based on the principles of participatory research, that attempts to address this challenge. Participatory approaches are characterised by a desire to use social science methods to solve practical problems and a commitment on the part of researchers to substantive and sustained collaboration with relevant stakeholders. We describe how we applied an emerging practical model of participatory research, the researcher-in-residence model, to evaluate a large-scale integrated care programme in the UK. We propose that the approach added value to the programme in a number of ways: by engaging stakeholders in using established evidence and with the benefits of rigorously evaluating their work, by providing insights for local stakeholders that they were either not familiar with or had not fully considered in relation to the development and implementation of the programme and by challenging established mindsets and norms. While there is still much to learn about the benefits and challenges of applying participatory approaches in the health sector, we demonstrate how using such approaches have the potential to help practitioners integrate care more effectively in their daily practice and help progress the academic study of integrated care.

Individuals with chronic hemiparetic stroke can correctly match forearm positions within a single arm

OBJECTIVE

Previous studies determined, using between arms position matching assessments, that at least one-half of individuals with stroke have an impaired position sense. We investigated whether individuals with chronic stroke who have impairments mirroring arm positions also have impairments identifying the location of each arm in space.

METHODS

Participants with chronic hemiparetic stroke and age-matched participants without neurological impairments (controls) performed a between forearms position matching task based on a clinical assessment and a single forearm position matching task, using passive and active movements, based on a robotic assessment.

RESULTS

12 out of our 14 participants with stroke who had clinically determined between forearms position matching impairments had greater errors than the controls in both their paretic and non-paretic arm when matching positions during passive movements; yet stroke participants performed comparable to the controls during active movements.

CONCLUSIONS

Many individuals with chronic stroke may have impairments matching positions in both their paretic and non-paretic arm if their arm is moved for them, yet not within either arm if these individuals control their own movements.

SIGNIFICANCE

The neural mechanisms governing arm location perception in the stroke population may differ depending on whether arm movements are made passively versus actively.

Balancing stakeholder needs in the evaluation of healthcare quality improvement

Quality improvement (QI) efforts affect a broader range of people than we often assume. These are the potential stakeholders for QI and its evaluation, and they have valuable perspectives to offer when they are consulted in planning, conducting and interpreting evaluations. QI practitioners are accustomed to consulting stakeholders to assess unintended consequences or assess patient experiences of care, but in many cases there are additional benefits to a broad inclusion of stakeholders. These benefits are better adherence to ethical standards, to assure that all legitimate interests take part, more useful and relevant evaluation information and better political buy-in to improve impact. Balancing various stakeholder needs for information requires skill for both politics and research management. These challenges have few pat answers, but several preferred practices, which are illustrated with practical examples.

Patient safety climate strength: a concept that requires more attention

Background

When patient safety climate (PSC) surveys are used in healthcare, reporting typically focuses on PSC level (mean or per cent positive scores). This paper explores how an additional focus on PSC strength can enhance the utility of PSC survey data.

Setting and participants

442 care providers from 24 emergency departments (EDs) across Canada.

Methods

We use anonymised data from the Can-PSCS PSC instrument collected in 2011 as part of the Qmentum accreditation programme. We examine differences in climate strength across EDs using the Rwg(j) and intraclass correlation coefficients measures of inter-rater agreement.

Results

Across the six survey dimensions, median Rwg(j) was sufficiently high to support shared climate perceptions (0.64–0.83), but varied widely across the 24 ED units. We provide an illustrative example showing vastly different climate strength (Rwg(j) range=0.17–0.86) for units with an equivalent level of PSC (eg, climate mean score=3).

Conclusions

Most PSC survey results focus solely on climate level. To facilitate improvement in PSC, we advocate a simple, holistic safety climate profile including three metrics: climate level (using mean or per cent positive climate scores), climate strength (using the Rwg(j), or SD as a proxy) and the shape of the distribution (using histograms to see the distribution of scores within units). In PSC research, we advocate paying attention to climate strength as an important variable in its own right. Focusing on PSC level and strength can further understanding of the extent to which PSC is a key variable in the domain of patient safety.

Secular trends and evaluation of complex interventions: the rising tide phenomenon

Evaluations of service delivery interventions with contemporaneous controls often yield null results, even when the intervention appeared promising in advance. There can be many reasons for null results. In this paper we introduce the concept of a ‘rising tide’ phenomenon being a possible explanation of null results. We note that evaluations of service delivery interventions often occur when awareness of the problems they intend to address is already heightened, and pressure to tackle them is mounting throughout a health system. An evaluation may therefore take place in a setting where the system as a whole is improving – where there is a pronounced temporal trend or a ‘rising tide causing all vessels to rise’. As a consequence, control sites in an intervention study will improve. This reduces the difference between intervention and control sites and predisposes the study to a null result, leading to the conclusion that the intervention has no effect. We discuss how a rising tide may be distinguished from other causes of improvement in both control and intervention groups, and give examples where the rising tide provides a convincing explanation of such a finding. We offer recommendations for interpretation of research findings where improvements in the intervention group are matched by improvements in the control group. Understanding the rising tide phenomenon is important for a more nuanced interpretation of null results arising in the context of system-wide improvement. Recognition that a rising tide may have predisposed to a null result in one health system cautions against generalising the result to another health system where strong secular trends are absent.

Development of the Quality Improvement Minimum Quality Criteria Set (QI-MQCS): a tool for critical appraisal of quality improvement intervention publications

Objective

Valid, reliable critical appraisal tools advance quality improvement (QI) intervention impacts by helping stakeholders identify higher quality studies. QI approaches are diverse and differ from clinical interventions. Widely used critical appraisal instruments do not take unique QI features into account and existing QI tools (eg, Standards for QI Reporting Excellence) are intended for publication guidance rather than critical appraisal. This study developed and psychometrically tested a critical appraisal instrument, the QI Minimum Quality Criteria Set (QI-MQCS) for assessing QI-specific features of QI publications.

Methods

Approaches to developing the tool and ensuring validity included a literature review, in-person and online survey expert panel input, and application to empirical examples. We investigated psychometric properties in a set of diverse QI publications (N=54) by analysing reliability measures and item endorsement rates and explored sources of disagreement between reviewers.

Results

The QI-MQCS includes 16 content domains to evaluate QI intervention publications: Organisational Motivation, Intervention Rationale, Intervention Description, Organisational Characteristics, Implementation, Study Design, Comparator Description, Data Sources, Timing, Adherence/Fidelity, Health Outcomes, Organisational Readiness, Penetration/Reach, Sustainability, Spread and Limitations. Median inter-rater agreement for QI-MQCS items was κ 0.57 (83% agreement). Item statistics indicated sufficient ability to differentiate between publications (median quality criteria met 67%). Internal consistency measures indicated coherence without excessive conceptual overlap (absolute mean interitem correlation=0.19). The critical appraisal instrument is accompanied by a user manual detailing What to consider, Where to look and How to rate.

Conclusions

We developed a ready-to-use, valid and reliable critical appraisal instrument applicable to healthcare QI intervention publications, but recognise scope for continuing refinement.

How to build up the actionable knowledge base: the role of 'best fit' framework synthesis for studies of improvement in healthcare

Increasing recognition of the role and value of theory in improvement work in healthcare offers the prospect of capitalising upon, and consolidating, actionable lessons from synthesis of improvement projects and initiatives. We propose that informed use of theory can (i) provide a mechanism by which to collect and organise data from a body of improvement work, (ii) offer a framework for analysis and identification of lessons learnt and (iii) facilitate an evaluation of the feasibility, effectiveness and acceptability of improvement programmes. Improvement practitioners can benefit from using an underpinning external structure as a lens by which to examine the specific achievements of their own projects alongside comparable initiatives led by others. We demonstrate the utility of a method known as ‘best fit framework synthesis’ (BFFS) in offering a ubiquitous and versatile means by which to collect, analyse and evaluate improvement work in healthcare. First reported in 2011, BFFS represents a pragmatic, flexible approach to integrating theory with findings from practice. A deductive phase, where a review team seeks to accommodate a substantial part of the data, is followed by an inductive phase, in which the team explores data not accommodated by the framework. We explore the potential for BFFS within improvement work by drawing upon the evidence synthesis methodology literature and practical examples of improvement work reported in BMJ Quality and Safety (2011–2015). We suggest four variants of BFFS that may have particular value in synthesising a body of improvement work. We conclude that BFFS, alongside other approaches that seek to optimise the contribution of theory to improvement work, represents one important enabling mechanism by which to establish the rigour and scientific credentials of the emerging discipline of ‘improvement science’.

How to study improvement interventions: a brief overview of possible study types

Improvement (defined broadly as purposive efforts to secure positive change) has become an increasingly important activity and field of inquiry within healthcare. This article offers an overview of possible methods for the study of improvement interventions. The choice of available designs is wide, but debates continue about how far improvement efforts can be simultaneously practical (aimed at producing change) and scientific (aimed at producing new knowledge), and whether the distinction between the practical and the scientific is a real and useful one. Quality improvement projects tend to be applied and, in some senses, self-evaluating. They are not necessarily directed at generating new knowledge, but reports of such projects if well conducted and cautious in their inferences may be of considerable value. They can be distinguished heuristically from research studies, which are motivated by and set out explicitly to test a hypothesis, or otherwise generate new knowledge, and from formal evaluations of improvement projects. We discuss variants of trial designs, quasi-experimental designs, systematic reviews, programme evaluations, process evaluations, qualitative studies, and economic evaluations. We note that designs that are better suited to the evaluation of clearly defined and static interventions may be adopted without giving sufficient attention to the challenges associated with the dynamic nature of improvement interventions and their interactions with contextual factors. Reconciling pragmatism and research rigour is highly desirable in the study of improvement. Trade-offs need to be made wisely, taking into account the objectives involved and inferences to be made.

Integrating empowerment evaluation and quality improvement to achieve healthcare improvement outcomes

While the body of evidence-based healthcare interventions grows, the ability of health systems to deliver these interventions effectively and efficiently lags behind. Quality improvement approaches, such as the model for improvement, have demonstrated some success in healthcare but their impact has been lessened by implementation challenges. To help address these challenges, we describe the empowerment evaluation approach that has been developed by programme evaluators and a method for its application (Getting To Outcomes (GTO)). We then describe how GTO can be used to implement healthcare interventions. An illustrative healthcare quality improvement example that compares the model for improvement and the GTO method for reducing hospital admissions through improved diabetes care is described. We conclude with suggestions for integrating GTO and the model for improvement.

How can healthcare standards be standardised?

International travel, medical tourism and trade have created a demand for reliable assessment of healthcare provision across borders, and for information which is accessible to patients, insurers and referring institutions. External assessment schemes for healthcare providers may be clustered into three types: statutory regulation and institutional licensing, International Standardization Organisation certification, and voluntary systems such as peer review and healthcare accreditation. Increasing complexity of healthcare provision, pressures for public accountability and expectations of professional self-governance place a burden on the inspectors and the inspected. If only to contain costs of external assessment and to increase access to reliable information for patients and insurers, the three approaches must work together rather than compete. This paper summarises the origins, aims, authority and methods of the three general models, describing current pressures and opportunities for convergence (between systems and across borders) in the UK and in Europe.

What to expect when you're evaluating healthcare improvement: a concordat approach to managing collaboration and uncomfortable realities

Evaluation of improvement initiatives in healthcare is essential to establishing whether interventions are effective and to understanding how and why they work in order to enable replication. Although valuable, evaluation is often complicated by tensions and friction between evaluators, implementers and other stakeholders. Drawing on the literature, we suggest that these tensions can arise from a lack of shared understanding of the goals of the evaluation; confusion about roles, relationships and responsibilities; data burdens; issues of data flows and confidentiality; the discomforts of being studied and the impact of disappointing or otherwise unwelcome results. We present a possible approach to managing these tensions involving the co-production and use of a concordat. We describe how we developed a concordat in the context of an evaluation of a complex patient safety improvement programme known as Safer Clinical Systems Phase 2. The concordat development process involved partners (evaluators, designers, funders and others) working together at the outset of the project to agree a set of principles to guide the conduct of the evaluation. We suggest that while the concordat is a useful resource for resolving conflicts that arise during evaluation, the process of producing it is perhaps even more important, helping to make explicit unspoken assumptions, clarify roles and responsibilities, build trust and establish open dialogue and shared understanding. The concordat we developed established some core principles that may be of value for others involved in evaluation to consider. But rather than seeing our document as a ready-made solution, there is a need for recognition of the value of the process of co-producing a locally agreed concordat in enabling partners in the evaluation to work together effectively.

How to study improvement interventions: a brief overview of possible study types

Improvement (defined broadly as purposive efforts to secure positive change) has become an increasingly important activity and field of inquiry within healthcare. This article offers an overview of possible methods for the study of improvement interventions. The choice of available designs is wide, but debates continue about how far improvement efforts can be simultaneously practical (aimed at producing change) and scientific (aimed at producing new knowledge), and whether the distinction between the practical and the scientific is a real and useful one. Quality improvement projects tend to be applied and, in some senses, self-evaluating. They are not necessarily directed at generating new knowledge, but reports of such projects if well conducted and cautious in their inferences may be of considerable value. They can be distinguished heuristically from research studies, which are motivated by and set out explicitly to test a hypothesis, or otherwise generate new knowledge, and from formal evaluations of improvement projects. We discuss variants of trial designs, quasi-experimental designs, systematic reviews, programme evaluations, process evaluations, qualitative studies, and economic evaluations. We note that designs that are better suited to the evaluation of clearly defined and static interventions may be adopted without giving sufficient attention to the challenges associated with the dynamic nature of improvement interventions and their interactions with contextual factors. Reconciling pragmatism and research rigour is highly desirable in the study of improvement. Trade-offs need to be made wisely, taking into account the objectives involved and inferences to be made.

Demystifying theory and its use in improvement

The role and value of theory in improvement work in healthcare has been seriously underrecognised. We join others in proposing that more informed use of theory can strengthen improvement programmes and facilitate the evaluation of their effectiveness. Many professionals, including improvement practitioners, are unfortunately mystified-and alienated-by theory, which discourages them from using it in their work. In an effort to demystify theory we make the point in this paper that, far from being discretionary or superfluous, theory ('reason-giving'), both informal and formal, is intimately woven into virtually all human endeavour. We explore the special characteristics of grand, mid-range and programme theory; consider the consequences of misusing theory or failing to use it; review the process of developing and applying programme theory; examine some emerging criteria of 'good' theory; and emphasise the value, as well as the challenge, of combining informal experience-based theory with formal, publicly developed theory. We conclude that although informal theory is always at work in improvement, practitioners are often not aware of it or do not make it explicit. The germane issue for improvement practitioners, therefore, is not whether they use theory but whether they make explicit the particular theory or theories, informal and formal, they actually use.

Designing quality improvement initiatives: the action effect method, a structured approach to identifying and articulating programme theory

BACKGROUND

The identification and articulation of programme theory can support effective design, execution and evaluation of quality improvement (QI) initiatives. Programme theory includes an agreed aim, potential interventions to achieve this aim, anticipated cause/effect relationships between the interventions and the aim and measures to monitor improvement. This paper outlines the approach used in a research and improvement programme to support QI initiatives in identifying and articulating programme theory: the action effect method.

BACKGROUND TO METHOD DEVELOPMENT

Building on a previously used QI method, the driver diagram, the action effect method was developed using co-design and iteration over four annual rounds of improvement initiatives. This resulted in a specification of the elements required to fully articulate the programme theory of a QI initiative.

THE ACTION EFFECT METHOD

The action effect method is a systematic and structured process to identify and articulate a QI initiative's programme theory. The method connects potential interventions and implementation activities with an overall improvement aim through a diagrammatic representation of hypothesised and evidenced cause/effect relationships. Measure concepts, in terms of service delivery and patient and system outcomes, are identified to support evaluation.

DISCUSSION AND CONCLUSIONS

The action effect method provides a framework to guide the execution and evaluation of a QI initiative, a focal point for other QI methods and a communication tool to engage stakeholders. A clear definition of what constitutes a well-articulated programme theory is provided to guide the use of the method and assessment of the fidelity of its application.

Hospital performance based on treatment delays: comparison of ranking methods

BACKGROUND

Reducing time-to-care is crucial in many acute and chronic diseases. Quality indicators based on target delays derived from guidelines are used to compare hospital performance but there is no accepted methodology for comparing performance when no target delay has been established.

AIM

To explore by different statistical methods the uncertainty in hospital comparisons that are based on delay indicators, when no target delay is available.

METHODS

Data for hospital door-to-needle time were extracted from a 2010 study of 1699 patients in 57 hospitals with ST-elevated myocardial infarction. We determined whether the times in each hospital were statistically different from the overall mean time or the median time for all hospitals by (i) one-way analysis of variance (ANOVA), (ii) non-parametric ANOVA with Nelson-Hsu adjustment (ANOVA R) and (iii) the proportional hazard model (PHM). We also tested for the assumptions underlying the methods: normal distribution for ANOVA, homogeneity of variances (homoscedasticity) for ANOVA and ANOVA R, and proportionality for PHM.

RESULTS

Door-to-needle times were available for 889 patients in 44 hospitals. Data distribution was not Gaussian. Test assumptions were verified for ANOVA R (homoscedasticity) for one data subset (>48-h times (48H) excluded) and for PHM (proportionality) for two data subsets (48H or >95th percentile (P95) times excluded). The same five significantly better performers were identified in each case (although ANOVA R missed one). ANOVA R (48H) identified two significantly poorer performers, PHM (48H) identified three and PHM (P95) just one. Poorer performers differed according to method.

CONCLUSIONS

The tested statistical methods yielded broadly similar results but no method was truly satisfactory. A transparency statement should therefore always specify the ranking method used to compare hospital performance.

A case report of evaluating a large-scale health systems improvement project in an uncontrolled setting: a quality improvement initiative in KwaZulu-Natal, South Africa

OBJECTIVE

New approaches are needed to evaluate quality improvement (QI) within large-scale public health efforts. This case report details challenges to large-scale QI evaluation, and proposes solutions relying on adaptive study design.

STUDY DESIGN

We used two sequential evaluative methods to study a QI effort to improve delivery of HIV preventive care in public health facilities in three districts in KwaZulu-Natal, South Africa, over a 3-year period. We initially used a cluster randomised controlled trial (RCT) design.

PRINCIPAL FINDINGS

During the RCT study period, tensions arose between intervention implementation and evaluation design due to loss of integrity of the randomisation unit over time, pressure to implement changes across the randomisation unit boundaries, and use of administrative rather than functional structures for the randomisation. In response to this loss of design integrity, we switched to a more flexible intervention design and a mixed-methods quasiexperimental evaluation relying on both a qualitative analysis and an interrupted time series quantitative analysis.

CONCLUSIONS

Cluster RCT designs may not be optimal for evaluating complex interventions to improve implementation in uncontrolled 'real world' settings. More flexible, context-sensitive evaluation designs offer a better balance of the need to adjust the intervention during the evaluation to meet implementation challenges while providing the data required to evaluate effectiveness. Our case study involved HIV care in a resource-limited setting, but these issues likely apply to complex improvement interventions in other settings.