A coproduced patient and public event: An approach to developing and prioritizing ambulance performance measures

The effectiveness of knowledge-sharing techniques and approaches in research funded by the National Institute for Health and Care Research (NIHR): a systematic review

BACKGROUND

The National Institute of Health and Care Research (NIHR), funds, enables and delivers world-leading health and social care research to improve people's health and wellbeing. To achieve this aim, effective knowledge sharing (two-way knowledge sharing between researchers and stakeholders to create new knowledge and enable change in policy and practice) is needed. To date, it is not known which knowledge sharing techniques and approaches are used or how effective these are in creating new knowledge that can lead to changes in policy and practice in NIHR funded studies.

METHODS

In this restricted systematic review, electronic databases [MEDLINE, The Health Management Information Consortium (including the Department of Health's Library and Information Services and King's Fund Information and Library Services)] were searched for published NIHR funded studies that described knowledge sharing between researchers and other stakeholders. One researcher performed title and abstract, full paper screening and quality assessment (Critical Appraisal Skills Programme qualitative checklist) with a 20% sample independently screened by a second reviewer. A narrative synthesis was adopted.

RESULTS

In total 9897 records were identified. After screening, 17 studies were included. Five explicit forms of knowledge sharing studies were identified: embedded models, knowledge brokering, stakeholder engagement and involvement of non-researchers in the research or service design process and organisational collaborative partnerships between universities and healthcare organisations. Collectively, the techniques and approaches included five types of stakeholders and worked with them at all stages of the research cycle, except the stage of formation of the research design and preparation of funding application. Seven studies (using four of the approaches) gave examples of new knowledge creation, but only one study (using an embedded model approach) gave an example of a resulting change in practice. The use of a theory, model or framework to explain the knowledge sharing process was identified in six studies.

CONCLUSIONS

Five knowledge sharing techniques and approaches were reported in the included NIHR funded studies, and seven studies identified the creation of new knowledge. However, there was little investigation of the effectiveness of these approaches in influencing change in practice or policy.

Methods to Generate Innovative Research Ideas and Improve Patient and Public Involvement in Modern Epidemiological Research: Review, Patient Viewpoint, and Guidelines for Implementation of a Digital Cohort Study

Background

Patient and public involvement (PPI) in research aims to increase the quality and relevance of research by incorporating the perspective of those ultimately affected by the research. Despite these potential benefits, PPI is rarely included in epidemiology protocols.

Objective

The aim of this study is to provide an overview of methods used for PPI and offer practical recommendations for its efficient implementation in epidemiological research.

Methods

We conducted a review on PPI methods. We mirrored it with a patient advocate’s viewpoint about PPI. We then identified key steps to optimize PPI in epidemiological research based on our review and the viewpoint of the patient advocate, taking into account the identification of barriers to, and facilitators of, PPI. From these, we provided practical recommendations to launch a patient-centered cohort study. We used the implementation of a new digital cohort study as an exemplary use case.

Results

We analyzed data from 97 studies, of which 58 (60%) were performed in the United Kingdom. The most common methods were workshops (47/97, 48%); surveys (33/97, 34%); meetings, events, or conferences (28/97, 29%); focus groups (25/97, 26%); interviews (23/97, 24%); consensus techniques (8/97, 8%); James Lind Alliance consensus technique (7/97, 7%); social media analysis (6/97, 6%); and experience-based co-design (3/97, 3%). The viewpoint of a patient advocate showed a strong interest in participating in research. The most usual PPI modalities were research ideas (60/97, 62%), co-design (42/97, 43%), defining priorities (31/97, 32%), and participation in data analysis (25/97, 26%). We identified 9 general recommendations and 32 key PPI-related steps that can serve as guidelines to increase the relevance of epidemiological studies.

Conclusions

PPI is a project within a project that contributes to improving knowledge and increasing the relevance of research. PPI methods are mainly used for idea generation. On the basis of our review and case study, we recommend that PPI be included at an early stage and throughout the research cycle and that methods be combined for generation of new ideas. For e-cohorts, the use of digital tools is essential to scale up PPI. We encourage investigators to rely on our practical recommendations to extend PPI in future epidemiological studies.

Patients as partners in health research: A scoping review

BACKGROUND

The role of patient involvement in health research has evolved over the past decade. Despite efforts to engage patients as partners, the role is not well understood. We undertook this review to understand the engagement practices of patients who assume roles as partners in health research.

METHODS

Using a recognized methodological approach, two academic databases (MEDLINE and EMBASE) and grey literature sources were searched. Findings were organized into one of the three higher levels of engagement, described by the Patient and Researcher Engagement framework developed by Manafo. We examined and quantified the supportive strategies used during involvement, used thematic analysis as described by Braun and Clarke and themed the purpose of engagement, and categorized the reported outcomes according to the CIHR Engagement Framework.

RESULTS

Out of 6621 records, 119 sources were included in the review. Thematic analysis of the purpose of engagement revealed five themes: documenting and advancing PPI, relevance of research, co-building, capacity building and impact on research. Improved research design was the most common reported outcome and the most common role for patient partners was as members of the research team, and the most commonly used strategy to support involvement was by meetings.

CONCLUSION

The evidence collected during this review advanced our understanding of the engagement of patients as research partners. As patient involvement becomes more mainstream, this knowledge will aid researchers and policy-makers in the development of approaches and tools to support engagement.

PATIENT/USER INVOLVEMENT

Patients led and conducted the grey literature search, including the synthesis and interpretation of the findings.

Evaluating health service outcomes of public involvement in health service design in high-income countries: a systematic review

BACKGROUND

Internationally, it is expected that health services will involve the public in health service design. Evaluation of public involvement has typically focused on the process and experiences for participants. Less is known about outcomes for health services. The aim of this systematic review was to a) identify and synthesise what is known about health service outcomes of public involvement and b) document how outcomes were evaluated.

METHODS

Searches were undertaken in MEDLINE, EMBASE, The Cochrane Library, PsycINFO, Web of Science, and CINAHL for studies that reported health service outcomes from public involvement in health service design. The review was limited to high-income countries and studies in English. Study quality was assessed using the Mixed Methods Appraisal Tool and critical appraisal guidelines for assessing the quality and impact of user involvement in health research. Content analysis was used to determine the outcomes of public involvement in health service design and how outcomes were evaluated.

RESULTS

A total of 93 articles were included. The majority were published in the last 5 years, were qualitative, and were located in the United Kingdom. A range of health service outcomes (discrete products, improvements to health services and system/policy level changes) were reported at various levels (service level, across services, and across organisations). However, evaluations of outcomes were reported in less than half of studies. In studies where outcomes were evaluated, a range of methods were used; most frequent were mixed methods. The quality of study design and reporting was inconsistent.

CONCLUSION

When reporting public involvement in health service design authors outline a range of outcomes for health services, but it is challenging to determine the extent of outcomes due to inadequate descriptions of study design and poor reporting. There is an urgent need for evaluations, including longitudinal study designs and cost-benefit analyses, to fully understand outcomes from public involvement in health service design.

Developing new ways of measuring the quality and impact of ambulance service care: the PhOEBE mixed-methods research programme

Background

Ambulance service quality measures have focused on response times and a small number of emergency conditions, such as cardiac arrest. These quality measures do not reflect the care for the wide range of problems that ambulance services respond to and the Prehospital Outcomes for Evidence Based Evaluation (PhOEBE) programme sought to address this.

Objectives

The aim was to develop new ways of measuring the impact of ambulance service care by reviewing and synthesising literature on prehospital ambulance outcome measures and using consensus methods to identify measures for further development; creating a data set linking routinely collected ambulance service, hospital and mortality data; and using the linked data to explore the development of case-mix adjustment models to assess differences or changes in processes and outcomes resulting from ambulance service care.

Design

A mixed-methods study using a systematic review and synthesis of performance and outcome measures reported in policy and research literature; qualitative interviews with ambulance service users; a three-stage consensus process to identify candidate indicators; the creation of a data set linking ambulance, hospital and mortality data; and statistical modelling of the linked data set to produce novel case-mix adjustment measures of ambulance service quality.

Setting

East Midlands and Yorkshire, England.

Participants

Ambulance services, patients, public, emergency care clinical academics, commissioners and policy-makers between 2011 and 2015.

Interventions

None.

Main outcome measures

Ambulance performance and quality measures.

Data sources

Ambulance call-and-dispatch and electronic patient report forms, Hospital Episode Statistics, accident and emergency and inpatient data, and Office for National Statistics mortality data.

Results

Seventy-two candidate measures were generated from systematic reviews in four categories: (1) ambulance service operations (n = 14), (2) clinical management of patients (n = 20), (3) impact of care on patients (n = 9) and (4) time measures (n = 29). The most common operations measures were call triage accuracy; clinical management was adherence to care protocols, and for patient outcome it was survival measures. Excluding time measures, nine measures were highly prioritised by participants taking part in the consensus event, including measures relating to pain, patient experience, accuracy of dispatch decisions and patient safety. Twenty experts participated in two Delphi rounds to refine and prioritise measures and 20 measures scored ≥ 8/9 points, which indicated good consensus. Eighteen patient and public representatives attending a consensus workshop identified six measures as important: time to definitive care, response time, reduction in pain score, calls correctly prioritised to appropriate levels of response, proportion of patients with a specific condition who are treated in accordance with established guidelines, and survival to hospital discharge for treatable emergency conditions. From this we developed six new potential indicators using the linked data set, of which five were constructed using case-mix-adjusted predictive models: (1) mean change in pain score; (2) proportion of serious emergency conditions correctly identified at the time of the 999 call; (3) response time (unadjusted); (4) proportion of decisions to leave a patient at scene that were potentially inappropriate; (5) proportion of patients transported to the emergency department by 999 emergency ambulance who did not require treatment or investigation(s); and (6) proportion of ambulance patients with a serious emergency condition who survive to admission, and to 7 days post admission. Two indicators (pain score and response times) did not need case-mix adjustment. Among the four adjusted indicators, we found that accuracy of call triage was 61%, rate of potentially inappropriate decisions to leave at home was 5–10%, unnecessary transport to hospital was 1.7–19.2% and survival to hospital admission was 89.5–96.4% depending on Clinical Commissioning Group area. We were unable to complete a fourth objective to test the indicators in use because of delays in obtaining data. An economic analysis using indicators (4) and (5) showed that incorrect decisions resulted in higher costs.

Limitations

Creation of a linked data set was complex and time-consuming and data quality was variable. Construction of the indicators was also complex and revealed the effects of other services on outcome, which limits comparisons between services.

Conclusions

We identified and prioritised, through consensus processes, a set of potential ambulance service quality measures that reflected preferences of services and users. Together, these encompass a broad range of domains relevant to the population using the emergency ambulance service. The quality measures can be used to compare ambulance services or regions or measure performance over time if there are improvements in mechanisms for linking data across services.

Future work

The new measures can be used to assess different dimensions of ambulance service delivery but current data challenges prohibit routine use. There are opportunities to improve data linkage processes and to further develop, validate and simplify these measures.

Funding

The National Institute for Health Research Programme Grants for Applied Research programme.

"About sixty per cent I want to do it": Health researchers' attitudes to, and experiences of, patient and public involvement (PPI)-A qualitative interview study

Background

Funders, policy‐makers and research organizations increasingly expect health researchers in the UK to involve patients and members of the public in research. It has been stated that it makes research “more effective, more credible and often more cost efficient.” However, the evidence base for this assertion is evolving and can be limited. There has been little research into how health researchers feel about involving people, how they go about it, how they manage formal policy rhetoric, and what happens in practice.

Objective

To explore researchers’ experiences and perceptions of patient and public involvement (PPI).

Methods

Semi‐structured interview study of 36 health researchers (both clinical and non‐clinical), with data collection and thematic analysis informed by the theoretical domains framework.

Results

In the course of our analysis, we developed four themes that encapsulate the participants’ experiences and perceptions of PPI. Participants expressed ambivalence, cynicism and enthusiasm about PPI, an activity that creates emotional labour, which is both rewarding and burdensome and requires practical and social support. It is operationalized in an academic context influenced by power and incentives.

Discussion and conclusions

Researchers’ experiences and attitudes towards patient and public involvement are a key factor in the successful embedding of involvement within the wider research culture. We call for a culture change that supports the development of effective organizational approach to support involvement.

Quality Is in the Eye of the Beholder-A Focus Group Study from the Perspective of Ambulance Clinicians, Physicians, and Managers

Quality within all areas of healthcare should be systemically monitored and ensured. However, the definition of quality is complex and diverse. In the ambulance service (AS), quality has traditionally been defined as response time, but this measurement eliminates the possibility of addressing other characteristics of quality, such as the care provided. This study aimed to explore what constitutes quality in the context of the ambulance service as experienced by ambulance clinicians, physicians, and managers. A focus group study was conducted with 18 participants. The three focus groups were analyzed with the focus group method developed by Kreuger and Casey. The participants highlighted patient involvement, information and care, as well as adherence to policies, regulations, and their own standards as representing quality in the AS. This study demonstrates that quality is in the eye of the beholder. As quality seems to be viewed similarly by patients and ambulance clinicians, physicians, and managers, stakeholders should aim for a paradigm shift where patients' experience of the care is just as important as various time measures.

A coproduced patient and public event: An approach to developing and prioritizing ambulance performance measures

BACKGROUND

Patient and public involvement (PPI) is recognized as an important component of high-quality health services research. PPI is integral to the Pre-hospital Outcomes for Evidence Based Evaluation (PhOEBE) programme. The PPI event described in detail in this article focusses on the process of involving patients and public representatives in identifying, prioritizing and refining a set of outcome measures that can be used to support ambulance service performance measurement.

OBJECTIVE

To obtain public feedback on little known, complex aspects of ambulance service performance measurement.

DESIGN

The event was codesigned and coproduced with the PhOEBE PPI reference group and PhOEBE research team. The event consisted of brief researcher-led presentations, group discussions facilitated by the PPI reference group members and electronic voting.

SETTING AND PARTICIPANTS

Data were collected from eighteen patient and public representatives who attended an event venue in Yorkshire.

RESULTS

The results of the PPI event showed that this interactive format and mode of delivery was an effective method to obtain public feedback and produced a clear indication of which ambulance performance measures were most highly favoured by event participants.

DISCUSSION AND CONCLUSIONS

The event highlighted valuable contributions the PPI reference group made to the design process, supporting participant recruitment and facilitation of group discussions. In addition, the positive team working experience of the event proved a catalyst for further improvements in PPI within the PhOEBE project.

Prioritizing novel and existing ambulance performance measures through expert and lay consensus: A three-stage multimethod consensus study

Background

Current ambulance quality and performance measures, such as response times, do not reflect the wider scope of care that services now provide. Using a three‐stage consensus process, we aimed to identify new ways of measuring ambulance service quality and performance that represent service provider and public perspectives.

Design

A multistakeholder consensus event, modified Delphi study, and patient and public consensus workshop.

Setting and participants

Representatives from ambulance services, patient and public involvement (PPI) groups, emergency care clinical academics, commissioners and policymakers.

Results

Nine measures/principles were highly prioritized by >75% of consensus event participants, including measures relating to pain, patient experience, accuracy of dispatch decisions and patient safety. Twenty experts participated in two Delphi rounds to further refine and prioritize measures; 20 measures in three domains scored ≥8/9, indicating good consensus, including proportion of calls correctly prioritized, time to definitive care and measures related to pain. Eighteen patient/public representatives attended a consensus workshop, and six measures were identified as important. These include time to definitive care, response time, reduction in pain scores, calls correctly prioritized to appropriate levels of response and survival to hospital discharge for treatable emergency conditions.

Conclusions

Using consensus methods, we identified a shortlist of ambulance outcome and performance measures that are important to ambulance clinicians and service providers, service users, commissioners, and clinical academics, reflecting current pre‐hospital ambulance care and services. The measures can potentially be used to assess pre‐hospital quality or performance over time, with most calculated using routinely available data.