A Systematic Review of Measurement Tools for the Proactive Assessment of Patient Safety in General Practice

Comparing rates of adverse events detected in incident reporting and the Global Trigger Tool: a systematic review

Many hospitals continue to use incident reporting systems (IRSs) as their primary patient safety data source. The information IRSs collect on the frequency of harm to patients [adverse events (AEs)] is generally of poor quality, and some incident types (e.g. diagnostic errors) are under-reported. Other methods of collecting patient safety information using medical record review, such as the Global Trigger Tool (GTT), have been developed. The aim of this study was to undertake a systematic review to empirically quantify the gap between the percentage of AEs detected using the GTT to those that are also detected via IRSs. The review was conducted in adherence to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. Studies published in English, which collected AE data using the GTT and IRSs, were included. In total, 14 studies met the inclusion criteria. All studies were undertaken in hospitals and were published between 2006 and 2022. The studies were conducted in six countries, mainly in the USA (nine studies). Studies reviewed 22 589 medical records using the GTT across 107 institutions finding 7166 AEs. The percentage of AEs detected using the GTT that were also detected in corresponding IRSs ranged from 0% to 37.4% with an average of 7.0% (SD 9.1; median 3.9 and IQR 5.2). Twelve of the fourteen studies found <10% of the AEs detected using the GTT were also found in corresponding IRSs. The >10-fold gap between the detection rates of the GTT and IRSs is strong evidence that the rate of AEs collected in IRSs in hospitals should not be used to measure or as a proxy for the level of safety of a hospital. IRSs should be recognized for their strengths which are to detect rare, serious, and new incident types and to enable analysis of contributing and contextual factors to develop preventive and corrective strategies. Health systems should use multiple patient safety data sources to prioritize interventions and promote a cycle of action and improvement based on data rather than merely just collecting and analysing information.

Development of the Self-Perceived Safety of Orthopedic Post-Surgery Inpatients (SPSOPSI) Scale

In recent decades, patient safety in orthopedics has gained increasing importance and has been regarded as a core concept of medical care quality. However, according to currently published studies, measurement instruments used to evaluate post-surgery orthopedic patient's perceptions are still very rare. This study aimed to develop a new measurement instrument, the self-perceived safety of orthopedic post-surgery inpatients (SPSOPSI) scale, which can provide healthcare workers with a better understanding of orthopedic patients' self-perceived safety and give more precise clinical suggestions. Item analysis and exploratory factor analysis (EFA) were used, and the results showed that the six-factor model is good-fit: root mean square residual (RMR) 0.00, root mean square error of approximation (RMSEA) 0.06, goodness-of-fit index (GFI) 0.90, comparative fit index (CFI) 0.98, incremental fit index (IFI) 0.98. The results showed the SPSOPSI scale is a valid and reliable tool for health care providers can use to evaluate orthopedic post-surgery patients' perceived safety.

Patients' perception of safety climate in Irish general practice: a cross-sectional study

BACKGROUND

Although patients have the potential to provide important information on patient safety, considerably fewer patient-report measures of safety climate (SC) have been applied in the primary care setting as compared to secondary care. Our aim was to examine the application of a patient-report measure of safety climate in an Irish population to understand patient perceptions of safety in general practice and identify potential areas for improvement. Specifically, our research questions were: 1. What are patients' perceptions of SC in Irish general practice? 2. Do patient risk factors impact perceptions of SC? 3. Do patient responses to an open-ended question about safety enhance our understanding of patient safety beyond that obtained from a quantitative measure of SC?

METHODS

The Patient Perspective of Safety in General Practice (PPS-GP) survey was distributed to primary care patients in Ireland. The survey consisted of both Likert-response items, and free-text entry questions in relation to the safety of care. A series of five separate hierarchical regressions were used to examine the relationship between a range of patient-related variables and each of the survey subscales. A deductive content analysis approach was used to code the free-text responses.

RESULTS

A total of 584 completed online and paper surveys were received. Respondents generally had positive perceptions of safety across all five SC subscales of the PPS-GP. Regarding patient risk factors, younger age and being of non-Irish nationality were consistently associated with more negative SC perceptions. Analysis of the free-text responses revealed considerably poorer patient perceptions (n = 85, 65.4%) of the safety experience in primary care.

CONCLUSION

Our findings indicate that despite being under-utilised, patients' perceptions are a valuable source of information for measuring SC, with promising implications for safety improvement in general practice. Further consideration should be given to how best to utilise this data in order to improve safety in primary care.

How safe is prehospital care? A systematic review

Background

As compared to other domains of healthcare, little is known about patient safety incidents (PSIs) in prehospital care. The aims of our systematic review were to identify how the prevalence and level of harm associated with PSIs in prehospital care are assessed; the frequency of PSIs in prehospital care; and the harm associated with PSIs in prehospital care.

Method

Searches were conducted of Medline, Web of Science, PsycInfo, CINAHL, Academic Search Complete and the grey literature. Reference lists of included studies and existing related reviews were also screened. English-language, peer-reviewed studies reporting data on number/frequency of PSIs and/or harm associated with PSIs were included. Two researchers independently extracted data from the studies and carried out a critical appraisal using the Quality Assessment Tool for Studies with Diverse Designs (QATSDD).

Results

Of the 22 included papers, 16 (73%) used data from record reviews, and 6 (27%) from incident reports. The frequency of PSIs in prehospital care was found to be a median of 5.9 per 100 records/transports/patients. A higher prevalence of PSIs was identified within studies that used record review data (9.9 per 100 records/transports/patients) as compared to incident reports (0.3 per records/transports/patients). Across the studies that reported harm, a median of 15.6% of PSIs were found to result in harm. Studies that utilized record review data reported that a median of 6.5% of the PSIs resulted in harm. For data from incident reporting systems, a median of 54.6% of incidents were associated with harm. The mean QATSDD score was 25.6 (SD = 4.1, range = 16–34).

Conclusions

This systematic review gives direction as to how to advance methods for identifying PSIs in prehospital care and assessing the extent to which patients are harmed.

A meta-review of methods of measuring and monitoring safety in primary care

Background

A major barrier to safety improvement in primary care is a lack of safety data. The aims of this systematic meta-review (registration: CRD42021224367) were to identify systematic reviews of studies that examine methods of measuring and monitoring safety in primary care; classify the methods of measuring and monitoring safety in the included systematic reviews using the five safety domains of Vincent et al.’s framework and use this information to make recommendations for improving the measurement and monitoring of safety in primary care.

Methods

Four databases (Medline, Academic Search Complete, Web of Science and CINAHL) and the grey literature were screened in November 2020, with searches updated in January 2021. Systematic reviews were included if they addressed the measurement of patient safety in primary care and were published in English. Studies were assessed using the Critical Appraisal Skills Programme for systematic reviews.

Results

A total of 6904 papers were screened, with 13 systematic reviews included. A commonly reported method of measuring ‘past harm’ was through patient record review. The most frequent methods for assessing the ‘reliability of safety critical processes’ were checklists, observations and surveys of staff. Methods used to assess ‘sensitivity to operations’ included observation, staff surveys, interviews, focus groups, active monitoring and simulated patients. Safety climate surveys were a commonly used as an approach to assess ‘anticipation and preparedness’. A number of the reviews concluded that safety data could, and should, be used for ‘integration and learning’. The main limitation of the meta-review was that it was of systematic reviews only.

Conclusions

Many of the methods for measuring and monitoring safety are readily available, quick to administer, do not require external involvement and are inexpensive. However, there is still a need to improve the psychometric properties of many measures. Researchers must support the development of psychometrically sound safety measures that do not over burden primary care practitioners. Policymakers must consider how primary care practitioners can be supported to implement these measures.

Analysis of Patient Safety Incidents in Primary Care Reported in an Electronic Registry Application

OBJECTIVES

(1) To describe the epidemiology of patient safety (PS) incidents registered in an electronic notification system in primary care (PC) health centres; (2) to define a risk map; and (3) to identify the critical areas where intervention is needed.

DESIGN

Descriptive analytical study of incidents reported from 1 January to 31 December 2018, on the TPSC Cloud™ platform (The Patient Safety Company) accessible from the corporate website (Intranet) of the regional public health service.

SETTING

24 Catalan Institute of Health PC health centres of the Tarragona region (Spain).

PARTICIPANTS

Professionals from the PC health centres and a Patient Safety Functional Unit.

MEASUREMENTS

Data obtained from records voluntarily submitted to an electronic, standardised and anonymised form. Data recorded: healthcare unit, notifier, type of incident, risk matrix, causal and contributing factors, preventability, level of resolution and improvement actions.

RESULTS

A total of 1544 reports were reviewed and 1129 PS incidents were analysed: 25.0% of incidents did not reach the patient; 66.5% reached the patient without causing harm, and 8.5% caused adverse events. Nurses provided half of the reports (48.5%), while doctors reported more adverse events (70.8%; p < 0.01). Of the 96 adverse events, 46.9% only required observation, 34.4% caused temporary damage that required treatment, 13.5% required (or prolonged) hospitalization, and 5.2% caused severe permanent damage and/or a situation close to death. Notably, 99.2% were considered preventable. The main critical areas were: communication (27.8%), clinical-administrative management (25.1%), care delivery (23.5%) and medicines (18.4%); few incidents were related to diagnosis (3.6%).

CONCLUSIONS

PS incident notification applications are adequate for reporting incidents and adverse events associated with healthcare. Approximately 75% and 10% of incidents reach the patient and cause some damage, respectively, and most cases are considered preventable. Adequate and strengthened risk management of critical areas is required to improve PS.

A comparative assessment of two tools designed to support patient safety culture in UK general practice

BACKGROUND

The NHS has recognised the importance of a high quality patient safety culture in the delivery of primary health care in the rapidly evolving environment of general practice. Two tools, PC-SafeQuest and MapSaf, were developed with the intention of assessing and improving patient safety culture in this setting. Both have been made widely available through their inclusion in the Royal College of General Practitioners' Patient Safety Toolkit and our work offerss a timely exploration of the tools to inform practice staff as to how each might be usefully applied and in which circumstances. Here we present a comparative analysis of their content, and describe the perspectives of staff on their design, outputs and the feasibility of their sustained use.

METHODS

We have used a content analysis to provide the context for the qualitative study of staff experiences of using the tools at a representative range of practices recruited from across the Midlands (UK). Data was collected through moderated focus groups using an identical topic guide.

RESULTS

A total of nine practices used the PC-SafeQuest tool and four the MapSaf tool. A total of 159 staff completed the PC-SafeQuest tool 52 of whom took part in the subsequent focus group discussions, and 25 staff completed the MapSaf tool all of whom contributed to the focus group discussions. PC-SafeQuest was perceived as quick and easy to use with direct questions pertinent to the work of GP practices providing useful quantitative insight into important areas of safety culture. Though MaPSaF was more logistically challenging, it created a forum for synchronous cross- practice discussions raising awareness of perceptions of safety culture across the practice team.

CONCLUSIONS

Both tools were able to promote reflective and reflexive practice either in individual staff members or across the broader practice team and the oversight they granted provided useful direction for senior staff looking to improve patient safety. Because PC SafeQuest can be easily disseminated and independently completed it is logistically suited to larger practice organisations, whereas the MapSaf tool lends itself to smaller practices where assembling staff in a single workshop is more readily achieved.

A systematic review of patient complaints about general practice

BACKGROUND

Health care complaints are an underutilized resource for quality and safety improvement. Most research on health care complaints is focused on secondary care. However, there is also a need to consider patient safety in general practice, and complaints could inform quality and safety improvement.

OBJECTIVE

This review aimed to synthesize the extant research on complaints in general practice.

METHODS

Five electronic databases were searched: Medline, Web of Science, CINAHL, PsycINFO and Academic Search Complete. Peer-reviewed studies describing the content, impact of and motivation for complaints were included and data extracted. Framework synthesis was conducted using the Healthcare Complaints Analysis Tool (HCAT) as an organizing framework. Methodological quality was appraised using the Quality Assessment Tool for Studies with Diverse Designs (QATSDD).

RESULTS

The search identified 2960 records, with 21 studies meeting inclusion criteria. Methodological quality was found to be variable. The contents of complaints were classified using the HCAT, with 126 complaints (54%) classified in the Clinical domain, 55 (23%) classified as Management and 54 (23%) classified as Relationships. Motivations identified for making complaints included quality improvement for other patients and monetary compensation. Complaints had both positive and negative impacts on individuals and systems involved.

CONCLUSION

This review highlighted the high proportion of clinical complaints in general practice compared to secondary care, patients' motivations for making complaints and the positive and negative impacts that complaints can have on health care systems. Future research focused on the reliable coding of complaints and their use to improve quality and safety in general practice is required.

Developing and evaluating a tool to measure general practice productivity: a multimethod study

Background

Systems for measuring the performance of general practices are extremely limited.

Objectives

The aim was to develop, pilot test and evaluate a measure of productivity that can be applied across all typical general practices in England, and that may result in improvements in practice, thereby leading to better patient outcomes.

Methods

Stage 1 – the approach used was based on the Productivity Measurement and Enhancement System (ProMES). Through 16 workshops with 80 general practice staff and 72 patient representatives, the objectives of general practices were identified, as were indicators that could measure these objectives and systems to convert the indicators into an effectiveness score and a productivity index. This was followed by a consensus exercise involving a face-to-face meeting with 16 stakeholders and an online survey with 27 respondents. An online version of the tool [termed the General Practice Effectiveness Tool (GPET)] and detailed guidance were created. Stage 2 – 51 practices were trained to use the GPET for up to 6 months, entering data on each indicator monthly and getting automated feedback on changes in effectiveness over time. The feasibility and acceptability of the GPET were examined via 38 telephone interviews with practice representatives, an online survey of practice managers and two focus groups with patient representatives.

Results

The workshops resulted in 11 objectives across four performance areas: (1) clinical care, (2) practice management, (3) patient focus and (4) external focus. These were measured by 52 indicators, gathered from clinical information systems, practice records, checklists, a short patient questionnaire and a short staff questionnaire. The consensus exercise suggested that this model was appropriate, but that the tool would be of more benefit in tracking productivity within practices than in performance management. Thirty-eight out of 51 practices provided monthly data, but only 28 practices did so for the full period. Limited time and personnel changes made participation difficult for some. Over the pilot period, practice effectiveness increased significantly. Perceptions of the GPET were varied. Usefulness was given an average rating of 4.5 out of 10.0. Ease of use was more positive, scoring 5.6 out of 10.0. Five indicators were highlighted as problematic to gather, and 27% of practices had difficulties entering data. Feedback from interviews suggested difficulties using the online system and finding time to make use of feedback. Most practices could not provide sufficient monthly financial data to calculate a conventional productivity index.

Limitations

It was not possible to create a measure that provides comparability between all practices, and most practices could not provide sufficient financial data to create a productivity index, leaving an effectiveness measure instead. Having a relatively small number of practices, with no control group, limited this study, and there was a limited timescale for the testing and evaluation.

Implications

The GPET has demonstrated some viability as a tool to aid practice improvement. The model devised could serve as a basis for measuring effectiveness in general practice more widely.

Future work

Some additional research is needed to refine the GPET. Enhanced testing with a control sample would evaluate whether or not it is the use of the GPET that leads to improved performance.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

A systematic review of 7 years of research on entrustable professional activities in graduate medical education, 2011-2018

PURPOSE

This review aimed to synthesise some of the extant work on the use of entrustable professional activities (EPAs) for postgraduate physicians, to assess the quality of the work and provide direction for future research and practice.

METHOD

Systematic searches were conducted within five electronic databases (Medline, Scopus, Web of Science, PsycINFO and CINAHL) in September 2018. Reference lists, Google Scholar and Google were also searched. Methodological quality was assessed using the Quality Assessment Tool for Studies with Diverse Designs (QATSDD).

RESULTS

In total, 49 studies were included, classified as Development of EPAs (n = 37; 76% of total included), Implementation and/or assessment of EPAs (n = 10; 20%), or both (n = 2; 4%). EPAs were described for numerous specialties, including internal medicine (n = 14; 36%), paediatrics (n = 8; 21%) and psychiatry (n = 4; 10%). Of the development studies, 92% utilised more than one method to generate EPAs. The two most commonly used methods were developing initial EPAs in a working group, (n = 27; 69%) and revising through deliberation (n = 21; 54%). Development papers were of variable quality (mean QATSDD score = 20, range 6-41). Implementation and assessment studies utilised methods that included observing trainee performance (n = 6; 50%) and enrolling trainees in competency-based curricula, which included EPAs (n = 4; 33%). The methodological quality of these implementation studies varied (mean QATSDD score = 19.5, range = 6-32).

CONCLUSIONS

This review highlighted a need for: (i) consideration of best practice guidelines for EPA development; (ii) focus on the methodological quality of research on EPA development and of EPAs, and (iii) further work investigating the implementation of EPAs in the curriculum.

Safety in primary care (SAP-C): a randomised, controlled feasibility study in two different healthcare systems

BACKGROUND

Patient safety research is conducted predominantly in hospital settings, with a dearth of insight from primary care, despite suggestions that 2.2% of primary care consultations result in a patient safety incident. This study aimed to assess the feasibility of an intervention intended to improve patient safety in general practice.

METHODS

A randomised controlled feasibility study was conducted with general practices in the Republic of Ireland (N = 9) and Northern Ireland (N = 2), randomly assigned to the intervention (N = 5) or control (N = 6) group. The nine-month intervention consisted of: 1) repeated safety climate (SC) measurement (using GP-SafeQuest questionnaire) and feedback (comparative anonymised practice-level SC data), and 2) patient record reviews using a specialised trigger tool to identify instances of undetected patient harm. For control practices, SC was measured at baseline and study end only. The intervention's perceived usefulness and feasibility were explored via an end-of-study questionnaire and semi-structured interviews.

RESULTS

Thirteen practices were invited; 11 participated; 10 completed the study. At baseline, 84.8% of intervention practice staff (39/46) and 77.8% (42/54) of control practice staff completed the SC questionnaire; at the study terminus, 78.3% (36/46) of intervention practice staff and 68.5% (37/54) of control practice staff did so. Changes in SC scores, indicating improvement, were observed among the intervention practices but not in the control group. The trigger tool was applied to 188 patient records; patient safety incidents of varying severity were detected in 19.1% (36/188). Overall, 59% of intervention practice team members completed the end-of-study questionnaire, with the majority in both healthcare systems responding positively about the intervention. Interviews (N = 9) identified the intervention's usefulness in informing practice management and patient safety issues, time as a barrier to its use, and the value of group discussion of feedback.

CONCLUSION

This feasibility study suggests that a definitive randomised controlled trial of the intervention is warranted. Our findings suggest that the intervention is feasible, useful, and sustainable. Practices were willing to be recruited into the study, response and retention rates were acceptable, and there is possible evidence of a positive effect of the intervention.

TRIAL REGISTRATION

The trial registration number is: ISRCTN11426121 (retrospectively registered 12th June 2018).

Potential value of patient record review to assess and improve patient safety in general practice: A systematic review

Background: There is limited research, and guidance, on how to address safety in general practice proactively.

Objectives: This review aimed to synthesize the literature describing the use of patient record review (PRR) to measure and improve patient safety in primary care. The PRR methodologies utilized and the resulting outcomes were examined.

Methods: Searches were conducted using Medline, Embase, CINAHL and PsycINFO in February 2017. Reference lists of included studies and existing review papers were also screened. English language, peer-reviewed studies that utilized PRR to identify patient safety incidents (PSIs) occurring in general practice were included. Two researchers independently extracted data from articles and applied the Quality Assessment Tool for Studies with Diverse Designs.

Results: A total of 3265 studies were screened, with 15 included. Trigger tools were the most frequent method used for the PRRs (n = 6). The mean number of safety incidents per 100 records was 12.6. Within studies, a mean of 30.6% of incidents were associated with severe harm (range 8.6–50%), and a mean of 55.6% of incidents was considered preventable (range 32.7–93.5%). The most commonly identified types of PSIs related to medication and prescribing, diagnosis, communication and treatment. Three studies reported on improvement actions taken after the PRRs.

Conclusion: This review suggests that PRR may be a promising means of proactively identifying patient safety incidents and informing improvements.