Evaluating instruments for regulation of health care in the Netherlands

[Reliability of peer review-like dialogue in the German statutory quality assurance program]

BACKGROUND

Quality measurement in the German statutory program for quality in health care follows a two-step process. For selected areas of health care, quality is measured via performance indicators (first step). Providers failing to achieve benchmarks in these indicators subsequently enter into a peer review process (second step) and are asked by the respective regional authority to provide a written statement regarding their indicator results. The statements are then evaluated by peers, with the goal to assess the provider's quality of care. In the past, similar peer review-based approaches to the measurement of health care quality in other countries have shown a tendency to lack reliability. So far, the reliability of this component of the German statutory program for quality in health care has not been investigated.

METHOD

Using logistic regression models, the influence of the respective regional authority on the peer review component of health care quality measurement in Germany was investigated using three exemplary indicators and data from 2016.

RESULTS

Both the probability that providers are asked to provide a statement as well as the results produced by the peer review process significantly depend on the regional authority in charge. This dependence cannot be fully explained by differences in the indicator results or by differences in case volume.

CONCLUSIONS

The present results are in accordance with earlier findings, which show low reliability for peer review-based approaches to quality measurement. Thus, different results produced by the peer review component of the quality measurement process may in part be due to differences in the way the review process is conducted. This heterogeneity among the regional authorities limits the reliability of this process. In order to increase reliability, the peer review process should be standardized to a higher degree, with clear review criteria, and the peers should undergo comprehensive training for the review process. Alternatively, the future peer review component could be adapted to focus rather on identification of improvement strategies than on reliable provider comparisons.

Regulation of long-term care for older persons: a scoping review of empirical research

OBJECTIVE

Not only care professionals are responsible for the quality of care but other stakeholders including regulators also play a role. Over the last decades, countries have increasingly invested in regulation of Long-Term Care (LTC) for older persons, raising the question of how regulation should be put into practice to guarantee or improve the quality of care. This scoping review aims to summarize the evidence on regulatory practices in LTC for older persons. It identifies empirical studies, documents the aims and findings, and describes research gaps to foster this field.

DESIGN

A literature search (in PubMed, Embase, Cinahl, APA PsycInfo and Scopus) was performed from inception up to December 12th, 2022. Thirty-one studies were included.

RESULTS

All included studies were from high-income countries, in particular Australia, the US and Northwestern Europe, and almost all focused on care provided in LTC facilities. The studies focused on different aspects of regulatory practice, including care users' experiences in collecting intelligence, impact of standards, regulatory systems and strategies, inspection activities and policies, perception and style of inspectors, perception and attitudes of inspectees and validity and reliability of inspection outcomes.

CONCLUSION

With increasingly fragmented and networked care providers, and an increasing call for person-centred care, more flexible forms of regulatory practice in LTC are needed, organized closer to daily practice, bottom-up. We hope that this scoping review will raise awareness of the importance of regulatory practice and foster research in this field, to improve the quality of LTC for older persons, and optimize their functional ability and well-being.

Determinants of regulatory compliance in health and social care services: A systematic review using the Consolidated Framework for Implementation Research

BACKGROUND

The delivery of high quality care is a fundamental goal for health systems worldwide. One policy tool to ensure quality is the regulation of services by an independent public authority. This systematic review seeks to identify determinants of compliance with such regulation in health and social care services.

METHODS

Searches were carried out on five electronic databases and grey literature sources. Quantitative, qualitative and mixed methods studies were eligible for inclusion. Titles and abstracts were screened by two reviewers independently. Determinants were identified from the included studies, extracted and allocated to constructs in the Consolidated Framework for Implementation Research (CFIR). The quality of included studies was appraised by two reviewers independently. The results were synthesised in a narrative review using the constructs of the CFIR as grouping themes.

RESULTS

The search yielded 7,500 articles for screening, of which 157 were included. Most studies were quantitative designs in nursing home settings and were conducted in the United States. Determinants were largely structural in nature and allocated most frequently to the inner and outer setting domains of the CFIR. The following structural characteristics and compliance were found to be positively associated: smaller facilities (measured by bed capacity); higher nurse-staffing levels; and lower staff turnover. A facility's geographic location and compliance was also associated. It was difficult to make findings in respect of process determinants as qualitative studies were sparse, limiting investigation of the processes underlying regulatory compliance.

CONCLUSION

The literature in this field has focused to date on structural attributes of compliant providers, perhaps because these are easier to measure, and has neglected more complex processes around the implementation of regulatory standards. A number of gaps, particularly in terms of qualitative work, are evident in the literature and further research in this area is needed to provide a clearer picture.

Assessing improvement capability in healthcare organisations: a qualitative study of healthcare regulatory agencies in the UK

Objectives

Healthcare regulatory agencies are increasingly concerned not just with assessing the current performance of the organisations they regulate, but with assessing their improvement capability to predict their future performance trajectory. This study examines how improvement capability is conceptualised and assessed by healthcare UK regulatory agencies.

Design

Qualitative analysis of data from six UK healthcare regulatory agencies was conducted. Three data sources were analysed using an a priori framework of eight dimensions of improvement capability identified from an extensive literature review.

Setting

The focus of the research study was the regulation of hospital-based care, which accounts for the majority of UK healthcare expenditure. Six UK regulatory agencies that review hospital care participated.

Participants

Data sources included interviews with regulatory staff (n = 48), policy documents (n = 90) and assessment reports (n = 30).

Intervention

None-this was a qualitative, observational study.

Results

This research study finds that of eight dimensions of improvement capability, process improvement and learning, and strategy and governance, dominate regulatory assessment practices. The dimension of service-user focus receives the least frequency of use. It may be that dimensions which are relatively easy to 'measure', such as documents for strategy and governance, dominate assessment processes, or there may be gaps in regulatory agencies' assessment instruments, deficits of expertise in improvement capability, or practical difficulties in operationalising regulatory agency intentions to reliably assess improvement capability.

Conclusions

The UK regulatory agencies seek to assess improvement capability to predict performance trajectories, but out of eight dimensions of improvement capability, two dominate assessment. Furthermore, the definition and meaning of assessment instruments requires development. This would strengthen the validity and reliability of agencies' assessment, diagnosis and prediction of performance trajectories, and support development of more appropriate regulatory performance interventions.

External inspection of compliance with standards for improved healthcare outcomes

BACKGROUND

Inspection systems are used in healthcare to promote quality improvements (i.e. to achieve changes in organisational structures or processes, healthcare provider behaviour and patient outcomes). These systems are based on the assumption that externally promoted adherence to evidence-based standards (through inspection/assessment) will result in higher quality of healthcare. However, the benefits of external inspection in terms of organisational-, provider- and patient-level outcomes are not clear. This is the first update of the original Cochrane review, published in 2011.

OBJECTIVES

To evaluate the effectiveness of external inspection of compliance with standards in improving healthcare organisation behaviour, healthcare professional behaviour and patient outcomes.

SEARCH METHODS

We searched the following electronic databases for studies up to 1 June 2015: the Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE, Embase, Database of Abstracts of Reviews of Effectiveness, HMIC, ClinicalTrials.gov and the World Health Organization International Clinical Trials Registry Platform. There was no language restriction and we included studies regardless of publication status. We also searched the reference lists of included studies and contacted authors of relevant papers, accreditation bodies and the International Organization for Standardization (ISO), regarding any further published or unpublished work. We also searched an online database of systematic reviews (PDQ-evidence.org).

SELECTION CRITERIA

We included randomised controlled trials (RCTs), non-randomised trials (NRCTs), interrupted time series (ITSs) and controlled before-after studies (CBAs) evaluating the effect of external inspection against external standards on healthcare organisation change, healthcare professional behaviour or patient outcomes in hospitals, primary healthcare organisations and other community-based healthcare organisations.

DATA COLLECTION AND ANALYSIS

Two review authors independently applied eligibility criteria, extracted data and assessed the risk of bias of each included study. Since meta-analysis was not possible, we produced a narrative results summary. We used the GRADE tool to assess the certainty of the evidence.

MAIN RESULTS

We did not identify any new eligible studies in this update. One cluster RCT involving 20 South African public hospitals and one ITS involving all acute hospital trusts in England, met the inclusion criteria. A trust is a National Health Service hospital which has opted to withdraw from local authority control and be managed by a trust instead.The cluster RCT reported mixed effects of external inspection on compliance with COHSASA (Council for Health Services Accreditation for South Africa) accreditation standards and eight indicators of hospital quality. Improved total compliance score with COHSASA accreditation standards was reported for 21/28 service elements: mean intervention effect was 30% (95% confidence interval (CI) 23% to 37%) (P < 0.001). The score increased from 48% to 78% in intervention hospitals, while remaining the same in control hospitals (43%). The median intervention effect for the indicators of hospital quality of care was 2.4% (range -1.9% to +11.8%).The ITS study evaluated compliance with policies to address healthcare-acquired infections and reported a mean reduction in MRSA (methicillin-resistant Staphylococcus aureus) infection rates of 100 cases per quarter (95% CI -221.0 to 21.5, P = 0.096) at three months' follow-up and an increase of 70 cases per quarter (95% CI -250.5 to 391.0; P = 0.632) at 24 months' follow-up. Regression analysis showed similar MRSA rates before and after the external inspection (difference in slope 24.27, 95% CI -10.4 to 58.9; P = 0.147).Neither included study reported data on unanticipated/adverse consequences or economic outcomes. The cluster RCT reported mainly outcomes related to healthcare organisation change, and no patient reported outcomes other than patient satisfaction.The certainty of the included evidence from both studies was very low. It is uncertain whether external inspection accreditation programmes lead to improved compliance with accreditation standards. It is also uncertain if external inspection infection programmes lead to improved compliance with standards, and if this in turn influences healthcare-acquired MRSA infection rates.

AUTHORS' CONCLUSIONS

The review highlights the paucity of high-quality controlled evaluations of the effectiveness and the cost-effectiveness of external inspection systems. If policy makers wish to understand the effectiveness of this type of intervention better, there needs to be further studies across a range of settings and contexts and studies reporting outcomes important to patients.

Are inspectors' assessments reliable? Ratings of NHS acute hospital trust services in England

The credibility of a regulator could be threatened if stakeholders perceive that assessments of performance made by its inspectors are unreliable. Yet there is little published research on the reliability of inspectors’ assessments of health care organizations’ services.

A retrospective review of how nonconformities are expressed and finalized in external inspections of health-care facilities

BACKGROUND

External inspections are widely used in health care as a means of improving the quality of care. However, the way external inspections affect the involved organization is poorly understood. A better understanding of these processes is important to improve our understanding of the varying effects of external inspections in different organizations. In turn, this can contribute to the development of more effective ways of conducting inspections. The way the inspecting organization states their grounds for noncompliant behavior and subsequently follows up to enforce the necessary changes can have implications for the inspected organization's change process. We explore how inspecting organizations express and state their grounds for noncompliant behavior and how they follow up to enforce improvements.

METHODS

We conducted a retrospective review, in which we performed a content analysis of the documents from 36 external inspections in Norway. Our analysis was guided by Donabedian's structure, process, and outcome model.

RESULTS

Deficiencies in the management system in combination with clinical work processes was considered as nonconformity by the inspecting organizations. Two characteristic patterns were identified in the way observations led to a statement of nonconformity: one in which it was clearly demonstrated how deficiencies in the management system could affect clinical processes, and one in which this connection was not demonstrated. Two characteristic patterns were also identified in the way the inspecting organization followed up and finalized their inspection: one in which the inspection was finalized solely based on the documented changes in structural deficiencies addressed in the nonconformity statement, and one based on the documented changes in structural and process deficiencies addressed in the nonconformity statement.

CONCLUSION

External inspections are performed to improve the quality of care. To accomplish this aim, we suggest that nonconformities should be grounded by observations that clearly demonstrate how deficiencies in the management system might affect the clinical processes, and that the inspection should be finalized based on documented changes in both structural and process deficiencies addressed in the nonconformity statement.

Experimental studies to improve the reliability and validity of regulatory judgments on health care in the Netherlands: a randomized controlled trial and before and after case study

RATIONALE, AIMS AND OBJECTIVES

We examined the effect of two interventions on both the reliability and validity of regulatory judgments: adjusting the regulatory instrument and attending a consensus meeting.

METHOD

We adjusted the regulatory instrument. With a randomized controlled trial (RCT) we examined the effect of the adjustments we made to the instrument. In the consensus meeting inspectors discussed cases and had to reach consensus about the order of the cases. We used a before and after case study to assess the effect of the consensus meeting. We compared the judgments assigned in the RCT with the unadjusted instrument with the judgments assigned with the unadjusted instrument after the consensus meeting. Moreover we explored the effect of increasing the number of inspectors per regulatory visit based on the estimates of the two interventions.

RESULTS

The consensus meeting improved the agreement between inspectors; the variance between inspectors was smallest (0.03) and the reliability coefficient was highest (0.59). Validity is assessed by examining the relation between the assigned judgments and the corporate standard and expressed by a correlation coefficient. This coefficient was highest after the consensus meeting (0.48). Adjustment of the instrument did not increase reliability and validity coefficients.

CONCLUSIONS

Participating in a consensus meeting improved reliability and validity. Increasing the number of inspectors resulted in both higher reliability and validity values. Organizing consensus meetings and increasing the number of inspectors per regulatory visit seem to be valuable interventions for improving regulatory judgments.

Including citizens in institutional reviews: expectations and experiences from the Dutch Healthcare Inspectorate

BACKGROUND

Recent changes in the structure and policy context of Dutch health care have placed the issue of citizen participation high on the agenda of the Dutch Healthcare Inspectorate (IGZ), which conducts quality and safety reviews in medical practices and health-care institutions. With a few exceptions, the potential role that citizens can play in the regulation of health-care institutions is overlooked in research on patient/citizen participation in health care.

OBJECTIVE

This research addressed the following question: What are the (political) expectations for increasing citizen participation in health-care regulation and how do these compare to regulators' expectations and experiences in practice?

DESIGN

Because of the largely explorative nature of this study, we used qualitative methods (document and web analysis, focus groups and interviews) to answer this question.

RESULTS

Our study shows that inspectors already have experience with participatory formats that lead to important information. There are three areas where the IGZ is currently increasing citizen participation: (i) providing individuals with information about inspectorate processes and activities, (ii) including patients as sources of information, and (iii) formally reviewing how citizen participation is ensured by health-care institutions. In situations where the patient has the clearest overview of the whole care trajectory, intensive methods of participation deliver valuable information.

CONCLUSIONS

It is important to target participation activities and to capitalize on existing opportunities and activities, rather than creating participation activities for the sake of participation. In this regard, further research on the effectiveness and efficacy of different participatory strategies is necessary.

Reducing interrater variability and improving health care: a meta-analytical review

OBJECTIVES

In the scientific literature about reliability, the main approach to increasing reliability seems to involve increasing the number of observers and improving the instrument used. Other aspects for improving reliability - like the training of raters - seem to receive less notice. It is worth asking whether this technical approach could be complemented by training the user of the instrument. A systematic meta-analytical review of the research literature was performed to answer this question and examine the effectiveness of planned interventions for improving interrater reliability of health care professionals.

METHOD

The databases of PubMed (MEDLINE), Embase, Omega and PsycINFO were searched. The inclusion criteria were met by 57 studies. Details extracted from the studies included the study design, the number of observers and the number of observed cases, the intervention, the type of instrument (whether or not it was highly technical), and statistical information about the agreement before and after the intervention. Interventions were categorized into three groups: training of professionals, improving the diagnostic instrument and a combination of training and improving the instrument. A meta-analysis was performed by means of linear regression.

RESULTS

The interventions were arranged according to their effectiveness in improving the diagnostic instrument (mean change: β = 0.13), training combined with improving the instrument (mean change: β = 0.10) and training (mean change: β = 0.09).

CONCLUSION

On average, although all types of interventions are effective, improving the diagnostic instrument seems to be the most effective. Especially when highly technical instruments were concerned, improvement proved to be very effective (β = 0.52). Because instrumental variables constitute a major source of error, improving the instrument is an important approach. However, this review offers solid arguments that can complement the literature and practice, with a focus on training the user of the instrument.