Less is more: Virginia's Performance Outcomes Measurement System

Patient-reported outcome measurements (PROMs) and provider assessment in mental health: a systematic review of the context of implementation

Purpose

To review and integrate the vast amount of literature yielded by recent growing interest in patient-reported outcome measurement and routine outcome measures (PROMs/ROMs), in order to suggest options and improvements for implementation. PROMs are the systematic assessment of service users’ health using standardized self-report measures. Specifically, for ROMs, it includes routine provider or caretaker assessment measures. Both are administered to ascertain routinely, the impact of treatment in mental health settings and to improve care. A review is needed because of the large differences in setting, conceptualization, practice and implementation. Here, we examine the different major projects worldwide.

Data sources

Psych INFO and PubMed including Medline, Biomed Central, EMBASE Psychiatry and Elsevier Science’s Direct.

Study selection

We conducted a systematized review of the literature published from 2000 to 2018 on the implementation and sustainability of PROMs and ROMs in mental health services for adults.

Data extraction, synthesis and Results

We described and characterized the programs in different countries worldwide. We identified 103 articles that met the inclusion criteria, representing over 80 PROMs/ROMs initiatives in 15 countries. National policy and structure of mental health services were found to be major factors in implementation. We discuss the great variability in PROMs/ROMs models in different countries, making suggestions for their streamlining and improvement.

Conclusion

We extracted valuable information on the different characteristics of the numerous PROMs/ROMs initiatives worldwide. However, in the absence of a strong nationwide policy effort and support, implementation seems scattered and irregular. Thus, development of the implementation of PROMs/ROMs is left to groups of enthusiastic clinicians and researchers, making sustainability problematic.

Performance indicators for public mental healthcare: a systematic international inventory

Background

The development and use of performance indicators (PI) in the field of public mental health care (PMHC) has increased rapidly in the last decade. To gain insight in the current state of PI for PMHC in nations and regions around the world, we conducted a structured review of publications in scientific peer-reviewed journals supplemented by a systematic inventory of PI published in policy documents by (non-) governmental organizations.

Methods

Publications on PI for PMHC were identified through database- and internet searches. Final selection was based on review of the full content of the publications. Publications were ordered by nation or region and chronologically. Individual PI were classified by development method, assessment level, care domain, performance dimension, diagnostic focus, and data source. Finally, the evidence on feasibility, data reliability, and content-, criterion-, and construct validity of the PI was evaluated.

Results

A total of 106 publications were included in the sample. The majority of the publications (n = 65) were peer-reviewed journal articles and 66 publications specifically dealt with performance of PMHC in the United States. The objectives of performance measurement vary widely from internal quality improvement to increasing transparency and accountability. The characteristics of 1480 unique PI were assessed. The majority of PI is based on stakeholder opinion, assesses care processes, is not specific to any diagnostic group, and utilizes administrative data sources. The targeted quality dimensions varied widely across and within nations depending on local professional or political definitions and interests. For all PI some evidence for the content validity and feasibility has been established. Data reliability, criterion- and construct validity have rarely been assessed. Only 18 publications on criterion validity were included. These show significant associations in the expected direction on the majority of PI, but mixed results on a noteworthy number of others.

Conclusions

PI have been developed for a broad range of care levels, domains, and quality dimensions of PMHC. To ensure their usefulness for the measurement of PMHC performance and advancement of transparency, accountability and quality improvement in PMHC, future research should focus on assessment of the psychometric properties of PI.

Performance measurement in mental health care: present situation and future possibilities

SUMMARY

This paper describes performance measurement and its indicators for mental health care services. Performance measurement can serve several goals such as accountability, quality improvement and performance management. For all three purposes structure, process and outcome indicators should be measured. Literature was retrieved from Medline and PsychInfo in order to see which performance indicators were used for the three purposes of performance measurement in mental health care. The indicators were classified in structure, process and outcome indicators. The results show no big differences in the indicators used among studies. Performance management is the performance measurement purpose most referred to, followed by accountability, and quality improvement. Outcome and process indicators are used most, structure indicators are in the minority. Several levels of measurement, that is national or service level, came forward in the literature review. To overcome misinterpretation of data and to be able to improve quality and manage performances, performance indicator sets should refer to structure, process and outcome. Indicators should be chosen carefully with the aim of the measurement taken into mind. Based on this review, a conceptual framework is presented to support managers in their decisions about which indictors can best be used for performance measurement. Additionally, a model that provides an understanding of the use of information gained by performance measurement is given.

Factors influencing consumer Outcome Monitoring in implementation of evidence-based practices: results from the National EBP Implementation Project

OBJECTIVE

This study reports participating agencies use of a single fidelity dimension, Outcome Monitoring, during the 2-year National EBP Implementation Project.

METHOD

The study involved an emerging theme qualitative approach for analyzing the implementation of fidelity achieved for five EBPs in 49 sites across eight states.

RESULTS

Twenty-seven percent of the sites reached a high level of implementation fidelity.

CONCLUSION

Results suggest that four interrelated factors contributed to whether agencies successfully implemented Outcome Monitoring-agency data collection methods, agency culture, practitioner skill, and how well "consumer outcomes" are defined.

Strategies for Implementing Performance Measurement in Behavioural Health Care Organisations

Performance measurement provides a basis for demonstrating accountability and a foundation for performance improvement. Nonetheless, practitioners tend to be cautious of attempts to assess the efficiency and effectiveness of behavioural services, particularly given the qualitative nature of some treatment interventions (such as process-oriented therapies) and the limita-tions associated with many mental health instruments. This article examines the rationale for resisting performance measurement and then describes actions that behavioural health care organisations can take to create a systematic approach to collecting and using these measures to improve organisational effectiveness and client services. Examples from the author's work setting, a publicly funded psychiatric hospital for children and adolescents, are provided to demonstrate how these actions have been operationalised.

Linking data to decision-making: applying qualitative data analysis methods and software to identify mechanisms for using outcomes data

A multiple case study was conducted to examine how staff in child out-of-home care programs used data from an Outcomes Management System (OMS) and other sources to inform decision-making. Data collection consisted of thirty-seven semi-structured interviews with clinicians, managers, and directors from two treatment foster care programs and two residential treatment centers, and individuals involved with developing the OMS; and observations of clinical and quality management meetings. Case study and grounded theory methodology guided analyses. The application of qualitative data analysis software is described. Results show that although staff rarely used data from the OMS, they did rely on other sources of systematically collected information to inform clinical, quality management, and program decisions. Analyses of how staff used these data suggest that improving the utility of OMS will involve encouraging staff to participate in data-based decision-making, and designing and implementing OMS in a manner that reflects how decision-making processes operate.