Addressing Missing Data in Patient-Reported Outcome Measures (PROMS): Implications for the Use of PROMS for Comparing Provider Performance

Can prediction models for hospital readmission be improved by incorporating patient-reported outcome measures? A systematic review and narrative synthesis

PURPOSE

To investigate the roles, challenges, and implications of using patient-reported outcome measures (PROMs) in predicting the risk of hospital readmissions.

METHODS

We systematically searched four bibliometric databases for peer-reviewed studies published in English between 1 January 2000 and 15 June 2023 and used validated PROMs to predict readmission risks for adult populations. Reported studies were analysed and narratively synthesised in accordance with the CHARMS and PRISMA guidelines.

RESULTS

Of the 2858 abstracts reviewed, 23 studies met predefined eligibility criteria, representing diverse geographic regions and medical specialties. Among those, 19 identified the positive contributions of PROMs in predicting readmission risks. Seven studies utilised generic PROMs exclusively, eleven used generic and condition-specific PROMs, while 5 focussed solely on condition-specific PROMs. Logistic regression was the most used modelling approach, with 13 studies aiming at predicting 30-day all-cause readmission risks. The c-statistic, ranging from 0.54 to 0.84, was reported in 22/23 studies as a measure of model discrimination. Nine studies reported model calibration in addition to c-statistic. Thirteen studies detailed their approaches to dealing with missing data.

CONCLUSION

Our study highlights the potential of PROMs to enhance predictive accuracy in readmission models, while acknowledging the diversity in data collection methods, readmission definitions, and model evaluation approaches. Recognizing that PROMs serve various purposes beyond readmission reduction, our study supports routine data collection and strategic integration of PROMs in healthcare practices to improve patient outcomes. To facilitate comparative analysis and broaden the use of PROMs in the prediction framework, it is imperative to consider the methodological aspects involved.

The Joint Awareness Score: A Shortened, Simplified, Improved Alternative to the Forgotten Joint Score

Background

The Forgotten Joint Score (FJS) is a 12-question patient-reported outcomes measure created to measure a patient's awareness of their artificial joint. The FJS has attained wide usage, though it is not without weaknesses. Our patients report that the semantics of the English translation are flawed and that the 5 answer options for each question are poorly differentiated. Additionally, the FJS will result in no score if 3 or more questions are unanswered. This prompted the development of an alternative patient-reported outcomes measure, the Joint Awareness Score (JAS), that builds upon the core concept of joint awareness underlying the FJS, but that is easier to understand and shorter to complete. We completed an exploratory, pilot study to evaluate this outcomes instrument. Our hypothesis is that the JAS will correlate strongly with the FJS and could be used as a substitute.

Methods

Knee arthroplasty patients in a prospective registry were administered the FJS and the JAS. Internal consistency and correlation were calculated with Cronbach's alpha and Pearson's correlation coefficient, respectively.

Results

This study included 174 patients. Cronbach's alpha for FJS was 0.97 for 6 months and 0.97 for 12 months, whereas JAS was 0.89 at 6 months and 0.85 at 12 months. Pearson correlation comparing FJS and JAS at 6 months was 0.88 (95% confidence interval: 0.83, 0.92) and 0.86 (95% confidence interval: 0.78, 0.92) at 12 months.

Conclusions

The Joint Awareness Score is a new patient-reported outcomes measure that is a substitute for the FJS, with half the number of questions, improved semantics, and simplified answers.

Handling Missing Data in Health Economics and Outcomes Research (HEOR): A Systematic Review and Practical Recommendations

BACKGROUND

Missing data in costs and/or health outcomes and in confounding variables can create bias in the inference of health economics and outcomes research studies, which in turn can lead to inappropriate policies. Most of the literature focuses on handling missing data in randomized controlled trials, which are not necessarily always the data used in health economics and outcomes research.

OBJECTIVES

We aimed to provide an overview on missing data issues and how to address incomplete data and report the findings of a systematic literature review of methods used to deal with missing data in health economics and outcomes research studies that focused on cost, utility, and patient-reported outcomes.

METHODS

A systematic search of papers published in English language until the end of the year 2020 was carried out in PubMed. Studies using statistical methods to handle missing data for analyses of cost, utility, or patient-reported outcome data were included, as were reviews and guidance papers on handling missing data for those outcomes. The data extraction was conducted with a focus on the context of the study, the type of missing data, and the methods used to tackle missing data.

RESULTS

From 1433 identified records, 40 papers were included. Thirteen studies were economic evaluations. Thirty studies used multiple imputation with 17 studies using multiple imputation by chained equation, while 15 studies used a complete-case analysis. Seventeen studies addressed missing cost data and 23 studies dealt with missing outcome data. Eleven studies reported a single method while 20 studies used multiple methods to address missing data.

CONCLUSIONS

Several health economics and outcomes research studies did not offer a justification of their approach of handling missing data and some used only a single method without a sensitivity analysis. This systematic literature review highlights the importance of considering the missingness mechanism and including sensitivity analyses when planning, analyzing, and reporting health economics and outcomes research studies.

Fixing the Leaky Pipe: How to Improve the Uptake of Patient-Reported Outcomes-Based Prognostic and Predictive Models in Cancer Clinical Practice

PURPOSE

This discussion paper outlines challenges and proposes solutions for successfully implementing prediction models that incorporate patient-reported outcomes (PROs) in cancer practice.

METHODS

We organized a full-day multidisciplinary meeting of people with expertise in cancer care delivery, PRO collection, PRO use in prediction modeling, computing, implementation, and decision science. The discussions presented here focused on identifying challenges to the development, implementation and use of prediction models incorporating PROs, and suggesting possible solutions.

RESULTS

Specific challenges and solutions were identified across three broad areas. (1) Understanding decision making and implementation: necessitating multidisciplinary collaboration in the early stages and throughout; early stakeholder engagement to define the decision problem and ensure acceptability of PROs in prediction; understanding patient/clinician interpretation of PRO predictions and uncertainty to optimize prediction impact; striving for model integration into existing electronic health records; and early regulatory alignment. (2) Recognizing the limitations to PRO collection and their impact on prediction: incorporating validated, clinically important PROs to maximize model generalizability and clinical engagement; and minimizing missing PRO data (resulting from both structural digital exclusion and time-varying factors) to avoid exacerbating existing inequalities. (3) Statistical and modeling challenges: incorporating statistical methods to address missing data; ensuring predictive modeling recognizes complex causal relationships; and considering temporal and geographic recalibration so that model predictions reflect the relevant population.

CONCLUSION

Developing and implementing PRO-based prediction models in cancer care requires extensive multidisciplinary working from the earliest stages, recognition of implementation challenges because of PRO collection and model presentation, and robust statistical methods to manage missing data, causality, and calibration. Prediction models incorporating PROs should be viewed as complex interventions, with their development and impact assessment carried out to reflect this.

Toward developing care outcome quality indicators for home care for older people: A prospective cohort study in Japan

INTRODUCTION

Care quality in Japan's long-term care (LTC) agencies, including home care, is the responsibility primarily of individual agencies, and the evaluation of service processes and outcomes is minimal.

OBJECTIVES

To describe the development of quality indicators for LTC (QIs-LTC) in Japan.

METHODS

QIs-LTC were developed through literature review and expert panel discussions and then were piloted and used in a 2-year longitudinal survey. The survey (launched in September 2019) targeted older people receiving home care (n = 1450), their family members (n = 880), their professional home care providers (n = 577), and managers of home care agencies (n = 122).

RESULTS

Across eight domains (maintaining dignity, minimizing symptoms and disease deterioration, maintaining nutritional status, maintaining bladder/bowel control, encouraging physical activities, experiencing sound sleep, maintaining serenity and contentedness, and maintaining family's well-being), 24 care quality objectives were set with 24 outcome QIs-LTC and 144 process QIs-LTC. In the survey, 84.8% of clients were using home care nursing, 26.3% were living alone, and 39.5% had dementia. In the month preceding the data collection, 13.9% of clients had a new disease or worsening of an existing disease, 8.8% were hospitalized at least once, and 47.9% did not participate in activities of interest. About 20% of clients' families were unable to spend time peacefully, and 52.8% were exhausted from the client's care.

CONCLUSIONS

The QIs-LTC developed in the current study are generic and client- and family-centered. They encompass objective and subjective information and would facilitate standardized monitoring if adopted and comparison between LTC settings, including home care. In addition, future research directives are outlined. Geriatr Gerontol Int 2023; 23: 383-394.

Use of patient-reported outcomes in heart failure: from clinical trials to routine practice

Heart failure (HF) is a complex syndrome that affects mortality/morbidity and acts at different levels in the patient's life, resulting in a drastic impairment in multiple aspects of daily activities (e.g. physical, mental/emotional, and social) and leading to a reduction in quality of life. The definition of disease status and symptom severity has been traditionally based on the physician assessment, while the patient's experience of disease has been long overlooked. The active participation of patients in their own care is necessary to better understand the perception of disease and the multiple aspects of life affected, and to improve adherence to treatments. Patient-reported outcomes (PROs) aim to switch traditional care to a more patient-centred approach. Although PROs demonstrated precision in the evaluation of disease status and have a good association with prognosis in several randomized controlled trials, their implementation into clinical practice is limited. This review discusses the modalities of use of PROs in HF, summarizes the most largely adopted PROs in HF care, and provides an overview on the application of PROs in trials and the potential for their transition to clinical practice. By discussing the advantages and the disadvantages of their use, the reasons limiting their application in daily clinical routine, and the strategies that may promote their implementation, this review aims to foster the systematic integration of the patient's standpoint in HF care.

Moving from development to implementation of digital innovations within the NHS: myHealthE, a remote monitoring system for tracking patient outcomes in child and adolescent mental health services

Objective

This paper aims to report our experience of developing, implementing, and evaluating myHealthE (MHE), a digital innovation for Child and Adolescents Mental Health Services (CAMHS), which automates the remote collection and reporting of Patient-Reported Outcome Measures (PROMs) into National Health Services (NHS) electronic healthcare records.

Methods

We describe the logistical and governance issues encountered in developing the MHE interface with patient-identifiable information, and the steps taken to overcome these development barriers. We describe the application's architecture and hosting environment to enable its operability within the NHS, as well as the capabilities needed within the technical team to bridge the gap between academic development and NHS operational teams.

Results

We present evidence on the feasibility and acceptability of this system within clinical services and the process of iterative development, highlighting additional functions that were incorporated to increase system utility.

Conclusion

This article provides a framework with which to plan, develop, and implement automated PROM collection from remote devices back to NHS infrastructure. The challenges and solutions described in this paper will be pertinent to other digital health innovation researchers aspiring to deploy interoperable systems within NHS clinical systems.

Predicting non-response in patient-reported outcome measures: results from the Swiss quality assurance programme in cardiac inpatient rehabilitation

BACKGROUND

Quality assurance programmes measure and compare certain health outcomes to ensure high-quality care in the health-care sector. The outcome of health-related quality of life is typically measured by patient-reported outcome measures (PROMs). However, certain patient groups are less likely to respond to PROMs than others. This non-response bias can potentially distort results in quality assurance programmes.

OBJECTIVE

Our study aims to identify relevant predictors of non-response during assessment using the PROM MacNew Heart Disease questionnaire in cardiac rehabilitation.

METHODS

This is a cross-sectional study based on data from the Swiss external quality assurance programme. All patients aged 18 years or older who underwent inpatient cardiac rehabilitation in 16 Swiss rehabilitation clinics between 2016 and 2019 were included. Patients' socio-demographic and basic medical data were analysed descriptively by comparing two groups: non-responders and responders. We used a random intercept logistic regression model to estimate the associations of patient characteristics and clinic differences with non-response.

RESULTS

Of 24 572 patients, there were 33.3% non-responders and 66.7% responders. The mean age was 70 years, and 31.0% were women. The regression model showed that being female was associated with non-response [odds ratio (OR) 1.22; 95% confidence interval (CI) 1.14-1.30], as well as having no supplementary health insurance (OR 1.49; 95% CI 1.39-1.59). Each additional year of age increased the chance of non-response by an OR of 1.02 (95% CI 1.02-1.02). Not being a first language speaker of German, French or Italian increased the chance of non-response by an OR of 6.94 (95% CI 6.03-7.99). Patients admitted directly from acute care had a higher chance of non-response (OR 1.23; 95% CI 1.10-1.38), as well as patients being discharged back into acute care after rehabilitation (OR 3.89; 95% CI 3.00-5.04). Each point on the cumulative illness rating scale total score increased the chance of non-response by an OR of 1.05 (95% CI 1.04-1.05). Certain diagnoses also influenced the chance of non-response. Even after adjustment for known confounders, response rates differed substantially between the 16 clinics.

CONCLUSION

We have found significant non-response bias among certain patient groups, as well as across different treatment facilities. Measures to improve response rates among patients with known barriers to participation, as well as among different treatment facilities, need to be considered, particularly when PROMs are being used for comparison of providers in quality assurance programmes or outcome evaluation.

Multivariable prediction models for long-term outcomes after hip fracture: A protocol for a systematic review

Background: Hip fracture results in high mortality and, for many survivors, long-term functional limitations. Multivariable prediction models for hip fracture outcomes have the potential to aid clinical-decision making as well as risk-adjustment in national audits of care. The aim of this study is to identify, critically appraise and synthesise published multivariable prediction models for long-term outcomes after hip fracture. Protocol: The systematic review will include a literature search of electronic databases (MEDLINE, Embase, Scopus, Web of Science and CINAHL) for journal articles. Search terms related to hip fracture, prognosis and outcomes will be included. Study selection criteria includes studies of people with hip fracture where the study aimed to predict one or more long-term outcomes through derivation or validation of a multivariable prediction model. Studies will be excluded if they focus only on the predictive value of individual factors, or only include patients with periprosthetic fractures, fractures managed non-surgically or younger patients. Covidence software will be used for data management. Two review authors will independently conduct study selection, data extraction and appraisal. Data will be extracted based on the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist. Risk of bias assessment will be conducted using the Prediction model Risk of Bias Assessment Tool (PROBAST). Characteristics and results of all studies will be narratively synthesised and presented in tables. Where the same model has been validated in multiple studies, a meta-analysis of discrimination and calibration will be conducted. Conclusions: This systematic review will aim to identify multivariable models for hip fracture outcome prognosis that have been derived using high quality methods. Results will highlight if current models have the potential for further assessment for use in both clinical decision making and improving methods of national hip fracture audits. PROSPERO registration: CRD42022330019 (25th May 2022).

Quality of patient-reported outcome reporting in trials of diabetes in pregnancy: A systematic review

AIMS

Patient-reported outcomes (PROs) are reports of the patient's health status that come directly from the patient without interpretation by the clinician or anyone else. They are increasingly used in randomised controlled trials (RCTs). In this systematic review we identified RCTs conducted in women with diabetes in pregnancy which included PROs in their primary or secondary outcomes. We then evaluated the quality of PRO reporting against an internationally accepted reporting framework (Consolidated Standards of Reporting Trials (CONSORT-PRO) guidelines).

METHODS

We searched online databases for studies published 2013-2021 using a combination of keywords. Two authors reviewed all abstracts independently. Data on study characteristics and the quality of PRO reporting were extracted from relevant studies. We conducted a multiple regression analysis to identify factors associated with high quality reporting.

RESULTS

We identified 7122 citations. Thirty-five articles were included for review. Only 17% of RCTs included a PRO as a primary or secondary outcome. Out of a maximum score of 100 the median score was 46, indicating sub-optimal reporting. A multiple regression analysis did not reveal any factors associated with high quality reporting.

CONCLUSIONS

Researchers should be mindful of the importance of PRO inclusion and reporting and include reliable PROs in trials.

Continuous care intervention with carbohydrate restriction improves physical function of the knees among patients with type 2 diabetes: a non-randomized study

Background

In a previous study, we assessed a novel, remotely monitored carbohydrate restricted diet regimen including nutritional ketosis in patients with type 2 diabetes and reported significant improvements in weight, glycemic control, abdominal fat and inflammation from baseline to 2 years. Knee outcome measures were collected as a secondary outcome in the trial. This study aims to assess the effect of this intervention on knee functional scores and to identify if changes in weight, central abdominal fat (CAF), glycemic status and high sensitivity C-reactive protein (hsCRP) were associated with its improvement.

Methods

This prospective analysis included continuous care intervention (CCI, n = 173) and usual care (UC, n = 69) trial participants with type 2 diabetes that reported knee pain at baseline. Knee outcome measures included the Knee injury and Osteoarthritis Outcome Score (KOOS) pain, symptoms, activities of daily living (ADL), sports and recreation function, and knee-related quality of life subscales, and total KOOS score were assessed from baseline to 2 years. Missing data at each time point were replaced with multiple imputation under the assumption of missing at random. To assess if the primary analysis of the knee scores changed under plausible missing not at random assumptions, sensitivity analysis was also performed using pattern mixture models. In CCI, we also assessed factors associated with the improvement of knee scores.

Results

In the primary analysis, CCI participants demonstrated a statistically significant improvement in total KOOS and all KOOS individual subscale scores at 1 year and maintained through 2 years as opposed to UC patients who showed no significant changes from baseline to 2 years. The significant improvement in total KOOS and its individual subscale scores from baseline to 2 years remained relatively stable in CCI in the sensitivity analysis under different missing not at random scenarios confirming the robustness of the findings from the primary analysis. Approximately 46% of the CCI participants met the 10 points minimal clinically important change at 2 years. A reduction in CAF was associated with improvement in total KOOS and KOOS ADL, while a decrease in hsCRP was associated with improvement in KOOS symptoms scores.

Conclusion

A very low carbohydrate intervention including nutritional ketosis resulted in significant improvements in knee pain and function among patients with T2D. The improvements in knee function were likely secondary to a reduction in central adiposity and inflammation. Future research on the applicability of this intervention in radiographically confirmed OA patients is important.

Trial registration

Clinical trial registration: NCT02519309 (10/08/2015).

Supplementary Information

The online version contains supplementary material available at 10.1186/s12891-022-05258-0.

Nonresponder bias in hand surgery: analysis of 1945 cases lost to follow-up over a 6-year period

The primary aim of this study was to identify factors associated with nonresponse to routinely collected patient-reported outcome measures (PROMs) after hand surgery. The secondary aim was to investigate the impact of nonresponder bias on postoperative PROMs. We identified 4357 patient episodes for which the patients received pre- and 1-year postoperative questionnaires. The response rate was 55%. Univariate and regression analyses were undertaken to determine factors predicting nonresponse. We developed a predictive model for the postoperative Quick version of the Disabilities of the Arm, Shoulder, and Hand (QuickDASH) scores for nonresponders using imputation. Younger age, increasing deprivation, higher comorbidity, worse preoperative QuickDASH scores and unemployment predicted nonresponse. No significant difference in mean postoperative QuickDASH score was observed between the responders, and the scores for the responders combined with the predicted scores for the nonresponders. Preoperative function was the primary predictor of postoperative outcome. These results challenge the dogma that 'loss to follow-up' automatically invalidates the results of a study.Level of evidence: III.

Impact of COVID‐19 social distancing measures on routine mental health care provision and treatment outcome for common mental disorders in the Netherlands

Objective

The uptake of digital interventions in mental health care (MHC) has been slow, as many therapists and patients believe that in‐person contact is essential for establishing a good working relationship and good outcomes in treatment. The public health policies regarding social distancing during the coronavirus disease‐2019 (COVID‐19) pandemic forced an abrupt transformation of MHC provisions for outpatients: Since mid‐March 2020, nearly all in‐person contact was replaced with videoconferencing. The COVID‐19 crisis offered a unique opportunity to investigate whether MHC with videoconferencing yields inferior results as compared to in‐person interventions.

Method

In a large urban MHC facility in the Netherlands, measurement‐based care is routine practice. Outcome data are regularly collected to support shared decision making and monitor patient progress. For this study, pretest and post‐test data were used to compare outcomes for three cohorts: treatments performed prior to, partially during and entirely during the COVID‐19 lockdown. Outcomes were compared in two large data sets: Basic MHC (N = 1392) and Specialized MHC (N = 1040).

Results

Therapeutic outcomes appeared robust for COVID‐19 conditions across the three cohorts: No differences in outcomes were found between treatments that were conducted during lockdown compared to in‐person treatments prior to COVID‐19, or treatments which started in‐person, but needed to be continued by means of videoconferencing.

Discussion

Videoconferencing care during the COVID‐19 pandemic had similar outcomes compared to traditional in‐person care. These real‐world results corroborate findings of previous randomized controlled studies and meta‐analyses in which videoconferencing and in‐person care has been directly compared in terms of clinical effectiveness.

The subjective knee value is a valid single-item survey to assess knee function in common knee disorders

INTRODUCTION

The patient's perspective plays a key role in judging the effect of knee disorders on physical function. We have introduced the Subjective Knee Value (SKV) to simplify the evaluation of individual's knee function by providing one simple question. The purpose of this prospective study was to validate the SKV with accepted multiple-item knee surveys across patients with orthopaedic knee disorders.

MATERIALS AND METHODS

Between January through March 2020, consecutive patients (n = 160; mean age 51 ± 18 years, range from 18 to 85 years, 54% women) attending the outpatient clinic for knee complaints caused by osteoarthritis (n = 69), meniscal lesion (n = 45), tear of the anterior cruciate ligament (n = 23) and focal chondral defect (n = 23) were invited to complete a knee-specific survey including the SKV along with the Knee Injury Osteoarthritis Outcome Score (KOOS) and the International Knee Documentation Committee subjective knee form (IKDC-S). The Pearson correlation coefficient was used to evaluate external validity between the SKV and each patient-reported outcome measure (PROM) separately. Furthermore, patient's compliance was assessed by comparing responding rates.

RESULTS

Overall, the SKV highly correlated with both the KOOS (R = 0.758, p < 0.05) and the IKDC-S (R = 0.802, p < 0.05). This was also demonstrated across all investigated diagnosis- and demographic-specific (gender, age) subgroups (range 0.509-0.936). No relevant floor/ceiling effects were noticed. The responding rate for the SKV (96%) was significantly higher when compared with those for the KOOS (81%) and the IKDC-S (83%) (p < 0.05).

CONCLUSION

At baseline, the SKV exhibits acceptable validity across all investigated knee-specific PROMs in a broad patient population with a wide array of knee disorders. The simplified survey format without compromising the precision to evaluate individual's knee function justifies implementation in daily clinical practice.

LEVEL OF EVIDENCE

II, cohort study (diagnosis).

Patient-reported outcome measures in ophthalmology: too difficult to read?

Objective

Patient-reported outcome measures (PROMs) are commonly used in clinical trials and research. Yet, in order to be effective, a PROM needs to be understandable to respondents. The aim of this cross-sectional analysis was to assess reading level of PROMs validated for use in common eye conditions.

Methods and analysis

Readability measures determine the level of education a person is expected to have attained to be able to read a passage of text; this was calculated using the Flesch-Kincaid Grade Level, FORCAST and Gunning-Fog tests within readability calculations software package Oleander Readability Studio 2012.1. Forty PROMs, previously validated for use in at least one of age-related macular degeneration, glaucoma and/or diabetic retinopathy, were identified for inclusion via a systematic literature search. The American Medical Association (AMA) and National Institutes of Health (NIH) recommend patient materials should not exceed a sixth-grade reading level. Number of PROMs exceeding this level was calculated.

Results

Median (IQR) readability scores were 7.9 (5.4-10.5), 9.9 (8.9-10.7) and 8.4 (6.9-11.1) for Flesch-Kincaid Grade Level, FORCAST and Gunning-Fog test, respectively. Depending on metric used, this meant 61% (95% CI 45% to 76%), 100% (95% CI 91% to 100%) and 80% (95% CI 65% to 91%) exceeded the recommended threshold.

Conclusion

Most PROMs commonly used in ophthalmology require a higher reading level than that recommended by the AMA and NIH and likely contain questions that are too difficult for many patients to read. Greater care is needed in designing PROMs appropriate for the literacy level of a population.

Handling Missing Data in the Short Form-12 Health Survey (SF-12): Concordance of Real Patient Data and Data Estimated by Missing Data Imputation Procedures

If information on single items in the Short Form–12 health survey (SF-12) is missing, the analysis of only complete cases causes a loss of statistical power and, in case of nonrandom missing data (MD), systematic bias. This study aimed at evaluating the concordance of real patient data and data estimated by different MD imputation procedures in the items of the SF-12 assessment. For this ends, MD were examined in a sample of 1,137 orthopedic patients. Additionally, MD were simulated (a) in the subsample of orthopedic patients exhibiting no MD (n = 810; 71%) as well as (b) in a sample of 6,970 respondents representing the German general population (95.8% participants with complete data) using logistic regression modelling. Simulated MD were replaced by mean values as well as regression-, expectation-maximization- (EM-), and multiple imputation estimates. Higher age and lower education were associated with enhanced probabilities of MD. In terms of accuracy in both data sets, the EM-procedure (ICC_2,1 = .33-.72) outperformed alternative estimation approaches substantially (e.g., regression imputation: ICC_2,1 = .18-.48). The EM-algorithm can be recommended to estimate MD in the items of the SF-12, because it reproduces the actual patient data most accurately.

Patient-Centered Outcomes in Radiation Oncology

Patient-reported outcome and health-related quality of life scales have the potential to engage patients and providers, allowing for better communication and shared decision-making in oncology care. When monitored longitudinally, they facilitate earlier interventions that may help with symptom management and improve traditional outcome metrics, including survival. Their use in clinical trials has allowed for changes in guidelines in the management of various cancers. The voice and experience of the patient, captured by these scales, enable providers to better detail the journey patients can expect to experience during and after treatment.

Estimating treatment effects under untestable assumptions with nonignorable missing data

Nonignorable missing data poses key challenges for estimating treatment effects because the substantive model may not be identifiable without imposing further assumptions. For example, the Heckman selection model has been widely used for handling nonignorable missing data but requires the study to make correct assumptions, both about the joint distribution of the missingness and outcome and that there is a valid exclusion restriction. Recent studies have revisited how alternative selection model approaches, for example estimated by multiple imputation (MI) and maximum likelihood, relate to Heckman-type approaches in addressing the first hurdle. However, the extent to which these different selection models rely on the exclusion restriction assumption with nonignorable missing data is unclear. Motivated by an interventional study (REFLUX) with nonignorable missing outcome data in half of the sample, this article critically examines the role of the exclusion restriction in Heckman, MI, and full-likelihood selection models when addressing nonignorability. We explore the implications of the different methodological choices concerning the exclusion restriction for relative bias and root-mean-squared error in estimating treatment effects. We find that the relative performance of the methods differs in practically important ways according to the relevance and strength of the exclusion restriction. The full-likelihood approach is less sensitive to alternative assumptions about the exclusion restriction than Heckman-type models and appears an appropriate method for handling nonignorable missing data. We illustrate the implications of method choice for inference in the REFLUX study, which evaluates the effect of laparoscopic surgery on long-term quality of life for patients with gastro-oseophageal reflux disease.

Developing a Preliminary Conceptual Framework for Guidelines on Inclusion of Patient Reported-Outcome Measures (PROMs) in Clinical Quality Registries

Purpose

Patient-centred and value-based health-care organisations are increasingly recognising the importance of the patient perspective in the measurement and evaluation of health outcomes. This has been primarily implemented using patient-reported outcome measures (PROMs). Clinical quality registries (CQRs) are specifically designed to improve direct clinical care, benchmark health-care provision and inform health service planning and policy. Despite CQRs having incorporated the patient perspective to support the evaluation of health-care provision, no evidence-based guidelines for inclusion of PROMs in CQRs exist. This has led to substantial heterogeneity in capturing and reporting PROMs within this setting. This publication is the first in a series describing the development of evidence-informed guidelines for PROMs inclusion within CQRs in Australia.

Methods

This study consisted of three components: 1) a literature review of existing evidence of guidelines, enablers, barriers, and lessons learnt of PROMs use within the CQRs setting; 2) a survey of Australian CQRs to determine current practices for PROMs use and reporting; and 3) development of a preliminary conceptual framework for PROMs inclusion in CQRs.

Results

Content analysis of the literature review and survey of 66 Australian registries elicited eight categories for the conceptual framework. The framework covers eight components: rationale, setting, ethics, selection of PROMs, administration, data management, statistical methods, feedback, and reporting.

Conclusion

We developed a preliminary conceptual framework, which classified findings, from both the literature and the survey, into broad categories ranging from initial development to outcome dissemination providing the structure for development of guidelines in the next phase of this project, engaging national and international leaders in health-related quality of life research, clinicians, researchers, patient advocates and consumers.

What Do U.K. Orthopedic Surgery Patients Think About PROMs? Evaluating the Evaluation and Explaining Missing Data

The NHS routinely evaluates the quality of life of patients receiving hip or knee replacement surgery using patient-reported outcome measures (PROMs), but some hospital completion rates are only 30%, restricting data usefulness. Statistics limit insights into how and why data are missing, so qualitative methods were used to explore this issue. Observation periods preceded semistructured interviews with 34 preoperative patients attending an orthopedic outpatient clinic. Interview themes covered: completion time/timing, orientation, setting, measures, and practicalities. Triangulated against observations, pragmatic barriers, and facilitators were considered. Refined themes included completion conditions, patient support, and national delivery. Simple improvements (e.g., quiet zone) could improve completion rates and reducing missing data. Reorganizing preoperative leaflets and their systematic distribution via standardized procedures could reassure patients, enhancing PROMs acceptance, while reducing inquiries and subsequent staff burden. Findings have implications for interpreting national statistics. They indicate that further debate about mandating preoperative PROMs is due.

Patient-Reported Outcomes: Understanding Surgical Efficacy and Quality from the Patient's Perspective

In surgery, quality assessment encourages improved care delivery, better outcomes, and helps determine surgical efficacy. Quality is important from a patient, provider, payer, and policy maker standpoint. However, given the growth of outpatient procedures, expansion of surgical indications to enhance function, and the decline of perioperative morbidity and mortality, many traditional quality metrics, such as mortality, readmissions, and complications, may not fully capture quality. As such, patient-reported outcomes (PROs) can be used to complement the established clinical outcomes and describe surgical efficacy and quality from the patient's point of view. Generic and disease-specific PRO measures capture health-related quality of life, functional status, and pain. These measures permit a more holistic understanding of how surgery affects different aspects of a patient's health, augment other clinical outcomes, and are commonly used to determine efficacy in clinical trials. Moreover, our national reimbursement structure is currently evolving to include PROs for certain surgical conditions in measures of quality and with direct linkage to payments. Even so, there continues to be challenges in the implementation of PRO measures in everyday surgical practice, with questions of optimal administration and how to integrate these measures into provider work flow. Despite these challenges, PROs provide vital information regarding surgical efficacy and quality and are critical in the delivery of patient-centered care.

Stability Enhanced Variable Selection for a Semiparametric Model with Flexible Missingness Mechanism and Its Application to the ChAMP Study

This paper is motivated by the analytical challenges we encounter when analyzing the ChAMP (Chondral Lesions And Meniscus Procedures) study, a randomized controlled trial to compare debridement to observation of chondral lesions in arthroscopic knee surgery. The main outcome, WOMAC (Western Ontario and McMaster Universities Osteoarthritis Index) pain score, is derived from the patient's responses to the questionnaire collected in the study. The major goal is to identify potentially important variables that contribute to this outcome. In this paper, the model of interest is a semiparametric model for the pain score. To address the missing data issue, we adopt a flexible missingness mechanism which is much more versatile in practice than a single parametric model. Then we propose a pairwise conditional likelihood approach to estimate the unknown parameter in the semiparametric model without the need of modeling its nonparametric counterpart nor the missingness mechanism. For variable selection we apply a regularization approach with a variety of stability enhanced tuning parameter selection methods. We conduct comprehensive simulation studies to evaluate the performance of the proposed method. We also apply the proposed method to the ChAMP study to demonstrate its usefulness.

Impact of missing data on bias and precision when estimating change in patient-reported outcomes from a clinical registry

BACKGROUND

Clinical registries, which capture information about the health and healthcare use of patients with a health condition or treatment, often contain patient-reported outcomes (PROs) that provide insights about the patient's perspectives on their health. Missing data can affect the value of PRO data for healthcare decision-making. We compared the precision and bias of several missing data methods when estimating longitudinal change in PRO scores.

METHODS

This research conducted analyses of clinical registry data and simulated data. Registry data were from a population-based regional joint replacement registry for Manitoba, Canada; the study cohort consisted of 5631 patients having total knee arthroplasty between 2009 and 2015. PROs were measured using the 12-item Short Form Survey, version 2 (SF-12v2) at pre- and post-operative occasions. The simulation cohort was a subset of 3000 patients from the study cohort with complete PRO information at both pre- and post-operative occasions. Linear mixed-effects models based on complete case analysis (CCA), maximum likelihood (ML) and multiple imputation (MI) without and with an auxiliary variable (MI-Aux) were used to estimate longitudinal change in PRO scores. In the simulated data, bias, root mean squared error (RMSE), and 95% confidence interval (CI) coverage and width were estimated under varying amounts and types of missing data.

RESULTS

Three thousand two hundred thirty (57.4%) patients in the study cohort had complete data on the SF-12v2 at both occasions. In this cohort, mixed-effects models based on CCA resulted in substantially wider 95% CIs than models based on ML and MI methods. The latter two methods produced similar estimates and 95% CI widths. In the simulation cohort, when 50% of the data were missing, the MI-Aux method, in which a single hypothetical auxiliary variable was strongly correlated (i.e., 0.8) with the outcome, reduced the 95% CI width by up to 14% and bias and RMSE by up to 50 and 45%, respectively, when compared with the MI method.

CONCLUSIONS

Missing data can substantially affect the precision of estimated change in PRO scores from clinical registry data. Inclusion of auxiliary information in MI models can increase precision and reduce bias, but identifying the optimal auxiliary variable(s) may be challenging.

Bias through selective inclusion and attrition: Representativeness when comparing provider performance with routine outcome monitoring data

Background

Observational research based on routine outcome monitoring is prone to missing data, and outcomes can be biased due to selective inclusion at baseline or selective attrition at posttest. As patients with complete data may not be representative of all patients of a provider, missing data may bias results, especially when missingness is not random but systematic.

Methods

The present study establishes clinical and demographic patient variables relevant for representativeness of the outcome information. It applies strategies to estimate sample selection bias (weighting by inclusion propensity) and selective attrition bias (multiple imputation based on multilevel regression analysis) and estimates the extent of their impact on an index of provider performance. The association between estimated bias and response rate is also investigated.

Results

Provider‐based analyses showed that in current practice, the effect of selective inclusion was minimal, but attrition had a more substantial effect, biasing results in both directions: overstating and understating performance. For 22% of the providers, attrition bias was estimated to be in excess of 0.05 ES. Bias was associated with overall response rate (r = .50). When selective inclusion and attrition bring providers' response below 50%, it is more likely that selection bias increased beyond a critical level, and conclusions on the comparative performance of such providers may be misleading.

Conclusions

Estimates of provider performance were biased by selection, especially by missing data at posttest. Results on the extent and direction of bias and minimal requirements for response rates to arrive at unbiased performance indicators are discussed.

Electronic capturing of patient-reported outcome measures on a touchscreen computer in clinical diabetes practice (the DiaPROM trial): a feasibility study

BACKGROUND

Living with type 1 diabetes (T1D) is demanding, and emotional problems may impair ability for diabetes self-management. Thus, diabetes guidelines recommend regular assessment of such problems. Using patient-reported outcome measures (PROMs) to assess diabetes-related distress and psychological well-being is considered useful. It has been proposed that future work should examine the use of PROMs to support the care of individual patients and improve the quality of health services. To our knowledge, the use of PROMs has not been systematically evaluated in diabetes care services in Norway. Electronically captured PROMs can be directly incorporated into electronic patient records. Thus, the study's overall aim was to examine the feasibility and acceptability of capturing PROMs electronically on a touchscreen computer in clinical diabetes practice.

METHODS

Adults with T1D age ≥ 40 years completed PROMs on a touchscreen computer at Haukeland University Hospital's diabetes outpatient clinic. We included 46 items related to diabetes-related distress, self-perceived diabetes competence, awareness of hypoglycaemia, occurrence of hyperglycaemia, hypoglycaemia and fluctuating glucose levels, routines for glucose monitoring, general well-being and health-related quality of life. Participants subsequently completed a paper-based questionnaire regarding comprehension and relevance of the PROMs, acceptance of the number of items and willingness to complete electronic PROMs annually. We wrote field notes in the outpatient clinic based on observations and comments from the invited participants.

RESULTS

During spring 2017, 69 participants (50.7% men), age 40 to 74 years, were recruited. Generally, the touchscreen computer functioned well technically. Median time spent completing the PROMs was 8 min 19 s. Twenty-nine (42.0%) participants completed the PROMs without missing items, with an 81.4% average instrument completion rate. Participants reported that the PROMs were comprehensible (n = 62) and relevant (n = 46) to a large or very large degree, with an acceptable number of items (n = 51). Moreover, 54 were willing to complete PROMs annually. Participants commented that the focus on living with diabetes was valued.

CONCLUSIONS

Capturing PROMs on a touchscreen computer in an outpatient clinic was technically and practically feasible. The participants found the PROMs to be relevant and acceptable with a manageable number of items, and reported willingness to complete PROMs annually.

Are cost differences between specialist and general hospitals compensated by the prospective payment system?

Prospective payment systems fund hospitals based on a fixed-price regime that does not directly distinguish between specialist and general hospitals. We investigate whether current prospective payments in England compensate for differences in costs between specialist orthopaedic hospitals and trauma and orthopaedics departments in general hospitals. We employ reference cost data for a sample of hospitals providing services in the trauma and orthopaedics specialty. Our regression results suggest that specialist orthopaedic hospitals have on average 13% lower profit margins. Under the assumption of break-even for the average trauma and orthopaedics department, two of the three specialist orthopaedic hospitals appear to make a loss on their activity. The same holds true for 33% of departments in our sample. Patient age and severity are the main drivers of such differences.

Association Between Symptom Duration and Patient-Reported Outcomes Before and After Hip Replacement Surgery

OBJECTIVE

Patients experience discomfort and compromised quality of life while waiting for hip replacement. Symptom duration may affect quality of life attained following surgery. We undertook this study to investigate the impact of symptom duration on patient-reported postsurgical outcomes from hip replacement surgery.

METHODS

National observational data collected before and after hip replacement surgery in England between 2009 and 2016 were used to investigate determinants of symptom duration prior to surgery and the relationship between symptom duration and presurgical and postsurgical patient-reported outcomes. Multivariable linear regression models were used to estimate associations between patient-reported outcomes and symptom duration, controlling for a range of covariates.

RESULTS

The sample included 209,192 patients; most (69%) experienced symptoms for 1-5 years. A few patients (14%) experienced symptoms for <1 year, for longer than 5 years (6-10 years [11%]), or for >10 years (5%). Symptom duration decreased overall over the studied time period and was shorter among patients who were male, older, and from areas of lesser deprivation. Patients with a symptom duration <1 year had better postsurgical pain and function outcomes (Oxford Hip Score [OHS] 0.875 [95% confidence interval (95% CI) 0.777, 0.973]) than those with 1-5 years symptom duration in an adjusted model. Conversely, those with symptom duration >5 years had increasingly poorer postsurgical outcomes (OHS -0.730 [95% CI -0.847, -0.613] for those with disease duration 6-10 years and OHS -1.112 [95% CI -1.278, -0.946] for those with disease duration >10 years).

CONCLUSION

Symptom duration prior to hip replacement has become more standardized in England over time. However, increasing duration remains a significant predictor of poorer outcomes after surgery.

A Bayesian framework for health economic evaluation in studies with missing data

Health economics studies with missing data are increasingly using approaches such as multiple imputation that assume that the data are "missing at random." This assumption is often questionable, as-even given the observed data-the probability that data are missing may reflect the true, unobserved outcomes, such as the patients' true health status. In these cases, methodological guidelines recommend sensitivity analyses to recognise data may be "missing not at random" (MNAR), and call for the development of practical, accessible approaches for exploring the robustness of conclusions to MNAR assumptions. Little attention has been paid to the problem that data may be MNAR in health economics in general and in cost-effectiveness analyses (CEA) in particular. In this paper, we propose a Bayesian framework for CEA where outcome or cost data are missing. Our framework includes a practical, accessible approach to sensitivity analysis that allows the analyst to draw on expert opinion. We illustrate the framework in a CEA comparing an endovascular strategy with open repair for patients with ruptured abdominal aortic aneurysm, and provide software tools to implement this approach.

Detecting and Visualizing Outliers in Provider Profiling Using Funnel Plots and Mixed Effects Models-An Example from Prescription Claims Data

When prescribing a drug for a patient, a physician also has to consider economic aspects. We were interested in the feasibility and validity of profiling based on funnel plots and mixed effect models for the surveillance of German ambulatory care physicians’ prescribing. We analyzed prescriptions issued to patients with a health insurance card attending neurologists’ and psychiatrists’ ambulatory practices in the German federal state of Saarland. The German National Association of Statutory Health Insurance Physicians developed a prescribing assessment scheme (PAS) which contains a systematic appraisal of the benefit of drugs for so far 12 different indications. The drugs have been classified on the basis of their clinical evidence as “standard”, “reserve” or “third level” medication. We had 152.583 prescriptions in 56 practices available for analysis. A total of 38.796 patients received these prescriptions. The funnel plot approach with additive correction for overdispersion was almost equivalent to a mixed effects model which directly took the multilevel structure of the data into account. In the first case three practices were labeled as outliers, the mixed effects model resulted in two outliers. We suggest that both techniques should be routinely applied within a surveillance system of prescription claims data.

Comparison of statistical approaches for analyzing incomplete longitudinal patient-reported outcome data in randomized controlled trials

Purpose

Missing data are a potential source of bias in the results of RCTs, but are often unavoidable in clinical research, particularly in patient-reported outcome measures (PROMs). Maximum likelihood (ML), multiple imputation (MI), and inverse probability weighting (IPW) can be used to handle incomplete longitudinal data. This paper compares their performance when analyzing PROMs, using a simulation study based on an RCT data set.

Methods

Realistic missing-at-random data were simulated based on patterns observed during the follow-up of the knee arthroscopy trial (ISRCTN45837371). Simulation scenarios covered different sample sizes, with missing PROM data in 10%–60% of participants. Monotone and nonmonotone missing data patterns were considered. Missing data were addressed by using ML, MI, and IPW and analyzed via multilevel mixed-effects linear regression models. Root mean square errors in the treatment effects were used as performance parameters across 1,000 simulations.

Results

Nonconvergence issues were observed for IPW at small sample sizes. The performance of all three approaches worsened with decreasing sample size and increasing proportions of missing data. MI and ML performed similarly when the MI model was restricted to baseline variables, but MI performed better when using postrandomization data in the imputation model and also in nonmonotone versus monotone missing data scenarios. IPW performed worse than ML and MI in all simulation scenarios.

Conclusion

When additional postrandomization information is available, MI can be beneficial over ML for handling incomplete longitudinal PROM data. IPW is not recommended for handling missing PROM data in the simulated scenarios.

Methods for the economic evaluation of changes to the organisation and delivery of health services: principal challenges and recommendations

There is a requirement for economic evaluation of health technologies seeking public funding across Europe. Changes to the organisation and delivery of health services, including changes to health policy, are not covered by such appraisals. These changes also have consequences for National Health Service (NHS) funds, yet undergo no mandatory cost-effectiveness assessment. The focus on health technologies may have occurred because larger-scale service changes pose more complex challenges to evaluators. This paper discusses the principal challenges faced when performing economic evaluations of changes to the organisation and delivery of health services and provides recommendations for overcoming them. The five principal challenges identified are as follows: undertakingex-anteevaluation; evaluating impacts in terms of quality-adjusted life years; assessing costs and opportunity costs; accounting for spillover effects; and generalisability. Of these challenges, methods for estimating the impact on costs and quality-adjusted life years are those most in need of development. Methods are available forex-anteevaluation, assessing opportunity costs and examining generalisability. However, these are rarely applied in practice. The general principles of assessing the cost-effectiveness of interventions should be applied to all NHS spending, not just that involving health technologies. Advancements in this area have the potential to improve the allocation of scarce NHS resources.

Quality-of-Life Metrics Correlate With Disease Severity in Idiopathic Subglottic Stenosis

OBJECTIVES

Idiopathic subglottic stenosis (ISGS) can have significant impacts on quality of life (QOL), but it remains unclear how patients' subjective responses correlate with objective measurement of disease severity. Peak expiratory flow percentage (PEF%) has been shown to be an effective measure of disease severity in subglottic stenosis. This study aims to identify the key QOL questions correlated with PEF% and proposes a statistical model for prediction of disease severity.

METHODS

Patients with ISGS presenting to an academic laryngologist were included retrospectively from 2012 to 2016. Peak expiratory flow percentage (age, sex, and height adjusted) was recorded for each visit, along with four validated QOL instruments (European QOL-Five Dimensions; RAND 36-Item Health Survey; Clinical COPD [Chronic Obstructive Pulmonary Disease] Questionnaire; and the Airway, Dyspnea, Voice, and Swallowing Summary Assessment). A stepwise multiple linear regression was used to identify statistically significant independent variables correlated with PEF%, and a model was built with these variables.

RESULTS

Thirty-two patients were included, with a total of 271 patient encounters. Overall scores from each of the four QOL instruments were correlated with PEF% values recorded each visit (P < 0.05). Question responses correlating most positively included overall breathlessness, difficulty catching breath, cough within the past week, dyspnea with moderate activity, perception that voice changes are restricting social life, and overall general health (all P < 0.01). A model constructed using six nonoverlapping questions yielded an adjusted R2 of 0.58.

CONCLUSION

Quality of life is correlated to PEF% in ISGS. Using a limited number of QOL questions, clinicians can predict objective worsening or improvement of disease severity, as measured by spirometry.

LEVEL OF EVIDENCE

2b. Laryngoscope, 2017.

Multidimensional performance assessment of public sector organisations using dominance criteria

Public sector organisations pursue multiple objectives and serve a number of stakeholders. But stakeholders are rarely explicit about the valuations they attach to different objectives, nor are these valuations likely to be identical. This complicates the assessment of their performance because no single set of weights can be chosen legitimately to aggregate outputs into unidimensional composite scores. We propose the use of dominance criteria in a multidimensional performance assessment framework to identify best practice and poor performance under relatively weak assumptions about stakeholders' preferences. We use as an example providers of hip replacement surgery in the English National Health Service and estimate multivariate multilevel models to study their performance in terms of length of stay, readmission rates, post-operative patient-reported health status and waiting time. We find substantial correlation between objectives and demonstrate that ignoring the correlation can lead to incorrect assessments of performance.

Elective hospital admissions: secondary data analysis and modelling with an emphasis on policies to moderate growth

Background

The English NHS faces financial pressures that may render the growth rates of elective admissions seen between 2001/2 and 2011/12 unsustainable. A better understanding of admissions growth, and the influence of policy, are needed to minimise the impact on health gain for patients.

Objectives

This project had several objectives: (1) to better understand the determinants of elective activity and policy to moderate growth at minimum health loss for patients; (2) to build a rich data set integrating health, practice and local area data to study general practitioner (GP) referrals and resulting admissions; (3) to predict patients whose treatment is unlikely to be cost-effective using patient-reported outcomes and to examine variation in provider performance; and (4) to study how policies that aim to reduce elective admissions may change demand for emergency care. The main drivers of elective admissions growth have increased either supply of or demand for care, and could include, for example, technical innovations or increased awareness of treatment benefits. Of the factors studied, neither system reform nor population ageing appears to be a key driver. The introduction of the prospective payment tariff ‘Payment by Results’ appears to have led to primary care trusts (PCTs) having increasingly similar lengths of stay. In deprived areas, increasing GP supply appears to moderate elective admissions. Reducing the incidence of single-handed practices tends to reduce referrals and admissions. Policies to reduce referrals are likely to reduce admissions but treatments may be particularly reduced in the lowest referring practices, in which resulting health loss may be greatest. In this model, per full-time equivalent, female and highly experienced GPs identify more patients admitted by specialists.

Results

It appears from our studies that some patient characteristics are associated with not achieving sufficient patient gain to warrant cost-effective treatment. The introduction of independent sector treatment centres is estimated to have caused an increase in emergency activity rates at local PCTs. The explanations offered for increasing elective admissions indicate that they are manageable by health policy.

Conclusions

Further work is required to understand some of the results identified, such as whether or not high-volume Clinical Commissioning Groups are fulfilling unmet need; why some practices refer at low rates relative to admissions; why the period effect, which results from factors that equally affect all in the study at a point in time, dominates in the age–period–cohort analysis; and exactly how the emergency and elective sections of hospital treatment interact. This project relies on the analysis of secondary data. This type of research does not easily facilitate the important input of clinical experts or service users. It would be beneficial if other methods, including surveys and consultation with key stakeholders, could be incorporated into future research now that we have uncovered important questions.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

Choice of hospital: Which type of quality matters?

The implications of hospital quality competition depend on what type of quality affects choice of hospital. Previous studies of quality and choice of hospitals have used crude measures of quality such as mortality and readmission rates rather than measures of the health gain from specific treatments. We estimate multinomial logit models of hospital choice by patients undergoing hip replacement surgery in the English NHS to test whether hospital demand responds to quality as measured by detailed patient reports of health before and after hip replacement. We find that a one standard deviation increase in average health gain increases demand by up to 10%. The more traditional measures of hospital quality are less important in determining hospital choice.

Challenges, solutions and future directions in the evaluation of service innovations in health care and public health

Evaluating service innovations in health care and public health requires flexibility, collaboration and pragmatism; this collection identifies robust, innovative and mixed methods to inform such evaluations.

Comparing the performance of English mental health providers in achieving patient outcomes

Evidence on provider payment systems that incorporate patient outcomes is limited for mental health care. In England, funding for mental health care services is changing to a prospective payment system with a future objective of linking some part of provider payment to outcomes. This research examines performance of mental health providers offering hospital and community services, in order to investigate if some are delivering better outcomes. Outcomes are measured using the Health of the Nation Outcome Scales (HoNOS) - a clinician-rated routine outcome measure (CROM) mandated for national use. We use data from the Mental Health Minimum Data Set (MHMDS) - a dataset on specialist mental health care with national coverage - for the years 2011/12 and 2012/13 with a final estimation sample of 305,960 observations with follow-up HoNOS scores. A hierarchical ordered probit model is used and outcomes are risk adjusted with independent variables reflecting demographic, need, severity and social indicators. A hierarchical linear model is also estimated with the follow-up total HoNOS score as the dependent variable and the baseline total HoNOS score included as a risk-adjuster. Provider performance is captured by a random effect that is quantified using Empirical Bayes methods. We find that worse outcomes are associated with severity and better outcomes with older age and social support. After adjusting outcomes for various risk factors, variations in performance are still evident across providers. This suggests that if the intention to link some element of provider payment to outcomes becomes a reality, some providers may gain financially whilst others may lose. The paper contributes to the limited literature on risk adjustment of outcomes and performance assessment of providers in mental health in the context of prospective activity-based payment systems.