A conceptual framework to develop a patient-reported experience measure of the quality of mental health care: a qualitative study of the PREMIUM project in France

Psychometric Assessment of an Item Bank for Adaptive Testing on Patient-Reported Experience of Care Environment for Severe Mental Illness: Validation Study

BACKGROUND

The care environment significantly influences the experiences of patients with severe mental illness and the quality of their care. While a welcoming and stimulating environment enhances patient satisfaction and health outcomes, psychiatric facilities often prioritize staff workflow over patient needs. Addressing these challenges is crucial to improving patient experiences and outcomes in mental health care.

OBJECTIVE

This study is part of the Patient-Reported Experience Measure for Improving Quality of Care in Mental Health (PREMIUM) project and aims to establish an item bank (PREMIUM-CE) and to develop computerized adaptive tests (CATs) to measure the experience of the care environment of adult patients with schizophrenia, bipolar disorder, or major depressive disorder.

METHODS

We performed psychometric analyses including assessments of item response theory (IRT) model assumptions, IRT model fit, differential item functioning (DIF), item bank validity, and CAT simulations.

RESULTS

In this multicenter cross-sectional study, 498 patients were recruited from outpatient and inpatient settings. The final PREMIUM-CE 13-item bank was sufficiently unidimensional (root mean square error of approximation=0.082, 95% CI 0.067-0.097; comparative fit index=0.974; Tucker-Lewis index=0.968) and showed an adequate fit to the IRT model (infit mean square statistic ranging between 0.7 and 1.0). DIF analysis revealed no item biases according to gender, health care settings, diagnosis, or mode of study participation. PREMIUM-CE scores correlated strongly with satisfaction measures (r=0.69-0.78; P<.001) and weakly with quality-of-life measures (r=0.11-0.21; P<.001). CAT simulations showed a strong correlation (r=0.98) between CAT scores and those of the full item bank, and around 79.5% (396/498) of the participants obtained a reliable score with the administration of an average of 7 items.

CONCLUSIONS

The PREMIUM-CE item bank and its CAT version have shown excellent psychometric properties, making them reliable measures for evaluating the patient experience of the care environment among adults with severe mental illness in both outpatient and inpatient settings. These measures are a valuable addition to the existing landscape of patient experience assessment, capturing what truly matters to patients and enhancing the understanding of their care experiences.

TRIAL REGISTRATION

ClinicalTrials.gov NCT02491866; https://clinicaltrials.gov/study/NCT02491866.

How can we improve the care of patients with schizophrenia in the real-world? A population-based cohort study of 456,003 patients

An important step to improve outcomes for patients with schizophrenia is to understand treatment patterns in routine practice. The aim of the current study was to describe the long-term management of patients with schizophrenia treated with antipsychotics (APs) in real-world practice. This population-based study included adults with schizophrenia and who had received ≥3 deliveries of an AP from 2012-2017, identified using a National Health Data System. Primary endpoints were real-life prescription patterns, patient characteristics, healthcare utilization, comorbidities and mortality. Of the 456,003 patients included, 96% received oral APs, 17.5% first-generation long-acting injectable APs (LAIs), and 16.1% second generation LAIs. Persistence rates at 24 months after treatment initiation were 23.9% (oral APs), 11.5% (first-generation LAIs) and 20.8% (second-generation LAIs). Median persistence of oral APs, first-generation LAIs and second-generation LAIs was 5.0, 3.3, and 6.1 months, respectively. Overall, 62.1% of patients were administered anxiolytics, 45.7% antidepressants and 28.5% anticonvulsants, these treatments being more frequently prescribed in women and patients aged ≥50 years. Dyslipidemia was the most frequent metabolic comorbidity (16.2%) but lipid monitoring was insufficient (median of one occasion). Metabolic comorbidities were more frequent in women. Standardized patient mortality remained consistently high between 2013 and 2015 (3.3-3.7 times higher than the general French population) with a loss of life expectancy of 17 years for men and 8 years for women. Cancer (20.2%) and cardiovascular diseases (17.2%) were the main causes of mortality, and suicide was responsible for 25.4% of deaths among 18-34-year-olds. These results highlight future priorities for care of schizophrenia patients. The global persistence of APs used in this population was low, whereas rates of psychiatric hospitalization remain high. More focus on specific populations is needed, such as patients aged >50 years to prevent metabolic disturbances and 18-34-year-olds to reduce suicide rates.

Untreated major depression in healthcare workers: Results from the nationwide AMADEUS survey

AIMS

Recent studies have suggested increased rates of depression in healthcare workers (HCWs), with direct impact on care quality and productivity. The objective was to determine the proportion of HCWs adequately treated for their major depression in a nationwide survey, the proportion of lifestyle risk factors for depression, and to determine if working in psychiatry department or professional status may modulate these outcomes.

DESIGN

The method followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement guidelines. A survey comprising the Center for Epidemiologic Studies-Depression Scale was sent to public and private national healthcare facilities.

RESULTS

Overall, 10,325 participants were recruited; 3438 (33.3%) [95% confidence interval 32.4-34.2] met likely diagnostic criteria for clinical depression. Almost 80% of them declared that they had a history of major depressive episode. However, only approximately 23% of them were treated with antidepressants and approximately 13% had a psychiatric follow-up. Depressed HCWs working in psychiatry departments received slightly better care for their depression, but they also consumed more anxiolytics and hypnotics and had more risk factors for depression (including smoking and hazardous drinking). We found specificities according to professions, physicians reported higher rates of hazardous drinking while nurses, nurse assistants, and health executives had higher rates of smoking and twice as much obesity as physicians.

CONCLUSION

Disseminating tools to detect major depression, programs destigmatizing depression and antidepressants, promoting physical activity, weight loss, tobacco cessation and reduced alcohol consumption are promising strategies to improve the care of major depression in HCWs.

RELEVANCE TO CLINICAL PRACTICE

Reducing depression in healthcare workers is necessary to improve the quality of care, to limit burnout, medical errors, absenteeism and turn-over and globally to improve the wellbeing at work.

Reducing Health Inequalities in Individuals with Severe Mental Disorders: Harnessing Real-World Data and Patient-Reported Measures

Mental disorders are the leading cause of diminished lifespans worldwide and make up 5 of the top 10 most significant causes of disability [...].

Zufriedenheit mit (teil-)stationärer psychiatrischer Behandlung: Effekte auf die physische und psychische Gesundheit

INTRODUCTION

Quality assessment from the patient's point of view makes it possible to identify negative quality developments at an early stage. The focus is not on the medical result, but on what the patient wants. Correlations between patient satisfaction and physical and psychological treatment outcome were already shown in the 1990s. However, studies using rather unspecific satisfaction measures are scarce. The aim of this study was to investigate the influence of patient satisfaction with the treatment and the therapies offered on the extent of recovery.

METHODS

In this prospective study, a questionnaire developed for the differentiated recording of patient satisfaction with the therapy offerings of the LWL-Klinik Dortmund was used in a day-care/hospital setting. The structure of the questionnaire was tested by means of explorative factor analysis. The factors generated in this way served as the basis for the hierarchical regression analyses in the further course. In addition to important treatment aspects from the patient's point of view, the subjective health status was recorded by means of SF-36.

RESULTS

105 patients participated in the study (64% female, 84% diagnosed with depression). Significant predictors for physical health were well-being after exercise therapy and satisfaction with the weekly structure of services. Significant predictors for mental health were age at onset of illness, age, perceived benefits from exercise therapy as well as occupational therapy, treatment duration and setting.

DISCUSSION

The demonstrated impact of patient satisfaction on mental health highlights the relevance of treatment quality improvement to recovery.

How subjective well-being, patient-reported clinical improvement (PROMs) and experience of care (PREMs) relate in an acute psychiatric care setting?

BACKGROUND

Patient-reported outcome measures (PROMs) and patient-reported experience measures (PREMs) are increasingly acknowledged as critical tools for enhancing patient-centred, value-based care. However, research is lacking on the impact of using standardized patient-reported indicators in acute psychiatric care. The aim of this study was to explore whether subjective well-being indicators (generic PROMs) are relevant for evaluating the quality of hospital care, distinct from measures of symptom improvement (disease-specific PROMs) and from PREMs.

METHODS

Two hundred and forty-eight inpatients admitted to a psychiatric university hospital were included in the study between January and June 2021. Subjective well-being was assessed using standardized generic PROMs on well-being, symptom improvement was assessed using standardized disease-specific PROMs, and experience of care using PREMs. PROMs were completed at admission and discharge, PREMs were completed at discharge. Clinicians rated their experience of providing treatment using adapted PREMs items.

RESULTS

Change in subjective well-being (PROMs) at discharge was significantly (p < 0.001), but moderately (r² = 28.5%), correlated to improvement in symptom outcomes, and weakly correlated to experience of care (PREMs) (r² = 11.0%), the latter being weakly explained by symptom changes (r² = 6.9%). Patients and clinicians assessed the experience of care differently.

CONCLUSIONS

This study supports the case for routinely measuring patients' subjective well-being to better capture the unmet needs of patients undergoing psychiatric hospital treatment, and the use of standardized patient-reported measures as key indicators of high quality of care across mental health services.

Development of the PREMIUM Computerized Adaptive Testing for Measuring the Quality of Information Delivered to Patients with Severe Mental Illnesses

Measuring the quality of information delivered to patients with severe mental illness (SMI), i.e., schizophrenia, bipolar disorders, and major depressive disorders, is essential to improve their quality of care. In this work, we described the different steps of the validation of the PREMIUM computerized adaptive testing (CAT) for measuring the quality of information delivered to patients with SMI. The PREMIUM item bank regarding information included 25 items. A total of 499 patients with schizophrenia (53%), bipolar disorders (26%), and major depressive disorders (22%) were recruited from numerous in- and outpatient settings in France. Unidimensionality, local independence, and monotonicity were verified for 19 items of the item bank. The psychometric properties were satisfactory for both internal (RMSEA = 0.069, CFI = 0.969, TLI = 0.963) and external validity (in particular, significant associations were found with age, educational level, and social functioning). The CAT exhibited satisfactory accuracy and precision (standard error of measurement <0.55 and root mean square error <0.3), with an average administration of eight items. This CAT may be used by healthcare professionals in psychiatric settings to accurately assess the patients’ experience with information.

Calibration and Validation of a PREMIUM-DT Item Bank to Measure the Experience of Drug Therapy for Patients with Severe Mental Illness

The aim of this study was to (1) calibrate an item bank to measure patients’ experience of drug therapy for adult patients with SMIs and (2) develop computerized adaptive testing (CAT) to improve its use in routine practice. This is a cross-sectional, multicentric study involving 541 patients with schizophrenia, bipolar disorder, and major depressive disorder. Analyses based on classical test and item response theories were performed. After 7 highly inter-correlated items and 4 items with low factor loadings were removed, the remaining 26 items were sufficiently unidimensional (RMSEA = 0.069, CFI = 0.969, TLI = 0.963) and showed adequate fit to the generalized partial credit model. There was no differential item functioning by gender, age, care setting, or diagnosis from moderate- to large-magnitude. The mean score was 46.0 ± 16.9 and was significantly higher for patients reporting good medication adherence. The resulting PREMIUM-DT item bank has strong psychometric properties, and CAT facilitates widespread use in clinical settings (an average of 8 items administered, corresponding to a reliability of >0.90). Our results suggest that practical information and information about the side effects of psychotropic treatments and how to cope with them should be targeted as a priority to improve patients’ experience of drug therapy.

Development and Calibration of the PREMIUM Item Bank for Measuring Respect and Dignity for Patients with Severe Mental Illness

Most patient-reported experience measures (PREMs) are paper-based, leading to a high burden for patients and care providers. The aim of this study was to (1) calibrate an item bank to measure patients’ experience of respect and dignity for adult patients with serious mental illnesses and (2) develop computerized adaptive testing (CAT) to improve the use of this PREM in routine practice. Patients with schizophrenia, bipolar disorder, and major depressive disorder were enrolled in this multicenter and cross-sectional study. Psychometric analyses were based on classical test and item response theories and included evaluations of unidimensionality, local independence, and monotonicity; calibration and evaluation of model fit; analyses of differential item functioning (DIF); testing of external validity; and finally, CAT development. A total of 458 patients participated in the study. Of the 24 items, 2 highly inter-correlated items were deleted. Factor analysis showed that the remaining items met the unidimensional assumption (RMSEA = 0.054, CFI = 0.988, TLI = 0.986). DIF analyses revealed no biases by sex, age, care setting, or diagnosis. External validity testing has generally supported our assumptions. CAT showed satisfactory accuracy and precision. This work provides a more accurate and flexible measure of patients’ experience of respect and dignity than that obtained from standard questionnaires.

The Validity of the SQoL-18 in Patients with Bipolar and Depressive Disorders: A Psychometric Study from the PREMIUM Project

The S-QoL 18 is a self-administered questionnaire that assesses quality of life (QoL) among individuals with schizophrenia. This study aims to validate the S-QoL 18 in bipolar and depressive disorders for a more widespread use in psychiatric settings. This study was conducted in a non-selected sample of individuals with bipolar and depressive disorders in the day hospital of a regional psychiatric academic hospital. Two-hundred and seventy-two stable outpatients with bipolar (n = 73) and recurrent and persistent depressive (n = 199) disorders were recruited over a 12 month-period. The S-QoL 18 was tested for construct validity, reliability, and external validity. The eight-factor structure of the S-QoL 18 was confirmed by confirmatory factor analysis (RMSEA = 0.075 (0.064–0.086), CFI = 0.972, TLI = 0.961). Internal consistency and reliability were satisfactory. External validity was confirmed via correlations between S-QoL 18 dimension scores, symptomatology, and functioning. The percentage of missing data for the eight dimensions did not exceed 5%. INFIT statistics were ranged from 0.7 to 1.2, ensuring that all items of the scale measured the same QoL concept. In conclusion, the S-QoL 18 appears to be a valid and reliable instrument for measuring QoL in patients with bipolar and depressive disorders. The S-QoL 18 may be used by healthcare professionals in clinical settings to accurately assess QoL in individuals with bipolar and depressive disorders, as well as in schizophrenia.

Needs and Experiences in Psychiatric Treatment (NEPT)- Piloting a Collaboratively Generated, Initial Research Tool to Evaluate Cross-Sectoral Mental Health Services

BACKGROUND

Research tools to evaluate institutions or interventions in the field of mental health have rarely been constructed by researchers with personal experience of using the mental health system ("experiential expertise"). This paper presents a preliminary tool that has been developed within a participatory-collaborative process evaluation as part of a controlled, multi-center, prospective cohort study (PsychCare) to evaluate psychiatric flexible and integrative treatment, FIT for short, models in Germany.

METHOD

The collaborative research team consisting of researchers with and without experiential expertise developed 12 experiential program components of FIT models by an iterative research process based on the Grounded Theory Methodology. These components were transformed into a preliminary research tool that was evaluated by a participatory expert panel, and during a pilot and validation study, the latter using a random sample of 327 users from 14 mental health departments. Internal consistency of the tool was tested using Cronbach's alpha. Construct validity was evaluated using a Principal Components Analysis (PCA) and a Jonckheere Terpstra test in relation to different implementation levels of the FIT model. Concurrent validity was tested against a German version of the Client Satisfaction Questionnaire (ZUF-8) using correlation analysis and a linear regression model.

RESULTS

The evaluation of the expert panel reduced 29 initial items to 16 that were further reduced to 11 items during the pilot study, resulting into a research tool (Needs and Experiences in Psychiatric Treatment-NEPT) that demonstrated good internal consistency (Cronbach's alpha of 0.89). PCA yielded a 1-component structure, which accounted for 49% of the total variance supporting the unidimensional structure of the tool. The total NEPT score increased alongside the increasing implementation of the FIT model (p < 0.05). There was evidence (p < 0.001) for convergent validity assessed against the ZUF-8 as criterion measure.

CONCLUSIONS

The NEPT tool seems to be promising for further development to assess the experiences with and fulfillment of needs of psychiatric care models from the perspective of users. This paper demonstrates that it is possible to use a participatory-collaborative approach within the methodologically rigorous confines of a prospective, controlled research design.