The importance of content and face validity in instrument development: lessons learnt from service users when developing the Recovering Quality of Life measure (ReQoL)

Validation of a high-fidelity training model for fetoscopic spina bifida surgery

Open fetal surgery for spina bifida (SB) is safe and effective yet invasive. The growing interest in fetoscopic SB repair (fSB-repair) prompts the need for appropriate training. We aimed to develop and validate a high-fidelity training model for fSB-repair. fSB-repair was simulated in the abdominal cavity and on the stomach of adult rabbits. Laparoscopic fetal surgeons served either as novices (n = 2) or experts (n = 3) based on their experience. Technical performance was evaluated using competency Cumulative Sum (CUSUM) analysis and the group splitting method. Main outcome measure for CUSUM competency was a composite binary outcome for surgical success, i.e. watertight repair, operation time ≤ 180 min and Objective-Structured-Assessment-of-Technical-Skills (OSATS) score ≥ 18/25. Construct validity was first confirmed since competency levels of novices and experts during their six first cases using both methods were significantly different. Criterion validity was also established as 33 consecutive procedures were needed for novices to reach competency using learning curve CUSUM, which is a number comparable to that of clinical fSB-repair. Finally, we surveyed expert fetal surgeons worldwide to assess face and content validity. Respondents (26/49; 53%) confirmed it with ≥ 71% of scores for overall realism ≥ 4/7 and usefulness ≥ 3/5. We propose to use our high-fidelity model to determine and shorten the learning curve of laparoscopic fetal surgeons and retain operative skills.

The quality in psychiatric care-Addiction outpatient instrument: Psychometric properties and patient views of the quality of care

Aim

The aim of the study was to evaluate the psychometric properties and factor structure of the Quality in Psychiatric Care—Addiction Outpatient (QPC‐AOP) instrument and to describe the experiences with the quality of care among addiction outpatients.

Design

The study has a cross‐sectional design.

Methods

A sample of 244 patients with addiction and psychiatric disorders completed the QPC‐AOP.

Results

Confirmatory factor analysis showed adequate to excellent goodness‐of‐fit indices supporting the 9‐factor structure of the QPC‐AOP. The results thus demonstrate that the concept of quality of care to a large extent is equivalent among outpatients from general psychiatry and from outpatient addiction services. Internal consistency for the full QPC‐AOP was adequate, but poor for some of the separate factors. The patients’ ratings of quality of care were generally high; the highest rating was for Encounter and the lowest for Discharge.

Quality of measures on respectful and disrespectful maternity care: A systematic review

This systematic review aimed to critique the process of development and psychometric properties of tools measuring respectful or disrespectful maternity care experienced by women during labor and birth in low- and middle-income countries. The MEDLINE, Embase, CINAHL, Web of Science, PubMed, and Cochrane Library electronic databases were systematically searched from their inception to February 2020. Methodological quality was assessed using the COnsensus-based Standards for the selection of health Measurement INstruments checklist. Six tools measuring respectful maternity care during the intrapartum period were identified. Measurement error, cross-cultural validity, and responsiveness were not evaluated by any tool developers, while structural validity, internal consistency, and hypothesis testing were the most frequently assessed measurement properties. Interestingly, this review could not identify any measures of disrespectful care even though most included measures focused on disrespect and abuse. No measure was of sufficient quality to determine women's experiences of disrespectful and respectful maternity care in low- and middle-income countries. New valid and reliable measures using rigorous approaches to tool development are required.

Estimating a Preference-Based Index for Mental Health From the Recovering Quality of Life Measure: Valuation of Recovering Quality of Life Utility Index

BACKGROUND

There are increasing concerns about the appropriateness of generic preference-based measures to capture health benefits in the area of mental health.

OBJECTIVES

The aim of this study is to estimate preference weights for a new measure, Recovering Quality of Life (ReQoL-10), to better capture the benefits of mental healthcare.

METHODS

Psychometric analyses of a larger sample of mental health service users (n = 4266) using confirmatory factor analyses and item response theory were used to derive a health state classification system and inform the selection of health states for utility assessment. A valuation survey with members of the UK public representative in terms of age, sex, and region was conducted using face-to-face interviewer administered time-trade-off with props. A series of regression models were fitted to the data and the best performing model selected for the scoring algorithm.

RESULTS

The ReQoL-Utility Index (UI) classification system comprises 6 mental health items and 1 physical health item. Sixty-four health states were valued by 305 participants. The preferred model was a random effects model, with significant and consistent coefficients and best model fit. Estimated utilities modeled for all health states ranged from -0.195 (state worse than dead) to 1 (best possible state).

CONCLUSIONS

The development of the ReQoL-UI is based on a novel application of item response theory methods for generating the classification system and selecting health states for valuation. Conventional time-trade-off was used to elicit utility values that are modeled to enable the generation of QALYs for use in cost-utility analysis of mental health interventions.

Enhancing 'meaningfulness' of functional assessments: UK adaptation of the Amsterdam IADL questionnaire

OBJECTIVE

Commonly used measures of instrumental activities of daily living (IADL) do not capture activities for a technologically advancing society. This study aimed to adapt the proxy/informant-based Amsterdam IADL Questionnaire (A-IADL-Q) for use in the UK and develop a self-report version.

DESIGN

An iterative mixed method cross-cultural adaptation of the A-IADL-Q and the development of a self-report version involving a three-step design: (1) interviews and focus groups with lay and professional stakeholders to assess face and content validity; (2) a questionnaire to measure item relevance to older adults in the U.K.; (3) a pilot of the adapted questionnaire in people with cognitive impairment.

SETTING

Community settings in the UK.

PARTICIPANTS

One hundred and forty-eight participants took part across the three steps: (1) 14 dementia professionals; 8 people with subjective cognitive decline (SCD), mild cognitive impairment (MCI), or dementia due to Alzheimer's disease; and 6 relatives of people with MCI or dementia; (2) 92 older adults without cognitive impairment; and (3) 28 people with SCD or MCI.

MEASUREMENTS

The cultural relevance and applicability of the A-IADL-Q scale items were assessed using a 6-point Likert scale. Cognitive and functional performance was measured using a battery of cognitive and functional measures.

RESULTS

Iterative modifications to the scale resulted in a 55-item adapted version appropriate for UK use (A-IADL-Q-UK). Pilot data revealed that the new and revised items performed well. Four new items correlated with the weighted average score (Kendall's Tau -.388, -.445, -.497, -.569). An exploratory analysis of convergent validity found correlations in the expected direction with cognitive and functional measures.

CONCLUSION

The A-IADL-Q-UK provides a measurement of functional decline for use in the UK that captures culturally relevant activities. A new self-report version has been developed and is ready for testing. Further evaluation of the A-IADL-Q-UK for construct validity is now needed.

Public preference for COVID-19 vaccines in China: A discrete choice experiment

BACKGROUND

As the coronavirus disease 2019 (COVID-19) pandemic is sweeping across the globe, there is an urgent need to develop effective vaccines as the most powerful strategy to end the pandemic. This study aimed to examine how factors related to vaccine characteristics, their social normative influence and convenience of vaccination can affect the public's preference for the uptake of the COVID-19 vaccine in China.

METHODS

An online discrete choice experiment (DCE) survey was administered to a sample of China's general population. Participants were asked to make a series of hypothetical choices and estimate their preference for different attributes of the vaccine. A mixed logit regression model was used to analyse the DCE data. Willingness to pay for each attribute was also calculated.

RESULTS

Data of 1236 participants who provided valid responses were included in the analysis. There was strong public preference for high effectiveness of the vaccine, followed by long protective duration, very few adverse events and being manufactured overseas. Price was the least important attribute affecting the public preference in selecting the COVID-19 vaccine.

CONCLUSIONS

The strong public preferences detected in this study should be considered when developing COVID-19 vaccination programme in China. The results provide useful information for policymakers to identify the individual and social values for a good vaccination strategy.

PATIENT OR PUBLIC CONTRIBUTION

The design of the experimental choices was fully based on interviews and focus group discussions participated by 26 Chinese people with diverse socio-economic backgrounds. Without their participation, the study would not be possible.

Improving quality of life assessments for high-need adult Medicaid service users with mental health conditions

PURPOSE

There is a lack of consensus on how to evaluate health and social service programs for people with mental health (MH) conditions. Having service users be the primary decision makers in selecting outcome measures can inform a meaningful evaluation strategy. We sought to identify the quality of life (QoL) survey preferences of high-need adult service users with MH conditions.

METHODS

A systematic review identified generic, self-reported QoL surveys with evidence of validity in MH populations of interest. An advisory panel selected the most promising surveys to assess the success of programs like Medicaid for MH service users. Three groups of high-need, adult service users with MH conditions and one group of direct care staff ranked the surveys from the advisory panel, and generated and ranked characteristics that were desirable or undesirable in a QoL survey.

RESULTS

Twenty-two surveys met the inclusion criteria. Of the six surveys selected by the advisory panel, groups of service users and direct care staff most preferred the Warwick-Edinburg Mental Well-being Scale (WEMWBS). The WEMWBS best embodied the features prioritized by the groups: to have a user-friendly format and positive focus, to be clearly worded and brief, and to avoid presumptive or unrealistic items. Service user groups appreciated survey topics most amenable to self-report, such as satisfaction with relationships.

CONCLUSION

Using QoL surveys that service users prefer can reduce the chance that deteriorating QoL is going unchecked, and increase the chance that decisions based on survey findings are meaningful to service users.

Second Victim Experience and Support Tool: Persian translation and psychometric characteristics evaluation

BACKGROUND

Healthcare providers involved in an unexpected adverse patient safety event are affected by negative outcomes of the event and are called 'second victims'. The Second Victim Experience and Support Tool has been introduced as a reliable and valid measure to assess the second victim responses and to support recourses of healthcare professionals.

AIM

The core objective of this survey was to assess the psychometric attributes of the scale within Iranian nurse.

METHODS

Having translated the scale based on the WHO guidelines and assessing its face validity, a board of 8 specialists assessed the content validity of the measure. In order to assess the psychometric characteristics of the questionnaire, a survey was conducted in 5 university-affiliated hospitals of Urmia. Confirmatory factor analysis is adopted to assess the construct validity. The internal consistency and test-retest reliability of the scale were determined.

RESULTS

The Persian version of the scale consists of 29 items and 9 subscales. Confirmatory factor analysis illustrated that the scale reflects a foolproof construct validity. Cronbach α coefficients revealed the reliability of the scale (0.68-0.88).

CONCLUSIONS

The survey proves to be a reliable and valid measure to evaluate the second victim responses and conceivable support resources of Iranian nurses or other Persian-speaking healthcare professionals of the region.

IMPLICATIONS FOR HEALTH POLICY AND PRACTICE

Health policymakers ought to build up large-scale surveys to determine the degree of second victimization of healthcare professionals so as to offer suitable support systems. Taking the scale into account would improve evaluation of the benefits of second victim support initiatives in nursing practice.

Self-Report Rating Scales to Guide Measurement-Based Care in Child and Adolescent Psychiatry

Measurement-based care involves the practice of systematically administrating rating scales to patients in order to use the collected information to enhance clinical evaluation, monitor treatment progress, and directly inform decisions relating to each patient's treatment. Rating scales must be psychometrically validated and efficiently administered within the practice setting. Brief rating scales that are available within the public domain may help to optimize workflows and prevent response fatigue. Clinicians should also have a sufficient understanding of the underlying psychometric properties of rating scales to accurately interpret changes in scores over time and use these results to appropriately direct care.

Assessment instruments for lower urinary tract dysfunction in children: Symptoms, characteristics and psychometric properties

INTRODUCTION

The use of instruments to assess symptoms of the lower urinary tract has increased mainly when it comes to pediatric urology. The search for a validated and reliable one for use in clinical practice should be a concern of the professionals involved in the care of these children.

OBJECTIVE

The aim of this study was to analyze the assessment instruments for lower urinary tract dysfunction in children regarding: symptoms, characteristics and psychometric properties.

STUDY DESIGN

PubMed and Latin American and Caribbean Literature in Health Sciences databases were searched. The PICO strategy was used to construct the research question and bibliographic search. Keywords included voiding disorders, LUTS, Child, Surveys and Questionnaires.

RESULTS

Nine articles were identified. The construct observed by all instruments was bladder dysfunction and seven instruments also assessed bowel dysfunction. Thirty different events were addressed. In eight instruments the respondents are the parents. The age of the target population ranged from three to eighteen years and regarding the number of items, an average of 21.8 items was observed. Six instruments showed good results of reliability and four presented good results of accuracy.

CONCLUSIONS

The instruments included a wide range of symptoms related to dysfunction. Only two reported all validation and reliability tests, however six instruments showed good reliability results, making them eligible for the cross-cultural adaptation and validation process for use in countries other than those of the instrument's origin.

An item response theory analysis of an item pool for the recovering quality of life (ReQoL) measure

Purpose

ReQoL-10 and ReQoL-20 have been developed for use as outcome measures with individuals aged 16 and over, experiencing mental health difficulties. This paper reports modelling results from the item response theory (IRT) analyses that were used for item reduction.

Methods

From several stages of preparatory work including focus groups and a previous psychometric survey, a pool of items was developed. After confirming that the ReQoL item pool was sufficiently unidimensional for scoring, IRT model parameters were estimated using Samejima’s Graded Response Model (GRM). All 39 mental health items were evaluated with respect to item fit and differential item function regarding age, gender, ethnicity, and diagnosis. Scales were evaluated regarding overall measurement precision and known-groups validity (by care setting type and self-rating of overall mental health).

Results

The study recruited 4266 participants with a wide range of mental health diagnoses from multiple settings. The IRT parameters demonstrated excellent coverage of the latent construct with the centres of item information functions ranging from − 0.98 to 0.21 and with discrimination slope parameters from 1.4 to 3.6. We identified only two poorly fitting items and no evidence of differential item functioning of concern. Scales showed excellent measurement precision and known-groups validity.

Conclusion

The results from the IRT analyses confirm the robust structure properties and internal construct validity of the ReQoL instruments. The strong psychometric evidence generated guided item selection for the final versions of the ReQoL measures.

Electronic supplementary material

The online version of this article (10.1007/s11136-020-02622-2) contains supplementary material, which is available to authorized users.

Development and psychometric properties of a new brief scale for subjective personal agency (SPA-5) in people with schizophrenia

AIMS

Personal agency is a variable which potentially facilitates personal recovery in people with serious mental illness. This study aimed to develop a new brief measure for subjective personal agency that can be completed by people with serious mental illness.

METHODS

Two focus group interviews were first conducted with 11 people with schizophrenia to understand the fundamental components of subjective personal agency for people with serious mental illness living in the community. One group comprised six people with schizophrenia living in the community, while the other consisted of five people with schizophrenia working as peer-support workers. We then developed scale items through collaboration with people with schizophrenia and qualitative analysis (stage 1). A cross-sectional survey was then conducted to test the psychometric properties of the new scale among service users with schizophrenia in 18 assertive community treatment teams (stage 2). Factor validity was tested via exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). We evaluated convergent validity with the Boston University Empowerment Scale (BUES), divergent validity with the global assessment of functioning (GAF), internal consistency, and test-retest reliability.

RESULTS

Seven items were included in the scale at stage 1. In stage 2, 195 participants completed this scale. EFA revealed a one-factor model with five items. CFA indicated good model fit (χ2 statistics [CMIN] = 8.445, df = 5 (CMIN/df = 1.689), p = 0.133, comparative fit index = 0.974, Tucker-Lewis fit index = 0.949, root mean square error of approximation = 0.077 and standardised root mean squared residual = 0.042). The new scale was significantly correlated with total BUES score (r = 0.526, p < 0.001), but not with GAF score. Cronbach's α for internal consistency was 0.79, and intra-class correlation coefficient for test-retest reliability was 0.70.

CONCLUSION

We developed a new, five-item Subjective Personal Agency scale (SPA-5) that can be completed by people with serious mental illness. Further studies are needed to confirm the results outside Japan.

Development of a new tool for the early identification of communication-support needs in people living with dementia: An Australian face-validation study

Language and communication difficulties are common in dementia but limited tools are available for a timely assessment of those individuals who experience these difficulties. The Communication-Support Needs Assessment Tool in Dementia (CoSNAT-D) was developed to assist in (a) the early identification of communication difficulties among people with dementia in the community context; and (b) determining the level of their communication support needs to guide appropriate service access. The CoSNAT-D was developed based on a literature review. The face validity was tested with end-users (people with dementia and support persons of people with dementia) regarding relevance, wording, syntax, appropriateness and comprehensiveness of the items of the tool. Data were collected using semi-structured phone interviews. Seven people with dementia and 15 carers participated in the study. Feedback regarding items' appropriateness was largely positive and minor changes were made to improve comprehensiveness. A new item was added to the original item pool. The interviews highlighted the importance of including people with dementia in the assessment processes and person-centered approaches in this context. The CoSNAT-D presents a first step for the early identification of individual support needs of people living with dementia and communication difficulties who live in the community. Items of the CoSNAT-D have been determined face-valid by end-users. The face-validated version of the tool is currently undergoing further testing to determine additional relevant psychometric properties.

Development and validation of a conceptual model and strategy adoption guidelines for persons with dementia at risk of getting lost

Various solutions are available to address critical wandering; however, no guideline exists to allow for the successful adoption of these strategies. This study developed and validated a conceptual model and series of guidelines to facilitate choice of wander-management strategies. Face and content validity of the model and guidelines were assessed through an online and paper survey, individual interviews and focus groups across stakeholders. Directed content analysis was performed. Overall impression of the conceptual model and guidelines was positive. Valued features included the guidelines' ability to help users in choosing proactive wander-management strategies. Suggested changes included revisions in terminology and more factors and strategies to be added to the model and guidelines. This work is being disseminated and used by persons with dementia, caregivers, health practitioners and community service providers to identify strategies to mitigate the risks associated with critical wandering.

An emerging framework for fully incorporating public involvement (PI) into patient-reported outcome measures (PROMs)

Patient-reported outcome measures (PROMs) are widely used in the United Kingdom (UK) and internationally to report and monitor patients’ subjective assessments of their symptoms and functional status and also their quality of life. Whilst the importance of involving the public in PROM development to increase the quality of the developed PROM has been highlighted this practice is not widespread. There is a lack of guidance on how public involvement (PI) could be embedded in the development of PROMs, where the roles can be more complex than in other types of research. This paper provides a timely review and sets out an emerging framework for fully incorporating PI into PROM development.

Public involvement in health outcomes research: lessons learnt from the development of the recovering quality of life (ReQoL) measures

Background

To provide a model for Public involvement (PI) in instrument development and other research based on lessons learnt in the co-production of a recently developed mental health patient reported outcome measure called Recovering Quality of Life (ReQoL). While service users contributed to the project as research participants, this paper focuses on the role of expert service users as research partners, hence referred to as expert service users or PI.

Methods

At every stage of the development, service users influenced the design, content and face validity of the measure, collaborating with other researchers, clinicians and stakeholders who were central to this research. Expert service users were integral to the Scientific Group which was the main decision-making body, and also provided advice through the Expert Service User Group.

Results

During the theme and item generation phase (stage 1) expert service users affirmed the appropriateness of the seven domains of the Patient Reported Outcome Measure (activity, hope, belonging and relationships, self-perception, wellbeing, autonomy, and physical health). Expert service users added an extra 58 items to the pool of 180 items and commented on the results from the face and content validity testing (stage 2) of a refined pool of 88. In the item reduction and scale generation phase (stage 3), expert service users contributed to discussions concerning the ordering and clustering of the themes and items and finalised the measures. Expert service users were also involved in the implementation and dissemination of ReQoL (stage 4). Expert service users contributed to the interpretation of findings, provided inputs at every stage of the project and were key decision-makers. The challenges include additional work to make the technical materials accessible, extra time to the project timescales, including time to achieve consensus from different opinions, sometimes strongly held, and extra costs.

Conclusion

This study demonstrates a successful example of how PI can be embedded in research, namely in instrument development. The rewards of doing so cannot be emphasised enough but there are challenges, albeit surmountable ones. Researchers should anticipate and address those challenges during the planning stage of the project.

Health-related quality of life among children, adolescents, and adults with bladder exstrophy-epispadias complex: a systematic review of the literature and recommendations for future research

PURPOSE

Bladder exstrophy-epispadias complex (BEEC) is a rare spectrum of genitourinary malformations. Children risk long-term urinary and genital dysfunctions. To achieve a comprehensive understanding, this study aimed to review the literature on generic and disease-specific health-related quality of life (HRQOL) in BEEC patients, and methodologies used.

METHODS

A literature search was conducted in Pubmed/CINAHL/Embase/PsycINFO/Cochrane, from inception to May 2018. A meta-analysis of HRQOL in BEEC patients compared to healthy references was performed.

RESULTS

Twenty-one articles (published 1994-2018), describing HRQOL of children and adolescents (n = 5) and adults only (n = 5), or integrated age populations (n = 11), were identified (median sample size 24, loss to follow-up 43%, response rate 84%). Overall HRQOL was reduced in BEEC patients compared to healthy references in 4/4 studies. Impaired physical or general health in BEEC patients has been described in 9 articles, diminished mental health in 11, restricted social health in 10, and sexual health/functioning or body perception impairments in 13 articles. Urinary incontinence was the most common factor related to worse HRQOL (12 studies). In six studies, HRQOL was better than healthy norms. In eligible studies (n = 5), the pooled estimate of the effect of BEEC indicated worse HRQOL for children and adults (0 > effect sizes < 0.5). Thirty-six HRQOL assessments were used, none developed and validated for BEEC.

CONCLUSIONS

HRQOL in BEEC patients may be negatively impacted, particularly considering mental and social HRQOL. Sexual health/functioning or body perception impairments may be present in adolescents and adults. However, HRQOL is heterogeneously assessed and subsequent findings are differently reported. Additional research is warranted and can be improved.

Exploring the item sets of the Recovering Quality of Life (ReQoL) measures using factor analysis

Purpose

This paper presents two studies exploring the latent structure of item sets used in the development of the Recovering Quality of Life mental health outcome measures: ReQoL-10 and ReQoL-20.

Method

In study 1, 2262 participants completed an initial set of 61 items. In study 2, 4266 participants completed a reduced set of 40 items. Study 2 evaluated two formats of the questionnaires: one version where the items were intermingled and one where the positively worded and negatively worded items were presented as two separate blocks. Exploratory and confirmatory factor analyses were conducted on both datasets where models were specified using ordinal treatment of the item responses. Dimensionality based on the conceptual framework and methods effects reflecting the mixture of positively worded and negatively worded items were explored. Factor invariance was tested across the intermingled and block formats.

Results

In both studies, a bi-factor model (study 1: RMSEA = 0.061; CFI = 0.954; study 2: RMSEA = 0.066; CFI = 0.971) with one general factor and two local factors (positively worded questions and negatively worded questions) was preferred. The loadings on the general factor were higher than on the two local factors suggesting that the ReQoL scale scores can be understood in terms of a general factor. Insignificant differences were found between the intermingled and block formats.

Conclusions

The analyses confirmed that the ReQoL item sets are sufficiently unidimensional to proceed to item response theory analysis. The model was robust across different ordering of positive and negative items.

Electronic supplementary material

The online version of this article (10.1007/s11136-018-2091-1) contains supplementary material, which is available to authorized users.

Integrating Qualitative and Quantitative Data in the Development of Outcome Measures: The Case of the Recovering Quality of Life (ReQoL) Measures in Mental Health Populations

While it is important to treat symptoms, there is growing recognition that in order to help people with mental health problems lead meaningful and fulfilling lives, it is crucial to capture the impact of their conditions on wider aspects of their social lives. We constructed two versions of the Recovering Quality of Life (ReQoL) measure&mdash;ReQoL-10 and ReQoL-20&mdash;for use in routine settings and clinical trials from a larger pool of items by combining qualitative and quantitative evidence covering six domains. Qualitative evidence was gathered through interviews and focus groups with over 76 service users, clinicians, and a translatability assessment. Psychometric evidence generated from data from over 6200 service users was obtained from confirmatory factor models and item response theory analyses. In this paper we present an approach based on a traffic light pictorial format that was developed to present qualitative and quantitative evidence to a group of service users, clinicians, and researchers to help to make the final selection. This work provides a pragmatic yet rigorous approach to combining qualitative and quantitative evidence to ensure that ReQoL is psychometrically robust and has high relevance to service users and clinicians. This approach can be extended to the development of patient reported outcome measures in general.