Assessment of response shift using two structural equation modeling techniques

Power analyses for measurement model misspecification and response shift detection with structural equation modeling

PURPOSE

Statistical power for response shift detection with structural equation modeling (SEM) is currently underreported. The present paper addresses this issue by providing worked-out examples and syntaxes of power calculations relevant for the statistical tests associated with the SEM approach for response shift detection.

METHODS

Power calculations and related sample-size requirements are illustrated for two modelling goals: (1) to detect misspecification in the measurement model, and (2) to detect response shift. Power analyses for hypotheses regarding (exact) overall model fit and the presence of response shift are demonstrated in a step-by-step manner. The freely available and user-friendly R-package lavaan and shiny-app 'power4SEM' are used for the calculations.

RESULTS

Using the SF-36 as an example, we illustrate the specification of null-hypothesis (H0) and alternative hypothesis (H1) models to calculate chi-square based power for the test on overall model fit, the omnibus test on response shift, and the specific test on response shift. For example, we show that a sample size of 506 is needed to reject an incorrectly specified measurement model, when the actual model has two-medium sized cross loadings. We also illustrate power calculation based on the RMSEA index for approximate fit, where H0 and H1 are defined in terms of RMSEA-values.

CONCLUSION

By providing accessible resources to perform power analyses and emphasizing the different power analyses associated with different modeling goals, we hope to facilitate the uptake of power analyses for response shift detection with SEM and thereby enhance the stringency of response shift research.

Response shift in coronary artery disease

PURPOSE

Patients with coronary artery disease (CAD) experience significant angina symptoms and lifestyle changes. Revascularization procedures can result in better patient-reported outcomes (PROs) than optimal medical therapy (OMT) alone. This study evaluates the impact of response shift (RS) on changes in PROs of patients with CAD across treatment strategies.

METHODS

Data were from patients with CAD in the Alberta Provincial Project on Outcome Assessment in Coronary Heart Disease (APPROACH) registry who completed the 16-item Canadian version of the Seattle Angina Questionnaire at 2 weeks and 1 year following a coronary angiogram. Multi-group confirmatory factor analysis (MG-CFA) was used to assess measurement invariance across treatment groups at week 2. Longitudinal MG-CFA was used to test for RS according to receipt of coronary artery bypass grafting (CABG), percutaneous coronary intervention (PCI), or optimal medical therapy (OMT) alone.

RESULTS

Of the 3116 patients included in the analysis, 443 (14.2%) received CABG, 2049(65.8%) PCI, and the remainder OMT alone. The MG-CFA revealed a partial-strong invariance across the treatment groups at 2 weeks (CFI = 0.98, RMSEA [90% CI] = 0.05 [0.03, 0.06]). Recalibration RS was detected on the Angina Symptoms and Burden subscale and its magnitude in the OMT, PCI, and CABG groups were 0.32, 0.28, and 0.53, respectively. After adjusting for RS effects, the estimated target changes were largest in the CABG group and negligible in the OMT group.

CONCLUSION

Adjusting for RS is recommended in studies that use SAQ-CAN to assess changes in patients with CAD who have received revascularization versus OMT alone.

Detection of response shift in health-related quality of life studies: a systematic review

Background

Response Shift (RS) refers to the idea that an individual may undergo changes in its health-related quality of life (HRQOL). If internal standard, values, or reconceptualization of HRQOL change over time, then answer to the same items by the same individuals may not be comparable over time. Traditional measures to evaluate RS is prone to bias and strong methodologies to study the existence of this phenomenon is required. The objective is to systematically identify, analyze, and synthesize the existing and recent evidence of statistical methods used for RS detection in HRQOL studies.

Methods

The analysis of selected studies between January 2010 and July 2020 was performed through a systematic review in MEDLINE/PubMed, Scopus, Web of Science, PsycINFO and Google Scholar databases. The search strategy used the terms “Health-Related Quality of Life” and “Response Shift” using the filters “Humans”, “Journal Article”, “English” and “2010/01/01–2020/07/31”. The search was made in August 2020.

Results

After considering the inclusion and exclusion criteria, from the total selected articles (675), 107 (15.9%) of the publications were included in the analysis. From these, 79 (71.0%) detected the existence of RS and 86 (80.4%) only used one detection method. The most used methods were Then Test (n = 41) and Oort’s Structural Equation Models (SEM) (n = 35). Other method used were Multiple Lineal Regression (n = 7), Mixed-Effect Regression (n = 6), Latent Trajectory Analysis (n = 6), Item Response Theory (n = 6), Logistics Regression (n = 5), Regression and Classification Trees (n = 4) and Relative Importance Method (n = 4). Most of these detected recalibration, including Then Test (n = 27), followed by Oort’s SEM that detected the higher combination of RS types: recalibration (n = 24), reprioritization (n = 13) and reconceptualization (n = 7).

Conclusions

There is a continuous interest of studying RS detection. Oort’s SEM becomes the most versatile method in its capability for detecting RS in all different types. Despite results from previous systematic reviews, same methods have been used during the last years. We observed the need to explore other alternative methods allowing same detection capacity with robust and highly precise methodology. The investigation on RS detection and types requires more study, therefore new opportunity grows to continue attending this phenomenon through a multidisciplinary perspective.

Response shift after coronary revascularization

PURPOSE

The aims of this study were to investigate (1) the extent to which response shift occurs among patients with coronary artery disease (CAD) after coronary revascularization, (2) whether the assessment of changes in health-related quality of life (HRQoL), controlled for response shift, yield more valid estimates of changes in HRQoL, as indicated by stronger associations with criterion measures of change, than without controlling for response shift, and (3) if occurrences of response shift are related to patient characteristics.

METHODS

Patients with CAD completed the SF-36 and the Seattle Angina Questionnaire (SAQ7) at baseline and 3 months after coronary revascularization. Sociodemographic, clinical and psychosocial variables were measured with the patient version of the New York Heart Association-class, Subjective Significance Questionnaire, Reconstruction of Life Events Questionnaire (RE-LIFE), and HEXACO personality inventory. Oort's Structural Equation Modeling (SEM) approach was used to investigate response shift.

RESULTS

191 patient completed questionnaires at baseline and at 3 months after treatment. The SF-36 showed recalibration and reprioritization response shift and the SAQ7 reconceptualization response shift. Controlling for these response shift effects did not result in more valid estimates of change. One significant association was found between reprioritization response shift and complete integration of having CAD into their life story, as indicated by the RE-LIFE.

CONCLUSION

Results indicate response shift in HRQoL following coronary revascularization. While we did not find an impact of response shift on the estimates of change, the SEM approach provides a more comprehensive insight into the different types of change in HRQoL following coronary revascularization.

Detecting response shift in health-related quality of life measurement among patients with hypertension using structural equation modeling

BACKGROUND

Outcomes derived from longitudinal self-reported health-related quality of life measures can be confounded by response shift. This study was aimed to detect response shift among patients with hypertension attending a community-based disease management program.

METHODS

240 consecutive consulting or follow-up patients with diagnosed hypertension were recruited. The Short Form 36-item Health Survey was self-administered at 12 community health service stations at baseline and four weeks after attending the program. The 4-step structural equation modeling approach assessed response shift.

RESULTS

Data from 203 (84.6%) patients were eligible for analyses (mean age 65.9 ± 10.8 years, 46.3% female). The results showed uniform recalibration of social functioning ([Formula: see text](1) = 22.98, P < 0.001), and non-uniform recalibration of role limitations due to physical problems ([Formula: see text](1) = 8.84, P = 0.003), and bodily pain ([Formula: see text](1) = 17.41, P < 0.001). The effects of response shift on social functioning were calculated as "small" (effect-size = 0.35), but changed the observed changes from improvement (effect-size = 0.25) to slight deterioration (effect-size = -0.10). After accounting for the response shift effect, the general physical health of participants was improved (effect-size = 0.37), while deterioration (effect-size = -0.21) in the general mental health was also found.

CONCLUSIONS

Recalibration existed among patients with hypertension attending the disease management program. The interventions in the program might act as a catalyst that induced the response shift. We conclude that response shift should be considered in hypertension research with longitudinal health-related quality of life data.

Consequences of ignoring the response-shift and measure non-invariant items in sleep studies: an empirical data based simulation of the treatment effect of CBT-I on dysfunctional sleep beliefs

INTRODUCTION

Previous studies have shown that psychological interventions do not only improve patients' mental symptoms (i.e., true change) but may also change the internal standards patients use to evaluate their symptoms (i.e., response shifts). Although the response shifts could reflect patients' cognitive changes toward their disorders as the interventions aim to achieve, failing to differentiate them from the true change during data analyses could bias the research conclusions. Considering this issue is seldom discussed in sleep studies, this study thus examined the impacts of response-shift items in an intervention study of cognitive behavioral therapy for insomnia (CBT-I) via empirical-data based simulations.

METHOD

We used longitudinal measurement invariance tests to identify the items in an abbreviated version of the Dysfunctional Beliefs and Attitudes about Sleep Scale that are non-invariant (response shifted) against CBT-I based on data from 114 insomnia patients. The partial invariance model built accordingly was then used as a population model for simulations to examine the impacts of the response-shift items on follow-up paired t-tests.

RESULTS

Invariance tests indicate CBT-I would lift the intercept of one item in DBAS-10 and cause non-uniform calibrations in three items. The following up simulations showed that failing to exclude the intercept-lifted item from the calculations of the subscale scores would lower the probability of using paired t-test to correctly detect the treatment effect by up to 53%.

CONCLUSIONS

We recommend sleep researchers to consider the issues of response-shift when assessing sleep-related constructs in interventional studies for insomnia.

How peoples’ ratings of dental implant treatment change over time?

Objectives

Dental implant treatment (DIT) improves peoples’ oral health-related quality of life (OHQoL). Assessment of longitudinal changes in OHRQoL may be undermined by response shift (RS). RS is the process by which quality of life changes, independent of health status as a result recalibration, reprioritization or reconceptualization. Thus, this study aimed to describe RS in the OHRQoL and perceived oral health of individuals receiving DIT and to compare the then-test, a self-anchored scale and the classification and regression trees (CRT) approaches for assessing RS.

Methods

OHRQoL was assessed in 100 patients receiving DIT using the OHIP-Edent (n = 100) and a self-anchored scale (n = 45) before placement of the final restoration and 3 to 6 months after treatment was completed. The OHIP-Edent was also used as a retrospective assessment at follow-up. CRT examined changes in the OHIP-Edent total score as a dependent variable with global changes in oral health and each OHIP-Edent subscale score as independent variables.

Results

OHRQoL and perceived oral health improved after treatment. The OHIP-Edent score decreased from 36.4 at baseline to 12.7 after treatment. On average, participants recalibrated their internal standard downwards (− 4.0 OHIP-Edent points). CRT detected downwards recalibration in 5% of participants and upwards in 15%. Reprioritization was observed in the social disability and psychological discomfort aspects of OHRQoL.

Conclusions

RS affects longitudinal assessments of OHRQoL in DIT, reducing the apparent magnitude of change. The then-test and CRT are valid and complementary methods to assess RS.

Response shift effects of quality of life assessments in breast cancer survivors

Measurements of quality of life (QoL) can be distorted by respondents adapting to new situations between measurement points and consequently having a changed frame of reference. To investigate this bias in breast cancer survivors, we compare their QoL with that of the general population and use two complementary methods for detecting this bias. Breast cancer survivors (n = 308, response rate: 91%) were tested with the QoL questionnaire European Organization for Research and Treatment of Cancer Quality of Life Questionnaire Core 30. Response shift was examined with the thentest (retrospective judgements) and with the structural equation modelling (SEM) approach. Compared with the general population, breast cancer survivors showed impaired QoL in all functioning scales (Hedges' g: -0.56 to -0.93) and symptom scales (Hedges' g: 0.28-0.74).The thentest method indicated recalibration effects in several dimensions including social functioning. The SEM method detected a non-uniform recalibration effect for social functioning from pretest to posttest and from pretest to thentest, but no effect between thentest and posttest. Breast cancer survivors' QoL is clearly diminished. Comparing the two approaches for detecting response shift showed that it is also useful to apply SEM to retrospective judgements and that this can reveal response shift effects that would otherwise be overlooked.

Using classification and regression tree modelling to investigate response shift patterns in dentine hypersensitivity

Background

Dentine hypersensitivity (DH) affects people’s quality of life (QoL). However changes in the internal meaning of QoL, known as Response shift (RS) may undermine longitudinal assessment of QoL. This study aimed to describe patterns of RS in people with DH using Classification and Regression Trees (CRT) and to explore the convergent validity of CRT with the then-test and ideals approaches.

Methods

Data from an 8-week clinical trial of mouthwashes for dentine hypersensitivity (n = 75) using the Dentine Hypersensitivity Experience Questionnaire (DHEQ) as the outcome measure, were analysed. CRT was used to examine 8-week changes in DHEQ total score as a dependent variable with clinical status for DH and each DHEQ subscale score (restrictions, coping, social, emotional and identity) as independent variables. Recalibration was inferred when the clinical change was not consistent with the DHEQ change score using a minimally important difference for DHEQ of 22 points. Reprioritization was inferred by changes in the relative importance of each subscale to the model over time.

Results

Overall, 50.7% of participants experienced a clinical improvement in their DH after treatment and 22.7% experienced an important improvement in their quality of life. Thirty-six per cent shifted their internal standards downward and 14.7% upwards, suggesting recalibration. Reprioritization occurred over time among the social and emotional impacts of DH.

Conclusions

CRT was a useful method to reveal both, the types and nature of RS in people with a mild health condition and demonstrated convergent validity with design based approaches to detect RS.

Personality and Perceived Health in Spousal Caregivers of Patients with Lung Cancer: The Roles of Neuroticism and Extraversion

Purpose. Family members' responsibilities for patients with cancer have increased dramatically over the past decade and will likely continue to rise. Given that caregiving is associated with declines in self-care, there is a need for research on caregivers' perceptions of their own health. The purpose of this study was to examine whether personality is associated with four self-report perceived health items from the SF-36. Methods. The sample consisted of 114 spouses of lung cancer patients who completed cross-sectional measures as part of a larger cohort study on adjustment to the diagnosis and treatment of lung cancer. Predictors of interest were Neuroticism and Extraversion scores from the NEO-FFI. Covariates were age, gender, conscientiousness, depressive symptoms, and objective illness burden. Results. Multivariate analyses revealed that caregivers with higher Extraversion scores were less likely to respond affirmatively to the item "I expect my health to get worse" (OR = 0.90, p < 0.05). Neuroticism was associated with poorer perceived health (ORs from 1.11 to 1.12, p's < 0.05). Conclusions. The present cross-sectional findings suggest that personality is associated with responses to SF-36 perceived health items beyond what can be accounted for by objective illness burden and other covariates. The potential overestimation of health among extraverted caregivers may have implications for their health outcomes.

An item-level response shift study on the change of health state with the rating of asthma-specific quality of life: a report from the PROMIS(®) Pediatric Asthma Study

PURPOSE

To examine item-level response shift associated with the change in asthma-related health state (i.e., change in asthma control status and global rating of change (GRC) in breathing problems).

METHODS

Study sample comprised 238 asthmatic children who were between 8 and 17.9 years and completed the Pediatric Asthma Quality of Life Questionnaire (PAQLQ) symptoms, emotion function, and activity limitation domains at baseline and a follow-up assessment. Structural equation modeling was implemented to assess item-level response shift associated with the change in asthma-related health state with the adjustment for the influence of confounding variables. The magnitude of item-level response shift and its influence on the change of domain scores was estimated using Cohen's effect sizes.

RESULTS

We found no instances of item-level response shift. However, two items were identified with measurement bias related to GRC due to breathing problems. Specifically, asthmatic children with better/about the same GRC due to breathing problems reported lower scores for one item in the emotional domain at follow-up compared to those with deteriorated GRC due to breathing problems. In addition, asthmatic children with better/about the same GRC due to breathing problems reported better scores for another item in the symptom domain at baseline compared to those with deteriorated GRC due to breathing problems. The impact of measurement bias was small and did not bias the change of domain scores over time.

CONCLUSIONS

No item-level response shift, but two instances of measurement bias, appears in asthmatic children. However, the impact of these measurement issues is negligible.

Associação entre a qualidade de vida e adesão à medicação de indivíduos hipertensos

Objetivo Avaliar a associação entre a qualidade de vida e a adesão a medicação de indivíduos hipertensos. Métodos Estudo transversal, realizado com 432 hipertensos cadastrados em sistema informatizado público federal. Os dados foram coletados no domicilio por entrevista estruturada com questões relacionadas a variáveis socioeconômicas, clínicas, avaliação da adesão ao tratamento e o WHOQOL-BREF para a qualidade de vida. Utilizou-se o teste de H de Kruskal-Wallis para medir a associação entre as escalas da qualidade de vida e a classificação da adesão ao tratamento. Resultados Os escores mais baixos estavam presente no domínio autoavaliação e os mais altos foram encontrados no social. Os indivíduos que possuíam adesão extrema ao tratamento anti-hipertensivo apresentaram escores mais altos na avaliação da qualidade de vida em comparação com indivíduos classificados como não adesão extrema ao tratamento anti-hipertensivo. Conclusão A associação entre a qualidade de vida e adesão a medicação em indivíduos hipertensos não foi preditiva, sendo que os melhores escores estavam presentes nos indivíduos hipertensos que apresentaram alta adesão a medicação e os piores escores da qualidade de vida se apresentaram nos indivíduos de não adesão extrema e limítrofe a não adesão total.

Overall performance of Oort's procedure for response shift detection at item level: a pilot simulation study

OBJECTIVE

This simulation study was designed to provide data on the performance of Oort's procedure (OP) for response shift (RS) detection (regarding type I error, power, and overall performance), according to sample characteristics, at item level. A specific objective was to assess the impact of using different information criteria (IC), as alternatives to the LRT (likelihood-ratio test), for global assessment of RS occurrence.

METHODS

Responses to five binary items at two times of measurement were simulated. Thirty-six combinations of sample characteristics [sample size (n), "true change," correlations between the two latent variables and presence/absence of uniform recalibration RS (ur)] were considered. A thousand datasets were generated for each combination. RS detection was performed on each dataset following OP. Type I error and power of the global assessment of RS occurrence, as well as overall performance of the OP, were assessed.

RESULTS

The estimated type I error was close to 5 % for the LRT and lower than 5 % for the IC. The estimated power was higher for the LRT as compared to the AIC, which was the highest among the other IC. For the LRT, the estimated power for n = 100 and for the combination of n = 200 and ur = 1 item was below 80 %. Otherwise, for other combinations of sample characteristics, the estimated power was above 90 %.

CONCLUSION

For the LRT, higher values of power were estimated compared to IC with appropriate values of type I error. These results were consistent with Oort's proposal to use the LRT as the criterion to assess global RS occurrence.

Evaluating response shift in training evaluation: comparing the retrospective pretest with an adapted measurement invariance approach in a classroom management training program

BACKGROUND

Response shift (RS) can threaten the internal validity of pre-post designs. As RS may indicate a redefinition of the target construct, its occurrence in training evaluation is rather likely. The most common approach to deal with RS is to implement a retrospective pretest (then-test) instead of the traditional pre-test. In health psychology, an adapted measurement invariance approach (MIad) was developed as an alternative technique to study RS. Results produced by identifying RS with the two approaches were rarely studied simultaneously or within an experimental framework.

OBJECTIVES

To study RS in two different treatment conditions and compare results produced by both techniques in identifying various types of RS. We further studied validity aspects of the then-test.

RESEARCH DESIGN

We evaluated RS by applying the then-test procedure (TP) and the measurement invariance apporach MIad within an experimental design: Participants either attended a short-term or a long-term classroom management training program.

SUBJECTS

Participants were 146 student teachers in their first year of master's study.

MEASURES

Pre (before training), post, and then self-ratings (after training) on classroom management knowledge were administered.

RESULTS

Results indicated that the two approaches do not yield the same results. The MIad identified more and also group-specific RS as opposed to the findings of the TP, which found less and only little evidence for group-specific RS.

CONCLUSIONS

Further research is needed to study the usability and validity of the respective approaches. In particular, the usability of the then-test seems to be challenged.

Influence of explanatory and confounding variables on HRQoL after controlling for measurement bias and response shift in measurement

The purpose of this study was to examine the influence of explanatory and confounding variables on health-related quality of life after accounting for response shift, measurement bias and response shift in measurement using structural equation modeling. Hypertensive patients with coronary artery disease randomized to anti-hypertensive treatment, completed the ShortForm-36 questionnaire at both baseline and 1 year (n = 788). Three measurement biases were found and all three were considered as response shift in measurement. Older patients reported worse scores for both physical functioning (PF) and role-physical at baseline and 1 year later compared to younger patients; and males reported better PF than females after conditioning on the latent trait of general physical health. Before controlling for response shift, patients' PF scores were not statistically different over time; however, PF scores significantly improved (p < 0.01) after controlling for recalibration response shift. Assessment of how patients perceive their change in health-related quality of life over time is warranted.

Item response theory and factor analysis as a mean to characterize occurrence of response shift in a longitudinal quality of life study in breast cancer patients

BACKGROUND

The occurrence of response shift (RS) in longitudinal health-related quality of life (HRQoL) studies, reflecting patient adaptation to disease, has already been demonstrated. Several methods have been developed to detect the three different types of response shift (RS), i.e. recalibration RS, 2) reprioritization RS, and 3) reconceptualization RS. We investigated two complementary methods that characterize the occurrence of RS: factor analysis, comprising Principal Component Analysis (PCA) and Multiple Correspondence Analysis (MCA), and a method of Item Response Theory (IRT).

METHODS

Breast cancer patients (n = 381) completed the EORTC QLQ-C30 and EORTC QLQ-BR23 questionnaires at baseline, immediately following surgery, and three and six months after surgery, according to the "then-test/post-test" design. Recalibration was explored using MCA and a model of IRT, called the Linear Logistic Model with Relaxed Assumptions (LLRA) using the then-test method. Principal Component Analysis (PCA) was used to explore reconceptualization and reprioritization.

RESULTS

MCA highlighted the main profiles of recalibration: patients with high HRQoL level report a slightly worse HRQoL level retrospectively and vice versa. The LLRA model indicated a downward or upward recalibration for each dimension. At six months, the recalibration effect was statistically significant for 11/22 dimensions of the QLQ-C30 and BR23 according to the LLRA model (p ≤ 0.001). Regarding the QLQ-C30, PCA indicated a reprioritization of symptom scales and reconceptualization via an increased correlation between functional scales.

CONCLUSIONS

Our findings demonstrate the usefulness of these analyses in characterizing the occurrence of RS. MCA and IRT model had convergent results with then-test method to characterize recalibration component of RS. PCA is an indirect method in investigating the reprioritization and reconceptualization components of RS.

Symptom distress predicts long-term health and well-being in allogeneic stem cell transplantation survivors

The number of survivors after allogeneic hematopoietic stem cell transplantation (HSCT) continues to increase, yet their survivorship experience has not been fully characterized. This study examines the health status and health-related quality of life (HRQL) of HSCT survivors. The aims of the study were to: (1) explore the baseline and change over time in these health outcomes, and (2) characterize subgroups experiencing adverse outcomes. In this longitudinal study, adults who survived >3 years from date of allogeneic HSCT completed a series of patient-reported outcome measures annually, including measures of health status, HRQL, and symptoms. Data were analyzed using hierarchical linear modeling. Subjects (N = 171) were on average 44 (±13.5) years of age and primarily male (62.6%); 40% were Hispanic. Mean scores for physical and mental health and HRQL were preserved relative to population norms. Hierarchical linear modeling revealed no significant change in the mean trajectories of these outcomes, although significant between-individual variability was observed. When controlling for demographic and clinical factors, physical symptom distress negatively affected all outcomes. The impact of symptom distress on physical health varied based on time since HSCT; impairment in physical health was greatest in survivors experiencing high symptom distress and who were within the first decade post transplantation. Extended treatment with systemic immunosuppressive therapy also predicted inferior physical health. These findings suggest that patient-centered outcomes are preserved relative to normative values and are generally stable after allogeneic HSCT, although survivors with persistent symptoms and those receiving systemic immunosuppression experience impairments in health status and HRQL.