Critical examination of current response shift methods and proposal for advancing new methods

Power analyses for measurement model misspecification and response shift detection with structural equation modeling

PURPOSE

Statistical power for response shift detection with structural equation modeling (SEM) is currently underreported. The present paper addresses this issue by providing worked-out examples and syntaxes of power calculations relevant for the statistical tests associated with the SEM approach for response shift detection.

METHODS

Power calculations and related sample-size requirements are illustrated for two modelling goals: (1) to detect misspecification in the measurement model, and (2) to detect response shift. Power analyses for hypotheses regarding (exact) overall model fit and the presence of response shift are demonstrated in a step-by-step manner. The freely available and user-friendly R-package lavaan and shiny-app 'power4SEM' are used for the calculations.

RESULTS

Using the SF-36 as an example, we illustrate the specification of null-hypothesis (H0) and alternative hypothesis (H1) models to calculate chi-square based power for the test on overall model fit, the omnibus test on response shift, and the specific test on response shift. For example, we show that a sample size of 506 is needed to reject an incorrectly specified measurement model, when the actual model has two-medium sized cross loadings. We also illustrate power calculation based on the RMSEA index for approximate fit, where H0 and H1 are defined in terms of RMSEA-values.

CONCLUSION

By providing accessible resources to perform power analyses and emphasizing the different power analyses associated with different modeling goals, we hope to facilitate the uptake of power analyses for response shift detection with SEM and thereby enhance the stringency of response shift research.

Response shift in coronary artery disease

PURPOSE

Patients with coronary artery disease (CAD) experience significant angina symptoms and lifestyle changes. Revascularization procedures can result in better patient-reported outcomes (PROs) than optimal medical therapy (OMT) alone. This study evaluates the impact of response shift (RS) on changes in PROs of patients with CAD across treatment strategies.

METHODS

Data were from patients with CAD in the Alberta Provincial Project on Outcome Assessment in Coronary Heart Disease (APPROACH) registry who completed the 16-item Canadian version of the Seattle Angina Questionnaire at 2 weeks and 1 year following a coronary angiogram. Multi-group confirmatory factor analysis (MG-CFA) was used to assess measurement invariance across treatment groups at week 2. Longitudinal MG-CFA was used to test for RS according to receipt of coronary artery bypass grafting (CABG), percutaneous coronary intervention (PCI), or optimal medical therapy (OMT) alone.

RESULTS

Of the 3116 patients included in the analysis, 443 (14.2%) received CABG, 2049(65.8%) PCI, and the remainder OMT alone. The MG-CFA revealed a partial-strong invariance across the treatment groups at 2 weeks (CFI = 0.98, RMSEA [90% CI] = 0.05 [0.03, 0.06]). Recalibration RS was detected on the Angina Symptoms and Burden subscale and its magnitude in the OMT, PCI, and CABG groups were 0.32, 0.28, and 0.53, respectively. After adjusting for RS effects, the estimated target changes were largest in the CABG group and negligible in the OMT group.

CONCLUSION

Adjusting for RS is recommended in studies that use SAQ-CAN to assess changes in patients with CAD who have received revascularization versus OMT alone.

Response shift results of quantitative research using patient-reported outcome measures: a descriptive systematic review

PURPOSE

The objective of this systematic review was to describe the prevalence and magnitude of response shift effects, for different response shift methods, populations, study designs, and patient-reported outcome measures (PROM)s.

METHODS

A literature search was performed in MEDLINE, PSYCINFO, CINAHL, EMBASE, Social Science Citation Index, and Dissertations & Theses Global to identify longitudinal quantitative studies that examined response shift using PROMs, published before 2021. The magnitude of each response shift effect (effect sizes, R-squared or percentage of respondents with response shift) was ascertained based on reported statistical information or as stated in the manuscript. Prevalence and magnitudes of response shift effects were summarized at two levels of analysis (study and effect levels), for recalibration and reprioritization/reconceptualization separately, and for different response shift methods, and population, study design, and PROM characteristics. Analyses were conducted twice: (a) including all studies and samples, and (b) including only unrelated studies and independent samples.

RESULTS

Of the 150 included studies, 130 (86.7%) detected response shift effects. Of the 4868 effects investigated, 793 (16.3%) revealed response shift. Effect sizes could be determined for 105 (70.0%) of the studies for a total of 1130 effects, of which 537 (47.5%) resulted in detection of response shift. Whereas effect sizes varied widely, most median recalibration effect sizes (Cohen's d) were between 0.20 and 0.30 and median reprioritization/reconceptualization effect sizes rarely exceeded 0.15, across the characteristics. Similar results were obtained from unrelated studies.

CONCLUSION

The results draw attention to the need to focus on understanding variability in response shift results: Who experience response shifts, to what extent, and under which circumstances?

Health change awareness and its association with weight loss following bariatric surgery

OBJECTIVE

Patients' ability to judge health change over time has important clinical implications for treatment, but is understudied in longitudinal contexts with meaningful health change. We assess patients' awareness of health change for 5 years following bariatric surgery, and its association with weight loss.

METHOD

Participants were part of the Longitudinal Assessment of Bariatric Surgery (N = 2,027). Perceived health change for each year was assessed by comparing it to self-reports of health on the SF-36 health survey. Participants were categorized as concordant when perceived and actual self-reported health change corresponded, and as discordant when they did not correspond.

RESULTS

Year-to-year concordance between perceived and actual self-reported health change occurred less than 50% of the time. Discordance between perceived and actual health was associated with weight loss following surgery. Discordant-positive participants who perceived their health change as more positive than was warranted lost more weight post-surgery and thus had lower body mass index scores than concordant participants. Conversely, discordant-negative participants who perceived their health as worse than what was warranted lost less weight post-surgery and thus had higher body mass index scores.

CONCLUSIONS

These results suggest that recollection of past health is generally poor and can be biased by salient factors during recall. Clinicians are advised to use caution when retrospective judgments of health are utilized. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Response shift in hearing related quality of life after cochlear implantation - effect size and clinical significance: a then-test study

BACKGROUND

Quality of life questionnaires are often used in the assessment of rehabilitation of hearing-impaired patients with a cochlear implant. However, a prospective study with a systematic retrospective evaluation of the preoperative quality of life after surgery has not yet been conducted and may reveal a change in internal standards, such as a response shift, due to the implantation and hearing rehabilitation.

METHODS

The Nijmegen Cochlear Implant Questionnaire (NCIQ) was used for assessing hearing related quality of life. It has three general domains (physical, psychological and social) and six subdomains. Seventeen patients were tested before (t₀) and retrospectively (then-test; pre-t₁) and acutely postoperative (post-t₁) after cochlear implantation. Observed changes, then-test changes, response shifts and effect sizes were calculated. Non-parametric statistical methods were used.

RESULTS

The NCIQ total score was 52.32 ± 18.69 (mean, standard deviation) for t₀, 59.29 ± 14.06 for pre-t₁ and 67.65 ± 26.02 for post-t₁ questioning. The observed change was statistically significant in all domains but in speech production. Response shift was statistically significant in the total score and in part of the domains. The effect sizes for the response shift were moderate (> 0.5) in the total score, psychological, social general scores and subdomains.

CONCLUSIONS

In this study we found that response shift does exist in adults with severe to profound hearing loss undergoing cochlear implantation. By advising the participants to deactivate the implant for the then-test, recall bias and noise were minimized. The clinical significance of the response shift was present in the total score and in the social and psychological domains.

TRIAL REGISTRATION

This study was retrospectively registered with the German Clinical Trial Register, TRN DRKS00029467, on 07/08/2022.

Impact of response shift effects in the assessment of self-reported depression during treatment: Insights from a rTMS versus Venlafaxine randomized controlled trial

PURPOSE

Patient-Reported Outcomes are essential to properly assess treatment effectiveness in randomized clinical trial (RCT) for Major Depressive Disorder (MDD). MDD self-assessment may vary over time depending on change in the meaning of patients' self-evaluation of depression, i.e. Response Shift (RS). Our aim was to investigate RS and its impact on different depression domains in a clinical trial comparing rTMS versus Venlafaxine.

METHODS

The occurrence and type of RS was determined using Structural Equation Modeling applied to change over time in 3 domains (Sad Mood, Performance Impairment, Negative Self-Reference) of the short-form Beck Depression Inventory (BDI-13) in a secondary analysis of a RCT on 170 patients with MDD treated by rTMS, venlafaxine or both.

RESULTS

RS was evidenced in the venlafaxine group in the Negative Self-Reference and Sad Mood domains.

CONCLUSION

RS effects differed between treatment arms in self-reported depression domains in patients with MDD. Ignoring RS would have led to a slight underestimation of depression improvement, depending on treatment group. Further investigations of RS and advancing new methods are needed to better inform decision making based on Patient-Reported Outcomes.

An experience- and preference-based EQ-5D-3L value set derived using 18 months of longitudinal data in patients who sustained a fracture: results from the ICUROS

INTRODUCTION

EQ-5D-3L preference-based value sets are predominately based on hypothetical health states and derived in cross-sectional settings. Therefore, we derived an experience-based value set from a prospective observational study.

METHODS

The International Costs and Utilities Related to Osteoporotic fractures Study (ICUROS) was a multinational study on fragility fractures, prospectively collecting EQ-5D-3L and Time trade-off (TTO) within two weeks after fracture (including pre-fracture recall), and at 4, 12, and 18 months thereafter. We derived an EQ-5D-3L value set by regressing the TTO values on the ten impairment levels in the EQ-5D-3L. We explored the potential for response shift and whether preferences for domains vary systematically with prior impairment in that domain. Finally, we compared the value set to 25 other EQ-5D-3L preference-based value sets.

RESULTS

TTO data were available for 12,954 EQ-5D-3L health states in 4683 patients. All coefficients in the value set had the expected sign, were statistically significant, and increased monotonically with severity of impairment. We found evidence for response shift in mobility, self-care, and usual activities. The value set had good agreement with the only other experience- and preference-based value set, but poor agreement with all hypothetical value sets.

CONCLUSIONS

We present an experience- and preference-based value set with high face validity. The study indicates that response shift may be important to account for when deriving value sets. Furthermore, the study suggests that perspective (experienced versus hypothetical) is more important than country setting or demographics for valuation of EQ-5D-3L health states.

Implications of the syntheses on definition, theory, and methods conducted by the Response Shift - in Sync Working Group

PURPOSE

Our aim is to advance response shift research by explicating the implications of published syntheses by the Response Shift - in Sync Working Group in an integrative way and suggesting ways for improving the quality of future response shift studies.

METHODS

Members of the Working Group further discussed the syntheses of the literature on definitions, theoretical underpinnings, operationalizations, and response shift methods. They outlined areas in need of further explication and refinement, and delineated additional implications for future research.

RESULTS

First, the proposed response shift definition was further specified and its implications for the interpretation of results explicated in relation to former, published definitions. Second, the proposed theoretical model was further explained in relation to previous theoretical models and its implications for formulating research objectives highlighted. Third, ways to explore alternative explanations per response shift method and their implications for response shift detection and explanation were delineated. The implications of the diversity of the response shift methods for response shift research were presented. Fourth, the implications of the need to enhance the quality and reporting of the response shift studies for future research were sketched.

CONCLUSION

With our work, we intend to contribute to a common language regarding response shift definitions, theory, and methods. By elucidating some of the major implications of earlier work, we hope to advance response shift research.

Measurement invariance in the social sciences: Historical development, methodological challenges, state of the art, and future perspectives

This review summarizes the current state of the art of statistical and (survey) methodological research on measurement (non)invariance, which is considered a core challenge for the comparative social sciences. After outlining the historical roots, conceptual details, and standard procedures for measurement invariance testing, the paper focuses in particular on the statistical developments that have been achieved in the last 10 years. These include Bayesian approximate measurement invariance, the alignment method, measurement invariance testing within the multilevel modeling framework, mixture multigroup factor analysis, the measurement invariance explorer, and the response shift-true change decomposition approach. Furthermore, the contribution of survey methodological research to the construction of invariant measurement instruments is explicitly addressed and highlighted, including the issues of design decisions, pretesting, scale adoption, and translation. The paper ends with an outlook on future research perspectives.

Social and therapeutic decline earlier than physical and psychological domains after discharge in heart failure patients: A patient-reported outcome measurements of latent transition analysis

Background

Among patients with chronic heart failure (CHF), response shifts are common in assessing treatment effects. However, few studies focused on potential response shifts in these patients.

Materials and methods

Data of CHF patient-reported outcome measures (PROMs) were obtained from three hospitals in Shanxi, China, from 2017 to 2019. A total of 497 patients were enrolled and followed up at 1 month and 6 months after discharge. Latent transition analysis (LTA) was employed to determine the longitudinal transition trajectories of latent subtypes in CHF patients in the physiological, psychological, social, and therapeutic domains.

Results

The patients were divided into high- and low-level groups in the four domains according to the LTA. One month after discharge, the physiological and psychological domains improved, while the social and therapeutic domains remained unchanged. Six months after discharge, the former remained stable, but the latter deteriorated. The factors affecting the state transition in four domains were as follows. The influencing factor of the physiological domains are gender, age, tea consumption, smoking, alcohol consumption, physical activity, and light diet; those of the psychological domain are gender, occupation, smoking, alcohol consumption, and physical activity; those of the social domains are age; those of the therapeutic domains are education and income.

Conclusion

The disease status of CHF patients has shifted over time. Risk factors accelerate the deterioration of patients’ condition. Furthermore, the risk factors of social and therapeutic domains deteriorate patients’ condition faster than those of physiological and psychological domains. Therefore, individualized intervention programs should be given for CHF patients who may be transferred to the low-level groups to maintain the treatment effect and improve the prognosis.

Response Shift After Cognitive Behavioral Therapy Targeting Severe Fatigue: Explorative Analysis of Three Randomized Controlled Trials

Background

Cognitive behavioral therapy (CBT) is an evidence-based intervention for severe fatigue. Changes in patients’ fatigue scores following CBT might reflect not only the intended relief in fatigue but also response shift, a change in the meaning of patients’ self-evaluation. Objectives were to (1) identify the occurrence of response shift in patients undergoing CBT, (2) determine the impact of response shift on the intervention effect, and (3) investigate whether changes in fatigue-related cognitions and perceptions, targeted during CBT, are associated with response shift.

Methods

Data of three randomized controlled trials testing the efficacy of CBT in individuals with chronic fatigue syndrome (CFS, n = 222), cancer (n = 123), and diabetes (n = 107) were re-analyzed. Fatigue severity was measured with 8 items from the Checklist Individual Strength, a valid and widely used self-report questionnaire. Structural equation modelling was applied to assess lack of longitudinal measurement invariance, as indication of response shift.

Results

As expected, in all three trials, response shift was indicated in the CBT groups, not the control groups. Response shift through reprioritization was indicated for the items “Physically, I feel exhausted” (CFS) and “I tire easily” (cancer, diabetes), which became less vs. more important to the measurement of fatigue, respectively. However, this did not affect the intervention effects. Some changes in cognitions and perceptions were associated with the response shifts.

Conclusions

CBT seems to induce response shift through reprioritization across patient groups, but its occurrence does not affect the intervention effect. Future research should corroborate these findings and investigate whether patients indeed change their understanding of fatigue.

Understanding Patients' Perceived Health After Critical Illness: Analysis of Two Prospective, Longitudinal Studies of ARDS Survivors

BACKGROUND

Perceived health is one of the strongest determinants of subjective well-being, but it has received little attention among survivors of ARDS.

RESEARCH QUESTION

How well do self-reported measures of physical, emotional, and social functioning predict perceived overall health (measured using the EQ-5D visual analog scale [EQ-5D-VAS]) among adult survivors of ARDS? Are demographic features, comorbidity, or severity of illness correlated with perceived health after controlling for self-reported functioning?

STUDY DESIGN AND METHODS

We analyzed the ARDSNet Long Term Outcomes Study (ALTOS) and Improving Care of Acute Lung Injury Patients (ICAP) Study, two longitudinal cohorts with a total of 823 survivors from 44 US hospitals, which prospectively assessed survivors at 6 and 12 months after ARDS. Perceived health, evaluated using the EQ-5D-VAS, was predicted using ridge regression and self-reported measures of physical, emotional, and social functioning. The difference between observed and predicted perceived health was termed perspective deviation (PD). Correlations between PD and demographics, comorbidities, and severity of illness were explored.

RESULTS

The correlation between observed and predicted EQ-5D-VAS scores ranged from 0.68 to 0.73 across the two cohorts and time points. PD ranged from -80 to +34 and was more than the minimum clinically important difference for 52% to 55% of survivors. Neither demographic features, comorbidity, nor severity of illness were correlated strongly with PD, with |r| < 0.25 for all continuous variables in both cohorts and time points. The correlation between PD at 6- and 12-month assessments was weak (ALTOS: r = 0.22, P < .001; ICAP: r = 0.20, P = .02).

INTERPRETATION

About half of survivors of ARDS showed clinically important differences in actual perceived health vs predicted perceived health based on self-reported measures of functioning. Survivors of ARDS demographic features, comorbidities, and severity of illness were correlated only weakly with perceived health after controlling for measures of perceived functioning, highlighting the challenge of predicting how individual patients will respond psychologically to new impairments after critical illness.

Performance of a Rasch-based method for group comparisons of longitudinal change and response shift at the item level in PRO data: A simulation study

The growing interest in patient perception and experience in healthcare has led to an increase in the use of patient-reported outcomes (PRO) data. However, chronically ill patients may regularly adapt to their disease and, as a consequence, might change their perception of the PRO being measured. This phenomenon named response shift (RS) may occur differently depending on clinical and individual characteristics. The RespOnse Shift ALgorithm at the Item level (ROSALI), a method for RS analysis at the item level based on Rasch models, has recently been extended to explore heterogeneity of item-level RS between two groups of patients. The performances of ROSALI in terms of RS detection at the item level and biases of estimated differences in latent variable means were assessed. A simulation study was performed to investigate four scenarios: no RS, RS in only one group, RS affecting both groups either in a similar or a different way. Performances of ROSALI were assessed using rates of false detection of RS when no RS was simulated and a set of criteria (presence of RS, correct identification of items and groups affected by RS) when RS was simulated. Rates of false detection of RS were low indicating that ROSALI satisfactorily prevents from mistakenly inferring RS. ROSALI is able to detect RS and identify the item and group(s) affected when RS affects all response categories of an item in the same way. The performances of ROSALI depend mainly on the sample size and the degree of heterogeneity of item-level RS.

Response shift in patient-reported outcomes: definition, theory, and a revised model

Purpose

The extant response shift definitions and theoretical response shift models, while helpful, also introduce predicaments and theoretical debates continue. To address these predicaments and stimulate empirical research, we propose a more specific formal definition of response shift and a revised theoretical model.

Methods

This work is an international collaborative effort and involved a critical assessment of the literature.

Results

Three main predicaments were identified. First, the formal definitions of response shift need further specification and clarification. Second, previous models were focused on explaining change in the construct intended to be measured rather than explaining the construct at multiple time points and neglected the importance of using at least two time points to investigate response shift. Third, extant models do not explicitly distinguish the measure from the construct. Here we define response shift as an effect occurring whenever observed change (e.g., change in patient-reported outcome measures (PROM) scores) is not fully explained by target change (i.e., change in the construct intended to be measured). The revised model distinguishes the measure (e.g., PROM) from the underlying target construct (e.g., quality of life) at two time points. The major plausible paths are delineated, and the underlying assumptions of this model are explicated.

Conclusion

It is our hope that this refined definition and model are useful in the further development of response shift theory. The model with its explicit list of assumptions and hypothesized relationships lends itself for critical, empirical examination. Future studies are needed to empirically test the assumptions and hypothesized relationships.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11136-021-02846-w.

Implications of response shift for micro-, meso-, and macro-level healthcare decision-making using results of patient-reported outcome measures

PURPOSE

Results of patient-reported outcome measures (PROMs) are increasingly used to inform healthcare decision-making. Research has shown that response shift can impact PROM results. As part of an international collaboration, our goal is to provide a framework regarding the implications of response shift at the level of patient care (micro), healthcare institute (meso), and healthcare policy (macro).

METHODS

Empirical evidence of response shift that can influence patients' self-reported health and preferences provided the foundation for development of the framework. Measurement validity theory, hermeneutic philosophy, and micro-, meso-, and macro-level healthcare decision-making informed our theoretical analysis.

RESULTS

At the micro-level, patients' self-reported health needs to be interpreted via dialogue with the clinician to avoid misinterpretation of PROM data due to response shift. It is also important to consider the potential impact of response shift on study results, when these are used to support decisions. At the meso-level, individual-level data should be examined for response shift before aggregating PROM data for decision-making related to quality improvement, performance monitoring, and accreditation. At the macro-level, critical reflection on the conceptualization of health is required to know whether response shift needs to be controlled for when PROM data are used to inform healthcare coverage.

CONCLUSION

Given empirical evidence of response shift, there is a critical need for guidelines and knowledge translation to avoid potential misinterpretations of PROM results and consequential biases in decision-making. Our framework with guiding questions provides a structure for developing strategies to address potential impacts of response shift at micro-, meso-, and macro-levels.