The Dosing of Mobile-Based Just-in-Time Adaptive Self-Management Prompts for Caregivers: Preliminary Findings From a Pilot Microrandomized Study

BACKGROUND

Caregivers of people with chronic illnesses often face negative stress-related health outcomes and are unavailable for traditional face-to-face interventions due to the intensity and constraints of their caregiver role. Just-in-time adaptive interventions (JITAIs) have emerged as a design framework that is particularly suited for interventional mobile health studies that deliver in-the-moment prompts that aim to promote healthy behavioral and psychological changes while minimizing user burden and expense. While JITAIs have the potential to improve caregivers' health-related quality of life (HRQOL), their effectiveness for caregivers remains poorly understood.

OBJECTIVE

The primary objective of this study is to evaluate the dose-response relationship of a fully automated JITAI-based self-management intervention involving personalized mobile app notifications targeted at decreasing the level of caregiver strain, anxiety, and depression. The secondary objective is to investigate whether the effectiveness of this mobile health intervention was moderated by the caregiver group. We also explored whether the effectiveness of this intervention was moderated by (1) previous HRQOL measures, (2) the number of weeks in the study, (3) step count, and (4) minutes of sleep.

METHODS

We examined 36 caregivers from 3 disease groups (10 from spinal cord injury, 11 from Huntington disease, and 25 from allogeneic hematopoietic cell transplantation) in the intervention arm of a larger randomized controlled trial (subjects in the other arm received no prompts from the mobile app) designed to examine the acceptability and feasibility of this intensive type of trial design. A series of multivariate linear models implementing a weighted and centered least squares estimator were used to assess the JITAI efficacy and effect.

RESULTS

We found preliminary support for a positive dose-response relationship between the number of administered JITAI messages and JITAI efficacy in improving caregiver strain, anxiety, and depression; while most of these associations did not meet conventional levels of significance, there was a significant association between high-frequency JITAI and caregiver strain. Specifically, administering 5-6 messages per week as opposed to no messages resulted in a significant decrease in the HRQOL score of caregiver strain with an estimate of -6.31 (95% CI -11.76 to -0.12; P=.046). In addition, we found that the caregiver groups and the participants' levels of depression in the previous week moderated JITAI efficacy.

CONCLUSIONS

This study provides preliminary evidence to support the effectiveness of the self-management JITAI and offers practical guidance for designing future personalized JITAI strategies for diverse caregiver groups.

TRIAL REGISTRATION

ClinicalTrials.gov NCT04556591; https://clinicaltrials.gov/ct2/show/NCT04556591.

Construct Validity of Pediatric PROMIS Computerized Adaptive Testing Measures in Children With Adolescent Idiopathic Scoliosis

BACKGROUND

The use of patient-reported outcome measures, especially Patient-Reported Outcomes Measurement Information System (PROMIS) measures, has increased in recent years. Given this growth, it is imperative to ensure that the measures being used are validated for the intended population(s)/disease(s). Our objective was to assess the construct validity of 8 PROMIS computer adaptive testing (CAT) measures among children with adolescent idiopathic scoliosis (AIS).

METHODS

We prospectively enrolled 200 children (aged 10 to 17 y) with AIS, who completed 8 PROMIS CATs (Anxiety, Depressive Symptoms, Mobility, Pain Behavior, Pain Interference, Peer Relationships, Physical Activity, Physical Stress Experiences) and the Scoliosis Research Society-22r questionnaire (SRS-22r) electronically. Treatment categories were observation, bracing, indicated for surgery, or postoperative from posterior spinal fusion. Construct validity was evaluated using known group analysis and convergent and discriminant validity analyses. Analysis of variance was used to identify differences in PROMIS T -scores by treatment category (known groups). The Spearman rank correlation coefficient ( rs ) was calculated between corresponding PROMIS and SRS-22r domains (convergent) and between unrelated PROMIS domains (discriminant). Floor/ceiling effects were calculated.

RESULTS

Among treatment categories, significant differences were found in PROMIS Mobility, Pain Behavior, Pain Interference, and Physical Stress Experiences and in all SRS-22r domains ( P <0.05) except Mental Health ( P =0.15). SRS-22r Pain was strongly correlated with PROMIS Pain Interference ( rs =-0.72) and Pain Behavior ( rs =-0.71) and moderately correlated with Physical Stress Experiences ( rs =-0.57). SRS-22r Mental Health was strongly correlated with PROMIS Depressive Symptoms ( rs =-0.72) and moderately correlated with Anxiety ( rs =-0.62). SRS-22r Function was moderately correlated with PROMIS Mobility ( rs =0.64) and weakly correlated with Physical Activity ( rs =0.34). SRS-22r Self-Image was weakly correlated with PROMIS Peer Relationships ( rs =0.33). All unrelated PROMIS CATs were weakly correlated (| rs |<0.40). PROMIS Anxiety, Mobility, Pain Behavior, and Pain Interference and SRS-22r Function, Pain, and Satisfaction displayed ceiling effects.

CONCLUSIONS

Evidence supports the construct validity of 6 PROMIS CATs in evaluating AIS patients. Ceiling effects should be considered when using specific PROMIS CATs.

LEVEL OF EVIDENCE

Level II, prognostic.

On the Performance of Semi- and Nonparametric Item Response Functions in Computer Adaptive Tests

Large-scale assessments often use a computer adaptive test (CAT) for selection of items and for scoring respondents. Such tests often assume a parametric form for the relationship between item responses and the underlying construct. Although semi- and nonparametric response functions could be used, there is scant research on their performance in a CAT. In this work, we compare parametric response functions versus those estimated using kernel smoothing and a logistic function of a monotonic polynomial. Monotonic polynomial items can be used with traditional CAT item selection algorithms that use analytical derivatives. We compared these approaches in CAT simulations with a variety of item selection algorithms. Our simulations also varied the features of the calibration and item pool: sample size, the presence of missing data, and the percentage of nonstandard items. In general, the results support the use of semi- and nonparametric item response functions in a CAT.

The PROMIS Upper Extremity Computer Adaptive Test Correlates With Previously Validated Metrics in Patients With Carpal Tunnel Syndrome

Background: The Patient-Reported Outcomes Measurement Information System Upper Extremity (PROMIS UE) computer adaptive test was developed to improve precision and reduce question burden. We hypothesized that in patients with carpal tunnel syndrome (CTS): (1) PROMIS UE would correlate with established patient-reported outcome measures (PROs); (2) the time and number of questions required would be lower than current metrics; (3) there would be no floor or ceiling effects; and (4) PROMIS UE would not correlate with disease severity. Methods: Patients undergoing electrodiagnostic evaluation found to have a primary diagnosis of unilateral CTS prospectively completed PROMIS UE, Quick Disabilities of the Arm, Shoulder and Hand (qDASH), and Boston Carpal Tunnel Syndrome Questionnaire (BCTQ). Electrophysiologic and clinical severity was recorded. The relationships among PROs were described with Spearman coefficients. A floor or ceiling effect was confirmed if >15% of patients achieved the lowest or highest possible score, respectively. Results: Fifty-one patients (average, 53.9 years) were enrolled. An excellent correlation was identified between PROMIS UE and qDASH (R = -0.76, P < .001). There was a good correlation between PROMIS UE and BCTQ (R = -0.58, P < 0.001). The PROMIS UE required less time and fewer questions than qDASH and BCTQ (P = .02 and P < .001). There were no floor or ceiling effects. Neither neurophysiologic nor clinical severity correlated with PROMIS UE (R = 0.24, P > .05 and R = -0.18, P > .05). Conclusions: The PROMIS UE has an excellent correlation with qDASH and a good correlation with BCTQ in patients with CTS. Furthermore, PROMIS UE required less time and fewer questions than established PROs. Used as a single PRO, PROMIS UE represents a practical alternative to current metrics in patients with CTS.

PROMIS physical function underperforms psychometrically relative to American Shoulder and Elbow Surgeons score in patients undergoing anatomic total shoulder arthroplasty

BACKGROUND

The purpose of this study was to evaluate the psychometric properties of the Patient-Reported Outcomes Measurement Information System (PROMIS) physical function computer adaptive test (PF-CAT) relative to the American Shoulder and Elbow Surgeons (ASES) score in patients with glenohumeral osteoarthritis undergoing primary anatomic total shoulder arthroplasty (TSA).

METHODS

A retrospective study of an institutional TSA registry was performed. Preoperative PROMIS PF-CAT and ASES scores were collected. Floor and ceiling effects were determined, and convergent validity was established through Pearson correlations. Rasch partial credit modeling was used for psychometric analysis of the validity of PF-CAT and ASES question items. Person-item maps were generated to characterize the distribution of question responses along the latent dimension of shoulder disability.

RESULTS

Responses from 179 patients (184 shoulders) were included. PF-CAT had a moderate correlation to ASES (r = 0.487; P < .001), with no floor or ceiling effects; ASES had a 1.1% floor effect and no ceiling effect. With iterative Rasch model item-reduction analysis eliminating poorly fitting question items, all possible PF-CAT items were eliminated after 6 iterations. With ASES, just 1 function question item was dropped. Person-item maps showed ASES to be superior to PROMIS PF-CAT psychometrically, with sequential and improved coverage of the latent dimension of shoulder disability.

CONCLUSION

Despite moderate correlation with ASES, PROMIS PF-CAT demonstrated inferior validity and psychometric properties in patients undergoing TSA. PF-CAT should not replace the ASES in this population of patients.

Correlation of PROMIS Physical Function, Pain Interference, and Depression in Pediatric and Adolescent Patients in the Ambulatory Sports Medicine Clinic

Background:

Recently, interest has increased in incorporating the National Institutes of Health Patient-Reported Outcomes Measurement Information System (PROMIS) outcomes into clinical and research applications in sports medicine. The PROMIS forms have not been studied in pediatric and adolescent sports medicine patients.

Purpose/Hypothesis:

The goal of this study was to determine the correlation between PROMIS Computer Adaptive Test (CAT) forms measuring physical function, pain interference, and depression in pediatric and adolescent patients seen in the ambulatory sports medicine clinic. We hypothesized that there would be a negative correlation between physical function and pain interference as well as depression, as has been demonstrated in adult patient populations.

Study Design:

Cross-sectional study; Level of evidence, 3.

Methods:

All patients aged 8 to 17 years seen by 3 shoulder and sports medicine providers were included in this study. Patients completed a series of PROMIS CAT forms at clinic visits, including the PROMIS-PF and PROMIS-UE (Physical Function and Upper Extremity; depending on the nature of the complaint), PROMIS-PI (Pain Interference), and PROMIS-Depression subscales. Pearson correlation coefficients were calculated between the PROMIS forms as well as with other patient demographic data.

Results:

A total of 236 patient visits (152 patients) were included in the study, comprising 712 total PROMIS CAT forms. A negative correlation was found between PROMIS-PF and both PROMIS-Depression (R = –0.34) and PROMIS-PI (R = –0.76). These correlations with PROMIS-Depression and PROMIS-PI were –0.21 and –0.75, respectively, when considering the PROMIS-UE CAT. Patient demographic information had minimal impact on PROMIS scores as well as on correlations between scores.

Conclusion:

Correlations between physical function, pain interference, and depression were found to be similar in pediatric patients as they are in adult patients, as measured by PROMIS CAT forms.

Translation and cross-cultural adaptation of the PEDI-CAT: Dutch version

PURPOSE

The PEDI-CAT measures daily functioning of children and youth, aged 1 to 21 years, with a variety of physical, cognitive and/or behavioral disabilities. In order to use an instrument in another culture or language, translation and cross-cultural validation are important, particularly for end-users. This study describes the process of translation and cross-cultural adaptation of the Dutch version of the PEDI-CAT.

METHODS

End-users were involved in all steps. First, the PEDI-CAT items were reviewed to determine whether the items were relevant and acceptable in the Dutch culture. Then, the PEDI-CAT was translated into Dutch using specific guidelines. Finally, the wording of the Dutch items and response options were reviewed and tested with 22 parents of children and adolescents with and without disabilities.

RESULTS

All 267 items and response options of the original PEDI-CAT were assessed as relevant and translated into Dutch. A selection of 175 items was tested with Think Aloud interviews which revealed that the translation of 46 items could be improved.

CONCLUSION

The role of end-users in the process of translation and cross-cultural adaptation was crucial. This collaborative process resulted in a Dutch version of the PEDI-CAT that has been optimally adapted to the Dutch language and culture.

Adaptive Testing With a Hierarchical Item Response Theory Model

The hierarchical item response theory (H-IRT) model is very flexible and allows a general factor and subfactors within an overall structure of two or more levels. When an H-IRT model with a large number of dimensions is used for an adaptive test, the computational burden associated with interim scoring and selection of subsequent items is heavy. An alternative approach for any high-dimension adaptive test is to reduce dimensionality for interim scoring and item selection and then revert to full dimensionality for final score reporting, thereby significantly reducing the computational burden. This study compared the accuracy and efficiency of final scoring for multidimensional, local multidimensional, and unidimensional item selection and interim scoring methods, using both simulated and real item pools. The simulation study was conducted under 10 conditions (i.e., five test lengths and two H-IRT models) with a simulated sample of 10,000 students. The study with the real item pool was conducted using item parameters from an actual 45-item adaptive test with a simulated sample of 10,000 students. Results indicate that the theta estimations provided by the local multidimensional and unidimensional item selection and interim scoring methods were relatively as accurate as the theta estimation provided by the multidimensional item selection and interim scoring method, especially during the real item pool study. In addition, the multidimensional method required the longest computation time and the unidimensional method required the shortest computation time.

Investigation of Response Changes in the GRE Revised General Test

Research on examinees' response changes on multiple-choice tests over the past 80 years has yielded some consistent findings, including that most examinees make score gains by changing answers. This study expands the research on response changes by focusing on a high-stakes admissions test-the Verbal Reasoning and Quantitative Reasoning measures of the GRE revised General Test. We analyzed data from 8,538 examinees for Quantitative and 9,140 for Verbal sections who took the GRE revised General Test in 12 countries. The analyses yielded findings consistent with prior research. In addition, as examinees' ability increases, the benefit of response changing increases. The study yielded significant implications for both test agencies and test takers. Computer adaptive tests often do not allow the test takers to review and revise. Findings from this study confirm the benefit of such features.

Scale refinement and initial evaluation of a behavioral health function measurement tool for work disability evaluation

OBJECTIVE

To use item response theory (IRT) data simulations to construct and perform initial psychometric testing of a newly developed instrument, the Social Security Administration Behavioral Health Function (SSA-BH) instrument, that aims to assess behavioral health functioning relevant to the context of work.

DESIGN

Cross-sectional survey followed by IRT calibration data simulations.

SETTING

Community.

PARTICIPANTS

Sample of individuals applying for Social Security Administration disability benefits: claimants (n=1015) and a normative comparative sample of U.S. adults (n=1000).

INTERVENTIONS

None.

MAIN OUTCOME MEASURE

SSA-BH measurement instrument.

RESULTS

IRT analyses supported the unidimensionality of 4 SSA-BH scales: mood and emotions (35 items), self-efficacy (23 items), social interactions (6 items), and behavioral control (15 items). All SSA-BH scales demonstrated strong psychometric properties including reliability, accuracy, and breadth of coverage. High correlations of the simulated 5- or 10-item computer adaptive tests with the full item bank indicated robust ability of the computer adaptive testing approach to comprehensively characterize behavioral health function along 4 distinct dimensions.

CONCLUSIONS

Initial testing and evaluation of the SSA-BH instrument demonstrated good accuracy, reliability, and content coverage along all 4 scales. Behavioral function profiles of Social Security Administration claimants were generated and compared with age- and sex-matched norms along 4 scales: mood and emotions, behavioral control, social interactions, and self-efficacy. Using the computer adaptive test-based approach offers the ability to collect standardized, comprehensive functional information about claimants in an efficient way, which may prove useful in the context of the Social Security Administration's work disability programs.

Symptom burden and comorbidities impact the consistency of responses on patient-reported functional outcomes

OBJECTIVE

To assess the influence of symptom intensity, mood, and comorbidities on patient-clinician agreement and the consistency of responses to functional patient-reported outcomes (PROs).

DESIGN

Two data sources were used. The first, a cross-sectional database of patients with breast cancer who completed functional PROs and were administered the FIM, was used to examine whether average pain intensity (as measured with an 11-point numeric rating scale [NRS]) and Rand Mental Health inventory scores differed among those rating their functional independence as different than clinicians. The second, a longitudinal database of 311 adults with late-stage lung cancer who completed the Activity Measure for Post Acute Care Computer Adaptive Test (AM PAC CAT) with differences between their expected and actual responses as reflected in their AM PAC CAT SEs.

SETTING

Two tertiary medical centers.

PARTICIPANTS

Data source #1, 163 women with stage IV breast cancer; data source #2, 311 adults with late-stage lung cancer.

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURES

Data source #1, FIM, pain NRS, Older Americans Resource Study activities of daily living subscale, Physical Function-10, Mental Health Inventory-17. Data source #2, AM PAC CAT and NRS symptom ratings.

RESULTS

Pain intensity was significantly higher when clinicians and patients disagreed regarding a patient's independence in the ability to transfer (NRS pain severity, 3.78 vs 2.40; P=.014), groom (3.71 vs 2.36, P=.009), bathe (3.76 vs 2.40, P=.016), and dress (3.09 vs 2.44, P=.034). The magnitude of AM PAC CAT SEs was significantly associated with the severity of participants' pain, dyspnea, and fatigue, as well as the presence of musculoskeletal disorders and coronary artery disease. Neither mood nor emotional distress was associated with clinician-patient agreement or AM PAC CAT SE.

CONCLUSIONS

Pain intensity is associated with disagreement between patients and clinicians about the patient's level of functioning. Moreover, physical symptoms (pain, dyspnea, fatigue) as well as specific medical comorbidities (musculoskeletal disorders, coronary artery disease), but not mood, are associated with inconsistency in patients' assessment of their functional abilities.

Development and evaluation of a computer adaptive test to assess anxiety in cardiovascular rehabilitation patients

OBJECTIVE

To develop and evaluate a computer adaptive test for the assessment of anxiety in cardiovascular rehabilitation patients (ACAT-cardio) that tailors an optimal test for each patient and enables precise and time-effective measurement.

DESIGN

Simulation study, validation study (against the anxiety subscale of the Hospital Anxiety and Depression Scale and the physical component summary scale of the 12-Item Short-Form Health Survey), and longitudinal study (beginning and end of rehabilitation).

SETTING

Cardiac rehabilitation centers.

PARTICIPANTS

Cardiovascular rehabilitation patients: simulation study sample (n=106; mean age, 57.8y; 25.5% women) and validation and longitudinal study sample (n=138; mean age, 58.6 and 57.9y, respectively; 16.7% and 12.1% women, respectively).

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURES

Hospital Anxiety and Depression Scale, 12-Item Short-Form Health Survey, and ACAT-cardio.

RESULTS

The mean number of items was 9.2 with an average processing time of 1:13 minutes when an SE ≤.50 was used as a stopping rule; with an SE ≤.32, there were 28 items and a processing time of 3:47 minutes. Validity could be confirmed via correlations between .68 and .81 concerning convergent validity (ACAT-cardio vs Hospital Anxiety and Depression Scale anxiety subscale) and correlations between -.47 and -.30 concerning discriminant validity (ACAT-cardio vs 12-Item Short-Form Health Survey physical component summary scale). Sensitivity to change was moderate to high with standardized response means between .45 and .82.

CONCLUSIONS

The ACAT-cardio shows good psychometric properties and provides the opportunity for an innovative and time-effective assessment of anxiety in cardiovascular rehabilitation. A more flexible stopping rule might further improve the ACAT-cardio. Additionally, testing in other cardiovascular populations would increase generalizability.