Feasibility of computerized adaptive testing for collection of patient-reported outcomes after inpatient rehabilitation

Test-Retest Reliability and Responsiveness of the Computerized Adaptive Testing System of the Functional Assessment of Stroke

OBJECTIVE

To examine the test-retest reliability, responsiveness, and clinical utility of the Computerized Adaptive Testing System of the Functional Assessment of Stroke (CAT-FAS) in persons with stroke.

DESIGN

Repeated measurements design.

SETTING

A department of rehabilitation in a medical center.

PARTICIPANTS

30 persons with chronic stroke (for test-retest reliability) and 65 persons with subacute stroke (for responsiveness) were recruited. To examine the test-retest reliability, the participants received measurements twice at 1-month intervals. To examine the responsiveness, the data were collected at admission and discharge from hospital.

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASUREMENT TOOL

CAT-FAS.

RESULTS

The intra-class correlation coefficients of the CAT-FAS were ≥0.82, indicating good to excellent test-retest reliability. The Kazis' effect size and standardized response mean of the CAT-FAS were ≥0.96, indicating good group-level responsiveness. For individual-level responsiveness, approximately two-thirds of the participants exceeded the conditional minimal detectable change. On average, the CAT-FAS was completed within 9 items and 3 minutes per administration.

CONCLUSIONS

Our results suggest the CAT-FAS is an efficient measurement tool with good to excellent test-retest reliability and responsiveness. In addition, the CAT-FAS can be used routinely in clinical settings to monitor progress of the crucial 4 domains for persons with stroke.

Feasibility of PROMIS using computerized adaptive testing during inpatient rehabilitation

BACKGROUND

There has been an increased significance on patient-reported outcomes in clinical settings. We aimed to evaluate the feasibility of administering patient-reported outcome measures by computerized adaptive testing (CAT) using a tablet computer with rehabilitation inpatients, assess workload demands on staff, and estimate the extent to which rehabilitation inpatients have elevated T-scores on six Patient Reported Outcomes Measurement Information System® (PROMIS®) measures.

METHODS

Patients (N = 108) with stroke, spinal cord injury, traumatic brain injury, and other neurological disorders participated in this study. PROMIS computerized adaptive tests (CAT) were administered via a web-based platform. Summary scores were calculated for six measures: Pain Interference, Sleep Disruption, Anxiety, Depression, Illness Impact Positive, and Illness Impact Negative. We calculated the percent of patients with T-scores equivalent to 2 standard deviations or greater above the mean.

RESULTS

During the first phase, we collected data from 19 of 49 patients; of the remainder, 61% were not available or had cognitive or expressive language impairments. In the second phase of the study, 40 of 59 patients participated to complete the assessment. The mean PROMIS T-scores were in the low 50 s, indicating an average symptom level, but 19-31% of patients had elevated T-scores where the patients needed clinical action.

CONCLUSIONS

The study demonstrated that PROMIS assessment using a CAT administration during an inpatient rehabilitation setting is feasible with the presence of a research staff member to complete PROMIS assessment.

Remote Follow-Up Technologies in Traumatic Brain Injury: A Scoping Review

Traumatic brain injury (TBI) remains a leading cause of death and disability worldwide. Motivations for outcome data collection in TBI are threefold: to improve patient outcomes, to facilitate research, and to provide the means and methods for wider injury surveillance. Such data play a pivotal role in population health, and ways to increase the reliability of data collection following TBI should be pursued. As a result, technology-aided follow-up of patients with neurotrauma is on the rise; there is, therefore, a need to describe how such technologies have been used. A scoping review was conducted and reported using the PRISMA extension (PRISMA-ScR). Five electronic databases (Embase, MEDLINE, Global Health, PsycInfo, and Scopus) were searched systematically using keywords derived from the concepts of "telemedicine," "TBI," "outcome assessment," and "patient-generated health data." Forty studies described follow-up technologies (FUTs) utilizing telephones (52.5%, n = 21), short message service (SMS; 10%, n = 4), smartphones (22.5%, n = 9), videoconferencing (10%, n = 4), digital assistants (2.5%, n = 1), and custom devices (2.5%, n = 1) among cohorts of patients with TBI of varying injury severity. Where reported, clinical facilitators, remote follow-up timing and intervals between sessions, synchronicity of follow-up instances, proxy involvement, outcome measures utilized, and technology evaluation efforts are described. FUTs can aid more temporally sensitive assessments and capture fluctuating sequelae, a benefit of particular relevance to TBI cohorts. However, the evidence base surrounding FUTs remains in its infancy, particularly with respect to large samples, low- and middle-income patient cohorts, and the validation of outcome measures for deployment via such remote technology.

A Mobile Health App for the Collection of Functional Outcomes After Inpatient Stroke Rehabilitation: Pilot Randomized Controlled Trial

BACKGROUND

Monitoring the functional status of poststroke patients after they transition home is significant for rehabilitation. Mobile health (mHealth) technologies may provide an opportunity to reach and follow patients post discharge. However, the feasibility and validity of functional assessments administered by mHealth technologies are unknown.

OBJECTIVE

This study aimed to evaluate the feasibility, validity, and reliability of functional assessments administered through the videoconference function of a mobile phone-based app compared with administration through the telephone function in poststroke patients after rehabilitation hospitalization.

METHODS

A randomized controlled trial was conducted in a rehabilitation hospital in Southeast China. Participants were randomly assigned to either a videoconference follow-up (n=60) or a telephone follow-up (n=60) group. We measured the functional status of participants in each group at 2-week and 3-month follow-up periods. Half the participants in each group were followed by face-to-face home visit assessments as the gold standard. Validity was assessed by comparing any score differences between videoconference follow-up and home visit assessments, as well as telephone follow-up and home visit assessments. Reliability was assessed by computing agreements between videoconference follow-up and home visit assessments, as well as telephone follow-up and home visit assessments. Feasibility was evaluated by the levels of completion, satisfaction, comfort, and confidence in the 2 groups.

RESULTS

Scores obtained from the videoconference follow-up were similar to those of the home visit assessment. However, most scores collected from telephone administration were higher than those of the home visit assessment. The agreement between videoconference follow-up and home visit assessments was higher than that between telephone follow-up and home visit assessments at all follow-up periods. In the telephone follow-up group, completion rates were 95% and 82% at 2-week and 3-month follow-up points, respectively. In the videoconference follow-up group, completion rates were 95% and 80% at 2-week and 3-month follow-up points, respectively. There were no differences in the completion rates between the 2 groups at all follow-up periods (X²₁=1.6, P=.21 for 2-week follow-up; X²₁=1.9, P=.17 for 3-month follow-up). Patients in the videoconference follow-up group perceived higher confidence than those in the telephone follow-up group at both 2-week and 3-month follow-up periods (X²₃=6.7, P=.04 for 2-week follow-up; X²₃=8.0, P=.04 for 3-month follow-up). The videoconference follow-up group demonstrated higher satisfaction than the telephone follow-up group at 3-month follow-up (X²₃=13.9; P=.03).

CONCLUSIONS

The videoconference follow-up assessment of functional status demonstrates higher validity and reliability, as well as higher confidence and satisfaction perceived by patients, than the telephone assessment. The videoconference assessment provides an efficient means of assessing functional outcomes of patients after hospital discharge. This method provides a novel solution for clinical trials requiring longitudinal assessments.

TRIAL REGISTRATION

chictr.org.cn: ChiCTR1900027626; http://www.chictr.org.cn/edit.aspx?pid=44831&htm=4.

Computerized adaptive testing to screen children for emotional and behavioral problems by preventive child healthcare

Background

Questionnaires to detect emotional and behavioral problems (EBP) in Preventive Child Healthcare (PCH) should be short which potentially affects validity and reliability. Simulation studies have shown that Computerized Adaptive Testing (CAT) could overcome these weaknesses. We studied the applicability (using the measures participation rate, satisfaction, and efficiency) and the validity of CAT in routine PCH practice.

Methods

We analyzed data on 461 children aged 10–11 years (response 41%), who were assessed during routine well-child examinations by PCH professionals. Before the visit, parents completed the CAT and the Child Behavior Checklist (CBCL). Satisfaction was measured by parent- and PCH professional-report. Efficiency of the CAT procedure was measured as number of items needed to assess whether a child has serious problems or not. Its validity was assessed using the CBCL as the criterion.

Results

Parents and PCH professionals rated the CAT on average as good. The procedure required at average 16 items to assess whether a child has serious problems or not. Agreement of scores on the CAT scales with corresponding CBCL scales was high (range of Spearman correlations 0.59–0.72). Area Under Curves (AUC) were high (range: 0.95–0.97) for the Psycat total, externalizing, and hyperactivity scales using corresponding CBCL scale scores as criterion. For the Psycat internalizing scale the AUC was somewhat lower but still high (0.86).

Conclusions

CAT is a valid procedure for the identification of emotional and behavioral problems in children aged 10–11 years. It may support the efficient and accurate identification of children with overall, and potentially also specific, emotional and behavioral problems in routine PCH.

Development of a Computerized Adaptive Testing System of the Functional Assessment of Stroke

OBJECTIVE

To develop a computerized adaptive testing system of the Functional Assessment of Stroke (CAT-FAS) to assess upper- and lower-extremity (UE/LE) motor function, postural control, and basic activities of daily living with optimal efficiency and without sacrificing psychometric properties in patients with stroke.

DESIGN

Simulation study.

SETTING

One rehabilitation unit in a medical center.

PARTICIPANTS

Patients with subacute stroke (N=301; mean age, 67.3±10.9; intracranial infarction, 74.5%).

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURES

The UE and LE subscales of the Fugl-Meyer Assessment, Postural Assessment Scale for Stroke Patients, and Barthel Index.

RESULTS

The CAT-FAS adopting the optimal stopping rule (limited reliability increase of <.010) had good Rasch reliability across the 4 domains (.88-.93) and needed few items for the whole administration (8.5 items on average). The concurrent validity (CAT-FAS vs original tests, Pearson r=.91-.95) and responsiveness (standardized response mean, .65-.76) of the CAT-FAS were good in patients with stroke.

CONCLUSIONS

We developed the CAT-FAS, and our results support that the CAT-FAS has sufficient efficiency, reliability, concurrent validity, and responsiveness in patients with stroke. The CAT-FAS can be used to simultaneously assess patients' functions of UE, LE, postural control, and basic activities of daily living using, on average, no more than 10 items; this efficiency is useful in reducing the assessment burdens for both clinicians and patients.

Applying Computerized Adaptive Testing to the Four-Dimensional Symptom Questionnaire (4DSQ): A Simulation Study

BACKGROUND

Efficient screening questionnaires are useful in general practice. Computerized adaptive testing (CAT) is a method to improve the efficiency of questionnaires, as only the items that are particularly informative for a certain responder are dynamically selected.

OBJECTIVE

The objective of this study was to test whether CAT could improve the efficiency of the Four-Dimensional Symptom Questionnaire (4DSQ), a frequently used self-report questionnaire designed to assess common psychosocial problems in general practice.

METHODS

A simulation study was conducted using a sample of Dutch patients visiting a general practitioner (GP) with psychological problems (n=379). Responders completed a paper-and-pencil version of the 50-item 4DSQ and a psychometric evaluation was performed to check if the data agreed with item response theory (IRT) assumptions. Next, a CAT simulation was performed for each of the four 4DSQ scales (distress, depression, anxiety, and somatization), based on the given responses as if they had been collected through CAT. The following two stopping rules were applied for the administration of items: (1) stop if measurement precision is below a predefined level, or (2) stop if more than half of the items of the subscale are administered.

RESULTS

In general, the items of each of the four scales agreed with IRT assumptions. Application of the first stopping rule reduced the length of the questionnaire by 38% (from 50 to 31 items on average). When the second stopping rule was also applied, the total number of items could be reduced by 56% (from 50 to 22 items on average).

CONCLUSIONS

CAT seems useful for improving the efficiency of the 4DSQ by 56% without losing a considerable amount of measurement precision. The CAT version of the 4DSQ may be useful as part of an online assessment to investigate the severity of mental health problems of patients visiting a GP. This simulation study is the first step needed for the development a CAT version of the 4DSQ. A CAT version of the 4DSQ could be of high value for Dutch GPs since increasing numbers of patients with mental health problems are visiting the general practice. In further research, the results of a real-time CAT should be compared with the results of the administration of the full scale.