Adaptive Measurement of Change: A Novel Method to Reduce Respondent Burden and Detect Significant Individual-Level Change in Patient-Reported Outcome Measures

Development and psychometric evaluation of item banks for memory and attention - supplements to the EORTC CAT Core instrument

BACKGROUND

Cancer patients may experience a decrease in cognitive functioning before, during and after cancer treatment. So far, the Quality of Life Group of the European Organisation for Research and Treatment of Cancer (EORTC QLG) developed an item bank to assess self-reported memory and attention within a single, cognitive functioning scale (CF) using computerized adaptive testing (EORTC CAT Core CF item bank). However, the distinction between different cognitive functions might be important to assess the patients' functional status appropriately and to determine treatment impact. To allow for such assessment, the aim of this study was to develop and psychometrically evaluate separate item banks for memory and attention based on the EORTC CAT Core CF item bank.

METHODS

In a multistep process including an expert-based content analysis, we assigned 44 items from the EORTC CAT Core CF item bank to the memory or attention domain. Then, we conducted psychometric analyses based on a sample used within the development of the EORTC CAT Core CF item bank. The sample consisted of 1030 cancer patients from Denmark, France, Poland, and the United Kingdom. We evaluated measurement properties of the newly developed item banks using confirmatory factor analysis (CFA) and item response theory model calibration.

RESULTS

Item assignment resulted in 31 memory and 13 attention items. Conducted CFAs suggested good fit to a 1-factor model for each domain and no violations of monotonicity or indications of differential item functioning. Evaluation of CATs for both memory and attention confirmed well-functioning item banks with increased power/reduced sample size requirements (for CATs ≥ 4 items and up to 40% reduction in sample size requirements in comparison to non-CAT format).

CONCLUSION

Two well-functioning and psychometrically robust item banks for memory and attention were formed from the existing EORTC CAT Core CF item bank. These findings could support further research on self-reported cognitive functioning in cancer patients in clinical trials as well as for real-word-evidence. A more precise assessment of attention and memory deficits in cancer patients will strengthen the evidence on the effects of cancer treatment for different cancer entities, and therefore contribute to shared and informed clinical decision-making.

Optimizing Patient-Reported Outcome Collection and Documentation in Medical Music Therapy: Process-Improvement Study

BACKGROUND

To measure the effectiveness of nonpharmacologic interventions delivered during clinical care, investigators need to ensure robust and routine data collection without disrupting individualized patient care or adding unnecessary documentation burden.

OBJECTIVE

A process-improvement study was undertaken to improve documentation consistency and increase the capture of patient-reported outcomes (PROs; ie, stress, pain, anxiety, and coping) within a medical music therapy (MT) team.

METHODS

We used 2 Plan-Do-Study-Act (PDSA) cycles to improve documentation processes among an MT team (13.3 clinical full-time equivalent staff). Trainings focused on providing skills and resources for optimizing pre- and postsession PRO collection, specific guidelines for entering session data in the electronic health record, and opportunities for the team to provide feedback. Two comparisons of therapists' PRO collection rates were conducted: (1) between the 6 months before PDSA Cycle 1 (T0) and PDSA Cycle 1 (T1), and (2) between T1 and PDSA Cycle 2 (T2).

RESULTS

Music therapists' rates of capturing any PRO within MT sessions increased significantly (P<.001) from T0 to T1 and from T1 to T2 for all domains, including stress (4/2758, 0.1% at T0; 1012/2786, 36.3% at T1; and 393/775, 50.7% at T2), pain (820/2758, 29.7% at T0; 1444/2786, 51.8% at T1; and 476/775, 61.4% at T2), anxiety (499/2758, 18.1% at T0; 950/2786, 34.1% at T1; and 400/775, 51.6% at T2), and coping (0/2758, 0% at T0; 571/2786, 20.5% at T1; and 319/775, 41.2% at T2). Music therapists' feedback and findings from a retrospective analysis were used to create an improved electronic health record documentation template.

CONCLUSIONS

Rates of PRO data collection improved within the medical MT team. Although the process improvement in this study was applied to a nonpharmacologic MT intervention, the principles are applicable to numerous inpatient clinical providers. As hospitals continue to implement nonpharmacologic therapies in response to the Joint Commission's recommendations, routine PRO collection will provide future researchers with the ability to evaluate the impact of these therapies on pain relief and opioid use.

Clinical Delivery and Effectiveness of Music Therapy in Hematology and Oncology: An EMMPIRE Retrospective Study

BACKGROUND

Music therapy (MT) has been shown to improve outcomes for patients with sickle cell disease (SCD) and patients with hematologic and/or oncologic conditions excluding SCD (HemOnc) in prior randomized trials. While few studies have described the clinical delivery (ie, volume, clinical settings, patient characteristics, referrals, and session characteristics) of MT and examined its real-world effectiveness, no studies have compared responses between hematology/oncology populations. The purpose of this study was to examine the clinical delivery and effectiveness of MT at a freestanding academic cancer center and compare the effectiveness of MT on pain, anxiety, and fatigue between adult patients in the HemOnc and SCD groups.

METHODS

A retrospective review was conducted of all MT sessions provided at a freestanding academic cancer center between January 2017 and July 2020. The unadjusted single-session effects of MT on pain, anxiety, and fatigue were assessed among patients reporting symptoms ≥1 out of 10 on a 0 to 10 scale. Adjustments were made for multiple sessions on the same patient using a mixed model to compare pre-session and change scores between the HemOnc and SCD groups. Patients' comments were analyzed using conventional qualitative content analysis.

RESULTS

Music therapists provided 4002 sessions to 1152 patients including 1012 in the HemOnc group and 140 in the SCD group. In the combined sample, statistically significant reductions in pain (1.48 units), anxiety (2.58 units), and fatigue (0.84 units) were observed, with changes in pain and anxiety exceeding clinically significant thresholds. After adjustment, the SCD group reported significantly greater pre-session pain (7.22 vs 5.81) and anxiety (6.11 vs 5.17) as well as greater anxiety reduction (2.89 vs 2.23) than the HemOnc group. Patients' comments contained themes including enjoyment, gratitude, and improvements in mood, pain, and anxiety.

CONCLUSIONS

This study supports the delivery and clinical effectiveness of MT for addressing the needs of patients throughout their course of treatment at an academic cancer center and justifies the inclusion of individuals with SCD within integrative oncology services.

Key considerations to reduce or address respondent burden in patient-reported outcome (PRO) data collection

Patient-reported outcomes (PROs) are used in clinical trials to provide evidence of the benefits and risks of interventions from a patient perspective and to inform regulatory decisions and health policy. The collection of PROs in routine practice can facilitate monitoring of patient symptoms; identification of unmet needs; prioritisation and/or tailoring of treatment to the needs of individual patients and inform value-based healthcare initiatives. However, respondent burden needs to be carefully considered and addressed to avoid high rates of missing data and poor reporting of PRO results, which may lead to poor quality data for regulatory decision making and/or clinical care.

Responsiveness and interpretation of the PROMIS Cancer Function Brief 3D Profile

BACKGROUND

Measuring function with valid and responsive tools in patients with cancer is essential for driving clinical decision-making and for the end points of clinical trials. Current patient-reported outcome measurements of function fall short for many reasons. This study evaluates the responsiveness of the Patient-Reported Outcomes Measurement Information System (PROMIS) Cancer Function Brief 3D Profile, a novel measure of function across multiple domains.

METHODS

Two hundred nine participants across five geographically distinct tertiary care centers completed the assessment and pain rating at two outpatient cancer rehabilitation clinic visits. Patients and providers completed a global rating of change measure at the second visit to indicate whether the patient was improving or worsening in function. Multiple response indices and linear models measured whether the measure was responsive to self-reported and clinician-rated changes over time. Correlations between changes in function and changes in anchors (pain rating and performance status) were also calculated.

RESULTS

Function as measured by the PROMIS Cancer Function Brief 3D Profile changed appropriately as both patients and clinicians rated change. Small to moderate effect sizes supported the tool's responsiveness. Function was moderately correlated with pain and more strongly correlated with performance status, and changes in function corresponded with changes in anchor variables. No floor/ceiling effect was found.

CONCLUSIONS

The PROMIS Cancer Function Brief 3D Profile is sensitive to changes over time in patients with cancer. The measure may be useful in clinical practice and as an end point in clinical trials.

LAY SUMMARY

We gave patients a questionnaire by which they told their physicians how well they were functioning, including how fatigued they were. This study tested that questionnaire to see whether the scores would change if patients got better or worse.

Patient-Reported Outcome Measures: An Exploration of Their Utility in Functional Assessment and Rehabilitation

Patient reported outcome measures (PROMs) have had a place in medicine for many years. However, their importance has grown rapidly in recent years due to a number of factors-not the least of which is an increased recognition of importance of the patient’ assessment of their care and outcomes. Despite their growing use, few of us have a good understanding of when they should be favored over alternative measures, how they are developed, the benefits of computerized administration, or how to leverage their incorporation into the electronic health record (EHR) and clinical decision making. This supplement is not intended to cover all aspects of the use of PROMs in clinical and rehabilitation care. Rather it’s goal is to provide an overview of their use and development, as well as the clinical relevance of their increasingly electronic, computerized, and EHR-based administration. It’s papers also highlight how advances in computing capabilities have permitted the use of newer modeling capabilities, such as multidimensional item response theory, to radically expand the capabilities of computerized PROMs.

The supplement emphasizes functional PROMs, however the described methods are easily applicable to other domains. More specially, its papers range from an introduction in the form of a historical overview of PROMs in rehabilitation medicine to the details of novel computerized approaches to improve their efficiency and capacity for individualized, clinically actionable assessments. In summary, we do hope that, whether you are a clinician, administrator, or psychometrician that you will find something of interest in this intriguing and increasingly important, aspect of medicine and rehabilitation.

Multidimensional Computerized Adaptive Testing: A Potential Path Toward the Efficient and Precise Assessment of Applied Cognition, Daily Activity, and Mobility for Hospitalized Patients

OBJECTIVE

To develop and evaluate an efficient and precise variable-length functional assessment of applied cognition, daily activity, and mobility to inform mobility preservation and rehabilitation service delivery among hospitalized patients.

DESIGN

A multidimensional item bank tapping into these dimensions was developed, with all items calibrated using a multidimensional graded response model. The items were adaptively selected from the item banks to maximize the test information, and the test ended when a joint stopping rule was satisfied. A simulation study was conducted based on the completed instrument, the Functional Assessment in Acute Care Multidimensional Computerized Adaptive Test (FAMCAT), to compare its measurement precision and efficiency capabilities relative to conventional unidimensional computerized adaptive testing. Precision was measured by the bias and root mean squared error between the estimated and true (ie, simulated) θ estimates, whereas efficiency was measured by average test length. Data were collected by an interviewer reading questions from a tablet computer and entering patients' responses.

SETTING

A large Midwestern hospital.

PARTICIPANTS

A total of 4143 patients hospitalized with medical diagnosis and/or surgical complications, with 2060 in the calibration sample and 2083 in the validation cohort.

INTERVENTION

Not applicable.

RESULTS

Among the 2083 patients in the validation sample, FAMCAT administration required an average of 6 (SD=3.11) minutes. Ninety-six percent had their tests terminated by the standard error rule after responding to an average of 22.05 (SD=7.98) items, whereas 15 were terminated by the change in θ rule, with an average test length of 45.27 (SD=11.49). The remaining 76 responded until reaching the maximum test length of 60 items.

CONCLUSIONS

The FAMCAT has the potential to satisfy the need for structured, frequent, and precise assessment of functional domains among hospitalized patients with medical diagnosis and/or surgical complications. The results are promising and may be informative for others who wish to develop similar instruments when concurrent assessment of correlated domains is required.

Correlation and Crosswalks between Patient-Reported Functional Outcomes and PROMIS Physical Function among Medically Ill Patients

OBJECTIVES

To develop scoring crosswalks between a new multi-domain patient-reported outcome measure, the Functional Assessment in Acute Care (FAMCAT) with a PROMIS measure of physical function and examine correlations with existing legacy instruments.

DESIGN

Cross-sectional, single-group design study Setting: Large Midwestern academic teaching hospital Participants: A sample of 1,885 patients (53% men, average age 62 (SD 16)) hospitalized on the general medical services between May 2016 - June 2017.

INTERVENTION

Not applicable.

MAIN OUTCOME MEASURES

Scores from the FAMCAT administered via computerized adaptive testing were compared to scores on the 8-item PROMIS Physical Function short form.

RESULTS

Correlations with the FAMCAT and the PROMIS PF were strong for initial scores (MCAT_Mobilty r=0.78, p<.0001; MCAT_DailyAct r=0.81, p<.0001). The Applied Cognition scale did not demonstrate adequate correlations, thus was not a candidate for crosswalk scores. While the MCAT_Mobility scale could be initially linked, subsequent analysis did not support a valid crosswalk. Linking criteria were applied with the Daily Activity scale to developing a final concordance table.

CONCLUSIONS

The FAMCAT Daily Activity scale yielded robust correlations to develop crosswalk scores with the PROMIS PF. The resulting crosswalk conversion metric may be useful to compare outcomes across these constructs for assessing functional abilities among patients on general medical services. The Applied Cognition and Basic Mobility scales did not meet criteria; therefore, alternate legacy instruments are needed to develop additional crosswalks.

Identifying Clinically Relevant Functional Strata to Direct Mobility Preservation among Patients Hospitalized with Medical Conditions

OBJECTIVE

To develop a system to guide interpretation of scores generated from the newly developed item response theory (IRT)-based, multi-dimensional computer adaptive test (FAMCAT) which assesses three important domains of function: mobility, daily activities, and applied cognition.

DESIGN

Cross-sectional data used to inform a modified-Delphi approach to establish FAMCAT cut points delineating various functional ability levels.

SETTING

Large Midwestern academic teaching hospital PARTICIPANTS: 2049 patients hospitalized to an inpatient medical service INTERVENTION: Not applicable.

MAIN OUTCOME MEASURES

FAMCAT Basic Mobility, Daily Activity, and Applied Cognition scores RESULTS: IRT-based score estimation data was successfully integrated with expert clinical feedback using a modified-Delphi process to arrive at consensus yielding 4 functional level strata (ranging from bed-based mobility to independent mobility) for both the FAMCAT Mobility and Daily Activity scales; 1 cut-point was supported to delineate two functional strata for Applied Cognition.

CONCLUSIONS

Meaningful cut-points were established for each FAMCAT domain using a data-informed, modified-Delphi process for achieving consensus. The resulting FAMCAT interpretation guide may be used to develop an ability-matched mobility preservation program and identify patients who may require a higher level of supervision based on the resulting FAMCAT scores.