It Is All About Value Now: The Data You Need to Collect and How to Do It: AOA Critical Issues

Collection of Patient-Reported Outcome Measures: Comparing Paper to Online Data Collection Among Knee Arthroplasty Patients

BACKGROUND

At our institution, patient-reported outcome measures were completed on paper forms until 2021, when we began sending emails to knee arthroplasty patients so they could complete surveys electronically. This study evaluated our transition from paper-based to electronic collection of the Knee injury and Osteoarthritis Outcome Score for Joint Replacement (KOOS JR).

METHODS

We compared 276 knee arthroplasty procedures performed from March 2020 through June 2020 that were eligible to complete paper KOOS JR surveys with 490 knee arthroplasty procedures performed from March 2022 through June 2022 that were eligible to complete electronic surveys. Survey completion rates at preoperative and 1-year follow-up were evaluated as well as the relative frequency of patients achieving a substantial clinical benefit (SCB) KOOS JR score increase of 20 points or more. Multivariate regressions were used to assess the potential influence of covariates, including age at surgery, sex, body mass index, type of insurance, surgery site, and the distance patients traveled to our institution.

RESULTS

Response rates for preoperative surveys completed within 90 days of surgery increased from 53% (146 of 276) with paper to 83% (406 of 490) with electronic surveys, while 1-year follow-up response rates improved from 38% (105 of 276) to 65% (320 of 490). Multivariate analyses indicated that only the survey type (paper or electronic) was associated with response rates. Electronic data collection also reduced incomplete (13 to 0.4%) and unnecessary (38 to 0.4%) surveys. The annual cost of data collection decreased from $140,696 with paper-based forms to $105,742 with electronic surveys. However, patients achieving a SCB declined from 81% (42 of 52) with paper to 64% (176 of 276) with electronic surveys (P = 0.02).

CONCLUSIONS

Compared to paper forms, electronic data collection at our institution increased follow-up rates and improved data quality at lower costs, but the relative frequency of patients reporting a SCB decreased.

Enhancing the Evaluation of Physical Function Following Orthopaedic Trauma Care: Comparison of PROMIS Computerized Adaptive Testing and Short Musculoskeletal Function Assessment

OBJECTIVES

To compare measurement properties of Patient-Reported Outcomes Measurement Information System (PROMIS) (physical function [PF] and pain interference [PI]) computerized adaptive testing to traditional Short Musculoskeletal Function Assessment (SMFA) (dysfunction index [DI] and bother index [BI]). To explore factors associated with PROMIS scores.

METHODS

DESIGN

Cross-sectional study.

SETTING

Level I Trauma Center.

PATIENT SELECTION CRITERIA

Isolated upper/lower extremity fracture patients were recruited from the orthopaedic trauma outpatient clinic (October 1, 2021 to January 1, 2023).

OUTCOME MEASURES

Correlations (Pearson), reliability (standard error [SE] [T score]), efficiency (amount of information per item [1 - SE2/Nitems]), and floor/ceiling effects were assessed. An r > 0.7 represented high correlation, and SE ≤ 2.2 represented sufficient reliability. Factors associated with worse PROMIS scores were also identified.

RESULTS

In total, 202 patients completed PROMs at median 98 days follow-up. Correlations between PROMIS-PF and SMFA-DI, and PROMIS-PI and SMFA-BI were -0.84 and 0.65. Reliability was very high for both instruments (mean SE 2.0 [PROMIS-PF], SE 2.1 [PROMIS-PI], and SE 1.2 [SMFA-DI], SE 1.8 [SMFA-BI]). Relative efficiency for PROMIS-PF versus SMFA-DI, and PROMIS-PI versus SMFA-BI was 7.8 (SD 2.5) and 4.1 (SD 1.7), respectively. Neither PROMIS nor SMFA exhibited floor/ceiling effects. In the multivariable regression analyses, elevated levels of depression, among other factors, showed an (independent) association with worse PROMIS-PF and PROMIS-PI scores.

CONCLUSIONS

PROMIS-PF and PROMIS-PI CATs showed a (high and moderate) correlation with SMFA and hence measure a comparable construct of physical function and discomfort. As computerized adaptive tests are much more efficient to administer, they present a compelling alternative to SMFA for evaluating impact of fracture treatment. The relation between symptoms of depression and PROMIS scores emphasizes the importance of psychosocial aspects of health in orthopaedic trauma patients.

LEVEL OF EVIDENCE

Prognostic Level III. See Instructions for Authors for a complete description of levels of evidence.

Defining Patient-relevant Thresholds and Change Scores for the HOOS JR and KOOS JR Anchored on the Patient-acceptable Symptom State Question

BACKGROUND

When evaluating the results of clinical research studies, readers need to know that patients perceive effect sizes, not p values. Knowing the minimum clinically important difference (MCID) and the patient-acceptable symptom state (PASS) threshold for patient-reported outcome measures helps us to ascertain whether our interventions result in improvements that are large enough for patients to care about, and whether our treatments alleviate patient symptoms sufficiently. Prior studies have developed the MCID and PASS threshold for the Hip Disability and Osteoarthritis Outcome Score for Joint Replacement (HOOS JR) and Knee Injury and Osteoarthritis Outcome Score for Joint Replacement (KOOS JR) anchored on satisfaction with surgery, but to our knowledge, neither the MCID nor the PASS thresholds for these instruments anchored on a single-item PASS question have been described.

QUESTIONS/PURPOSES

(1) What are the MCID (defined here as the HOOS/KOOS JR change score associated with achieving PASS) and PASS threshold for the HOOS JR and KOOS JR anchored on patient responses to the single-item PASS instrument? (2) How do patient demographic factors such as age, gender, and BMI correlate with MCID and PASS thresholds using the single-item PASS instrument?

METHODS

Between July 2020 and September 2021, a total of 10,970 patients underwent one primary unilateral THA or TKA and completed at least one of the three surveys (preoperative HOOS or KOOS JR, 1-year postoperative HOOS or KOOS JR, and 1-year postoperative single-item anchor) at one large, academic medical center. Of those, only patients with data for all three surveys were eligible, leaving 13% (1465 total; 783 THAs and 682 TKAs) for analysis. Despite this low percentage, the overall sample size was large, and there was little difference between completers and noncompleters in terms of demographics or baseline patient-reported outcome measure scores. Patients undergoing bilateral total joint arthroplasty or revision total joint arthroplasty and those without all three surveys at 1 year of follow-up were excluded. A receiver operating characteristic curve analysis, leveraging a 1-year, single-item PASS (that is, "Do you consider that your current state is satisfactory?" with possible answers of "yes" or "no") as the anchor was then used to establish the MCID and PASS thresholds among the 783 included patients who underwent primary unilateral THA and 682 patients who underwent primary unilateral TKA. We also explored the associations of age at the time of surgery (younger than 65 years or 65 years and older), gender (men or women), BMI (< 30 or ≥ 30 kg/m 2 ), and baseline Patient-Reported Outcome Measure Information System-10 physical and mental component scores (< 50 or ≥ 50) for each of the MCID and PASS thresholds through stratified analyses.

RESULTS

For the HOOS JR, the MCID associated with the PASS was 23 (95% CI 18 to 31), with an area under the receiver operating characteristic curve of 0.75, and the PASS threshold was 81 (95% CI 77 to 85), with an area under the receiver operating characteristic curve of 0.81. For the KOOS JR, the MCID was 16 (95% CI 14 to 18), with an area under the receiver operating characteristic curve of 0.75, and the PASS threshold was 71 (95% CI 66 to 73) with an area under the receiver operating characteristic curve of 0.84. Stratified analyses indicated higher change scores and PASS threshold for younger men undergoing THA and higher PASS thresholds for older women undergoing TKA.

CONCLUSION

Here, we demonstrated the utility of a single patient-centered anchor question, raising the question as to whether simply collecting a postoperative PASS is an easier way to measure success than collecting preoperative and postoperative patient-reported outcome measures and then calculating MCIDs and the substantial clinical benefit.

LEVEL OF EVIDENCE

Level III, therapeutic study.

Younger Age, Capsular Repair, and Larger Preoperative Alpha Angles Are Associated With Earlier Achievement of Clinically Meaningful Improvement After Hip Arthroscopy for Femoroacetabular Impingement Syndrome

PURPOSE

The purpose of the study was to analyze demographic, radiographic, and intraoperative factors that influence the time to achieve the minimal clinically important difference (MCID) and maximum outcome improvement satisfaction threshold (MOIT) after primary hip arthroscopy for femoroacetabular impingement syndrome (FAIS) and labral tear.

METHODS

Included patients had undergone hip arthroscopy with labral repair or reconstruction for FAIS with labral tear between February 2008 and October 2018. Patients were excluded if they had a prior ipsilateral hip surgery, prior hip conditions, a Tonnis grade > 1, or were unwilling to participate. Multiple demographic, radiographic, and intraoperative variables were collected. The modified Harris Hip Score (mHHS) and Non-Arthritic Hip Score (NAHS) were collected before surgery and at 3 months, 1 year, and 2 years after surgery. The MCID and MOIT for the mHHS and NAHS were either calculated or determined through previously published values. A time-to-event analysis was performed to determine variables predictive of early or delayed achievement of MCID or MOIT. Early achievement was defined as achieving MCID or MOIT at the 3-month timepoint.

RESULTS

Six hundred thirty-two hips (632 patients) were included. Of those that achieved MCID and MOIT, 428 (73.0%) and 414 (73.0%) patients achieved MCID and 253 (47.9%) and 264 (52.5%) patients achieved MOIT by 3 months after surgery for mHHS and NAHS, respectively. Younger age, capsular repair, and increasing alpha angle were associated with earlier achievement for either MCID or MOIT. Increasing age, worker's compensation claims, and higher baseline patient-reported outcome measure scores were associated with delayed achievement for either MCID or MOIT.

CONCLUSIONS

Most of the patients who achieved MCID and MOIT for mHHS and NAHS did so by 3 months after surgery. Younger age, capsular repair, and increasing alpha angle were associated with earlier achievement of MCID and MOIT after hip arthroscopy.

LEVEL OF EVIDENCE

Level IV, case series.

Substantial Inconsistency and Variability Exists Among Minimum Clinically Important Differences for Shoulder Arthroplasty Outcomes: A Systematic Review

BACKGROUND

As the value of patient-reported outcomes becomes increasingly recognized, minimum clinically important difference (MCID) thresholds have seen greater use in shoulder arthroplasty. However, MCIDs are unique to certain populations, and variation in the modes of calculation in this field may be of concern. With the growing utilization of MCIDs within the field and value-based care models, a detailed appraisal of the appropriateness of MCID use in the literature is necessary and has not been systematically reviewed.

QUESTIONS/PURPOSES

We performed a systematic review of MCID quantification in existing studies on shoulder arthroplasty to answer the following questions: (1) What is the range of values reported for the MCID in commonly used shoulder arthroplasty patient-reported outcome measures (PROMs)? (2) What percentage of studies use previously existing MCIDs versus calculating a new MCID? (3) What techniques for calculating the MCID were used in studies where a new MCID was calculated?

METHODS

The Embase, PubMed, and Ovid/MEDLINE databases were queried from December 2008 through December 2020 for total shoulder arthroplasty and reverse total shoulder arthroplasty articles reporting an MCID value for various PROMs. Two reviewers (DAK, MAM) independently screened articles for eligibility, specifically identifying articles that reported MCID values for PROMs after shoulder arthroplasty, and extracted data for analysis. Each study was classified into two categories: those referencing a previously defined MCID and those using a newly calculated MCID. Methods for determining the MCID for each study and the variability of reported MCIDs for each PROM were recorded. The number of patients, age, gender, BMI, length of follow-up, surgical indications, and surgical type were extracted for each article. Forty-three articles (16,408 patients) with a mean (range) follow-up of 20 months (0.75 to 68) met the inclusion criteria. The median (range) BMI of patients was 29.3 kg/m2 (28.0 to 32.2 kg/m2), and the median (range) age was 68 years (53 to 84). There were 17 unique PROMs with MCID values. Of the 112 MCIDs reported, the most common PROMs with MCIDs were the American Shoulder and Elbow Surgeons (ASES) (23% [26 of 112]), the Simple Shoulder Test (SST) (17% [19 of 112]), and the Constant (15% [17 of 112]).

RESULTS

The ranges of MCID values for each PROM varied widely (ASES: 6.3 to 29.5; SST: 1.4 to 4.0; Constant: -0.3 to 12.8). Fifty-six percent (24 of 43) of studies used previously established MCIDs, with 46% (11 of 24) citing one study. Forty-four percent (19 of 43) of studies established new MCIDs, and the most common technique was anchor-based (37% [7 of 19]), followed by distribution (21% [4 of 19]).

CONCLUSION

There is substantial inconsistency and variability in the quantification and reporting of MCID values in shoulder arthroplasty studies. Many shoulder arthroplasty studies apply previously published MCID values with variable ranges of follow-up rather than calculating population-specific thresholds. The use of previously calculated MCIDs may be acceptable in specific situations; however, investigators should select an anchor-based MCID calculated from a patient population as similar as possible to their own. This practice is preferable to the use of distribution-approach MCID methods. Alternatively, authors may consider using substantial clinical benefit or patient-acceptable symptom state to assess outcomes after shoulder arthroplasty.

CLINICAL RELEVANCE

Although MCIDs may provide a useful effect-size based alternative to the traditional p value, care must be taken to use an MCID that is appropriate for the particular patient population being studied.

Defining the Patient Acceptable Symptom State for the HOOS JR and KOOS JR After Primary Total Joint Arthroplasty

BACKGROUND

It is essential to quantify an acceptable outcome after total joint arthroplasty (TJA) in order to understand quality of care. The purpose of this study was to define patient acceptable symptom state (PASS) thresholds for the Knee injury and Osteoarthritis Outcome Score, Joint Replacement (KOOS JR) and the Hip disability and Osteoarthritis Outcome Score, Joint Replacement (HOOS JR) after TJA.

METHODS

A receiver operating characteristic (ROC) curve analysis, leveraging 2-year satisfaction of "moderate improvement" or better as the anchor, was used to establish PASS thresholds among 5,216 patients who underwent primary total hip arthroplasty and 4,036 who underwent primary total knee arthroplasty from 2007 to 2012 with use of an institutional registry. Changes in PASS thresholds were explored by stratifying and recalculating these thresholds by age at the time of surgery (<70 or ≥70 years of age), sex (men or women), body mass index (BMI; <30 or ≥30 kg/m2), and baseline Short Form-36 (SF-36) physical and mental component scores (<50 or ≥50).

RESULTS

The HOOS JR PASS threshold was 76.7 (area under the ROC curve [AUC] = 0.91), which was achieved by 4,334 patients (83.1%). The KOOS JR PASS threshold was 63.7 (AUC = 0.89), which was achieved by 3,461 patients (85.8%). Covariate stratification demonstrated that PASS thresholds were higher in men compared with women, and in those with higher preoperative SF-36 physical and mental scores (≥50) compared with lower SF-36 scores (<50). Results differed between instruments for BMI and age: higher BMI was associated with a lower PASS threshold for the HOOS JR but a higher PASS threshold for the KOOS JR. The HOOS JR PASS threshold was higher in patients who were <70 years of age compared with those who were ≥70 years of age, but was equivalent for the KOOS JR.

CONCLUSIONS

The PASS thresholds for the HOOS JR and KOOS JR at 2 years after TJA were 76.7 and 63.7, respectively. The PASS thresholds were associated with certain preoperative covariates, suggesting that an acceptable symptom state after TJA is influenced by patient-specific factors.

LEVEL OF EVIDENCE

Prognostic Level IV. See Instructions for Authors for a complete description of levels of evidence.

Large Heterogeneity Among Minimal Clinically Important Differences for Hip Arthroscopy Outcomes: A Systematic Review of Reporting Trends and Quantification Methods

PURPOSE

To perform a systematic review of reporting trends and quantification methods for the minimal clinically important difference (MCID) within the hip arthroscopy literature.

METHODS

Cochrane, PubMed, and OVID/MEDLINE databases were queried for hip arthroscopy articles that reported the MCID. Studies were classified as (1) calculating new MCID values for their specific study-population or (2) referencing previously established MCID values. Data pertaining to patient demographics, study characteristics, outcome measures, method of MCID quantification, MCID value, anchor questions, measurement error, and study from which referenced MCID values were obtained were extracted.

RESULTS

A total of 59 articles with 18,830 patients (19,867 hips) was included. A total of 19 unique outcome measures was reported. A total of 33 (n = 55.9%) studies (follow-up range 6-60 months) used previously established MCID values to assess their study population (MCID values established at a follow-up range 6-31 months). The remaining 26 studies (44.1%) performed new MCID calculations. The MCID values were inconsistent and varied widely (Hip Outcome Score-Activities of Daily Living: 5.0-15.4; Hip Outcome Score-Sports Subscale: 6-25; modified Harris hip score: 2.4-20.9). Among the 33 studies that used previously established MCID values, 10 different studies were cited as the reference. Among the remaining 26 studies that calculated a new MCID value, the most common method was 0.5 standard deviation method (n = 21, 80.8%). Only 3 of 26 (11.5%) studies reported a measurement of error in conjunction with their MCID values.

CONCLUSIONS

Inconsistencies in MCID reporting and quantification methods led to a wide range of MCID values for commonly administered outcome measures within the hip arthroscopy literature-even for the same outcome measures. The majority of studies referenced previously established MCID values with variable ranges of follow-up and applied those values to assess their specific study population at varying follow-ups.

LEVEL OF EVIDENCE

IV, systematic review.