Analysis of change in patient-reported outcome measures with floor and ceiling effects using the multilevel Tobit model: a simulation study and an example from a National Joint Register using body mass index and the Oxford Hip Score

Do the Revision Rates of Arthroplasty Surgeons Correlate With Postoperative Patient-reported Outcome Measure Scores? A Study From the Australian Orthopaedic Association National Joint Replacement Registry

BACKGROUND

Patient-reported outcome measures (PROMs) are a pragmatic and efficient means to evaluate the functional quality of arthroplasty beyond revision rates, which are used by most joint replacement registries to judge success. The relationship between these two measures of quality-revision rates and PROMs-is unknown, and not every procedure with a poor functional result is revised. It is logical-although still untested-that higher cumulative revision rates correlate inversely with PROMs for individual surgeons; more revisions are associated with lower PROM scores.

QUESTIONS/PURPOSES

We used data from a large national joint replacement registry to ask: (1) Does a surgeon's early THA cumulative percent revision (CPR) rate and (2) early TKA CPR rate correlate with the postoperative PROMs of patients undergoing primary THA and TKA, respectively, who have not undergone revision?

METHODS

Elective primary THA and TKA procedures in patients with a primary diagnosis of osteoarthritis that were performed between August 2018 and December 2020 and registered in the Australian Orthopaedic Association National Joint Replacement Registry PROMs program were eligible. THAs and TKAs were eligible for inclusion in the primary analysis if 6-month postoperative PROMs were available, the operating surgeon was clearly identified, and the surgeon had performed at least 50 primary THAs or TKAs. Based on the inclusion criteria, 17,668 THAs were performed at eligible sites. We excluded 8878 procedures that were not matched to the PROMs program, leaving 8790 procedures. A further 790 were excluded because they were performed by unknown or ineligible surgeons or were revised, leaving 8000 procedures performed by 235 eligible surgeons, including 4256 (53%; 3744 cases of missing data) patients who had postoperative Oxford Hip Scores and 4242 (53%; 3758 cases of missing data) patients who had a postoperative EQ-VAS score recorded. Complete covariate data were available for 3939 procedures for the Oxford Hip Score and for 3941 procedures for the EQ-VAS. A total of 26,624 TKAs were performed at eligible sites. We excluded 12,685 procedures that were not matched to the PROMs program, leaving 13,939 procedures. A further 920 were excluded because they were performed by unknown or ineligible surgeons, or because they were revisions, leaving 13,019 procedures performed by 276 eligible surgeons, including 6730 (52%; 6289 cases of missing data) patients who had had postoperative Oxford Knee Scores and 6728 (52%; 6291 cases of missing data) patients who had a postoperative EQ-VAS score recorded. Complete covariate data were available for 6228 procedures for the Oxford Knee Score and for 6241 procedures for the EQ-VAS. The Spearman correlation between the operating surgeon's 2-year CPR and 6-month postoperative EQ-VAS Health and Oxford Hip or Oxford Knee Score was evaluated for THA and TKA procedures where a revision had not been performed. Associations between postoperative Oxford and EQ-VAS scores and a surgeon's 2-year CPR were estimated based on multivariate Tobit regressions and a cumulative link model with a probit link, adjusting for patient age, gender, ASA score, BMI category, preoperative PROMs, as well as surgical approach for THA. Missing data were accounted for using multiple imputation, with models assuming they were missing at random and a worst-case scenario.

RESULTS

Of the eligible THA procedures, postoperative Oxford Hip Score and surgeon 2-year CPR were correlated so weakly as to be clinically irrelevant (Spearman correlation ρ = -0.09; p < 0.001), and the correlation with postoperative EQ-VAS was close to zero (ρ = -0.02; p = 0.25). Of the eligible TKA procedures, postoperative Oxford Knee Score and EQ-VAS and surgeon 2-year CPR were correlated so weakly as to be clinically irrelevant (ρ = -0.04; p = 0.004 and ρ = 0.03; p = 0.006, respectively). All models accounting for missing data found the same result.

CONCLUSION

A surgeon's 2-year CPR did not exhibit a clinically relevant correlation with PROMs after THA or TKA, and all surgeons had similar postoperative Oxford scores. PROMs, revision rates, or both may be inaccurate or imperfect indicators of successful arthroplasty. Missing data may limit the findings of this study, although the results were consistent under a variety of different missing data scenarios. Innumerable factors contribute to arthroplasty results, including patient-related variables, differences in implant design, and the technical quality of the procedure. PROMs and revision rates may be analyzing two different facets of function after arthroplasty. Although surgeon variables are associated with revision rates, patient factors may exert a stronger influence on functional outcomes. Future research should identify variables that correlate with functional outcome. Additionally, given the gross level of function that Oxford scores record, outcome measures that can identify clinically meaningful functional differences are required. The use of Oxford scores in national arthroplasty registries may rightfully be questioned.

LEVEL OF EVIDENCE

Level III, therapeutic study.

How do Patient-reported Outcome Scores in International Hip and Knee Arthroplasty Registries Compare?

BACKGROUND

Patient-reported outcome measures (PROMs) are the only systematic approach through which the patient's perspective can be considered by surgeons (in determining a procedure's efficacy or appropriateness) or healthcare systems (in the context of value-based healthcare). PROMs in registries enable international comparison of patient-centered outcomes after total joint arthroplasty, but the extent to which those scores may vary between different registry populations has not been clearly defined.

QUESTIONS/PURPOSES

(1) To what degree do mean change in general and joint-specific PROM scores vary across arthroplasty registries, and to what degree is the proportion of missing PROM scores in an individual registry associated with differences in the mean reported change scores? (2) Do PROM scores vary with patient BMI across registries? (3) Are comorbidity levels comparable across registries, and are they associated with differences in PROM scores?

METHODS

Thirteen national, regional, or institutional registries from nine countries reported aggregate PROM scores for patients who had completed PROMs preoperatively and 6 and/or 12 months postoperatively. The requested aggregate PROM scores were the EuroQol-5 Dimension Questionnaire (EQ-5D) index values, on which score 1 reflects "full health" and 0 reflects "as bad as death." Joint-specific PROMs were the Oxford Knee Score (OKS) and the Oxford Hip Score (OHS), with total scores ranging from 0 to 48 (worst-best), and the Hip Disability and Osteoarthritis Outcome Score-Physical Function shortform (HOOS-PS) and the Knee Injury and Osteoarthritis Outcome Score-Physical Function shortform (KOOS-PS) values, scored 0 to 100 (worst-best). Eligible patients underwent primary unilateral THA or TKA for osteoarthritis between 2016 and 2019. Registries were asked to exclude patients with subsequent revisions within their PROM collection period. Raw aggregated PROM scores and scores adjusted for age, gender, and baseline values were inspected descriptively. Across all registries and PROMs, the reported percentage of missing PROM data varied from 9% (119 of 1354) to 97% (5305 of 5445). We therefore graphically explored whether PROM scores were associated with the level of data completeness. For each PROM cohort, chi-square tests were performed for BMI distributions across registries and 12 predefined PROM strata (men versus women; age 20 to 64 years, 65 to 74 years, and older than 75 years; and high or low preoperative PROM scores). Comorbidity distributions were evaluated descriptively by comparing proportions with American Society of Anesthesiologists (ASA) physical status classification of 3 or higher across registries for each PROM cohort.

RESULTS

The mean improvement in EQ-5D index values (10 registries) ranged from 0.16 to 0.33 for hip registries and 0.12 to 0.25 for knee registries. The mean improvement in the OHS (seven registries) ranged from 18 to 24, and for the HOOS-PS (three registries) it ranged from 29 to 35. The mean improvement in the OKS (six registries) ranged from 15 to 20, and for the KOOS-PS (four registries) it ranged from 19 to 23. For all PROMs, variation was smaller when adjusting the scores for differences in age, gender, and baseline values. After we compared the registries, there did not seem to be any association between the level of missing PROM data and the mean change in PROM scores. The proportions of patients with BMI 30 kg/m 2 or higher ranged from 16% to 43% (11 hip registries) and from 35% to 62% (10 knee registries). Distributions of patients across six BMI categories differed across hip and knee registries. Further, for all PROMs, distributions also differed across 12 predefined PROM strata. For the EQ-5D, patients in the younger age groups (20 to 64 years and 65 to 74 years) had higher proportions of BMI measurements greater than 30 kg/m 2 than older patients, and patients with the lowest baseline scores had higher proportions of BMI measurements more than 30 kg/m 2 compared with patients with higher baseline scores. These associations were similar for the OHS and OKS cohorts. The proportions of patients with ASA Class at least 3 ranged across registries from 6% to 35% (eight hip registries) and from 9% to 42% (nine knee registries).

CONCLUSION

Improvements in PROM scores varied among international registries, which may be partially explained by differences in age, gender, and preoperative scores. Higher BMI tended to be associated with lower preoperative PROM scores across registries. Large variation in BMI and comorbidity distributions across registries suggest that future international studies should consider the effect of adjusting for these factors. Although we were not able to evaluate its effect specifically, missing PROM data is a recurring challenge for registries. Demonstrating generalizability of results and evaluating the degree of response bias is crucial in using registry-based PROMs data to evaluate differences in outcome. Comparability between registries in terms of specific PROMs collection, postoperative timepoints, and demographic factors to enable confounder adjustment is necessary to use comparison between registries to inform and improve arthroplasty care internationally.

LEVEL OF EVIDENCE

Level III, therapeutic study.

Collection and Reporting of Patient-reported Outcome Measures in Arthroplasty Registries: Multinational Survey and Recommendations

BACKGROUND

Patient-reported outcome measures (PROMs) are validated questionnaires that are completed by patients. Arthroplasty registries vary in PROM collection and use. Current information about registry collection and use of PROMs is important to help improve methods of PROM data analysis, reporting, comparison, and use toward improving clinical practice.

QUESTIONS/PURPOSES

To characterize PROM collection and use by registries, we asked: (1) What is the current practice of PROM collection by arthroplasty registries that are current or former members of the International Society of Arthroplasty Registries, and are there sufficient similarities in PROM collection between registries to enable useful international comparisons that could inform the improvement of arthroplasty care? (2) How do registries differ in PROM administration and demographic, clinical, and comorbidity index variables collected for case-mix adjustment in data analysis and reporting? (3) What quality assurance methods are used for PROMs, and how are PROM results reported and used by registries? (4) What recommendations to arthroplasty registries may improve PROM reporting and facilitate international comparisons?

METHODS

An electronic survey was developed with questions about registry structure and collection, analysis, reporting, and use of PROM data and distributed to directors or senior administrators of 39 arthroplasty registries that were current or former members of the International Society of Arthroplasty Registries. In all, 64% (25 of 39) of registries responded and completed the survey. Missing responses from incomplete surveys were captured by contacting the registries, and up to three reminder emails were sent to nonresponding registries. Recommendations about PROM collection were drafted, revised, and approved by the International Society of Arthroplasty Registries PROMs Working Group members.

RESULTS

Of the 25 registries that completed the survey, 15 collected generic PROMs, most frequently the EuroQol-5 Dimension survey; 16 collected joint-specific PROMs, most frequently the Knee Injury and Osteoarthritis Outcome Score and Hip Disability and Osteoarthritis Outcome Score; and 11 registries collected a satisfaction item. Most registries administered PROM questionnaires within 3 months before and 1 year after surgery. All 16 registries that collected PROM data collected patient age, sex or gender, BMI, indication for the primary arthroplasty, reason for revision arthroplasty, and a comorbidity index, most often the American Society of Anesthesiologists classification. All 16 registries performed regular auditing and reporting of data quality, and most registries reported PROM results to hospitals and linked PROM data to other data sets such as hospital, medication, billing, and emergency care databases. Recommendations for transparent reporting of PROMs were grouped into four categories: demographic and clinical, survey administration, data analysis, and results.

CONCLUSION

Although registries differed in PROM collection and use, there were sufficient similarities that may enable useful data comparisons. The International Society of Arthroplasty Registries PROMs Working Group recommendations identify issues that may be important to most registries such as the need to make decisions about survey times and collection methods, as well as how to select generic and joint-specific surveys, handle missing data and attrition, report data, and ensure representativeness of the sample.

CLINICAL RELEVANCE

By collecting PROMs, registries can provide patient-centered data to surgeons, hospitals, and national entities to improve arthroplasty care.