The reliability and validity of lung cancer and melanoma clinical quality survival measures

OBJECTIVE

To develop a risk adjustment approach and test reliability and validity for oncology survival measures.

DATA SOURCES AND STUDY SETTING

We used the National Cancer Institute's Surveillance, Epidemiology, and End Results (SEER)-Medicare data from 2010 to 2013, with mortality data through 2015.

STUDY DESIGN

We developed 2-year risk-standardized survival rates (RSSR) for melanoma, non-small cell lung cancer (NSCLC), and small cell lung cancer (SCLC). Patients were attributed to group practices based on the plurality of visits. We identified the risk-adjustment variables via bootstrap and calculated the RSSRs. Reliability was tested via three approaches: (1) signal-to-noise ratio (SNR) reliability, (2) split-half, and (3) test-retest using bootstrap. We tested known group validity by stage at diagnosis using Cohen's d.

DATA COLLECTION/EXTRACTION METHODS

We selected all patients enrolled in Medicare and linked to SEER during the measurement period with an incident first primary diagnosis of stage I-IV melanoma, NSCLC, or SCLC. We excluded patients with missing data on month and/or stage of diagnosis.

PRINCIPAL FINDINGS

Results are based on patients with melanoma (n = 4344); NSCLC (n = 16,080); and SCLC (n = 2807) diagnosed between 2012 and 2013. The median (interquartile range) for the RSSRs at the group practice-level were 0.89 (0.83-0.87) for melanoma, 0.37 (0.30-0.43) for NSCLC, and 0.19 (0.11-0.25) for SCLC. C-statistics for the models ranged from 0.725 to 0.825. The reliability varied by approach with median SNR 0.20, 0.25, and 0.13; median test-retest 0.59, 0.57, and 0.56; median split-half reliability 0.21, 0.29, and 0.29 for melanoma, NSCLC, and SCLC, respectively. Cohen's d for stage I-IIIa and IIIb+ was 1.27, 0.86, 0.60 for melanoma, NSCLC, and SCLC, respectively.

CONCLUSIONS

Our results suggest that these cancer survival measures demonstrated adequate test-retest reliability and expected findings for the known-group validity analysis. If data limitations and feasibility challenges can be addressed, implementation of these quality measures may provide a survival metric used for oncology quality improvement efforts.

Real-World Adherence to Patient-Reported Outcome Monitoring as a Cancer Care Quality Metric

PURPOSE:

Routine collection of patient-reported outcomes (PROs) for patients with advanced solid malignancies is an evidence-based practice and critical component of high-quality cancer care, but real-world adherence is poorly characterized. We sought to describe real-world adherence to PRO monitoring and its potential predictors.

METHODS:

We conducted a retrospective cross-sectional study using deidentified electronic health record data from a National Cancer Institute Cancer Center, encompassing one academic and two community sites. Participants included individuals with lung cancer receiving systemic therapy from January 1 to December 31, 2019. The primary outcome was patient-level adherence, defined as the proportion of treatment visits during which a PRO questionnaire (spanning symptoms, functional status, and global quality-of-life domains) was completed within 30 days. Practice-level performance was calculated as unadjusted mean patient-level adherence. We modeled patient-level adherence using multivariable ordinary least squares regression and identified covariates associated with adherence using a significance threshold of P < .05.

RESULTS:

In 2019, there were 18,604 encounters for 1,105 patients with lung cancer (mean [standard deviation] age 65.8 [10.2] years; 621 [56.2%] female; 216 [19.6%] Black) receiving systemic therapy. The mean patient-level PRO adherence ranged from 27.2% to 70.0% across sites and was 49.4% overall. Advanced age (≥ 65 years) and Black or African American race were negatively associated with PRO adherence ( P < .01).

CONCLUSION:

Across this real-world cohort of patients undergoing treatment for lung cancer, adherence to PRO monitoring lagged that achieved in seminal clinical trials, with potential age- and race-based disparities, demonstrating an implementation gap that could be addressed with standardized reporting of an adherence-based quality metric.

Practice-site-level measures of primary care comprehensiveness and their associations with patient outcomes

OBJECTIVES

To develop two practice-site-level measures of comprehensiveness and examine their associations with patient outcomes, and how their performance differs from physician-level measures.

DATA SOURCES

Medicare fee-for-service claims.

STUDY DESIGN

We calculated practice-site-level comprehensiveness measures (new problem management and involvement in patient conditions) across 5286 primary care physicians (PCPs) at 1339 practices in the Comprehensive Primary Care initiative evaluation in 2013. We assessed their associations with practices' attributed beneficiaries' 2014 total Medicare expenditures, hospitalization rates, ED visit rates. We also examined variation in PCPs' comprehensiveness across PCPs within practices versus between primary care practices. Finally, we compared associations of practice-site and PCP-level measures with outcomes.

PRINCIPAL FINDINGS

The measures had good variation across primary care practices, strong validity, and high reliability. Receiving primary care from a practice at the 75th versus 25th percentile on the involvement in patient conditions measure was associated with $21.93 (2.8%) lower total Medicare expenditures per beneficiary per month (P < .01). Receiving primary care from a practice at the 75th versus 25th percentile on the new problem management measure was associated with $14.77 (1.9%) lower total Medicare expenditures per beneficiary per month (P < .05); 8.84 (3.0%) fewer hospitalizations (P < .001), and 21.27 (3.1%) fewer ED visits per thousand beneficiaries per year (P < .01). PCP comprehensiveness varied more within than between practices.

CONCLUSIONS

More comprehensive primary care practices had lower Medicare FFS expenditures, hospitalization, and ED visit rates. Both PCP and practice-site level comprehensiveness measures had strong construct and predictive validity; PCP-level measures were more precise.

New approaches to measuring the comprehensiveness of primary care physicians

OBJECTIVE

To develop claims-based measures of comprehensiveness of primary care physicians (PCPs) and summarize their associations with health care utilization and cost.

DATA SOURCES AND STUDY SETTING

A total of 5359 PCPs caring for over 1 million Medicare fee-for-service beneficiaries from 1404 practices.

STUDY DESIGN

We developed Medicare claims-based measures of physician comprehensiveness (involvement in patient conditions and new problem management) and used a previously developed range of services measure. We analyzed the association of PCPs' comprehensiveness in 2013 with their beneficiaries' emergency department, hospitalizations rates, and ambulatory care-sensitive condition (ACSC) admissions (each per 1000 beneficiaries per year), and Medicare expenditures (per beneficiary per month) in 2014, adjusting for beneficiary, physician, practice, and market characteristics, and clustering.

PRINCIPAL FINDINGS

Each measure varied across PCPs and had low correlation with the other measures-as intended, they capture different aspects of comprehensiveness. For patients whose PCPs' comprehensiveness score was at the 75th vs 25th percentile (more vs less comprehensive), patients had lower service use (P < 0.05) in one or more measures: involvement with patient conditions: total Medicare expenditures, -$17.4 (-2.2 percent); hospitalizations, -5.5 (-1.9 percent); emergency department (ED) visits, -16.3 (-2.4 percent); new problem management: total Medicare expenditures, -$13.3 (-1.7 percent); hospitalizations, -7.0 (-2.4 percent); ED visits, -19.7 (-2.9 percent); range of services: ED visits, -17.1 (-2.5 percent). There were no significant associations between the comprehensiveness measures and ACSC admission rates.

CONCLUSIONS

These measures demonstrate strong content and predictive validity and reliability. Medicare beneficiaries of PCPs providing more comprehensive care had lower hospitalization rates, ED visits, and total Medicare expenditures.

Development and Validation of the Modified Patient-Centered Medical Home Assessment for the Comprehensive Primary Care Initiative

OBJECTIVE

To describe the modified Patient-Centered Medical Home Assessment (M-PCMH-A) survey module developed to track primary care practices' care delivery approaches over time, assess whether its underlying factor structure is reliable, and produce factor scores that provide a more reliable summary measure of the practice's care delivery than would a simple average of question responses.

DATA SOURCES/STUDY SETTING

Survey data collected from diverse practices participating in the Comprehensive Primary Care (CPC) initiative in 2012 (n = 497) and 2014 (n = 493) and matched comparison practices in 2014 (n = 423).

STUDY DESIGN

Confirmatory factor analysis.

DATA COLLECTION

Thirty-eight questions organized in six domains: Access and Continuity of Care, Planned Care for Chronic Conditions and Preventive Care, Risk-Stratified Care Management, Patient and Caregiver Engagement, Coordination of Care across the Medical Neighborhood, and Continuous Data-Driven Improvement.

PRINCIPAL FINDINGS

Confirmatory factor analysis suggested using seven factors (splitting one domain into two), reassigning two questions to different domain factors, and removing one question, resulting in high reliability, construct validity, and stability in all but one factor. The seven factors together formed a single higher-order factor summary measure. Factor scores guard against potential biases from equal weighting.

CONCLUSIONS

The M-PCMH-A can validly and reliably track primary care delivery across practices and over time using factors representing seven key components of care as well as an overall score. Researchers should calculate factor loadings for their specific data if possible, but average scores may be suitable if they cannot use factor analysis due to resource or sample constraints.