A conceptual framework for external validity

Computational Phenomapping of Randomized Clinical Trials to Enable Assessment of their Real-world Representativeness and Personalized Inference

Importance

Randomized clinical trials (RCTs) are the standard for defining an evidence-based approach to managing disease, but their generalizability to real-world patients remains challenging to quantify.

Objective

To develop a multidimensional patient variable mapping algorithm to quantify the similarity and representation of electronic health record (EHR) patients corresponding to an RCT and estimate the putative treatment effects in real-world settings based on individual treatment effects observed in an RCT.

Design

A retrospective analysis of the Treatment of Preserved Cardiac Function Heart Failure with an Aldosterone Antagonist Trial (TOPCAT; 2006-2012) and a multi-hospital patient cohort from the electronic health record (EHR) in the Yale New Haven Hospital System (YNHHS; 2015-2023).

Setting

A multicenter international RCT (TOPCAT) and multi-hospital patient cohort (YNHHS).

Participants

All TOPCAT participants and patients with heart failure with preserved ejection fraction (HFpEF) and ≥1 hospitalization within YNHHS.

Exposures

63 pre-randomization characteristics measured across the TOPCAT and YNNHS cohorts.

Main Outcomes and Measures

Real-world generalizability of the RCT TOPCAT using a multidimensional phenotypic distance metric between TOPCAT and YNHHS cohorts. Estimation of the individualized treatment effect of spironolactone use on all-cause mortality within the YNHHS cohort based on phenotypic distance from the TOPCAT cohort.

Results

There were 3,445 patients in TOPCAT and 11,712 HFpEF patients across five hospital sites. Across the 63 TOPCAT variables mapped by clinicians to the EHR, there were larger differences between TOPCAT and each of the 5 EHR sites (median SMD 0.200, IQR 0.037-0.410) than between the 5 EHR sites (median SMD 0.062, IQR 0.010-0.130). The synthesis of these differences across covariates using our multidimensional similarity score also suggested substantial phenotypic dissimilarity between the TOPCAT and EHR cohorts. By phenotypic distance, a majority (55%) of TOPCAT participants were closer to each other than any individual EHR patient. Using a TOPCAT-derived model of individualized treatment benefit from spironolactone, those predicted to derive benefit and receiving spironolactone in the EHR cohorts had substantially better outcomes compared with predicted benefit and not receiving the medication (HR 0.74, 95% CI 0.62-0.89).

Conclusions and Relevance

We propose a novel approach to evaluating the real-world representativeness of RCT participants against corresponding patients in the EHR across the full multidimensional spectrum of the represented phenotypes. This enables the evaluation of the implications of RCTs for real-world patients.

KEY POINTS

Question: How can we examine the multi-dimensional generalizability of randomized clinical trials (RCT) to real-world patient populations?Findings: We demonstrate a novel phenotypic distance metric comparing an RCT to real-world populations in a large multicenter RCT of heart failure patients and the corresponding patients in multisite electronic health records (EHRs). Across 63 pre-randomization characteristics, pairwise assessments of members of the RCT and EHR cohorts were more discordant from each other than between members of the EHR cohort (median standardized mean difference 0.200 [0.037-0.410] vs 0.062 [0.010-0.130]), with a majority (55%) of RCT participants closer to each other than any individual EHR patient. The approach also enabled the quantification of expected real world outcomes based on effects observed in the RCT.Meaning: A multidimensional phenotypic distance metric quantifies the generalizability of RCTs to a given population while also offering an avenue to examine expected real-world patient outcomes based on treatment effects observed in the RCT.

A Novel Digital Twin Strategy to Examine the Implications of Randomized Control Trials for Real-World Populations

Randomized clinical trials (RCTs) are essential to guide medical practice; however, their generalizability to a given population is often uncertain. We developed a statistically informed Generative Adversarial Network (GAN) model, RCT-Twin-GAN, that leverages relationships between covariates and outcomes and generates a digital twin of an RCT (RCT-Twin) conditioned on covariate distributions from a second patient population. We used RCT-Twin-GAN to reproduce treatment effect outcomes of the Systolic Blood Pressure Intervention Trial (SPRINT) and the Action to Control Cardiovascular Risk in Diabetes (ACCORD) Blood Pressure Trial, which tested the same intervention but had different treatment effect results. To demonstrate treatment effect estimates of each RCT conditioned on the other RCT patient population, we evaluated the cardiovascular event-free survival of SPRINT digital twins conditioned on the ACCORD cohort and vice versa (SPRINT-conditioned ACCORD twins). The conditioned digital twins were balanced by the intervention arm (mean absolute standardized mean difference (MASMD) of covariates between treatment arms 0.019 (SD 0.018), and the conditioned covariates of the SPRINT-Twin on ACCORD were more similar to ACCORD than a sprint (MASMD 0.0082 SD 0.016 vs. 0.46 SD 0.20). Most importantly, across iterations, SPRINT conditioned ACCORD-Twin datasets reproduced the overall non-significant effect size seen in ACCORD (5-year cardiovascular outcome hazard ratio (95% confidence interval) of 0.88 (0.73-1.06) in ACCORD vs median 0.87 (0.68-1.13) in the SPRINT conditioned ACCORD-Twin), while the ACCORD conditioned SPRINT-Twins reproduced the significant effect size seen in SPRINT (0.75 (0.64-0.89) vs median 0.79 (0.72-0.86)) in ACCORD conditioned SPRINT-Twin). Finally, we describe the translation of this approach to real-world populations by conditioning the trials on an electronic health record population. Therefore, RCT-Twin-GAN simulates the direct translation of RCT-derived treatment effects across various patient populations with varying covariate distributions.

Reproductive and obstetrical outcomes after treatment of retained products of conception: hysteroscopic removal vs ultrasound-guided electric vacuum aspiration, a prospective follow-up study

BACKGROUND

Traditionally, curettage has been the most widely performed surgical intervention for removing retained products of conception. However, hysteroscopic removal is increasingly performed as an alternative because of the potentially lower risk of intrauterine adhesions and higher rates of complete removal. Until recently, studies comparing curettage with hysteroscopic removal regarding reproductive and obstetrical outcomes were limited, and data conflicting.

OBJECTIVE

This study aimed to assess reproductive and obstetrical outcomes in women wishing to conceive after removal of retained products of conception by hysteroscopy or ultrasound-guided electric vacuum aspiration.

STUDY DESIGN

This was a prospective long-term follow-up study, conducted in 3 teaching hospitals and 1 university hospital. Patients were included from April 2015 until June 2022 for follow-up, either in a randomized controlled, nonblinded trial on the risk of intrauterine adhesions after removal of retained products of conception, or in a cohort alongside the randomized trial. Women with an ultrasonographic image suggestive of retained products of conception ranging from 1 to 4 cm were eligible. Surgical procedures in the randomized controlled trial were hysteroscopic morcellation or ultrasound-guided electric vacuum aspiration. In the cohort study, hysteroscopic treatment included hysteroscopic morcellation or cold loop resection compared with ultrasound-guided electric vacuum aspiration.

RESULTS

A total of 261 out of 305 patients (85.6%) were available for follow-up after removal of retained products of conception, resulting in a cohort of 171 women after hysteroscopic removal and 90 women after removal by ultrasound-guided vacuum aspiration. Respectively, 92 of 171 women (53.8%) in the hysteroscopic removal group and 56 of 90 (62.2%) in the electric vacuum aspiration group wished to conceive (P=.192). Subsequent pregnancy rates were 88 of 91 (96.7%) after hysteroscopic removal and 52 of 56 (92.9%) after electric vacuum aspiration (P=.428). The live birth rates were 61 of 80 (76.3%) and 37 of 48 (77.1%) after hysteroscopic removal and electric vacuum aspiration, respectively (P=.914), with 8 of 88 pregnancies (9.1%) in the hysteroscopic removal group and 4 of 52 (7.7%) in the electric vacuum aspiration group still ongoing at follow-up (P=1.00). The median time to conception was 8.2 weeks (interquartile range, 5.0-17.2) in the hysteroscopic removal group and 6.9 weeks (interquartile range, 5.0-12.1) in the electric vacuum aspiration group (P=.262). The overall placental complication rate was 13 of 80 (16.3%) in the hysteroscopic removal group and 11 of 48 (22.9%) in the electric vacuum aspiration group (P=.350).

CONCLUSION

Hysteroscopic removal and ultrasound-guided electric vacuum aspiration of retained products of conception seem to have no significantly different effects on subsequent live birth rate, pregnancy rate, time to conception, or pregnancy complications. Reproductive and obstetrical outcomes after removal of retained products of conception are reassuring, albeit with a high risk of placental complications.

Hysteroscopic morcellation vs. curettage for removal of retained products of conception: a multicenter randomized controlled trial

OBJECTIVE

To study the comparison between hysteroscopic morcellation (HM) of retained products of conception (RPOC) with ultrasound (US)-guided electric vacuum aspiration in terms of intrauterine adhesion (IUA) formation, efficacy, and complications.

DESIGN

A randomized controlled, nonblinded trial.

SETTING

Three teaching hospitals and one university hospital from April 2015 to June 2022.

PATIENTS

A total of 133 women with RPOC on US, ranging from 1-4 cm, were randomized to receive either HM or electric vacuum aspiration.

INTERVENTION

Hysteroscopic morcellation was performed with the TruClear System (Medtronic, Minneapolis, MN, USA). Electric vacuum aspiration was performed using an 8- or 10-mm flexible plastic Karman cannula under US guidance. Women allocated to vacuum aspiration underwent the procedure as soon as possible.

MAIN OUTCOME MEASURES

In the HM group, an office diagnostic hysteroscopy was planned a minimum of 6 weeks after the end of pregnancy, followed by retained product of conception removal at least 8 weeks after the end of the pregnancy. Postoperatively, an office second-look hysteroscopy was scheduled to assess the primary outcome of IUAs.

RESULTS

Postoperative IUAs were seen in 14.3% (9/63) of patients in the HM group and 20.6% (13/64) of patients in the vacuum aspiration group (-6% [-19.1% to 7.1%]). Significantly more RPOC were removed completely by HM compared with vacuum aspiration (95.2% vs. 82.5% (-14% [-24.9% to -3.1%]), and additional operative hysteroscopy was less frequently necessary in the HM group (12.5%) compared with the vacuum aspiration group (31.3%) (-20.1% [-34.3% to -6%]). The median operating time was shorter for vacuum aspiration compared with HM (5.80 minutes vs. 7.15 minutes). No differences were observed between HM and vacuum aspiration for the occurrence of intraoperative or postoperative complications (5.5% vs. 5.0% and 2.7% vs. 1.3%, respectively).

CONCLUSION

In our randomized controlled trial, no significant differences were found in the occurrence of IUAs and complications. However, the RPOC were more often completely removed by HM than vacuum aspiration, and the HM group required fewer additional hysteroscopic treatments.

CLINICAL TRIAL REGISTRATION NUMBER

NTR4923 (https://trialsearch.who.int/Trial2.aspx?TrialID=NTR4923). Date of registration: November 23, 2014, date of initial participant enrollment: January 1, 2015.

A data-driven approach to optimizing clinical study eligibility criteria

OBJECTIVE

Feasible, safe, and inclusive eligibility criteria are crucial to successful clinical research recruitment. Existing expert-centered methods for eligibility criteria selection may not be representative of real-world populations. This paper presents a novel model called OPTEC (OPTimal Eligibility Criteria) based on the Multiple Attribute Decision Making method boosted by an efficient greedy algorithm.

METHODS

It systematically identifies the optimal criteria combination for a given medical condition with the optimal tradeoff among feasibility, patient safety, and cohort diversity. The model offers flexibility in attribute configurations and generalizability to various clinical domains. The model was evaluated on two clinical domains (i.e., Alzheimer's disease and Neoplasm of pancreas) using two datasets (i.e., MIMIC-III dataset and NewYork-Presbyterian/Columbia University Irving Medical Center (NYP/CUIMC) database).

RESULTS

We simulated the process of automatically optimizing eligibility criteria according to user-specified prioritization preferences and generated recommendations based on the top-ranked criteria combination accordingly (top 0.41-2.75%) with OPTEC. Harnessing the power of the model, we designed an interactive criteria recommendation system and conducted a case study with an experienced clinical researcher using the think-aloud protocol.

CONCLUSIONS

The results demonstrated that OPTEC could be used to recommend feasible eligibility criteria combinations, and to provide actionable recommendations for clinical study designers to construct a feasible, safe, and diverse cohort definition during early study design.

Industrial Sectors’ Perceptions about the Benefits of Implementing ISO 14001 Standard: MANOVA and Discriminant Analysis Approach

The most frequent drawback of ISO 14001 observed in existing studies relates to the cost of certification and implementation process. This drawback requires scaling the benefits of adopting the standard to assign organizations limited resources based on each benefit scale. This paper reports the first research results that scale the benefits of adopting the standard. A quantitative method was adopted, where data were collected using a questionnaire survey. A total of 120 respondents were recruited from organizations operating in six industrial sectors to take part in the study. MANOVA and discriminant analysis methodologies were used to analyze the 14 most cited benefits in the literature on adopting the standard. A novel feature of our approach is the comprehensive statistical analysis of the collected data, which yields robust results due to assumption satisfaction. The results demonstrated that the mean vector of the benefits was not equal per each sector. Environmental management and indicator dimensions can discriminate sectors more than the environmental awareness dimension. This study provides insights into the necessity of assessing the benefits of ISO 14001 adaptation that helps organizations allocate their limited resources optimally and support the listing of standard key performance indicators in ISO 14001. In addition, it calls for combining ISO 9001:2015, ISO 14001:2015, and ISO 45001:2018 in one standard.