Multivariate analysis of the population representativeness of related clinical studies

Assessing the use of prescription drugs and dietary supplements in obese respondents in the National Health and Nutrition Examination Survey

Introduction

Obesity is a common disease and a known risk factor for many other conditions such as hypertension, type 2 diabetes, and cancer. Treatment options for obesity include lifestyle changes, pharmacotherapy, and surgical interventions such as bariatric surgery. In this study, we examine the use of prescription drugs and dietary supplements by the individuals with obesity.

Methods

We conducted a cross-sectional analysis of the National Health and Nutrition Examination Survey (NHANES) data 2003–2018. We used multivariate logistic regression to analyze the correlations of demographics and obesity status with the use of prescription drugs and dietary supplement use. We also built machine learning models to classify prescription drug and dietary supplement use using demographic data and obesity status.

Results

Individuals with obesity are more likely to take cardiovascular agents (OR = 2.095, 95% CI 1.989–2.207) and metabolic agents (OR = 1.658, 95% CI 1.573–1.748) than individuals without obesity. Gender, age, race, poverty income ratio, and insurance status are significantly correlated with dietary supplement use. The best performing model for classifying prescription drug use had the accuracy of 74.3% and the AUROC of 0.82. The best performing model for classifying dietary supplement use had the accuracy of 65.3% and the AUROC of 0.71.

Conclusions

This study can inform clinical practice and patient education of the use of prescription drugs and dietary supplements and their correlation with obesity.

Quantifying representativeness in randomized clinical trials using machine learning fairness metrics

OBJECTIVE

We help identify subpopulations underrepresented in randomized clinical trials (RCTs) cohorts with respect to national, community-based or health system target populations by formulating population representativeness of RCTs as a machine learning (ML) fairness problem, deriving new representation metrics, and deploying them in easy-to-understand interactive visualization tools.

MATERIALS AND METHODS

We represent RCT cohort enrollment as random binary classification fairness problems, and then show how ML fairness metrics based on enrollment fraction can be efficiently calculated using easily computed rates of subpopulations in RCT cohorts and target populations. We propose standardized versions of these metrics and deploy them in an interactive tool to analyze 3 RCTs with respect to type 2 diabetes and hypertension target populations in the National Health and Nutrition Examination Survey.

RESULTS

We demonstrate how the proposed metrics and associated statistics enable users to rapidly examine representativeness of all subpopulations in the RCT defined by a set of categorical traits (eg, gender, race, ethnicity, smoking status, and blood pressure) with respect to target populations.

DISCUSSION

The normalized metrics provide an intuitive standardized scale for evaluating representation across subgroups, which may have vastly different enrollment fractions and rates in RCT study cohorts. The metrics are beneficial complements to other approaches (eg, enrollment fractions) used to identify generalizability and health equity of RCTs.

CONCLUSION

By quantifying the gaps between RCT and target populations, the proposed methods can support generalizability evaluation of existing RCT cohorts. The interactive visualization tool can be readily applied to identified underrepresented subgroups with respect to any desired source or target populations.

"Include me if you can"-reasons for low enrollment of pediatric patients in a psychopharmacological trial

BACKGROUND

Low recruitment in clinical trials is a common and costly problem which undermines medical research. This study aimed to investigate the challenges faced in recruiting children and adolescents with obsessive-compulsive disorder and autism spectrum disorder for a randomized, double-blind, placebo-controlled clinical trial and to analyze reasons for non-participation. The trial was part of the EU FP7 project TACTICS (Translational Adolescent and Childhood Therapeutic Interventions in Compulsive Syndromes).

METHODS

Demographic data on pre-screening patients were collected systematically, including documented reasons for non-participation. Findings were grouped according to content, and descriptive statistical analyses of the data were performed.

RESULTS

In total, n = 173 patients were pre-screened for potential participation in the clinical trial. Of these, only five (2.9%) were eventually enrolled. The main reasons for non-inclusion were as follows: failure to meet all inclusion criteria/meeting one or more of the exclusion criteria (n = 73; 42.2%), no interest in the trial or trials in general (n = 40; 23.1%), and not wanting changes to current therapy/medication (n = 14; 8.1%).

CONCLUSIONS

The findings from this study add valuable information to the existing knowledge on reasons for low clinical trial recruitment rates in pediatric psychiatric populations. Low enrollment and high exclusion rates raise the question of whether such selective study populations are representative of clinical patient cohorts. Consequently, the generalizability of the results of such trials may be limited. The present findings will be useful in the development of improved recruitment strategies and may guide future research in establishing the measurement of representativeness to ensure enhanced external validity in psychopharmacological clinical trials in pediatric populations.

TRIAL REGISTRATION

EudraCT 2014-003080-38 . Registered on 14 July 2014.

Novel stepwise approach to assess representativeness of a large multicenter observational cohort of tuberculosis patients: The example of RePORT Brazil

BACKGROUND

A major goal of tuberculosis (TB) epidemiological studies is to obtain results that can be generalized to the larger population with TB. The ability to extrapolate findings on the determinants of TB treatment outcomes is also important.

METHODS

We compared baseline clinical and demographic characteristics and determinants of anti-TB treatment outcomes between persons enrolled in the Regional Prospective Observational Research in Tuberculosis (RePORT)-Brazil cohort between June 2015 and June 2019, and the registry of TB cases reported to the Brazilian National TB Program (Information System for Notifiable Diseases [SINAN]) during the same time period. Multivariable regression models adjusted for the study site were performed using second-generation p-values, a novel statistical approach. Associations with unfavorable treatment outcomes were tested for both RePORT-Brazil and SINAN cohorts.

FINDINGS

A total of 1,060 culture-confirmed TB patients were enrolled in RePORT-Brazil and 455,873 TB cases were reported to SINAN. Second-generation p-value analyses revealed that the cohorts were strikingly similar with regard to sex, age, use of antiretroviral therapy and positive initial smear sputum microscopy. However, diabetes, HIV infection, and smoking were more frequently documented in RePORT-Brazil. Illicit drug use, the presence of diabetes, and history of prior TB were associated with unfavorable TB treatment outcomes; illicit drug use was associated with such outcomes in both cohorts.

CONCLUSIONS

There were important similarities in demographic characteristics and determinants of clinical outcomes between the RePORT-Brazil cohort and the Brazilian National registry of TB cases.

Using Real-World Data to Rationalize Clinical Trials Eligibility Criteria Design: A Case Study of Alzheimer's Disease Trials

Low trial generalizability is a concern. The Food and Drug Administration had guidance on broadening trial eligibility criteria to enroll underrepresented populations. However, investigators are hesitant to do so because of concerns over patient safety. There is a lack of methods to rationalize criteria design. In this study, we used data from a large research network to assess how adjustments of eligibility criteria can jointly affect generalizability and patient safety (i.e the number of serious adverse events [SAEs]). We first built a model to predict the number of SAEs. Then, leveraging an a priori generalizability assessment algorithm, we assessed the changes in the number of predicted SAEs and the generalizability score, simulating the process of dropping exclusion criteria and increasing the upper limit of continuous eligibility criteria. We argued that broadening of eligibility criteria should balance between potential increases of SAEs and generalizability using donepezil trials as a case study.

Reporting of demographic data and representativeness in machine learning models using electronic health records

OBJECTIVE

The development of machine learning (ML) algorithms to address a variety of issues faced in clinical practice has increased rapidly. However, questions have arisen regarding biases in their development that can affect their applicability in specific populations. We sought to evaluate whether studies developing ML models from electronic health record (EHR) data report sufficient demographic data on the study populations to demonstrate representativeness and reproducibility.

MATERIALS AND METHODS

We searched PubMed for articles applying ML models to improve clinical decision-making using EHR data. We limited our search to papers published between 2015 and 2019.

RESULTS

Across the 164 studies reviewed, demographic variables were inconsistently reported and/or included as model inputs. Race/ethnicity was not reported in 64%; gender and age were not reported in 24% and 21% of studies, respectively. Socioeconomic status of the population was not reported in 92% of studies. Studies that mentioned these variables often did not report if they were included as model inputs. Few models (12%) were validated using external populations. Few studies (17%) open-sourced their code. Populations in the ML studies include higher proportions of White and Black yet fewer Hispanic subjects compared to the general US population.

DISCUSSION

The demographic characteristics of study populations are poorly reported in the ML literature based on EHR data. Demographic representativeness in training data and model transparency is necessary to ensure that ML models are deployed in an equitable and reproducible manner. Wider adoption of reporting guidelines is warranted to improve representativeness and reproducibility.

Assessing the Use and Perception of Dietary Supplements Among Obese Patients with National Health and Nutrition Examination Survey

Complementary alternative medicine, especially dietary supplements (DS), has gained increasing popularity for weight loss due to its availability without prescription, price, and ease of use. Besides weight loss, there are various perceived, potential benefits linked to DS use. However, health consumers with limited health literacy may not adequately know the benefits and risk of overdose for DS. In this project, we aim to gain a better understanding of the use of DS products among obese people as well as the perceived benefits of these products. We identified obese adults after combining the National Health and Nutrition Examination Survey data collected from 2003 to 2014. We found that there is a knowledge gap between the reported benefits of major DS by obese adults and the existing DS knowledge base and label database. This gap may inform the design of patient education material on DS usage in the future.

Precision Dosing Priority Criteria: Drug, Disease, and Patient Population Variables

The administered dose of a drug modulates whether patients will experience optimal effectiveness, toxicity including death, or no effect at all. Dosing is particularly important for diseases and/or drugs where the drug can decrease severe morbidity or prolong life. Likewise, dosing is important where the drug can cause death or severe morbidity. Since we believe there are many examples where more precise dosing could benefit patients, it is worthwhile to consider how to prioritize drug–disease targets. One key consideration is the quality of information available from which more precise dosing recommendations can be constructed. When a new more precise dosing scheme is created and differs significantly from the approved label, it is important to consider the level of proof necessary to either change the label and/or change clinical practice. The cost and effort needed to provide this proof should also be considered in prioritizing drug–disease precision dosing targets. Although precision dosing is being promoted and has great promise, it is underutilized in many drugs and disease states. Therefore, we believe it is important to consider how more precise dosing is going to be delivered to high priority patients in a timely manner. If better dosing schemes do not change clinical practice resulting in better patient outcomes, then what is the use? This review paper discusses variables to consider when prioritizing precision dosing candidates while highlighting key examples of precision dosing that have been successfully used to improve patient care.

Clinical Trial Generalizability Assessment in the Big Data Era: A Review

Clinical studies, especially randomized, controlled trials, are essential for generating evidence for clinical practice. However, generalizability is a long‐standing concern when applying trial results to real‐world patients. Generalizability assessment is thus important, nevertheless, not consistently practiced. We performed a systematic review to understand the practice of generalizability assessment. We identified 187 relevant articles and systematically organized these studies in a taxonomy with three dimensions: (i) data availability (i.e., before or after trial (a priori vs. a posteriori generalizability)); (ii) result outputs (i.e., score vs. nonscore); and (iii) populations of interest. We further reported disease areas, underrepresented subgroups, and types of data used to profile target populations. We observed an increasing trend of generalizability assessments, but < 30% of studies reported positive generalizability results. As a priori generalizability can be assessed using only study design information (primarily eligibility criteria), it gives investigators a golden opportunity to adjust the study design before the trial starts. Nevertheless, < 40% of the studies in our review assessed a priori generalizability. With the wide adoption of electronic health records systems, rich real‐world patient databases are increasingly available for generalizability assessment; however, informatics tools are lacking to support the adoption of generalizability assessment practice.

Assessing the Validity of a a priori Patient-Trial Generalizability Score using Real-world Data from a Large Clinical Data Research Network: A Colorectal Cancer Clinical Trial Case Study

Existing trials had not taken enough consideration of their population representativeness, which can lower the effectiveness when the treatment is applied in real-world clinical practice. We analyzed the eligibility criteria of Bevacizumab colorectal cancer treatment trials, assessed their a priori generalizability, and examined how it affects patient outcomes when applied in real-world clinical settings. To do so, we extracted patient-level data from a large collection of electronic health records (EHRs) from the OneFlorida consortium. We built a zero-inflated negative binomial model using a composite patient-trial generalizability (cPTG) score to predict patients' clinical outcomes (i.e., number of serious adverse events, [SAEs]). Our study results provide a body of evidence that 1) the cPTG scores can predict patient outcomes; and 2) patients who are more similar to the study population in the trials that were used to develop the treatment will have a significantly lower possibility to experience serious adverse events.

The next generation of precision medicine: observational studies, electronic health records, biobanks and continuous monitoring

Precision medicine can utilize new techniques in order to more effectively translate research findings into clinical practice. In this article, we first explore the limitations of traditional study designs, which stem from (to name a few): massive cost for the assembly of large patient cohorts; non-representative patient data; and the astounding complexity of human biology. Second, we propose that harnessing electronic health records and mobile device biometrics coupled to longitudinal data may prove to be a solution to many of these problems by capturing a 'real world' phenotype. We envision that future biomedical research utilizing more precise approaches to patient care will utilize continuous and longitudinal data sources.

Prevalence of Multiple Chronic Conditions Among Older Adults in Florida and the United States: Comparative Analysis of the OneFlorida Data Trust and National Inpatient Sample

Background

Older patients with multiple chronic conditions are often faced with increased health care needs and subsequent higher medical costs, posing significant financial burden to patients, their caregivers, and the health care system. The increasing adoption of electronic health record systems and the proliferation of clinical data offer new opportunities for prevalence studies and for population health assessment. The last few years have witnessed an increasing number of clinical research networks focused on building large collections of clinical data from electronic health records and claims to make it easier and less costly to conduct clinical research.

Objective

The aim of this study was to compare the prevalence of common chronic conditions and multiple chronic conditions in older adults between Florida and the United States using data from the OneFlorida Clinical Research Consortium and the Healthcare Cost and Utilization Project (HCUP) National Inpatient Sample (NIS).

Methods

We first analyzed the basic demographic characteristics of the older adults in 3 datasets—the 2013 OneFlorida data, the 2013 HCUP NIS data, and the combined 2012 to 2016 OneFlorida data. Then we analyzed the prevalence of each of the 25 chronic conditions in each of the 3 datasets. We stratified the analysis of older adults with hypertension, the most prevalent condition. Additionally, we examined trends (ie, overall trends and then by age, race, and gender) in the prevalence of discharge records representing multiple chronic conditions over time for the OneFlorida (2012-2016) and HCUP NIS cohorts (2003-2013).

Results

The rankings of the top 10 prevalent conditions are the same across the OneFlorida and HCUP NIS datasets. The most prevalent multiple chronic conditions of 2 conditions among the 3 datasets were—hyperlipidemia and hypertension; hypertension and ischemic heart disease; diabetes and hypertension; chronic kidney disease and hypertension; anemia and hypertension; and hyperlipidemia and ischemic heart disease. We observed increasing trends in multiple chronic conditions in both data sources.

Conclusions

The results showed that chronic conditions and multiple chronic conditions are prevalent in older adults across Florida and the United States. Even though slight differences were observed, the similar estimates of prevalence of chronic conditions and multiple chronic conditions across OneFlorida and HCUP NIS suggested that clinical research data networks such as OneFlorida, built from heterogeneous data sources, can provide rich data resources for conducting large-scale secondary data analyses.

The representativeness of eligible patients in type 2 diabetes trials: a case study using GIST 2.0

Objective

The population representativeness of a clinical study is influenced by how real-world patients qualify for the study. We analyze the representativeness of eligible patients for multiple type 2 diabetes trials and the relationship between representativeness and other trial characteristics.

Methods

Sixty-nine study traits available in the electronic health record data for 2034 patients with type 2 diabetes were used to profile the target patients for type 2 diabetes trials. A set of 1691 type 2 diabetes trials was identified from ClinicalTrials.gov, and their population representativeness was calculated using the published Generalizability Index of Study Traits 2.0 metric. The relationships between population representativeness and number of traits and between trial duration and trial metadata were statistically analyzed. A focused analysis with only phase 2 and 3 interventional trials was also conducted.

Results

A total of 869 of 1691 trials (51.4%) and 412 of 776 phase 2 and 3 interventional trials (53.1%) had a population representativeness of <5%. The overall representativeness was significantly correlated with the representativeness of the Hba1c criterion. The greater the number of criteria or the shorter the trial, the less the representativeness. Among the trial metadata, phase, recruitment status, and start year were found to have a statistically significant effect on population representativeness. For phase 2 and 3 interventional trials, only start year was significantly associated with representativeness.

Conclusions

Our study quantified the representativeness of multiple type 2 diabetes trials. The common low representativeness of type 2 diabetes trials could be attributed to specific study design requirements of trials or safety concerns. Rather than criticizing the low representativeness, we contribute a method for increasing the transparency of the representativeness of clinical trials.

Assessing the population representativeness of colorectal cancer treatment clinical trials

The generalizability (external validity) of clinical trials has long been a concern for both clinical research community as well as the general public. Results of trials that do not represent the target population may not be applicable to the broader patient population. In this study, we used a previously published metric Generalizability Index for Study Traits (GIST) to assess the population representativeness of colorectal cancer (CRC) treatment trials. Our analysis showed that the quantitative eligibility criteria of CRC trials are in general not restrictive. However, the qualitative eligibility criteria in these trials are with moderate or strict restrictions, which may impact their population representativeness of the real-world patient population.

Comparing and Contrasting A Priori and A Posteriori Generalizability Assessment of Clinical Trials on Type 2 Diabetes Mellitus

Clinical trials are indispensable tools for evidence-based medicine. However, they are often criticized for poor generalizability. Traditional trial generalizability assessment can only be done after the trial results are published, which compares the enrolled patients with a convenience sample of real-world patients. However, the proliferation of electronic data in clinical trial registries and clinical data warehouses offer a great opportunity to assess the generalizability during the design phase of a new trial. In this work, we compared and contrasted a priori (based on eligibility criteria) and a posteriori (based on enrolled patients) generalizability of Type 2 diabetes clinical trials. Further, we showed that comparing the study population selected by the clinical trial eligibility criteria to the real-world patient population is a good indicator of the generalizability of trials. Our findings demonstrate that the a priori generalizability of a trial is comparable to its a posteriori generalizability in identifying restrictive quantitative eligibility criteria.

Correlating eligibility criteria generalizability and adverse events using Big Data for patients and clinical trials

Randomized controlled trials can benefit from proactive assessment of how well their participant selection strategies during the design of eligibility criteria can influence the study generalizability. In this paper, we present a quantitative metric called generalizability index for study traits 2.0 (GIST 2.0) to assess the a priori generalizability (based on population representativeness) of a clinical trial by accounting for the dependencies among multiple eligibility criteria. The metric was evaluated on 16 sepsis trials identified from ClinicalTrials.gov, with their adverse event reports extracted from the trial results sections. The correlation between GIST scores and adverse events was analyzed. We found that the GIST 2.0 score was significantly correlated with total adverse events and serious adverse events (weighted correlation coefficients of 0.825 and 0.709, respectively, with P < 0.01). This study exemplifies the promising use of Big Data in electronic health records and ClinicalTrials.gov for optimizing eligibility criteria design for clinical studies.

GIST 2.0: A scalable multi-trait metric for quantifying population representativeness of individual clinical studies

The design of randomized controlled clinical studies can greatly benefit from iterative assessments of population representativeness of eligibility criteria. We propose a multi-trait metric - GIST 2.0 that can compute the a priori generalizability based on the population representativeness of a clinical study by explicitly modeling the dependencies among all eligibility criteria. We evaluate this metric on twenty clinical studies of two diseases and analyze how a study's eligibility criteria affect its generalizability (collectively and individually). We statistically analyze the effects of trial setting, trait selection and trait summarizing technique on GIST 2.0. Finally we provide theoretical as well as empirical validations for the expected properties of GIST 2.0.