Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine

OBJECTIVE

Applications of machine learning in healthcare are of high interest and have the potential to improve patient care. Yet, the real-world accuracy of these models in clinical practice and on different patient subpopulations remains unclear. To address these important questions, we hosted a community challenge to evaluate methods that predict healthcare outcomes. We focused on the prediction of all-cause mortality as the community challenge question.

MATERIALS AND METHODS

Using a Model-to-Data framework, 345 registered participants, coalescing into 25 independent teams, spread over 3 continents and 10 countries, generated 25 accurate models all trained on a dataset of over 1.1 million patients and evaluated on patients prospectively collected over a 1-year observation of a large health system.

RESULTS

The top performing team achieved a final area under the receiver operator curve of 0.947 (95% CI, 0.942-0.951) and an area under the precision-recall curve of 0.487 (95% CI, 0.458-0.499) on a prospectively collected patient cohort.

DISCUSSION

Post hoc analysis after the challenge revealed that models differ in accuracy on subpopulations, delineated by race or gender, even when they are trained on the same data.

CONCLUSION

This is the largest community challenge focused on the evaluation of state-of-the-art machine learning methods in a healthcare system performed to date, revealing both opportunities and pitfalls of clinical AI.

Large trans-ethnic meta-analysis identifies AKR1C4 as a novel gene associated with age at menarche

STUDY QUESTION

Does the expansion of genome-wide association studies (GWAS) to a broader range of ancestries improve the ability to identify and generalise variants associated with age at menarche (AAM) in European populations to a wider range of world populations?

SUMMARY ANSWER

By including women with diverse and predominantly non-European ancestry in a large-scale meta-analysis of AAM with half of the women being of African ancestry, we identified a new locus associated with AAM in African-ancestry participants, and generalised loci from GWAS of European ancestry individuals.

WHAT IS KNOWN ALREADY

AAM is a highly polygenic puberty trait associated with various diseases later in life. Both AAM and diseases associated with puberty timing vary by race or ethnicity. The majority of GWAS of AAM have been performed in European ancestry women.

STUDY DESIGN, SIZE, DURATION

We analysed a total of 38 546 women who did not have predominantly European ancestry backgrounds: 25 149 women from seven studies from the ReproGen Consortium and 13 397 women from the UK Biobank. In addition, we used an independent sample of 5148 African-ancestry women from the Southern Community Cohort Study (SCCS) for replication.

PARTICIPANTS/MATERIALS, SETTING, METHODS

Each AAM GWAS was performed by study and ancestry or ethnic group using linear regression models adjusted for birth year and study-specific covariates. ReproGen and UK Biobank results were meta-analysed using an inverse variance-weighted average method. A trans-ethnic meta-analysis was also carried out to assess heterogeneity due to different ancestry.

MAIN RESULTS AND THE ROLE OF CHANCE

We observed consistent direction and effect sizes between our meta-analysis and the largest GWAS conducted in European or Asian ancestry women. We validated four AAM loci (1p31, 6q16, 6q22 and 9q31) with common genetic variants at P < 5 × 10-7. We detected one new association (10p15) at P < 5 × 10-8 with a low-frequency genetic variant lying in AKR1C4, which was replicated in an independent sample. This gene belongs to a family of enzymes that regulate the metabolism of steroid hormones and have been implicated in the pathophysiology of uterine diseases. The genetic variant in the new locus is more frequent in African-ancestry participants, and has a very low frequency in Asian or European-ancestry individuals.

LARGE SCALE DATA

N/A.

LIMITATIONS, REASONS FOR CAUTION

Extreme AAM (<9 years or >18 years) were excluded from analysis. Women may not fully recall their AAM as most of the studies were conducted many years later. Further studies in women with diverse and predominantly non-European ancestry are needed to confirm and extend these findings, but the availability of such replication samples is limited.

WIDER IMPLICATIONS OF THE FINDINGS

Expanding association studies to a broader range of ancestries or ethnicities may improve the identification of new genetic variants associated with complex diseases or traits and the generalisation of variants from European-ancestry studies to a wider range of world populations.

STUDY FUNDING/COMPETING INTEREST(S)

Funding was provided by CHARGE Consortium grant R01HL105756-07: Gene Discovery For CVD and Aging Phenotypes and by the NIH grant U24AG051129 awarded by the National Institute on Aging (NIA). The authors have no conflict of interest to declare.

The syndemic effects of mental ill health, household hunger, and intimate partner violence on adherence to antiretroviral therapy among pregnant women living with HIV in Yaoundé, Cameroon

Background

This research advances understanding of interrelationships among three barriers to adherence to antiretroviral therapy (ART) among pregnant women living with HIV (WLWH) in Cameroon: probable common mental disorders (CMD), intimate partner violence (IPV), and hunger.

Methods

The sample included 220 pregnant WLWH in Cameroon. Multivariable modified Poisson regression was conducted to assess the relationship between IPV, hunger, and CMD on ART adherence.

Results

Almost half (44%) of participants recently missed/mistimed an ART dose. Probable CMD was associated with greater risk of missed/mistimed ART dose (aRR 1.5 [95% CI 1.1, 1.9]). Hunger was associated with greater risk of missed/mistimed ART dose among those who reported IPV (aRR 1.9 [95% CI 1.2, 2.8]), but not among those who did not (aRR 0.8 [95% CI 0.2, 2.3]).

Conclusion

Suboptimal ART adherence, CMD, and IPV were common among pregnant WLWH in Cameroon. Pregnant WLWH experiencing IPV and hunger may be especially vulnerable to suboptimal ART adherence.