Evaluating the Effectiveness of Personalized Medicine With Software

Measuring the Performance of Survival Models to Personalize Treatment Choices

Various statistical and machine learning algorithms can be used to predict treatment effects at the patient level using data from randomized clinical trials (RCTs). Such predictions can facilitate individualized treatment decisions. Recently, a range of methods and metrics were developed for assessing the accuracy of such predictions. Here, we extend these methods, focusing on the case of survival (time-to-event) outcomes. We start by providing alternative definitions of the participant-level treatment benefit; subsequently, we summarize existing and propose new measures for assessing the performance of models estimating participant-level treatment benefits. We explore metrics assessing discrimination and calibration for benefit and decision accuracy. These measures can be used to assess the performance of statistical as well as machine learning models and can be useful during model development (i.e., for model selection or for internal validation) or when testing a model in new settings (i.e., in an external validation). We illustrate methods using simulated data and real data from the OPERAM trial, an RCT in multimorbid older people, which randomized participants to either standard care or a pharmacotherapy optimization intervention. We provide R codes for implementing all models and measures.

Advancing the personalized advantage index (PAI): a systematic review and application in two large multi-site samples in anxiety disorders

BACKGROUND

The Personalized Advantage Index (PAI) shows promise as a method for identifying the most effective treatment for individual patients. Previous studies have demonstrated its utility in retrospective evaluations across various settings. In this study, we explored the effect of different methodological choices in predictive modelling underlying the PAI.

METHODS

Our approach involved a two-step procedure. First, we conducted a review of prior studies utilizing the PAI, evaluating each study using the Prediction model study Risk Of Bias Assessment Tool (PROBAST). We specifically assessed whether the studies adhered to two standards of predictive modeling: refraining from using leave-one-out cross-validation (LOO CV) and preventing data leakage. Second, we examined the impact of deviating from these methodological standards in real data. We employed both a traditional approach violating these standards and an advanced approach implementing them in two large-scale datasets, PANIC-net (n = 261) and Protect-AD (n = 614).

RESULTS

The PROBAST-rating revealed a substantial risk of bias across studies, primarily due to inappropriate methodological choices. Most studies did not adhere to the examined prediction modeling standards, employing LOO CV and allowing data leakage. The comparison between the traditional and advanced approach revealed that ignoring these standards could systematically overestimate the utility of the PAI.

CONCLUSION

Our study cautions that violating standards in predictive modeling may strongly influence the evaluation of the PAI's utility, possibly leading to false positive results. To support an unbiased evaluation, crucial for potential clinical application, we provide a low-bias, openly accessible, and meticulously annotated script implementing the PAI.

Uncovering individualised treatment effects for educational trials

Large-scale Randomised Controlled Trials (RCTs) are widely regarded as "the gold standard" for testing the causal effects of school-based interventions. RCTs typically present the statistical significance of the average treatment effect (ATE), which captures the effect an intervention has had on average for a given population. However, key decisions in child health and education are often about individuals who may be very different from those averages. One way to identify heterogeneous treatment effects across different individuals, not captured by the ATE, is to conduct subgroup analyses. For example, free school meal (FSM) pupils as required for projects funded by the Education Endowment Foundation (EEF) in England. These subgroup analyses, as we demonstrate in 48 EEF-funded RCTs involving over 200,000 students, are usually not standardised across studies and offer flexible degrees of freedom to researchers, potentially leading to mixed, if not misleading, results. Here, we develop and deploy an alternative to ATE and subgroup analysis, a machine-learning and regression-based framework to predict individualised treatment effects (ITEs). ITEs could show where an intervention worked, for which individuals, and to what extent. Our findings have implications for decision-makers in fields like education, healthcare, law, and clinical practices concerning children and adolescents.

Measuring the performance of prediction models to personalize treatment choice

When data are available from individual patients receiving either a treatment or a control intervention in a randomized trial, various statistical and machine learning methods can be used to develop models for predicting future outcomes under the two conditions, and thus to predict treatment effect at the patient level. These predictions can subsequently guide personalized treatment choices. Although several methods for validating prediction models are available, little attention has been given to measuring the performance of predictions of personalized treatment effect. In this article, we propose a range of measures that can be used to this end. We start by defining two dimensions of model accuracy for treatment effects, for a single outcome: discrimination for benefit and calibration for benefit. We then amalgamate these two dimensions into an additional concept, decision accuracy, which quantifies the model's ability to identify patients for whom the benefit from treatment exceeds a given threshold. Subsequently, we propose a series of performance measures related to these dimensions and discuss estimating procedures, focusing on randomized data. Our methods are applicable for continuous or binary outcomes, for any type of prediction model, as long as it uses baseline covariates to predict outcomes under treatment and control. We illustrate all methods using two simulated datasets and a real dataset from a trial in depression. We implement all methods in the R package predieval. Results suggest that the proposed measures can be useful in evaluating and comparing the performance of competing models in predicting individualized treatment effect.

Mental health care for older adults: recent advances and new directions in clinical practice and research

The world's population is aging, bringing about an ever-greater burden of mental disorders in older adults. Given multimorbidities, the mental health care of these people and their family caregivers is labor-intensive. At the same time, ageism is a big problem for older people, with and without mental disorders. Positive elements of aging, such as resilience, wisdom and prosocial behaviors, need to be highlighted and promoted, both to combat stigma and to help protect and improve mental health in older adults. The positive psychiatry of aging is not an oxymoron, but a scientific construct strongly informed by research evidence. We champion a broader concept of geriatric psychiatry - one that encompasses health as well as illness. In the present paper, we address these issues in the context of four disorders that are the greatest source of years lived with disability: neurocognitive disorders, major depression, schizophrenia, and substance use disorders. We emphasize the need for implementation of multidisciplinary team care, with comprehensive assessment, clinical management, intensive outreach, and coordination of mental, physical and social health services. We also underscore the need for further research into moderators and mediators of treatment response variability. Because optimal care of older adults with mental disorders is both patient-focused and family-centered, we call for further research into enhancing the well-being of family caregivers. To optimize both the safety and efficacy of pharmacotherapy, further attention to metabolic, cardiovascular and neurological tolerability is much needed, together with further development and testing of medications that reduce the risk for suicide. At the same time, we also address positive aging and normal cognitive aging, both as an antidote to ageism and as a catalyst for change in the way we think about aging per se and late-life mental disorders more specifically. It is in this context that we provide directions for future clinical care and research.

Approaches to priority identification in digital health in ten countries of the Global Digital Health Partnership

BACKGROUND

To promote shared digital health best practices in a global context, as agreed within the Global Digital Health Partnership (GDHP), one of the most important topics to evaluate is the ability to detect what participating countries believe to be priorities suitable to improve their healthcare systems. No previously published scientific papers investigated these aspects as a cross-country comparison.

OBJECTIVE

The aim of this paper is to present results concerning the priorities identification section of the Evidence and Evaluation survey addressed to GDHP members in 2021, comparing countries' initiatives and perspectives for the future of digital health based on internationally agreed developments.

METHODS

This survey followed a cross-sectional study approach. An online survey was addressed to the stakeholders of 29 major countries.

RESULTS

Ten out of 29 countries answered the survey. The mean global score of 3.54 out of 5, calculated on the whole data set, demonstrates how the global attention to a digital evolution in health is shared by most of the evaluated countries.

CONCLUSION

The resulting insights on the differences between digital health priority identification among different GDHP countries serves as a starting point to coordinate further progress on digital health worldwide and foster evidence-based collaboration.

Personalized optimal nutrition lifestyle for self obesity management using metaalgorithms

Precision medicine applies machine learning methods to estimate the personalized optimal treatment decision based on individual information, such as genetic data and medical history. The main purpose of self obesity management is to develop a personalized optimal life plan that is easy to implement and adhere to, thereby reducing the incidence of obesity and obesity-related diseases. The methodology comprises three components. First, we apply catboost, random forest and lasso covariance test to evaluate the importance of individual features in forecasting body mass index. Second, we apply metaalgorithms to estimate the personalized optimal decision on alcohol, vegetable, high caloric food and daily water intake respectively for each individual. Third, we propose new metaalgorithms named SX and SXwint learners to compute the personalized optimal decision and compare their performances with other prevailing metalearners. We find that people who receive individualized optimal treatment options not only have lower obesity levels than others, but also have lower obesity levels than those who receive ’one-for-all’ treatment options. In conclusion, all metaalgorithms are effective at estimating the personalized optimal decision, where SXwint learner shows the best performance on daily water intake.