Power analysis in a SMART design: sample size estimation for determining the best embedded dynamic treatment regime

Understanding Physician's Perspectives on AI in Health Care: Protocol for a Sequential Multiple Assignment Randomized Vignette Study

BACKGROUND

As the availability and performance of artificial intelligence (AI)-based clinical decision support (CDS) systems improve, physicians and other care providers poised to be on the front lines will be increasingly tasked with using these tools in patient care and incorporating their outputs into clinical decision-making processes. Vignette studies provide a means to explore emerging hypotheses regarding how context-specific factors, such as clinical risk, the amount of information provided about the AI, and the AI result, may impact physician acceptance and use of AI-based CDS tools. To best anticipate how such factors influence the decision-making of frontline physicians in clinical scenarios involving AI decision-support tools, hypothesis-driven research is needed that enables scenario testing before the implementation and deployment of these tools.

OBJECTIVE

This study's objectives are to (1) design an original, web-based vignette-based survey that features hypothetical scenarios based on emerging or real-world applications of AI-based CDS systems that will vary systematically by features related to clinical risk, the amount of information provided about the AI, and the AI result; and (2) test and determine causal effects of specific factors on the judgments and perceptions salient to physicians' clinical decision-making.

METHODS

US-based physicians with specialties in family or internal medicine will be recruited through email and mail (target n=420). Through a web-based survey, participants will be randomized to a 3-part "sequential multiple assignment randomization trial (SMART) vignette" detailing a hypothetical clinical scenario involving an AI decision support tool. The SMART vignette design is similar to the SMART design but adapted to a survey design. Each respondent will be randomly assigned to 1 of the possible vignette variations of the factors we are testing at each stage, which include the level of clinical risk, the amount of information provided about the AI, and the certainty of the AI output. Respondents will be given questions regarding their hypothetical decision-making in response to the hypothetical scenarios.

RESULTS

The study is currently in progress and data collection is anticipated to be completed in 2024.

CONCLUSIONS

The web-based vignette study will provide information on how contextual factors such as clinical risk, the amount of information provided about an AI tool, and the AI result influence physicians' reactions to hypothetical scenarios that are based on emerging applications of AI in frontline health care settings. Our newly proposed "SMART vignette" design offers several benefits not afforded by the extensively used traditional vignette design, due to the 2 aforementioned features. These advantages are (1) increased validity of analyses targeted at understanding the impact of a factor on the decision outcome, given previous outcomes and other contextual factors; and (2) balanced sample sizes across groups. This study will generate a better understanding of physician decision-making within this context.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/54787.

SMART Binary: New Sample Size Planning Resources for SMART Studies with Binary Outcome Measurements

Sequential Multiple-Assignment Randomized Trials (SMARTs) play an increasingly important role in psychological and behavioral health research. This experimental approach enables researchers to answer scientific questions about how to sequence and match interventions to the unique, changing needs of individuals. A variety of sample size planning resources for SMART studies have been developed, enabling researchers to plan SMARTs for addressing different types of scientific questions. However, relatively limited attention has been given to planning SMARTs with binary (dichotomous) outcomes, which often require higher sample sizes relative to continuous outcomes. Existing resources for estimating sample size requirements for SMARTs with binary outcomes do not consider the potential to improve power by including a baseline measurement and/or multiple repeated outcome measurements. The current paper addresses this issue by providing sample size planning simulation procedures and approximate formulas for two-wave repeated measures binary outcomes (i.e., two measurement times for the outcome variable, before and after intervention delivery). The simulation results agree well with the formulas. We also discuss how to use simulations to calculate power for studies with more than two outcome measurement occasions. Results show that having at least one repeated measurement of the outcome can substantially improve power under certain conditions.

Interim monitoring of sequential multiple assignment randomized trials using partial information

The sequential multiple assignment randomized trial (SMART) is the gold standard trial design to generate data for the evaluation of multistage treatment regimes. As with conventional (single-stage) randomized clinical trials, interim monitoring allows early stopping; however, there are few methods for principled interim analysis in SMARTs. Because SMARTs involve multiple stages of treatment, a key challenge is that not all enrolled participants will have progressed through all treatment stages at the time of an interim analysis. Wu et al. (2021) propose basing interim analyses on an estimator for the mean outcome under a given regime that uses data only from participants who have completed all treatment stages. We propose an estimator for the mean outcome under a given regime that gains efficiency by using partial information from enrolled participants regardless of their progression through treatment stages. Using the asymptotic distribution of this estimator, we derive associated Pocock and O'Brien-Fleming testing procedures for early stopping. In simulation experiments, the estimator controls type I error and achieves nominal power while reducing expected sample size relative to the method of Wu et al. (2021). We present an illustrative application of the proposed estimator based on a recent SMART evaluating behavioral pain interventions for breast cancer patients.

Sequential multiple assignment randomised controlled trial protocol for developing an adaptive intervention to improve depressive symptoms among family caregivers of people with dementia

OBJECTIVES

Family caregivers of people with dementia (FC-of-PWD) suffer from a high level of stress and depressive symptoms, which usually require different interventions at different stages. Although some standalone interventions such as behavioural activation (BA) and mindfulness practice (MP) have been shown to be potentially effective at reducing depressive symptoms, the best sequence and combination of these interventions for caregivers are unknown. This study aims to develop and identify a two-stage adaptive intervention with prespecified rules guiding whether, how or when to offer different interventions initially/over time to reduce depressive symptoms in FG-of-PWD.

METHODS

A sequential multiple assignment randomised trial design will be adopted. 272 FG-of-PWD with mild to moderate depressive symptoms will be recruited from the community. Four two-stage, embedded adaptive interventions involving BA and MP of different sequences and dosages (eg, 8 weeks of BA followed by booster sessions for responders and 8 weeks of MP for non-responders) will be assigned to the participants following a set of decision rules. The primary outcomes will be depressive symptoms (measured using the Patient Health Questionnaire-9), assessed after the second stage of the intervention. Other outcomes, such as positive aspects of caregiving (measured using the Positive Aspects of Caregiving Scale), sleep quality (measured using the Pittsburgh Sleep Quality Index) and time points will also be assessed. The analyses will follow the intention-to-treat principle. Several process indicators (eg, engagement in meaningful activities and level of mindfulness) will also be assessed. The findings will have strong implications for the further development of psychosocial adaptive interventions to reduce depressive symptoms among FC-of-PWD.

ETHICS AND DISSEMINATION

This study has received ethical approval from the Human Research Ethics Committee at The Hong Kong Polytechnic University (HSEARS20211223001). The findings will be presented at academic conferences and submitted to peer-reviewed journals for publication.

TRIAL REGISTRATION NUMBER

NCT05634317.

Bayesian sample size calculations for comparing two strategies in SMART studies

In the management of most chronic conditions characterized by the lack of universally effective treatments, adaptive treatment strategies (ATSs) have grown in popularity as they offer a more individualized approach. As a result, sequential multiple assignment randomized trials (SMARTs) have gained attention as the most suitable clinical trial design to formalize the study of these strategies. While the number of SMARTs has increased in recent years, sample size and design considerations have generally been carried out in frequentist settings. However, standard frequentist formulae require assumptions on interim response rates and variance components. Misspecifying these can lead to incorrect sample size calculations and correspondingly inadequate levels of power. The Bayesian framework offers a straightforward path to alleviate some of these concerns. In this paper, we provide calculations in a Bayesian setting to allow more realistic and robust estimates that account for uncertainty in inputs through the 'two priors' approach. Additionally, compared to the standard frequentist formulae, this methodology allows us to rely on fewer assumptions, integrate pre-trial knowledge, and switch the focus from the standardized effect size to the MDD. The proposed methodology is evaluated in a thorough simulation study and is implemented to estimate the sample size for a full-scale SMART of an internet-based adaptive stress management intervention on cardiovascular disease patients using data from its pilot study conducted in two Canadian provinces.

Sample size estimation for comparing dynamic treatment regimens in a SMART: A Monte Carlo-based approach and case study with longitudinal overdispersed count outcomes

Dynamic treatment regimens (DTRs), also known as treatment algorithms or adaptive interventions, play an increasingly important role in many health domains. DTRs are motivated to address the unique and changing needs of individuals by delivering the type of treatment needed, when needed, while minimizing unnecessary treatment. Practically, a DTR is a sequence of decision rules that specify, for each of several points in time, how available information about the individual's status and progress should be used in practice to decide which treatment (e.g. type or intensity) to deliver. The sequential multiple assignment randomized trial (SMART) is an experimental design widely used to empirically inform the development of DTRs. Sample size planning resources for SMARTs have been developed for continuous, binary, and survival outcomes. However, an important gap exists in sample size estimation methodology for SMARTs with longitudinal count outcomes. Furthermore, in many health domains, count data are overdispersed-having variance greater than their mean. We propose a Monte Carlo-based approach to sample size estimation applicable to many types of longitudinal outcomes and provide a case study with longitudinal overdispersed count outcomes. A SMART for engaging alcohol and cocaine-dependent patients in treatment is used as motivation.

The Sequential Multiple Assignment Randomized Trial for Controlling Infectious Diseases: A Review of Recent Developments

Infectious diseases have posed severe threats to public health across the world. Effective prevention and control of infectious diseases in the long term requires adapting interventions based on epidemiological evidence. The sequential multiple assignment randomized trial (SMART) is a multistage randomized trial that can provide valid evidence of when and how to adapt interventions for controlling infectious diseases based on evolving epidemiological evidence. We review recent developments in SMARTs to bring wider attention to the potential benefits of employing SMARTs in constructing effective adaptive interventions for controlling infectious diseases and other threats to public health. We discuss 2 example SMARTs for infectious diseases and summarize recent developments in SMARTs from the varied aspects of design, analysis, cost, and ethics. Public health investigators are encouraged to familiarize themselves with the related materials we discuss and collaborate with experts in SMARTs to translate the methodological developments into preeminent public health research. (Am J Public Health. 2023;113(1):49-59. https://doi.org/10.2105/AJPH.2022.307135).

Use of Sequential Multiple Assignment Randomized Trials (SMARTs) in oncology: systematic review of published studies

Sequential multiple assignments randomized trials (SMARTs) are a type of experimental design where patients may be randomised multiple times according to pre-specified decision rules. The present work investigates the state-of-the-art of SMART designs in oncology, focusing on the discrepancy between the available methodological approaches in the statistical literature and the procedures applied within cancer clinical trials. A systematic review was conducted, searching PubMed, Embase and CENTRAL for protocols or reports of results of SMART designs and registrations of SMART designs in clinical trial registries applied to solid tumour research. After title/abstract and full-text screening, 33 records were included. Fifteen were reports of trials' results, four were trials' protocols and fourteen were trials' registrations. The study design was defined as SMART by only one out of fifteen trial reports. Conversely, 13 of 18 study protocols and trial registrations defined the study design SMART. Furthermore, most of the records considered each stage separately in the analysis, without considering treatment regimens embedded in the trial. SMART designs in oncology are still limited. Study powering and analysis is mainly based on statistical approaches traditionally used in single-stage parallel trial designs. Formal reporting guidelines for SMART designs are needed.

MCMTC: A Pragmatic Framework for Selecting an Experimental Design to Inform the Development of Digital Interventions

Advances in digital technologies have created unprecedented opportunities to deliver effective and scalable behavior change interventions. Many digital interventions include multiple components, namely several aspects of the intervention that can be differentiated for systematic investigation. Various types of experimental approaches have been developed in recent years to enable researchers to obtain the empirical evidence necessary for the development of effective multiple-component interventions. These include factorial designs, Sequential Multiple Assignment Randomized Trials (SMARTs), and Micro-Randomized Trials (MRTs). An important challenge facing researchers concerns selecting the right type of design to match their scientific questions. Here, we propose MCMTC – a pragmatic framework that can be used to guide investigators interested in developing digital interventions in deciding which experimental approach to select. This framework includes five questions that investigators are encouraged to answer in the process of selecting the most suitable design: (1) Multiple-component intervention: Is the goal to develop an intervention that includes multiple components; (2) Component selection: Are there open scientific questions about the selection of specific components for inclusion in the intervention; (3) More than a single component: Are there open scientific questions about the inclusion of more than a single component in the intervention; (4) Timing: Are there open scientific questions about the timing of component delivery, that is when to deliver specific components; and (5) Change: Are the components in question designed to address conditions that change relatively slowly (e.g., over months or weeks) or rapidly (e.g., every day, hours, minutes). Throughout we use examples of tobacco cessation digital interventions to illustrate the process of selecting a design by answering these questions. For simplicity we focus exclusively on four experimental approaches—standard two- or multi-arm randomized trials, classic factorial designs, SMARTs, and MRTs—acknowledging that the array of possible experimental approaches for developing digital interventions is not limited to these designs.

Optimal allocation to treatments in a sequential multiple assignment randomized trial

One of the main questions in the design of a trial is how many subjects should be assigned to each treatment condition. Previous research has shown that equal randomization is not necessarily the best choice. We study the optimal allocation for a novel trial design, the sequential multiple assignment randomized trial, where subjects receive a sequence of treatments across various stages. A subject's randomization probabilities to treatments in the next stage depend on whether he or she responded to treatment in the current stage. We consider a prototypical sequential multiple assignment randomized trial design with two stages. Within such a design, many pairwise comparisons of treatment sequences can be made, and a multiple-objective optimal design strategy is proposed to consider all such comparisons simultaneously. The optimal design is sought under either a fixed total sample size or a fixed budget. A Shiny App is made available to find the optimal allocations and to evaluate the efficiency of competing designs. As the optimal design depends on the response rates to first-stage treatments, maximin optimal design methodology is used to find robust optimal designs. The proposed methodology is illustrated using a sequential multiple assignment randomized trial example on weight loss management.

Adaptive randomization in a two-stage sequential multiple assignment randomized trial

Sequential multiple assignment randomized trials (SMARTs) are systematic and efficient media for comparing dynamic treatment regimes (DTRs), where each patient is involved in multiple stages of treatment with the randomization at each stage depending on the patient's previous treatment history and interim outcomes. Generally, patients enrolled in SMARTs are randomized equally to ethically acceptable treatment options regardless of how effective those treatments were during the previous stages, which results in some undesirable consequences in practice, such as low recruitment, less retention, and lower treatment adherence. In this article, we propose a response-adaptive SMART (RA-SMART) design where the allocation probabilities are imbalanced in favor of more promising treatments based on the accumulated information on treatment efficacy from previous patients and stages. The operating characteristics of the RA-SMART design relative to SMART design, including the consistency and efficiency of estimated response rate under each DTR, the power of identifying the optimal DTR, and the number of patients treated with the optimal and the worst DTRs, are assessed through extensive simulation studies. Some practical suggestions are discussed in the conclusion.

Basal Glucose Control in Type 1 Diabetes Using Deep Reinforcement Learning: An In Silico Validation

People with Type 1 diabetes (T1D) require regular exogenous infusion of insulin to maintain their blood glucose concentration in a therapeutically adequate target range. Although the artificial pancreas and continuous glucose monitoring have been proven to be effective in achieving closed-loop control, significant challenges still remain due to the high complexity of glucose dynamics and limitations in the technology. In this work, we propose a novel deep reinforcement learning model for single-hormone (insulin) and dual-hormone (insulin and glucagon) delivery. In particular, the delivery strategies are developed by double Q-learning with dilated recurrent neural networks. For designing and testing purposes, the FDA-accepted UVA/Padova Type 1 simulator was employed. First, we performed long-term generalized training to obtain a population model. Then, this model was personalized with a small data-set of subject-specific data. In silico results show that the single and dual-hormone delivery strategies achieve good glucose control when compared to a standard basal-bolus therapy with low-glucose insulin suspension. Specifically, in the adult cohort (n = 10), percentage time in target range 70, 180 mg/dL improved from 77.6% to 80.9% with single-hormone control, and to 85.6% with dual-hormone control. In the adolescent cohort (n = 10), percentage time in target range improved from 55.5% to [Formula: see text] with single-hormone control, and to 78.8% with dual-hormone control. In all scenarios, a significant decrease in hypoglycemia was observed. These results show that the use of deep reinforcement learning is a viable approach for closed-loop glucose control in T1D.

SMARTp: A SMART design for nonsurgical treatments of chronic periodontitis with spatially referenced and nonrandomly missing skewed outcomes

This paper proposes dynamic treatment regimes (DTRs) as effective individualized treatment strategies for managing chronic periodontitis. The proposed DTRs are studied via SMARTp-a two-stage sequential multiple assignment randomized trial (SMART) design. For this design, we propose a statistical analysis plan and a novel cluster-level sample size calculation method that factors in typical features of periodontal responses such as non-Gaussianity, spatial clustering, and nonrandom missingness. Here, each patient is viewed as a cluster, and a tooth within a patient's mouth is viewed as an individual unit inside the cluster, with the tooth-level covariance structure described by a conditionally autoregressive structure. To accommodate possible skewness and tail behavior, the tooth-level clinical attachment level (CAL) response is assumed to be skew-t, with the nonrandomly missing structure captured via a shared parameter model corresponding to the missingness indicator. The proposed method considers mean comparison for the regimes with or without sharing an initial treatment, where the expected values and corresponding variances or covariance for the sample means of a pair of DTRs are derived by the inverse probability weighting and method of moments. Simulation studies are conducted to investigate the finite-sample performance of the proposed sample size formulas under a variety of outcome-generating scenarios. An R package SMARTp implementing our sample size formula is available at the Comprehensive R Archive Network for free download.