Circulating proteins to predict COVID-19 severity

Predicting COVID-19 severity is difficult, and the biological pathways involved are not fully understood. To approach this problem, we measured 4701 circulating human protein abundances in two independent cohorts totaling 986 individuals. We then trained prediction models including protein abundances and clinical risk factors to predict COVID-19 severity in 417 subjects and tested these models in a separate cohort of 569 individuals. For severe COVID-19, a baseline model including age and sex provided an area under the receiver operator curve (AUC) of 65% in the test cohort. Selecting 92 proteins from the 4701 unique protein abundances improved the AUC to 88% in the training cohort, which remained relatively stable in the testing cohort at 86%, suggesting good generalizability. Proteins selected from different COVID-19 severity were enriched for cytokine and cytokine receptors, but more than half of the enriched pathways were not immune-related. Taken together, these findings suggest that circulating proteins measured at early stages of disease progression are reasonably accurate predictors of COVID-19 severity. Further research is needed to understand how to incorporate protein measurement into clinical care.

Low-Rank Representation of Reinforcement Learning Policies

We propose a general framework for policy representation for reinforcement learning tasks. This framework involves finding a low-dimensional embedding of the policy on a reproducing kernel Hilbert space (RKHS). The usage of RKHS based methods allows us to derive strong theoretical guarantees on the expected return of the reconstructed policy. Such guarantees are typically lacking in black-box models, but are very desirable in tasks requiring stability and convergence guarantees. We conduct several experiments on classic RL domains. The results confirm that the policies can be robustly represented in a low-dimensional space while the embedded policy incurs almost no decrease in returns.

Biomedical Research & Informatics Living Laboratory for Innovative Advances of New Technologies in Community Mobility Rehabilitation: Protocol for a longitudinal evaluation of mobility outcomes (Preprint)

Background

Rapid advances in technologies over the past 10 years have enabled large-scale biomedical and psychosocial rehabilitation research to improve the function and social integration of persons with physical impairments across the lifespan. The Biomedical Research and Informatics Living Laboratory for Innovative Advances of New Technologies (BRILLIANT) in community mobility rehabilitation aims to generate evidence-based research to improve rehabilitation for individuals with acquired brain injury (ABI).

Objective

This study aims to (1) identify the factors limiting or enhancing mobility in real-world community environments (public spaces, including the mall, home, and outdoors) and understand their complex interplay in individuals of all ages with ABI and (2) customize community environment mobility training by identifying, on a continuous basis, the specific rehabilitation strategies and interventions that patient subgroups benefit from most. Here, we present the research and technology plan for the BRILLIANT initiative.

Methods

A cohort of individuals, adults and children, with ABI (N=1500) will be recruited. Patients will be recruited from the acute care and rehabilitation partner centers within 4 health regions (living labs) and followed throughout the continuum of rehabilitation. Participants will also be recruited from the community. Biomedical, clinician-reported, patient-reported, and brain imaging data will be collected. Theme 1 will implement and evaluate the feasibility of collecting data across BRILLIANT living labs and conduct predictive analyses and artificial intelligence (AI) to identify mobility subgroups. Theme 2 will implement, evaluate, and identify community mobility interventions that optimize outcomes for mobility subgroups of patients with ABI.

Results

The biomedical infrastructure and equipment have been established across the living labs, and development of the clinician- and patient-reported outcome digital solutions is underway. Recruitment is expected to begin in May 2022.

Conclusions

The program will develop and deploy a comprehensive clinical and community-based mobility-monitoring system to evaluate the factors that result in poor mobility, and develop personalized mobility interventions that are optimized for specific patient subgroups. Technology solutions will be designed to support clinicians and patients to deliver cost-effective care and the right intervention to the right person at the right time to optimize long-term functional potential and meaningful participation in the community.

International Registered Report Identifier (IRRID)

PRR1-10.2196/12506

COVID-19 Prognosis via Self-Supervised Representation Learning and Multi-Image Prediction

The rapid spread of COVID-19 cases in recent months has strained hospital resources, making rapid and accurate triage of patients presenting to emergency departments a necessity. Machine learning techniques using clinical data such as chest X-rays have been used to predict which patients are most at risk of deterioration. We consider the task of predicting two types of patient deterioration based on chest X-rays: adverse event deterioration (i.e., transfer to the intensive care unit, intubation, or mortality) and increased oxygen requirements beyond 6 L per day. Due to the relative scarcity of COVID-19 patient data, existing solutions leverage supervised pretraining on related non-COVID images, but this is limited by the differences between the pretraining data and the target COVID-19 patient data. In this paper, we use self-supervised learning based on the momentum contrast (MoCo) method in the pretraining phase to learn more general image representations to use for downstream tasks. We present three results. The first is deterioration prediction from a single image, where our model achieves an area under receiver operating characteristic curve (AUC) of 0.742 for predicting an adverse event within 96 hours (compared to 0.703 with supervised pretraining) and an AUC of 0.765 for predicting oxygen requirements greater than 6 L a day at 24 hours (compared to 0.749 with supervised pretraining). We then propose a new transformer-based architecture that can process sequences of multiple images for prediction and show that this model can achieve an improved AUC of 0.786 for predicting an adverse event at 96 hours and an AUC of 0.848 for predicting mortalities at 96 hours. A small pilot clinical study suggested that the prediction accuracy of our model is comparable to that of experienced radiologists analyzing the same information.

The Bottleneck Simulator: A Model-Based Deep Reinforcement Learning Approach

Deep reinforcement learning has recently shown many impressive successes. However, one major obstacle towards applying such methods to real-world problems is their lack of data-efficiency. To this end, we propose the Bottleneck Simulator: a model-based reinforcement learning method which combines a learned, factorized transition model of the environment with rollout simulations to learn an effective policy from few examples. The learned transition model employs an abstract, discrete (bottleneck) state, which increases sample efficiency by reducing the number of model parameters and by exploiting structural properties of the environment. We provide a mathematical analysis of the Bottleneck Simulator in terms of fixed points of the learned policy, which reveals how performance is affected by four distinct sources of error: an error related to the abstract space structure, an error related to the transition model estimation variance, an error related to the transition model estimation bias, and an error related to the transition model class bias. Finally, we evaluate the Bottleneck Simulator on two natural language processing tasks: a text adventure game and a real-world, complex dialogue response selection task. On both tasks, the Bottleneck Simulator yields excellent performance beating competing approaches.

Transparency and reproducibility in artificial intelligence

Breakthroughs in artificial intelligence (AI) hold enormous potential as it can automate complex tasks and go even beyond human performance. In their study, McKinney et al. showed the high potential of AI for breast cancer screening. However, the lack of methods’ details and algorithm code undermines its scientific value. Here, we identify obstacles hindering transparent and reproducible AI research as faced by McKinney et al., and provide solutions to these obstacles with implications for the broader field.

Development of a polygenic risk score to improve screening for fracture risk: A genetic risk prediction study

BACKGROUND

Since screening programs identify only a small proportion of the population as eligible for an intervention, genomic prediction of heritable risk factors could decrease the number needing to be screened by removing individuals at low genetic risk. We therefore tested whether a polygenic risk score for heel quantitative ultrasound speed of sound (SOS)-a heritable risk factor for osteoporotic fracture-can identify low-risk individuals who can safely be excluded from a fracture risk screening program.

METHODS AND FINDINGS

A polygenic risk score for SOS was trained and selected in 2 separate subsets of UK Biobank (comprising 341,449 and 5,335 individuals). The top-performing prediction model was termed "gSOS", and its utility in fracture risk screening was tested in 5 validation cohorts using the National Osteoporosis Guideline Group clinical guidelines (N = 10,522 eligible participants). All individuals were genome-wide genotyped and had measured fracture risk factors. Across the 5 cohorts, the average age ranged from 57 to 75 years, and 54% of studied individuals were women. The main outcomes were the sensitivity and specificity to correctly identify individuals requiring treatment with and without genetic prescreening. The reference standard was a bone mineral density (BMD)-based Fracture Risk Assessment Tool (FRAX) score. The secondary outcomes were the proportions of the screened population requiring clinical-risk-factor-based FRAX (CRF-FRAX) screening and BMD-based FRAX (BMD-FRAX) screening. gSOS was strongly correlated with measured SOS (r2 = 23.2%, 95% CI 22.7% to 23.7%). Without genetic prescreening, guideline recommendations achieved a sensitivity and specificity for correct treatment assignment of 99.6% and 97.1%, respectively, in the validation cohorts. However, 81% of the population required CRF-FRAX tests, and 37% required BMD-FRAX tests to achieve this accuracy. Using gSOS in prescreening and limiting further assessment to those with a low gSOS resulted in small changes to the sensitivity and specificity (93.4% and 98.5%, respectively), but the proportions of individuals requiring CRF-FRAX tests and BMD-FRAX tests were reduced by 37% and 41%, respectively. Study limitations include a reliance on cohorts of predominantly European ethnicity and use of a proxy of fracture risk.

CONCLUSIONS

Our results suggest that the use of a polygenic risk score in fracture risk screening could decrease the number of individuals requiring screening tests, including BMD measurement, while maintaining a high sensitivity and specificity to identify individuals who should be recommended an intervention.

[Health technology assessment of innovative medical devices: Timing and decision at national and local level]

INTRODUCTION

The Medical Device Committee (CODIMS) evaluates all innovative medical devices (MD) before their introduction in the hospitals of the Assistance publique-hôpitaux de Paris (AP-HP). At the national level, the Medical Device and Health Technology Evaluation Committee (CNEDiMTS) provides recommendation for MD with respects to reimbursement by the National Health Insurance Fund. The aim of this study is to compare the recommendations of both committees and to analyze their timing on a six-year period.

MATERIAL AND METHOD

We selected all innovative MD assessed by the CODIMS between 2013 and 2018. We retrieved all the recommendations for these MD from the CNEDiMTS. We performed quantitative and qualitative analysis of data collected.

RESULTS

On 30 innovative MD assessed by both the CODIMS and the CNEDiMTS, 11 (37%) evaluations were performed by the CODIMS before the CNEDiMTS evaluation. They occurred approximately a year before the CNEDiMTS recommendation (an average of 378 days). Among the 25 MD with a recommendation of both committees, the two opinions were consistent in 88 per cent of all cases.

DISCUSSION/CONCLUSION

This study highlights that there is a good consistency between the recommendations of both committees. This suggests that the MD evaluations conducted at the hospital level are relevant and timely. Finally, a better coordination between the national and local levels should be promoted for the MD assessment.

On Overfitting and Asymptotic Bias in Batch Reinforcement Learning with Partial Observability

This paper provides an analysis of the tradeoff between asymptotic bias (suboptimality with unlimited data) and overfitting (additional suboptimality due to limited data) in the context of reinforcement learning with partial observability. Our theoretical analysis formally characterizes that while potentially increasing the asymptotic bias, a smaller state representation decreases the risk of overfitting. This analysis relies on expressing the quality of a state representation by bounding $L_1$ error terms of the associated belief states.  Theoretical results are empirically illustrated when the state representation is a truncated history of observations, both on synthetic POMDPs and on a large-scale POMDP in the context of smartgrids, with real-world data. Finally, similarly to known results in the fully observable setting, we also briefly discuss and empirically illustrate how using function approximators and adapting the discount factor may enhance the tradeoff between asymptotic bias and overfitting in the partially observable context.

A decision-theoretic approach for the collaborative control of a smart wheelchair

While assistive robot technology is quickly progressing, several challenges remain to make this technology truly usable and useful for humans. One of the aspects that is particularly important is in defining control protocols that allow both the human and the robot technology to contribute to the best of their abilities. In this paper we propose a framework for the collaborative control of a smart wheelchair designed for individuals with mobility impairments. Our approach is based on a decision-theoretic model of control, and accepts commands from both the human user and robot controller. We use a Partially Observable Markov Decision Process to optimize the collaborative action choice, which allows the system to take into account uncertainty in the user intent, in the command and in the environment. The system is deployed and validated on the SmartWheeler platform, and experiments with 8 users show the improvement in usability and navigation efficiency that are achieved with this form of collaborative control.

Engagement and learning in simulation: recommendations of the Simnovate Engaged Learning Domain Group

Background

Health professions education (HPE) is based on deliberate learning activities and clinical immersion to achieve clinical competence. Simulation is a tool that helps bridge the knowledge-to-action gap through deliberate learning. This paper considers how to optimally engage learners in simulation activities as part of HPE.

Methods

The Simnovate Engaged Learning Domain Group undertook 3 teleconferences to survey the current concepts regarding pervasive learning. Specific attention was paid to engagement in the learning process, with respect to fidelity, realism and emotions, and the use of narratives in HPE simulation.

Results

This paper found that while many types of simulation exist, the current ways to categorise the types of simulation do not sufficiently describe what a particular simulation will entail. This paper introduces a novel framework to describe simulation by deconstructing a simulation activity into 3 core characteristics (scope, modality and environment). Then, the paper discusses how engagement is at the heart of the learning process, but remained an understudied phenomenon with respect to HPE simulation. Building on the first part, a conceptual framework for engaged learning in HPE simulation was derived, with potential use across all HPE methods.

Discussion

The framework considers how the 3 characteristics of simulation interplay with the dimensions of fidelity (physical, conceptual and emotional), and how these can be conveyed by and articulated through beauty (as a proxy for efficiency) as coexisting factors to drive learner engagement. This framework leads to the translation of deliberately taught knowledge, skills and attitudes into clinical competence and subsequent performance.

Differences in the Evacuation Behaviour of Office and Apartment Building Occupants

An experiment was designed to observe evacuation times and occupant movement in two office buildings during a simulated fire emergency and to compare these results with previous studies of evacuation drills in apartment buildings. The evacuation drills were recorded using videocameras located throughout the buildings. The results were analyzed with respect to occupant behaviour, occupants' time to start to evacuate, occupants' time to reach an outside exit, and the occupants' speed while travelling in the stairways. A comparison of the results from this office buildings study with previous studies involving evacuations of midrise and highrise apartment buildings reveals many interesting differences. The physical organization of the buildings, evacuation strategies, and the occupants' characteristics, behaviour and movement are discussed.

The study showed that apartment building occupants delay their evacuation more than office building occupants, either by a long preparation time or because they cannot hear the alarm from their apartments. Travelling speeds are slowest in midrise apartment buildings where the population is more diverse and includes children, elderly people and occupants with limitations. The more structured evacuation plans, the presence of fire wardens and the easier access to fire safety information also contribute to the efficiency of evacuation procedures in office buildings.

Modeling Glucagon Action in Patients With Type 1 Diabetes

The dual-hormone artificial pancreas is an emerging technology to treat type 1 diabetes (T1D). It consists of a glucose sensor, infusion pumps, and a dosing algorithm that directs hormonal delivery. Preclinical optimization of dosing algorithms using computer simulations has the potential to accelerate the pace of development for this technology. However, current simulation environments consider glucose regulation models that either do not include glucagon action submodels or include submodels that were proposed without comparison to other candidate models. We consider here nine candidate models of glucagon action featuring a number of possible characteristics: insulin-independent glucagon action, insulin/glucagon ratio effect on hepatic glucose production, insulin-dependent suppression of glucagon action, and the effect of rate of change of glucagon. To assess the models, we use measurements of plasma insulin, plasma glucagon, and endogenous glucose production collected from experiments involving eight subjects with T1D who receive four subcutaneous glucagon boluses. We estimate each model's parameters using a Bayesian approach, and the models are contrasted based on the deviance information criterion. The model achieving the best fit features insulin-dependent suppression of glucagon action and incorporates effects of both glucagon levels and its rate of change.

A multiple imputation strategy for sequential multiple assignment randomized trials

Sequential multiple assignment randomized trials (SMARTs) are increasingly being used to inform clinical and intervention science. In a SMART, each patient is repeatedly randomized over time. Each randomization occurs at a critical decision point in the treatment course. These critical decision points often correspond to milestones in the disease process or other changes in a patient's health status. Thus, the timing and number of randomizations may vary across patients and depend on evolving patient-specific information. This presents unique challenges when analyzing data from a SMART in the presence of missing data. This paper presents the first comprehensive discussion of missing data issues typical of SMART studies: we describe five specific challenges and propose a flexible imputation strategy to facilitate valid statistical estimation and inference using incomplete data from a SMART. To illustrate these contributions, we consider data from the Clinical Antipsychotic Trial of Intervention and Effectiveness, one of the most well-known SMARTs to date.

Automatic Detection and Classification of Unsafe Events During Power Wheelchair Use

Using a powered wheelchair (PW) is a complex task requiring advanced perceptual and motor control skills. Unfortunately, PW incidents and accidents are not uncommon and their consequences can be serious. The objective of this paper is to develop technological tools that can be used to characterize a wheelchair user’s driving behavior under various settings. In the experiments conducted, PWs are outfitted with a datalogging platform that records, in real-time, the 3-D acceleration of the PW. Data collection was conducted over 35 different activities, designed to capture a spectrum of PW driving events performed at different speeds (collisions with fixed or moving objects, rolling on incline plane, and rolling across multiple types obstacles). The data was processed using time-series analysis and data mining techniques, to automatically detect and identify the different events. We compared the classification accuracy using four different types of time-series features: 1) time-delay embeddings; 2) time-domain characterization; 3) frequency-domain features; and 4) wavelet transforms. In the analysis, we compared the classification accuracy obtained when distinguishing between safe and unsafe events during each of the 35 different activities. For the purposes of this study, unsafe events were defined as activities containing collisions against objects at different speed, and the remainder were defined as safe events. We were able to accurately detect 98% of unsafe events, with a low (12%) false positive rate, using only five examples of each activity. This proof-of-concept study shows that the proposed approach has the potential of capturing, based on limited input from embedded sensors, contextual information on PW use, and of automatically characterizing a user’s PW driving behavior.

Policy Iteration Based on Stochastic Factorization

When a transition probability matrix is represented as the product of two stochastic matrices, one can swap the factors of the multiplication to obtain another transition matrix that retains some fundamental characteristics of the original. Since the derived matrix can be much smaller than its precursor, this property can be exploited to create a compact version of a Markov decision process (MDP), and hence to reduce the computational cost of dynamic programming. Building on this idea, this paper presents an approximate policy iteration algorithm called policy iteration based on stochastic factorization, or PISF for short. In terms of computational complexity, PISF replaces standard policy iteration's cubic dependence on the size of the MDP with a function that grows only linearly with the number of states in the model. The proposed algorithm also enjoys nice theoretical properties: it always terminates after a finite number of iterations and returns a decision policy whose performance only depends on the quality of the stochastic factorization. In particular, if the approximation error in the factorization is sufficiently small, PISF computes the optimal value function of the MDP. The paper also discusses practical ways of factoring an MDP and illustrates the usefulness of the proposed algorithm with an application involving a large-scale decision problem of real economical interest.

Design and validation of an intelligent wheelchair towards a clinically-functional outcome

BACKGROUND

Many people with mobility impairments, who require the use of powered wheelchairs, have difficulty completing basic maneuvering tasks during their activities of daily living (ADL). In order to provide assistance to this population, robotic and intelligent system technologies have been used to design an intelligent powered wheelchair (IPW). This paper provides a comprehensive overview of the design and validation of the IPW.

METHODS

The main contributions of this work are three-fold. First, we present a software architecture for robot navigation and control in constrained spaces. Second, we describe a decision-theoretic approach for achieving robust speech-based control of the intelligent wheelchair. Third, we present an evaluation protocol motivated by a meaningful clinical outcome, in the form of the Robotic Wheelchair Skills Test (RWST). This allows us to perform a thorough characterization of the performance and safety of the system, involving 17 test subjects (8 non-PW users, 9 regular PW users), 32 complete RWST sessions, 25 total hours of testing, and 9 kilometers of total running distance.

RESULTS

User tests with the RWST show that the navigation architecture reduced collisions by more than 60% compared to other recent intelligent wheelchair platforms. On the tasks of the RWST, we measured an average decrease of 4% in performance score and 3% in safety score (not statistically significant), compared to the scores obtained with conventional driving model. This analysis was performed with regular users that had over 6 years of wheelchair driving experience, compared to approximately one half-hour of training with the autonomous mode.

CONCLUSIONS

The platform tested in these experiments is among the most experimentally validated robotic wheelchairs in realistic contexts. The results establish that proficient powered wheelchair users can achieve the same level of performance with the intelligent command mode, as with the conventional command mode.

Time Series Analysis Using Geometric Template Matching

We present a novel framework for analyzing univariate time series data. At the heart of the approach is a versatile algorithm for measuring the similarity of two segments of time series called geometric template matching (GeTeM). First, we use GeTeM to compute a similarity measure for clustering and nearest-neighbor classification. Next, we present a semi-supervised learning algorithm that uses the similarity measure with hierarchical clustering in order to improve classification performance when unlabeled training data are available. Finally, we present a boosting framework called TDEBOOST, which uses an ensemble of GeTeM classifiers. TDEBOOST augments the traditional boosting approach with an additional step in which the features used as inputs to the classifier are adapted at each step to improve the training error. We empirically evaluate the proposed approaches on several datasets, such as accelerometer data collected from wearable sensors and ECG data.

Adaptive control of epileptiform excitability in an in vitro model of limbic seizures

Deep brain stimulation (DBS) is a promising tool for treating drug-resistant epileptic patients. Currently, the most common approach is fixed-frequency stimulation (periodic pacing) by means of stimulating devices that operate under open-loop control. However, a drawback of this DBS strategy is the impossibility of tailoring a personalized treatment, which also limits the optimization of the stimulating apparatus. Here, we propose a novel DBS methodology based on a closed-loop control strategy, developed by exploiting statistical machine learning techniques, in which stimulation parameters are adapted to the current neural activity thus allowing for seizure suppression that is fine-tuned on the individual scale (adaptive stimulation). By means of field potential recording from adult rat hippocampus-entorhinal cortex (EC) slices treated with the convulsant drug 4-aminopyridine we determined the effectiveness of this approach compared to low-frequency periodic pacing, and found that the closed-loop stimulation strategy: (i) has similar efficacy as low-frequency periodic pacing in suppressing ictal-like events but (ii) is more efficient than periodic pacing in that it requires less electrical pulses. We also provide evidence that the closed-loop stimulation strategy can alternatively be employed to tune the frequency of a periodic pacing strategy. Our findings indicate that the adaptive stimulation strategy may represent a novel, promising approach to DBS for individually-tailored epilepsy treatment.

[Computerized prescription for medical devices: examples, interest and limits]

INTRODUCTION

Related to the good practice contract implemented in hospitals, the prescription dedicated to medical devices, such as pharmaceuticals, could promote safety and good practice.

MATERIAL AND METHOD

We attempted to implement a computerized prescription for medical devices. In order to illustrate the method, two examples were selected: the Negative Pressure Wound Therapy (NPWT) and the Drug Eluting Stents (DES).

RESULTS

In partnership with the medical teams was elaborated a computerized protocol which included all the needed items for the good use of NPWT. For DES, a pre-existing questionnaire was used. We updated it in order to integrate new items such as the prescriber's name, the patient's name, the characteristics of the wound, the DES references and the indications.

DISCUSSION AND CONCLUSION

Computerized prescriptions for high-risk and expensive medical devices seem to be an interesting approach to guarantee the patient care safety and to reduce the budget impacts. In order to monitor the indications funded as fee-for-service medical devices, a prescription will emerge as a gold standard in the future in France. Eventually, this study highlights a new activity of clinical pharmacy for hospital pharmacists dealing with medical devices.

Mobility profile and wheelchair driving skills of powered wheelchair users: sensor-based event recognition using a support vector machine classifier

This paper presents a method to automatically recognize events and driving activities during the use of a powered wheelchair (PW). The method uses a support vector machine classifier, trained from sensor-based data from a datalogging platform installed on the PW. Data from a 3D accelerometer positioned on the back of the PW were collected in a laboratory space during PW driving tasks. 16-segmented events and driving activities (i.e. impacts from different side on different objects, rolling down or up on incline surface, going across threshold of different height) were performed repeatedly (n=25 trials) by one operator at three different speeds (slow, normal, high). We present results from an experiment aiming to classify five different events and driving activities from the sensor data acquired using the datalogging platform. Classification results show the ability of the proposed method to reliably segment 100% of events, and to identify the correct event type in 80% of events.

Non-Deterministic Policies in Markovian Decision Processes

Markovian processes have long been used to model stochastic environments. Reinforcement learning has emerged as a framework to solve sequential planning and decision-making problems in such environments. In recent years, attempts were made to apply methods from reinforcement learning to construct decision support systems for action selection in Markovian environments. Although conventional methods in reinforcement learning have proved to be useful in problems concerning sequential decision-making, they cannot be applied in their current form to decision support systems, such as those in medical domains, as they suggest policies that are often highly prescriptive and leave little room for the user's input. Without the ability to provide flexible guidelines, it is unlikely that these methods can gain ground with users of such systems. This paper introduces the new concept of non-deterministic policies to allow more flexibility in the user's decision-making process, while constraining decisions to remain near optimal solutions. We provide two algorithms to compute non-deterministic policies in discrete domains. We study the output and running time of these method on a set of synthetic and real-world problems. In an experiment with human subjects, we show that humans assisted by hints based on non-deterministic policies outperform both human-only and computer-only agents in a web navigation task.

Treating epilepsy via adaptive neurostimulation: a reinforcement learning approach

This paper presents a new methodology for automatically learning an optimal neurostimulation strategy for the treatment of epilepsy. The technical challenge is to automatically modulate neurostimulation parameters, as a function of the observed EEG signal, so as to minimize the frequency and duration of seizures. The methodology leverages recent techniques from the machine learning literature, in particular the reinforcement learning paradigm, to formalize this optimization problem. We present an algorithm which is able to automatically learn an adaptive neurostimulation strategy directly from labeled training data acquired from animal brain tissues. Our results suggest that this methodology can be used to automatically find a stimulation strategy which effectively reduces the incidence of seizures, while also minimizing the amount of stimulation applied. This work highlights the crucial role that modern machine learning techniques can play in the optimization of treatment strategies for patients with chronic disorders such as epilepsy.

AAAI 2008 Workshop Reports

AAAI was pleased to present the AAAI-08 Workshop Program, held Sunday and Monday, July 13–14, in Chicago, Illinois, USA. The program included the following 15 workshops: Advancements in POMDP Solvers; AI Education Workshop Colloquium; Coordination, Organizations, Institutions, and Norms in Agent Systems, Enhanced Messaging; Human Implications of Human-Robot Interaction; Intelligent Techniques for Web Personalization and Recommender Systems; Metareasoning: Thinking about Thinking; Multidisciplinary Workshop on Advances in Preference Handling; Search in Artificial Intelligence and Robotics; Spatial and Temporal Reasoning; Trading Agent Design and Analysis; Transfer Learning for Complex Tasks; What Went Wrong and Why: Lessons from AI Research and Applications; and Wikipedia and Artificial Intelligence: An Evolving Synergy.

MDPs with Non-Deterministic Policies

Markov Decision Processes (MDPs) have been extensively studied and used in the context of planning and decision-making, and many methods exist to find the optimal policy for problems modelled as MDPs. Although finding the optimal policy is sufficient in many domains, in certain applications such as decision support systems where the policy is executed by a human (rather than a machine), finding all possible near-optimal policies might be useful as it provides more flexibility to the person executing the policy. In this paper we introduce the new concept of non-deterministic MDP policies, and address the question of finding near-optimal non-deterministic policies. We propose two solutions to this problem, one based on a Mixed Integer Program and the other one based on a search algorithm. We include experimental results obtained from applying this framework to optimize treatment choices in the context of a medical decision support system.

Online Planning Algorithms for POMDPs

Partially Observable Markov Decision Processes (POMDPs) provide a rich framework for sequential decision-making under uncertainty in stochastic domains. However, solving a POMDP is often intractable except for small problems due to their complexity. Here, we focus on online approaches that alleviate the computational complexity by computing good local policies at each decision step during the execution. Online algorithms generally consist of a lookahead search to find the best action to execute at each time step in an environment. Our objectives here are to survey the various existing online POMDP methods, analyze their properties and discuss their advantages and disadvantages; and to thoroughly evaluate these online approaches in different environments under various metrics (return, error bound reduction, lower bound improvement). Our experimental results indicate that state-of-the-art online heuristic search methods can handle large POMDP domains efficiently.

Model-Based Bayesian Reinforcement Learning in Large Structured Domains

Model-based Bayesian reinforcement learning has generated significant interest in the AI community as it provides an elegant solution to the optimal exploration-exploitation tradeoff in classical reinforcement learning. Unfortunately, the applicability of this type of approach has been limited to small domains due to the high complexity of reasoning about the joint posterior over model parameters. In this paper, we consider the use of factored representations combined with online planning techniques, to improve scalability of these methods. The main contribution of this paper is a Bayesian framework for learning the structure and parameters of a dynamical system, while also simultaneously planning a (near-)optimal sequence of actions.

Online Planning Algorithms for POMDPs

Partially Observable Markov Decision Processes (POMDPs) provide a rich framework for sequential decision-making under uncertainty in stochastic domains. However, solving a POMDP is often intractable except for small problems due to their complexity. Here, we focus on online approaches that alleviate the computational complexity by computing good local policies at each decision step during the execution. Online algorithms generally consist of a lookahead search to find the best action to execute at each time step in an environment. Our objectives here are to survey the various existing online POMDP methods, analyze their properties and discuss their advantages and disadvantages; and to thoroughly evaluate these online approaches in different environments under various metrics (return, error bound reduction, lower bound improvement). Our experimental results indicate that state-of-the-art online heuristic search methods can handle large POMDP domains efficiently.

A Variance Analysis for POMDP Policy Evaluation

Partially Observable Markov Decision Processes have been studied widely as a model for decision making under uncertainty, and a number of methods have been developed to find the solutions for such processes. Such studies often involve calculation of the value function of a specific policy, given a model of the transition and observation probabilities, and the reward. These models can be learned using labeled samples of on-policy trajectories. However, when using empirical models, some bias and variance terms are introduced into the value function as a result of imperfect models. In this paper, we propose a method for estimating the bias and variance of the value function in terms of the statistics of the empirical transition and observation model. Such error terms can be used to meaningfully compare the value of different policies. This is an important result for sequential decision-making, since it will allow us to provide more formal guarantees about the quality of the policies we implement. To evaluate the precision of the proposed method, we provide supporting experiments on problems from the field of robotics and medical decision making.

Theoretical Analysis of Heuristic Search Methods for Online POMDPs

Planning in partially observable environments remains a challenging problem, despite significant recent advances in offline approximation techniques. A few online methods have also been proposed recently, and proven to be remarkably scalable, but without the theoretical guarantees of their offline counterparts. Thus it seems natural to try to unify offline and online techniques, preserving the theoretical properties of the former, and exploiting the scalability of the latter. In this paper, we provide theoretical guarantees on an anytime algorithm for POMDPs which aims to reduce the error made by approximate offline value iteration algorithms through the use of an efficient online searching procedure. The algorithm uses search heuristics based on an error analysis of lookahead search, to guide the online search towards reachable beliefs with the most potential to reduce error. We provide a general theorem showing that these search heuristics are admissible, and lead to complete and ε-optimal algorithms. This is, to the best of our knowledge, the strongest theoretical result available for online POMDP solution methods. We also provide empirical evidence showing that our approach is also practical, and can find (provably) near-optimal solutions in reasonable time.

Constructing evidence-based treatment strategies using methods from computer science

This paper details a new methodology, instance-based reinforcement learning, for constructing adaptive treatment strategies from randomized trials. Adaptive treatment strategies are operationalized clinical guidelines which recommend the next best treatment for an individual based on his/her personal characteristics and response to earlier treatments. The instance-based reinforcement learning methodology comes from the computer science literature, where it was developed to optimize sequences of actions in an evolving, time varying system. When applied in the context of treatment design, this method provides the means to evaluate both the therapeutic and diagnostic effects of treatments in constructing an adaptive treatment strategy. The methodology is illustrated with data from the STAR*D trial, a multi-step randomized study of treatment alternatives for individuals with treatment-resistant major depressive disorder.

Anytime Point-Based Approximations for Large POMDPs

The Partially Observable Markov Decision Process has long been recognized as a rich framework for real-world planning and control problems, especially in robotics. However exact solutions in this framework are typically computationally intractable for all but the smallest problems. A well-known technique for speeding up POMDP solving involves performing value backups at specific belief points, rather than over the entire belief simplex. The efficiency of this approach, however, depends greatly on the selection of points. This paper presents a set of novel techniques for selecting informative belief points which work well in practice. The point selection procedure is combined with point-based value backups to form an effective anytime POMDP algorithm called Point-Based Value Iteration (PBVI). The first aim of this paper is to introduce this algorithm and present a theoretical analysis justifying the choice of belief selection technique. The second aim of this paper is to provide a thorough empirical comparison between PBVI and other state-of-the-art POMDP methods, in particular the Perseus algorithm, in an effort to highlight their similarities and differences. Evaluation is performed using both standard POMDP domains and realistic robotic tasks.