Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack

A survey on clinical natural language processing in the United Kingdom from 2007 to 2022

Much of the knowledge and information needed for enabling high-quality clinical research is stored in free-text format. Natural language processing (NLP) has been used to extract information from these sources at scale for several decades. This paper aims to present a comprehensive review of clinical NLP for the past 15 years in the UK to identify the community, depict its evolution, analyse methodologies and applications, and identify the main barriers. We collect a dataset of clinical NLP projects (n = 94; £ = 41.97 m) funded by UK funders or the European Union's funding programmes. Additionally, we extract details on 9 funders, 137 organisations, 139 persons and 431 research papers. Networks are created from timestamped data interlinking all entities, and network analysis is subsequently applied to generate insights. 431 publications are identified as part of a literature review, of which 107 are eligible for final analysis. Results show, not surprisingly, clinical NLP in the UK has increased substantially in the last 15 years: the total budget in the period of 2019-2022 was 80 times that of 2007-2010. However, the effort is required to deepen areas such as disease (sub-)phenotyping and broaden application domains. There is also a need to improve links between academia and industry and enable deployments in real-world settings for the realisation of clinical NLP's great potential in care delivery. The major barriers include research and development access to hospital data, lack of capable computational resources in the right places, the scarcity of labelled data and barriers to sharing of pretrained models.

Ethical considerations for precision psychiatry: A roadmap for research and clinical practice

Precision psychiatry is an emerging field with transformative opportunities for mental health. However, the use of clinical prediction models carries unprecedented ethical challenges, which must be addressed before accessing the potential benefits of precision psychiatry. This critical review covers multidisciplinary areas, including psychiatry, ethics, statistics and machine-learning, healthcare and academia, as well as input from people with lived experience of mental disorders, their family, and carers. We aimed to identify core ethical considerations for precision psychiatry and mitigate concerns by designing a roadmap for research and clinical practice. We identified priorities: learning from somatic medicine; identifying precision psychiatry use cases; enhancing transparency and generalizability; fostering implementation; promoting mental health literacy; communicating risk estimates; data protection and privacy; and fostering the equitable distribution of mental health care. We hope this blueprint will advance research and practice and enable people with mental health problems to benefit from precision psychiatry.

Clinical prediction models in psychiatry: a systematic review of two decades of progress and challenges

Recent years have seen the rapid proliferation of clinical prediction models aiming to support risk stratification and individualized care within psychiatry. Despite growing interest, attempts to synthesize current evidence in the nascent field of precision psychiatry have remained scarce. This systematic review therefore sought to summarize progress towards clinical implementation of prediction modeling for psychiatric outcomes. We searched MEDLINE, PubMed, Embase, and PsychINFO databases from inception to September 30, 2020, for English-language articles that developed and/or validated multivariable models to predict (at an individual level) onset, course, or treatment response for non-organic psychiatric disorders (PROSPERO: CRD42020216530). Individual prediction models were evaluated based on three key criteria: (i) mitigation of bias and overfitting; (ii) generalizability, and (iii) clinical utility. The Prediction model Risk Of Bias ASsessment Tool (PROBAST) was used to formally appraise each study's risk of bias. 228 studies detailing 308 prediction models were ultimately eligible for inclusion. 94.5% of developed prediction models were deemed to be at high risk of bias, largely due to inadequate or inappropriate analytic decisions. Insufficient internal validation efforts (within the development sample) were also observed, while only one-fifth of models underwent external validation in an independent sample. Finally, our search identified just one published model whose potential utility in clinical practice was formally assessed. Our findings illustrated significant growth in precision psychiatry with promising progress towards real-world application. Nevertheless, these efforts have been inhibited by a preponderance of bias and overfitting, while the generalizability and clinical utility of many published models has yet to be formally established. Through improved methodological rigor during initial development, robust evaluations of reproducibility via independent validation, and evidence-based implementation frameworks, future research has the potential to generate risk prediction tools capable of enhancing clinical decision-making in psychiatric care.

Patient and public involvement to build trust in artificial intelligence: A framework, tools, and case studies

Artificial intelligence (AI) is increasingly taking on a greater role in healthcare. However, hype and negative news reports about AI abound. Integrating patient and public involvement (PPI) in healthcare AI projects may help in adoption and acceptance of these technologies.

We argue that AI algorithms should also be co-designed with patients and healthcare workers.

We specifically suggest (1) including patients with lived experience of the disease, and (2) creating a research advisory group (RAG) and using these group meetings to walk patients through the process of AI model building, starting with simple (e.g., linear) models.

We present a framework, case studies, best practices, and tools for applying participative data science to healthcare, enabling data scientists, clinicians, and patients to work together. The strategy of co-designing with patients can help set more realistic expectations for all stakeholders, since conventional narratives of AI revolve around dystopia or limitless optimism.

Hype and negative news reports about artificial intelligence (AI) abound. Involving patients in healthcare AI projects may help in adoption and acceptance of these technologies. We argue that AI algorithms should be co-designed with patients and healthcare workers.

We show examples of how to involve patients in AI research and how patients can build trust in algorithms. We share some best practices, case studies, a framework, and computational tools.

Avenues for future work include guidelines for patient and public involvement in AI healthcare research for funding bodies and regulatory agencies.

An understanding of what AI can and cannot do, and a realistic appraisal of risks and benefits, may help in adoption and democratize access to AI for healthcare.

Building a Precision Medicine Delivery Platform for Clinics: The University of California, San Francisco, BRIDGE Experience

Despite an ever-expanding number of analytics with the potential to impact clinical care, the field currently lacks point-of-care technological tools that allow clinicians to efficiently select disease-relevant data about their patients, algorithmically derive clinical indices (eg, risk scores), and view these data in straightforward graphical formats to inform real-time clinical decisions. Thus far, solutions to this problem have relied on either bottom-up approaches that are limited to a single clinic or generic top-down approaches that do not address clinical users’ specific setting-relevant or disease-relevant needs. As a road map for developing similar platforms, we describe our experience with building a custom but institution-wide platform that enables economies of time, cost, and expertise. The BRIDGE platform was designed to be modular and scalable and was customized to data types relevant to given clinical contexts within a major university medical center. The development process occurred by using a series of human-centered design phases with extensive, consistent stakeholder input. This institution-wide approach yielded a unified, carefully regulated, cross-specialty clinical research platform that can be launched during a patient’s electronic health record encounter. The platform pulls clinical data from the electronic health record (Epic; Epic Systems) as well as other clinical and research sources in real time; analyzes the combined data to derive clinical indices; and displays them in simple, clinician-designed visual formats specific to each disorder and clinic. By integrating an application into the clinical workflow and allowing clinicians to access data sources that would otherwise be cumbersome to assemble, view, and manipulate, institution-wide platforms represent an alternative approach to achieving the vision of true personalized medicine.

Diagnosis, prognosis, and treatment of brief psychotic episodes: a review and research agenda

Brief psychotic episodes represent an intriguing paradox in clinical psychiatry because they elude the standard knowledge that applies to the persisting psychotic disorders such as schizophrenia. This Review describes key diagnostic considerations such as conceptual foundations, current psychiatric classification versus research-based operationalisations, epidemiology, and sociocultural variations; prognostic aspects including the risk of psychosis recurrence, types of psychotic recurrences, other clinical outcomes, prognostic factors; and therapeutic issues such as treatment guidelines and unmet need of care. The advances and challenges associated with the scientific evidence are used to set a research agenda in this area. We conclude that brief psychotic episodes can be reconceptualised within a clinical staging model to promote innovative translational research and improve our understanding and treatment of psychotic disorders.

A class-contrastive human-interpretable machine learning approach to predict mortality in severe mental illness

Machine learning (ML), one aspect of artificial intelligence (AI), involves computer algorithms that train themselves. They have been widely applied in the healthcare domain. However, many trained ML algorithms operate as ‘black boxes’, producing a prediction from input data without a clear explanation of their workings. Non-transparent predictions are of limited utility in many clinical domains, where decisions must be justifiable. Here, we apply class-contrastive counterfactual reasoning to ML to demonstrate how specific changes in inputs lead to different predictions of mortality in people with severe mental illness (SMI), a major public health challenge. We produce predictions accompanied by visual and textual explanations as to how the prediction would have differed given specific changes to the input. We apply it to routinely collected data from a mental health secondary care provider in patients with schizophrenia. Using a data structuring framework informed by clinical knowledge, we captured information on physical health, mental health, and social predisposing factors. We then trained an ML algorithm and other statistical learning techniques to predict the risk of death. The ML algorithm predicted mortality with an area under receiver operating characteristic curve (AUROC) of 0.80 (95% confidence intervals [0.78, 0.82]). We used class-contrastive analysis to produce explanations for the model predictions. We outline the scenarios in which class-contrastive analysis is likely to be successful in producing explanations for model predictions. Our aim is not to advocate for a particular model but show an application of the class-contrastive analysis technique to electronic healthcare record data for a disease of public health significance. In patients with schizophrenia, our work suggests that use or prescription of medications like antidepressants was associated with lower risk of death. Abuse of alcohol/drugs and a diagnosis of delirium were associated with higher risk of death. Our ML models highlight the role of co-morbidities in determining mortality in patients with schizophrenia and the need to manage co-morbidities in these patients. We hope that some of these bio-social factors can be targeted therapeutically by either patient-level or service-level interventions. Our approach combines clinical knowledge, health data, and statistical learning, to make predictions interpretable to clinicians using class-contrastive reasoning. This is a step towards interpretable AI in the management of patients with schizophrenia and potentially other diseases.

Using Natural Language Processing on Electronic Health Records to Enhance Detection and Prediction of Psychosis Risk

BACKGROUND

Using novel data mining methods such as natural language processing (NLP) on electronic health records (EHRs) for screening and detecting individuals at risk for psychosis.

METHOD

The study included all patients receiving a first index diagnosis of nonorganic and nonpsychotic mental disorder within the South London and Maudsley (SLaM) NHS Foundation Trust between January 1, 2008, and July 28, 2018. Least Absolute Shrinkage and Selection Operator (LASSO)-regularized Cox regression was used to refine and externally validate a refined version of a five-item individualized, transdiagnostic, clinically based risk calculator previously developed (Harrell's C = 0.79) and piloted for implementation. The refined version included 14 additional NLP-predictors: tearfulness, poor appetite, weight loss, insomnia, cannabis, cocaine, guilt, irritability, delusions, hopelessness, disturbed sleep, poor insight, agitation, and paranoia.

RESULTS

A total of 92 151 patients with a first index diagnosis of nonorganic and nonpsychotic mental disorder within the SLaM Trust were included in the derivation (n = 28 297) or external validation (n = 63 854) data sets. Mean age was 33.6 years, 50.7% were women, and 67.0% were of white race/ethnicity. Mean follow-up was 1590 days. The overall 6-year risk of psychosis in secondary mental health care was 3.4 (95% CI, 3.3-3.6). External validation indicated strong performance on unseen data (Harrell's C 0.85, 95% CI 0.84-0.86), an increase of 0.06 from the original model.

CONCLUSIONS

Using NLP on EHRs can considerably enhance the prognostic accuracy of psychosis risk calculators. This can help identify patients at risk of psychosis who require assessment and specialized care, facilitating earlier detection and potentially improving patient outcomes.

Establishing a clinical service to prevent psychosis: What, how and when? Systematic review

The first rate-limiting step to successfully translate prevention of psychosis in to clinical practice is to establish specialised Clinical High Risk for Psychosis (CHR-P) services. This study systematises the knowledge regarding CHR-P services and provides guidelines for translational implementation. We conducted a PRISMA/MOOSE-compliant (PROSPERO-CRD42020163640) systematic review of Web of Science to identify studies until 4/05/2020 reporting on CHR-P service configuration, outreach strategy and referrals, service user characteristics, interventions, and outcomes. Fifty-six studies (1998-2020) were included, encompassing 51 distinct CHR-P services across 15 countries and a catchment area of 17,252,666 people. Most services (80.4%) consisted of integrated multidisciplinary teams taking care of CHR-P and other patients. Outreach encompassed active (up to 97.6%) or passive (up to 63.4%) approaches: referrals came mostly (90%) from healthcare agencies. CHR-P individuals were more frequently males (57.2%). Most (70.6%) services accepted individuals aged 12-35 years, typically assessed with the CAARMS/SIPS (83.7%). Baseline comorbid mental conditions were reported in two-third (69.5%) of cases, and unemployment in one third (36.6%). Most services provided up to 2-years (72.4%), of clinical monitoring (100%), psychoeducation (81.1%), psychosocial support (73%), family interventions (73%), individual (67.6%) and group (18.9%) psychotherapy, physical health interventions (37.8%), antipsychotics (87.1%), antidepressants (74.2%), anxiolytics (51.6%), and mood stabilisers (38.7%). Outcomes were more frequently ascertained clinically (93.0%) and included: persistence of symptoms/comorbidities (67.4%), transition to psychosis (53.5%), and functional status (48.8%). We provide ten practical recommendations for implementation of CHR-P services. Health service knowledge summarised by the current study will facilitate translational efforts for implementation of CHR-P services worldwide.

Antipsychotic use in dementia: the relationship between neuropsychiatric symptom profiles and adverse outcomes

Antipsychotic treatments are associated with safety concerns in people with dementia. The authors aimed to investigate whether risk of adverse outcomes related to antipsychotic prescribing differed according to major neuropsychiatric syndromes-specifically psychosis, agitation, or a combination. A cohort of 10,106 patients with a diagnosis of dementia was assembled from a large dementia care database in South East London. Neuropsychiatric symptoms closest to first dementia diagnosis were determined according to the Health of the Nation Outcome Scales' mental and behavioural problem scores and the sample was divided into four groups: 'agitation and psychosis', 'agitation, but no psychosis', 'psychosis, but no agitation', and 'neither psychosis nor agitation'. Antipsychotic prescription in a one-year window around first dementia diagnosis was ascertained as exposure variable through natural language processing from free text. Cox regression models were used to analyse associations of antipsychotic prescription with all-cause and stroke-specific mortality, emergency hospitalisation and hospitalised stroke adjusting for sixteen potential confounders including demographics, cognition, functioning, as well as physical and mental health. Only in the group 'psychosis, but no agitation' (n = 579), 30% of whom were prescribed an antipsychotic, a significant antipsychotic-associated increased risk of hospitalised stroke was present after adjustment (adjusted hazard ratio (HR) 2.16; 95% confidence interval (CI) 1.09-4.25). An increased antipsychotic-related all-cause (adjusted HR 1.14; 95% CI 1.04-1.24) and stroke-specific mortality risk (adjusted HR 1.28; 95% CI 1.01-1.63) was detected in the whole sample, but no interaction between the strata and antipsychotic-related mortality. In conclusion, the adverse effects of antipsychotics in dementia are complex. Stroke risk may be highest when used in patients presenting with psychosis without agitation, indicating the need for novel interventions for this group.

Real-world implementation of precision psychiatry: Transdiagnostic risk calculator for the automatic detection of individuals at-risk of psychosis

BACKGROUND

Risk estimation models integrated into Electronic Health Records (EHRs) can deliver innovative approaches in psychiatry, but clinicians' endorsement and their real-world usability are unknown. This study aimed to investigate the real-world feasibility of implementing an individualised, transdiagnostic risk calculator to automatically screen EHRs and detect individuals at-risk for psychosis.

METHODS

Feasibility implementation study encompassing an in-vitro phase (March 2018 to May 2018) and in-vivo phase (May 2018 to April 2019). The in-vitro phase addressed implementation barriers and embedded the risk calculator (predictors: age, gender, ethnicity, index cluster diagnosis, age*gender) into the local EHR. The in-vivo phase investigated the real-world feasibility of screening individuals accessing secondary mental healthcare at the South London and Maudsley NHS Trust. The primary outcome was adherence of clinicians to automatic EHR screening, defined by the proportion of clinicians who responded to alerts from the risk calculator, over those contacted.

RESULTS

In-vitro phase: implementation barriers were identified/overcome with clinician and service user engagement, and the calculator was successfully integrated into the local EHR through the CogStack platform. In-vivo phase: 3722 individuals were automatically screened and 115 were detected. Clinician adherence was 74% without outreach and 85% with outreach. One-third of clinicians responded to the first email (37.1%) or phone calls (33.7%). Among those detected, cumulative risk of developing psychosis was 12% at six-month follow-up.

CONCLUSION

This is the first implementation study suggesting that combining precision psychiatry and EHR methods to improve detection of individuals with emerging psychosis is feasible. Future psychiatric implementation research is urgently needed.

Implementing Precision Psychiatry: A Systematic Review of Individualized Prediction Models for Clinical Practice

BACKGROUND

The impact of precision psychiatry for clinical practice has not been systematically appraised. This study aims to provide a comprehensive review of validated prediction models to estimate the individual risk of being affected with a condition (diagnostic), developing outcomes (prognostic), or responding to treatments (predictive) in mental disorders.

METHODS

PRISMA/RIGHT/CHARMS-compliant systematic review of the Web of Science, Cochrane Central Register of Reviews, and Ovid/PsycINFO databases from inception until July 21, 2019 (PROSPERO CRD42019155713) to identify diagnostic/prognostic/predictive prediction studies that reported individualized estimates in psychiatry and that were internally or externally validated or implemented. Random effect meta-regression analyses addressed the impact of several factors on the accuracy of prediction models.

FINDINGS

Literature search identified 584 prediction modeling studies, of which 89 were included. 10.4% of the total studies included prediction models internally validated (n = 61), 4.6% models externally validated (n = 27), and 0.2% (n = 1) models considered for implementation. Across validated prediction modeling studies (n = 88), 18.2% were diagnostic, 68.2% prognostic, and 13.6% predictive. The most frequently investigated condition was psychosis (36.4%), and the most frequently employed predictors clinical (69.5%). Unimodal compared to multimodal models (β = .29, P = .03) and diagnostic compared to prognostic (β = .84, p < .0001) and predictive (β = .87, P = .002) models were associated with increased accuracy.

INTERPRETATION

To date, several validated prediction models are available to support the diagnosis and prognosis of psychiatric conditions, in particular, psychosis, or to predict treatment response. Advancements of knowledge are limited by the lack of implementation research in real-world clinical practice. A new generation of implementation research is required to address this translational gap.