Machine learning in medicine: a practical introduction to natural language processing

Controlling Inputter Variability in Vignette Studies Assessing Web-Based Symptom Checkers: Evaluation of Current Practice and Recommendations for Isolated Accuracy Metrics

BACKGROUND

The rapid growth of web-based symptom checkers (SCs) is not matched by advances in quality assurance. Currently, there are no widely accepted criteria assessing SCs' performance. Vignette studies are widely used to evaluate SCs, measuring the accuracy of outcome. Accuracy behaves as a composite metric as it is affected by a number of individual SC- and tester-dependent factors. In contrast to clinical studies, vignette studies have a small number of testers. Hence, measuring accuracy alone in vignette studies may not provide a reliable assessment of performance due to tester variability.

OBJECTIVE

This study aims to investigate the impact of tester variability on the accuracy of outcome of SCs, using clinical vignettes. It further aims to investigate the feasibility of measuring isolated aspects of performance.

METHODS

Healthily's SC was assessed using 114 vignettes by 3 groups of 3 testers who processed vignettes with different instructions: free interpretation of vignettes (free testers), specified chief complaints (partially free testers), and specified chief complaints with strict instruction for answering additional symptoms (restricted testers). κ statistics were calculated to assess agreement of top outcome condition and recommended triage. Crude and adjusted accuracy was measured against a gold standard. Adjusted accuracy was calculated using only results of consultations identical to the vignette, following a review and selection process. A feasibility study for assessing symptom comprehension of SCs was performed using different variations of 51 chief complaints across 3 SCs.

RESULTS

Intertester agreement of most likely condition and triage was, respectively, 0.49 and 0.51 for the free tester group, 0.66 and 0.66 for the partially free group, and 0.72 and 0.71 for the restricted group. For the restricted group, accuracy ranged from 43.9% to 57% for individual testers, averaging 50.6% (SD 5.35%). Adjusted accuracy was 56.1%. Assessing symptom comprehension was feasible for all 3 SCs. Comprehension scores ranged from 52.9% and 68%.

CONCLUSIONS

We demonstrated that by improving standardization of the vignette testing process, there is a significant improvement in the agreement of outcome between testers. However, significant variability remained due to uncontrollable tester-dependent factors, reflected by varying outcome accuracy. Tester-dependent factors, combined with a small number of testers, limit the reliability and generalizability of outcome accuracy when used as a composite measure in vignette studies. Measuring and reporting different aspects of SC performance in isolation provides a more reliable assessment of SC performance. We developed an adjusted accuracy measure using a review and selection process to assess data algorithm quality. In addition, we demonstrated that symptom comprehension with different input methods can be feasibly compared. Future studies reporting accuracy need to apply vignette testing standardization and isolated metrics.

Advancing medical imaging with language models: featuring a spotlight on ChatGPT

This review paper aims to serve as a comprehensive guide and instructional resource for researchers seeking to effectively implement language models in medical imaging research. First, we presented the fundamental principles and evolution of language models, dedicating particular attention to large language models. We then reviewed the current literature on how language models are being used to improve medical imaging, emphasizing a range of applications such as image captioning, report generation, report classification, findings extraction, visual question response systems, interpretable diagnosis and so on. Notably, the capabilities of ChatGPT were spotlighted for researchers to explore its further applications. Furthermore, we covered the advantageous impacts of accurate and efficient language models in medical imaging analysis, such as the enhancement of clinical workflow efficiency, reduction of diagnostic errors, and assistance of clinicians in providing timely and accurate diagnoses. Overall, our goal is to have better integration of language models with medical imaging, thereby inspiring new ideas and innovations. It is our aspiration that this review can serve as a useful resource for researchers in this field, stimulating continued investigative and innovative pursuits of the application of language models in medical imaging.

Use of Natural Language Understanding to Facilitate Surgical De-Escalation of Axillary Staging in Patients With Breast Cancer

PURPOSE

Natural language understanding (NLU) may be particularly well equipped for enhanced data capture from the electronic health record given its examination of both content-driven and context-driven extraction.

METHODS

We developed and applied a NLU model to examine rates of pathological node positivity (pN+) and rates of lymphedema to determine whether omission of routine axillary staging could be extended to younger patients with estrogen receptor-positive (ER+)/cN0 disease.

RESULTS

We found that rates of pN+ and arm lymphedema were similar between patients age 55-69 years and ≥70 years, with rates of lymphedema exceeding rates of pN+ for clinical stage T1c and smaller disease.

CONCLUSION

Data from our NLU model suggest that omission of sentinel lymph node biopsy might be extended beyond Choosing Wisely recommendations, limited to those older than 70 years and to all postmenopausal women with early-stage ER+/cN0 disease. These data support the recently reported SOUND trial results and provide additional granularity to facilitate surgical de-escalation.

Application of natural language processing to predict final recommendation of Brazilian health technology assessment reports

INTRODUCTION

Health technology assessment (HTA) plays a vital role in healthcare decision-making globally, necessitating the identification of key factors impacting evaluation outcomes due to the significant workload faced by HTA agencies.

OBJECTIVES

The aim of this study was to predict the approval status of evaluations conducted by the Brazilian Committee for Health Technology Incorporation (CONITEC) using natural language processing (NLP).

METHODS

Data encompassing CONITEC's official report summaries from 2012 to 2022. Textual data was tokenized for NLP analysis. Least Absolute Shrinkage and Selection Operator, logistic regression, support vector machine, random forest, neural network, and extreme gradient boosting (XGBoost), were evaluated for accuracy, area under the receiver operating characteristic curve (ROC AUC) score, precision, and recall. Cluster analysis using the k-modes algorithm categorized entries into two clusters (approved, rejected).

RESULTS

The neural network model exhibited the highest accuracy metrics (precision at 0.815, accuracy at 0.769, ROC AUC at 0.871, and recall at 0.746), followed by XGBoost model. The lexical analysis uncovered linguistic markers, like references to international HTA agencies' experiences and government as demandant, potentially influencing CONITEC's decisions. Cluster and XGBoost analyses emphasized that approved evaluations mainly concerned drug assessments, often government-initiated, while non-approved ones frequently evaluated drugs, with the industry as the requester.

CONCLUSIONS

NLP model can predict health technology incorporation outcomes, opening avenues for future research using HTA reports from other agencies. This model has the potential to enhance HTA system efficiency by offering initial insights and decision-making criteria, thereby benefiting healthcare experts.

Identifying signs and symptoms of urinary tract infection from emergency department clinical notes using large language models

BACKGROUND

Natural language processing (NLP) tools including recently developed large language models (LLMs) have myriad potential applications in medical care and research, including the efficient labeling and classification of unstructured text such as electronic health record (EHR) notes. This opens the door to large-scale projects that rely on variables that are not typically recorded in a structured form, such as patient signs and symptoms.

OBJECTIVES

This study is designed to acquaint the emergency medicine research community with the foundational elements of NLP, highlighting essential terminology, annotation methodologies, and the intricacies involved in training and evaluating NLP models. Symptom characterization is critical to urinary tract infection (UTI) diagnosis, but identification of symptoms from the EHR has historically been challenging, limiting large-scale research, public health surveillance, and EHR-based clinical decision support. We therefore developed and compared two NLP models to identify UTI symptoms from unstructured emergency department (ED) notes.

METHODS

The study population consisted of patients aged ≥ 18 who presented to an ED in a northeastern U.S. health system between June 2013 and August 2021 and had a urinalysis performed. We annotated a random subset of 1250 ED clinician notes from these visits for a list of 17 UTI symptoms. We then developed two task-specific LLMs to perform the task of named entity recognition: a convolutional neural network-based model (SpaCy) and a transformer-based model designed to process longer documents (Clinical Longformer). Models were trained on 1000 notes and tested on a holdout set of 250 notes. We compared model performance (precision, recall, F1 measure) at identifying the presence or absence of UTI symptoms at the note level.

RESULTS

A total of 8135 entities were identified in 1250 notes; 83.6% of notes included at least one entity. Overall F1 measure for note-level symptom identification weighted by entity frequency was 0.84 for the SpaCy model and 0.88 for the Longformer model. F1 measure for identifying presence or absence of any UTI symptom in a clinical note was 0.96 (232/250 correctly classified) for the SpaCy model and 0.98 (240/250 correctly classified) for the Longformer model.

CONCLUSIONS

The study demonstrated the utility of LLMs and transformer-based models in particular for extracting UTI symptoms from unstructured ED clinical notes; models were highly accurate for detecting the presence or absence of any UTI symptom on the note level, with variable performance for individual symptoms.

Natural language processing of online support group postings reveals patients' perspectives on strategies for managing psoriasis

Psoriasis is a chronic skin disorder, and patients encounter high physical and psychosocial burdens. Social media forums feature extensive patient-generated comments. We hypothesized that analyzing patient-posted comments using natural language processing would provide insights into patient engagements, sentiments, concerns, and support, which are vital for the holistic management of psoriasis. We collected 32,000 active user comments posted on Reddit. We applied Latent Dirichlet Allocation to categorize posts into popular topics and employed spectral clustering to establish cohesive themes and word representation frequency within these topics. We sorted posts into 29 significant topics of discussion and categorized them into four categories: management (37.48%), emotion (21.57%), presentation (19.79%), and others (3.57%). The frequent posts on management were diet (7.23%), biologics (6.95%), and adverse effects (3.88%). The emotion category comprised negative sentiments (11.02%), encouragement (5.49%), and gratitude (5.06%). The presentation topic included a discussion of scalp (5.69%), flare-timing (3.63%), and arthritis (2.64%). Others comprised differential diagnosis (5.01%), leaky gut (4.12%), and referrals (3.70%). This study identified patients' experiences and perspectives associated with psoriasis, which should be considered to tailor support systems to improve their quality of life.

Predicting Response to In-Hospital Pulmonary Rehabilitation in Individuals Recovering From Exacerbations of Chronic Obstructive Pulmonary Disease

BACKGROUND

Predicting the response to pulmonary rehabilitation (PR) could be valuable in defining admission priorities. We aimed to investigate whether the response of individuals recovering from a COPD exacerbation (ECOPD) could be forecasted using machine learning approaches.

METHOD

This multicenter, retrospective study recorded data on anthropometrics, demographics, physiological characteristics, post-PR changes in six-minute walking distance test (6MWT), Medical Research Council scale for dyspnea (MRC), Barthel Index dyspnea (BId), COPD assessment test (CAT) and proportion of participants reaching the minimal clinically important difference (MCID). The ability of multivariate approaches (linear regression, quantile regression, regression trees, and conditional inference trees) in predicting changes in each outcome measure has been assessed.

RESULTS

Individuals with lower baseline 6MWT, as well as those with less severe airway obstruction or admitted from acute care hospitals, exhibited greater improvements in 6MWT, whereas older as well as more dyspnoeic individuals had a lower forecasted improvement. Individuals with more severe CAT and dyspnea, and lower 6MWT had a greater potential improvement in CAT. More dyspnoeic individuals were also more likely to show improvement in BId and MRC. The Mean Absolute Error estimates of change prediction were 44.70m, 3.22 points, 5.35 points, and 0.32 points for 6MWT, CAT, BId, and MRC respectively. Sensitivity and specificity in discriminating individuals reaching the MCID of outcomes ranged from 61.78% to 98.99% and from 14.00% to 71.20%, respectively.

CONCLUSION

While the assessed models were not entirely satisfactory, predictive equations derived from clinical practice data might help in forecasting the response to PR in individuals recovering from an ECOPD. Future larger studies will be essential to confirm the methodology, variables, and utility.

Risk Management and Patient Safety in the Artificial Intelligence Era: A Systematic Review

BACKGROUND

Healthcare systems represent complex organizations within which multiple factors (physical environment, human factor, technological devices, quality of care) interconnect to form a dense network whose imbalance is potentially able to compromise patient safety. In this scenario, the need for hospitals to expand reactive and proactive clinical risk management programs is easily understood, and artificial intelligence fits well in this context. This systematic review aims to investigate the state of the art regarding the impact of AI on clinical risk management processes. To simplify the analysis of the review outcomes and to motivate future standardized comparisons with any subsequent studies, the findings of the present review will be grouped according to the possibility of applying AI in the prevention of the different incident type groups as defined by the ICPS.

MATERIALS AND METHODS

On 3 November 2023, a systematic review of the literature according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines was carried out using the SCOPUS and Medline (via PubMed) databases. A total of 297 articles were identified. After the selection process, 36 articles were included in the present systematic review.

RESULTS AND DISCUSSION

The studies included in this review allowed for the identification of three main "incident type" domains: clinical process, healthcare-associated infection, and medication. Another relevant application of AI in clinical risk management concerns the topic of incident reporting.

CONCLUSIONS

This review highlighted that AI can be applied transversely in various clinical contexts to enhance patient safety and facilitate the identification of errors. It appears to be a promising tool to improve clinical risk management, although its use requires human supervision and cannot completely replace human skills. To facilitate the analysis of the present review outcome and to enable comparison with future systematic reviews, it was deemed useful to refer to a pre-existing taxonomy for the identification of adverse events. However, the results of the present study highlighted the usefulness of AI not only for risk prevention in clinical practice, but also in improving the use of an essential risk identification tool, which is incident reporting. For this reason, the taxonomy of the areas of application of AI to clinical risk processes should include an additional class relating to risk identification and analysis tools. For this purpose, it was considered convenient to use ICPS classification.

Use of natural language understanding to facilitate surgical de-escalation of axillary staging in patients with breast cancer

Natural language understanding (NLU) may be particularly well-equipped for enhanced data capture from the electronic health record (EHR) given its examination of both content- and context-driven extraction. We developed and applied a NLU model to examine rates of pathological node positivity (pN+) and rates of lymphedema to determine if omission of routine axillary staging could be extended to younger patients with ER+/cN0 disease. We found that rates of pN+ and arm lymphedema were similar between patients 55-69yo and ≥70yo, with rates of lymphedema exceeding rates of pN+ for clinical stage T1c and smaller disease. Data from our NLU model suggest that omission of SLNB might be extended beyond Choosing Wisely recommendations, limited to those over 70 years old, to all postmenopausal women with early-stage ER+/cN0 disease. These data support the recently-reported SOUND trial results and provide additional granularity to facilitate surgical de-escalation.

Machine learning for hospital readmission prediction in pediatric population

BACKGROUND AND OBJECTIVE

Pediatric readmissions are a burden on patients, families, and the healthcare system. In order to identify patients at higher readmission risk, more accurate techniques, as machine learning (ML), could be a good strategy to expand the knowledge in this area. The aim of this study was to develop predictive models capable of identifying children and adolescents at high risk of potentially avoidable 30-day readmission using ML.

METHODS

Retrospective cohort study was carried out with 9,080 patients under 18 years old admitted to a tertiary university hospital. Demographic, clinical, and biochemical data were collected from electronic databases. We randomly divided the dataset into training (75 %) and testing (25 %), applied downsampling, repeated cross-validation with five folds and ten repetitions, and the hyperparameter was optimized of each technique using a grid search via racing with ANOVA models. We applied six ML classification algorithms to build the predictive models, including classification and regression tree (CART), random forest (RF), gradient boosting machine (GBM), extreme gradient boosting (XGBoost), decision tree and logistic regression (LR). The area under the receiver operating curve (AUC), sensitivity, specificity, Youden's J-index and accuracy were used to evaluate the performance of each model.

RESULTS

The avoidable 30-day hospital readmissions rate was 9.5 %. Some algorithms presented similar AUC, both in the dataset training and in the dataset testing, such as XGBoost, RF, GBM and CART. Considering the Youden's J-index, the algorithm that presented the best index was XGBoost with bagging imputation, with AUC of 0.814 (J-index of 0.484). Cancer diagnosis, age, red blood cells, leukocytes, red cell distribution width and sodium levels, elective admission, and multimorbidity were the most important characteristics to classify between readmission and non-readmission groups.

CONCLUSION

Machine learning approaches, especially XGBoost, can predict potentially avoidable 30-day pediatric hospital readmission into tertiary assistance. If implemented in the computer hospital system, our model can help in the early and more accurate identification of patients at readmission risk, targeting health strategic interventions.

Natural language processing to automate a web-based model of care and modernize skin cancer multidisciplinary team meetings

BACKGROUND

Cancer multidisciplinary team (MDT) meetings are under intense pressure to reform given the rapidly rising incidence of cancer and national mandates for protocolized streaming of cases. The aim of this study was to validate a natural language processing (NLP)-based web platform to automate evidence-based MDT decisions for skin cancer with basal cell carcinoma as a use case.

METHODS

A novel and validated NLP information extraction model was used to extract perioperative tumour and surgical factors from histopathology reports. A web application with a bespoke application programming interface used data from this model to provide an automated clinical decision support system, mapped to national guidelines and generating a patient letter to communicate ongoing management. Performance was assessed against retrospectively derived recommendations by two independent and blinded expert clinicians.

RESULTS

There were 893 patients (1045 lesions) used to internally validate the model. High accuracy was observed when compared against human predictions, with an overall value of 0.92. Across all classifiers the virtual skin MDT was highly specific (0.96), while sensitivity was lower (0.72).

CONCLUSION

This study demonstrates the feasibility of a fully automated, virtual, web-based service model to host the skin MDT with good system performance. This platform could be used to support clinical decision-making during MDTs as 'human in the loop' approach to aid protocolized streaming. Future prospective studies are needed to validate the model in tumour types where guidelines are more complex.

Optimizing artificial intelligence in sepsis management: Opportunities in the present and looking closely to the future

Sepsis remains a major challenge internationally for healthcare systems. Its incidence is rising due to poor public awareness and delays in its recognition and subsequent management. In sepsis, mortality increases with every hour left untreated. Artificial intelligence (AI) is transforming worldwide healthcare delivery at present. This review has outlined how AI can augment strategies to address this global disease burden. AI and machine learning (ML) algorithms can analyze vast quantities of increasingly complex clinical datasets from electronic medical records to assist clinicians in diagnosing and treating sepsis earlier than traditional methods. Our review highlights how these models can predict the risk of sepsis and organ failure even before it occurs. This gives providers additional time to plan and execute treatment plans, thereby avoiding increasing complications associated with delayed diagnosis of sepsis. The potential for cost savings with AI implementation is also discussed, including improving workflow efficiencies, reducing administrative costs, and improving healthcare outcomes. Despite these advantages, clinicians have been slow to adopt AI into clinical practice. Some of the limitations posed by AI solutions include the lack of diverse data sets for model building so that they are widely applicable for routine clinical use. Furthermore, the subsequent algorithms are often based on complex mathematics leading to clinician hesitancy to embrace such technologies. Finally, we highlight the need for robust political and regulatory frameworks in this area to achieve the trust and approval of clinicians and patients to implement this transformational technology.

Natural Language Processing Applied to Clinical Documentation in Post-acute Care Settings: A Scoping Review

OBJECTIVES

To determine the scope of the application of natural language processing to free-text clinical notes in post-acute care and provide a foundation for future natural language processing-based research in these settings.

DESIGN

Scoping review; reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines.

SETTING AND PARTICIPANTS

Post-acute care (ie, home health care, long-term care, skilled nursing facilities, and inpatient rehabilitation facilities).

METHODS

PubMed, Cumulative Index of Nursing and Allied Health Literature, and Embase were searched in February 2023. Eligible studies had quantitative designs that used natural language processing applied to clinical documentation in post-acute care settings. The quality of each study was appraised.

RESULTS

Twenty-one studies were included. Almost all studies were conducted in home health care settings. Most studies extracted data from electronic health records to examine the risk for negative outcomes, including acute care utilization, medication errors, and suicide mortality. About half of the studies did not report age, sex, race, or ethnicity data or use standardized terminologies. Only 8 studies included variables from socio-behavioral domains. Most studies fulfilled all quality appraisal indicators.

CONCLUSIONS AND IMPLICATIONS

The application of natural language processing is nascent in post-acute care settings. Future research should apply natural language processing using standardized terminologies to leverage free-text clinical notes in post-acute care to promote timely, comprehensive, and equitable care. Natural language processing could be integrated with predictive models to help identify patients who are at risk of negative outcomes. Future research should incorporate socio-behavioral determinants and diverse samples to improve health equity in informatics tools.

The use of natural language processing in detecting and predicting falls within the healthcare setting: a systematic review

Falls are a common problem associated with significant morbidity, mortality, and economic costs. Current fall prevention policies in local healthcare settings are often guided by information provided by fall risk assessment tools, incident reporting, and coding data. This review was conducted with the aim of identifying studies which utilized natural language processing (NLP) for the automated detection and prediction of falls in the healthcare setting. The databases Ovid Medline, Ovid Embase, Ovid Emcare, PubMed, CINAHL, IEEE Xplore, and Ei Compendex were searched from 2012 until April 2023. Retrospective derivation, validation, and implementation studies wherein patients experienced falls within a healthcare setting were identified for inclusion. The initial search yielded 2611 publications for title and abstract screening. Full-text screening was conducted on 105 publications, resulting in 26 unique studies that underwent qualitative analyses. Studies applied NLP towards falls risk factor identification, known falls detection, future falls prediction, and falls severity stratification with reasonable success. The NLP pipeline was reviewed in detail between studies and models utilizing rule-based, machine learning (ML), deep learning (DL), and hybrid approaches were examined. With a growing literature surrounding falls prediction in both inpatient and outpatient environments, the absence of studies examining the impact of these models on patient and system outcomes highlights the need for further implementation studies. Through an exploration of the application of NLP techniques, it may be possible to develop models with higher performance in automated falls prediction and detection.

Heralding a new era of oxytocinergic research: New tools, new problems?

According to classic neuroendocrinology, hypothalamic oxytocin cells can be categorized into parvo- and magnocellular neurons. However, research in the last decade provided ample evidence that this black-and-white model of oxytocin neurons is most likely oversimplified. Novel genetic, functional and morphological studies indicate that oxytocin neurons might be organized in functional modules and suggest the existence of five or more distinct oxytocinergic subpopulations. However, many of these novel, automated high-throughput techniques might be inherently biased and interpretation of acquired data needs to be approached with caution to enable drawing sound and reliable conclusions. In addition, the recent finding that astrocytes in various brain regions express functional oxytocin receptors represents a paradigm shift and challenges the view that oxytocin primarily acts as a direct peptidergic neurotransmitter. This review highlights the latest technical advances in oxytocinergic research, puts recent studies on the oxytocin system into context and formulates various provocative ideas based on novel findings that challenges various prevailing hypotheses and dogmas about oxytocinergic modulation.

A neural network-enhanced methodology for the rapid establishment of local DRLs in interventional radiology: EVAR as a case example

PURPOSE

To develop a neural network-enhanced workflow for the automatic and rapid establishment/update of local diagnostic reference levels (DRLs) in interventional radiology (IR) using endovascular aneurysm repair (EVAR) procedures as a case example.

METHODS

Radiation dose reports were collected retrospectively for 46 consecutive EVAR procedures. These reports served as demonstrative data for the development of the proposed methodology. An algorithm was developed to receive multiple dose reports, automatically extract the kerma area product (KAP), air kerma (Ka,r), number of exposure images, and fluoroscopy time (FT) from each report and calculate the first, second, third quartiles as well as the maximum and minimum values of the extracted parameters. To extract the values of interest from the dose reports, Tesseract, an open-source optical character recognition (OCR) engine was employed. Furthermore, the accuracy and time efficiency of the proposed methodology were assessed. Specifically, the values extracted from the algorithm were compared with the ground truth values and the algorithm's processing time was compared with the respective time needed to manually extract and process the values of interest.

RESULTS

The OCR-based algorithm managed to correctly recognize 182 from the 184 target values, resulting in an accuracy of 99%. Moreover, the proposed pipeline reduced the processing time for the establishment of DRLs by 98%. DRL value for EVAR procedures, set as the third quartile of KAP was found to be 551 Gy*cm2.

CONCLUSION

An accurate and time-efficient workflow was developed for the establishment of local DRLs in interventional radiology.

Artificial intelligence for the vascular surgeon

In recent years, artificial intelligence (AI) has permeated different aspects of vascular surgery to solve challenges in clinical practice. Although AI in vascular surgery is still in its early stages, there have been promising developments in its applications to vascular diagnosis, risk stratification, and outcome prediction. By establishing a baseline knowledge of AI, vascular surgeons are better equipped to use and interpret the data from these types of projects. This review aims to provide an overview of the fundamentals of AI and highlight its role in helping vascular surgeons overcome the challenges of clinical practice. In addition, we discuss the limitations of AI and how they affect AI applications.

A New Tool for Holistic Residency Application Review: Using Natural Language Processing of Applicant Experiences to Predict Interview Invitation

PROBLEM

Reviewing residency application narrative components is time intensive and has contributed to nearly half of applications not receiving holistic review. The authors developed a natural language processing (NLP)-based tool to automate review of applicants' narrative experience entries and predict interview invitation.

APPROACH

Experience entries (n = 188,500) were extracted from 6,403 residency applications across 3 application cycles (2017-2019) at 1 internal medicine program, combined at the applicant level, and paired with the interview invitation decision (n = 1,224 invitations). NLP identified important words (or word pairs) with term frequency-inverse document frequency, which were used to predict interview invitation using logistic regression with L1 regularization. Terms remaining in the model were analyzed thematically. Logistic regression models were also built using structured application data and a combination of NLP and structured data. Model performance was evaluated on never-before-seen data using area under the receiver operating characteristic and precision-recall curves (AUROC, AUPRC).

OUTCOMES

The NLP model had an AUROC of 0.80 (vs chance decision of 0.50) and AUPRC of 0.49 (vs chance decision of 0.19), showing moderate predictive strength. Phrases indicating active leadership, research, or work in social justice and health disparities were associated with interview invitation. The model's detection of these key selection factors demonstrated face validity. Adding structured data to the model significantly improved prediction (AUROC 0.92, AUPRC 0.73), as expected given reliance on such metrics for interview invitation.

NEXT STEPS

This model represents a first step in using NLP-based artificial intelligence tools to promote holistic residency application review. The authors are assessing the practical utility of using this model to identify applicants screened out using traditional metrics. Generalizability must be determined through model retraining and evaluation at other programs. Work is ongoing to thwart model "gaming," improve prediction, and remove unwanted biases introduced during model training.

Health Equity in Clinical Research Informatics

OBJECTIVES

Through a scoping review, we examine in this survey what ways health equity has been promoted in clinical research informatics with patient implications and especially published in the year of 2021 (and some in 2022).

METHOD

A scoping review was conducted guided by using methods described in the Joanna Briggs Institute Manual. The review process consisted of five stages: 1) development of aim and research question, 2) literature search, 3) literature screening and selection, 4) data extraction, and 5) accumulate and report results.

RESULTS

From the 478 identified papers in 2021 on the topic of clinical research informatics with focus on health equity as a patient implication, 8 papers met our inclusion criteria. All included papers focused on artificial intelligence (AI) technology. The papers addressed health equity in clinical research informatics either through the exposure of inequity in AI-based solutions or using AI as a tool for promoting health equity in the delivery of healthcare services. While algorithmic bias poses a risk to health equity within AI-based solutions, AI has also uncovered inequity in traditional treatment and demonstrated effective complements and alternatives that promotes health equity.

CONCLUSIONS

Clinical research informatics with implications for patients still face challenges of ethical nature and clinical value. However, used prudently-for the right purpose in the right context-clinical research informatics could bring powerful tools in advancing health equity in patient care.

Identifying epilepsy surgery candidates with natural language processing: A systematic review

INTRODUCTION

Epilepsy surgery is an underutilised, efficacious management strategy for selected individuals with drug-resistant epilepsy. Natural language processing (NLP) may aid in the identification of patients who are suitable to undergo evaluation for epilepsy surgery. The feasibility of this approach is yet to be determined.

METHOD

In accordance with the PRISMA guidelines, a systematic review of the databases PubMed, EMBASE and Cochrane library was performed. This systematic review was prospectively registered on PROSPERO.

RESULTS

6 studies fulfilled inclusion criteria. The majority of included studies reported on datasets from only a single centre, with one study utilising data from two centres and one study six centres. The most commonly employed algorithms were support vector machines (5/6), with only one study utilising NLP strategies such as random forest models and gradient boosted machines. However, the results are promising, with all studies demonstrating moderate to high levels of performance in the identification of patients who may be suitable to undergo epilepsy surgery evaluation. Furthermore, multiple studies demonstrated that NLP could identify such patients 1-2 years prior to the treating clinicians instigating referral. However, no studies were identified that have evaluated the influence of implementing such algorithms on healthcare systems or patient outcomes.

CONCLUSIONS

NLP is a promising approach to aid in the identification of patients that may be suitable to undergo epilepsy surgery evaluation. Further studies are required examining diverse datasets with additional analytical methodologies. Studies evaluating the impact of implementation of such algorithms would be beneficial.

The value of a spaceflight clinical decision support system for earth-independent medical operations

As NASA prepares for crewed lunar missions over the next several years, plans are also underway to journey farther into deep space. Deep space exploration will require a paradigm shift in astronaut medical support toward progressively earth-independent medical operations (EIMO). The Exploration Medical Capability (ExMC) element of NASA's Human Research Program (HRP) is investigating the feasibility and value of advanced capabilities to promote and enhance EIMO. Currently, astronauts rely on real-time communication with ground-based medical providers. However, as the distance from Earth increases, so do communication delays and disruptions. Moreover, resupply and evacuation will become increasingly complex, if not impossible, on deep space missions. In contrast to today's missions in low earth orbit (LEO), where most medical expertise and decision-making are ground-based, an exploration crew will need to autonomously detect, diagnose, treat, and prevent medical events. Due to the sheer amount of pre-mission training required to execute a human spaceflight mission, there is often little time to devote exclusively to medical training. One potential solution is to augment the long duration exploration crew's knowledge, skills, and abilities with a clinical decision support system (CDSS). An analysis of preliminary data indicates the potential benefits of a CDSS to mission outcomes when augmenting cognitive and procedural performance of an autonomous crew performing medical operations, and we provide an illustrative scenario of how such a CDSS might function.

Urban resilience to socioeconomic disruptions during the COVID-19 pandemic: Evidence from China

The COVID-19 pandemic and the associated restrictions have raised the awareness of building pandemic-resilient cities. Prior studies often evaluated the resilience of one type of urban system while lacking a comparison across various urban subsystems. This study fills this gap by measuring and comparing the adaptive resilience to the pandemic of various urban subsystems in Chinese cities. We propose a novel outcome measurement of the pandemic's socioeconomic impacts on cities, i.e., the citizens' complaints data, and use its temporal changes to measure cities' adaptive resilience to the pandemic. We find a wide range of urban subsystems were severely shocked by the pandemic, including the urban economy, construction-and-housing sector, welfare system, and education system. Different urban subsystems exhibit divergent degrees of adaptive resilience to the pandemic. Using cluster analysis, we also identify three types of cities with different patterns of adaptive resilience: cities whose general economies were the least resilient, cities whose construction-and-housing system was the least resilient, and cities that were mostly affected by restriction measures. Our findings contribute to the understanding of the pandemic's socioeconomic costs and help identify the divergent resilience of different urban subsystems so as to develop targeted policy interventions to improve cities' resilience to the pandemic.

Early Identification of Family Physicians Using Qualitative Admissions Data

BACKGROUND AND OBJECTIVES

The medical community has been concerned about the shortage of family physicians for decades. Identification of likely family medicine (FM) student matches early in medical school is an efficient recruitment tool. The objective of this study was to analyze qualitative data from medical school applications to establish themes that differentiate future family physicians from their non-FM counterparts.

METHODS

We conducted a qualitative analysis of admissions essays from two groups of 2010-2019 medical school graduates: a study group of students who matched to FM (n=135) and a random sample comparison group of non-FM matches (n=136). We utilized a natural language modeling platform to recognize semantic patterns in the data. This platform generated keywords for each sample, which then guided a more traditional content analysis of the qualitative data for themes.

RESULTS

The two groups shared two themes: emotions and science/academics, but with some differences in thematic emphasis. The study group tended toward more positive emotions and the comparison group tended to utilize more specialized scientific language. The study group exhibited two unique themes: special interests in service and community/people. A secondary theme of religious faith was evident in the FM study group. The comparison group exhibited two unique themes: lab/clinical research and career aspirations.

CONCLUSIONS

Aided by machine learning, a novel analytical approach revealed key differences between FM and non-FM student application materials. Findings suggest qualitative application data may contain identifiable thematic differences when comparing students who eventually match into FM residency programs to those who match into other specialties. Assessing student potential for FM could help guide recruitment and mentorship activities.

Developing AI enabled sensors and decision support for military operators in the field

Wearable sensors enable down range data collection of physiological and cognitive performance of the warfighter. However, autonomous teams may find the sensor data impractical to interpret and hence influence real-time decisions without the support of subject matter experts. Decision support tools can reduce the burden of interpreting physiological data in the field and incorporate a systems perspective where noisy field data can contain useful additional signals. We present a methodology of how artificial intelligence can be used for modeling human performance with decision-making to achieve actionable decision support. We provide a framework for systems design and advancing from the laboratory to real world environments. The result is a validated measure of down-range human performance with a low burden of operation.

Development and international validation of logistic regression and machine-learning models for the prediction of 10-year molar loss

AIM

To develop and validate models based on logistic regression and artificial intelligence for prognostic prediction of molar survival in periodontally affected patients.

MATERIALS AND METHODS

Clinical and radiographic data from four different centres across four continents (two in Europe, one in the United States, and one in China) including 515 patients and 3157 molars were collected and used to train and test different types of machine-learning algorithms for their prognostic ability of molar loss over 10 years. The following models were trained: logistic regression, support vector machine, K-nearest neighbours, decision tree, random forest, artificial neural network, gradient boosting, and naive Bayes. In addition, different models were aggregated by means of the ensembled stacking method. The primary outcome of the study was related to the prediction of overall molar loss (MLO) in patients after active periodontal treatment.

RESULTS

The general performance in the external validation settings (aggregating three cohorts) revealed that the ensembled model, which combined neural network and logistic regression, showed the best performance among the different models for the prediction of MLO with an area under the curve (AUC) = 0.726. The neural network model showed the best AUC of 0.724 for the prediction of periodontitis-related molar loss. In addition, the ensembled model showed the best calibration performance.

CONCLUSIONS

Through a multi-centre collaboration, both prognostic models for the prediction of molar loss were developed and externally validated. The ensembled model showed the best performance in terms of both discrimination and validation, and it is made freely available to clinicians for widespread use in clinical practice.

Machine learning in medicine: a practical introduction to techniques for data pre-processing, hyperparameter tuning, and model comparison

BACKGROUND

There is growing enthusiasm for the application of machine learning (ML) and artificial intelligence (AI) techniques to clinical research and practice. However, instructions on how to develop robust high-quality ML and AI in medicine are scarce. In this paper, we provide a practical example of techniques that facilitate the development of high-quality ML systems including data pre-processing, hyperparameter tuning, and model comparison using open-source software and data.

METHODS

We used open-source software and a publicly available dataset to train and validate multiple ML models to classify breast masses into benign or malignant using mammography image features and patient age. We compared algorithm predictions to the ground truth of histopathologic evaluation. We provide step-by-step instructions with accompanying code lines.

FINDINGS

Performance of the five algorithms at classifying breast masses as benign or malignant based on mammography image features and patient age was statistically equivalent (P > 0.05). Area under the receiver operating characteristics curve (AUROC) for the logistic regression with elastic net penalty was 0.89 (95% CI 0.85 - 0.94), for the Extreme Gradient Boosting Tree 0.88 (95% CI 0.83 - 0.93), for the Multivariate Adaptive Regression Spline algorithm 0.88 (95% CI 0.83 - 0.93), for the Support Vector Machine 0.89 (95% CI 0.84 - 0.93), and for the neural network 0.89 (95% CI 0.84 - 0.93).

INTERPRETATION

Our paper allows clinicians and medical researchers who are interested in using ML algorithms to understand and recreate the elements of a comprehensive ML analysis. Following our instructions may help to improve model generalizability and reproducibility in medical ML studies.

Ethnicity influences phenotype and clinical outcomes: Comparing a South American with a North American inflammatory bowel disease cohort

Inflammatory bowel disease (IBD), including ulcerative colitis (UC) and Crohn disease (CD), has emerged as a global disease with an increasing incidence in developing and newly industrialized regions such as South America. This global rise offers the opportunity to explore the differences and similarities in disease presentation and outcomes across different genetic backgrounds and geographic locations. Our study includes 265 IBD patients. We performed an exploratory analysis of the databases of Chilean and North American IBD patients to compare the clinical phenotypes between the cohorts. We employed an unsupervised machine-learning approach using principal component analysis, uniform manifold approximation, and projection, among others, for each disease. Finally, we predicted the cohort (North American vs Chilean) using a random forest. Several unsupervised machine learning methods have separated the 2 main groups, supporting the differences between North American and Chilean patients with each disease. The variables that explained the loadings of the clinical metadata on the principal components were related to the therapies and disease extension/location at diagnosis. Our random forest models were trained for cohort classification based on clinical characteristics, obtaining high accuracy (0.86 = UC; 0.79 = CD). Similarly, variables related to therapy and disease extension/location had a high Gini index. Similarly, univariate analysis showed a later CD age at diagnosis in Chilean IBD patients (37 vs 24; P = .005). Our study suggests a clinical difference between North American and Chilean IBD patients: later CD age at diagnosis with a predominantly less aggressive phenotype (39% vs 54% B1) and more limited disease, despite fewer biological therapies being used in Chile for both diseases.

Development and validation of an automated basal cell carcinoma histopathology information extraction system using natural language processing

Introduction

Routinely collected healthcare data are a powerful research resource, but often lack detailed disease-specific information that is collected in clinical free text such as histopathology reports. We aim to use natural Language Processing (NLP) techniques to extract detailed clinical and pathological information from histopathology reports to enrich routinely collected data.

Methods

We used the general architecture for text engineering (GATE) framework to build an NLP information extraction system using rule-based techniques. During validation, we deployed our rule-based NLP pipeline on 200 previously unseen, de-identified and pseudonymised basal cell carcinoma (BCC) histopathological reports from Swansea Bay University Health Board, Wales, UK. The results of our algorithm were compared with gold standard human annotation by two independent and blinded expert clinicians involved in skin cancer care.

Results

We identified 11,224 items of information with a mean precision, recall, and F1 score of 86.0% (95% CI: 75.1-96.9), 84.2% (95% CI: 72.8-96.1), and 84.5% (95% CI: 73.0-95.1), respectively. The difference between clinician annotator F1 scores was 7.9% in comparison with 15.5% between the NLP pipeline and the gold standard corpus. Cohen's Kappa score on annotated tokens was 0.85.

Conclusion

Using an NLP rule-based approach for named entity recognition in BCC, we have been able to develop and validate a pipeline with a potential application in improving the quality of cancer registry data, supporting service planning, and enhancing the quality of routinely collected data for research.

Cardiovascular Disease Detection using Ensemble Learning

One of the most challenging tasks for clinicians is detecting symptoms of cardiovascular disease as earlier as possible. Many individuals worldwide die each year from cardiovascular disease. Since heart disease is a major concern, it must be dealt with timely. Multiple variables affecting health, such as excessive blood pressure, elevated cholesterol, an irregular pulse rate, and many more, make it challenging to diagnose cardiac disease. Thus, artificial intelligence can be useful in identifying and treating diseases early on. This paper proposes an ensemble-based approach that uses machine learning (ML) and deep learning (DL) models to predict a person’s likelihood of developing cardiovascular disease. We employ six classification algorithms to predict cardiovascular disease. Models are trained using a publicly available dataset of cardiovascular disease cases. We use random forest (RF) to extract important cardiovascular disease features. The experiment results demonstrate that the ML ensemble model achieves the best disease prediction accuracy of 88.70%.

An Intelligent Hybrid Sentiment Analyzer for Personal Protective Medical Equipments Based on Word Embedding Technique: The COVID-19 Era

Due to the accelerated growth of symmetrical sentiment data across different platforms, experimenting with different sentiment analysis (SA) techniques allows for better decision-making and strategic planning for different sectors. Specifically, the emergence of COVID-19 has enriched the data of people’s opinions and feelings about medical products. In this paper, we analyze people’s sentiments about the products of a well-known e-commerce website named Alibaba.com. People’s sentiments are experimented with using a novel evolutionary approach by applying advanced pre-trained word embedding for word presentations and combining them with an evolutionary feature selection mechanism to classify these opinions into different levels of ratings. The proposed approach is based on harmony search algorithm and different classification techniques including random forest, k-nearest neighbor, AdaBoost, bagging, SVM, and REPtree to achieve competitive results with the least possible features. The experiments are conducted on five different datasets including medical gloves, hand sanitizer, medical oxygen, face masks, and a combination of all these datasets. The results show that the harmony search algorithm successfully reduced the number of features by 94.25%, 89.5%, 89.25%, 92.5%, and 84.25% for the medical glove, hand sanitizer, medical oxygen, face masks, and whole datasets, respectively, while keeping a competitive performance in terms of accuracy and root mean square error (RMSE) for the classification techniques and decreasing the computational time required for classification.