Automatic Construction of Chinese Herbal Prescriptions From Tongue Images Using CNNs and Auxiliary Latent Therapy Topics

PrescDRL: deep reinforcement learning for herbal prescription planning in treatment of chronic diseases

Treatment planning for chronic diseases is a critical task in medical artificial intelligence, particularly in traditional Chinese medicine (TCM). However, generating optimized sequential treatment strategies for patients with chronic diseases in different clinical encounters remains a challenging issue that requires further exploration. In this study, we proposed a TCM herbal prescription planning framework based on deep reinforcement learning for chronic disease treatment (PrescDRL). PrescDRL is a sequential herbal prescription optimization model that focuses on long-term effectiveness rather than achieving maximum reward at every step, thereby ensuring better patient outcomes. We constructed a high-quality benchmark dataset for sequential diagnosis and treatment of diabetes and evaluated PrescDRL against this benchmark. Our results showed that PrescDRL achieved a higher curative effect, with the single-step reward improving by 117% and 153% compared to doctors. Furthermore, PrescDRL outperformed the benchmark in prescription prediction, with precision improving by 40.5% and recall improving by 63%. Overall, our study demonstrates the potential of using artificial intelligence to improve clinical intelligent diagnosis and treatment in TCM.

Development and Application of Traditional Chinese Medicine Using AI Machine Learning and Deep Learning Strategies

Traditional Chinese medicine (TCM) has been used for thousands of years and has been proven to be effective at treating many complicated illnesses with minimal side effects. The application and advancement of TCM are, however, constrained by the absence of objective measuring standards due to its relatively abstract diagnostic methods and syndrome differentiation theories. Ongoing developments in machine learning (ML) and deep learning (DL), specifically in computer vision (CV) and natural language processing (NLP), offer novel opportunities to modernize TCM by exploring the profound connotations of its theory. This review begins with an overview of the ML and DL methods employed in TCM; this is followed by practical instances of these applications. Furthermore, extensive discussions emphasize the mature integration of ML and DL in TCM, such as tongue diagnosis, pulse diagnosis, and syndrome differentiation treatment, highlighting their early successful application in the TCM field. Finally, this study validates the accomplishments and addresses the problems and challenges posed by the application and development of TCM powered by ML and DL. As ML and DL techniques continue to evolve, modern technology will spark new advances in TCM.

Two-stream vision transformer based multi-label recognition for TCM prescriptions construction

Traditional Chinese medicine (TCM) observation diagnosis images (including facial and tongue images) provide essential human body information, holding significant importance in clinical medicine for diagnosis and treatment. TCM prescriptions, known for their simplicity, non-invasiveness, and low side effects, have been widely applied worldwide. Exploring automated herbal prescription construction based on visual diagnosis holds vital value in delving into the correlation between external features and herbal prescriptions and offering medical services in mobile healthcare systems. To effectively integrate multi-perspective visual diagnosis images and automate prescription construction, this study proposes a multi-herb recommendation framework based on Visual Transformer and multi-label classification. The framework comprises three key components: image encoder, label embedding module, and cross-modal fusion classification module. The image encoder employs a dual-stream Visual Transformer to learn dependencies between different regions of input images, capturing both local and global features. The label embedding module utilizes Graph Convolutional Networks to capture associations between diverse herbal labels. Finally, two Multi-Modal Factorized Bilinear modules are introduced as effective components to fuse cross-modal vectors, creating an end-to-end multi-label image-herb recommendation model. Through experimentation with real facial and tongue images and generating prescription data closely resembling real samples. The precision is 50.06 %, the recall rate is 48.33 %, and the F1-score is 49.18 %. This study validates the feasibility of automated herbal prescription construction from the perspective of visual diagnosis. Simultaneously, it provides valuable insights for constructing herbal prescriptions automatically from more physical information.

Integrating artificial intelligence into the modernization of traditional Chinese medicine industry: a review

Traditional Chinese medicine (TCM) is the practical experience and summary of the Chinese nation for thousands of years. It shows great potential in treating various chronic diseases, complex diseases and major infectious diseases, and has gradually attracted the attention of people all over the world. However, due to the complexity of prescription and action mechanism of TCM, the development of TCM industry is still in a relatively conservative stage. With the rise of artificial intelligence technology in various fields, many scholars began to apply artificial intelligence technology to traditional Chinese medicine industry and made remarkable progress. This paper comprehensively summarizes the important role of artificial intelligence in the development of traditional Chinese medicine industry from various aspects, including new drug discovery, data mining, quality standardization and industry technology of traditional Chinese medicine. The limitations of artificial intelligence in these applications are also emphasized, including the lack of pharmacological research, database quality problems and the challenges brought by human-computer interaction. Nevertheless, the development of artificial intelligence has brought new opportunities and innovations to the modernization of traditional Chinese medicine. Integrating artificial intelligence technology into the comprehensive application of Chinese medicine industry is expected to overcome the major problems faced by traditional Chinese medicine industry and further promote the modernization of the whole traditional Chinese medicine industry.

MLP-Like Model With Convolution Complex Transformation for Auxiliary Diagnosis Through Medical Images

Medical images such as facial and tongue images have been widely used for intelligence-assisted diagnosis, which can be regarded as the multi-label classification task for disease location (DL) and disease nature (DN) of biomedical images. Compared with complicated convolutional neural networks and Transformers for this task, recent MLP-like architectures are not only simple and less computationally expensive, but also have stronger generalization capabilities. However, MLP-like models require better input features from the image. Thus, this study proposes a novel convolution complex transformation MLP-like (CCT-MLP) model for the multi-label DL and DN recognition task for facial and tongue images. Notably, the convolutional Tokenizer and multiple convolutional layers are first used to extract the better shallow features from input biomedical images to make up for the loss of spatial information obtained by the simple MLP structure. Subsequently, the Channel-MLP architecture with complex transformations is used to extract deep-level contextual features. In this way, multi-channel features are extracted and mixed to perform the multi-label classification of the input biomedical images. Experimental results on our constructed multi-label facial and tongue image datasets demonstrate that our method outperforms existing methods in terms of both accuracy (Acc) and mean average precision (mAP).

A Review on Electronic Health Record Text-Mining for Biomedical Name Entity Recognition in Healthcare Domain

Biomedical-named entity recognition (bNER) is critical in biomedical informatics. It identifies biomedical entities with special meanings, such as people, places, and organizations, as predefined semantic types in electronic health records (EHR). bNER is essential for discovering novel knowledge using computational methods and Information Technology. Early bNER systems were configured manually to include domain-specific features and rules. However, these systems were limited in handling the complexity of the biomedical text. Recent advances in deep learning (DL) have led to the development of more powerful bNER systems. DL-based bNER systems can learn the patterns of biomedical text automatically, making them more robust and efficient than traditional rule-based systems. This paper reviews the healthcare domain of bNER, using DL techniques and artificial intelligence in clinical records, for mining treatment prediction. bNER-based tools are categorized systematically and represent the distribution of input, context, and tag (encoder/decoder). Furthermore, to create a labeled dataset for our machine learning sentiment analyzer to analyze the sentiment of a set of tweets, we used a manual coding approach and the multi-task learning method to bias the training signals with domain knowledge inductively. To conclude, we discuss the challenges facing bNER systems and future directions in the healthcare field.

Development of a tongue image-based machine learning tool for the diagnosis of gastric cancer: a prospective multicentre clinical cohort study

BACKGROUND

Tongue images (the colour, size and shape of the tongue and the colour, thickness and moisture content of the tongue coating), reflecting the health state of the whole body according to the theory of traditional Chinese medicine (TCM), have been widely used in China for thousands of years. Herein, we investigated the value of tongue images and the tongue coating microbiome in the diagnosis of gastric cancer (GC).

METHODS

From May 2020 to January 2021, we simultaneously collected tongue images and tongue coating samples from 328 patients with GC (all newly diagnosed with GC) and 304 non-gastric cancer (NGC) participants in China, and 16 S rDNA was used to characterize the microbiome of the tongue coating samples. Then, artificial intelligence (AI) deep learning models were established to evaluate the value of tongue images and the tongue coating microbiome in the diagnosis of GC. Considering that tongue imaging is more convenient and economical as a diagnostic tool, we further conducted a prospective multicentre clinical study from May 2020 to March 2022 in China and recruited 937 patients with GC and 1911 participants with NGC from 10 centres across China to further evaluate the role of tongue images in the diagnosis of GC. Moreover, we verified this approach in another independent external validation cohort that included 294 patients with GC and 521 participants with NGC from 7 centres. This study is registered at ClinicalTrials.gov, NCT01090362.

FINDINGS

For the first time, we found that both tongue images and the tongue coating microbiome can be used as tools for the diagnosis of GC, and the area under the curve (AUC) value of the tongue image-based diagnostic model was 0.89. The AUC values of the tongue coating microbiome-based model reached 0.94 using genus data and 0.95 using species data. The results of the prospective multicentre clinical study showed that the AUC values of the three tongue image-based models for GCs reached 0.88-0.92 in the internal verification and 0.83-0.88 in the independent external verification, which were significantly superior to the combination of eight blood biomarkers.

INTERPRETATION

Our results suggest that tongue images can be used as a stable method for GC diagnosis and are significantly superior to conventional blood biomarkers. The three kinds of tongue image-based AI deep learning diagnostic models that we developed can be used to adequately distinguish patients with GC from participants with NGC, even early GC and precancerous lesions, such as atrophic gastritis (AG).

FUNDING

The National Key R&D Program of China (2021YFA0910100), Program of Zhejiang Provincial TCM Sci-tech Plan (2018ZY006), Medical Science and Technology Project of Zhejiang Province (2022KY114, WKJ-ZJ-2104), Zhejiang Provincial Research Center for Upper Gastrointestinal Tract Cancer (JBZX-202006), Natural Science Foundation of Zhejiang Province (HDMY22H160008), Science and Technology Projects of Zhejiang Province (2019C03049), National Natural Science Foundation of China (82074245, 81973634, 82204828), and Chinese Postdoctoral Science Foundation (2022M713203).

A survey of artificial intelligence in tongue image for disease diagnosis and syndrome differentiation

The rapid development of artificial intelligence technology has gradually extended from the general field to all walks of life, and intelligent tongue diagnosis is the product of a miraculous connection between this new discipline and traditional disciplines. We reviewed the deep learning methods and machine learning applied in tongue image analysis that have been studied in the last 5 years, focusing on tongue image calibration, detection, segmentation, and classification of diseases, syndromes, and symptoms/signs. Introducing technical evolutions or emerging technologies were applied in tongue image analysis; as we have noticed, attention mechanism, multiscale features, and prior knowledge were successfully applied in it, and we emphasized the value of combining deep learning with traditional methods. We also pointed out two major problems concerned with data set construction and the low reliability of performance evaluation that exist in this field based on the basic essence of tongue diagnosis in traditional Chinese medicine. Finally, a perspective on the future of intelligent tongue diagnosis was presented; we believe that the self-supervised method, multimodal information fusion, and the study of tongue pathology will have great research significance.

Personalized Intelligent Syndrome Differentiation Guided By TCM Consultation Philosophy

Traditional Chinese Medicine (TCM) is one of the oldest medical systems in the world, and inquiry is an essential part of TCM diagnosis. The development of artificial intelligence has led to the proposal of several computational TCM diagnostic methods. However, there are few research studies among them, and they have the following flaws: (1) insufficient engagement with the patient, (2) barren TCM consultation philosophy, and (3) inadequate validation of the method. As TCM inquiry knowledge is abstract and there are few relevant datasets, we devise a novel knowledge representation technique. The mapping of symptoms and syndromes is constructed based on the diagnostics of traditional Chinese medicine. As a guide, the inquiry knowledge base is constructed utilizing the “Ten Brief Inquiries,” TCM's domain knowledge. Subsequently, a corresponding assessment approach is proposed for an intelligent consultation model for syndrome differentiation. We establish three criteria: the quality of the generated question-answer pairs, the accuracy of model identification, and the average number of questions. Three TCM specialists are asked to undertake a manual evaluation of the model separately. The results reveal that our approach is capable of pretty accurate syndrome differentiation. Furthermore, the model's question and answer pairs for simulated consultations are relevant, accurate, and efficient.

Deep learning and machine intelligence: New computational modeling techniques for discovery of the combination rules and pharmacodynamic characteristics of Traditional Chinese Medicine

It has been increasingly accepted that Multi-Ingredient-Based interventions provide advantages over single-target therapy for complex diseases. With the growing development of Traditional Chinese Medicine (TCM) and continually being refined of a holistic view, "multi-target" and "multi-pathway" integration characteristics of which are being accepted. However, its effector substances, efficacy targets, especially the combination rules and mechanisms remain unclear, and more powerful strategies to interpret the synergy are urgently needed. Artificial intelligence (AI) and computer vision lead to a rapidly expanding in many fields, including diagnosis and treatment of TCM. AI technology significantly improves the reliability and accuracy of diagnostics, target screening, and new drug research. While all AI techniques are capable of matching models to biological big data, the specific methods are complex and varied. Retrieves literature by the keywords such as "artificial intelligence", "machine learning", "deep learning", "traditional Chinese medicine" and "Chinese medicine". Search the application of computer algorithms of TCM between 2000 and 2021 in PubMed, Web of Science, China National Knowledge Infrastructure (CNKI), Elsevier and Springer. This review concentrates on the application of computational in herb quality evaluation, drug target discovery, optimized compatibility and medical diagnoses of TCM. We describe the characteristics of biological data for which different AI techniques are applicable, and discuss some of the best data mining methods and the problems faced by deep learning and machine learning methods applied to Chinese medicine.

Weakly Supervised Deep Learning for Tooth-Marked Tongue Recognition

The recognition of tooth-marked tongues has important value for clinical diagnosis of traditional Chinese medicine. Tooth-marked tongue is often related to spleen deficiency, cold dampness, sputum, effusion, and blood stasis. The clinical manifestations of patients with tooth-marked tongue include loss of appetite, borborygmus, gastric distention, and loose stool. Traditional clinical tooth-marked tongue recognition is conducted subjectively based on the doctor’s visual observation, and its performance is affected by the doctor’s subjectivity, experience, and environmental lighting changes. In addition, the tooth marks typically have various shapes and colors on the tongue, which make it very challenging for doctors to identify tooth marks. The existing methods based on deep learning have made great progress for tooth-marked tongue recognition, but there are still shortcomings such as requiring a large amount of manual labeling of tooth marks, inability to detect and locate the tooth marks, and not conducive to clinical diagnosis and interpretation. In this study, we propose an end-to-end deep neural network for tooth-marked tongue recognition based on weakly supervised learning. Note that the deep neural network only requires image-level annotations of tooth-marked or non-tooth marked tongues. In this method, a deep neural network is trained to classify tooth-marked tongues with the image-level annotations. Then, a weakly supervised tooth-mark detection network (WSTDN) as an architecture variant of the pre-trained deep neural network is proposed for the tooth-marked region detection. Finally, the WSTDN is re-trained and fine-tuned using only the image-level annotations to simultaneously realize the classification of the tooth-marked tongue and the positioning of the tooth-marked region. Experimental results of clinical tongue images demonstrate the superiority of the proposed method compared with previously reported deep learning methods for tooth-marked tongue recognition. The proposed tooth-marked tongue recognition model may provide important syndrome diagnosis and efficacy evaluation methods, and contribute to the understanding of ethnopharmacological mechanisms.

TCMNER and PubMed: A Novel Chinese Character-Level-Based Model and a Dataset for TCM Named Entity Recognition

Intelligent traditional Chinese medicine (TCM) has become a popular research field by means of prospering of deep learning technology. Important achievements have been made in such representative tasks as automatic diagnosis of TCM syndromes and diseases and generation of TCM herbal prescriptions. However, one unavoidable issue that still hinders its progress is the lack of labeled samples, i.e., the TCM medical records. As an efficient tool, the named entity recognition (NER) models trained on various TCM resources can effectively alleviate this problem and continuously increase the labeled TCM samples. In this work, on the basis of in-depth analysis, we argue that the performance of the TCM named entity recognition model can be better by using the character-level representation and tagging and propose a novel word-character integrated self-attention module. With the help of TCM doctors and experts, we define 5 classes of TCM named entities and construct a comprehensive NER dataset containing the standard content of the publications and the clinical medical records. The experimental results on this dataset demonstrate the effectiveness of the proposed module.

Tongue image quality assessment based on a deep convolutional neural network

BACKGROUND

Tongue diagnosis is an important research field of TCM diagnostic technology modernization. The quality of tongue images is the basis for constructing a standard dataset in the field of tongue diagnosis. To establish a standard tongue image database in the TCM industry, we need to evaluate the quality of a massive number of tongue images and add qualified images to the database. Therefore, an automatic, efficient and accurate quality control model is of significance to the development of intelligent tongue diagnosis technology for TCM.

METHODS

Machine learning methods, including Support Vector Machine (SVM), Random Forest (RF), Gradient Boosting Decision Tree (GBDT), Adaptive Boosting Algorithm (Adaboost), Naïve Bayes, Decision Tree (DT), Residual Neural Network (ResNet), Convolution Neural Network developed by Visual Geometry Group at University of Oxford (VGG), and Densely Connected Convolutional Networks (DenseNet), were utilized to identify good-quality and poor-quality tongue images. Their performances were made comparisons by using metrics such as accuracy, precision, recall, and F1-Score.

RESULTS

The experimental results showed that the accuracy of the three deep learning models was more than 96%, and the accuracy of ResNet-152 and DenseNet-169 was more than 98%. The model ResNet-152 obtained accuracy of 99.04%, precision of 99.05%, recall of 99.04%, and F1-score of 99.05%. The performances were better than performances of other eight models. The eight models are VGG-16, DenseNet-169, SVM, RF, GBDT, Adaboost, Naïve Bayes, and DT. ResNet-152 was selected as quality-screening model for tongue IQA.

CONCLUSIONS

Our research findings demonstrate various CNN models in the decision-making process for the selection of tongue image quality assessment and indicate that applying deep learning methods, specifically deep CNNs, to evaluate poor-quality tongue images is feasible.

Cross domains adversarial learning for Chinese named entity recognition for online medical consultation

Deep learning methods have been applied to Chinese named entity recognition for the online medical consultation. They require a large number of marked samples. However, no such database is available at present. This paper begins with constructing a larger labelled Chinese texts database for the online medical consultation. Second, a basic framework unit is proposed, which is pre-trained by the transfer learning from both Bidirectional language model and Mask language model trained on the larger unlabelled data. Finally, cross domains adversarial learning (CDAL) for Chinese named entity recognition is proposed to further improve the performance, which not only uses the pre-trained basic framework unit, but also uses the adversarial multi-task learning on both electronic medical record texts and online medical consultation texts. Experimental results validate the effectiveness of CDAL.

Grouping attributes zero-shot learning for tongue constitution recognition

Traditional Chinese Medicine (TCM) considers that the personal constitution determines the occurrence trend and therapeutic effects of certain diseases, which can be recognized by machine learning through tongue images. However, current machine learning methods are confronted with two challenges. First, there are not some larger tongue image databases available. Second, they do not use the domain knowledge of TCM, so that the imbalance of constitution categories cannot be solved. Therefore, this paper proposes a new constitution recognition method based on the zero-shot learning with the knowledge of TCM. To further improve the performance, a new zero-shot learning method is proposed by grouping attributes and learning discriminant latent features, which can better solve the imbalance problem of constitution categories. Experimental results on our constructed databases validate the proposed methods.