The National Library of Medicine Pill Image Recognition Challenge: An Initial Report

High accurate and explainable multi-pill detection framework with graph neural network-assisted multimodal data fusion

Due to the significant resemblance in visual appearance, pill misuse is prevalent and has become a critical issue, responsible for one-third of all deaths worldwide. Pill identification, thus, is a crucial concern that needs to be investigated thoroughly. Recently, several attempts have been made to exploit deep learning to tackle the pill identification problem. However, most published works consider only single-pill identification and fail to distinguish hard samples with identical appearances. Also, most existing pill image datasets only feature single pill images captured in carefully controlled environments under ideal lighting conditions and clean backgrounds. In this work, we are the first to tackle the multi-pill detection problem in real-world settings, aiming at localizing and identifying pills captured by users during pill intake. Moreover, we also introduce a multi-pill image dataset taken in unconstrained conditions. To handle hard samples, we propose a novel method for constructing heterogeneous a priori graphs incorporating three forms of inter-pill relationships, including co-occurrence likelihood, relative size, and visual semantic correlation. We then offer a framework for integrating a priori with pills' visual features to enhance detection accuracy. Our experimental results have proved the robustness, reliability, and explainability of the proposed framework. Experimentally, it outperforms all detection benchmarks in terms of all evaluation metrics. Specifically, our proposed framework improves COCO mAP metrics by 9.4% over Faster R-CNN and 12.0% compared to vanilla YOLOv5. Our study opens up new opportunities for protecting patients from medication errors using an AI-based pill identification solution.

A Novel Two-Stage Induced Deep Learning System for Classifying Similar Drugs with Diverse Packaging

Dispensing errors play a crucial role in various medical errors, unfortunately emerging as the third leading cause of death in the United States. This alarming statistic has spurred the World Health Organization (WHO) into action, leading to the initiation of the Medication Without Harm Campaign. The primary objective of this campaign is to prevent dispensing errors from occurring and ensure patient safety. Due to the rapid development of deep learning technology, there has been a significant increase in the development of automatic dispensing systems based on deep learning classification to avoid dispensing errors. However, most previous studies have focused on developing deep learning classification systems for unpackaged pills or drugs with the same type of packaging. However, in the actual dispensing process, thousands of similar drugs with diverse packaging within a healthcare facility greatly increase the risk of dispensing errors. In this study, we proposed a novel two-stage induced deep learning (TSIDL)-based system to classify similar drugs with diverse packaging efficiently. The results demonstrate that the proposed TSIDL method outperforms state-of-the-art CNN models in all classification metrics. It achieved a state-of-the-art classification accuracy of 99.39%. Moreover, this study also demonstrated that the TSIDL method achieved an inference time of only 3.12 ms per image. These results highlight the potential of real-time classification for similar drugs with diverse packaging and their applications in future dispensing systems, which can prevent dispensing errors from occurring and ensure patient safety efficiently.

An Accurate Deep Learning-Based System for Automatic Pill Identification: Model Development and Validation

BACKGROUND

Medication errors account for a large proportion of all medical errors. In most homes, patients take a variety of medications for a long period. However, medication errors frequently occur because patients often throw away the containers of their medications.

OBJECTIVE

We proposed a deep learning-based system for reducing medication errors by accurately identifying prescription pills. Given the pill images, our system located the pills in the respective pill databases in South Korea and the United States.

METHODS

We organized the system into a pill recognition step and pill retrieval step, and we applied deep learning models to train not only images of the pill but also imprinted characters. In the pill recognition step, there are 3 modules that recognize the 3 features of pills and their imprints separately and correct the recognized imprint to fit the actual data. We adopted image classification and text detection models for the feature and imprint recognition modules, respectively. In the imprint correction module, we introduced a language model for the first time in the pill identification system and proposed a novel coordinate encoding technique for effective correction in the language model. We identified pills using similarity scores of pill characteristics with those in the database.

RESULTS

We collected the open pill database from South Korea and the United States in May 2022. We used a total of 24,404 pill images in our experiments. The experimental results show that the predicted top-1 candidates achieve accuracy levels of 85.6% (South Korea) and 74.5% (United States) for the types of pills not trained on 2 different databases (South Korea and the United States). Furthermore, the predicted top-1 candidate accuracy of our system was 78% with consumer-granted images, which was achieved by training only 1 image per pill. The results demonstrate that our system could identify and retrieve new pills without additional model updates. Finally, we confirmed through an ablation study that the language model that we emphasized significantly improves the pill identification ability of the system.

CONCLUSIONS

Our study proposes the possibility of reducing medical errors by showing that the introduction of artificial intelligence can identify numerous pills with high precision in real time. Our study suggests that the proposed system can reduce patients' misuse of medications and help medical staff focus on higher-level tasks by simplifying time-consuming lower-level tasks such as pill identification.

Development of an Automatic Pill Image Data Generation System

OBJECTIVES

Since the easiest way to identify pills and obtain information about them is to distinguish them visually, many studies on image processing technology exist. However, no automatic system for generating pill image data has yet been developed. Therefore, we propose a system for automatically generating image data by taking pictures of pills from various angles. This system is referred to as the pill filming system in this paper.

METHODS

We designed the pill filming system to have three components: structure, controller, and a graphical user interface (GUI). This system was manufactured with black polylactic acid using a 3D printer for lightweight and easy manufacturing. The mainboard controls data storage, and the entire process is managed through the GUI. After one reciprocating movement of the seesaw, the web camera at the top shoots the target pill on the stage. This image is then saved in a specific directory on the mainboard.

RESULTS

The pill filming system completes its workflow after generating 300 pill images. The total time to collect data per pill takes 21 minutes and 25 seconds. The generated image size is 1280 × 960 pixels, the horizontal and vertical resolutions are both 96 DPI (dot per inch), and the file extension is .jpg.

CONCLUSIONS

This paper proposes a system that can automatically generate pill image data from various angles. The pill observation data from various angles include many cases. In addition, the data collected in the same controlled environment have a uniform background, making it easy to process the images. Large quantities of high-quality data from the pill filming system can contribute to various studies using pill images.

A Study of the Recent Trends of Immunology: Key Challenges, Domains, Applications, Datasets, and Future Directions

The human immune system is very complex. Understanding it traditionally required specialized knowledge and expertise along with years of study. However, in recent times, the introduction of technologies such as AIoMT (Artificial Intelligence of Medical Things), genetic intelligence algorithms, smart immunological methodologies, etc., has made this process easier. These technologies can observe relations and patterns that humans do and recognize patterns that are unobservable by humans. Furthermore, these technologies have also enabled us to understand better the different types of cells in the immune system, their structures, their importance, and their impact on our immunity, particularly in the case of debilitating diseases such as cancer. The undertaken study explores the AI methodologies currently in the field of immunology. The initial part of this study explains the integration of AI in healthcare and how it has changed the face of the medical industry. It also details the current applications of AI in the different healthcare domains and the key challenges faced when trying to integrate AI with healthcare, along with the recent developments and contributions in this field by other researchers. The core part of this study is focused on exploring the most common classifications of health diseases, immunology, and its key subdomains. The later part of the study presents a statistical analysis of the contributions in AI in the different domains of immunology and an in-depth review of the machine learning and deep learning methodologies and algorithms that can and have been applied in the field of immunology. We have also analyzed a list of machine learning and deep learning datasets about the different subdomains of immunology. Finally, in the end, the presented study discusses the future research directions in the field of AI in immunology and provides some possible solutions for the same.

Comparison of RetinaNet, SSD, and YOLO v3 for real-time pill identification

BACKGROUND

The correct identification of pills is very important to ensure the safe administration of drugs to patients. Here, we use three current mainstream object detection models, namely RetinaNet, Single Shot Multi-Box Detector (SSD), and You Only Look Once v3(YOLO v3), to identify pills and compare the associated performance.

METHODS

In this paper, we introduce the basic principles of three object detection models. We trained each algorithm on a pill image dataset and analyzed the performance of the three models to determine the best pill recognition model. The models were then used to detect difficult samples and we compared the results.

RESULTS

The mean average precision (MAP) of RetinaNet reached 82.89%, but the frames per second (FPS) is only one third of YOLO v3, which makes it difficult to achieve real-time performance. SSD does not perform as well on the indicators of MAP and FPS. Although the MAP of YOLO v3 is slightly lower than the others (80.69%), it has a significant advantage in terms of detection speed. YOLO v3 also performed better when tasked with hard sample detection, and therefore the model is more suitable for deployment in hospital equipment.

CONCLUSION

Our study reveals that object detection can be applied for real-time pill identification in a hospital pharmacy, and YOLO v3 exhibits an advantage in detection speed while maintaining a satisfactory MAP.

Effects of Background Colors, Flashes, and Exposure Values on the Accuracy of a Smartphone-Based Pill Recognition System Using a Deep Convolutional Neural Network: Deep Learning and Experimental Approach

BACKGROUND

Pill image recognition systems are difficult to develop due to differences in pill color, which are influenced by external factors such as the illumination from and the presence of a flash.

OBJECTIVE

In this study, the differences in color between reference images and real-world images were measured to determine the accuracy of a pill recognition system under 12 real-world conditions (ie, different background colors, the presence and absence of a flash, and different exposure values [EVs]).

METHODS

We analyzed 19 medications with different features (ie, different colors, shapes, and dosages). The average color difference was calculated based on the color distance between a reference image and a real-world image.

RESULTS

For images with black backgrounds, as the EV decreased, the top-1 and top-5 accuracies increased independently of the presence of a flash. The top-5 accuracy for images with black backgrounds increased from 26.8% to 72.6% when the flash was on and increased from 29.5% to 76.8% when the flash was off as the EV decreased. However, the top-5 accuracy increased from 62.1% to 78.4% for images with white backgrounds when the flash was on. The best top-1 accuracy was 51.1% (white background; flash on; EV of +2.0). The best top-5 accuracy was 78.4% (white background; flash on; EV of 0).

CONCLUSIONS

The accuracy generally increased as the color difference decreased, except for images with black backgrounds and an EV of -2.0. This study revealed that background colors, the presence of a flash, and EVs in real-world conditions are important factors that affect the performance of a pill recognition model.