An adaptive multi-modal hybrid model for classifying thyroid nodules by combining ultrasound and infrared thermal images

BACKGROUND

Two types of non-invasive, radiation-free, and inexpensive imaging technologies that are widely employed in medical applications are ultrasound (US) and infrared thermography (IRT). The ultrasound image obtained by ultrasound imaging primarily expresses the size, shape, contour boundary, echo, and other morphological information of the lesion, while the infrared thermal image obtained by infrared thermography imaging primarily describes its thermodynamic function information. Although distinguishing between benign and malignant thyroid nodules requires both morphological and functional information, present deep learning models are only based on US images, making it possible that some malignant nodules with insignificant morphological changes but significant functional changes will go undetected.

RESULTS

Given the US and IRT images present thyroid nodules through distinct modalities, we proposed an Adaptive multi-modal Hybrid (AmmH) classification model that can leverage the amalgamation of these two image types to achieve superior classification performance. The AmmH approach involves the construction of a hybrid single-modal encoder module for each modal data, which facilitates the extraction of both local and global features by integrating a CNN module and a Transformer module. The extracted features from the two modalities are then weighted adaptively using an adaptive modality-weight generation network and fused using an adaptive cross-modal encoder module. The fused features are subsequently utilized for the classification of thyroid nodules through the use of MLP. On the collected dataset, our AmmH model respectively achieved 97.17% and 97.38% of F1 and F2 scores, which significantly outperformed the single-modal models. The results of four ablation experiments further show the superiority of our proposed method.

CONCLUSIONS

The proposed multi-modal model extracts features from various modal images, thereby enhancing the comprehensiveness of thyroid nodules descriptions. The adaptive modality-weight generation network enables adaptive attention to different modalities, facilitating the fusion of features using adaptive weights through the adaptive cross-modal encoder. Consequently, the model has demonstrated promising classification performance, indicating its potential as a non-invasive, radiation-free, and cost-effective screening tool for distinguishing between benign and malignant thyroid nodules. The source code is available at https://github.com/wuliZN2020/AmmH .

Development of Automatic Portable Pathology Scanner and Its Evaluation for Clinical Practice

Digital pathological scanners transform traditional glass slides into whole slide images (WSIs), which significantly improve the efficiency of pathological diagnosis and promote the development of digital pathology. However, the huge economic burden limits the spread and application of general WSI scanners in relatively remote and backward regions. In this paper, we develop an automatic portable cytopathology scanner based on mobile internet, Landing-Smart, to avert the above problems. Landing-Smart is a tiny device with a size of 208 mm × 107 mm × 104 mm and a weight of 1.8 kg, which integrates four main components including a smartphone, a glass slide carrier, an electric controller, and an optical imaging unit. By leveraging a simple optical imaging unit to substitute the sophisticated but complex conventional light microscope, the cost of Landing-Smart is less than $3000, much cheaper than general WSI scanners. On the one hand, Landing-Smart utilizes the built-in camera of the smartphone to acquire field of views (FoVs) in the section one by one. On the other hand, it uploads the images to the cloud server in real time via mobile internet, where the image processing and stitching method is implemented to generate the WSI of the cytological sample. The practical assessment of 209 cervical cytological specimens has demonstrated that Landing-Smart is comparable to general digital scanners in cytopathology diagnosis. Landing-Smart provides an effective tool for preliminary cytological screening in underdeveloped areas.