A novel approach for identification of zoonotic trypanosome utilizing deep metric learning and vector database-based image retrieval system

Trypanosomiasis, a significant health concern in South America, South Asia, and Southeast Asia, requires active surveys to effectively control the disease. To address this, we have developed a hybrid model that combines deep metric learning (DML) and image retrieval. This model is proficient at identifying Trypanosoma species in microscopic images of thin-blood film examinations. Utilizing the ResNet50 backbone neural network, a trained-model has demonstrated outstanding performance, achieving an accuracy exceeding 99.71 % and up to 96 % in recall. Acknowledging the necessity for automated tools in field scenarios, we demonstrated the potential of our model as an autonomous screening approach. This was achieved by using prevailing convolutional neural network (CNN) applications, and vector database based-images returned by the KNN algorithm. This achievement is primarily attributed to the implementation of the Triplet Margin Loss function as 98 % of precision. The robustness of the model demonstrated in five-fold cross-validation highlights the ResNet50 neural network, based on DML, as a state-of-the-art CNN model as AUC >98 %. The adoption of DML significantly improves the performance of the model, remaining unaffected by variations in the dataset and rendering it a useful tool for fieldwork studies. DML offers several advantages over conventional classification model to manage large-scale datasets with a high volume of classes, enhancing scalability. The model has the capacity to generalize to novel classes that were not encountered during training, proving particularly advantageous in scenarios where new classes may consistently emerge. It is also well suited for applications requiring precise recognition, especially in discriminating between closely related classes. Furthermore, the DML exhibits greater resilience to issues related to class imbalance, as it concentrates on learning distances or similarities, which are more tolerant to such imbalances. These contributions significantly make the effectiveness and practicality of DML model, particularly in in fieldwork research.

Evaluation of retrieval accuracy and visual similarity in content-based image retrieval of chest CT for obstructive lung disease

The aim of our study was to assess the performance of content-based image retrieval (CBIR) for similar chest computed tomography (CT) in obstructive lung disease. This retrospective study included patients with obstructive lung disease who underwent volumetric chest CT scans. The CBIR database included 600 chest CT scans from 541 patients. To assess the system performance, follow-up chest CT scans of 50 patients were evaluated as query cases, which showed the stability of the CT findings between baseline and follow-up chest CT, as confirmed by thoracic radiologists. The CBIR system retrieved the top five similar CT scans for each query case from the database by quantifying and comparing emphysema extent and size, airway wall thickness, and peripheral pulmonary vasculatures in descending order from the database. The rates of retrieval of the same pairs of query CT scans in the top 1-5 retrievals were assessed. Two expert chest radiologists evaluated the visual similarities between the query and retrieved CT scans using a five-point scale grading system. The rates of retrieving the same pairs of query CTs were 60.0% (30/50) and 68.0% (34/50) for top-three and top-five retrievals. Radiologists rated 64.8% (95% confidence interval 58.8-70.4) of the retrieved CT scans with a visual similarity score of four or five and at least one case scored five points in 74% (74/100) of all query cases. The proposed CBIR system for obstructive lung disease integrating quantitative CT measures demonstrated potential for retrieving chest CT scans with similar imaging phenotypes. Further refinement and validation in this field would be valuable.

Sketch-based semantic retrieval of medical images

The volume of medical images stored in hospitals is rapidly increasing; however, the utilization of these accumulated medical images remains limited. Existing content-based medical image retrieval (CBMIR) systems typically require example images, leading to practical limitations, such as the lack of customizable, fine-grained image retrieval, the inability to search without example images, and difficulty in retrieving rare cases. In this paper, we introduce a sketch-based medical image retrieval (SBMIR) system that enables users to find images of interest without the need for example images. The key concept is feature decomposition of medical images, which allows the entire feature of a medical image to be decomposed into and reconstructed from normal and abnormal features. Building on this concept, our SBMIR system provides an easy-to-use two-step graphical user interface: users first select a template image to specify a normal feature and then draw a semantic sketch of the disease on the template image to represent an abnormal feature. The system integrates both types of input to construct a query vector and retrieves reference images. For evaluation, ten healthcare professionals participated in a user test using two datasets. Consequently, our SBMIR system enabled users to overcome previous challenges, including image retrieval based on fine-grained image characteristics, image retrieval without example images, and image retrieval for rare cases. Our SBMIR system provides on-demand, customizable medical image retrieval, thereby expanding the utility of medical image databases.

Medical Image Retrieval via Nearest Neighbor Search on Pre-trained Image Features

Nearest neighbor search, also known as NNS, is a technique used to locate the points in a high-dimensional space closest to a given query point. This technique has multiple applications in medicine, such as searching large medical imaging databases, disease classification, and diagnosis. However, when the number of points is significantly large, the brute-force approach for finding the nearest neighbor becomes computationally infeasible. Therefore, various approaches have been developed to make the search faster and more efficient to support the applications. With a focus on medical imaging, this paper proposes DenseLinkSearch (DLS), an effective and efficient algorithm that searches and retrieves the relevant images from heterogeneous sources of medical images. Towards this, given a medical database, the proposed algorithm builds an index that consists of pre-computed links of each point in the database. The search algorithm utilizes the index to efficiently traverse the database in search of the nearest neighbor. We also explore the role of medical image feature representation in content-based medical image retrieval tasks. We propose a Transformer-based feature representation technique that outperformed the existing pre-trained Transformer-based approaches on benchmark medical image retrieval datasets. We extensively tested the proposed NNS approach and compared the performance with state-of-the-art NNS approaches on benchmark datasets and our created medical image datasets. The proposed approach outperformed the existing approaches in terms of retrieving accurate neighbors and retrieval speed. In comparison to the existing approximate NNS approaches, our proposed DLS approach outperformed them in terms of lower average time per query and ≥ 99% R@10 on 11 out of 13 benchmark datasets. We also found that the proposed medical feature representation approach is better for representing medical images compared to the existing pre-trained image models. The proposed feature extraction strategy obtained an improvement of 9.37%, 7.0%, and 13.33% in terms of P@5, P@10, and P@20, respectively, in comparison to the best-performing pre-trained image model. The source code and datasets of our experiments are available at https://github.com/deepaknlp/DLS.

Histopathological Image Deep Feature Representation for CBIR in Smart PACS

Pathological Anatomy is moving toward computerizing processes mainly due to the extensive digitization of histology slides that resulted in the availability of many Whole Slide Images (WSIs). Their use is essential, especially in cancer diagnosis and research, and raises the pressing need for increasingly influential information archiving and retrieval systems. Picture Archiving and Communication Systems (PACSs) represent an actual possibility to archive and organize this growing amount of data. The design and implementation of a robust and accurate methodology for querying them in the pathology domain using a novel approach are mandatory. In particular, the Content-Based Image Retrieval (CBIR) methodology can be involved in the PACSs using a query-by-example task. In this context, one of many crucial points of CBIR concerns the representation of images as feature vectors, and the accuracy of retrieval mainly depends on feature extraction. Thus, our study explored different representations of WSI patches by features extracted from pre-trained Convolution Neural Networks (CNNs). In order to perform a helpful comparison, we evaluated features extracted from different layers of state-of-the-art CNNs using different dimensionality reduction techniques. Furthermore, we provided a qualitative analysis of obtained results. The evaluation showed encouraging results for our proposed framework.

A clinically motivated self-supervised approach for content-based image retrieval of CT liver images

Deep learning-based approaches for content-based image retrieval (CBIR) of computed tomography (CT) liver images is an active field of research, but suffer from some critical limitations. First, they are heavily reliant on labeled data, which can be challenging and costly to acquire. Second, they lack transparency and explainability, which limits the trustworthiness of deep CBIR systems. We address these limitations by: (1) Proposing a self-supervised learning framework that incorporates domain-knowledge into the training procedure, and, (2) by providing the first representation learning explainability analysis in the context of CBIR of CT liver images. Results demonstrate improved performance compared to the standard self-supervised approach across several metrics, as well as improved generalization across datasets. Further, we conduct the first representation learning explainability analysis in the context of CBIR, which reveals new insights into the feature extraction process. Lastly, we perform a case study with cross-examination CBIR that demonstrates the usability of our proposed framework. We believe that our proposed framework could play a vital role in creating trustworthy deep CBIR systems that can successfully take advantage of unlabeled data.

Co-attention enabled content-based image retrieval

Content-based image retrieval (CBIR) aims to provide the most similar images to a given query. Feature extraction plays an essential role in retrieval performance within a CBIR pipeline. Current CBIR studies would either uniformly extract feature information from the input image and use it directly or employ some trainable spatial weighting module which is then used for similarity comparison between pairs of query and candidate matching images. These spatial weighting modules are normally query non-sensitive and only based on the knowledge learned during the training stage. They may focus towards incorrect regions, especially when the target image is not salient or is surrounded by distractors. This paper proposes an efficient query sensitive co-attention¹ mechanism for large-scale CBIR tasks. In order to reduce the extra computation cost required by the query sensitivity to the co-attention mechanism, the proposed method employs clustering of the selected local features. Experimental results indicate that the co-attention maps can provide the best retrieval results on benchmark datasets under challenging situations, such as having completely different image acquisition conditions between the query and its match image.

ExpertosLF: dynamic late fusion of CBIR systems using online learning with relevance feedback

One of the main challenges in CBIR systems is to choose discriminative and compact features, among dozens, to represent the images under comparison. Over the years, a great effort has been made to combine multiple features, mainly using early, late, and hierarchical fusion techniques. Unveiling the perfect combination of features is highly domain-specific and dependent on the type of image. Thus, the process of designing a CBIR system for new datasets or domains involves a huge experimentation overhead, leading to multiple fine-tuned CBIR systems. It would be desirable to dynamically find the best combination of CBIR systems without needing to go through such extensive experimentation and without requiring previous domain knowledge. In this paper, we propose ExpertosLF, a model-agnostic interpretable late fusion technique based on online learning with expert advice, which dynamically combines CBIR systems without knowing a priori which ones are the best for a given domain. At each query, ExpertosLF takes advantage of user's feedback to determine each CBIR contribution in the ensemble for the following queries. ExpertosLF produces an interpretable ensemble that is independent of the dataset and domain. Moreover, ExpertosLF is designed to be modular, and scalable. Experiments on 13 benchmark datasets from the Biomedical, Real, and Sketch domains revealed that: (i) ExpertosLF surpasses the performance of state of the art late-fusion techniques; (ii) it successfully and quickly converges to the performance of the best CBIR sets across domains without any previous domain knowledge (in most cases, fewer than 25 queries need to receive human feedback).

Content-based medical image retrieval system for lung diseases using deep CNNs

Content-based image retrieval (CBIR) systems are designed to retrieve images that are relevant, based on detailed analysis of latent image characteristics, thus eliminating the dependency of natural language tags, text descriptions, or keywords associated with the images. A CBIR system maintains high-level image visuals in the form of feature vectors, which the retrieval engine leverages for similarity-based matching and ranking for a given query image. In this paper, a CBIR system is proposed for the retrieval of medical images (CBMIR) for enabling the early detection and classification of lung diseases based on lung X-ray images. The proposed CBMIR system is built on the predictive power of deep neural models for the identification and classification of disease-specific features using transfer learning based models trained on standard COVID-19 Chest X-ray image datasets. Experimental evaluation on the standard dataset revealed that the proposed approach achieved an improvement of 49.71% in terms of precision, averaging across various distance metrics. Also, an improvement of 26.55% was observed in the area under precision-recall curve (AUPRC) values across all subclasses.

Securing content-based image retrieval on the cloud using generative models

Content-based image retrieval (CBIR) with deep neural networks (DNNs) on the cloud has tremendous business and technical advantages to handle large-scale image repositories. However, cloud-based CBIR service raises challenges in image data and DNN model security. Typically, users who wish to request CBIR services on the cloud require their input images remaining confidential. On the other hand, image owners may intentionally (or unintentionally) upload adversarial examples to the cloud servers, which potentially leads to the misbehavior of CBIR services. Generative Adversarial Networks (GANs) can be utilized to defense against such malicious behavior. However, the GANs model, if not well protected, can be easily abused by the cloud to reconstruct the users' original image data. In this paper, we focus on the problem of secure generative model evaluation and secure gradient descent (GD) computation in GANs. We propose two secure generative model evaluation algorithms and two secure minimizer protocols. Furthermore, we propose and implement Sec-Defense-Gan, a secure image reconstruction framework which can keep the image data, the generative model details and corresponding outputs confidential from the cloud. Finally, We carried out a set of benchmarks over two public available image datasets to show the performance and correctness of Sec-Defense-Gan.

Towards non-data-hungry and fully-automated diagnosis of breast cancer from mammographic images

Analysing local texture and generating features are two key issues for automatic cancer detection in mammographic images. Recent researches have shown that deep neural networks provide a promising alternative to hand-driven features which suffer from curse of dimensionality and low accuracy rates. However, large and balanced training data are foremost requirements for deep learning-based models and these data are not always available publicly. In this work, we propose a fully-automated method for breast cancer diagnosis that performs training using small sets of data. Feature extraction from mammographic images is performed using a genetic-programming-based descriptor that exploits statistics on a local binary pattern-like local distribution defined in each pixel. The effectiveness of the suggested method is demonstrated on two challenging datasets, (1) the digital database for screening mammography and (2) the mammographic image analysis society digital mammogram database, for content-based image retrieval as well as for abnormality/malignancy classification. The experimental results show that the proposed method outperforms or achieves comparable results with deep learning-based methods even those with transfer learning and/or data-augmentation.

Detection of pneumonia infection in lungs from chest X-ray images using deep convolutional neural network and content-based image retrieval techniques

In this research, A Deep Convolutional Neural Network was proposed to detect Pneumonia infection in the lung using Chest X-ray images. The proposed Deep CNN models were trained with a Pneumonia Chest X-ray Dataset containing 12,000 images of infected and not infected chest X-ray images. The dataset was preprocessed and developed from the Chest X-ray8 dataset. The Content-based image retrieval technique was used to annotate the images in the dataset using Metadata and further contents. The data augmentation techniques were used to increase the number of images in each of class. The basic manipulation techniques and Deep Convolutional Generative Adversarial Network (DCGAN) were used to create the augmented images. The VGG19 network was used to develop the proposed Deep CNN model. The classification accuracy of the proposed Deep CNN model was 99.34 percent in the unseen chest X-ray images. The performance of the proposed deep CNN was compared with state-of-the-art transfer learning techniques such as AlexNet, VGG16Net and InceptionNet. The comparison results show that the classification performance of the proposed Deep CNN model was greater than the other techniques.

Content-Based Image Retrieval of Chest CT with Convolutional Neural Network for Diffuse Interstitial Lung Disease: Performance Assessment in Three Major Idiopathic Interstitial Pneumonias

Objective

To assess the performance of content-based image retrieval (CBIR) of chest CT for diffuse interstitial lung disease (DILD).

Materials and Methods

The database was comprised by 246 pairs of chest CTs (initial and follow-up CTs within two years) from 246 patients with usual interstitial pneumonia (UIP, n = 100), nonspecific interstitial pneumonia (NSIP, n = 101), and cryptogenic organic pneumonia (COP, n = 45). Sixty cases (30-UIP, 20-NSIP, and 10-COP) were selected as the queries. The CBIR retrieved five similar CTs as a query from the database by comparing six image patterns (honeycombing, reticular opacity, emphysema, ground-glass opacity, consolidation and normal lung) of DILD, which were automatically quantified and classified by a convolutional neural network. We assessed the rates of retrieving the same pairs of query CTs, and the number of CTs with the same disease class as query CTs in top 1–5 retrievals. Chest radiologists evaluated the similarity between retrieved CTs and queries using a 5-scale grading system (5-almost identical; 4-same disease; 3-likelihood of same disease is half; 2-likely different; and 1-different disease).

Results

The rate of retrieving the same pairs of query CTs in top 1 retrieval was 61.7% (37/60) and in top 1–5 retrievals was 81.7% (49/60). The CBIR retrieved the same pairs of query CTs more in UIP compared to NSIP and COP (p = 0.008 and 0.002). On average, it retrieved 4.17 of five similar CTs from the same disease class. Radiologists rated 71.3% to 73.0% of the retrieved CTs with a similarity score of 4 or 5.

Conclusion

The proposed CBIR system showed good performance for retrieving chest CTs showing similar patterns for DILD.

Content-Based Image Retrieval System for Pulmonary Nodules Using Optimal Feature Sets and Class Membership-Based Retrieval

Lung cancer manifests itself in the form of lung nodules, the diagnosis of which is essential to plan the treatment. Automated retrieval of nodule cases will assist the budding radiologists in self-learning and differential diagnosis. This paper presents a content-based image retrieval (CBIR) system for lung nodules using optimal feature sets and learning to enhance the performance of retrieval. The classifiers with more features suffer from the curse of dimensionality. Like classification schemes, we found that the optimal feature set selected using the minimal-redundancy-maximal-relevance (mRMR) feature selection technique improves the precision performance of simple distance-based retrieval (SDR). The performance of the classifier is always superior to SDR, which leans researchers towards conventional classifier-based retrieval (CCBR). While CCBR improves the average precision and provides 100% precision for correct classification, it fails for misclassification leading to zero retrieval precision. The class membership-based retrieval (CMR) is found to bridge this gap for texture-based retrieval. Here, CMR is proposed for nodule retrieval using shape-, margin-, and texture-based features. It is found again that optimal feature set is important for the classifier used in CMR as well as for the feature set used for retrieval, which may lead to different feature sets. The proposed system is evaluated using two independent databases from two continents: a public database LIDC/IDRI and a private database PGIMER-IITKGP, using three distance metrics, i.e., Canberra, City block, and Euclidean. The proposed CMR-based retrieval system with optimal feature sets performs better than CCBR and SDR with optimal features in terms of average precision. Apart from average precision and standard deviation of precision, the fraction of queries with zero precision retrieval is also measured.

Breast cancer diagnosis from histopathological images using textural features and CBIR

Currently, breast cancer diagnosis is an extensively researched topic. An effective method to diagnose breast cancer is to use histopathological images. However, extracting features from these images is a challenging task. Thus, we propose a method that uses phylogenetic diversity indexes to characterize images for creating a model to classify histopathological breast images into four classes - invasive carcinoma, in situ carcinoma, normal tissue, and benign lesion. The classifiers used were the most robust ones according to the existing literature: XGBoost, random forest, multilayer perceptron, and support vector machine. Moreover, we performed content-based image retrieval to confirm the classification results and suggest a ranking for sets of images that were not labeled. The results obtained were considerably robust and proved to be effective for the composition of a CADx system to help specialists at large medical centers.

Polyp fingerprint: automatic recognition of colorectal polyps' unique features

BACKGROUND

Content-based image retrieval (CBIR) is an application of machine learning used to retrieve images by similarity on the basis of features. Our objective was to develop a CBIR system that could identify images containing the same polyp ('polyp fingerprint').

METHODS

A machine learning technique called Bag of Words was used to describe each endoscopic image containing a polyp in a unique way. The system was tested with 243 white light images belonging to 99 different polyps (for each polyp there were at least two images representing it in two different temporal moments). Images were acquired in routine colonoscopies at Hospital Clínic using high-definition Olympus endoscopes. The method provided for each image the closest match within the dataset.

RESULTS

The system matched another image of the same polyp in 221/243 cases (91%). No differences were observed in the number of correct matches according to Paris classification (protruded: 90.7% vs. non-protruded: 91.3%) and size (< 10 mm: 91.6% vs. > 10 mm: 90%).

CONCLUSIONS

A CBIR system can match accurately two images containing the same polyp, which could be a helpful aid for polyp image recognition.

Automated Anatomic Labeling Architecture for Content Discovery in Medical Imaging Repositories

The combination of textual data with visual features is known to enhance medical image search capabilities. However, the most advanced imaging archives today only index the studies' available meta-data, often containing limited amounts of clinically useful information. This work proposes an anatomic labeling architecture, integrated with an open source archive software, for improved multimodal content discovery in real-world medical imaging repositories. The proposed solution includes a technical specification for classifiers in an extensible medical imaging archive, a classification database for querying over the extracted information, and a set of proof-of-concept convolutional neural network classifiers for identifying the presence of organs in computed tomography scans. The system automatically extracts the anatomic region features, which are saved in the proposed database for later consumption by multimodal querying mechanisms. The classifiers were evaluated with cross-validation, yielding a best F1-score of 96% and an average accuracy of 97%. We expect these capabilities to become common-place in production environments in the future, as automated detection solutions improve in terms of accuracy, computational performance, and interoperability.

Generating region proposals for histopathological whole slide image retrieval

BACKGROUND AND OBJECTIVE

Content-based image retrieval is an effective method for histopathological image analysis. However, given a database of huge whole slide images (WSIs), acquiring appropriate region-of-interests (ROIs) for training is significant and difficult. Moreover, histopathological images can only be annotated by pathologists, resulting in the lack of labeling information. Therefore, it is an important and challenging task to generate ROIs from WSI and retrieve image with few labels.

METHODS

This paper presents a novel unsupervised region proposing method for histopathological WSI based on Selective Search. Specifically, the WSI is over-segmented into regions which are hierarchically merged until the WSI becomes a single region. Nucleus-oriented similarity measures for region mergence and Nucleus-Cytoplasm color space for histopathological image are specially defined to generate accurate region proposals. Additionally, we propose a new semi-supervised hashing method for image retrieval. The semantic features of images are extracted with Latent Dirichlet Allocation and transformed into binary hashing codes with Supervised Hashing.

RESULTS

The methods are tested on a large-scale multi-class database of breast histopathological WSIs. The results demonstrate that for one WSI, our region proposing method can generate 7.3 thousand contoured regions which fit well with 95.8% of the ROIs annotated by pathologists. The proposed hashing method can retrieve a query image among 136 thousand images in 0.29 s and reach precision of 91% with only 10% of images labeled.

CONCLUSIONS

The unsupervised region proposing method can generate regions as predictions of lesions in histopathological WSI. The region proposals can also serve as the training samples to train machine-learning models for image retrieval. The proposed hashing method can achieve fast and precise image retrieval with small amount of labels. Furthermore, the proposed methods can be potentially applied in online computer-aided-diagnosis systems.

Content-Based Image Retrieval System for Pulmonary Nodules: Assisting Radiologists in Self-Learning and Diagnosis of Lung Cancer

Visual information of similar nodules could assist the budding radiologists in self-learning. This paper presents a content-based image retrieval (CBIR) system for pulmonary nodules, observed in lung CT images. The reported CBIR systems of pulmonary nodules cannot be put into practice as radiologists need to draw the boundary of nodules during query formation and feature database creation. In the proposed retrieval system, the pulmonary nodules are segmented using a semi-automated technique, which requires a seed point on the nodule from the end-user. The involvement of radiologists in feature database creation is also reduced, as only a seed point is expected from radiologists instead of manual delineation of the boundary of the nodules. The performance of the retrieval system depends on the accuracy of the segmentation technique. Several 3D features are explored to improve the performance of the proposed retrieval system. A set of relevant shape and texture features are considered for efficient representation of the nodules in the feature space. The proposed CBIR system is evaluated for three configurations such as configuration-1 (composite rank of malignancy "1","2" as benign and "4","5" as malignant), configuration-2 (composite rank of malignancy "1","2", "3" as benign and "4","5" as malignant), and configuration-3 (composite rank of malignancy "1","2" as benign and "3","4","5" as malignant). Considering top 5 retrieved nodules and Euclidean distance metric, the precision achieved by the proposed method for configuration-1, configuration-2, and configuration-3 are 82.14, 75.91, and 74.27 %, respectively. The performance of the proposed CBIR system is close to the most recent technique, which is dependent on radiologists for manual segmentation of nodules. A computer-aided diagnosis (CAD) system is also developed based on CBIR paradigm. Performance of the proposed CBIR-based CAD system is close to performance of the CAD system using support vector machine.

Content-based image retrieval for Lung Nodule Classification Using Texture Features and Learned Distance Metric

Similarity measurement of lung nodules is a critical component in content-based image retrieval (CBIR), which can be useful in differentiating between benign and malignant lung nodules on computer tomography (CT). This paper proposes a new two-step CBIR scheme (TSCBIR) for computer-aided diagnosis of lung nodules. Two similarity metrics, semantic relevance and visual similarity, are introduced to measure the similarity of different nodules. The first step is to search for K most similar reference ROIs for each queried ROI with the semantic relevance metric. The second step is to weight each retrieved ROI based on its visual similarity to the queried ROI. The probability is computed to predict the likelihood of the queried ROI depicting a malignant lesion. In order to verify the feasibility of the proposed algorithm, a lung nodule dataset including 366 nodule regions of interest (ROIs) is assembled from LIDC-IDRI lung images on CT scans. Three groups of texture features are implemented to represent a nodule ROI. Our experimental results on the assembled lung nodule dataset show good performance improvement over existing popular classifiers.

Neighbourhood search feature selection method for content-based mammogram retrieval

Content-based image retrieval plays an increasing role in the clinical process for supporting diagnosis. This paper proposes a neighbourhood search method to select the near-optimal feature subsets for the retrieval of mammograms from the Mammographic Image Analysis Society (MIAS) database. The features based on grey level cooccurrence matrix, Daubechies-4 wavelet, Gabor, Cohen-Daubechies-Feauveau 9/7 wavelet and Zernike moments are extracted from mammograms available in the MIAS database to form the combined or fused feature set for testing various feature selection methods. The performance of feature selection methods is evaluated using precision, storage requirement and retrieval time measures. Using the proposed method, a significant improvement is achieved in mean precision rate and feature dimension. The results show that the proposed method outperforms the state-of-the-art feature selection methods.

A Multimodal Search Engine for Medical Imaging Studies

The use of digital medical imaging systems in healthcare institutions has increased significantly, and the large amounts of data in these systems have led to the conception of powerful support tools: recent studies on content-based image retrieval (CBIR) and multimodal information retrieval in the field hold great potential in decision support, as well as for addressing multiple challenges in healthcare systems, such as computer-aided diagnosis (CAD). However, the subject is still under heavy research, and very few solutions have become part of Picture Archiving and Communication Systems (PACS) in hospitals and clinics. This paper proposes an extensible platform for multimodal medical image retrieval, integrated in an open-source PACS software with profile-based CBIR capabilities. In this article, we detail a technical approach to the problem by describing its main architecture and each sub-component, as well as the available web interfaces and the multimodal query techniques applied. Finally, we assess our implementation of the engine with computational performance benchmarks.

A new method of content based medical image retrieval and its applications to CT imaging sign retrieval

This paper proposes a new method of content based medical image retrieval through considering fused, context-sensitive similarity. Firstly, we fuse the semantic and visual similarities between the query image and each image in the database as their pairwise similarities. Then, we construct a weighted graph whose nodes represent the images and edges measure their pairwise similarities. By using the shortest path algorithm over the weighted graph, we obtain a new similarity measure, context-sensitive similarity measure, between the query image and each database image to complete the retrieval process. Actually, we use the fused pairwise similarity to narrow down the semantic gap for obtaining a more accurate pairwise similarity measure, and spread it on the intrinsic data manifold to achieve the context-sensitive similarity for a better retrieval performance. The proposed method has been evaluated on the retrieval of the Common CT Imaging Signs of Lung Diseases (CISLs) and achieved not only better retrieval results but also the satisfactory computation efficiency.

Automatic weighing attribute to retrieve similar lung cancer nodules

BACKGROUND

Cancer is a disease characterized as an uncontrolled growth of abnormal cells that invades neighboring tissues and destroys them. Lung cancer is the primary cause of cancer-related deaths in the world, and it diagnosis is a complex task for specialists and it presents some big challenges as medical image interpretation process, pulmonary nodule detection and classification. In order to aid specialists in the early diagnosis of lung cancer, computer assistance must be integrated in the imaging interpretation and pulmonary nodule classification processes. Methods of Content-Based Image Retrieval (CBIR) have been described as one promising technique to computer-aided diagnosis and is expected to aid radiologists on image interpretation with a second opinion. However, CBIR presents some limitations: image feature extraction process and appropriate similarity measure. The efficiency of CBIR systems depends on calculating image features that may be relevant to the case similarity analysis. When specialists classify a nodule, they are supported by information from exams, images, etc. But each information has more or less weight over decision making about nodule malignancy. Thus, finding a way to measure the weight allows improvement of image retrieval process through the assignment of higher weights to that attributes that best characterize the nodules.

METHODS

In this context, the aim of this work is to present a method to automatically calculate attribute weights based on local learning to reflect the interpretation on image retrieval process. The process consists of two stages that are performed sequentially and cyclically: Evaluation Stage and Training Stage. At each iteration the weights are adjusted according to retrieved nodules. After some iterations, it is possible reach a set of attribute weights that optimize the recovery of similar nodes.

RESULTS

The results achieved by updated weights were promising because was possible increase precision by 10% to 6% on average to retrieve of benign and malignant nodules, respectively, with recall of 25% compared with tests without weights associated to attributes in similarity metric. The best result, we reaching values over 100% of precision average until thirtieth lung cancer nodule retrieved.

CONCLUSIONS

Based on the results, WED applied to the three vectors used attributes (3D TA, 3D MSA and InV), with weights adjusted by the process, always achieved better results than those found with ED. With the weights, the Precision was increased on average by 17.3% compared with using ED.

Adapting content-based image retrieval techniques for the semantic annotation of medical images

The automatic annotation of medical images is a prerequisite for building comprehensive semantic archives that can be used to enhance evidence-based diagnosis, physician education, and biomedical research. Annotation also has important applications in the automatic generation of structured radiology reports. Much of the prior research work has focused on annotating images with properties such as the modality of the image, or the biological system or body region being imaged. However, many challenges remain for the annotation of high-level semantic content in medical images (e.g., presence of calcification, vessel obstruction, etc.) due to the difficulty in discovering relationships and associations between low-level image features and high-level semantic concepts. This difficulty is further compounded by the lack of labelled training data. In this paper, we present a method for the automatic semantic annotation of medical images that leverages techniques from content-based image retrieval (CBIR). CBIR is a well-established image search technology that uses quantifiable low-level image features to represent the high-level semantic content depicted in those images. Our method extends CBIR techniques to identify or retrieve a collection of labelled images that have similar low-level features and then uses this collection to determine the best high-level semantic annotations. We demonstrate our annotation method using retrieval via weighted nearest-neighbour retrieval and multi-class classification to show that our approach is viable regardless of the underlying retrieval strategy. We experimentally compared our method with several well-established baseline techniques (classification and regression) and showed that our method achieved the highest accuracy in the annotation of liver computed tomography (CT) images.

Differential geometry-based techniques for characterization of boundary roughness of pulmonary nodules in CT images

PURPOSE

Boundary roughness of a pulmonary nodule is an important indication of its malignancy. The irregularity of the shape of a nodule is represented in terms of a few diagnostic characteristics such as spiculation, lobulation, and sphericity. Quantitative characterization of these diagnostic characteristics is essential for designing a content-based image retrieval system and computer-aided system for diagnosis of lung cancer.

METHODS

This paper presents differential geometry-based techniques for computation of spiculation, lobulation, and sphericity using the binary mask of the segmented nodule. These shape features are computed in 3D considering complete nodule.

RESULTS

The performance of the proposed and competing methods is evaluated in terms of the precision, mean similarity, and normalized discounted cumulative gain on 891 nodules of Lung Image Database Consortium and Image Database Resource Initiative. The proposed methods are comparable to or better than gold standard technique. The reproducibility of proposed feature extraction techniques is evaluated using RIDER coffee break data set. The mean and standard deviation of the percent change of spiculation, lobulation, and sphericity are [Formula: see text], [Formula: see text], and [Formula: see text] %, respectively.

CONCLUSION

The prior works of computation of spiculation, lobulation, and sphericity require a set of four ground truths from radiologists and, hence, can not be used in practice. The proposed methods do not require ground truth information of nodules from radiologists, and hence, it can be used in real-life computer-aided diagnosis system for lung cancer.

Texture based feature extraction methods for content based medical image retrieval systems

The developments of content based image retrieval (CBIR) systems used for image archiving are continued and one of the important research topics. Although some studies have been presented general image achieving, proposed CBIR systems for archiving of medical images are not very efficient. In presented study, it is examined the retrieval efficiency rate of spatial methods used for feature extraction for medical image retrieval systems. The investigated algorithms in this study depend on gray level co-occurrence matrix (GLCM), gray level run length matrix (GLRLM), and Gabor wavelet accepted as spatial methods. In the experiments, the database is built including hundreds of medical images such as brain, lung, sinus, and bone. The results obtained in this study shows that queries based on statistics obtained from GLCM are satisfied. However, it is observed that Gabor Wavelet has been the most effective and accurate method.

Content-based image retrieval for brain MRI: an image-searching engine and population-based analysis to utilize past clinical data for future diagnosis

Radiological diagnosis is based on subjective judgment by radiologists. The reasoning behind this process is difficult to document and share, which is a major obstacle in adopting evidence-based medicine in radiology. We report our attempt to use a comprehensive brain parcellation tool to systematically capture image features and use them to record, search, and evaluate anatomical phenotypes. Anatomical images (T1-weighted MRI) were converted to a standardized index by using a high-dimensional image transformation method followed by atlas-based parcellation of the entire brain. We investigated how the indexed anatomical data captured the anatomical features of healthy controls and a population with Primary Progressive Aphasia (PPA). PPA was chosen because patients have apparent atrophy at different degrees and locations, thus the automated quantitative results can be compared with trained clinicians' qualitative evaluations. We explored and tested the power of individual classifications and of performing a search for images with similar anatomical features in a database using partial least squares-discriminant analysis (PLS-DA) and principal component analysis (PCA). The agreement between the automated z-score and the averaged visual scores for atrophy (r = 0.8) was virtually the same as the inter-evaluator agreement. The PCA plot distribution correlated with the anatomical phenotypes and the PLS-DA resulted in a model with an accuracy of 88% for distinguishing PPA variants. The quantitative indices captured the main anatomical features. The indexing of image data has a potential to be an effective, comprehensive, and easily translatable tool for clinical practice, providing new opportunities to mine clinical databases for medical decision support.

•

Brain parcellation tools define structures automatically and convert images into standardized and quantitative matrices.

•

We tested if an automated tool and the resultant vector of structural volumes can accurately capture anatomical phenotypes.

•

The agreement between visual and automated atrophy detection was virtually the same as the inter-evaluator agreement.

•

The quantitative indices captured the main anatomical features in brains with atrophy in different degrees and location.

•

The image quantification has potential to be an effective, comprehensive, and easily translatable tool for clinical practice.

A graph-based approach for the retrieval of multi-modality medical images

In this paper, we address the retrieval of multi-modality medical volumes, which consist of two different imaging modalities, acquired sequentially, from the same scanner. One such example, positron emission tomography and computed tomography (PET-CT), provides physicians with complementary functional and anatomical features as well as spatial relationships and has led to improved cancer diagnosis, localisation, and staging. The challenge of multi-modality volume retrieval for cancer patients lies in representing the complementary geometric and topologic attributes between tumours and organs. These attributes and relationships, which are used for tumour staging and classification, can be formulated as a graph. It has been demonstrated that graph-based methods have high accuracy for retrieval by spatial similarity. However, naïvely representing all relationships on a complete graph obscures the structure of the tumour-anatomy relationships. We propose a new graph structure derived from complete graphs that structurally constrains the edges connected to tumour vertices based upon the spatial proximity of tumours and organs. This enables retrieval on the basis of tumour localisation. We also present a similarity matching algorithm that accounts for different feature sets for graph elements from different imaging modalities. Our method emphasises the relationships between a tumour and related organs, while still modelling patient-specific anatomical variations. Constraining tumours to related anatomical structures improves the discrimination potential of graphs, making it easier to retrieve similar images based on tumour location. We evaluated our retrieval methodology on a dataset of clinical PET-CT volumes. Our results showed that our method enabled the retrieval of multi-modality images using spatial features. Our graph-based retrieval algorithm achieved a higher precision than several other retrieval techniques: gray-level histograms as well as state-of-the-art methods such as visual words using the scale- invariant feature transform (SIFT) and relational matrices representing the spatial arrangements of objects.

Software for automated classification of probe-based confocal laser endomicroscopy videos of colorectal polyps

AIM

To support probe-based confocal laser endomicroscopy (pCLE) diagnosis by designing software for the automated classification of colonic polyps.

METHODS

Intravenous fluorescein pCLE imaging of colorectal lesions was performed on patients undergoing screening and surveillance colonoscopies, followed by polypectomies. All resected specimens were reviewed by a reference gastrointestinal pathologist blinded to pCLE information. Histopathology was used as the criterion standard for the differentiation between neoplastic and non-neoplastic lesions. The pCLE video sequences, recorded for each polyp, were analyzed off-line by 2 expert endoscopists who were blinded to the endoscopic characteristics and histopathology. These pCLE videos, along with their histopathology diagnosis, were used to train the automated classification software which is a content-based image retrieval technique followed by k-nearest neighbor classification. The performance of the off-line diagnosis of pCLE videos established by the 2 expert endoscopists was compared with that of automated pCLE software classification. All evaluations were performed using leave-one-patient-out cross-validation to avoid bias.

RESULTS

Colorectal lesions (135) were imaged in 71 patients. Based on histopathology, 93 of these 135 lesions were neoplastic and 42 were non-neoplastic. The study found no statistical significance for the difference between the performance of automated pCLE software classification (accuracy 89.6%, sensitivity 92.5%, specificity 83.3%, using leave-one-patient-out cross-validation) and the performance of the off-line diagnosis of pCLE videos established by the 2 expert endoscopists (accuracy 89.6%, sensitivity 91.4%, specificity 85.7%). There was very low power (< 6%) to detect the observed differences. The 95% confidence intervals for equivalence testing were: -0.073 to 0.073 for accuracy, -0.068 to 0.089 for sensitivity and -0.18 to 0.13 for specificity. The classification software proposed in this study is not a "black box" but an informative tool based on the query by example model that produces, as intermediate results, visually similar annotated videos that are directly interpretable by the endoscopist.

CONCLUSION

The proposed software for automated classification of pCLE videos of colonic polyps achieves high performance, comparable to that of off-line diagnosis of pCLE videos established by expert endoscopists.

Prototypes for content-based image retrieval in clinical practice

Content-based image retrieval (CBIR) has been proposed as key technology for computer-aided diagnostics (CAD). This paper reviews the state of the art and future challenges in CBIR for CAD applied to clinical practice.We define applicability to clinical practice by having recently demonstrated the CBIR system on one of the CAD demonstration workshops held at international conferences, such as SPIE Medical Imaging, CARS, SIIM, RSNA, and IEEE ISBI. From 2009 to 2011, the programs of CADdemo@CARS and the CAD Demonstration Workshop at SPIE Medical Imaging were sought for the key word "retrieval" in the title. The systems identified were analyzed and compared according to the hierarchy of gaps for CBIR systems.In total, 70 software demonstrations were analyzed. 5 systems were identified meeting the criterions. The fields of application are (i) bone age assessment, (ii) bone fractures, (iii) interstitial lung diseases, and (iv) mammography. Bridging the particular gaps of semantics, feature extraction, feature structure, and evaluation have been addressed most frequently.In specific application domains, CBIR technology is available for clinical practice. While system development has mainly focused on bridging content and feature gaps, performance and usability have become increasingly important. The evaluation must be based on a larger set of reference data, and workflow integration must be achieved before CBIR-CAD is really established in clinical practice.

Content-based image retrieval applied to BI-RADS tissue classification in screening mammography

AIM

To present a content-based image retrieval (CBIR) system that supports the classification of breast tissue density and can be used in the processing chain to adapt parameters for lesion segmentation and classification.

METHODS

Breast density is characterized by image texture using singular value decomposition (SVD) and histograms. Pattern similarity is computed by a support vector machine (SVM) to separate the four BI-RADS tissue categories. The crucial number of remaining singular values is varied (SVD), and linear, radial, and polynomial kernels are investigated (SVM). The system is supported by a large reference database for training and evaluation. Experiments are based on 5-fold cross validation.

RESULTS

Adopted from DDSM, MIAS, LLNL, and RWTH datasets, the reference database is composed of over 10 000 various mammograms with unified and reliable ground truth. An average precision of 82.14% is obtained using 25 singular values (SVD), polynomial kernel and the one-against-one (SVM).

CONCLUSION

Breast density characterization using SVD allied with SVM for image retrieval enable the development of a CBIR system that can effectively aid radiologists in their diagnosis.

CBMIR: Content-based Image Retrieval Algorithm for Medical Image Databases

We propose a novel algorithm for the retrieval of images from medical image databases by content. The aim of this article is to present a content-based retrieval algorithm that is robust to scaling, with translation of objects within an image. For the best result and efficient representation and retrieval of medical images, attention is focused on the methodology, and the content of medical images is represented by the regions and relationships between such objects or regions of the Image Attributes (IA) of the objects. The CBMIR employs a new model in which each image is first decomposed into regions. The similarity measurement between images is developed based on a scheme that integrates the properties of all the regions in the images using regional matching. The method can answer queries by example. The efficiency and performance of the presented method has been evaluated using a dataset of about 5,000 simulated, but realistic computed tomography and magnetic resonance images, from which the original images are selected from three large medical image databases. The results of our experiments show more than a 93 percent success rate, which is satisfactory.