Automated Pulmonary Tuberculosis Severity Assessment on Chest X-rays

According to the 2022 World Health Organization's Global Tuberculosis (TB) report, an estimated 10.6 million people fell ill with TB, and 1.6 million died from the disease in 2021. In addition, 2021 saw a reversal of a decades-long trend of declining TB infections and deaths, with an estimated increase of 4.5% in the number of people who fell ill with TB compared to 2020, and an estimated yearly increase of 450,000 cases of drug resistant TB. Estimating the severity of pulmonary TB using frontal chest X-rays (CXR) can enable better resource allocation in resource constrained settings and monitoring of treatment response, enabling prompt treatment modifications if disease severity does not decrease over time. The Timika score is a clinically used TB severity score based on a CXR reading. This work proposes and evaluates three deep learning-based approaches for predicting the Timika score with varying levels of explainability. The first approach uses two deep learning-based models, one to explicitly detect lesion regions using YOLOV5n and another to predict the presence of cavitation using DenseNet121, which are then utilized in score calculation. The second approach uses a DenseNet121-based regression model to directly predict the affected lung percentage and another to predict cavitation presence using a DenseNet121-based classification model. Finally, the third approach directly predicts the Timika score using a DenseNet121-based regression model. The best performance is achieved by the second approach with a mean absolute error of 13-14% and a Pearson correlation of 0.7-0.84 using three held-out datasets for evaluating generalization.

Multi-omic profiling of follicular lymphoma reveals changes in tissue architecture and enhanced stromal remodeling in high-risk patients

Follicular lymphoma (FL) is a generally incurable malignancy that evolves from developmentally blocked germinal center (GC) B cells. To promote survival and immune escape, tumor B cells undergo significant genetic changes and extensively remodel the lymphoid microenvironment. Dynamic interactions between tumor B cells and the tumor microenvironment (TME) are hypothesized to contribute to the broad spectrum of clinical behaviors observed among FL patients. Despite the urgent need, existing clinical tools do not reliably predict disease behavior. Using a multi-modal strategy, we examined cell-intrinsic and -extrinsic factors governing progression and therapeutic outcomes in FL patients enrolled onto a prospective clinical trial. By leveraging the strengths of each platform, we identify several tumor-specific features and microenvironmental patterns enriched in individuals who experience early relapse, the most high-risk FL patients. These features include stromal desmoplasia and changes to the follicular growth pattern present 20 months before first progression and first relapse.

Harmonizing the Generation and Pre-publication Stewardship of FAIR Image data

Together with the molecular knowledge of genes and proteins, biological images promise to significantly enhance the scientific understanding of complex cellular systems and to advance predictive and personalized therapeutic products for human health. For this potential to be realized, quality-assured image data must be shared among labs at a global scale to be compared, pooled, and reanalyzed, thus unleashing untold potential beyond the original purpose for which the data was generated. There are two broad sets of requirements to enable image data sharing in the life sciences. One set of requirements is articulated in the companion White Paper entitled "Enabling Global Image Data Sharing in the Life Sciences," which is published in parallel and addresses the need to build the cyberinfrastructure for sharing the digital array data (arXiv:2401.13023 [q-bio.OT], https://doi.org/10.48550/arXiv.2401.13023). In this White Paper, we detail a broad set of requirements, which involves collecting, managing, presenting, and propagating contextual information essential to assess the quality, understand the content, interpret the scientific implications, and reuse image data in the context of the experimental details. We start by providing an overview of the main lessons learned to date through international community activities, which have recently made considerable progress toward generating community standard practices for imaging Quality Control (QC) and metadata. We then provide a clear set of recommendations for amplifying this work. The driving goal is to address remaining challenges, and democratize access to common practices and tools for a spectrum of biomedical researchers, regardless of their expertise, access to resources, and geographical location.

Tuberculosis Chest X-Ray Image Retrieval System Using Deep Learning Based Biomarker Predictions

The world health organization's global tuberculosis (TB) report for 2022 identifies TB, with an estimated 1.6 million, as a leading cause of death. The number of new cases has risen since 2020, particularly the number of new drug-resistant cases, estimated at 450,000 in 2021. This is concerning, as treatment of patients with drug resistant TB is complex and may not always be successful. The NIAID TB Portals program is an international consortium with a primary focus on patient centric data collection and analysis for drug resistant TB. The data includes images, their associated radiological findings, clinical records, and socioeconomic information. This work describes a TB Portals' Chest X-ray based image retrieval system which enables precision medicine. An input image is used to retrieve similar images and the associated patient specific information, thus facilitating inspection of outcomes and treatment regimens from comparable patients. Image similarity is defined using clinically relevant biomarkers: gender, age, body mass index (BMI), and the percentage of lung affected per sextant. The biomarkers are predicted using variations of the DenseNet169 convolutional neural network. A multi-task approach is used to predict gender, age and BMI incorporating transfer learning from an initial training on the NIH Clinical Center CXR dataset to the TB portals dataset. The resulting gender AUC, age and BMI mean absolute errors were 0.9854, 4.03years and 1.67 k g m 2 . For the percentage of sextant affected by lesions the mean absolute errors ranged between 7% to 12% with higher error values in the middle and upper sextants which exhibit more variability than the lower sextants. The retrieval system is currently available from https://rap.tbportals.niaid.nih.gov/find_similar_cxr.

Combining Radiological and Genomic TB Portals Data for Drug Resistance Analysis

Tuberculosis (TB) drug resistance is a worldwide public health problem. It decreases the likelihood of a positive outcome for the individual patient and increases the likelihood of disease spread. Therefore, early detection of TB drug resistance is crucial for improving outcomes and controlling disease transmission. While drug-sensitive tuberculosis cases are declining worldwide because of effective treatment, the threat of drug-resistant tuberculosis is growing, and the success rate of drug-resistant tuberculosis treatment is only around 60%. The TB Portals program provides a publicly accessible repository of TB case data with an emphasis on collecting drug-resistant cases. The dataset includes multi-modal information such as socioeconomic/geographic data, clinical characteristics, pathogen genomics, and radiological features. The program is an international collaboration whose participants are typically under a substantial burden of drug-resistant tuberculosis, with data collected from standard clinical care provided to the patients. Consequentially, the TB Portals dataset is heterogenous in nature, with data representing multiple treatment centers in different countries and containing cross-domain information. This study presents the challenges and methods used to address them when working with this real-world dataset. Our goal was to evaluate whether combining radiological features derived from a chest X-ray of the host and genomic features from the pathogen can potentially improve the identification of the drug susceptibility type, drug-sensitive (DS-TB) or drug-resistant (DR-TB), and the length of the first successful drug regimen. To perform these studies, significantly imbalanced data needed to be processed, which included a much larger number of DR-TB cases than DS-TB, many more cases with radiological findings than genomic ones, and the sparse high dimensional nature of the genomic information. Three evaluation studies were carried out. First, the DR-TB/DS-TB classification model achieved an average accuracy of 92.4% when using genomic features alone or when combining radiological and genomic features. Second, the regression model for the length of the first successful treatment had a relative error of 53.5% using radiological features, 25.6% using genomic features, and 22.0% using both radiological and genomic features. Finally, the relative error of the third regression model predicting the length of the first treatment using the most common drug combination varied depending on the feature type used. When using radiological features alone, the relative error was 17.8%. For genomic features alone, the relative error increased to 19.9%. The model had a relative error of 19.0% when both radiological and genomic features were combined. Although combining radiological and genomic features did not improve upon the use of genomic features when classifying DR-TB/DS-TB, the combination of the two feature types improved the relative error of the predictive model for the length of the first successful treatment. Furthermore, the regression model trained on radiological features achieved the best performance when predicting the treatment length of the most common drug combination.

3D-IBEX: Achieving multiplex 3-dimensional imaging for deep phenotyping of cells in tissues

2D multiplexed antibody-based imaging provides a framework to study cell-cell interactions in thin tissue sections but lacks the ability to interrogate spatial relationships in larger 3D anatomical structures. Several methods for 3D imaging of optically cleared tissue exist, but current approaches do not provide the marker depth afforded by high content 2D imaging. To overcome this constraint, we combined a fast hydrophilic tissue clearing technique, clearing-enhanced 3D (Ce3D), with the recently developed Iterative Bleaching Extends Multiplexity (IBEX) technique, and created 3D-IBEX. 3D-IBEX offers a high-resolution multiplex 3D imaging method. It has been used to analyze murine lung, lymph nodes, retina, and cornea as well as human retina and jejunum, yielding seamless views of large-scale tissue structures while obtaining single-cell resolution of multiple markers, permitting the identification of discrete cell subsets and structures. To date, up to 15 markers have been visualized in a single sample using 3 iterative staining and imaging cycles. This represents a major advance in the emerging field of multiplexed high-resolution 3D imaging but requires further refinements to address issues such as slow antibody penetration, the need for new computational algorithms to process large datasets, and implementation of methods for optimal registration of iterative images. When fully realized, 3D-IBEX will open new avenues for characterization of rare cell subsets missed in thin tissue sections, visualization of structures like nerves and vessels whose trajectories in a tissue are disrupted by sectioning, and delineation of mesoscale tissue domains that contribute to homeostasis and are often the substrates for pathology.

Intramural NIAID Research Opportunities (INRO) program.

Generalization Challenges in Drug-Resistant Tuberculosis Detection from Chest X-rays

Classification of drug-resistant tuberculosis (DR-TB) and drug-sensitive tuberculosis (DS-TB) from chest radiographs remains an open problem. Our previous cross validation performance on publicly available chest X-ray (CXR) data combined with image augmentation, the addition of synthetically generated and publicly available images achieved a performance of 85% AUC with a deep convolutional neural network (CNN). However, when we evaluated the CNN model trained to classify DR-TB and DS-TB on unseen data, significant performance degradation was observed (65% AUC). Hence, in this paper, we investigate the generalizability of our models on images from a held out country's dataset. We explore the extent of the problem and the possible reasons behind the lack of good generalization. A comparison of radiologist-annotated lesion locations in the lung and the trained model's localization of areas of interest, using GradCAM, did not show much overlap. Using the same network architecture, a multi-country classifier was able to identify the country of origin of the X-ray with high accuracy (86%), suggesting that image acquisition differences and the distribution of non-pathological and non-anatomical aspects of the images are affecting the generalization and localization of the drug resistance classification model as well. When CXR images were severely corrupted, the performance on the validation set was still better than 60% AUC. The model overfitted to the data from countries in the cross validation set but did not generalize to the held out country. Finally, we applied a multi-task based approach that uses prior TB lesions location information to guide the classifier network to focus its attention on improving the generalization performance on the held out set from another country to 68% AUC.

IBEX: an iterative immunolabeling and chemical bleaching method for high-content imaging of diverse tissues

High-content imaging is needed to catalog the variety of cellular phenotypes and multicellular ecosystems present in metazoan tissues. We recently developed iterative bleaching extends multiplexity (IBEX), an iterative immunolabeling and chemical bleaching method that enables multiplexed imaging (>65 parameters) in diverse tissues, including human organs relevant for international consortia efforts. IBEX is compatible with >250 commercially available antibodies and 16 unique fluorophores, and can be easily adopted to different imaging platforms using slides and nonproprietary imaging chambers. The overall protocol consists of iterative cycles of antibody labeling, imaging and chemical bleaching that can be completed at relatively low cost in 2-5 d by biologists with basic laboratory skills. To support widespread adoption, we provide extensive details on tissue processing, curated lists of validated antibodies and tissue-specific panels for multiplex imaging. Furthermore, instructions are included on how to automate the method using competitively priced instruments and reagents. Finally, we present a software solution for image alignment that can be executed by individuals without programming experience using open-source software and freeware. In summary, IBEX is a noncommercial method that can be readily implemented by academic laboratories and scaled to achieve high-content mapping of diverse tissues in support of a Human Reference Atlas or other such applications.

Differentiating between drug-sensitive and drug-resistant tuberculosis with machine learning for clinical and radiological features

Background

Tuberculosis (TB) drug resistance is a worldwide public health problem that threatens progress made in TB care and control. Early detection of drug resistance is important for disease control, with discrimination between drug-resistant TB (DR-TB) and drug-sensitive TB (DS-TB) still being an open problem. The objective of this work is to investigate the relevance of readily available clinical data and data derived from chest X-rays (CXRs) in DR-TB prediction and to investigate the possibility of applying machine learning techniques to selected clinical and radiological features for discrimination between DR-TB and DS-TB. We hypothesize that the number of sextants affected by abnormalities such as nodule, cavity, collapse and infiltrate may serve as a radiological feature for DR-TB identification, and that both clinical and radiological features are important factors for machine classification of DR-TB and DS-TB.

Methods

We use data from the NIAID TB Portals program (https://tbportals.niaid.nih.gov), 1,455 DR-TB cases and 782 DS-TB cases from 11 countries. We first select three clinical features and 26 radiological features from the dataset. Then, we perform Pearson's chi-squared test to analyze the significance of the selected clinical and radiological features. Finally, we train machine classifiers based on different features and evaluate their ability to differentiate between DR-TB and DS-TB.

Results

Pearson's chi-squared test shows that two clinical features and 23 radiological features are statistically significant regarding DR-TB vs. DS-TB. A ten-fold cross-validation using a support vector machine shows that automatic discrimination between DR-TB and DS-TB achieves an average accuracy of 72.34% and an average AUC value of 78.42%, when combing all 25 statistically significant features.

Conclusions

Our study suggests that the number of affected lung sextants can be used for predicting DR-TB, and that automatic discrimination between DR-TB and DS-TB is possible, with a combination of clinical features and radiological features providing the best performance.

Identifying Drug-Resistant Tuberculosis in Chest Radiographs: Evaluation of CNN Architectures and Training Strategies

Tuberculosis (TB) is a serious infectious disease that mainly affects the lungs. Drug resistance to the disease makes it more challenging to control. Early diagnosis of drug resistance can help with decision making resulting in appropriate and successful treatment. Chest X-rays (CXRs) have been pivotal to identifying tuberculosis and are widely available. In this work, we utilize CXRs to distinguish between drug-resistant and drug-sensitive tuberculosis. We incorporate Convolutional Neural Network (CNN) based models to discriminate the two types of TB, and employ standard and deep learning based data augmentation methods to improve the classification. Using labeled data from NIAID TB Portals and additional non-labeled sources, we were able to achieve an Area Under the ROC Curve (AUC) of up to 85% using a pretrained InceptionV3 network.

DicomAnnotator: a Configurable Open-Source Software Program for Efficient DICOM Image Annotation

Modern, supervised machine learning approaches to medical image classification, image segmentation, and object detection usually require many annotated images. As manual annotation is usually labor-intensive and time-consuming, a well-designed software program can aid and expedite the annotation process. Ideally, this program should be configurable for various annotation tasks, enable efficient placement of several types of annotations on an image or a region of an image, attribute annotations to individual annotators, and be able to display Digital Imaging and Communications in Medicine (DICOM)-formatted images. No current open-source software program fulfills these requirements. To fill this gap, we developed DicomAnnotator, a configurable open-source software program for DICOM image annotation. This program fulfills the above requirements and provides user-friendly features to aid the annotation process. In this paper, we present the design and implementation of DicomAnnotator. Using spine image annotation as a test case, our evaluation showed that annotators with various backgrounds can use DicomAnnotator to annotate DICOM images efficiently. DicomAnnotator is freely available at https://github.com/UW-CLEAR-Center/DICOM-Annotator under the GPLv3 license.

Dual-view light-sheet imaging through a tilted glass interface using a deformable mirror

Light-sheet microscopy has become indispensable for imaging developing organisms, and imaging from multiple directions (views) is essential to improve its spatial resolution. We combine multi-view light-sheet microscopy with microfluidics using adaptive optics (deformable mirror) which corrects aberrations introduced by the 45^o-tilted glass coverslip. The optimal shape of the deformable mirror is computed by an iterative algorithm that optimizes the point-spread function in two orthogonal views. Simultaneous correction in two optical arms is achieved via a knife-edge mirror that splits the excitation path and combines the detection paths. Our design allows multi-view light-sheet microscopy with microfluidic devices for precisely controlled experiments and high-content screening.

PD-1 blockade exacerbates Mycobacterium tuberculosis infection in rhesus macaques

Boosting immune cell function by targeting the coinhibitory receptor PD-1 may have applications in the treatment of chronic infections. Here, we examine the role of PD-1 during Mycobacterium tuberculosis (Mtb) infection of rhesus macaques. Animals treated with anti-PD-1 monoclonal antibody developed worse disease and higher granuloma bacterial loads compared with isotype control-treated monkeys. PD-1 blockade increased the number and functionality of granuloma Mtb-specific CD8 T cells. In contrast, Mtb-specific CD4 T cells in anti-PD-1-treated macaques were not increased in number or function in granulomas, expressed increased levels of CTLA-4, and exhibited reduced intralesional trafficking in live imaging studies. In granulomas of anti-PD-1-treated animals, multiple proinflammatory cytokines were elevated, and more cytokines correlated with bacterial loads, leading to the identification of a role for caspase 1 in the exacerbation of tuberculosis after PD-1 blockade. Last, increased Mtb bacterial loads after PD-1 blockade were found to associate with the composition of the intestinal microbiota before infection in individual macaques. Therefore, PD-1-mediated coinhibition is required for control of Mtb infection in macaques, perhaps because of its role in dampening detrimental inflammation and allowing for normal CD4 T cell responses.

A distance map regularized CNN for cardiac cine MR image segmentation

PURPOSE

Cardiac image segmentation is a critical process for generating personalized models of the heart and for quantifying cardiac performance parameters. Fully automatic segmentation of the left ventricle (LV), the right ventricle (RV), and the myocardium from cardiac cine MR images is challenging due to variability of the normal and abnormal anatomy, as well as the imaging protocols. This study proposes a multi-task learning (MTL)-based regularization of a convolutional neural network (CNN) to obtain accurate segmenation of the cardiac structures from cine MR images.

METHODS

We train a CNN network to perform the main task of semantic segmentation, along with the simultaneous, auxiliary task of pixel-wise distance map regression. The network also predicts uncertainties associated with both tasks, such that their losses are weighted by the inverse of their corresponding uncertainties. As a result, during training, the task featuring a higher uncertainty is weighted less and vice versa. The proposed distance map regularizer is a decoder network added to the bottleneck layer of an existing CNN architecture, facilitating the network to learn robust global features. The regularizer block is removed after training, so that the original number of network parameters does not change. The trained network outputs per-pixel segmentation when a new patient cine MR image is provided as an input.

RESULTS

We show that the proposed regularization method improves both binary and multi-class segmentation performance over the corresponding state-of-the-art CNN architectures. The evaluation was conducted on two publicly available cardiac cine MRI datasets, yielding average Dice coefficients of 0.84 ± 0.03 and 0.91 ± 0.04. We also demonstrate improved generalization performance of the distance map regularized network on cross-dataset segmentation, showing as much as 42% improvement in myocardium Dice coefficient from 0.56 ± 0.28 to 0.80 ± 0.14.

CONCLUSIONS

We have presented a method for accurate segmentation of cardiac structures from cine MR images. Our experiments verify that the proposed method exceeds the segmentation performance of three existing state-of-the-art methods. Furthermore, several cardiac indices that often serve as diagnostic biomarkers, specifically blood pool volume, myocardial mass, and ejection fraction, computed using our method are better correlated with the indices computed from the reference, ground truth segmentation. Hence, the proposed method has the potential to become a non-invasive screening and diagnostic tool for the clinical assessment of various cardiac conditions, as well as a reliable aid for generating patient specific models of the cardiac anatomy for therapy planning, simulation, and guidance.

Correction to: SimpleITK Image-Analysis Notebooks: a Collaborative Environment for Education and Reproducible Research

This paper had published originally without open access, but has since been republished with open access.

SimpleITK Image-Analysis Notebooks: a Collaborative Environment for Education and Reproducible Research

Modern scientific endeavors increasingly require team collaborations to construct and interpret complex computational workflows. This work describes an image-analysis environment that supports the use of computational tools that facilitate reproducible research and support scientists with varying levels of software development skills. The Jupyter notebook web application is the basis of an environment that enables flexible, well-documented, and reproducible workflows via literate programming. Image-analysis software development is made accessible to scientists with varying levels of programming experience via the use of the SimpleITK toolkit, a simplified interface to the Insight Segmentation and Registration Toolkit. Additional features of the development environment include user friendly data sharing using online data repositories and a testing framework that facilitates code maintenance. SimpleITK provides a large number of examples illustrating educational and research-oriented image analysis workflows for free download from GitHub under an Apache 2.0 license: github.com/InsightSoftwareConsortium/SimpleITK-Notebooks.

Left Ventricle Segmentation and Quantification from Cardiac Cine MR Images via Multi-task Learning

Segmentation of the left ventricle and quantification of various cardiac contractile functions is crucial for the timely diagnosis and treatment of cardiovascular diseases. Traditionally, the two tasks have been tackled independently. Here we propose a convolutional neural network based multi-task learning approach to perform both tasks simultaneously, such that, the network learns better representation of the data with improved generalization performance. Probabilistic formulation of the problem enables learning the task uncertainties during the training, which are used to automatically compute the weights for the tasks. We performed a five fold cross-validation of the myocardium segmentation obtained from the proposed multi-task network on 97 patient 4-dimensional cardiac cine-MRI datasets available through the STA-COM LV segmentation challenge against the provided gold-standard myocardium segmentation, obtaining a Dice overlap of 0.849 ± 0.036 and mean surface distance of 0.274 ± 0.083 mm, while simultaneously estimating the myocardial area with mean absolute difference error of 205 ± 198 mm2.

Image Segmentation, Registration and Characterization in R with SimpleITK

Many types of medical and scientific experiments acquire raw data in the form of images. Various forms of image processing and image analysis are used to transform the raw image data into quantitative measures that are the basis of subsequent statistical analysis. In this article we describe the SimpleITK R package. SimpleITK is a simplified interface to the insight segmentation and registration toolkit (ITK). ITK is an open source C++ toolkit that has been actively developed over the past 18 years and is widely used by the medical image analysis community. SimpleITK provides packages for many interpreter environments, including R. Currently, it includes several hundred classes for image analysis including a wide range of image input and output, filtering operations, and higher level components for segmentation and registration. Using SimpleITK, development of complex combinations of image and statistical analysis procedures is feasible. This article includes several examples of computational image analysis tasks implemented using SimpleITK, including spherical marker localization, multi-modal image registration, segmentation evaluation, and cell image analysis.

Cine Cardiac MRI Slice Misalignment Correction Towards Full 3D Left Ventricle Segmentation

Accurate segmentation of the left ventricle (LV) blood-pool and myocardium is required to compute cardiac function assessment parameters or generate personalized cardiac models for pre-operative planning of minimally invasive therapy. Cardiac Cine Magnetic Resonance Imaging (MRI) is the preferred modality for high resolution cardiac imaging thanks to its capability of imaging the heart throughout the cardiac cycle, while providing tissue contrast superior to other imaging modalities without ionizing radiation. However, there exists an inevitable misalignment between the slices in cine MRI due to the 2D + time acquisition, rendering 3D segmentation methods ineffective. A large part of published work on cardiac MR image segmentation focuses on 2D segmentation methods that yield good results in mid-slices, however with less accurate results for the apical and basal slices. Here, we propose an algorithm to correct for the slice misalignment using a Convolutional Neural Network (CNN)-based regression method, and then perform a 3D graph-cut based segmentation of the LV using atlas shape prior. Our algorithm is able to reduce the median slice misalignment error from 3.13 to 2.07 pixels, and obtain the blood-pool segmentation with an accuracy characterized by a 0.904 mean dice overlap and 0.56 mm mean surface distance with respect to the gold-standard blood-pool segmentation for 9 test cine MR datasets.

The National Library of Medicine Pill Image Recognition Challenge: An Initial Report

In January 2016 the U.S. National Library of Medicine announced a challenge competition calling for the development and discovery of high-quality algorithms and software that rank how well consumer images of prescription pills match reference images of pills in its authoritative RxIMAGE collection. This challenge was motivated by the need to easily identify unknown prescription pills both by healthcare personnel and the general public. Potential benefits of this capability include confirmation of the pill in settings where the documentation and medication have been separated, such as in a disaster or emergency; and confirmation of a pill when the prescribed medication changes from brand to generic, or for any other reason the shape and color of the pill change. The data for the competition consisted of two types of images, high quality macro photographs, reference images, and consumer quality photographs of the quality we expect users of a proposed application to acquire. A training dataset consisting of 2000 reference images and 5000 corresponding consumer quality images acquired from 1000 pills was provided to challenge participants. A second dataset acquired from 1000 pills with similar distributions of shape and color was reserved as a segregated testing set. Challenge submissions were required to produce a ranking of the reference images, given a consumer quality image as input. Determination of the winning teams was done using the mean average precision quality metric, with the three winners obtaining mean average precision scores of 0.27, 0.09, and 0.08. In the retrieval results, the correct image was amongst the top five ranked images 43%, 12%, and 11% of the time, out of 5000 query/consumer images. This is an initial promising step towards development of an NLM software system and application-programming interface facilitating pill identification. The training dataset will continue to be freely available online at: http://pir.nlm.nih.gov/challenge/submission.html.

Interactive initialization of 2D/3D rigid registration

PURPOSE

Registration is one of the key technical components in an image-guided navigation system. A large number of 2D/3D registration algorithms have been previously proposed, but have not been able to transition into clinical practice. The authors identify the primary reason for the lack of adoption with the prerequisite for a sufficiently accurate initial transformation, mean target registration error of about 10 mm or less. In this paper, the authors present two interactive initialization approaches that provide the desired accuracy for x-ray/MR and x-ray/CT registration in the operating room setting.

METHODS

The authors have developed two interactive registration methods based on visual alignment of a preoperative image, MR, or CT to intraoperative x-rays. In the first approach, the operator uses a gesture based interface to align a volume rendering of the preoperative image to multiple x-rays. The second approach uses a tracked tool available as part of a navigation system. Preoperatively, a virtual replica of the tool is positioned next to the anatomical structures visible in the volumetric data. Intraoperatively, the physical tool is positioned in a similar manner and subsequently used to align a volume rendering to the x-ray images using an augmented reality (AR) approach. Both methods were assessed using three publicly available reference data sets for 2D/3D registration evaluation.

RESULTS

In the authors' experiments, the authors show that for x-ray/MR registration, the gesture based method resulted in a mean target registration error (mTRE) of 9.3 ± 5.0 mm with an average interaction time of 146.3 ± 73.0 s, and the AR-based method had mTREs of 7.2 ± 3.2 mm with interaction times of 44 ± 32 s. For x-ray/CT registration, the gesture based method resulted in a mTRE of 7.4 ± 5.0 mm with an average interaction time of 132.1 ± 66.4 s, and the AR-based method had mTREs of 8.3 ± 5.0 mm with interaction times of 58 ± 52 s.

CONCLUSIONS

Based on the authors' evaluation, the authors conclude that the registration approaches are sufficiently accurate for initializing 2D/3D registration in the OR setting, both when a tracking system is not in use (gesture based approach), and when a tracking system is already in use (AR based approach).

When change happens: computer assistance and image guidance for minimally invasive therapy

Computer-assisted interventions are medical procedures that rely on image guidance and computer-based systems to provide visualisation and navigation information to the clinician, when direct vision of the sites or targets to be treated is not available, during minimally invasive procedures. Recent advances in medical image acquisition and processing, accompanied by technological breakthroughs in image fusion, visualisation and display have accelerated the adoption of minimally invasive approaches for a variety of medical procedures. This Letter is intended to serve as a brief overview of available image guidance and computer-assisted technology in the context of popular minimally invasive applications, while outlining some of the limitations and challenges in the transition from laboratory to clinical care.

Treatment planning and image guidance for radiofrequency ablation of large tumors

This article addresses the two key challenges in computer-assisted percutaneous tumor ablation: planning multiple overlapping ablations for large tumors while avoiding critical structures, and executing the prescribed plan. Toward semiautomatic treatment planning for image-guided surgical interventions, we develop a systematic approach to the needle-based ablation placement task, ranging from preoperative planning algorithms to an intraoperative execution platform. The planning system incorporates clinical constraints on ablations and trajectories using a multiple objective optimization formulation, which consists of optimal path selection and ablation coverage optimization based on integer programming. The system implementation is presented and validated in both phantom and animal studies. The presented system can potentially be further extended for other ablation techniques such as cryotherapy.

Image-guided navigation: a cost effective practical introduction using the Image-Guided Surgery Toolkit (IGSTK)

Teaching the key technical aspects of image-guided interventions using a hands-on approach is a challenging task. This is primarily due to the high cost and lack of accessibility to imaging and tracking systems. We provide a software and data infrastructure which addresses both challenges. Our infrastructure allows students, patients, and clinicians to develop an understanding of the key technologies by using them, and possibly by developing additional components and integrating them into a simple navigation system which we provide. Our approach requires minimal hardware, LEGO blocks to construct a phantom for which we provide CT scans, and a webcam which when combined with our software provides the functionality of a tracking system. A premise of this approach is that tracking accuracy is sufficient for our purpose. We evaluate the accuracy provided by a consumer grade webcam and show that it is sufficient for educational use. We provide an open source implementation of all the components required for a basic image-guided navigation as part of the Image-Guided Surgery Toolkit (IGSTK). It has long been known that in education there is no substitute for hands-on experience, to quote Sophocles, "One must learn by doing the thing; for though you think you know it, you have no certainty, until you try.". Our work provides this missing capability in the context of image-guided navigation. Enabling a wide audience to learn and experience the use of a navigation system.

Agile Methods for Open Source Safety-Critical Software

The introduction of software technology in a life-dependent environment requires the development team to execute a process that ensures a high level of software reliability and correctness. Despite their popularity, agile methods are generally assumed to be inappropriate as a process family in these environments due to their lack of emphasis on documentation, traceability, and other formal techniques. Agile methods, notably Scrum, favor empirical process control, or small constant adjustments in a tight feedback loop. This paper challenges the assumption that agile methods are inappropriate for safety-critical software development. Agile methods are flexible enough to encourage the rightamount of ceremony; therefore if safety-critical systems require greater emphasis on activities like formal specification and requirements management, then an agile process will include these as necessary activities. Furthermore, agile methods focus more on continuous process management and code-level quality than classic software engineering process models. We present our experiences on the image-guided surgical toolkit (IGSTK) project as a backdrop. IGSTK is an open source software project employing agile practices since 2004. We started with the assumption that a lighter process is better, focused on evolving code, and only adding process elements as the need arose. IGSTK has been adopted by teaching hospitals and research labs, and used for clinical trials. Agile methods have matured since the academic community suggested they are not suitable for safety-critical systems almost a decade ago, we present our experiences as a case study for renewing the discussion.

A High Speed High Resolution Contact Line Imager Using Amorphous Silicon Alloy Pin Diodes

We report the development of a very high speed high resolution contact line imager using amorphous silicon alloy PIN diodes both as photosensing elements and as isolation diodes in the multiplexing scheme. High speed is achieved by reading the integrated photocurrent in 8μsec and using current integration times less than Imsec. For 200 dots/inch, the scan speed at an illumination of 5×10−4W/cm2 is over 1000 lines/sec. This allows the reading of an A4 (8½″×11″) page in less than 2.0 sec. At this light level, the signal to noise ratio is greater than 40dB. The photosensor array can be used in the true direct contact mode and the multiplexed addressing scheme gives a substantial reduction in the number of peripheral IC chips necessary for operation.

Evaluation of a 4D cone-beam CT reconstruction approach using a simulation framework

Current image-guided navigation systems for thoracic abdominal interventions utilize three dimensional (3D) images acquired at breath-hold. As a result they can only provide guidance at a specific point in the respiratory cycle. The intervention is thus performed in a gated manner, with the physician advancing only when the patient is at the same respiratory cycle in which the 3D image was acquired. To enable a more continuous workflow we propose to use 4D image data. We describe an approach to constructing a set of 4D images from a diagnostic CT acquired at breath-hold and a set of intraoperative cone-beam CT (CBCT) projection images acquired while the patient is freely breathing. Our approach is based on an initial reconstruction of a gated 4D CBCT data set. The 3D CBCT images for each respiratory phase are then non-rigidly registered to the diagnostic CT data. Finally the diagnostic CT is deformed based on the registration results, providing a 4D data set with sufficient quality for navigation purposes. In this work we evaluate the proposed reconstruction approach using a simulation framework. A 3D CBCT dataset of an anthropomorphic phantom is deformed using internal motion data acquired from an animal model to create a ground truth 4D CBCT image. Simulated projection images are then created from the 4D image and the known CBCT scan parameters. Finally, the original 3D CBCT and the simulated X-ray images are used as input to our reconstruction method. The resulting 4D data set is then compared to the known ground truth by normalized cross correlation(NCC). We show that the deformed diagnostic CTs are of better quality than the gated reconstructions with a mean NCC value of 0.94 versus a mean 0.81 for the reconstructions.

OpenIGTLink: an open network protocol for image-guided therapy environment

BACKGROUND

With increasing research on system integration for image-guided therapy (IGT), there has been a strong demand for standardized communication among devices and software to share data such as target positions, images and device status.

METHOD

We propose a new, open, simple and extensible network communication protocol for IGT, named OpenIGTLink, to transfer transform, image and status messages. We conducted performance tests and use-case evaluations in five clinical and engineering scenarios.

RESULTS

The protocol was able to transfer position data with submillisecond latency up to 1024 fps and images with latency of <10 ms at 32 fps. The use-case tests demonstrated that the protocol is feasible for integrating devices and software.

CONCLUSION

The protocol proved capable of handling data required in the IGT setting with sufficient time resolution and latency. The protocol not only improves the interoperability of devices and software but also promotes transitions of research prototypes to clinical applications.

Localizing spherical fiducials in C-arm based cone-beam CT

PURPOSE

C-arm based cone-beam CT (CBCT) imaging enables the in situ acquisition of three dimensional images. In the context of image-guided interventions, this technology potentially reduces the complexity of a procedure's workflow. Instead of acquiring the preoperative volumetric images in a separate location and transferring the patient to the interventional suite, both imaging and intervention are carried out in the same location. A key component in image-guided interventions is image to patient registration. The most common registration approach, in clinical use, is based on fiducial markers placed on the patient's skin which are then localized in the volumetric image and in the interventional environment. When using C-arm CBCT, this registration approach is challenging as in many cases the small size of the volumetric reconstruction cannot include both the skin fiducials and the organ of interest.

METHODS

In this article the author shows that fiducial localization outside the reconstructed volume is possible if the projection images from which the reconstruction was obtained are available. By replacing direct fiducial localization in the volumetric images with localization in the projection images, the author obtains the fiducial coordinates in the volume's coordinate system even when the fiducials are outside the reconstructed region.

RESULTS

The approach was evaluated using two types of spherical fiducials, clinically used 4 mm diameter markers and a custom phantom embedded with 6 mm diameter markers that is part of a commercial navigation system. In all cases, the method localized all fiducials, including those that were outside the reconstructed volume. The method's mean (std) localization error as evaluated using fiducials that were directly localized in the CBCT reconstruction was 0.55 (0.22) mm for the 4 mm markers and 0.51(0.18) mm for the 6 mm markers.

CONCLUSIONS

Based on the evaluations the author concludes that the proposed localization approach is sufficiently accurate to augment or replace direct volumetric fiducial localization for thoracic-abdominal interventions. This allows the physician to position fiducials in a more flexible manner, relaxing the requirement that both the organ of interest and skin surface be contained in the volumetric reconstruction.

Electromagnetic tracking in the clinical environment

When choosing an electromagnetic tracking system (EMTS) for image-guided procedures several factors must be taken into consideration. Among others these include the system's refresh rate, the number of sensors that need to be tracked, the size of the navigated region, the system interaction with the environment, whether the sensors can be embedded into the tools and provide the desired transformation data, and tracking accuracy and robustness. To date, the only factors that have been studied extensively are the accuracy and the susceptibility of EMTSs to distortions caused by ferromagnetic materials. In this paper the authors shift the focus from analysis of system accuracy and stability to the broader set of factors influencing the utility of EMTS in the clinical environment. The authors provide an analysis based on all of the factors specified above, as assessed in three clinical environments. They evaluate two commercial tracking systems, the Aurora system from Northern Digital Inc., and the 3D Guidance system with three different field generators from Ascension Technology Corp. The authors show that these systems are applicable to specific procedures and specific environments, but that currently, no single system configuration provides a comprehensive solution across procedures and environments.

Designing Tracking Software for Image-Guided Surgery Applications: IGSTK Experience

OBJECTIVE: Many image-guided surgery applications require tracking devices as part of their core functionality. The Image-Guided Surgery Toolkit (IGSTK) was designed and developed to interface tracking devices with software applications incorporating medical images. METHODS: IGSTK was designed as an open source C++ library that provides the basic components needed for fast prototyping and development of image-guided surgery applications. This library follows a component-based architecture with several components designed for specific sets of image-guided surgery functions. At the core of the toolkit is the tracker component that handles communication between a control computer and navigation device to gather pose measurements of surgical instruments present in the surgical scene. The representations of the tracked instruments are superimposed on anatomical images to provide visual feedback to the clinician during surgical procedures. RESULTS: The initial version of the IGSTK toolkit has been released in the public domain and several trackers are supported. The toolkit and related information are available at www.igstk.org. CONCLUSION: With the increased popularity of minimally invasive procedures in health care, several tracking devices have been developed for medical applications. Designing and implementing high-quality and safe software to handle these different types of trackers in a common framework is a challenging task. It requires establishing key software design principles that emphasize abstraction, extensibility, reusability, fault-tolerance, and portability. IGSTK is an open source library that satisfies these needs for the image-guided surgery community.

The image-guided surgery toolkit IGSTK: an open source C++ software toolkit

This paper presents an overview of the image-guided surgery toolkit (IGSTK). IGSTK is an open source C++ software library that provides the basic components needed to develop image-guided surgery applications. It is intended for fast prototyping and development of image-guided surgery applications. The toolkit was developed through a collaboration between academic and industry partners. Because IGSTK was designed for safety-critical applications, the development team has adopted lightweight software processes that emphasizes safety and robustness while, at the same time, supporting geographically separated developers. A software process that is philosophically similar to agile software methods was adopted emphasizing iterative, incremental, and test-driven development principles. The guiding principle in the architecture design of IGSTK is patient safety. The IGSTK team implemented a component-based architecture and used state machine software design methodologies to improve the reliability and safety of the components. Every IGSTK component has a well-defined set of features that are governed by state machines. The state machine ensures that the component is always in a valid state and that all state transitions are valid and meaningful. Realizing that the continued success and viability of an open source toolkit depends on a strong user community, the IGSTK team is following several key strategies to build an active user community. These include maintaining a users and developers’ mailing list, providing documentation (application programming interface reference document and book), presenting demonstration applications, and delivering tutorial sessions at relevant scientific conferences.

Precision targeting of liver lesions using a novel electromagnetic navigation device in physiologic phantom and swine

Radiofrequency ablation of primary and metastatic liver tumors is becoming a potential alternative to surgical resection. We propose a novel system that uses real-time electromagnetic position sensing of the needle tip to help with precision guidance into a liver tumor. The purpose of this study was to evaluate this technology in phantom and animal models. Using an electromagnetic navigation device, instrumented 18 g needles were advanced into radioopaque tumor targets in a respiratory liver phantom. The phantom featured a moving liver target that simulated cranio-caudal liver motion due to respiration. Skin-to-target path planning and real-time needle guidance were provided by a custom-designed software interface based on pre-operative 1 mm CT data slices. Needle probes were advanced using only the electromagnetic navigation device and software display. No conventional real-time imaging was used to assist in advancing the needle to the target. Two experienced operators (interventional radiologists) and two inexperienced ones (residents) used the system. The same protocol was then also used in two anesthetized 45 kg Yorkshire swine where radioopaque agar nodules were injected into the liver to serve as targets. A total of 76 tumor targeting attempts were performed in the liver phantom, and 32 attempts were done in the swine. The average time for path planning was 30 s in the phantom, and 63 s in the swine. The median time for the actual needle puncture to reach the desired target was 33 s in the phantom, and 42 s in the swine. The average registration error between the CT coordinate system and electromagnetic coordinate system was 1.4 mm (SD 0.3 mm) in the phantom, and 1.9 mm (SD 0.4 mm) in the swine. The median distance from the final needle tip position to the center of the tumor was 6.4 mm (SD 3.3 mm, n=76) in the phantom, and 8.3 mm (SD 3.7 mm, n=32) in the swine. There was no statistical difference in the planning time, procedure time, or accuracy of needle placement between experienced and inexperienced operators. The novel electromagnetic navigation system allows probe delivery into hepatic tumors of a physiologic phantom and live anesthetized swine. The system allows less experienced operators to perform equally well as experienced radiologists in terms of procedure time and accuracy of needle probe delivery.

Precise robot-assisted guide positioning for distal locking of intramedullary nails

This paper presents a novel image-guided robot-based system to assist orthopedic surgeons in performing distal locking of long bone intramedullary nails. The system consists of a bone-mounted miniature robot fitted with a drill guide that provides rigid mechanical guidance for hand-held drilling of the distal screws' pilot holes. The robot is automatically positioned so that the drill guide and nail distal locking axes coincide, using a single fluoroscopic X-ray image. Since the robot is rigidly attached to the intramedullary nail or bone, no leg immobilization or real-time tracking is required. We describe the system and protocol and present a method for accurate and robust drill guide and nail hole localization and registration. The in vitro system accuracy experiments for fronto-parallel viewing show a mean angular error of 1.3 degrees (std = 0.4 degrees ) between the computed drill guide axes and the actual locking holes axes, and a mean 3.0 mm error (std = 1.1 mm) in the entry and exit drill point, which is adequate for successfully locking the nail.

Long bone panoramas from fluoroscopic X-ray images

This paper presents a new method for creating a single panoramic image of a long bone from several individual fluoroscopic X-ray images. Panoramic images are useful preoperatively for diagnosis, and intraoperatively for long bone fragment alignment, for making anatomical measurements, and for documenting surgical outcomes. Our method composes individual overlapping images into an undistorted panoramic view that is the equivalent of a single X-ray image with a wide field of view. The correlations between the images are established from the graduations of a radiolucent ruler imaged alongside the long bone. Unlike existing methods, ours uses readily available hardware, requires a simple image acquisition protocol with minimal user input, and works with existing fluoroscopic C-arm units without modifications. It is robust and accurate, producing panoramas whose quality and spatial resolution is comparable to that of the individual images. The method has been successfully tested on in vitro and clinical cases.

Gradient-based 2-D/3-D rigid registration of fluoroscopic X-ray to CT

We present a gradient-based method for rigid registration of a patient preoperative computed tomography (CT) to its intraoperative situation with a few fluoroscopic X-ray images obtained with a tracked C-arm. The method is noninvasive, anatomy-based, requires simple user interaction, and includes validation. It is generic and easily customizable for a variety of routine clinical uses in orthopaedic surgery. Gradient-based registration consists of three steps: 1) initial pose estimation; 2) coarse geometry-based registration on bone contours, and; 3) fine gradient projection registration (GPR) on edge pixels. It optimizes speed, accuracy, and robustness. Its novelty resides in using volume gradients to eliminate outliers and foreign objects in the fluoroscopic X-ray images, in speeding up computation, and in achieving higher accuracy. It overcomes the drawbacks of intensity-based methods, which are slow and have a limited convergence range, and of geometry-based methods, which depend on the image segmentation quality. Our simulated, in vitro, and cadaver experiments on a human pelvis CT, dry vertebra, dry femur, fresh lamb hip, and human pelvis under realistic conditions show a mean 0.5-1.7 mm (0.5-2.6 mm maximum) target registration accuracy.

Comparative in vitro study of contact- and image-based rigid registration for computer-aided surgery

We present an in vitro study of rigid registration methods for computer-aided surgery. The goals of the study were to obtain accuracy measures empirically under optimal laboratory conditions, and to identify the weak links in the registration chain. Specifically, we investigated two common registration methods (contact-based registration and image-based landmark registration) and established a framework for comparing the accuracy of both methods. The phantoms, protocols, and algorithms for tool tip calibration, contact-based registration with an optical tracker, fluoroscopic X-ray camera calibration, and fluoroscopic X-ray image-based landmark registration are described. Average accuracies of 0.5 mm (1.5 mm maximum) and 2.75 mm (3.4 mm maximum) were found for contact-based and image-based landmark registration, respectively. Based on these findings, the camera calibration was identified as being the main source of error in image-based landmark registration. Protocol improvements and algorithmic refinements to improve the accuracy of image-based landmark registration are proposed.

Ethnobotanical survey in the Palestinian area: a classification of the healing potential of medicinal plants

An ethnobotanical survey was carried out in the West Bank to evaluate the relative efficacy of the plants used to treat skin diseases and prostate cancer. A total number of 102 informants, 30 years and older and either native born or had been living in the West Bank for more than 30 years, were interviewed using a previously prepared questionnaire. Of about 165 plant species mentioned by the informants, 63 (38.1%) were mentioned by three or more informants. On the basis of their primary uses, 21 of these plants were reported to relieve skin disorders, 17 for urinary system disorders, 16 for gastric disorders, nine for cancer and prostate disorders, eight for arthritis, five for respiratory problems, and five for other ailments. Indices on fidelity levels (FLs), relative popularity level (RPL), and rank-order priority (ROP) were calculated. Plants were classified in two groups: 'popular' (RPL=1) or 'unpopular' (RPL<1). The following plant species were classified as popular in this study: Teucrium polium, Matricaria aurea, Urtica pilulifera, Paronychia argentea, Petroselinum sativum, and Salvia fruticosa. The remaining 57 species were classified as 'unpopular'. Fifty-nine plants were claimed to be effective against cancer and prostate disorders, which include Arum dioscorides, U. pilulifera, Allium sativum, Viscum cruciatum, and Allium cepa.

Nematicidal activity of essential oils and their components against the root-knot nematode

ABSTRACT Nematicidal activity of essential oils extracted from 27 spices and aromatic plants were evaluated in vitro and in pot experiments. Twelve of the twenty-seven essential oils immobilized more than 80% of juveniles of the root-knot nematode Meloidogyne javanica at a concentration of 1,000 mul/liter. At this concentration, most of these oils also inhibited nematode hatching. Essential oils of Carum carvi, Foeniculum vulgare, Mentha rotundifolia, and Mentha spicata showed the highest nematicidal activity among the in vitro tested oils. These oils and those from Origanum vulgare, O. syriacum, and Coridothymus capitatus mixed in sandy soil at concentrations of 100 and 200 mg/kg reduced the root galling of cucumber seedlings in pot experiments. The main components of these essential oils were tested for their nematicidal activity. Carvacrol, t-anethole, thymol, and (+)-carvone immobilized the juveniles and inhibited hatching at >125 mul/liter in vitro. Most of these components mixed in sandy soil at concentrations of 75 and 150 mg/kg reduced root galling of cucumber seedlings. In 3-liter pot experiments, nematicidal activity of the essential oils and their components was confirmed at 200 and 150 mg/kg, respectively. The results suggest that the essential oils and their main components may serve as nematicides.

FRACAS: a system for computer-aided image-guided long bone fracture surgery

This article describes FRACAS, a computer-integrated orthopedic system for assisting surgeons in performing closed medullary nailing of long bone fractures. FRACAS's goal is to reduce the surgeon's cumulative exposure to radiation and surgical complications associated with alignment and positioning errors of bone fragments, nail insertion, and distal screw locking. It replaces uncorrelated, static fluoroscopic images with a virtual reality display of three-dimensional bone models created from preoperative computed tomography and tracked intraoperatively in real time. Fluoroscopic images are used to register the bone models to the intraoperative situation and to verify that the registration is maintained. This article describes the system concept, software prototypes of preoperative modules (modeling, nail selection, and visualization), intraoperative modules (fluoroscopic image processing and tracking), and preliminary in vitro experimental results to date. Our experiments suggest that the modeling, nail selection, and visualization modules yield adequate results and that fluoroscopic image processing with submillimetric accuracy is practically feasible on clinical images.

Cholesterol and triglyceride reduction in rats fed Matthiola incana seed oil rich in (n-3) fatty acids

Seeds of Matthiola incana contain oil rich (55-65%) in (n-3) linolenic acid. Selected lines were developed and evaluated for their agronomic and chemical parameters. Extracted oil was fed for 6 weeks to rats, which were compared with rats fed a diet containing coconut oil or sunflower oil. Cholesterol levels were significantly lowest in rats fed diets rich in M. incana oil (27% reduction), and triglycerides were significantly lower in rats receiving either M. incana or sunflower oil (36% reduction). The contents of arachidonic acid and other (n-6) fatty acids were significantly the lowest in the liver and plasma of rats that had received M. incana oil. The levels of (n-3) fatty acids were significantly greater in both the liver and plasma of rats fed M. incana oil. The ratio of (n-3)/(n-6) long-chain fatty acids in the plasma was 7 times higher in rats fed with M. incana oil than in those fed with sunflower oil and 6 times higher than in those fed coconut oil. The results demonstrate for the first time a beneficial effect of dietary M. incana oil in reducing cholesterol levels and increasing (n-3) fatty acid levels in the plasma. This new, terrestrial plant source of (n-3) fatty acids could replace marine oils and thereby contribute beneficially to the human diet.