Investigating keypoint descriptors for camera relocalization in endoscopy surgery

PURPOSE

Recent advances in computer vision and machine learning have resulted in endoscopic video-based solutions for dense reconstruction of the anatomy. To effectively use these systems in surgical navigation, a reliable image-based technique is required to constantly track the endoscopic camera's position within the anatomy, despite frequent removal and re-insertion. In this work, we investigate the use of recent learning-based keypoint descriptors for six degree-of-freedom camera pose estimation in intraoperative endoscopic sequences and under changes in anatomy due to surgical resection.

METHODS

Our method employs a dense structure from motion (SfM) reconstruction of the preoperative anatomy, obtained with a state-of-the-art patient-specific learning-based descriptor. During the reconstruction step, each estimated 3D point is associated with a descriptor. This information is employed in the intraoperative sequences to establish 2D-3D correspondences for Perspective-n-Point (PnP) camera pose estimation. We evaluate this method in six intraoperative sequences that include anatomical modifications obtained from two cadaveric subjects.

RESULTS

Show that this approach led to translation and rotation errors of 3.9 mm and 0.2 radians, respectively, with 21.86% of localized cameras averaged over the six sequences. In comparison to an additional learning-based descriptor (HardNet++), the selected descriptor can achieve a better percentage of localized cameras with similar pose estimation performance. We further discussed potential error causes and limitations of the proposed approach.

CONCLUSION

Patient-specific learning-based descriptors can relocalize images that are well distributed across the inspected anatomy, even where the anatomy is modified. However, camera relocalization in endoscopic sequences remains a persistently challenging problem, and future research is necessary to increase the robustness and accuracy of this technique.

Video-based assessment of intraoperative surgical skill

PURPOSE

Surgeons' skill in the operating room is a major determinant of patient outcomes. Assessment of surgeons' skill is necessary to improve patient outcomes and quality of care through surgical training and coaching. Methods for video-based assessment of surgical skill can provide objective and efficient tools for surgeons. Our work introduces a new method based on attention mechanisms and provides a comprehensive comparative analysis of state-of-the-art methods for video-based assessment of surgical skill in the operating room.

METHODS

Using a dataset of 99 videos of capsulorhexis, a critical step in cataract surgery, we evaluated image feature-based methods and two deep learning methods to assess skill using RGB videos. In the first method, we predict instrument tips as keypoints and predict surgical skill using temporal convolutional neural networks. In the second method, we propose a frame-wise encoder (2D convolutional neural network) followed by a temporal model (recurrent neural network), both of which are augmented by visual attention mechanisms. We computed the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and predictive values through fivefold cross-validation.

RESULTS

To classify a binary skill label (expert vs. novice), the range of AUC estimates was 0.49 (95% confidence interval; CI = 0.37 to 0.60) to 0.76 (95% CI = 0.66 to 0.85) for image feature-based methods. The sensitivity and specificity were consistently high for none of the methods. For the deep learning methods, the AUC was 0.79 (95% CI = 0.70 to 0.88) using keypoints alone, 0.78 (95% CI = 0.69 to 0.88) and 0.75 (95% CI = 0.65 to 0.85) with and without attention mechanisms, respectively.

CONCLUSION

Deep learning methods are necessary for video-based assessment of surgical skill in the operating room. Attention mechanisms improved discrimination ability of the network. Our findings should be evaluated for external validity in other datasets.

Reconstructing the nasal septum from instrument motion during septoplasty surgery

Purpose: Surgery involves modifying anatomy to achieve a goal. Reconstructing anatomy can facilitate surgical care through surgical planning, real-time decision support, or anticipating outcomes. Tool motion is a rich source of data that can be used to quantify anatomy. Our work develops and validates a method for reconstructing the nasal septum from unstructured motion of the Cottle elevator during the elevation phase of septoplasty surgery, without need to explicitly delineate the surface of the septum. Approach: The proposed method uses iterative closest point registration to initially register a template septum to the tool motion. Subsequently, statistical shape modeling with iterative most likely oriented point registration is used to fit the reconstructed septum to Cottle tip position and orientation during flap elevation. Regularization of the shape model and transformation is incorporated. The proposed methods were validated on 10 septoplasty surgeries performed on cadavers by operators of varying experience level. Preoperative CT images of the cadaver septums were segmented as ground truth. Results: We estimated reconstruction error as the difference between the projections of the Cottle tip onto the surface of the reconstructed septum and the ground-truth septum segmented from the CT image. We found translational differences of 2.74 ( 2.06 - 2.81 )    mm and a rotational differences of 8.95 ( 7.11 - 10.55 )    deg between the reconstructed septum and the ground-truth septum [median (interquartile range)], given the optimal regularization parameters. Conclusions: Accurate reconstruction of the nasal septum can be achieved from tool tracking data during septoplasty surgery on cadavers. This enables understanding of the septal anatomy without need for traditional medical imaging. This result may be used to facilitate surgical planning, intraoperative care, or skills assessment.

SAGES consensus recommendations on an annotation framework for surgical video

BACKGROUND

The growing interest in analysis of surgical video through machine learning has led to increased research efforts; however, common methods of annotating video data are lacking. There is a need to establish recommendations on the annotation of surgical video data to enable assessment of algorithms and multi-institutional collaboration.

METHODS

Four working groups were formed from a pool of participants that included clinicians, engineers, and data scientists. The working groups were focused on four themes: (1) temporal models, (2) actions and tasks, (3) tissue characteristics and general anatomy, and (4) software and data structure. A modified Delphi process was utilized to create a consensus survey based on suggested recommendations from each of the working groups.

RESULTS

After three Delphi rounds, consensus was reached on recommendations for annotation within each of these domains. A hierarchy for annotation of temporal events in surgery was established.

CONCLUSIONS

While additional work remains to achieve accepted standards for video annotation in surgery, the consensus recommendations on a general framework for annotation presented here lay the foundation for standardization. This type of framework is critical to enabling diverse datasets, performance benchmarks, and collaboration.

Ethical implications of AI in robotic surgical training: A Delphi consensus statement

CONTEXT

As the role of AI in healthcare continues to expand there is increasing awareness of the potential pitfalls of AI and the need for guidance to avoid them.

OBJECTIVES

To provide ethical guidance on developing narrow AI applications for surgical training curricula. We define standardised approaches to developing AI driven applications in surgical training that address current recognised ethical implications of utilising AI on surgical data. We aim to describe an ethical approach based on the current evidence, understanding of AI and available technologies, by seeking consensus from an expert committee.

EVIDENCE ACQUISITION

The project was carried out in 3 phases: (1) A steering group was formed to review the literature and summarize current evidence. (2) A larger expert panel convened and discussed the ethical implications of AI application based on the current evidence. A survey was created, with input from panel members. (3) Thirdly, panel-based consensus findings were determined using an online Delphi process to formulate guidance. 30 experts in AI implementation and/or training including clinicians, academics and industry contributed. The Delphi process underwent 3 rounds. Additions to the second and third-round surveys were formulated based on the answers and comments from previous rounds. Consensus opinion was defined as ≥ 80% agreement.

EVIDENCE SYNTHESIS

There was 100% response from all 3 rounds. The resulting formulated guidance showed good internal consistency, with a Cronbach alpha of >0.8. There was 100% consensus that there is currently a lack of guidance on the utilisation of AI in the setting of robotic surgical training. Consensus was reached in multiple areas, including: 1. Data protection and privacy; 2. Reproducibility and transparency; 3. Predictive analytics; 4. Inherent biases; 5. Areas of training most likely to benefit from AI.

CONCLUSIONS

Using the Delphi methodology, we achieved international consensus among experts to develop and reach content validation for guidance on ethical implications of AI in surgical training. Providing an ethical foundation for launching narrow AI applications in surgical training. This guidance will require further validation.

PATIENT SUMMARY

As the role of AI in healthcare continues to expand there is increasing awareness of the potential pitfalls of AI and the need for guidance to avoid them.In this paper we provide guidance on ethical implications of AI in surgical training.

From the Dexterous Surgical Skill to the Battlefield-A Robotics Exploratory Study

INTRODUCTION

Short response time is critical for future military medical operations in austere settings or remote areas. Such effective patient care at the point of injury can greatly benefit from the integration of semi-autonomous robotic systems. To achieve autonomy, robots would require massive libraries of maneuvers collected with the goal of training machine learning algorithms. Although this is attainable in controlled settings, obtaining surgical data in austere settings can be difficult. Hence, in this article, we present the Dexterous Surgical Skill (DESK) database for knowledge transfer between robots. The peg transfer task was selected as it is one of the six main tasks of laparoscopic training. In addition, we provide a machine learning framework to evaluate novel transfer learning methodologies on this database.

METHODS

A set of surgical gestures was collected for a peg transfer task, composed of seven atomic maneuvers referred to as surgemes. The collected Dexterous Surgical Skill dataset comprises a set of surgical robotic skills using the four robotic platforms: Taurus II, simulated Taurus II, YuMi, and the da Vinci Research Kit. Then, we explored two different learning scenarios: no-transfer and domain-transfer. In the no-transfer scenario, the training and testing data were obtained from the same domain; whereas in the domain-transfer scenario, the training data are a blend of simulated and real robot data, which are tested on a real robot.

RESULTS

Using simulation data to train the learning algorithms enhances the performance on the real robot where limited or no real data are available. The transfer model showed an accuracy of 81% for the YuMi robot when the ratio of real-tosimulated data were 22% to 78%. For the Taurus II and the da Vinci, the model showed an accuracy of 97.5% and 93%, respectively, training only with simulation data.

CONCLUSIONS

The results indicate that simulation can be used to augment training data to enhance the performance of learned models in real scenarios. This shows potential for the future use of surgical data from the operating room in deployable surgical robots in remote areas.

Deep Learning Method for Automated Classification of Anteroposterior and Posteroanterior Chest Radiographs

Ensuring correct radiograph view labeling is important for machine learning algorithm development and quality control of studies obtained from multiple facilities. The purpose of this study was to develop and test the performance of a deep convolutional neural network (DCNN) for the automated classification of frontal chest radiographs (CXRs) into anteroposterior (AP) or posteroanterior (PA) views. We obtained 112,120 CXRs from the NIH ChestX-ray14 database, a publicly available CXR database performed in adult (106,179 (95%)) and pediatric (5941 (5%)) patients consisting of 44,810 (40%) AP and 67,310 (60%) PA views. CXRs were used to train, validate, and test the ResNet-18 DCNN for classification of radiographs into anteroposterior and posteroanterior views. A second DCNN was developed in the same manner using only the pediatric CXRs (2885 (49%) AP and 3056 (51%) PA). Receiver operating characteristic (ROC) curves with area under the curve (AUC) and standard diagnostic measures were used to evaluate the DCNN's performance on the test dataset. The DCNNs trained on the entire CXR dataset and pediatric CXR dataset had AUCs of 1.0 and 0.997, respectively, and accuracy of 99.6% and 98%, respectively, for distinguishing between AP and PA CXR. Sensitivity and specificity were 99.6% and 99.5%, respectively, for the DCNN trained on the entire dataset and 98% for both sensitivity and specificity for the DCNN trained on the pediatric dataset. The observed difference in performance between the two algorithms was not statistically significant (p = 0.17). Our DCNNs have high accuracy for classifying AP/PA orientation of frontal CXRs, with only slight reduction in performance when the training dataset was reduced by 95%. Rapid classification of CXRs by the DCNN can facilitate annotation of large image datasets for machine learning and quality assurance purposes.

Transition state clustering: Unsupervised surgical trajectory segmentation for robot learning

Demonstration trajectories collected from a supervisor in teleoperation are widely used for robot learning, and temporally segmenting the trajectories into shorter, less-variable segments can improve the efficiency and reliability of learning algorithms. Trajectory segmentation algorithms can be sensitive to noise, spurious motions, and temporal variation. We present a new unsupervised segmentation algorithm, transition state clustering (TSC), which leverages repeated demonstrations of a task by clustering segment endpoints across demonstrations. TSC complements any motion-based segmentation algorithm by identifying candidate transitions, clustering them by kinematic similarity, and then correlating the kinematic clusters with available sensory and temporal features. TSC uses a hierarchical Dirichlet process Gaussian mixture model to avoid selecting the number of segments a priori. We present simulated results to suggest that TSC significantly reduces the number of false-positive segments in dynamical systems observed with noise as compared with seven probabilistic and non-probabilistic segmentation algorithms. We additionally compare algorithms that use piecewise linear segment models, and find that TSC recovers segments of a generated piecewise linear trajectory with greater accuracy in the presence of process and observation noise. At the maximum noise level, TSC recovers the ground truth 49% more accurately than alternatives. Furthermore, TSC runs 100× faster than the next most accurate alternative autoregressive models, which require expensive Markov chain Monte Carlo (MCMC)-based inference. We also evaluated TSC on 67 recordings of surgical needle passing and suturing. We supplemented the kinematic recordings with manually annotated visual features that denote grasp and penetration conditions. On this dataset, TSC finds 83% of needle passing transitions and 73% of the suturing transitions annotated by human experts.

Public Opinion and Policy Change in the American States

This article explores the relationship between public opinion and nonincremental policy change by extending the analysis of Wright et al. (1985, 1987). We develop a two-step model in which we first relate the level of a relevant outcome measure of a policy to the degree of opinion liberalism, the strategy of Wright and his colleagues. Then we posit that policy shocks will move the policy system into greater policy-opinion congruence. The model is tested for two policy areas that have undergone nonincremental change over the late 1970s and early 1980s: tax policy and education policy.

Task-Level vs. Segment-Level Quantitative Metrics for Surgical Skill Assessment

OBJECTIVE

Task-level metrics of time and motion efficiency are valid measures of surgical technical skill. Metrics may be computed for segments (maneuvers and gestures) within a task after hierarchical task decomposition. Our objective was to compare task-level and segment (maneuver and gesture)-level metrics for surgical technical skill assessment.

DESIGN

Our analyses include predictive modeling using data from a prospective cohort study. We used a hierarchical semantic vocabulary to segment a simple surgical task of passing a needle across an incision and tying a surgeon's knot into maneuvers and gestures. We computed time, path length, and movements for the task, maneuvers, and gestures using tool motion data. We fit logistic regression models to predict experience-based skill using the quantitative metrics. We compared the area under a receiver operating characteristic curve (AUC) for task-level, maneuver-level, and gesture-level models.

SETTING

Robotic surgical skills training laboratory.

PARTICIPANTS

In total, 4 faculty surgeons with experience in robotic surgery and 14 trainee surgeons with no or minimal experience in robotic surgery.

RESULTS

Experts performed the task in shorter time (49.74s; 95% CI = 43.27-56.21 vs. 81.97; 95% CI = 69.71-94.22), with shorter path length (1.63m; 95% CI = 1.49-1.76 vs. 2.23; 95% CI = 1.91-2.56), and with fewer movements (429.25; 95% CI = 383.80-474.70 vs. 728.69; 95% CI = 631.84-825.54) than novices. Experts differed from novices on metrics for individual maneuvers and gestures. The AUCs were 0.79; 95% CI = 0.62-0.97 for task-level models, 0.78; 95% CI = 0.6-0.96 for maneuver-level models, and 0.7; 95% CI = 0.44-0.97 for gesture-level models. There was no statistically significant difference in AUC between task-level and maneuver-level (p = 0.7) or gesture-level models (p = 0.17).

CONCLUSIONS

Maneuver-level and gesture-level metrics are discriminative of surgical skill and can be used to provide targeted feedback to surgical trainees.

Fundus image mosaicking for information augmentation in computer-assisted slit-lamp imaging

Laser photocoagulation is currently the standard treatment for sight-threatening diseases worldwide, namely diabetic retinopathy and retinal vein occlusions. The slit lamp biomicroscope is the most commonly used device for this procedure, specially for the treatment of the eye periphery. However, only a small portion of the retina can be visualized through the biomicroscope, complicating the task of localizing and identifying surgical targets, increasing treatment duration and patient discomfort. In order to assist surgeons, we propose a method for creating intraoperative retina maps for view expansion using a slit-lamp device. Based on the mosaicking method described by Richa et al, 2012, the proposed method is a combination of direct and feature-based methods, suitable for the textured nature of the human retina. In this paper, we describe three major enhancements to the original formulation. The first is a visual tracking method using local illumination compensation to cope with the challenging visualization conditions. The second is an efficient pixel selection scheme for increased computational efficiency. The third is an entropy-based mosaic update method to dynamically improve the retina map during exploration. To evaluate the performance of the proposed method, we conducted several experiments on human subjects with a computer-assisted slit-lamp prototype. We also demonstrate the practical value of the system for photo documentation, diagnosis and intraoperative navigation.

Ultrasound elastography as a tool for imaging guidance during prostatectomy: initial experience

BACKGROUND

During laparoscopic or robotic assisted laparoscopic prostatectomy, the surgeon lacks tactile feedback which can help him tailor the size of the excision. Ultrasound elastography (USE) is an emerging imaging technology which maps the stiffness of tissue. In the paper we are evaluating USE as a palpation equivalent tool for intraoperative image guided robotic assisted laparoscopic prostatectomy.

MATERIAL/METHODS

Two studies were performed: 1) A laparoscopic ultrasound probe was used in a comparative study of manual palpation versus USE in detecting tumor surrogates in synthetic and ex-vivo tissue phantoms; N=25 participants (students) were asked to provide the presence, size and depth of these simulated lesions, and 2) A standard ultrasound probe was used for the evaluation of USE on ex-vivo human prostate specimens (N=10 lesions in N=6 specimens) to differentiate hard versus soft lesions with pathology correlation. Results were validated by pathology findings, and also by in-vivo and ex-vivo MR imaging correlation.

RESULTS

In the comparative study, USE displayed higher accuracy and specificity in tumor detection (sensitivity=84%, specificity=74%). Tumor diameters and depths were better estimated using USE versus with manual palpation. USE also proved consistent in identification of lesions in ex-vivo prostate specimens; hard and soft, malignant and benign, central and peripheral.

CONCLUSIONS

USE is a strong candidate for assisting surgeons by providing palpation equivalent evaluation of the tumor location, boundaries and extra-capsular extension. The results encourage us to pursue further testing in the robotic laparoscopic environment.

3D Sensing Algorithms Towards Building an Intelligent Intensive Care Unit

Intensive Care Units (ICUs) are chaotic places where hundreds of tasks are carried out by many different people. Timely and coordinated execution of these tasks are directly related to quality of patient outcomes. An improved understanding of the current care process can aid in improving quality. Our goal is to build towards a system that automatically catalogs various tasks being performed by the bedside. We propose a set of techniques using computer vision and machine learning to develop a system that passively senses the environment and identifies seven common actions such as documenting, checking up on a patient, and performing a procedure. Preliminary evaluation of our system on 5.5 hours of data from the Pediatric ICU obtains overall task recognition accuracy of 70%. Furthermore, we show how it can be used to summarize and visualize tasks. Our system provides a significant departure from current approaches used for quality improvement. With further improvement, we think that such a system could realistically be deployed in the ICU.

Vision-based proximity detection in retinal surgery

In retinal surgery, surgeons face difficulties such as indirect visualization of surgical targets, physiological tremor, and lack of tactile feedback, which increase the risk of retinal damage caused by incorrect surgical gestures. In this context, intraocular proximity sensing has the potential to overcome current technical limitations and increase surgical safety. In this paper, we present a system for detecting unintentional collisions between surgical tools and the retina using the visual feedback provided by the opthalmic stereo microscope. Using stereo images, proximity between surgical tools and the retinal surface can be detected when their relative stereo disparity is small. For this purpose, we developed a system comprised of two modules. The first is a module for tracking the surgical tool position on both stereo images. The second is a disparity tracking module for estimating a stereo disparity map of the retinal surface. Both modules were specially tailored for coping with the challenging visualization conditions in retinal surgery. The potential clinical value of the proposed method is demonstrated by extensive testing using a silicon phantom eye and recorded rabbit in vivo data.

A Clinical Pilot Study of a Modular Video-CT Augmentation System for Image-Guided Skull Base Surgery

Augmentation of endoscopic video with preoperative or intraoperative image data [e.g., planning data and/or anatomical segmentations defined in computed tomography (CT) and magnetic resonance (MR)], can improve navigation, spatial orientation, confidence, and tissue resection in skull base surgery, especially with respect to critical neurovascular structures that may be difficult to visualize in the video scene. This paper presents the engineering and evaluation of a video augmentation system for endoscopic skull base surgery translated to use in a clinical study. Extension of previous research yielded a practical system with a modular design that can be applied to other endoscopic surgeries, including orthopedic, abdominal, and thoracic procedures. A clinical pilot study is underway to assess feasibility and benefit to surgical performance by overlaying CT or MR planning data in real-time, high-definition endoscopic video. Preoperative planning included segmentation of the carotid arteries, optic nerves, and surgical target volume (e.g., tumor). An automated camera calibration process was developed that demonstrates mean re-projection accuracy (0.7±0.3) pixels and mean target registration error of (2.3±1.5)mm. An IRB-approved clinical study involving fifteen patients undergoing skull base tumor surgery is underway in which each surgery includes the experimental video-CT system deployed in parallel to the standard-of-care (un-augmented) video display. Questionnaires distributed to one neurosurgeon and two otolaryngologists are used to assess primary outcome measures regarding the benefit to surgical confidence in localizing critical structures and targets by means of video overlay during surgical approach, resection, and reconstruction.

Objective measures for longitudinal assessment of robotic surgery training

OBJECTIVES

Current robotic training approaches lack the criteria for automatically assessing and tracking (over time) technical skills separately from clinical proficiency. We describe the development and validation of a novel automated and objective framework for the assessment of training.

METHODS

We are able to record all system variables (stereo instrument video, hand and instrument motion, buttons and pedal events) from the da Vinci surgical systems using a portable archival system integrated with the robotic surgical system. Data can be collected unsupervised, and the archival system does not change system operations in any way. Our open-ended multicenter protocol is collecting surgical skill benchmarking data from 24 trainees to surgical proficiency, subject only to their continued availability. Two independent experts performed structured (objective structured assessment of technical skills) assessments on longitudinal data from 8 novice and 4 expert surgeons to generate baseline data for training and to validate our computerized statistical analysis methods in identifying the ranges of operational and clinical skill measures.

RESULTS

Objective differences in operational and technical skill between known experts and other subjects were quantified. The longitudinal learning curves and statistical analysis for trainee performance measures are reported. Graphic representations of the skills developed for feedback to the trainees are also included.

CONCLUSIONS

We describe an open-ended longitudinal study and automated motion recognition system capable of objectively differentiating between clinical and technical operational skills in robotic surgery. Our results have demonstrated a convergence of trainee skill parameters toward those derived from expert robotic surgeons during the course of our training protocol.

Assessing system operation skills in robotic surgery trainees

BACKGROUND

With increased use of robotic surgery in specialties including urology, development of training methods has also intensified. However, current approaches lack the ability to discriminate between operational and surgical skills.

METHODS

An automated recording system was used to longitudinally (monthly) acquire instrument motion/telemetry and video for four basic surgical skills - suturing, manipulation, transection, and dissection. Statistical models were then developed to discriminate the human-machine skill differences between practicing expert surgeons and trainees.

RESULTS

Data from six trainees and two experts was analyzed to validate the first ever statistical models of operational skills, and demonstrate classification with very high accuracy (91.7% for masters, and 88.2% for camera motion) and sensitivity.

CONCLUSIONS

The paper reports on a longitudinal study aimed at tracking robotic surgery trainees to proficiency, and methods capable of objectively assessing operational and technical skills that would be used in assessing trainee progress at the participating institutions.

Assessment of Crohn's disease lesions in wireless capsule endoscopy images

Capsule endoscopy (CE) provides noninvasive access to a large part of the small bowel that is otherwise inaccessible without invasive and traumatic treatment. However, it also produces large amounts of data (approximately 50,000 images) that must be then manually reviewed by a clinician. Such large datasets provide an opportunity for application of image analysis and supervised learning methods. Automated analysis of CE images has only focused on detection, and often only for bleeding. Compared to these detection approaches, we explored assessment of discrete disease for lesions created by mucosal inflammation in Crohn's disease (CD). Our work is the first study to systematically explore supervised classification for CD lesions, a classifier cascade to classify discrete lesions, as well as quantitative assessment of lesion severity. We used a well-developed database of 47 studies for evaluation of these methods. The developed methods show high agreement with ground truth severity ratings manually assigned by an expert, and good precision ( > 90% for lesion detection) and recall ( > 90%) for lesions of varying severity.

Augmenting capsule endoscopy diagnosis: a similarity learning approach

The current procedure for diagnosis of Crohn's disease (CD) from Capsule Endoscopy is a tedious manual process which requires the clinician to visually inspect large video sequences for matching and categorization of diseased areas (lesions). Automated methods for matching and classification can help improve this process by reducing diagnosis time and improving consistency of categorization. In this paper, we propose a novel SVM-based similarity learning method for distinguishing between correct and incorrect matches in Capsule Endoscopy (CE). We also show that this can be used in conjunction with a voting scheme to categorize lesion images. Results show that our methods outperform standard classifiers in discriminating similar from dissimilar lesion images, as well as in lesion categorization. We also show that our methods drastically reduce the complexity (training time) by requiring only one half of the data for training, without compromising the accuracy of the classifier.

Quantum phase transition in a 1D transport model with boson-affected hopping: Luttinger liquid versus charge-density-wave behavior

We solve a very general two-channel fermion-boson model describing charge transport within some background medium by means of a refined pseudosite density-matrix renormalization group technique. Performing a careful finite-size scaling analysis, we determine the ground-state phase diagram and convincingly prove that the model exhibits a metal-insulator quantum phase transition for the half-filled band case. In order to characterize the metallic and insulating regimes we calculate, besides the local particle densities and fermion-boson correlation functions, the kinetic energy, the charge-structure factor, the Luttinger liquid charge exponent, and the single-particle excitation gap for a one-dimensional infinite system.

Data-derived models for segmentation with application to surgical assessment and training

This paper addresses automatic skill assessment in robotic minimally invasive surgery. Hidden Markov models (HMMs) are developed for individual surgical gestures (or surgemes) that comprise a typical bench-top surgical training task. It is known that such HMMs can be used to recognize and segment surgemes in previously unseen trials. Here, the topology of each surgeme HMM is designed in a data-driven manner, mixing trials from multiple surgeons with varying skill levels, resulting in HMM states that model skill-specific sub-gestures. The sequence of HMM states visited while performing a surgeme are therefore indicative of the surgeon's skill level. This expectation is confirmed by the average edit distance between the state-level "transcripts" of the same surgeme performed by two surgeons with different expertise levels. Some surgemes are further shown to be more indicative of skill than others.

Ultrasound elastography: a dynamic programming approach

This paper introduces a 2-D strain imaging technique based on minimizing a cost function using dynamic programming (DP). The cost function incorporates similarity of echo amplitudes and displacement continuity. Since tissue deformations are smooth, the incorporation of the smoothness into the cost function results in reduced decorrelation noise. As a result, the method generates high-quality strain images of freehand palpation elastography with up to 10% compression, showing that the method is more robust to signal decorrelation (caused by scatterer motion in high axial compression and nonaxial motions of the probe) in comparison to the standard correlation techniques. The method operates in less than 1 s and is thus also potentially suitable for real time elastography.

Anatomical reconstruction from endoscopic images: toward quantitative endoscopy

BACKGROUND

Recent advances in computational image processing have made it possible to reconstruct camera motion and scene geometry from a series of monocular images. By applying these methods to endoscopic image sequences, it is possible to create detailed, quantitative anatomic reconstructions. Such anatomic reconstructions have many potential clinical uses. Our objectives in this study are to (1) develop a process flow for reconstruction from endoscopic image sequences and (2) present results supporting the hypothesis that such reconstructions can be computed.

METHODS

We first outline the overall process flow for endoscopic reconstruction. Then, we present an instantiation of this process flow using recently developed methods in computational vision. We apply these methods to cadaverous specimens for which ground truth endoscopic motion is known.

RESULTS

We are able to produce consistent estimates of endoscopic motion and dense reconstructions of the surrounding anatomy for >65% of 1373 image pairs.

CONCLUSION

Our study indicates that processing endoscopic images to produce anatomic structure is feasible. Such reconstructions have high potential clinical value for intraoperative navigation, diagnosis, and treatment planning.

Cooperative robot assistant for retinal microsurgery

This paper describes the development and results of initial testing of a cooperative robot assistant for retinal microsurgery. In the cooperative control paradigm, the surgeon and the robot share control of a tool attached to the robot through a force sensor. The system senses forces exerted by the operator on the tool and uses this information in various control modes to provide smooth, tremor-free, precise positional control and force scaling. The robot manipulator is specifically designed with retinal microsurgery in mind, having high efficacy, flexibility and ergonomics while meeting the accuracy and safety requirements of microsurgery. We have tested this robot on a biological model and we report the results for reliably cannulating approximately 80 microm diameter veins (equivalent in size to human retinal veins). We also describe improvements to the robot and the experimental setup facilitating more advanced set of experiments.

Ultrasound monitoring of tissue ablation via deformation model and shape priors

A rapid approach to monitor ablative therapy through optimizing shape and elasticity parameters is introduced. Our motivating clinical application is targeting and intraoperative monitoring of hepatic tumor thermal ablation, but the method translates to the generic problem of encapsulated stiff masses (solid organs, tumors, ablated lesions, etc.) in ultrasound imaging. The approach involves the integration of the following components: a biomechanical computational model of the tissue, a correlation approach to estimate/track tissue deformation, and an optimization method to solve the inverse problem and recover the shape parameters in the volume of interest. Successful convergence and reliability studies were conducted on simulated data. Then ex-vivo studies were performed on 18 ex-vivo bovine liver samples previously ablated under ultrasound monitoring in controlled laboratory environment. While B-mode ultrasound does not clearly identify the development of necrotic lesions, the proposed technique can potentially segment the ablation zone. The same framework can also yield both partial and full elasticity reconstruction.

AFM imaging of protein movements: histone H2A-H2B release during nucleosome remodeling

Being able to follow assembly/disassembly reactions of biomolecular complexes directly at the single molecule level would be very useful. Here, we use an AFM technique that can simultaneously obtain topographic images and identify the locations of a specific type of protein within those images to monitor the histone H2A component of nucleosomes acted on by human Swi-Snf, an ATP-dependent nucleosome remodeling complex. Activation of remodeling results in significant H2A release from nucleosomes, based on recognition imaging and nucleosome height changes, and changes in the recognition patterns of H2A associated directly with hSwi-Snf complexes.

Bootstrapped ultrasound calibration

This paper introduces an enhanced (bootstrapped) method for tracked ultrasound probe calibration. Prior to calibration, a position sensor is used to track an ultrasound probe in 3D space, while the US image is used to determine calibration target locations within the image. From this information, an estimate of the transformation matrix of the scan plane with respect to the position sensor is computed. While all prior calibration methods terminate at this phase, we use this initial calibration estimate to bootstrap an additional optimization of the transformation matrix on independent data to yield the minimum reconstruction error on calibration targets. The bootstrapped workflow makes use of a closed-form calibration solver and associated sensitivity analysis, allowing for rapid and robust convergence to an optimal calibration matrix. Bootstrapping demonstrates superior reconstruction accuracy.

Real-Time Endoscopic Mosaicking

With the advancement of minimally invasive techniques for surgical and diagnostic procedures, there is a growing need for the development of methods for improved visualization of internal body structures. Video mosaicking is one method for doing this. This approach provides a broader field of view of the scene by stitching together images in a video sequence. Of particular importance is the need for online processing to provide real-time feedback and visualization for image-guided surgery and diagnosis. We propose a method for online video mosaicking applied to endoscopic imagery, with examples in microscopic retinal imaging and catadioptric endometrial imaging.

Using atomic force microscopy to study nucleosome remodeling on individual nucleosomal arrays in situ

In eukaryotes, genomic processes like transcription, replication, repair, and recombination typically require alterations in nucleosome structure on specific DNA regions to operate. ATP-dependent nucleosome remodeling complexes provide a major mechanism for carrying out such alterations in vivo. To learn more about the action of these important complexes, we have utilized an atomic force microscopy in situ technique that permits comparison of the same individual molecules before and after activation of a particular process, in this case nucleosome remodeling. This direct approach was used to look for changes induced by the action of the human Swi-Snf remodeling complex on individual, single-copy mouse mammary tumor virus promoter nucleosomal arrays. Using this technique, we detect a variety of changes on remodeling. Many of these changes are larger in scale than suggested from previous studies and involve a number of DNA-mediated events, including a preference for the removal of a complete turn (80 basepairs) of nucleosomal DNA. The latter result raises the possibility of an unanticipated mode of human Swi-Snf interaction with the nucleosome, namely via the 11-nm histone surface.

Nucleosomal arrays can be salt-reconstituted on a single-copy MMTV promoter DNA template: their properties differ in several ways from those of comparable 5S concatameric arrays

Subsaturated nucleosomal arrays were reconstituted on a single-copy MMTV promoter DNA fragment by salt dialysis procedures and studied by atomic force microscopy. Up to an occupation level of approximately eight nucleosomes on this 1900 bp template, salt reconstitution produces nucleosomal arrays which look very similar to comparably loaded 5S rDNA nucleosomal arrays; i.e., nucleosomes are dispersed on the DNA template. Thus, at these occupation levels, the single-copy MMTV template forms arrays suitable for biophysical analyses. A quantitative comparison of the population features of subsaturated MMTV and 5S arrays detects differences between the two: a requirement for higher histone levels to achieve a given level of nucleosome occupation on MMTV templates, indicating that nucleosome loading is thermodynamically less favorable on this template; a preference for pairwise nucleosome occupation of the MMTV (but not the 5S) template at midrange occupation levels; and an enhanced salt stability for nucleosomes on MMTV versus 5S arrays, particularly in the midrange of array occupation. When average occupation levels exceed approximately eight nucleosomes per template, MMTV arrays show a significant level of mainly intramolecular compaction; 5S arrays do not. Taken together, these results show clearly that the nature of the underlying DNA template can affect the physical properties of nucleosomal arrays. DNA sequence-directed differences in the physical properties of chromatin may have important consequences for functional processes such as gene regulation.

Ethanol decreases expression of p21 and increases hyperphosphorylated pRb in cell lines of squamous cell carcinomas of the head and neck

BACKGROUND

Alcohol increases the risk of cancers of the upper aerodigestive tract, but the biological mechanisms of this ethanol effect are still unclear. We recently reported that ethanol is able to induce in vitro proliferation accompanied by an increased number of cells in the S phase of the cell cycle in squamous cell carcinoma cell lines of the head and neck (SCCHN). In the current study we investigated the influence of ethanol over a limited period of time (96 hr) on cell cycle-regulating proteins involved in G1/S phase transition.

METHODS

Synchronized cells of SCCHN cell lines JPPA (larynx) and SCC 9 and SCC 25 (tongue), as well as HaCaT (human immortalized keratinocytes)-used as a control-were cultured for 96 hr in the presence or absence of ethanol (10-3M). At several time intervals the expression of cyclin D1 and p21 and the phosphorylation status of the retinoblastoma protein (pRb) were determined by Western or Northern Blot analysis, or both.

RESULTS

Ethanol had no influence on the protein expression of cyclin D1. In contrast, a distinct downregulation of p21 at the protein as well as the mRNA level could be detected. Furthermore, as a downstream event, the hyperphosphorylated form of the pRb increased.

CONCLUSIONS

In the acute alcohol in vitro experiments, the marked downregulation of the important cell cycle inhibitor p21 and the corresponding increase of hyperphosphorylated pRb accelerate the progression of cells from the G1 to the S phase in the cell cycle. The importance of these data and their relevance to in vivo conditions remain speculative, but it could be a critical step in the multistep process of SCCHN carcinogenesis induced by ethanol.

c-Jun regulation in rat neonatal motoneurons postaxotomy

Motoneurons respond to peripheral nerve transection by either regenerative or degenerative events depending on their state of maturation. Since the expression of c-Jun has been involved in the early signalling of the regenerative process that follows nerve transection in adults, we have investigated c-Jun on rat neonatal axotomized motoneurons during the period in which neuronal death is induced. Changes in levels of c-Jun protein and its mRNA were determined by means of quantitative immunocytochemistry and in situ hybridization. Three hours after nerve transection performed on postnatal day (P)3, c-Jun protein and mRNA is induced in axotomized spinal cord motoneurons, and high levels were reached between 1 and 10 days after. This response is associated with a detectable c-Jun activation by phosphorylation on serine 63. No changes were found in the levels of activating transcription factor -2. Most of dying motoneurons were not labelled by either a specific c-Jun antibody or a c-jun mRNA probe. However, dying motoneurons were specifically stained by a polyclonal anti c-Jun antibody, indicating that some c-Jun antibodies react with unknown epitopes, probably distinct from c-Jun p39, that are specifically associated with apoptosis.

A novel role for protein tyrosine phosphatase shp1 in controlling glial activation in the normal and injured nervous system

Tyrosine phosphorylation regulated by protein tyrosine kinases and phosphatases plays an important role in the activation of glial cells. Here we examined the expression of intracellular protein tyrosine phosphatase SHP1 in the normal and injured adult rat and mouse CNS. Our study showed that in the intact CNS, SHP1 was expressed in astrocytes as well as in pyramidal cells in hippocampus and cortex. Axotomy of peripheral nerves and direct cortical lesion led to a massive upregulation of SHP1 in activated microglia and astrocytes, whereas the neuronal expression of SHP1 was not affected. In vitro experiments revealed that in astrocytes, SHP1 associates with epidermal growth factor (EGF)-receptor, whereas in microglia, SHP1 associates with colony-stimulating factor (CSF)-1-receptor. In postnatal and adult moth-eaten viable (me(v)/me(v)) mice, which are characterized by reduced SHP1 activity, a strong increase in reactive astrocytes, defined by GFAP immunoreactivity, was observed throughout the intact CNS, whereas neither the morphology nor the number of microglial cells appeared modified. Absence of (3)[H]-thymidine-labeled nuclei indicated that astrocytic proliferation does not occur. In response to injury, cell number as well as proliferation of microglia were reduced in me(v)/me(v) mice, whereas the posttraumatic astrocytic reaction did not differ from wild-type littermates. The majority of activated microglia in mutant mice showed rounded and ameboid morphology. However, the regeneration rate after facial nerve injury in me(v)/me(v) mice was similar to that in wild-type littermates. These results emphasize that SHP1 as a part of different signaling pathways plays an important role in the global regulation of astrocytic and microglial activation in the normal and injured CNS.

Neuronal MCP-1 expression in response to remote nerve injury

Direct injury of the brain is followed by inflammatory responses regulated by cytokines and chemoattractants secreted from resident glia and invading cells of the peripheral immune system. In contrast, after remote lesion of the central nervous system, exemplified here by peripheral transection or crush of the facial and hypoglossal nerve, the locally observed inflammatory activation is most likely triggered by the damaged cells themselves, that is, the injured neurons. The authors investigated the expression of the chemoattractants monocyte chemoattractant protein MCP-1, regulation on activation normal T-cell expressed and secreted (RANTES), and interferon-gamma inducible protein IP10 after peripheral nerve lesion of the facial and hypoglossal nuclei. In situ hybridization and immunohistochemistry revealed an induction of neuronal MCP-1 expression within 6 hours postoperation, reaching a peak at 3 days and remaining up-regulated for up to 6 weeks. MCP-1 expression was almost exclusively confined to neurons but was also present on a few scattered glial cells. The authors found no alterations in the level of expression and cellular distribution of RANTES or IP10, which were both confined to neurons. Protein expression of the MCP-1 receptor CCR2 did not change. MCP-1, expressed by astrocytes and activated microglia, has been shown to be crucial for monocytic, or T-cell chemoattraction, or both. Accordingly, expression of MCP-1 by neurons and its corresponding receptor in microglia suggests that this chemokine is involved in neuron and microglia interaction.

1,25(OH)2 vitamin D3 induces elevated expression of the cell cycle-regulating genes P21 and P27 in squamous carcinoma cell lines of the head and neck

The biologically active form of vitamin D3, 1,25-dihydroxyvitamin D3 [1,25(OH)2D3], inhibits proliferation and induces differentiation for various malignant cells, including squamous cell carcinoma cell lines of the head and neck (SCCHN). These effects are due to an arrest of cells in the G0/G1 phase of the cell cycle and are predominantly mediated by the vitamin D receptor. To further explore the molecular mechanisms of the antiproliferative activity in SCCHN we studied the influence of 1,25(OH)2D3 on the expression of the G1 phase-regulating proteins cyclin D1, p21 and p27. Furthermore, as a direct target of G1 protein complexes, we investigated the phosphorylation status of the retinoblastoma protein (pRb). Synchronized cells of 2 SCCHN cell lines [JPPA (laryngeal carcinoma) and SCC 9 (tongue carcinoma)] and human immortalized keratinocytes (HaCaT) were cultured for 96 h in the presence or absence (ethanol as control) of 1,25(OH)2D3 (10(-7) M). At various time intervals the cell cycle status was detected by fluorescence-activated cell sorting (FACS) analysis and in parallel the expression of cell cycle-regulating proteins was determined at the protein and mRNA levels. In all cell lines tested 1,25(OH)2D3 caused an arrest of cells in the G0/G1 phase of the cell cycle and markedly induced the expression of the inhibitors p21 and p27. No influence was detectable on the expression of cyclin D1. Induction of p21 and p27 mRNA revealed transcriptional regulation by the vitamin D receptor. Simultaneously, hyperphosphorylated pRb was transformed to the hypophosphorylated form. Our results demonstrate that the biologically active form of vitamin D3 directly regulates the expression of p21 and p27, inducing a G0/G1 phase arrest: one mechanism by which 1,25(OH)2D3 controls cell proliferation inSCCHN.