Tracking and mapping in medical computer vision: A review

As computer vision algorithms increase in capability, their applications in clinical systems will become more pervasive. These applications include: diagnostics, such as colonoscopy and bronchoscopy; guiding biopsies, minimally invasive interventions, and surgery; automating instrument motion; and providing image guidance using pre-operative scans. Many of these applications depend on the specific visual nature of medical scenes and require designing algorithms to perform in this environment. In this review, we provide an update to the field of camera-based tracking and scene mapping in surgery and diagnostics in medical computer vision. We begin with describing our review process, which results in a final list of 515 papers that we cover. We then give a high-level summary of the state of the art and provide relevant background for those who need tracking and mapping for their clinical applications. After which, we review datasets provided in the field and the clinical needs that motivate their design. Then, we delve into the algorithmic side, and summarize recent developments. This summary should be especially useful for algorithm designers and to those looking to understand the capability of off-the-shelf methods. We maintain focus on algorithms for deformable environments while also reviewing the essential building blocks in rigid tracking and mapping since there is a large amount of crossover in methods. With the field summarized, we discuss the current state of the tracking and mapping methods along with needs for future algorithms, needs for quantification, and the viability of clinical applications. We then provide some research directions and questions. We conclude that new methods need to be designed or combined to support clinical applications in deformable environments, and more focus needs to be put into collecting datasets for training and evaluation.

Six Degree-of-Freedom Measurements of Human Mild Traumatic Brain Injury

This preliminary study investigated whether direct measurement of head rotation improves prediction of mild traumatic brain injury (mTBI). Although many studies have implicated rotation as a primary cause of mTBI, regulatory safety standards use 3 degree-of-freedom (3DOF) translation-only kinematic criteria to predict injury. Direct 6DOF measurements of human head rotation (3DOF) and translation (3DOF) have not been previously available to examine whether additional DOFs improve injury prediction. We measured head impacts in American football, boxing, and mixed martial arts using 6DOF instrumented mouthguards, and predicted clinician-diagnosed injury using 12 existing kinematic criteria and 6 existing brain finite element (FE) criteria. Among 513 measured impacts were the first two 6DOF measurements of clinically diagnosed mTBI. For this dataset, 6DOF criteria were the most predictive of injury, more than 3DOF translation-only and 3DOF rotation-only criteria. Peak principal strain in the corpus callosum, a 6DOF FE criteria, was the strongest predictor, followed by two criteria that included rotation measurements, peak rotational acceleration magnitude and Head Impact Power (HIP). These results suggest head rotation measurements may improve injury prediction. However, more 6DOF data is needed to confirm this evaluation of existing injury criteria, and to develop new criteria that considers directional sensitivity to injury.

Registration of 3D ultrasound through an air-tissue boundary

In this study we evaluated a new method for registering three-dimensional ultrasound (3DUS) data to external coordinate systems. First, 3DUS was registered to the stereo endoscope of a da Vinci Surgical System by placing a registration tool against an air-tissue boundary so that the 3DUS could image ultrasound fiducials while the stereo endoscope could image camera markers on the same tool. The common points were used to solve the registration between the 3DUS and camera coordinate systems. The target registration error (TRE) when imaging through a PVC tissue phantom ranged from 3.85 1.76 mm to 1.82 1.03 mm using one to four registration tool positions. TRE when imaging through an ex-vivo liver tissue sample ranged from 2.36 1.01 mm to 1.51 0.70 mm using one to four registration tool positions. Second, using a similar method, 3DUS was registered to the kinematic coordinate system of a da Vinci Surgical System by using the da Vinci surgical manipulators to identify common points on an air-tissue boundary. TRE when imaging through a PVC tissue phantom was 0.95 0.38 mm. This registration method is simpler and potentially more accurate than methods using commercial motion tracking systems. This method may be useful in the future in augmented reality systems for laparoscopic and robotic-assisted surgery.

Tissue tracking and registration for image-guided surgery

Vision-based tracking of tissue is a key component to enable augmented reality during a surgical operation. Conven- tional tracking techniques in computer vision rely on identifying strong edge features or distinctive textures in a well-lit environ- ment; however endoscopic tissue images do not have strong edge features, are poorly lit and exhibit a high degree of specular reflection. Therefore, prior work in achieving densely populated 3D features for describing tissue surface profiles require complex image processing techniques and have been limited in providing stable, long-term tracking or real-time processing. In this paper, we present an integrated framework for ac- curately tracking tissue in surgical stereo-cameras at real-time speeds. We use a combination of the STAR feature detector and Binary Robust Independent Elementary Features to acquire salient features that can be persistently tracked at high frame rates. The features are then used to acquire a densely-populated map of the deformations of tissue surface in 3D. We evaluate the method against popular feature algorithms in in-vivo animal study video sequences, and we also apply the proposed method to human partial nephrectomy video sequences. We extend the salient feature framework to support region tracking in order to maintain the spatial correspondence of a tracked region of tissue or a medical image registration to the surrounding tissue. In-vitro tissue studies show registration accuracies of 1.3-3.3 mm using a rigid-body transformation method.

Indirect low-intensity ultrasonic stimulation for tissue engineering

Low-intensity ultrasound (LIUS) treatment has been shown to increase mass transport, which could benefit tissue grafts during the immediate postimplant period, when blood supply to the implanted tissue is suboptimal. In this in vitro study, we investigated effects of LIUS stimulation on dye diffusion, proliferation, metabolism, and tropomyosin expression of muscle cells (C2C12) and on tissue viability and gene expression of human adipose tissue organoids. We found that LIUS increased dye diffusion within adjacent tissue culture wells and caused anisotropic diffusion patterns. This effect was confirmed by a hydrophone measurement resulting in acoustic pressure 150–341 Pa in wells. Cellular studies showed that LIUS significantly increased proliferation, metabolic activity, and expression of tropomyosin. Adipose tissue treated with LIUS showed significantly increased metabolic activity and the cells had similar morphology to normal unilocular adipocytes. Gene analysis showed that tumor necrosis factor-alpha expression (a marker for tissue damage) was significantly lower for stimulated organoids than for control groups. Our data suggests that LIUS could be a useful modality for improving graft survival in vivo.

A robust uniaxial force sensor for minimally invasive surgery

This paper presents a novel miniature uniaxial force sensor for use within a beating heart during mitral valve annuloplasty. The sensor measures 5.5 mm in diameter and 12 mm in length and provides a hollow core to pass instrumentation. A soft elastomer flexure design maintains a waterproof seal. Fiber optic transduction eliminates electrical circuitry within the heart, and acetal components minimize ultrasound-imaging artifacts. Calibration uses a nonlinear viscoelastic method, and in vitro tests demonstrate a 0-4-N force range with rms errors of 0.13 N (< 3.2%). In vivo tests provide the first endocardial measurements of tissue-minimally invasive surgery instrument interaction forces in a beating heart.

Robotic force stabilization for beating heart intracardiac surgery

The manipulation of fast moving, delicate tissues in beating heart procedures presents a considerable challenge to surgeons. We present a new robotic force stabilization system that assists surgeons by maintaining a constant contact force with the beating heart. The system incorporates a novel, miniature uniaxial force sensor that is mounted to surgical instrumentation to measure contact forces during surgical manipulation. Using this sensor in conjunction with real-time tissue motion information derived from 3D ultrasound, we show that a force controller with feed-forward motion terms can provide safe and accurate force stabilization in an in vivo contact task against the beating mitral valve annulus. This confers a 50% reduction in force fluctuations when compared to a standard force controller and a 75% reduction in fluctuations when compared to manual attempts to maintain the same force.

Possible-word constraints in Cantonese speech segmentation

A Cantonese syllable-spotting experiment was conducted to examine whether the Possible-Word Constraint (PWC), proposed by Norris, McQueen, Cutler, and Butterfield (1997), can apply in Cantonese speech segmentation. In the experiment, listeners were asked to spot out the target Cantonese syllable from a series of nonsense sound strings. Results suggested that listeners found it more difficult to spot out the target syllable [kDm1] in the nonsense sound strings that attached with a single consonant [tkDm1] than in the nonsense sound strings that attached either with a vowel [a:kDm1] or a pseudo-syllable [khow1kDm1]. Finally, the current set of results further supported that the PWC appears to be a language-universal mechanism in segmenting continuous speech.

A preliminary study of subjective frequency estimates of words spoken in Cantonese

A database is presented of the subjective frequency estimates for a set of 30 Chinese homophones. The estimates are based on analysis of responses from a simple listening task by 120 University students. On the listening task, they are asked to mention the first meaning thought of upon hearing a Chinese homophone by writing down the corresponding Chinese characters. There was correlation of .66 between the frequency of spoken and written words, suggesting distributional information about the lexical representations is generally independent of modality. These subjective frequency counts should be useful in the construction of material sets for research on word recognition using spoken Chinese (Cantonese).

An outbreak of influenza a caused by imported virus in the United States, July 1999

We report 32 cases of culture-proven influenza A (A/Sydney) caused by virus imported into mainland US military barracks from Puerto Rico in July 1999. Despite the fact that the shelf life of the influenza vaccine is 18 months and that the outbreak strain was a component of the previous year's vaccine, no vaccine was available from manufacturers, owing to US Food and Drug Administration regulations. Formal consideration should be given to extending the date of expiration and to maintaining a supply of the influenza vaccine year-round.

Reanalysis of stone manipulation prior to extracorporeal shock wave lithotripsy for management of upper ureteral calculi

BACKGROUND

In situ extracorporeal shock wave lithotripsy (ESWL) for upper ureteral stones is controversial, in contrast to ESWL treatment of the majority of other renal calculi. Some urologists recommend retrograde mobilization of the calculi into the renal pelvis prior to ESWL as a routine procedure, while others report little advantage in manipulating a ureteral stone into the kidney before treatment by ESWL. To better understand this controversial issue, we reanalyzed clinical studies and the mechanism of stone disintegration in ESWL.

METHODS

We performed a computerized MEDLINE search and manual bibliographic review of relevant peer reviewed reports from 1980 to 1998. Ten reports were analyzed with respect to stone-free rate and the important findings are summarized.

RESULTS

Two of 10 data sets are statistically significant (p < 0.05) to verify a higher stone-free rate and another five reports also revealed that higher stone-free rates could be expected with retrograde stone manipulation into the renal collecting system before treatment by ESWL.

CONCLUSIONS

Higher stone-free rates with retrograde stone manipulation into the renal collecting system before treatment by ESWL is statistically significant and theoretically supported by the mechanism of stone disintegration.

The enzymic synthesis of GM1b: rat-brain CMP-N-acetylneuraminic acid: asialo-GM1 sialyltransferase

An enzyme which catalyzes the transfer of N-acetylneuraminic acid (NeuNAc) to a tetrahexosylceramide (asialo-GM1) in young rat brain is described. The enzymic product is a new monosialoganglioside containing a neuraminidase-labile neuraminic acid, GM1b. The activity of this sialyltransferase is higher in fetal and young rat brains. The enzyme exhibits a pH optimum of 6.5 in cacodylate buffer. The incorporation of radioactivity into GM1b is stimulated in the presence of asialo-GM1 and CMP-NeuNAc and is dependent on the quantity added. The detergent mixture, Tween 80 and CF54, is required for optimal activity. Recent demonstration of the natural occurrence of GM1b in the free cell types of rat ascites hepatopa cells suggests a functional importance of this CMP-Neu-NAc:asialo-GM1 sialyltransferase in the in vivo formation of this novel monosialoganglioside.

Chemical structure of monosialoganglioside GM1b biosynthesized in vitro

The structure of a neuraminidase-labile monosialoganglioside which is formed in vivo from asialoganglioside (galactosyl (beta, 1 in equilibrium 3) N-acetylgalactosaminyl (beta, 1 in equilibrium 4) galactosyl (beta, 1 in equilibrium 4) glucosyl (1 in equilibrium 1) ceramide) and cytidine-5'-monophospho-N-acetylneuraminic acid in the presence of young rat brain sialytransferase has been established. This monosialoganglioside contains a neuraminidase-labile N-acetylneuraminyl group which is linked at position C-3 of the terminal galactosyl unit. This result was obtained by ultramicro scale permethylation of radioactive neuraminidase-labile monosialoganglioside biosynthesized from asialoganglioside labeled with tritium in the terminal galactose.

Function of arginase in lactating mammary gland

The potential for a considerable formation of ornithine exists in lactating mammary gland because of its arginase content. Late in lactation arginase reaches an activity in the gland higher than that present in any rat tissue except liver. Occurrence of the urea cycle can be excluded since two enzymes for the further reaction of ornithine in the cycle, carbamoyl phosphate synthetase I and ornithine carbamoyltransferase, are both absent from this tissue. Instead, carbamoyl phosphate synthetase II appears early in lactation, associated with accumulation of aspartate carbamoyltransferase and DNA, consistent with the proposed role of these enzymes in pyrimidine synthesis. The facts require another physiological role for arginase apart from its known function in the urea cycle. Significant activity of ornithine aminotransferase develops in mammary gland in close parallel with the arginase. By this reaction, ornithine can be converted into glutamic semialdehyde and subsequently into proline. The enzymic composition of the lactating mammary gland is therefore appropriate for the major conversion of arginine into proline that is known to occur in the intact gland.

Frog brain uridine diphosphate galactose-N-acetylgalactosaminyl-N-acetylneuraminylgalactosylglucosylceramide galactosyltransferase

1. The enzyme that catalyses the transfer of galactose from UDP-galactose to N-acetylgalactosaminyl-(1-->4)-N-acetylneuraminyl-(2-->3)-galactosyl-(1-->4)-glucosylceramide (G(M2)) was found mainly in the heavy- and light-microsomal fractions of the adult frog brain. 2. The subcellular distribution of the enzyme, UDP-galactose-G(M2) galactosyltransferase, parallels that of gangliosides in adult frog brain. 3. The enzymic activity was first detected at late gastrulation (Shumway stage 11(1/2)) and increased until the completion of the operculum (Shumway stage 25) and then decreased in the tadpoles. 4. In adult frog brain, the enzyme exhibited a pH optimum of 7.2-7.3 in both cacodylate and tris buffers. The enzyme required 10mm-Mn(2+) for maximal activity and the K(m) for Mn(2+) was determined as 2.2mm. The half-maximal velocity was obtained at a G(M2) concentration of 0.18mm. Inhibition of the enzymic reaction was found when the G(M2) concentration was greater than 1.38mm. 5. The enzymic activity was also inhibited by the products in the pathway of ganglioside synthesis, i.e. either by a mixture of gangliosides or by individual ganglioside components. The most active inhibitor was disialoganglioside. The degree of inhibition is a function of the individual ganglioside concentration. 6. A product-inhibition mechanism for the regulation of ganglioside biosynthesis is discussed.