Automatic landmark detection and mapping for 2D/3D registration with BoneNet

A deep learning-based multi-view approach to automatic 3D landmarking and deformity assessment of lower limb

Anatomical Landmark detection in CT-Scan images is widely used in the identification of skeletal disorders. However, the traditional process of manually detecting anatomical landmarks, especially in three dimensions, is both time-consuming and prone to human errors. We propose a novel, deep-learning-based approach to automatic detection of 3D landmarks in CT images of the lower limb. We generate multiple view renderings of the scanned limb and then integrate them, using a pyramid-style convolutional neural network, to build a 3D model of the bone and to determine the spatial coordinates of the landmarks. Those landmarks are then used to calculate key anatomical indicators that would enable the reliable diagnosis of skeletal disorders. To evaluate the performance of the proposed approach we compare its predicted landmark coordinates and resulting anatomical indicators (both 2D and 3D) with those determined by human experts. The average coordinate error (difference between automatically and manually determined coordinates) of the landmarks was 2.05 ± 1.36 mm on test data, whereas the average angular error (difference between automatically and manually calculated angles in three and two dimensions) on the same dataset was 0.53 ± 0.66° and 0.74 ± 0.87°, respectively. Our proposed deep-learning-based approach not only outperforms the traditional landmark detection and indicator assessment methods in terms of speed and accuracy but also improves the credibility of the ensuing diagnoses by avoiding manual landmarking errors.

Automated Landmark Annotation for Morphometric Analysis of Distal Femur and Proximal Tibia

Manual anatomical landmarking for morphometric knee bone characterization in orthopedics is highly time-consuming and shows high operator variability. Therefore, automation could be a substantial improvement for diagnostics and personalized treatments relying on landmark-based methods. Applications include implant sizing and planning, meniscal allograft sizing, and morphological risk factor assessment. For twenty MRI-based 3D bone and cartilage models, anatomical landmarks were manually applied by three experts, and morphometric measurements for 3D characterization of the distal femur and proximal tibia were calculated from all observations. One expert performed the landmark annotations three times. Intra- and inter-observer variations were assessed for landmark position and measurements. The mean of the three expert annotations served as the ground truth. Next, automated landmark annotation was performed by elastic deformation of a template shape, followed by landmark optimization at extreme positions (highest/lowest/most medial/lateral point). The results of our automated annotation method were compared with ground truth, and percentages of landmarks and measurements adhering to different tolerances were calculated. Reliability was evaluated by the intraclass correlation coefficient (ICC). For the manual annotations, the inter-observer absolute difference was 1.53 ± 1.22 mm (mean ± SD) for the landmark positions and 0.56 ± 0.55 mm (mean ± SD) for the morphometric measurements. Automated versus manual landmark extraction differed by an average of 2.05 mm. The automated measurements demonstrated an absolute difference of 0.78 ± 0.60 mm (mean ± SD) from their manual counterparts. Overall, 92% of the automated landmarks were within 4 mm of the expert mean position, and 95% of all morphometric measurements were within 2 mm of the expert mean measurements. The ICC (manual versus automated) for automated morphometric measurements was between 0.926 and 1. Manual annotations required on average 18 min of operator interaction time, while automated annotations only needed 7 min of operator-independent computing time. Considering the time consumption and variability among observers, there is a clear need for a more efficient, standardized, and operator-independent algorithm. Our automated method demonstrated excellent accuracy and reliability for landmark positioning and morphometric measurements. Above all, this automated method will lead to a faster, scalable, and operator-independent morphometric analysis of the knee.

Enhancing the depth perception of DSA images with 2D-3D registration

Objective

Today, cerebrovascular disease has become an important health hazard. Therefore, it is necessary to perform a more accurate and less time-consuming registration of preoperative three-dimensional (3D) images and intraoperative two-dimensional (2D) projection images which is very important for conducting cerebrovascular disease interventions. The 2D-3D registration method proposed in this study is designed to solve the problems of long registration time and large registration errors in 3D computed tomography angiography (CTA) images and 2D digital subtraction angiography (DSA) images.

Methods

To make a more comprehensive and active diagnosis, treatment and surgery plan for patients with cerebrovascular diseases, we propose a weighted similarity measure function, the normalized mutual information-gradient difference (NMG), which can evaluate the 2D-3D registration results. Then, using a multi-resolution fusion optimization strategy, the multi-resolution fused regular step gradient descent optimization (MR-RSGD) method is presented to attain the optimal value of the registration results in the process of the optimization algorithm.

Result

In this study, we adopt two datasets of the brain vessels to validate and obtain similarity metric values which are 0.0037 and 0.0003, respectively. Using the registration method proposed in this study, the time taken for the experiment was calculated to be 56.55s and 50.8070s, respectively, for the two sets of data. The results show that the registration methods proposed in this study are both better than the Normalized Mutual (NM) and Normalized Mutual Information (NMI).

Conclusion

The experimental results in this study show that in the 2D-3D registration process, to evaluate the registration results more accurately, we can use the similarity metric function containing the image gray information and spatial information. To improve the efficiency of the registration process, we can choose the algorithm with gradient optimization strategy. Our method has great potential to be applied in practical interventional treatment for intuitive 3D navigation.