Shape-aware surface reconstruction from sparse 3D point-clouds

A feasibility study of tumor motion monitoring for SBRT of lung cancer based on 3D point cloud detection and stacking ensemble learning

PURPOSE

To construct a tumor motion monitoring model for stereotactic body radiation therapy (SBRT) of lung cancer from a feasibility perspective.

METHODS

A total of 32 treatment plans for 22 patients were collected, whose planning CT and the centroid position of the planning target volume (PTV) were used as the reference. Images of different respiratory phases in 4DCT were acquired to redefine the targets and obtain the floating PTV centroid positions. In accordance with the planning CT and CBCT registration parameters, data augmentation was accomplished, yielding 2130 experimental recordings for analysis. We employed a stacking multi-learning ensemble approach to fit the 3D point cloud variations of body surface and the change of target position to construct the tumor motion monitoring model, and the prediction accuracy was assess using root mean squared error (RMSE) and R-Square (R2).

RESULTS

The prediction displacement of the stacking ensemble model shows a high degree of agreement with the reference value in each direction. In the first layer of model, the X direction (RMSE =0.019 ∼ 0.145mm, R2 =0.9793∼0.9996) and the Z direction (RMSE = 0.051 ∼ 0.168 mm, R2 = 0.9736∼0.9976) show the best results, while the Y direction ranked behind (RMSE = 0.088 ∼ 0.224 mm, R2 = 0.9553∼ 0.9933). The second layer model summarizes the advantages of unit models of first layer, and RMSE of 0.015 mm, 0.083 mm, 0.041 mm, and R2 of 0.9998, 0.9931, 0.9984 respectively for X, Y, Z were obtained.

CONCLUSIONS

The tumor motion monitoring method for SBRT of lung cancer has potential application of non-ionization, non-invasive, markerless, and real-time.

Using a statistical shape model to estimate the knee landmarks for aligning femurs for femoral finite element models

BACKGROUND AND OBJECTIVE

The BOne Strength (BOS) score is a CT-based tool to assess fracture risk for patients with femoral bone metastases using finite element (FE) models. Until now, the knee joint center (KJC) and centers of the condyles (CoCs) were needed to create the FE model, hence BOS scores of incompletely scanned femurs could not be calculated. In this study, a statistical shape model (SSM) was used to align FE models of femurs with a removed knee anatomy. The aim was to determine the effect of using an SSM with different proximal femur fractions on KJC and CoC locations, and on the BOS score.

METHODS

QCT scans of 117 femurs were used to generate patient-specific FE models of the proximal femur. These models were aligned using the knee joint center (KJC), center of condyles (CoC) and femoral head center. The femurs were artificially shortened by removing 30 %, 50 % or 70 % of the femur. A recently developed SSM was used to reconstruct the distal femur. For each of the femur fractions, the difference between the original and SSM-reconstructed KJC and CoC were determined and the BOS scores were calculated.

RESULTS

Although the individual differences between the original and SSM-reconstructed KJC and CoC location could be large, the effect on the individual BOS scores was limited. The SSM-reconstructed BOS scores were highly correlated to the original BOS scores.

CONCLUSION

Using SSM to align femurs with a removed knee anatomy resulted in varying estimation of knee anatomy between patients but relatively accurate BOS scores.

Scalp surface estimation and head registration using sparse sampling and 3D statistical models

Registering the head and estimating the scalp surface are important for various biomedical procedures, including those using neuronavigation to localize brain stimulation or recording. However, neuronavigation systems rely on manually-identified fiducial head targets and often require a patient-specific MRI for accurate registration, limiting adoption. We propose a practical technique capable of inferring the scalp shape and use it to accurately register the subject's head. Our method does not require anatomical landmark annotation or an individual MRI scan, yet achieves accurate registration of the subject's head and estimation of its surface. The scalp shape is estimated from surface samples easily acquired using existing pointer tools, and registration exploits statistical head model priors. Our method allows for the acquisition of non-trivial shapes from a limited number of data points while leveraging their object class priors, surpassing the accuracy of common reconstruction and registration methods using the same tools. The proposed approach is evaluated in a virtual study with head MRI data from 1152 subjects, achieving an average reconstruction root-mean-square error of 2.95 mm, which outperforms a common neuronavigation technique by 2.70 mm. We also characterize the error under different conditions and provide guidelines for efficient sampling. Furthermore, we demonstrate and validate the proposed method on data from 50 subjects collected with conventional neuronavigation tools and setup, obtaining an average root-mean-square error of 2.89 mm; adding landmark-based registration improves this error to 2.63 mm. The simulation and experimental results support the proposed method's effectiveness with or without landmark annotation, highlighting its broad applicability.

Learning continuous shape priors from sparse data with neural implicit functions

Statistical shape models are an essential tool for various tasks in medical image analysis, including shape generation, reconstruction and classification. Shape models are learned from a population of example shapes, which are typically obtained through segmentation of volumetric medical images. In clinical practice, highly anisotropic volumetric scans with large slice distances are prevalent, e.g., to reduce radiation exposure in CT or image acquisition time in MR imaging. For existing shape modeling approaches, the resolution of the emerging model is limited to the resolution of the training shapes. Therefore, any missing information between slices prohibits existing methods from learning a high-resolution shape prior. We propose a novel shape modeling approach that can be trained on sparse, binary segmentation masks with large slice distances. This is achieved through employing continuous shape representations based on neural implicit functions. After training, our model can reconstruct shapes from various sparse inputs at high target resolutions beyond the resolution of individual training examples. We successfully reconstruct high-resolution shapes from as few as three orthogonal slices. Furthermore, our shape model allows us to embed various sparse segmentation masks into a common, low-dimensional latent space - independent of the acquisition direction, resolution, spacing, and field of view. We show that the emerging latent representation discriminates between healthy and pathological shapes, even when provided with sparse segmentation masks. Lastly, we qualitatively demonstrate that the emerging latent space is smooth and captures characteristic modes of shape variation. We evaluate our shape model on two anatomical structures: the lumbar vertebra and the distal femur, both from publicly available datasets.

Bone surface reconstruction and clinical features estimation from sparse landmarks and Statistical Shape Models: a feasibility study on the femur

In this study, we investigated a method allowing the determination of the femur bone surface as well as its mechanical axis from some easy-to-identify bony landmarks. The reconstruction of the whole femur is therefore performed from these landmarks using a Statistical Shape Model (SSM). The aim of this research is therefore to assess the impact of the number, the position, and the accuracy of the landmarks for the reconstruction of the femur and the determination of its related mechanical axis, an important clinical parameter to consider for the lower limb analysis. Two statistical femur models were created from our in-house dataset and a publicly available dataset. Both were evaluated in terms of average point-to-point surface distance error and through the mechanical axis of the femur. Furthermore, the clinical impact of using landmarks on the skin in replacement of bony landmarks is investigated. The predicted proximal femurs from bony landmarks were more accurate compared to on-skin landmarks while both had less than 3.5^∘ degrees mechanical axis angle deviation error. The results regarding the non-invasive determination of the mechanical axis are very encouraging and could open very interesting clinical perspectives for the analysis of the lower limb either for orthopedics or functional rehabilitation.

A New Simplification Algorithm for Scattered Point Clouds with Feature Preservation

High-precision and high-density three-dimensional point cloud models usually contain redundant data, which implies extra time and hardware costs in the subsequent data processing stage. To analyze and extract data more effectively, the point cloud must be simplified before data processing. Given that point cloud simplification must be sensitive to features to ensure that more valid information can be saved, in this paper, a new simplification algorithm for scattered point clouds with feature preservation, which can reduce the amount of data while retaining the features of data, is proposed. First, the Delaunay neighborhood of the point cloud is constructed, and then the edge points of the point cloud are extracted by the edge distribution characteristics of the point cloud. Second, the moving least-square method is used to obtain the normal vector of the point cloud and the valley ridge points of the model. Then, potential feature points are identified further and retained on the basis of the discrete gradient idea. Finally, non-feature points are extracted. Experimental results show that our method can be applied to models with different curvatures and effectively avoid the hole phenomenon in the simplification process. To further improve the robustness and anti-noise ability of the method, the neighborhood of the point cloud can be extended to multiple levels, and a balance between simplification speed and accuracy needs to be found.

Endoscopic image feature matching via motion consensus and global bilateral regression

BACKGROUND AND OBJECTIVE

Feature matching of endoscopic images is of crucial importance in many clinical applications, such as object tracking and surface reconstruction. However, with the presence of low texture, specular reflections and deformations, the feature matching methods of natural scene are facing great challenges in minimally invasive surgery (MIS) scenarios. We propose a novel motion consensus-based method for endoscopic image feature matching to address these problems.

METHODS

Our method starts by correcting the radial distortion with the spherical projection model and removing the specular reflection regions with an adaptive detection method, which helps to eliminate the image distortion and to reduce the quantity of outliers. We solve the matching problem with a two-stage strategy that progressively estimates a consensus of inliers; the result is a precisely smoothed motion field. First, we construct a spatial motion field from candidate feature matches and estimate its maximum posterior with expectation maximization algorithm, which is computationally efficient and able to obtain smoothed motion field quickly. Second, we extend the smoothed motion field to the affine domain and refine it with bilateral regression to preserve locally subtle motions. The true matches can be identified by checking the difference of feature motion against the estimated field.

RESULTS

Evaluations are implemented on two simulation datasets of deformation (218 images) and four different types of endoscopic datasets (1032 images). Our method is compared with three other state-of-the-art methods and achieves the best performance on affine transformation and nonrigid deformation simulations, with inlier ratio of 86.7% and 94.3%, sensitivity of 90.0% and 96.2%, precision of 88.2% and 93.9%, and F1-Score of 89.1% and 95.0%, respectively. On clinical datasets evaluations, the proposed method achieves an average reprojection error of 3.7 pixels and a consistent performance in multi-image correspondence of sequential images. Furthermore, we also present a surface reconstruction result from rhinoscopic images to validate the reliability of our method, which shows high-quality feature matching results.

CONCLUSIONS

The proposed motion consensus-based feature matching method is proved effective and robust for endoscopic images correspondence. This demonstrates its capability to generate reliable feature matches for surface reconstruction and other meaningful applications in MIS scenarios.

Reconstruction of the lower limb bones from digitised anatomical landmarks using statistical shape modelling

BACKGROUND

Bone shapes strongly influence force and moment predictions of kinematic and musculoskeletal models used in motion analysis. The precise determination of joint reference frames is essential for accurate predictions. Since clinical motion analysis typically does not include medical imaging, from which bone shapes may be obtained, scaling methods using reference subjects to create subject-specific bone geometries are widely used.

RESEARCH QUESTION

This study investigated if lower limb bone shape predictions from skin-based measurements, utilising an underlying statistical shape model (SSM) that corrects for soft tissue artefacts in digitisation, can be used to improve conventional linear scaling methods of bone geometries.

METHODS

SSMs created from 35 healthy adult femurs and tibiae/fibulae were used to reconstruct bone shapes by minimising the distance between anatomical landmarks on the models and those digitised in the motion laboratory or on medical images. Soft tissue artefacts were quantified from magnetic resonance images and then used to predict distances between landmarks digitised on the skin surface and bone. Reconstruction results were compared to linearly scaled models by measuring root mean squared distances to segmented surfaces, calculating differences of commonly used anatomical measures and the errors in the prediction of the hip joint centre.

RESULTS

SSM reconstructed surface predictions from varying landmark sets from skin and bone landmarks were more accurate compared to linear scaling methods (2.60-2.95 mm vs. 3.66-3.87 mm median error; p < 0.05). No significant differences were found between SSM reconstructions from bony landmarks and SSM reconstructions from digitised landmarks obtained in the motion lab and therefore reconstructions using skin landmarks are as accurate as reconstructions from landmarks obtained from medical images.

SIGNIFICANCE

These results indicate that SSM reconstructions can be used to increase the accuracy in obtaining bone shapes from surface digitised experimental data acquired in motion lab environments.

Deformable multimodal registration for navigation in beating-heart cardiac surgery

PURPOSE

Minimally invasive beating-heart surgery is currently performed using endoscopes and without navigation. Registration of intraoperative ultrasound to a preoperative cardiac CT scan is a valuable step toward image-guided navigation.

METHODS

The registration was achieved by first extracting a representative point set from each ultrasound image in the sequence using a deformable registration. A template shape representing the cardiac chambers was deformed through a hierarchy of affine transformations to match each ultrasound image using a generalized expectation maximization algorithm. These extracted point sets were matched to the CT by exhaustively searching over a large number of precomputed slices of 3D geometry. The result is a similarity transformation mapping the intraoperative ultrasound to preoperative CT.

RESULTS

Complete data sets were acquired for four patients. Transesophageal echocardiography ultrasound sequences were deformably registered to a model of oriented points with a mean error of 2.3 mm. Ultrasound and CT scans were registered to a mean of 3 mm, which is comparable to the error of 2.8 mm expected by merging ultrasound registration with uncertainty of cardiac CT.

CONCLUSION

The proposed algorithm registered 3D CT with dynamic 2D intraoperative imaging. The algorithm aligned the images in both space and time, needing neither dynamic CT imaging nor intraoperative electrocardiograms. The accuracy was sufficient for navigation in thoracoscopically guided beating-heart surgery.

Statistical Shape Models: Understanding and Mastering Variation in Anatomy

In our chapter we are describing how to reconstruct three-dimensional anatomy from medical image data and how to build Statistical 3D Shape Models out of many such reconstructions yielding a new kind of anatomy that not only allows quantitative analysis of anatomical variation but also a visual exploration and educational visualization. Future digital anatomy atlases will not only show a static (average) anatomy but also its normal or pathological variation in three or even four dimensions, hence, illustrating growth and/or disease progression.Statistical Shape Models (SSMs) are geometric models that describe a collection of semantically similar objects in a very compact way. SSMs represent an average shape of many three-dimensional objects as well as their variation in shape. The creation of SSMs requires a correspondence mapping, which can be achieved e.g. by parameterization with a respective sampling. If a corresponding parameterization over all shapes can be established, variation between individual shape characteristics can be mathematically investigated.We will explain what Statistical Shape Models are and how they are constructed. Extensions of Statistical Shape Models will be motivated for articulated coupled structures. In addition to shape also the appearance of objects will be integrated into the concept. Appearance is a visual feature independent of shape that depends on observers or imaging techniques. Typical appearances are for instance the color and intensity of a visual surface of an object under particular lighting conditions, or measurements of material properties with computed tomography (CT) or magnetic resonance imaging (MRI). A combination of (articulated) Statistical Shape Models with statistical models of appearance lead to articulated Statistical Shape and Appearance Models (a-SSAMs).After giving various examples of SSMs for human organs, skeletal structures, faces, and bodies, we will shortly describe clinical applications where such models have been successfully employed. Statistical Shape Models are the foundation for the analysis of anatomical cohort data, where characteristic shapes are correlated to demographic or epidemiologic data. SSMs consisting of several thousands of objects offer, in combination with statistical methods or machine learning techniques, the possibility to identify characteristic clusters, thus being the foundation for advanced diagnostic disease scoring.

Automated segmentation of knee bone and cartilage combining statistical shape knowledge and convolutional neural networks: Data from the Osteoarthritis Initiative

We present a method for the automated segmentation of knee bones and cartilage from magnetic resonance imaging (MRI) that combines a priori knowledge of anatomical shape with Convolutional Neural Networks (CNNs). The proposed approach incorporates 3D Statistical Shape Models (SSMs) as well as 2D and 3D CNNs to achieve a robust and accurate segmentation of even highly pathological knee structures. The shape models and neural networks employed are trained using data from the Osteoarthritis Initiative (OAI) and the MICCAI grand challenge "Segmentation of Knee Images 2010" (SKI10), respectively. We evaluate our method on 40 validation and 50 submission datasets from the SKI10 challenge. For the first time, an accuracy equivalent to the inter-observer variability of human readers is achieved in this challenge. Moreover, the quality of the proposed method is thoroughly assessed using various measures for data from the OAI, i.e. 507 manual segmentations of bone and cartilage, and 88 additional manual segmentations of cartilage. Our method yields sub-voxel accuracy for both OAI datasets. We make the 507 manual segmentations as well as our experimental setup publicly available to further aid research in the field of medical image segmentation. In conclusion, combining localized classification via CNNs with statistical anatomical knowledge via SSMs results in a state-of-the-art segmentation method for knee bones and cartilage from MRI data.

Review of 2-D/3-D Reconstruction Using Statistical Shape and Intensity Models and X-Ray Image Synthesis: Toward a Unified Framework

Patient-specific three-dimensional (3-D) bone models are useful for a number of clinical applications such as surgery planning, postoperative evaluation, as well as implant and prosthesis design. Two-dimensional-to-3-D (2-D/3-D) reconstruction, also known as model-to-modality or atlas-based 2-D/3-D registration, provides a means of obtaining a 3-D model of a patient's bones from their 2-D radiographs when 3-D imaging modalities are not available. The preferred approach for estimating both shape and density information (that would be present in a patient's computed tomography data) for 2-D/3-D reconstruction makes use of digitally reconstructed radiographs and deformable models in an iterative, non-rigid, intensity-based approach. Based on a large number of state-of-the-art 2-D/3-D bone reconstruction methods, a unified mathematical formulation of the problem is proposed in a common conceptual framework, using unambiguous terminology. In addition, shortcomings, recent adaptations, and persisting challenges are discussed along with insights for future research.

Weighted regularized statistical shape space projection for breast 3D model reconstruction

The use of 3D imaging has increased as a practical and useful tool for plastic and aesthetic surgery planning. Specifically, the possibility of representing the patient breast anatomy in a 3D shape and simulate aesthetic or plastic procedures is a great tool for communication between surgeon and patient during surgery planning. For the purpose of obtaining the specific 3D model of the breast of a patient, model-based reconstruction methods can be used. In particular, 3D morphable models (3DMM) are a robust and widely used method to perform 3D reconstruction. However, if additional prior information (i.e., known landmarks) is combined with the 3DMM statistical model, shape constraints can be imposed to improve the 3DMM fitting accuracy. In this paper, we present a framework to fit a 3DMM of the breast to two possible inputs: 2D photos and 3D point clouds (scans). Our method consists in a Weighted Regularized (WR) projection into the shape space. The contribution of each point in the 3DMM shape is weighted allowing to assign more relevance to those points that we want to impose as constraints. Our method is applied at multiple stages of the 3D reconstruction process. Firstly, it can be used to obtain a 3DMM initialization from a sparse set of 3D points. Additionally, we embed our method in the 3DMM fitting process in which more reliable or already known 3D points or regions of points, can be weighted in order to preserve their shape information. The proposed method has been tested in two different input settings: scans and 2D pictures assessing both reconstruction frameworks with very positive results.

Research on Biomimetic Models and Nanomechanical Behaviour of Membranous Wings of Chinese Bee Apis cerana cerana Fabricius

The structures combining the veins and membranes of membranous wings of the Chinese bee Apis cerana cerana Fabricius into a whole have excellent load-resisting capacity. The membranous wings of Chinese bees were taken as research objects and the mechanical properties of a biomimetic model of membranous wings as targets. In order to understand and learn from the biosystem and then make technical innovation, the membranous wings of Chinese bees were simulated and analysed with reverse engineering and finite element method. The deformations and stress states of the finite element model of membranous wings were researched under the concentrated force, uniform load, and torque. It was found that the whole model deforms evenly and there are no unusual deformations arising. The displacements and deformations are small and transform uniformly. It was indicated that the veins and membranes combine well into a whole to transmit loads effectively, which illustrates the membranous wings of Chinese bees having excellent integral mechanical behaviour and structure stiffness. The realization of structure models of the membranous wings of Chinese bees and analysis of the relativity of structures and performances or functions will provide an inspiration for designing biomimetic thin-film materials with superior load-bearing capacity.