Deep Learning-Based Femoral Cartilage Automatic Segmentation in Ultrasound Imaging for Guidance in Robotic Knee Arthroscopy

Ultrasound-based 3D bone modelling in computer assisted orthopedic surgery - a review and future challenges

Computer-assisted orthopedic surgery requires precise representations of bone surfaces. To date, computed tomography constitutes the gold standard, but comes with a number of limitations, including costs, radiation and availability. Ultrasound has potential to become an alternative to computed tomography, yet suffers from low image quality and limited field-of-view. These shortcomings may be addressed by a fully automatic segmentation and model-based completion of 3D bone surfaces from ultrasound images. This survey summarizes the state-of-the-art in this field by introducing employed algorithms, and determining challenges and trends. For segmentation, a clear trend toward machine learning-based algorithms can be observed. For 3D bone model completion however, none of the published methods involve machine learning. Furthermore, data sets and metrics are identified as weak spots in current research, preventing development and evaluation of models that generalize well.

Clinical Applications, Challenges, and Recommendations for Artificial Intelligence in Musculoskeletal and Soft-Tissue Ultrasound: AJR Expert Panel Narrative Review

Artificial intelligence (AI) is increasingly used in clinical practice for musculoskeletal imaging tasks, such as disease diagnosis and image reconstruction. AI applications in musculoskeletal imaging have focused primarily on radiography, CT, and MRI. Although musculoskeletal ultrasound stands to benefit from AI in similar ways, such applications have been relatively underdeveloped. In comparison with other modalities, ultrasound has unique advantages and disadvantages that must be considered in AI algorithm development and clinical translation. Challenges in developing AI for musculoskeletal ultrasound involve both clinical aspects of image acquisition and practical limitations in image processing and annotation. Solutions from other radiology subspecialties (e.g., crowdsourced annotations coordinated by professional societies), along with use cases (most commonly rotator cuff tendon tears and palpable soft-tissue masses), can be applied to musculoskeletal ultrasound to help develop AI. To facilitate creation of high-quality imaging datasets for AI model development, technologists and radiologists should focus on increasing uniformity in musculoskeletal ultrasound performance and increasing annotations of images for specific anatomic regions. This Expert Panel Narrative Review summarizes available evidence regarding AI's potential utility in musculoskeletal ultrasound and challenges facing its development. Recommendations for future AI advancement and clinical translation in musculoskeletal ultrasound are discussed.

Knee Bone Models From Ultrasound

The number of total knee arthroplasties performed worldwide is on the rise. Patient-specific planning and implants may improve surgical outcomes but require 3-D models of the bones involved. Ultrasound (US) may become a cheap and nonharmful imaging modality if the shortcomings of segmentation techniques in terms of automation, accuracy, and robustness are overcome; furthermore, any kind of US-based bone reconstruction must involve some kind of model completion to handle occluded areas, for example, the frontal femur. A fully automatic and robust processing pipeline is proposed, generating full bone models from 3-D freehand US scanning. A convolutional neural network (CNN) is combined with a statistical shape model (SSM) to segment and extrapolate the bone surface. We evaluate the method in vivo on ten subjects, comparing the US-based model to a magnetic resonance imaging (MRI) reference. The partial freehand 3-D record of the femur and tibia bones deviate by 0.7-0.8 mm from the MRI reference. After completion, the full bone model shows an average submillimetric error in the case of the femur and 1.24 mm in the case of the tibia. Processing of the images is performed in real time, and the final model fitting step is computed in less than a minute. It took an average of 22 min for a full record per subject.

Osteo-NeT: An Automated System for Predicting Knee Osteoarthritis from X-ray Images Using Transfer-Learning-Based Neural Networks Approach

Knee osteoarthritis is a challenging problem affecting many adults around the world. There are currently no medications that cure knee osteoarthritis. The only way to control the progression of knee osteoarthritis is early detection. Currently, X-ray imaging is a central technique used for the prediction of osteoarthritis. However, the manual X-ray technique is prone to errors due to the lack of expertise of radiologists. Recent studies have described the use of automated systems based on machine learning for the effective prediction of osteoarthritis from X-ray images. However, most of these techniques still need to achieve higher predictive accuracy to detect osteoarthritis at an early stage. This paper suggests a method with higher predictive accuracy that can be employed in the real world for the early detection of knee osteoarthritis. In this paper, we suggest the use of transfer learning models based on sequential convolutional neural networks (CNNs), Visual Geometry Group 16 (VGG-16), and Residual Neural Network 50 (ResNet-50) for the early detection of osteoarthritis from knee X-ray images. In our analysis, we found that all the suggested models achieved a higher level of predictive accuracy, greater than 90%, in detecting osteoarthritis. However, the best-performing model was the pretrained VGG-16 model, which achieved a training accuracy of 99% and a testing accuracy of 92%.

Deep Learning-Based Medical Images Segmentation of Musculoskeletal Anatomical Structures: A Survey of Bottlenecks and Strategies

By leveraging the recent development of artificial intelligence algorithms, several medical sectors have benefited from using automatic segmentation tools from bioimaging to segment anatomical structures. Segmentation of the musculoskeletal system is key for studying alterations in anatomical tissue and supporting medical interventions. The clinical use of such tools requires an understanding of the proper method for interpreting data and evaluating their performance. The current systematic review aims to present the common bottlenecks for musculoskeletal structures analysis (e.g., small sample size, data inhomogeneity) and the related strategies utilized by different authors. A search was performed using the PUBMED database with the following keywords: deep learning, musculoskeletal system, segmentation. A total of 140 articles published up until February 2022 were obtained and analyzed according to the PRISMA framework in terms of anatomical structures, bioimaging techniques, pre/post-processing operations, training/validation/testing subset creation, network architecture, loss functions, performance indicators and so on. Several common trends emerged from this survey; however, the different methods need to be compared and discussed based on each specific case study (anatomical region, medical imaging acquisition setting, study population, etc.). These findings can be used to guide clinicians (as end users) to better understand the potential benefits and limitations of these tools.

Deep convolutional feature details for better knee disorder diagnoses in magnetic resonance images

Convolutional neural networks (CNNs) applied to magnetic resonance imaging (MRI) have demonstrated their ability in the automatic diagnosis of knee injuries. Despite the promising results, the currently available solutions do not take into account the particular anatomy of knee disorders. Existing works have shown that injuries are localized in small-sized knee regions near the center of MRI scans. Based on such insights, we propose MRPyrNet, a CNN architecture capable of extracting more relevant features from these regions. Our solution is composed of a Feature Pyramid Network with Pyramidal Detail Pooling, and can be plugged into any existing CNN-based diagnostic pipeline. The first module aims to enhance the CNN intermediate features to better detect the small-sized appearance of disorders, while the second one captures such kind of evidence by maintaining its detailed information. An extensive evaluation campaign is conducted to understand in-depth the potential of the proposed solution. The experimental results achieved demonstrate that the application of MRPyrNet to baseline methodologies improves their diagnostic capability, especially in the case of anterior cruciate ligament tear and meniscal tear because of MRPyrNet's ability in exploiting the relevant appearance features of such disorders. Code is available at https://github.com/matteo-dunnhofer/MRPyrNet.

Discussion on the possibility of multi-layer intelligent technologies to achieve the best recover of musculoskeletal injuries: Smart materials, variable structures, and intelligent therapeutic planning

Although intelligent technologies has facilitated the development of precise orthopaedic, simple internal fixation, ligament reconstruction or arthroplasty can only relieve pain of patients in short-term. To achieve the best recover of musculoskeletal injuries, three bottlenecks must be broken through, which includes scientific path planning, bioactive implants and personalized surgical channels building. As scientific surgical path can be planned and built by through AI technology, 4D printing technology can make more bioactive implants be manufactured, and variable structures can establish personalized channels precisely, it is possible to achieve satisfied and effective musculoskeletal injury recovery with the progress of multi-layer intelligent technologies (MLIT).

Validating a Semi-Automated Technique for Segmenting Femoral Articular Cartilage on Ultrasound Images

OBJECTIVE

To validate a semi-automated technique to segment ultrasound-assessed femoral cartilage without compromising segmentation accuracy to a traditional manual segmentation technique in participants with an anterior cruciate ligament injury (ACL).

DESIGN

We recruited 27 participants with a primary unilateral ACL injury at a pre-operative clinic visit. One investigator performed a transverse suprapatellar ultrasound scan with the participant's ACL injured knee in maximum flexion. Three femoral cartilage ultrasound images were recorded. A single expert reader manually segmented the femoral cartilage cross-sectional area in each image. In addition, we created a semi-automatic program to segment the cartilage using a random walker-based method. We quantified the average cartilage thickness and echo-intensity for the manual and semi-automated segmentations. Intraclass correlation coefficients (ICC_2,k) and Bland-Altman plots were used to validate the semi-automated technique to the manual segmentation for assessing average cartilage thickness and echo-intensity. A dice correlation coefficient was used to quantify the overlap between the segmentations created with the semi-automated and manual techniques.

RESULTS

For average cartilage thickness, there was excellent reliability (ICC_2,k = 0.99) and a small mean difference (+0.8%) between the manual and semi-automated segmentations. For average echo-intensity, there was excellent reliability (ICC_2,k = 0.97) and a small mean difference (-2.5%) between the manual and semi-automated segmentations. The average dice correlation coefficient between the manual segmentation and semi-automated segmentation was 0.90, indicating high overlap between techniques.

CONCLUSIONS

Our novel semi-automated segmentation technique is a valid method that requires less technical expertise and time than manual segmentation in patients after ACL injury.

Ultrasound Imaging Offers Promising Alternative to Create 3-D Models for Personalised Auricular Implants

Three-dimensional imaging and advanced manufacturing are being applied in health care research to create novel diagnostic and surgical planning methods, as well as personalised treatments and implants. For ear reconstruction, where a cartilage-shaped implant is embedded underneath the skin to re-create shape and form, volumetric imaging and segmentation processing to capture patient anatomy are particularly challenging. Here, we introduce 3-D ultrasound (US) as an available option for imaging the external ear and underlying auricular cartilage structure, and compare it with computed tomography (CT) and magnetic resonance imaging (MRI) against micro-CT (µCT) as a high-resolution reference (gold standard). US images were segmented to create 3-D models of the auricular cartilage and compared against models generated from µCT to assess accuracy. We found that CT was significantly less accurate than the other methods (root mean square [RMS]: 1.30 ± 0.5 mm) and had the least contrast between tissues. There was no significant difference between MRI (RMS: 0.69 ± 0.2 mm) and US (0.55 ± 0.1 mm). US was also the least expensive imaging method at half the cost of MRI. These results unveil a novel use of ultrasound imaging that has not been presented before, as well as support its more widespread use in biofabrication as a low-cost imaging technique to create patient-specific 3D models and implants.

Deep learning segmentation of transverse musculoskeletal ultrasound images for neuromuscular disease assessment

Ultrasound imaging is a patient-friendly and robust technique for studying physiological and pathological muscles. An automatic deep learning (DL) system for the analysis of ultrasound images could be useful to support an expert operator, allowing the study of large datasets requiring less human interaction. The purpose of this study is to present a deep learning algorithm for the cross-sectional area (CSA) segmentation in transverse musculoskeletal ultrasound images, providing a quantitative grayscale analysis which is useful for studying muscles, and to validate the results in a large dataset. The dataset included 3917 images of biceps brachii, tibialis anterior and gastrocnemius medialis acquired on 1283 subjects (mean age 50 ± 21 years, 729 male). The algorithm was based on multiple deep-learning architectures, and its performance was compared to a manual expert segmentation. We compared the mean grayscale value inside the automatic and manual CSA using Bland-Altman plots and a correlation analysis. Classification in healthy and abnormal muscles between automatic and manual segmentation were compared using the grayscale value z-scores. In the test set, a Precision of 0.88 ± 0.12 and a Recall of 0.92 ± 0.09 was achieved. The network segmentation performance was slightly less in abnormal muscles, without a loss of discrimination between healthy and abnormal muscle images. Bland-Altman plots showed no clear trend in the error distribution and the two readings have a 0.99 Pearson's correlation coefficient (p < 0.001, test set). The ICC(A, 1) calculated between the z-score readings was 0.99. The algorithm achieves robust CSA segmentation performance and gives mean grayscale level information comparable to a manual operator. This could provide a helpful tool for clinicians in neuromuscular disease diagnosis and follow-up. The entire dataset and code are made available for the research community.

Development and Validation of a Deep Learning Model to Quantify Interstitial Fibrosis and Tubular Atrophy From Kidney Ultrasonography Images

IMPORTANCE

Interstitial fibrosis and tubular atrophy (IFTA) is a strong indicator of decline in kidney function and is measured using histopathological assessment of kidney biopsy core. At present, a noninvasive test to assess IFTA is not available.

OBJECTIVE

To develop and validate a deep learning (DL) algorithm to quantify IFTA from kidney ultrasonography images.

DESIGN, SETTING, AND PARTICIPANTS

This was a single-center diagnostic study of consecutive patients who underwent native kidney biopsy at John H. Stroger Jr. Hospital of Cook County, Chicago, Illinois, between January 1, 2014, and December 31, 2018. A DL algorithm was trained, validated, and tested to classify IFTA from kidney ultrasonography images. Of 6135 Crimmins-filtered ultrasonography images, 5523 were used for training (5122 images) and validation (401 images), and 612 were used to test the accuracy of the DL system. Kidney segmentation was performed using the UNet architecture, and classification was performed using a convolution neural network-based feature extractor and extreme gradient boosting. IFTA scored by a nephropathologist on trichrome stained kidney biopsy slide was used as the reference standard. IFTA was divided into 4 grades (grade 1, 0%-24%; grade 2, 25%-49%; grade 3, 50%-74%; and grade 4, 75%-100%). Data analysis was performed from December 2019 to May 2020.

MAIN OUTCOMES AND MEASURES

Prediction of IFTA grade was measured using the metrics precision, recall, accuracy, and F1 score.

RESULTS

This study included 352 patients (mean [SD] age 47.43 [14.37] years), of whom 193 (54.82%) were women. There were 159 patients with IFTA grade 1 (2701 ultrasonography images), 74 patients with IFTA grade 2 (1239 ultrasonography images), 41 patients with IFTA grade 3 (701 ultrasonography images), and 78 patients with IFTA grade 4 (1494 ultrasonography images). Kidney ultrasonography images were segmented with 91% accuracy. In the independent test set, the point estimates for performance matrices showed precision of 0.8927 (95% CI, 0.8682-0.9172), recall of 0.8037 (95% CI, 0.7722-0.8352), accuracy of 0.8675 (95% CI, 0.8406-0.8944), and an F1 score of 0.8389 (95% CI, 0.8098-0.8680) at the image level. Corresponding estimates at the patient level were precision of 0.9003 (95% CI, 0.8644-0.9362), recall of 0.8421 (95% CI, 0.7984-0.8858), accuracy of 0.8955 (95% CI, 0.8589-0.9321), and an F1 score of 0.8639 (95% CI, 0.8228-0.9049). Accuracy at the patient level was highest for IFTA grade 1 and IFTA grade 4. The accuracy (approximately 90%) remained high irrespective of the timing of ultrasonography studies and the biopsy diagnosis. The predictive performance of the DL system did not show significant improvement when combined with baseline clinical characteristics.

CONCLUSIONS AND RELEVANCE

These findings suggest that a DL algorithm can accurately and independently predict IFTA from kidney ultrasonography images.

Artificial intelligence in musculoskeletal ultrasound imaging

Ultrasonography (US) is noninvasive and offers real-time, low-cost, and portable imaging that facilitates the rapid and dynamic assessment of musculoskeletal components. Significant technological improvements have contributed to the increasing adoption of US for musculoskeletal assessments, as artificial intelligence (AI)-based computer-aided detection and computer-aided diagnosis are being utilized to improve the quality, efficiency, and cost of US imaging. This review provides an overview of classical machine learning techniques and modern deep learning approaches for musculoskeletal US, with a focus on the key categories of detection and diagnosis of musculoskeletal disorders, predictive analysis with classification and regression, and automated image segmentation. Moreover, we outline challenges and a range of opportunities for AI in musculoskeletal US practice.

Vessel membrane segmentation and calcification location in intravascular ultrasound images using a region detector and an effective selection strategy

BACKGROUND AND OBJECTIVE

Segmenting vessel membranes and locating the calcific region in intravascular ultrasound (IVUS) images aid physicians in the diagnosis of atherosclerosis. However, the manual extraction of the media adventitia (MA)/lumen border and calcification location are cumbersome due to the excessive number of IVUS frames. Moreover, most existing (semi-)automatic detection methods cannot achieve both vessel membrane extraction and calcification location simultaneously, and they are unable to detect vessel membranes in IVUS frames from different acquisition systems.

METHOD

A fully automatic approach is proposed based on extremal regions and a flexible selection strategy to extract vessel membranes in different IVUS frames and locate the calcific region in high-frequency ones. Three main steps are included in the algorithm. First, a region detector is employed to extract extremal regions from an IVUS image. Then, according to the selection strategy, a part of the extracted regions is selected. At the same time, the calcification is located according to its special acoustic properties. Next, approximate MA and lumen border segmentation is achieved based on the selected extremal regions and the located calcification in polar coordinates. Finally, the final segmentation results are obtained by smoothing the approximate values.

RESULT

To demonstrate the feasibility of the method, it was evaluated based on a standard public dataset. Furthermore, to quantitatively evaluate the segmentation performance, the Hausdorff distance (HD), Jaccard measure (JM) and percentage of area difference (PAD) were used. The results show that a mean HD of 1.13/1.21 mm, a mean JM of 0.83/0.77 and a mean PAD of 0.11/0.23 are achieved for MA/lumen border detection in 77 40-MHz IVUS images. For MA/lumen border extraction in 435 20-MHz IVUS frames, the average HD, JM and PAD values are 0.47/0.28 mm, 0.84/0.89 and 0.13/0.10, respectively. In addition, the approach successfully achieves calcification location in 40-MHz IVUS frames. In comparison with other published methods, the method proposed in this study is competitive.

CONCLUSION

According to these results, our strategy can extract MA/lumen borders in different IVUS frames and effectively locate calcification in high-frequency IVUS frames.