Autonomous Scanning Target Localization for Robotic Lung Ultrasound Imaging

A review of robot-assisted ultrasound examination: Systems and technology

BACKGROUND

At present, the number and overall level of ultrasound (US) doctors cannot meet the medical needs, and the medical ultrasound robots will largely solve the shortage of medical resources.

METHODS

According to the degree of automation, the handheld, semi-automatic and automatic ultrasound examination robot systems are summarised. Ultrasound scanning path planning and robot control are the keys to ensure that the robot systems can obtain high-quality images. Therefore, the ultrasound scanning path planning and control methods are summarised. The research progress and future trends are discussed.

RESULTS

A variety of ultrasound robot systems have been applied to various medical works. With the continuous improvement of automation, the systems provide high-quality ultrasound images and image guidance for clinicians.

CONCLUSION

Although the development of medical ultrasound robot still faces challenges, with the continuous progress of robot technology and communication technology, medical ultrasound robot will have great development potential and broad application space.

A fully autonomous robotic ultrasound system for thyroid scanning

The current thyroid ultrasound relies heavily on the experience and skills of the sonographer and the expertise of the radiologist, and the process is physically and cognitively exhausting. In this paper, we report a fully autonomous robotic ultrasound system, which is able to scan thyroid regions without human assistance and identify malignant nod- ules. In this system, human skeleton point recognition, reinforcement learning, and force feedback are used to deal with the difficulties in locating thyroid targets. The orientation of the ultrasound probe is adjusted dynamically via Bayesian optimization. Experimental results on human participants demonstrated that this system can perform high-quality ultrasound scans, close to manual scans obtained by clinicians. Additionally, it has the potential to detect thyroid nodules and provide data on nodule characteristics for American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) calculation.

Autonomous ultrasound scanning robotic system based on human posture recognition and image servo control: an application for cardiac imaging

In traditional cardiac ultrasound diagnostics, the process of planning scanning paths and adjusting the ultrasound window relies solely on the experience and intuition of the physician, a method that not only affects the efficiency and quality of cardiac imaging but also increases the workload for physicians. To overcome these challenges, this study introduces a robotic system designed for autonomous cardiac ultrasound scanning, with the goal of advancing both the degree of automation and the quality of imaging in cardiac ultrasound examinations. The system achieves autonomous functionality through two key stages: initially, in the autonomous path planning stage, it utilizes a camera posture adjustment method based on the human body's central region and its planar normal vectors to achieve automatic adjustment of the camera's positioning angle; precise segmentation of the human body point cloud is accomplished through efficient point cloud processing techniques, and precise localization of the region of interest (ROI) based on keypoints of the human body. Furthermore, by applying isometric path slicing and B-spline curve fitting techniques, it independently plans the scanning path and the initial position of the probe. Subsequently, in the autonomous scanning stage, an innovative servo control strategy based on cardiac image edge correction is introduced to optimize the quality of the cardiac ultrasound window, integrating position compensation through admittance control to enhance the stability of autonomous cardiac ultrasound imaging, thereby obtaining a detailed view of the heart's structure and function. A series of experimental validations on human and cardiac models have assessed the system's effectiveness and precision in the correction of camera pose, planning of scanning paths, and control of cardiac ultrasound imaging quality, demonstrating its significant potential for clinical ultrasound scanning applications.

Rib region detection for scanning path planning for fully automated robotic abdominal ultrasonography

PURPOSE

Scanning path planning is an essential technology for fully automated ultrasound (US) robotics. During biliary scanning, the subcostal boundary is critical body surface landmarks for scanning path planning but are often invisible, depending on the individual. This study developed a method of estimating the rib region for scanning path planning toward fully automated robotic US systems.

METHODS

We proposed a method for determining the rib region using RGB-D images and respiratory variation. We hypothesized that detecting the rib region would be possible based on changes in body surface position due to breathing. We generated a depth difference image by finding the difference between the depth image taken at the resting inspiratory position and the depth image taken at the maximum inspiratory position, which clearly shows the rib region. The boundary position of the subcostal was then determined by applying training using the YOLOv5 object detection model to this depth difference image.

RESULTS

In the experiments with healthy subjects, the proposed method of rib detection using the depth difference image marked an intersection over union (IoU) of 0.951 and average confidence of 0.77. The average error between the ground truth and predicted positions was 16.5 mm in 3D space. The results were superior to rib detection using only the RGB image.

CONCLUSION

The proposed depth difference imaging method, which measures respiratory variation, was able to accurately estimate the rib region without contact and physician intervention. It will be useful for planning the scan path during the biliary imaging.

Development and preliminary testing of a prior knowledge-based visual navigation system for cardiac ultrasound scanning

Purpose

Ultrasound is widely used to diagnose disease and guide surgery because it is versatile, inexpensive and radiation-free. However, image acquisition is dependent on the operation of a professional sonographer, which is a difficult skill to learn for a wider range of non-sonographers.

Methods

We propose a prior knowledge-based visual navigation method to obtain three important standard ultrasound views of the heart, based on the sonographer's skill learning and augmented reality prompts. The key information about the probe movement was captured using vision-based tracking and normalisation methods on 14 volunteers, based on a professional sonographer's practice. An augmented reality-based navigation method was then proposed to guide operators with no ultrasound experience to find standard views of the heart in a second set of three volunteers.

Results

Through quantitative analysis and qualitative scoring, the results showed that the proposed method can effectively guide non-sonographers to obtain standard views with diagnostic value.

Conclusion

It is believed that the method proposed in this paper has clear application value in primary care, and expansion of the data will allow the accuracy of the navigation to be further improved.

Fully Automatic Dual-Probe Lung Ultrasound Scanning Robot for Screening Triage

Two-dimensional lung ultrasound (LUS) has widely emerged as a rapid and noninvasive imaging tool for the detection and diagnosis of coronavirus disease 2019 (COVID-19). However, image differences will be magnified due to changes in ultrasound (US) imaging experience, such as US probe attitude control and force control, which will directly affect the diagnosis results. In addition, the risk of virus transmission between sonographer and patients is increased due to frequent physical contact. In this study, a fully automatic dual-probe US scanning robot for the acquisition of LUS images is proposed and developed. Furthermore, the trajectory was optimized based on the velocity look-ahead strategy, the stability of contact force of the system and the scanning efficiency were improved by 24.13% and 29.46%, respectively. Also, the control ability of the contact force of robotic automatic scanning was 34.14 times higher than that of traditional manual scanning, which significantly improves the smoothness of scanning. Importantly, there was no significant difference in image quality obtained by robotic automatic scanning and manual scanning. Furthermore, the scanning time for a single person is less than 4 min, which greatly improves the efficiency of screening triage of group COVID-19 diagnosis and suspected patients and reduces the risk of virus exposure and spread.

Visual Perception and Convolutional Neural Network-Based Robotic Autonomous Lung Ultrasound Scanning Localization System

Under the situation of severe COVID-19 epidemic, lung ultrasound (LUS) has been proved to be an effective and convenient method to diagnose and evaluate the extent of respiratory disease. However, the traditional clinical ultrasound (US) scanning requires doctors not only to be in close contact with patients but also to have rich experience. In order to alleviate the shortage of medical resources and reduce the work stress and risk of infection for doctors, we propose a visual perception and convolutional neural network (CNN)-based robotic autonomous LUS scanning localization system to realize scanned target recognition, probe pose solution and movement, and the acquisition of US images. The LUS scanned targets are identified through the target segmentation and localization algorithm based on the improved CNN, which is using the depth camera to collect the image information; furthermore, the method based on multiscale compensation normal vector is used to solve the attitude of the probe; finally, a position control strategy based on force feedback is designed to optimize the position and attitude of the probe, which can not only obtain high-quality US images but also ensure the safety of patients and the system. The results of human LUS scanning experiment verify the accuracy and feasibility of the system. The positioning accuracy of the scanned targets is 15.63 ± 0.18 mm, and the distance accuracy and rotation angle accuracy of the probe position calculation are 6.38 ± 0.25 mm and 8.60° ±2.29° , respectively. More importantly, the obtained high-quality US images can clearly capture the main pathological features of the lung. The system is expected to be applied in clinical practice.

A-SEE: Active-Sensing End-effector Enabled Probe Self-Normal-Positioning for Robotic Ultrasound Imaging Applications

Conventional manual ultrasound (US) imaging is a physically demanding procedure for sonographers. A robotic US system (RUSS) has the potential to overcome this limitation by automating and standardizing the imaging procedure. It also extends ultrasound accessibility in resource-limited environments with the shortage of human operators by enabling remote diagnosis. During imaging, keeping the US probe normal to the skin surface largely benefits the US image quality. However, an autonomous, real-time, low-cost method to align the probe towards the direction orthogonal to the skin surface without pre-operative information is absent in RUSS. We propose a novel end-effector design to achieve self-normal-positioning of the US probe. The end-effector embeds four laser distance sensors to estimate the desired rotation towards the normal direction. We then integrate the proposed end-effector with a RUSS system which allows the probe to be automatically and dynamically kept to normal direction during US imaging. We evaluated the normal positioning accuracy and the US image quality using a flat surface phantom, an upper torso mannequin, and a lung ultrasound phantom. Results show that the normal positioning accuracy is 4.17 ± 2.24 degrees on the flat surface and 14.67 ± 8.46 degrees on the mannequin. The quality of the RUSS collected US images from the lung ultrasound phantom was equivalent to that of the manually collected ones.