Towards Fully Autonomous Ultrasound Scanning Robot With Imitation Learning Based on Clinical Protocols

Automatic robotic doppler sonography of leg arteries

PURPOSE

Robot-assisted systems offer an opportunity to support the diagnostic and therapeutic treatment of vascular diseases to reduce radiation exposure and support the limited medical staff in vascular medicine. In the diagnosis and follow-up care of vascular pathologies, Doppler ultrasound has become the preferred diagnostic tool. The study presents a robotic system for automatic Doppler ultrasound examinations of patients' leg vessels.

METHODS

The robotic system consists of a redundant 7 DoF serial manipulator, to which a 3D ultrasound probe is attached. A compliant control was employed, whereby the transducer was guided along the vessel with a defined contact force. Visual servoing was used to correct the position of the probe during the scan so that the vessel can always be properly visualized. To track the vessel's position, methods based on template matching and Doppler sonography were used.

RESULTS

Our system was able to successfully scan the femoral artery of seven volunteers automatically for a distance of 20 cm. In particular, our approach using Doppler ultrasound data showed high robustness and an accuracy of 10.7 (±3.1) px in determining the vessel's position and thus outperformed our template matching approach, whereby an accuracy of 13.9 (±6.4) px was achieved.

CONCLUSIONS

The developed system enables automated robotic ultrasound examinations of vessels and thus represents an opportunity to reduce radiation exposure and staff workload. The integration of Doppler ultrasound improves the accuracy and robustness of vessel tracking, and could thus contribute to the realization of routine robotic vascular examinations and potential endovascular interventions.

Cardiac ultrasound simulation for autonomous ultrasound navigation

Introduction

Ultrasound is well-established as an imaging modality for diagnostic and interventional purposes. However, the image quality varies with operator skills as acquiring and interpreting ultrasound images requires extensive training due to the imaging artefacts, the range of acquisition parameters and the variability of patient anatomies. Automating the image acquisition task could improve acquisition reproducibility and quality but training such an algorithm requires large amounts of navigation data, not saved in routine examinations.

Methods

We propose a method to generate large amounts of ultrasound images from other modalities and from arbitrary positions, such that this pipeline can later be used by learning algorithms for navigation. We present a novel simulation pipeline which uses segmentations from other modalities, an optimized volumetric data representation and GPU-accelerated Monte Carlo path tracing to generate view-dependent and patient-specific ultrasound images.

Results

We extensively validate the correctness of our pipeline with a phantom experiment, where structures' sizes, contrast and speckle noise properties are assessed. Furthermore, we demonstrate its usability to train neural networks for navigation in an echocardiography view classification experiment by generating synthetic images from more than 1,000 patients. Networks pre-trained with our simulations achieve significantly superior performance in settings where large real datasets are not available, especially for under-represented classes.

Discussion

The proposed approach allows for fast and accurate patient-specific ultrasound image generation, and its usability for training networks for navigation-related tasks is demonstrated.

A review of robot-assisted ultrasound examination: Systems and technology

BACKGROUND

At present, the number and overall level of ultrasound (US) doctors cannot meet the medical needs, and the medical ultrasound robots will largely solve the shortage of medical resources.

METHODS

According to the degree of automation, the handheld, semi-automatic and automatic ultrasound examination robot systems are summarised. Ultrasound scanning path planning and robot control are the keys to ensure that the robot systems can obtain high-quality images. Therefore, the ultrasound scanning path planning and control methods are summarised. The research progress and future trends are discussed.

RESULTS

A variety of ultrasound robot systems have been applied to various medical works. With the continuous improvement of automation, the systems provide high-quality ultrasound images and image guidance for clinicians.

CONCLUSION

Although the development of medical ultrasound robot still faces challenges, with the continuous progress of robot technology and communication technology, medical ultrasound robot will have great development potential and broad application space.

Autonomous control of an ultrasound probe for intra-operative ultrasonography using vision-based shape sensing of pneumatically attachable flexible rails

PURPOSE

In robotic-assisted minimally invasive surgery, surgeons often use intra-operative ultrasound to visualise endophytic structures and localise resection margins. This must be performed by a highly skilled surgeon. Automating this subtask may reduce the cognitive load for the surgeon and improve patient outcomes.

METHODS

We demonstrate vision-based shape sensing of the pneumatically attachable flexible (PAF) rail by using colour-dependent image segmentation. The shape-sensing framework is evaluated on known curves ranging from r = 30 to r = 110 mm, replicating curvatures in a human kidney. The shape sensing is then used to inform path planning of a collaborative robot arm paired with an intra-operative ultrasound probe. We execute 15 autonomous ultrasound scans of a tumour-embedded kidney phantom and retrieve viable ultrasound images, as well as seven freehand ultrasound scans for comparison.

RESULTS

The vision-based sensor is shown to have comparable sensing accuracy with FBGS-based systems. We find the RMSE of the vision-based shape sensing of the PAF rail compared with ground truth to be 0.4975 ± 0.4169 mm. The ultrasound images acquired by the robot and by the human were evaluated by two independent clinicians. The median score across all criteria for both readers was '3-good' for human and '4-very good' for robot.

CONCLUSION

We have proposed a framework for autonomous intra-operative US scanning using vision-based shape sensing to inform path planning. Ultrasound images were evaluated by clinicians for sharpness of image, clarity of structures visible, and contrast of solid and fluid areas. Clinicians evaluated that robot-acquired images were superior to human-acquired images in all metrics. Future work will translate the framework to a da Vinci surgical robot.

A fully autonomous robotic ultrasound system for thyroid scanning

The current thyroid ultrasound relies heavily on the experience and skills of the sonographer and the expertise of the radiologist, and the process is physically and cognitively exhausting. In this paper, we report a fully autonomous robotic ultrasound system, which is able to scan thyroid regions without human assistance and identify malignant nod- ules. In this system, human skeleton point recognition, reinforcement learning, and force feedback are used to deal with the difficulties in locating thyroid targets. The orientation of the ultrasound probe is adjusted dynamically via Bayesian optimization. Experimental results on human participants demonstrated that this system can perform high-quality ultrasound scans, close to manual scans obtained by clinicians. Additionally, it has the potential to detect thyroid nodules and provide data on nodule characteristics for American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) calculation.

Robotic ultrasound imaging: State-of-the-art and future perspectives

Ultrasound (US) is one of the most widely used modalities for clinical intervention and diagnosis due to the merits of providing non-invasive, radiation-free, and real-time images. However, free-hand US examinations are highly operator-dependent. Robotic US System (RUSS) aims at overcoming this shortcoming by offering reproducibility, while also aiming at improving dexterity, and intelligent anatomy and disease-aware imaging. In addition to enhancing diagnostic outcomes, RUSS also holds the potential to provide medical interventions for populations suffering from the shortage of experienced sonographers. In this paper, we categorize RUSS as teleoperated or autonomous. Regarding teleoperated RUSS, we summarize their technical developments, and clinical evaluations, respectively. This survey then focuses on the review of recent work on autonomous robotic US imaging. We demonstrate that machine learning and artificial intelligence present the key techniques, which enable intelligent patient and process-specific, motion and deformation-aware robotic image acquisition. We also show that the research on artificial intelligence for autonomous RUSS has directed the research community toward understanding and modeling expert sonographers' semantic reasoning and action. Here, we call this process, the recovery of the "language of sonography". This side result of research on autonomous robotic US acquisitions could be considered as valuable and essential as the progress made in the robotic US examination itself. This article will provide both engineers and clinicians with a comprehensive understanding of RUSS by surveying underlying techniques. Additionally, we present the challenges that the scientific community needs to face in the coming years in order to achieve its ultimate goal of developing intelligent robotic sonographer colleagues. These colleagues are expected to be capable of collaborating with human sonographers in dynamic environments to enhance both diagnostic and intraoperative imaging.

Towards fully automated robotic platform for remote auscultation

BACKGROUND

Since most developed countries are facing an increase in the number of patients per healthcare worker due to a declining birth rate and an ageing population, relatively simple and safe diagnosis tasks may need to be performed using robotics and automation technologies, without specialists and hospitals. This study presents an automated robotic platform for remote auscultation, which is a highly cost-effective screening tool for detecting abnormal clinical signs.

METHOD

The developed robotic platform is composed of a 6-degree-of-freedom cooperative robotic arm, LiDAR camera, and a spring-based mechanism holding an electric stethoscope. The platform enables autonomous stethoscope positioning based on external body information acquired using the LiDAR camera-based multi-way registration; the platform also ensures safe and flexible contact, maintaining the contact force within a certain range through the passive-actuated mechanism.

RESULTS

Our preliminary results confirm that the robotic platform enables estimation of the landing positions required for cardiac examinations based on the depth and landmark information of the body surface. It also handles the stethoscope while maintaining the contact force without relying on the push-in displacement by the robotic arm.

CONCLUSION

The developed robotic platform enables the estimation of the landing positions and handling the stethoscope while maintaining the contact force, which promises the potential of automatic remote auscultation.

A Novel Ultrasound Robot with Force/torque Measurement and Control for Safe and Efficient Scanning

Medical ultrasound is of increasing importance in medical diagnosis and intraoperative assistance and possesses great potential advantages when integrated with robotics. However, some concerns, including the operation efficiency, operation safety, image quality, and comfort of patients, remain after introducing robotics into medical ultrasound. In this paper, an ultrasound robot integrating a force control mechanism, force/torque measurement mechanism, and online adjustment method, is proposed to overcome the current limitations. The ultrasound robot can measure operating forces and torques, provide adjustable constant operating forces, eliminate great operating forces introduced by accidental operations, and achieve various scanning depths based on clinical requirements. The proposed ultrasound robot would potentially facilitate sonographers to find the targets quickly, improve operation safety and efficiency, and decrease patients' discomfort. Simulations and experiments were carried out to evaluate the performance of the ultrasound robot. Experimental results show that the proposed ultrasound robot is able to detect operating force in the z-direction and torques around the x- and y- directions with errors of 3.53% F.S., 6.68% F.S., and 6.11% F.S., respectively, maintain the constant operating force with errors of less than 0.57N, and achieve various scanning depths for target searching and imaging. This proposed ultrasound robot has good performance and would potentially be used in medical ultrasound.

Towards Autonomous Robotic Minimally Invasive Ultrasound Scanning and Vessel Reconstruction on Non-Planar Surfaces

Autonomous robotic Ultrasound (US) scanning has been the subject of research for more than 2 decades. However, little work has been done to apply this concept into a minimally invasive setting, in which accurate force sensing is generally not available and robot kinematics are unreliable due to the tendon-driven, compliant robot structure. As a result, the adequate orientation of the probe towards the tissue surface remains unknown and the anatomy reconstructed from scan may become highly inaccurate. In this work we present solutions to both of these challenges: an attitude sensor fusion scheme for improved kinematic sensing and a visual, deep learning based algorithm to establish and maintain contact between the organ surface and the US probe. We further introduce a novel scheme to estimate and orient the probe perpendicular to the center line of a vascular structure. Our approach enables, for the first time, to autonomously scan across a non-planar surface and navigate along an anatomical structure with a robotically guided minimally invasive US probe. Our experiments on a vessel phantom with a convex surface confirm a significant improvement of the reconstructed curved vessel geometry, with our approach strongly reducing the mean positional error and variance. In the future, our approach could help identify vascular structures more effectively and help pave the way towards semi-autonomous assistance during partial hepatectomy and the potential to reduce procedure length and complication rates.

Remote Ultrasound Scan Procedures with Medical Robots: Towards New Perspectives between Medicine and Engineering

Background

This review explores state-of-the-art teleoperated robots for medical ultrasound scan procedures, providing a comprehensive look including the recent trends arising from the COVID-19 pandemic.

Methods

Physicians' experience is included to indicate the importance of their role in the design of improved medical robots. From this perspective, novel classes of equipment for remote diagnostics based on medical robotics are discussed in terms of innovative engineering technologies.

Results

Relevant literature is reviewed under the system engineering point of view, organizing the discussion on the basis of the main technological focus of each contribution.

Conclusions

This contribution is aimed at stimulating new research to obtain faster results on teleoperated robotics for ultrasound diagnostics in response to the high demand raised by the ongoing pandemic.