Novel clinical device tracking and tissue event characterization using proximally placed audio signal acquisition and processing

Vibro-acoustic sensing of tissue-instrument-interactions allows a differentiation of biological tissue in computerised palpation

BACKGROUND

The shift towards minimally invasive surgery is associated with a significant reduction of tactile information available to the surgeon, with compensation strategies ranging from vision-based techniques to the integration of sensing concepts into surgical instruments. Tactile information is vital for palpation tasks such as the differentiation of tissues or the characterisation of surfaces. This work investigates a new sensing approach to derive palpation-related information from vibration signals originating from instrument-tissue-interactions.

METHODS

We conducted a feasibility study to differentiate three non-animal and three animal tissue specimens based on palpation of the surface. A sensor configuration was mounted at the proximal end of a standard instrument opposite the tissue-interaction point. Vibro-acoustic signals of 1680 palpation events were acquired, and the time-varying spectrum was computed using Continuous-Wavelet-Transformation. For validation, nine spectral energy-related features were calculated for a subsequent classification using linear Support Vector Machine and k-Nearest-Neighbor.

RESULTS

Indicators derived from the vibration signal are highly stable in a set of palpations belonging to the same tissue specimen, regardless of the palpating subject. Differences in the surface texture of the tissue specimens reflect in those indicators and can serve as a basis for differentiation. The classification following a supervised learning approach shows an accuracy of >93.8% for the three-tissue classification tasks and decreases to 78.8% for a combination of all six tissues.

CONCLUSIONS

Simple features derived from the vibro-acoustic signals facilitate the differentiation between biological tissues, showing the potential of the presented approach to provide information related to the interacting tissue. The results encourage further investigation of a yet little-exploited source of information in minimally invasive surgery.

Vibro-Acoustic Sensing of Instrument Interactions as a Potential Source of Texture-Related Information in Robotic Palpation

The direct tactile assessment of surface textures during palpation is an essential component of open surgery that is impeded in minimally invasive and robot-assisted surgery. When indirectly palpating with a surgical instrument, the structural vibrations from this interaction contain tactile information that can be extracted and analysed. This study investigates the influence of the parameters contact angle α and velocity v→ on the vibro-acoustic signals from this indirect palpation. A 7-DOF robotic arm, a standard surgical instrument, and a vibration measurement system were used to palpate three different materials with varying α and v→. The signals were processed based on continuous wavelet transformation. They showed material-specific signatures in the time-frequency domain that retained their general characteristic for varying α and v→. Energy-related and statistical features were extracted, and supervised classification was performed, where the testing data comprised only signals acquired with different palpation parameters than for training data. The classifiers support vector machine and k-nearest neighbours provided 99.67% and 96.00% accuracy for the differentiation of the materials. The results indicate the robustness of the features against variations in the palpation parameters. This is a prerequisite for an application in minimally invasive surgery but needs to be confirmed in realistic experiments with biological tissues.

Surgeons' requirements for a surgical support system to improve laparoscopic access

Creating surgical access is a critical step in laparoscopic surgery. Surgeons have to insert a sharp instrument such as the Veress needle or a trocar into the patient’s abdomen until the peritoneal cavity is reached. They solely rely on their experience and distorted tactile feedback in that process, leading to a complication rate as high as 14% of all cases. Recent studies have shown the feasibility of surgical support systems that provide intraoperative feedback regarding the insertion process to improve laparoscopic access outcomes. However, to date, the surgeons’ requirements for such support systems remain unclear. This research article presents the results of an explorative study that aimed to acquire data about the information that helps surgeons improve laparoscopic access outcomes. The results indicate that feedback regarding the reaching of the peritoneal cavity is of significant importance and should be presented visually or acoustically. Finally, a solution should be straightforward and intuitive to use, should support or even improve the clinical workflow, but also cheap enough to facilitate its usage rate. While this study was tailored to laparoscopic access, its results also apply to other minimally invasive procedures.

The Advances in Computer Vision That Are Enabling More Autonomous Actions in Surgery: A Systematic Review of the Literature

This is a review focused on advances and current limitations of computer vision (CV) and how CV can help us obtain to more autonomous actions in surgery. It is a follow-up article to one that we previously published in Sensors entitled, “Artificial Intelligence Surgery: How Do We Get to Autonomous Actions in Surgery?” As opposed to that article that also discussed issues of machine learning, deep learning and natural language processing, this review will delve deeper into the field of CV. Additionally, non-visual forms of data that can aid computerized robots in the performance of more autonomous actions, such as instrument priors and audio haptics, will also be highlighted. Furthermore, the current existential crisis for surgeons, endoscopists and interventional radiologists regarding more autonomy during procedures will be discussed. In summary, this paper will discuss how to harness the power of CV to keep doctors who do interventions in the loop.

Study of needle punctures into soft tissue through audio and force sensing: can audio be a simple alternative for needle guidance?

PURPOSE

Percutaneous needle insertion is one of the most common minimally invasive procedures. The clinician's experience and medical imaging support are essential to the procedure's safety. However, imaging comes with inaccuracies due to artifacts, and therefore sensor-based solutions were proposed to improve accuracy. However, sensors are usually embedded in the needle tip, leading to design limitations. A novel concept was proposed for capturing tip-tissue interaction information through audio sensing, showing promising results for needle guidance. This work demonstrates that this audio approach can provide important puncture information by comparing audio and force signal dynamics during insertion.

METHODS

An experimental setup for inserting a needle into soft tissue was prepared. Audio and force signals were synchronously recorded at four different insertion velocities, and a dataset of 200 recordings was acquired. Indicators related to different aspects of the force and audio were compared through signal-to-signal and event-to-event correlation analysis.

RESULTS

High signal-to-signal correlations between force and audio indicators regardless of the insertion velocity were obtained. The force curvature indicator obtained the best correlation performances to audio with more than [Formula: see text] of the correlations higher than 0.6. The event-to-event correlation analysis shows that a puncture event in the force is generally identifiable in audio and that their intensities firmly related.

CONCLUSIONS

Audio contains valuable information for monitoring needle tip/tissue interaction. Significant dynamics obtained from a well-known sensor as force can also be extracted from audio, regardless of insertion velocities.

Artificial Intelligence Surgery: How Do We Get to Autonomous Actions in Surgery?

Most surgeons are skeptical as to the feasibility of autonomous actions in surgery. Interestingly, many examples of autonomous actions already exist and have been around for years. Since the beginning of this millennium, the field of artificial intelligence (AI) has grown exponentially with the development of machine learning (ML), deep learning (DL), computer vision (CV) and natural language processing (NLP). All of these facets of AI will be fundamental to the development of more autonomous actions in surgery, unfortunately, only a limited number of surgeons have or seek expertise in this rapidly evolving field. As opposed to AI in medicine, AI surgery (AIS) involves autonomous movements. Fortuitously, as the field of robotics in surgery has improved, more surgeons are becoming interested in technology and the potential of autonomous actions in procedures such as interventional radiology, endoscopy and surgery. The lack of haptics, or the sensation of touch, has hindered the wider adoption of robotics by many surgeons; however, now that the true potential of robotics can be comprehended, the embracing of AI by the surgical community is more important than ever before. Although current complete surgical systems are mainly only examples of tele-manipulation, for surgeons to get to more autonomously functioning robots, haptics is perhaps not the most important aspect. If the goal is for robots to ultimately become more and more independent, perhaps research should not focus on the concept of haptics as it is perceived by humans, and the focus should be on haptics as it is perceived by robots/computers. This article will discuss aspects of ML, DL, CV and NLP as they pertain to the modern practice of surgery, with a focus on current AI issues and advances that will enable us to get to more autonomous actions in surgery. Ultimately, there may be a paradigm shift that needs to occur in the surgical community as more surgeons with expertise in AI may be needed to fully unlock the potential of AIS in a safe, efficacious and timely manner.

Real-time acoustic sensing and artificial intelligence for error prevention in orthopedic surgery

In this work, we developed and validated a computer method capable of robustly detecting drill breakthrough events and show the potential of deep learning-based acoustic sensing for surgical error prevention. Bone drilling is an essential part of orthopedic surgery and has a high risk of injuring vital structures when over-drilling into adjacent soft tissue. We acquired a dataset consisting of structure-borne audio recordings of drill breakthrough sequences with custom piezo contact microphones in an experimental setup using six human cadaveric hip specimens. In the following step, we developed a deep learning-based method for the automated detection of drill breakthrough events in a fast and accurate fashion. We evaluated the proposed network regarding breakthrough detection sensitivity and latency. The best performing variant yields a sensitivity of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$93.64 \pm 2.42$$\end{document}93.64±2.42% for drill breakthrough detection in a total execution time of 139.29\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hbox { ms}}$$\end{document}ms. The validation and performance evaluation of our solution demonstrates promising results for surgical error prevention by automated acoustic-based drill breakthrough detection in a realistic experiment while being multiple times faster than a surgeon’s reaction time. Furthermore, our proposed method represents an important step for the translation of acoustic-based breakthrough detection towards surgical use.

Auscultation System for Acquisition of Vascular Sounds - Towards Sound-Based Monitoring of the Carotid Artery

Introduction

Atherosclerotic diseases of the carotid are a primary cause of cerebrovascular events such as stroke. For the diagnosis and monitoring angiography, ultrasound- or magnetic resonance-based imaging is used which requires costly hardware. In contrast, the auscultation of carotid sounds and screening for bruits - audible patterns related to turbulent blood flow - is a simple examination with comparably little technical demands. It can indicate atherosclerotic diseases and justify further diagnostics but is currently subjective and examiner dependent.

Methods

We propose an easy-to-use computer-assisted auscultation system for a stable and reproducible acquisition of vascular sounds of the carotid. A dedicated skin-transducer-interface was incorporated into a handheld device. The interface comprises two bell-shaped structures, one with additional acoustic membrane, to ensure defined skin contact and a stable propagation path of the sound. The device is connected wirelessly to a desktop application allowing real-time visualization, assessment of signal quality and input of supplementary information along with storage of recordings in a database. An experimental study with 5 healthy subjects was conducted to evaluate usability and stability of the device. Five recordings per carotid served as data basis for a wavelet-based analysis of the stability of spectral characteristics of the recordings.

Results

The energy distribution of the wavelet-based stationary spectra proved stable for measurements of a particular carotid with the majority of the energy located between 3 and 40 Hz. Different spectral properties of the carotids of one individual indicate the presence of sound characteristics linked to the particular vessel. User-dependent parameters such as variations of the applied contact pressure appeared to have minor influence on the general stability.

Conclusion

The system provides a platform for reproducible carotid auscultation and the creation of a database of pathological vascular sounds, which is a prerequisite to investigate sound-based vascular monitoring.

Surgical audio information as base for haptic feedback in robotic-assisted procedures

This work aims to demonstrate the feasibility that haptic information can be acquired from a da Vinci robotic tool using audio sensing according to sensor placement requirements in a real clinical scenario. For that, two potential audio sensor locations were studied using an experimental setup for performing, in a repeatable way, interactions of a da Vinci forceps with three different tissues. The obtained audio signals were assessed in terms of their resulting signal-to-noise-ratio (SNR) and their capability to distinguish between different tissues. A spectral energy distribution analysis using Discrete Wavelet Transformation was performed to extract signal signatures from the tested tissues. Results show that a high SNR was obtained in most of the audio recordings acquired from both studied positions. Additionally, evident spectral energy-related patterns could be extracted from the audio signals allowing us to distinguish between different palpated tissues.

Frequency and average gray-level information for thermal ablation status in ultrasound B-Mode sequences

During thermal ablation in a target tissue the information about temperature is crucial for decision making of successful therapy. An observable temporal and spatial temperature propagation would give a visual feedback of irreversible cell damage of the target tissue. Potential temperature features in ultrasound (US) B-Mode image sequences during radiofrequency (RF) ablation in ex-vivo porcine liver were found and analysed. These features could help to detect the transition between reversible and irreversible damage of the ablated target tissue. Experimental RF ablations of ex-vivo porcine liver were imaged with US B-Mode imaging and image sequences were recorded. Temperature was simultaneously measured within the liver tissue around a bipolar RF needle electrode. In the B-Mode images, regions of interest (ROIs) around the centre of the measurement spots were analysed in post-processing using average gray-level (AVGL) compared against temperature. The pole of maximum energy level in the time-frequency domain of the AVGL changes was investigated in relation to the measured temperatures. Frequency shifts of the pole were observed which could be related to transitions between the states of tissue damage.

Acoustic signal analysis of instrument-tissue interaction for minimally invasive interventions

Purpose

Minimally invasive surgery (MIS) has become the standard for many surgical procedures as it minimizes trauma, reduces infection rates and shortens hospitalization. However, the manipulation of objects in the surgical workspace can be difficult due to the unintuitive handling of instruments and limited range of motion. Apart from the advantages of robot-assisted systems such as augmented view or improved dexterity, both robotic and MIS techniques introduce drawbacks such as limited haptic perception and their major reliance on visual perception.

Methods

In order to address the above-mentioned limitations, a perception study was conducted to investigate whether the transmission of intra-abdominal acoustic signals can potentially improve the perception during MIS. To investigate whether these acoustic signals can be used as a basis for further automated analysis, a large audio data set capturing the application of electrosurgery on different types of porcine tissue was acquired. A sliding window technique was applied to compute log-mel-spectrograms, which were fed to a pre-trained convolutional neural network for feature extraction. A fully connected layer was trained on the intermediate feature representation to classify instrument–tissue interaction.

Results

The perception study revealed that acoustic feedback has potential to improve the perception during MIS and to serve as a basis for further automated analysis. The proposed classification pipeline yielded excellent performance for four types of instrument–tissue interaction (muscle, fascia, liver and fatty tissue) and achieved top-1 accuracies of up to 89.9%. Moreover, our model is able to distinguish electrosurgical operation modes with an overall classification accuracy of 86.40%.

Conclusion

Our proof-of-principle indicates great application potential for guidance systems in MIS, such as controlled tissue resection. Supported by a pilot perception study with surgeons, we believe that utilizing audio signals as an additional information channel has great potential to improve the surgical performance and to partly compensate the loss of haptic feedback.

Assessment of Sound Features for Needle Perforation Event Detection

This paper studies the use of non-invasive acoustic emission recordings for clinical device tracking. In particular, audio signals recorded at the proximal end of a needle are used to detect perforation events that occur when the needle tip crosses internal tissue layers.A comparative study is performed to assess the capacity of different features and envelopes in detecting perforation events. The results obtained from the considered experimental setup show a statistically significant correlation between the extracted envelopes and the perforation events, thus leading the way for future development of perforation detection algorithms.

Texture differentiation using audio signal analysis with robotic interventional instruments

Robotic minimally invasive surgery (RMIS) has played an important role in the last decades. In traditional surgery, surgeons rely on palpation using their hands. However, during RMIS, surgeons use the visual-haptics technique to compensate the missing sense of touch. Various sensors have been widely used to retrieve this natural sense, but there are still issues like integration, costs, sterilization and the small sensing area that prevent such approaches from being applied. A new method based on acoustic emission has been recently proposed for acquiring audio information from tool-tissue interaction during minimally invasive procedures that provide user guidance feedback. In this work the concept was adapted for acquiring audio information from a RMIS grasper and a first proof of concept is presented. Interactions of the grasper with various artificial and biological texture samples were recorded and analyzed using advanced signal processing and a clear correlation between audio spectral components and the tested texture were identified.