Acoustic speech analysis of patients with decompensated heart failure: A pilot study

Guess What We Can Hear-Novel Voice Biomarkers for the Remote Detection of Disease

The advancement of digital biomarkers and the provision of remote health care greatly progressed during the coronavirus disease 2019 global pandemic. Combining voice/speech data with artificial intelligence and machine-based learning offers a novel solution to the growing demand for telemedicine. Voice biomarkers, obtained from the extraction of characteristic acoustic and linguistic features, are associated with a variety of diseases and even coronavirus disease 2019. In the current review, we (1) describe the basis on which digital voice biomarkers could facilitate "telemedicine," (2) discuss potential mechanisms that may explain the association between voice biomarkers and disease, (3) offer a novel classification system to conceptualize voice biomarkers depending on different methods for recording and analyzing voice/speech samples, (4) outline evidence revealing an association between voice biomarkers and a number of disease states, and (5) describe the process of developing a voice biomarker from recording, storing voice samples, and extracting acoustic and linguistic features relevant to training and testing deep and machine-based learning algorithms to detect disease. We further explore several important future considerations in this area of research, including the necessity for clinical trials and the importance of safeguarding data and individual privacy. To this end, we searched PubMed and Google Scholar to identify studies evaluating the relationship between voice/speech features and biomarkers and various diseases. Search terms included digital biomarker, telemedicine, voice features, voice biomarker, speech features, speech biomarkers, acoustics, linguistics, cardiovascular disease, neurologic disease, psychiatric disease, and infectious disease. The search was limited to studies published in English in peer-reviewed journals between 1980 and the present. To identify potential studies not captured by our database search strategy, we also searched studies listed in the bibliography of relevant publications and reviews.

Acoustic Voice and Speech Biomarkers of Treatment Status during Hospitalization for Acute Decompensated Heart Failure

This study investigates acoustic voice and speech features as biomarkers for acute decompensated heart failure (ADHF), a serious escalation of heart failure symptoms including breathlessness and fatigue. ADHF-related systemic fluid accumulation in the lungs and laryngeal tissues is hypothesized to affect phonation and respiration for speech. A set of daily spoken recordings from 52 patients undergoing inpatient ADHF treatment was analyzed to identify voice and speech biomarkers for ADHF and to examine the trajectory of biomarkers during treatment. Results indicated that speakers produce more stable phonation, a more creaky voice, faster speech rates, and longer phrases after ADHF treatment compared to their pre-treatment voices. This project builds on work to develop a method of monitoring ADHF using speech biomarkers and presents a more detailed understanding of relevant voice and speech features.

Speech biomarkers of risk factors for vascular dementia in people with mild cognitive impairment

Introduction

In this study we intend to use speech analysis to analyze the cognitive impairments caused by pathologies of vascular origin such as diabetes, hypertension, hypercholesterolemia and heart disease, predictors of the development of vascular dementia.

Methods

In this study, 40 participants with mild cognitive impairment were asked to read while being recorded and they were inquired about their history of the aforementioned conditions. Their speech was then analyzed.

Results

We found that some speech parameters of frequencies and syllabic rhythm vary due to these pathologies. In addition, we conducted a discriminant analysis in which we found that diabetes and hypertension can be predicted with an accuracy over 95% with few speech parameters, and hypercholesterolemia and heart disease with an accuracy over 80%.

Discussion

The predictor parameters found are heterogeneous, including voice quality, amplitude, frequency, and rhythm parameters. This result may lead to investigate why such important qualitative changes occur in the voice of older adults with these pathologies. Rather than trying to find a diagnostic procedure already existing in classical medicine, we expect this finding to contribute to explore the causes and concomitant pathologies of these diseases. We discuss the implications of behavioral traits, such as speech, as digital biomarkers.

Evaluation of Speech and Pause Alterations in Patients With Acute and Chronic Heart Failure

Background

Acute heart failure is the most frequent cause of unplanned hospital admission in elderly patients. Various biomarkers have been evaluated to better assess the status of these patients and prevent decompensation. Recently, voice has been suggested as a cost‐effective and noninvasive way to monitor disease progression. This study evaluates speech and pause alterations in patients with acute decompensated and stable heart failure. Specifically, we aim to identify a vocal biomarker that could be used to monitor patients with heart failure and to prevent decompensation.

Methods and Results

Speech and pause patterns were evaluated in 68 patients with acute and 36 patients with stable heart failure. Voice recordings were performed using a web‐browser based application that consisted of 5 tasks. Speech and pause patterns were automatically extracted and compared between acute and stable patients and with clinical markers. Compared with stable patients, pause ratio was up to 14.9% increased in patients with acute heart failure. This increase was largely independent of sex, age, and ejection fraction and persisted in patients with lower degrees of edema or dyspnea. Furthermore, pause ratio was positively correlated with NT‐proBNP (N‐terminal pro‐B‐type natriuretic peptide) after controlling for acute versus stable heart failure. Collectively, our findings indicate that the pause ratio could be useful in identifying acute heart failure, particularly in patients who do not display traditional indicators of decompensation.

Conclusions

Speech and pause patterns are altered in patients with acute heart failure. Particularly, we identified pause ratio as an easily interpretable vocal biomarker to support the monitoring of heart failure decompensation.

Remote Speech Analysis in the Evaluation of Hospitalized Patients With Acute Decompensated Heart Failure

OBJECTIVES

This study assessed the performance of an automated speech analysis technology in detecting pulmonary fluid overload in patients with acute decompensated heart failure (ADHF).

BACKGROUND

Pulmonary edema is the main cause of heart failure (HF)-related hospitalizations and a key predictor of poor postdischarge prognosis. Frequent monitoring is often recommended, but signs of decompensation are often missed. Voice and sound analysis technologies have been shown to successfully identify clinical conditions that affect vocal cord vibration mechanics.

METHODS

Adult patients with ADHF (n = 40) recorded 5 sentences, in 1 of 3 languages, using HearO, a proprietary speech processing and analysis application, upon admission (wet) to and discharge (dry) from the hospital. Recordings were analyzed for 5 distinct speech measures (SMs), each a distinct time, frequency resolution, and linear versus perceptual (ear) model; mean change from baseline SMs was calculated.

RESULTS

In total, 1,484 recordings were analyzed. Discharge recordings were successfully tagged as distinctly different from baseline (wet) in 94% of cases, with distinct differences shown for all 5 SMs in 87.5% of cases. The largest change from baseline was documented for SM2 (218%). Unsupervised, blinded clustering of untagged admission and discharge recordings of 9 patients was further demonstrated for all 5 SMs.

CONCLUSIONS

Automated speech analysis technology can identify voice alterations reflective of HF status. This platform is expected to provide a valuable contribution to in-person and remote follow-up of patients with HF, by alerting to imminent deterioration, thereby reducing hospitalization rates. (Clinical Evaluation of Cordio App in Adult Patients With CHF; NCT03266029).

Reducing the Heart Failure Burden in Romania by Predicting Congestive Heart Failure Using Artificial Intelligence: Proof of Concept

Due to population aging, we are currently confronted with an increased number of chronic heart failure patients. The primary purpose of this study was to implement a noncontact system that can predict heart failure exacerbation through vocal analysis. We designed the system to evaluate the voice characteristics of every patient, and we used the identified variations as an input for a machine-learning-based approach. We collected data from a total of 16 patients, 9 men and 7 women, aged 65–91 years old, who agreed to take part in the study, with a detailed signed informed consent. We included hospitalized patients admitted with cardiogenic acute pulmonary edema in the study, regardless of the precipitation cause or other known cardiovascular comorbidities. There were no specific exclusion criteria, except age (which had to be over 18 years old) and patients with speech inabilities. We then recorded each patient’s voice twice a day, using the same smartphone, Lenovo P780, from day one of hospitalization—when their general status was critical—until the day of discharge, when they were clinically stable. We used the New York Heart Association Functional Classification (NYHA) classification system for heart failure to include the patients in stages based on their clinical evolution. Each voice recording has been accordingly equated and subsequently introduced into the machine-learning algorithm. We used multiple machine-learning techniques for classification in order to detect which one turns out to be more appropriate for the given dataset and the one that can be the starting point for future developments. We used algorithms such as Artificial Neural Networks (ANN), Support Vector Machine (SVM) and K-Nearest Neighbors (KNN). After integrating the information from 15 patients, the algorithm correctly classified the 16th patient into the third NYHA stage at hospitalization and second NYHA stage at discharge, based only on his voice recording. The KNN algorithm proved to have the best classification accuracy, with a value of 0.945. Voice is a cheap and easy way to monitor a patient’s health status. The algorithm we have used for analyzing the voice provides highly accurate preliminary results. We aim to obtain larger datasets and compute more complex voice analyzer algorithms to certify the outcomes presented.

Enlarging the spectrum of cluster headache: Extracranial autonomic involvement revealed by voice analysis

Background

People with cluster headache (CH) are frequently burdened by misdiagnosis or diagnostic delay. The peculiar somatic and behavioral changes characterizing patients with CH are not useful to improve diagnostic accuracy. In our clinical experience, we noticed a typical voice quality with low and croaking tone in patients with CH.

In this cross‐sectional study, we evaluated, by digital voice analysis, whether it is possible to identify typical voice quality characterizing patients with CH compared with healthy controls (HCs). Furthermore, to investigate whether putative differences in voice characteristics could be underpinned by constitutional aspects or pathological processes of vocal cords, subjects underwent a videolaryngostroboscopy. Smoking habits and alcohol consumption were specifically investigated.

Methods

After conducting digital recording of the voices from both patients with CH and HCs in a soundproof insulated cabin in the laboratory of the Audiology Department, a set of voice parameters was analyzed. We included the measures of fundamental frequency, calculations of jitter and shimmer, and noise‐to‐harmonics ratios as well as quantities related to the spectral tilt (i.e., H1–H2, H1–A1, H1–A2, and H1–A3) in 20 patients with CH and in 13 HCs. A videolaryngostroboscopy was performed in all subjects.

Results

Patients with CH, explored during the cluster bout period, showed significantly lower second harmonic (H1–H2) values compared with HCs (−6.9 ± 7.6 vs. 2.1 ± 6.7, p = 0.002), usually characterizing the so‐called creaky voice. By using a laryngoscopy investigation, a significantly higher prevalence of mild to moderate vocal cord edema and laryngopharyngeal reflux signs were found in patients with CH (100% of patients with CH vs. 15% of HC, p < 0.001).

Conclusion

Creaky phonation is a “physiological mode of laryngeal operation” usually underpinned by shortened and thickened vocal folds. Creaky voice phonation can be due to a vocal fold's reduced capability to become slack or flaccid secondary to vocal cord edema underpinned by laryngopharyngeal reflux affecting the phonatory mechanisms in patients with CH. The laryngopharyngeal reflux may represent a dysautonomic sign related to the increased parasympathetic tone during in‐bout period, reinforcing the hypothesis of an extracranial autonomic dysfunction as part of CH clinical picture.

Natural Language Processing as an Emerging Tool to Detect Late-Life Depression

Late-life depression (LLD) is a major public health concern. Despite the availability of effective treatments for depression, barriers to screening and diagnosis still exist. The use of current standardized depression assessments can lead to underdiagnosis or misdiagnosis due to subjective symptom reporting and the distinct cognitive, psychomotor, and somatic features of LLD. To overcome these limitations, there has been a growing interest in the development of objective measures of depression using artificial intelligence (AI) technologies such as natural language processing (NLP). NLP approaches focus on the analysis of acoustic and linguistic aspects of human language derived from text and speech and can be integrated with machine learning approaches to classify depression and its severity. In this review, we will provide rationale for the use of NLP methods to study depression using speech, summarize previous research using NLP in LLD, compare findings to younger adults with depression and older adults with other clinical conditions, and discuss future directions including the use of complementary AI strategies to fully capture the spectrum of LLD.

Noninvasive Vocal Biomarker is Associated With Severe Acute Respiratory Syndrome Coronavirus 2 Infection

Objective

To investigate the association of voice analysis with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection.

Patients and Methods

A vocal biomarker, a unitless scalar with a value between 0 and 1, was developed based on 434 voice samples. The biomarker training was followed by a prospective, multicenter, observational study. All subjects were tested for SARS-CoV-2, had their voice recorded to a smartphone application, and gave their informed consent to participate in the study. The association of SARS-CoV-2 infection with the vocal biomarker was evaluated.

Results

The final study population included 80 subjects with a median age of 29 [range, 23 to 36] years, of whom 68% were men. Forty patients were positive for SARS-CoV-2. Infected patients were 12 times more likely to report at least one symptom (odds ratio, 11.8; P<.001). The vocal biomarker was significantly higher among infected patients (OR, 0.11; 95% CI, 0.06 to 0.17 vs OR, 0.19; 95% CI, 0.12 to 0.3; P=.001). The area under the receiver operating characteristic curve evaluating the association of the vocal biomarker with SARS-CoV-2 status was 72%. With a biomarker threshold of 0.115, the results translated to a sensitivity and specificity of 85% (95% CI, 70% to 94%) and 53% (95% CI, 36% to 69%), respectively. When added to a self-reported symptom classifier, the area under the curve significantly improved from 0.775 to 0.85.

Conclusion

Voice analysis is associated with SARS-CoV-2 status and holds the potential to improve the accuracy of self-reported symptom-based screening tools. This pilot study suggests a possible role for vocal biomarkers in screening for SARS-CoV-2-infected subjects.

Feasibility of remote speech analysis in evaluation of dynamic fluid overload in heart failure patients undergoing haemodialysis treatment

Aims

This study aimed to assess the ability of a voice analysis application to discriminate between wet and dry states in chronic heart failure (CHF) patients undergoing regular scheduled haemodialysis treatment due to volume overload as a result of their chronic renal failure.

Methods and results

In this single‐centre, observational study, five patients with CHF, peripheral oedema of ≥2, and pulmonary congestion‐related dyspnoea, undergoing haemodialysis three times per week, recorded five sentences into a standard smartphone/tablet before and after haemodialysis. Recordings were provided that same noon/early evening and the next morning and evening. Patient weight was measured at the hospital before and after each haemodialysis session. Recordings were analysed by a smartphone application (app) algorithm, to compare speech measures (SMs) of utterances collected over time. On average, patients provided recordings throughout 25.8 ± 3.9 dialysis treatment cycles, resulting in a total of 472 recordings. Weight changes of 1.95 ± 0.64 kg were documented during cycles. Median baseline SM prior to dialysis was 0.87 ± 0.17, and rose to 1.07 ± 0.15 following the end of the dialysis session, at noon (P = 0.0355), and remained at a similar level until the following morning (P = 0.007). By the evening of the day following dialysis, SMs returned to baseline levels (0.88 ± 0.19). Changes in patient weight immediately after dialysis positively correlated with SM changes, with the strongest correlation measured the evening of the dialysis day [slope: −0.40 ± 0.15 (95% confidence interval: −0.71 to −0.10), P = 0.0096].

Conclusions

The fluid‐controlled haemodialysis model demonstrated the ability of the app algorithm to identify cyclic changes in SMs, which reflected bodily fluid levels. The voice analysis platform bears considerable potential as a harbinger of impending fluid overload in a range of clinical scenarios, which will enhance monitoring and triage efforts, ultimately optimizing remote CHF management.

Digital Health Applications in Heart Failure: a Critical Appraisal of Literature

Purpose of the review

Despite advancements in the diagnostic and therapeutic armamentarium, heart failure (HF) remains a major public health concern in the USA and worldwide. Digital health applications hold promise to bridge this gap and improve HF care. This review will provide the reader with a concise overview of the current digital health applications in HF, the main challenges to its use, and discuss the future of digital health for promoting care for HF patients.

Recent findings

Emerging evidence continues to support the potential role of digital health across the continuum of HF disease process including primary prevention, early detection, disease management, and reducing associated morbidity. There is also increasing emphasis on the need to pursue rigorous investigations to validate these promising claims, with some successful stories that have changed clinical practices.

Summary

Digital health technologies have emerged as potentially useful tools to complement HF care in both research and clinical realms. As digital technologies continue to play an increasing role in transforming healthcare delivery, creating the framework for its effective use would be necessary to ensure that digital health applications consistently improve outcomes and enhance care for HF patients.

Feasibility of Incorporating Voice Technology and Virtual Assistants in Cardiovascular Care and Clinical Trials

PURPOSE OF REVIEW

With the rising cost of cardiovascular clinical trials, there is interest in determining whether new technologies can increase cost effectiveness. This review focuses on current and potential uses of voice-based technologies, including virtual assistants, in cardiovascular clinical trials.

RECENT FINDINGS

Numerous potential uses for voice-based technologies have begun to emerge within cardiovascular medicine. Voice biomarkers, subtle changes in speech parameters, have emerged as a potential tool to diagnose and monitor many cardiovascular conditions, including heart failure, coronary artery disease, and pulmonary hypertension. With the increasing use of virtual assistants, numerous pilot studies have examined whether these devices can supplement initiatives to promote transitional care, physical activity, smoking cessation, and medication adherence with promising initial results. Additionally, these devices have demonstrated the ability to streamline data collection by administering questionnaires accurately and reliably. With the use of these technologies, there are several challenges that must be addressed before wider implementation including respecting patient privacy, maintaining regulatory standards, acceptance by patients and healthcare providers, determining the validity of voice-based biomarkers and endpoints, and increased accessibility.

SUMMARY

Voice technology represents a novel and promising tool for cardiovascular clinical trials; however, research is still required to understand how it can be best harnessed.

The Future of Wearables in Heart Failure Patients

The adoption of mobile health (mHealth) devices is creating a unique opportunity to improve heart failure (HF) care. The rise of mHealth is driven by multiple factors including consumerism, policy changes in health care, and innovations in technology. Wearable health devices are one aspect of mHealth that may improve the delivery of HF care by allowing for medical data collection outside of a clinician's office or hospital. Wearable devices are externally applied and capture functional or physiological data in order to monitor and improve patients' health. Most wearable sensors capture data continuously and may be incorporated into accessories (e.g., a watch or clothing) or may be applied as a cutaneous patch. Wearable devices are often paired with another device, such as a smartphone, to collect, interpret, or transmit data. This study assessed the potential applications of wearable devices in HF care, summarizes available data for wearables, and discusses the future of wearables for improving the health of patients with HF.

Reimagining What We Measure in Atherosclerosis-a "Phenotype Stack"

The term phenotype is so commonly used that we often assume that we each mean the same thing. The general definition, the set of observable characteristics of an individual resulting from the interaction of their genotype with the environment, is often left to the eye of the beholder. Whether applied to the multiple levels of biological phenomena or the intact human being, our ability to characterize, classify, and analyze phenotype has been limited by measurement deficits, computing limitations, and a culture that avoids the generalizable. With the advent of modern technology, there is the potential for a revolution in phenotyping, which incorporates old and new in structured ways to dramatically advance basic understanding of biology and behavior and to lead to major improvements in clinical care and public health. This revolution in how we think about phenotypes will require a radical change in the scale at which biomedicine operates with significant changes in the unit of action, which will have far-reaching implications for how care, translation, and discovery are implemented.

Non-invasive vocal biomarker is associated with pulmonary hypertension

Emerging data suggest that noninvasive voice biomarker analysis is associated with coronary artery disease. We recently showed that a vocal biomarker was associated with hospitalization and heart failure in patients with heart failure. We evaluate the association between a vocal biomarker and invasively measured indices of pulmonary hypertension (PH). Patients were referred for an invasive cardiac hemodynamic study between January 2017 and December 2018, and had their voices recorded on three separate occasions to their smartphone prior to each study. A pre-established vocal biomarker was determined based on each individual recording. The intra-class correlation co-efficient between the separate voice recording biomarker values for each individual participant was 0.829 (95% CI 0.740-0.889) implying very good agreement between values. Thus, the mean biomarker was calculated for each patient. Patients were divided into two groups: high pulmonary arterial pressure (PAP) defined as ≥ 35 mmHg (moderate or greater PH), versus lower PAP. Eighty three patients, mean age 61.6 ± 15.1 years, 37 (44.6%) male, were included. Patients with a high mean PAP (≥ 35 mmHg) had on average significantly higher values of the mean voice biomarker compared to those with a lower mean PAP (0.74 ± 0.85 vs. 0.40 ± 0.88 p = 0.046). Multivariate logistic regression showed that an increase in the mean voice biomarker by 1 unit was associated with a high PAP, odds ratio 2.31, 95% CI 1.05-5.07, p = 0.038. This study shows a relationship between a noninvasive vocal biomarker and an invasively derived hemodynamic index related to PH obtained during clinically indicated cardiac catheterization. These results may have important practical clinical implications for telemedicine and remote monitoring of patients with heart failure and PH.

Vocal Biomarker Is Associated With Hospitalization and Mortality Among Heart Failure Patients

Background The purpose of this article is to evaluate the association of voice signal analysis with adverse outcome among patients with congestive heart failure (CHF). Methods and Results The study cohort included 10 583 patients who were registered to a call center of patients who had chronic conditions including CHF in Israel between 2013 and 2018. A total of 223 acoustic features were extracted from 20 s of speech for each patient. A biomarker was developed based on a training cohort of non-CHF patients (N=8316). The biomarker was tested on a mutually exclusive CHF study cohort (N=2267) and was evaluated as a continuous and ordinal (4 quartiles) variable. Median age of the CHF study population was 77 (interquartile range 68-83) and 63% were men. During a median follow-up of 20 months (interquartile range 9-34), 824 (36%) patients died. Kaplan-Meier survival analysis showed higher cumulative probability of death with increasing quartiles (23%, 29%, 38%, and 54%; P<0.001). Survival analysis with adjustment to known predictors of poor survival demonstrated that each SD increase in the biomarker was associated with a significant 32% increased risk of death during follow-up (95% CI, 1.24-1.41, P<0.001) and that compared with the lowest quartile, patients in the highest quartile were 96% more likely to die (95% CI, 1.59-2.42, P<0.001). The model consistently demonstrated an independent association of the biomarker with hospitalizations during follow-up (P<0.001). Conclusions Noninvasive vocal biomarker is associated with adverse outcome among CHF patients, suggesting a possible role for voice analysis in telemedicine and CHF patient care.

Identifying a creak probability threshold for an irregular pitch period detection algorithm

Irregular pitch periods (IPPs) are associated with grammatically, pragmatically, and clinically significant types of nonmodal phonation, but are challenging to identify. Automatic detection of IPPs is desirable because accurately hand-identifying IPPs is time-consuming and requires training. The authors evaluated an algorithm developed for creaky voice analysis to automatically identify IPPs in recordings of American English conversational speech. To determine a perceptually relevant threshold probability, frame-by-frame creak probabilities were compared to hand labels, yielding a threshold of approximately 0.02. These results indicate a generally good agreement between hand-labeled IPPs and automatic detection, calling for future work investigating effects of linguistic and prosodic context.