Towards optimizing electrode configurations for silent speech recognition based on high-density surface electromyography

Speech synthesis from three-axis accelerometer signals using conformer-based deep neural network

Silent speech interfaces (SSIs) have emerged as innovative non-acoustic communication methods, and our previous study demonstrated the significant potential of three-axis accelerometer-based SSIs to identify silently spoken words with high classification accuracy. The developed accelerometer-based SSI with only four accelerometers and a small training dataset outperformed a conventional surface electromyography (sEMG)-based SSI. In this study, motivated by the promising initial results, we investigated the feasibility of synthesizing spoken speech from three-axis accelerometer signals. This exploration aimed to assess the potential of accelerometer-based SSIs for practical silent communication applications. Nineteen healthy individuals participated in our experiments. Five accelerometers were attached to the face to acquire speech-related facial movements while the participants read 270 Korean sentences aloud. For the speech synthesis, we used a convolution-augmented Transformer (Conformer)-based deep neural network model to convert the accelerometer signals into a Mel spectrogram, from which an audio waveform was synthesized using HiFi-GAN. To evaluate the quality of the generated Mel spectrograms, ten-fold cross-validation was performed, and the Mel cepstral distortion (MCD) was chosen as the evaluation metric. As a result, an average MCD of 5.03 ± 0.65 was achieved using four optimized accelerometers based on our previous study. Furthermore, the quality of generated Mel spectrograms was significantly enhanced by adding one more accelerometer attached under the chin, achieving an average MCD of 4.86 ± 0.65 (p < 0.001, Wilcoxon signed-rank test). Although an objective comparison is difficult, these results surpass those obtained using conventional SSIs based on sEMG, electromagnetic articulography, and electropalatography with the fewest sensors and a similar or smaller number of sentences to train the model. Our proposed approach will contribute to the widespread adoption of accelerometer-based SSIs, leveraging the advantages of accelerometers like low power consumption, invulnerability to physiological artifacts, and high portability.

A data-efficient and easy-to-use lip language interface based on wearable motion capture and speech movement reconstruction

Lip language recognition urgently needs wearable and easy-to-use interfaces for interference-free and high-fidelity lip-reading acquisition and to develop accompanying data-efficient decoder-modeling methods. Existing solutions suffer from unreliable lip reading, are data hungry, and exhibit poor generalization. Here, we propose a wearable lip language decoding technology that enables interference-free and high-fidelity acquisition of lip movements and data-efficient recognition of fluent lip language based on wearable motion capture and continuous lip speech movement reconstruction. The method allows us to artificially generate any wanted continuous speech datasets from a very limited corpus of word samples from users. By using these artificial datasets to train the decoder, we achieve an average accuracy of 92.0% across individuals (n = 7) for actual continuous and fluent lip speech recognition for 93 English sentences, even observing no training burn on users because all training datasets are artificially generated. Our method greatly minimizes users' training/learning load and presents a data-efficient and easy-to-use paradigm for lip language recognition.

Differentiation of Bolus Texture During Deglutition via High-Density Surface Electromyography: A Pilot Study

OBJECTIVE

Swallowing is a complex neuromuscular task. There is limited spatiotemporal data on normative surface electromyographic signal during swallow, particularly across standard textures. We hypothesize the pattern of electromyographic signal of the anterior neck varies cranio-caudally, that laterality can be evaluated, and categorization of bolus texture can be differentiated by high-density surface electromyography (HDsEMG) through signal analysis.

METHODS

An HDsEMG grid of 20 electrodes captured electromyographic activity in eight healthy adult subjects across 240 total swallows. Participants swallowed five standard textures: saliva, thin liquid, puree, mixed consistency, and dry solid. Data were bandpass filtered, underwent functional alignment of signal, and then placed into binary classifier receiver operating characteristic (ROC) curves. Muscular activity was visualized by creating two-dimensional EMG heat maps.

RESULTS

Signal analysis results demonstrated a positive correlation between signal amplitude and bolus texture. Greater differences of amplitude in the cranial most region of the array when compared to the caudal most region were noted in all subjects. Lateral comparison of the array revealed symmetric power levels across all subjects and textures. ROC curves demonstrated the ability to correctly classify textures within subjects in 6 of 10 texture comparisons.

CONCLUSION

This pilot study suggests that utilizing HDsEMG during deglutition can noninvasively differentiate swallows of varying texture noninvasively. This may prove useful in future diagnostic and behavioral swallow applications.

LEVEL OF EVIDENCE

4 Laryngoscope, 133:2695-2703, 2023.

Wearable bioelectronics fabricated in situ on skins

In recent years, wearable bioelectronics has rapidly expanded for diagnosing, monitoring, and treating various pathological conditions from the skin surface. Although the devices are typically prefabricated as soft patches for general usage, there is a growing need for devices that are customized in situ to provide accurate data and precise treatment. In this perspective, the state-of-the-art in situ fabricated wearable bioelectronics are summarized, focusing primarily on Drawn-on-Skin (DoS) bioelectronics and other in situ fabrication methods. The advantages and limitations of these technologies are evaluated and potential future directions are suggested for the widespread adoption of these technologies in everyday life.

Customizable, reconfigurable, and anatomically coordinated large-area, high-density electromyography from drawn-on-skin electrode arrays

Accurate anatomical matching for patient-specific electromyographic (EMG) mapping is crucial yet technically challenging in various medical disciplines. The fixed electrode construction of multielectrode arrays (MEAs) makes it nearly impossible to match an individual's unique muscle anatomy. This mismatch between the MEAs and target muscles leads to missing relevant muscle activity, highly redundant data, complicated electrode placement optimization, and inaccuracies in classification algorithms. Here, we present customizable and reconfigurable drawn-on-skin (DoS) MEAs as the first demonstration of high-density EMG mapping from in situ-fabricated electrodes with tunable configurations adapted to subject-specific muscle anatomy. The DoS MEAs show uniform electrical properties and can map EMG activity with high fidelity under skin deformation-induced motion, which stems from the unique and robust skin-electrode interface. They can be used to localize innervation zones (IZs), detect motor unit propagation, and capture EMG signals with consistent quality during large muscle movements. Reconfiguring the electrode arrangement of DoS MEAs to match and extend the coverage of the forearm flexors enables localization of the muscle activity and prevents missed information such as IZs. In addition, DoS MEAs customized to the specific anatomy of subjects produce highly informative data, leading to accurate finger gesture detection and prosthetic control compared with conventional technology.

Programmable living assembly of materials by bacterial adhesion

The field of engineered living materials aims to construct functional materials with desirable properties of natural living systems. A recent study demonstrated the programmed self-assembly of bacterial populations by engineered adhesion. Here we use this strategy to engineer self-healing living materials with versatile functions. Bacteria displaying outer membrane-anchored nanobody-antigen pairs are cultured separately and, when mixed, adhere to each other to enable processing into functional materials, which we term living assembled material by bacterial adhesion (LAMBA). LAMBA is programmable and can be functionalized with extracellular moieties up to 545 amino acids. Notably, the adhesion between nanobody-antigen pairs in LAMBA leads to fast recovery under stretching or bending. By exploiting this feature, we fabricated wearable LAMBA sensors that can detect bioelectrical or biomechanical signals. Our work establishes a scalable approach to produce genetically editable and self-healable living functional materials that can be applied in biomanufacturing, bioremediation and soft bioelectronics assembly.