Utilization of Smartphone Depth Mapping Cameras for App-Based Grading of Facial Movement Disorders: Development and Feasibility Study

Depth sensor technology in facial movement analysis: A comparative evaluation with marker-based motion analysis

OBJECTIVE

This study evaluates the reliability and agreement of depth sensor technology compared to marker-based motion analysis for facial movement assessment. Depth sensors, such as the Kinect-V2, offer a non-invasive alternative, but their accuracy in facial kinematics remains uncertain.

METHOD

100 healthy participants (50 male, 50 female) performed six facial movements- opening -mouth, smiling, eyebrow-lifting, forced-eye-closure, whistling, and frowning. These were recorded simultaneously using a marker-based motion system and a Kinect-V2 depth sensor. Data were analyzed for asymmetry, intra-method reliability using intraclass correlation coefficients (ICC), and agreement via Bland-Altman analysis.

RESULTS

Bland-Altman analysis showed mean biases for facial movements: opening-mouth (-0.99), smiling (2.7), eyebrow-lifting (-1.85), forced-eye-closure (-1.77), whistling (11.59), and frowning (20.82). Mean asymmetry values using the marker-based system vs. depth sensor: smiling (8.16%vs.4.22%), eyebrow-lifting (7.32%vs.6.88%), eye-closure (8.42%vs.5.39%), and frowning (11.50vs.13.86%). ICC values ranged from 0.41 (forced-eye-closure) to 0.80 (eyebrow lifting) for the marker-based system and 0.61 (forced-eye-closure) to 0.85 (mouth opening) for the depth sensor.

CONCLUSIONS

While depth sensors show strong intra-method reliability, they demonstrate biases and broader limits of agreement for subtle expressions. Further algorithmic improvements are needed for clinical applications.

Smartphone applications for facial scanning: A technical and scoping review

INTRODUCTION

Facial scanning through smartphone scanning applications (SSA) is increasingly being used for medical applications as cost-effective, chairside method. However, clinical validation is lacking. This review aims to address: (1) Which SSA could perform facial scanning? (2) Which SSA can be clinically used? (3) Which SSA have been reported and scientifically validated for medical applications?

METHODS

Technical search for SSA designed for face or object scanning was conducted on Google, Apple App Store, and Google Play Store from August 2022 to December 2023. Literature search was performed on PubMed, Cochrane, EMBASE, MEDLINE, Scopus, IEEE Xplore, ACM Digital Library, Clinicaltrials.gov, ICTRP (WHO) and preprints up to 2023. Eligibility criteria included English-written scientific articles incorporating at least one SSA for clinical purposes. SSA selection and data extraction were executed by one reviewer, validated by second, with third reviewer being consulted for discordances.

RESULTS

Sixty-three applications designed for three-dimensional object scanning were retrieved, with 52 currently offering facial scanning capabilities. Fifty-six scientific articles, comprising two case reports, 16 proof-of-concepts and 38 experimental studies were analysed. Thirteen applications (123D Catch, 3D Creator, Bellus 3D Dental Pro, Bellus 3D Face app, Bellus 3D Face Maker, Capture, Heges, Metascan, Polycam, Scandy Pro, Scaniverse, Tap tap tap and Trnio) were reported in literature for digital workflow integration, comparison or proof-of-concept studies.

CONCLUSION

Fifty-two SSA can perform facial scanning currently and can be used clinically, offering cost-effectiveness, portability and user-friendliness. Although clinical validation is crucial, only 13 SSA were scientifically validated, underlying awareness of potential pitfalls and limitations.

Innovative mobile app solution for facial nerve rehabilitation: a usability analysis

Background

Facial palsy after vestibular schwannoma surgery is temporary in many cases but can significantly affect patients' quality of life. Physical training-initially guided and subsequently performed by the patient-is of paramount importance for recovery of facial nerve function. The introduction of medical application software (apps) might improve therapy by maintaining motivation for daily home-based training and surveilling patients' rehabilitation progress.

Methods

We developed a mobile app, "FACEsemper", for home-based facial nerve rehabilitation. This app guides patients through a daily training program comprising six variable exercises, each performed in three repetitions. The app allows the user to customize the exercise intensity for different facial areas and includes a reminder function for daily training. Additional features include photo documentation, a calendar function, training report generation, and the possibility of direct communication with the attending physician. The app's usability was prospectively investigated with 27 subjects, including 8 physicians, 9 patients with facial palsy and 10 healthy subjects, over a two-week period. Usability was assessed using various self-rating questionnaires (i.e., mHealth App Usability Questionnaire, MAUQ; System Usability Scale, SUS; Visual Aesthetics of Apps Inventory, VisAAI) and scores were compared across the groups.

Results

The participants reported an average smartphone use of 12.19 years and completed a mean number of 290 ± 163 facial exercises during the study period. Patients used the app significantly more frequently than the other two groups (p = 0.017). The average total scores of the questionnaires were: MAUQ 5.67/7, SUS 89.6/100, VisAAI 5.88/7 and specific rating 6.13/7. In particular, the simplicity of use and craftsmanship of the app were rated very highly. Usability scores did not significantly differ between groups. A primary limitation identified was malfunction of the daily reminder feature in some Android versions.

Conclusion

This usability study demonstrated a positive user experience and excellent usability of the FACEsemper app. However, some limitations and areas for improvement were identified. As a next step, the app should be evaluated in a large patient cohort with facial palsy to determine its potential medical benefits for facial rehabilitation compared to traditional training methods.

[Telemedicine for patients with facial palsy : Current developments and options in otorhinolaryngologic treatment]

Digitalization is also becoming increasingly important in medicine. The COVID-19 pandemic has further accelerated this process and politicians are trying to create a framework for successful knowledge transfer and better digital medical care. This article describes the role of telemedicine in the treatment of patients suffering from facial nerve palsy. Facial nerve palsy has a wide range of effects, from limitations in facial mobility to psychological sequelae. While many of the acute, idiopathic facial nerve palsies improve after a few weeks, around a third of those affected develop synkinesis, involuntary movements that have lifelong functional and psychological consequences. Treatment includes various modalities, from medication and surgery to movement training. Telemedicine offers innovative solutions in cases of regional underuse, but also in the treatment of chronic facial nerve palsies. The article defines the term "telemedicine" in the current context and presents different types of application. A detailed analysis of the application scenarios of telemedicine in facial nerve palsy patients shows that despite a lack of evidence, many potentially useful concepts exist.

Toward an application of automatic evaluation system for central facial palsy using two simple evaluation indices in emergency medicine

A stroke is a medical emergency and thus requires immediate treatment. Paramedics should accurately assess suspected stroke patients and promptly transport them to a hospital with stroke care facilities; however, current assessment procedures rely on subjective visual assessment. We aim to develop an automatic evaluation system for central facial palsy (CFP) that uses RGB cameras installed in an ambulance. This paper presents two evaluation indices, namely the symmetry of mouth movement and the difference in mouth shape, respectively, extracted from video frames. These evaluation indices allow us to quantitatively evaluate the degree of facial palsy. A classification model based on these indices can discriminate patients with CFP. The results of experiments using our dataset show that the values of the two evaluation indices are significantly different between healthy subjects and CFP patients. Furthermore, our classification model achieved an area under the curve of 0.847. This study demonstrates that the proposed automatic evaluation system has great potential for quantitatively assessing CFP patients based on two evaluation indices.

Mathematical Validation of the Modified Sunnybrook Facial Grading System Using Four-dimensional Imaging

Objectives

Despite the limited validity of the Sunnybrook grading index, it is routinely used for the clinical evaluation of facial palsy. This study aimed to assess the dynamic asymmetry in unilateral facial palsy and mathematically validate a modified version of the Sunnybrook facial grading system.

Methods

The Sunnybrook facial grading system was modified to provide more descriptions of the measured parameters of the distorted facial expression in unilateral facial paralysis. This correlation study was conducted on 16 patients with unilateral facial palsy and a matched control group. Three-dimensional video recordings of six facial expressions - rest, maximum smile, cheek puff, lip purse, eyebrow raising, and eye closure - were used for each case in the analysis. Advanced geometric morphometrics were applied to quantify facial asymmetry and morphology throughout the course of each expression. Seven professional assessors graded facial asymmetry for the 16 cases, twice, using the modified Sunnybrook index. Cross-correlations between the objective mathematical measurements and the subjective clinical grades were calculated.

Results

The inter- and intra-observer reproducibility of the modified Sunnybrook index was high (r = -0.8). Significant positive correlations were detected between the clinical grading of facial palsy and the mathematical measurements at rest, maximum smile, lip purse, and raising of eyebrows. The correlations between the modified Sunnybrook index and mathematical measurements were poor for cheek puff and forceful eye closure.

Conclusions

The modified Sunnybrook grading index proved reproducible and mathematically valid for the grading of unilateral facial paralysis in most facial expressions, except for cheek puff and forceful eye closure.

Towards Preventing Gaps in Health Care Systems through Smartphone Use: Analysis of ARKit for Accurate Measurement of Facial Distances in Different Angles

There is a growing consensus in the global health community that the use of communication technologies will be an essential factor in ensuring universal health coverage of the world's population. New technologies can only be used profitably if their accuracy is sufficient. Therefore, we explore the feasibility of using Apple's ARKit technology to accurately measure the distance from the user's eye to their smartphone screen. We developed an iOS application for measuring eyes-to-phone distances in various angles, using the built-in front-facing-camera and TrueDepth sensor. The actual position of the phone is precisely controlled and recorded, by fixing the head position and placing the phone in a robotic arm. Our results indicate that ARKit is capable of producing accurate measurements, with overall errors ranging between 0.88% and 9.07% from the actual distance, across various head positions. The accuracy of ARKit may be impacted by several factors such as head size, position, device model, and temperature. Our findings suggest that ARKit is a useful tool in the development of applications aimed at preventing eye damage caused by smartphone use.

Use of a Three-dimensional Smartphone Scanner in the Quantitative Analysis of Repaired Unilateral Cleft Lip during the Coronavirus Disease Pandemic

Cleft lip, a major craniofacial abnormality, is highly prevalent among people with low socioeconomic status in Indonesia. Direct two-dimensional measurement of the affected region is the gold standard for surgical preparation; however, its compliance and usability are limited in pediatric patients. Modern smartphones, including iPhones, are equipped with high-resolution cameras, which can record images and videos of a face. Here, we investigated whether a three-dimensional (3D) smartphone scanner can be used for the facial measurements of patients with unilateral cleft lip.

The Feasibility of An Automatical Facial Evaluation System Providing Objective and Reliable Results for Facial Palsy

Facial palsy would lead to a series of physical and mental problems, as facial function plays an important role in various aspects of daily life. However, the current strategies for evaluating facial function relied heavily on raters and the results varied from the experience of raters. Thus, an objective and accurate facial evaluation system is always claimed. In this study, a customized automatical facial evaluation system (AFES) was proposed, which might have the potential to be employed as an adjunctive and efficient assessing method in clinic. In order to investigate the feasibility of AFES, ninety-two participants with facial palsy were recruited and received scale-based subjective manual evaluation (including mHBGS and mSFGS) and objective automatical evaluation of AFES (including aHBGS, aSFGS and indicators of facial regional features) at enrollment and after two weeks. The correlations between the results of the two methods were analyzed and the participants were stratified according to the severity of facial function for further analyses. Strong positive correlations between manual and automatical HBGS and SFGS were observed and higher correlations were reported in the participants with normal-mild and moderate facial palsy. Significant improvements in clinical scales and indicator of eye synkinesis were found in forty-two participants in two weeks. Furthermore, some of the indicators were correlated with scale scores (I4, I7) and one of them presented a significant change between the baseline evaluation and follow-up evaluation (I7). According to the results, AFES could be considered as a viable method to perform objective and reliable evaluation for patients with facial palsy and provide clarified results for prognosis.

Towards a Reliable and Rapid Automated Grading System in Facial Palsy Patients: Facial Palsy Surgery Meets Computer Science

Background: Reliable, time- and cost-effective, and clinician-friendly diagnostic tools are cornerstones in facial palsy (FP) patient management. Different automated FP grading systems have been developed but revealed persisting downsides such as insufficient accuracy and cost-intensive hardware. We aimed to overcome these barriers and programmed an automated grading system for FP patients utilizing the House and Brackmann scale (HBS). Methods: Image datasets of 86 patients seen at the Department of Plastic, Hand, and Reconstructive Surgery at the University Hospital Regensburg, Germany, between June 2017 and May 2021, were used to train the neural network and evaluate its accuracy. Nine facial poses per patient were analyzed by the algorithm. Results: The algorithm showed an accuracy of 100%. Oversampling did not result in altered outcomes, while the direct form displayed superior accuracy levels when compared to the modular classification form (n = 86; 100% vs. 99%). The Early Fusion technique was linked to improved accuracy outcomes in comparison to the Late Fusion and sequential method (n = 86; 100% vs. 96% vs. 97%). Conclusions: Our automated FP grading system combines high-level accuracy with cost- and time-effectiveness. Our algorithm may accelerate the grading process in FP patients and facilitate the FP surgeon’s workflow.

[Diagnostics and Therapy of Idiopathic Facial Palsy (Bell's Palsy)]

The purpose of this review is to report the knowledge for otolaryngologists on standard of care, latest advances, interesting new findings and controversies about the treatment of Bell's palsy. This review is focusing on the acute phase of the disease. The chronic phase, with incomplete, incorrect or no recovery of the palsy, is described briefly. Treatment with prednisolone alone within 72 hours after onset still is the cornerstone of the treatment. The role of antivirals still is unclear. Since 2009 no new and breakthrough clinical trials with influence on the treatment standards have been performed. A study to clarify the role of prednisolone treatment in children is ongoing. Patient-related outcome measures like the Facial Clinimetric Evaluation Scale and the Facial Disability Index are important tools to assess the subjective severity of the disease and psychosocial impact of Bell's palsy next to the motor deficits. Simplified subjective electronic grading systems like the eFACE and first automated image analysis systems have been introduced. Studies clarifying the role of antivirals for severe cases are urgently needed as well as studies on the role of salvage second line therapy after insufficient response to initial corticosteroid treatment. An international consensus on the outcome measures in diagnostics and follow-up is also needed.

Effect of an Intensified Combined Electromyography and Visual Feedback Training on Facial Grading in Patients With Post-paralytic Facial Synkinesis

Background: There is no current standard for facial synkinesis rehabilitation programs. The benefit and stability of effect of an intensified 10-day facial training combining electromyography and visual biofeedback training was evaluated.

Methods: Fifty-four patients (77.8% female; median age: 49.5 years) with post-paralytic facial synkinesis (median time to onset of paralysis: 31.1 months) were included in retrospective longitudinal study between January 2013 and June 2016. Facial function was assesses at baseline (T0), first days of training (T1), last day of training (T2), and follow-up visit (T3) at a median time of 6 months later using the House-Brackmann (HB) facial nerve grading system, Stennert index (SI), Facial Nerve Grading System 2.0 (FNGS 2.0), and Sunnybrook Facial Grading System (SFGS). Pairwise comparisons between the time points with post-hoc Bonferroni correction were performed.

Results: No significant changes of the gradings and subscores were seen between T0 and T1 (all p > 0.01). The 10-day combined and intensified feedback training between T1 and T2 improved facial symmetry and decreased synkinetic activity. Facial grading with the FNGS 2.0 or the SFGS were most suited to depict the training effect. FNGS 2.0, regional score, FNGS 2.0, synkinesis score, and FNGS 2.0 total score improved significantly (all p ≤ 0.0001). Both, the FNGS 2.0 and the SFGS showed the strongest improvement in the nasolabial fold/zygomatic and the oral region. Neither the age of the patient (r = 0.168; p = 0.224), the gender (r = 0.126; p = 0.363) nor the length of the interval between onset of the palsy and training start (r = 0.011; p = 0.886) correlated with the changes of the SFGS between T1 and T2. The results remained stable between T2 and T3 without any further significant change.

Conclusion: Intensified daily combined electromyography and visual biofeedback training over 10 days was effective in patients with facial synkinesis and benefits were stable 6 months after therapy.