3D Video Analysis of Facial Movements

Outcomes of Free Flap Transfer in Facial Reanimation: A Review

Free functional muscle transfer is is an option for reanimating the face in chronic facial nerve paralysis. The optimal outcome in these patients is the ability to restore a spontaneous smile in response to emotion. We discuss the role of free functional muscle transfer in facial paralysis treatment, the choices of nerve used in reconstruction surgery, and the application of different types of muscle flaps in facial reanimation. In this paper, we review the relevant and up-to-date academic literature regarding the outcomes of free functional muscle flap transfer in facial paralysis patients.

Novel Baseline Facial Muscle Database Using Statistical Shape Modeling and In Silico Trials toward Decision Support for Facial Rehabilitation

Backgrounds and Objective: Facial palsy is a complex pathophysiological condition affecting the personal and professional lives of the involved patients. Sudden muscle weakness or paralysis needs to be rehabilitated to recover a symmetric and expressive face. Computer-aided decision support systems for facial rehabilitation have been developed. However, there is a lack of facial muscle baseline data to evaluate the patient states and guide as well as optimize the rehabilitation strategy. In this present study, we aimed to develop a novel baseline facial muscle database (static and dynamic behaviors) using the coupling between statistical shape modeling and in-silico trial approaches. Methods: 10,000 virtual subjects (5000 males and 5000 females) were generated from a statistical shape modeling (SSM) head model. Skull and muscle networks were defined so that they statistically fit with the head shapes. Two standard mimics: smiling and kissing were generated. The muscle strains of the lengths in neutral and mimic positions were computed and recorded thanks to the muscle insertion and attachment points on the animated head and skull meshes. For validation, five head and skull meshes were reconstructed from the five computed tomography (CT) image sets. Skull and muscle networks were then predicted from the reconstructed head meshes. The predicted skull meshes were compared with the reconstructed skull meshes based on the mesh-to-mesh distance metrics. The predicted muscle lengths were also compared with those manually defined on the reconstructed head and skull meshes. Moreover, the computed muscle lengths and strains were compared with those in our previous studies and the literature. Results: The skull prediction's median deviations from the CT-based models were 2.2236 mm, 2.1371 mm, and 2.1277 mm for the skull shape, skull mesh, and muscle attachment point regions, respectively. The median deviation of the muscle lengths was 4.8940 mm. The computed muscle strains were compatible with the reported values in our previous Kinect-based method and the literature. Conclusions: The development of our novel facial muscle database opens new avenues to accurately evaluate the facial muscle states of facial palsy patients. Based on the evaluated results, specific types of facial mimic rehabilitation exercises can also be selected optimally to train the target muscles. In perspective, the database of the computed muscle lengths and strains will be integrated into our available clinical decision support system for automatically detecting malfunctioning muscles and proposing patient-specific rehabilitation serious games.

Accuracy and Reliability of 3D Imaging for Facial Movement Evaluation: Validation of the VECTRA H1

Three-dimensional imaging can be used to obtain objective assessments of facial morphology that is useful in a variety of clinical settings. The VECTRA H1 is unique in that it is relatively inexpensive, handheld, and does not require standardized environmental conditions for image capture. Although it provides accurate measurements when imaging relaxed facial expressions, the clinical evaluation of many disorders involves the assessment of facial morphology when performing facial movements. The aim of this study was to assess the accuracy and reliability of the VECTRA H1, specifically when imaging facial movement.

Methods

The accuracy, intrarater, and interrater reliability of the VECTRA H1 were assessed when imaging four facial expressions: eyebrow lift, smile, snarl, and lip pucker. Fourteen healthy adult subjects had the distances between 13 fiducial facial landmarks measured at rest and the terminal point of each of the four movements by digital caliper and by the VECTRA H1. Intraclass correlation and Bland-Altman limits of agreement were used to determine agreement between measures. The agreement between measurements obtained by five different reviewers was evaluated by intraclass correlation to determine interrater reliability.

Results

Median correlation between digital caliper and VECTRA H1 measurements ranged from 0.907 (snarl) to 0.921 (smile). Median correlation was very good for both intrarater (0.960-0.975) and interrater reliability (0.997-0.999). The mean absolute error between modalities, and both within and between raters was less than 2 mm for all movements tested.

Conclusion

The VECTRA H1 met acceptable standards for the assessment of facial morphology when imaging facial movements.

"Introduction of a low-cost and automated four-dimensional assessment system of the face."

SUMMARY

Existing automated objective grading systems either fail to consider the face's complex 3D morphology or suffer from poor feasibility and usability. Consumer-based Red Green Blue Depth (RGB-D) sensors and/or smartphone integrated 3D hardware can inexpensively collect detailed four-dimensional facial data in real-time but are yet to be incorporated into a practical system. This study aims to evaluate the feasibility of a proof-of-concept automated 4D facial assessment system using an RGB-D sensor (termed OpenFAS) for use in a standard clinical environment. This study was performed on normal adult volunteers and patients with facial nerve palsy (FNP). The setup consists of the Intel RealSense SR300 connected to a laptop running the OpenFAS application. The subject sequentially mimics the facial expressions shown on screen. Each frame is landmarked, and automatic anthropometric calculations are performed. Any errors during each session were noted. Landmarking accuracy was estimated by comparing the 'ground-truth position' of landmarks annotated manually to those placed automatically. 18 participants were included in the study, nine healthy participants and nine patients with FNP. Each session was standardized at approximately 106 seconds. 61.8% of landmarks were automatically annotated within approximately 1.575mm of their ground-truth locations. Our findings support that OpenFAS is usable and feasible in routine settings, laying down the critical groundwork for a facial assessment system that addresses the shortcomings of existing tools. However, the iteration of OpenFAS presented in this study is undoubtedly nascent with future work including improvements to landmarking accuracy, analyses components, and RGB-D technology required before clinical application.

Kinect-driven Patient-specific Head, Skull, and Muscle Network Modelling for Facial Palsy Patients

BACKGROUND AND OBJECTIVE

Facial palsy negatively affects both professional and personal life qualities of involved patients. Classical facial rehabilitation strategies can recover facial mimics into their normal and symmetrical movements and appearances. However, there is a lack of objective, quantitative, and in-vivo facial texture and muscle activation bio-feedbacks for personalizing rehabilitation programs and diagnosing recovering progresses. Consequently, this study proposed a novel patient-specific modelling method for generating a full patient specific head model from a visual sensor and then computing the facial texture and muscle activation in real-time for further clinical decision making.

METHODS

The modeling workflow includes (1) Kinect-to-head, (2) head-to-skull, and (3) muscle network definition & generation processes. In the Kinect-to-head process, subject-specific data acquired from a new user in neutral mimic were used for generating his/her geometrical head model with facial texture. In particular, a template head model was deformed to optimally fit with high-definition facial points acquired by the Kinect sensor. Moreover, the facial texture was also merged from his/her facial images in left, right, and center points of view. In the head-to-skull process, a generic skull model was deformed so that its shape was statistically fitted with his/her geometrical head model. In the muscle network definition & generation process, a muscle network was defined from the head and skull models for computing muscle strains during facial movements. Muscle insertion points and muscle attachment points were defined as vertex positions on the head model and the skull model respectively based on the standard facial anatomy. Three healthy subjects and two facial palsy patients were selected for validating the proposed method. In neutral positions, magnetic resonance imaging (MRI)-based head and skull models were compared with Kinect-based head and skull models. In mimic positions, infrared depth-based head models in smiling and [u]-pronouncing mimics were compared with appropriate animated Kinect-driven head models. The Hausdorff distance metric was used for these comparisons. Moreover, computed muscle lengths and strains in the tested facial mimics were validated with reported values in literature.

RESULTS

With the current hardware configuration, the patient-specific head model with skull and muscle network could be fast generated within 17.16±0.37s and animated in real-time with the framerate of 40 fps. In neutral positions, the best mean error was 1.91 mm for the head models and 3.21 mm for the skull models. On facial regions, the best mean errors were 1.53 mm and 2.82 mm for head and skull models respectively. On muscle insertion/attachment point regions, the best mean errors were 1.09 mm and 2.16 mm for head and skull models respectively. In mimic positions, these errors were 2.02 mm in smiling mimics and 2.00 mm in [u]-pronouncing mimics for the head models on facial regions. All above error values were computed on a one-time validation procedure. Facial muscles exhibited muscle shortening and muscle elongating for smiling and pronunciation of sound [u] respectively. Extracted muscle features (i.e. muscle length and strain) are in agreement with experimental and literature data.

CONCLUSIONS

This study proposed a novel modeling method for fast generating and animating patient-specific biomechanical head model with facial texture and muscle activation bio-feedbacks. The Kinect-driven muscle strains could be applied for further real-time muscle-oriented facial paralysis grading and other facial analysis applications.

Quantifying Pediatric Facial Palsy: : Using Digital Image Correlation to Objectively Characterize Pediatric Facial Symmetry

OBJECTIVE

This study introduces digital image correlation (DIC) as a novel technology to objectively quantify pediatric facial symmetry.

DESIGN

Descriptive cohort study of patients' facial symmetry as measured by DIC.

SETTING

Academic tertiary care hospital.

PATIENTS

9 of 12 identified facial palsy and 13 of 26 identified control subjects participated.

INTERVENTIONS

DIC was used to quantify facial strain and symmetry as patients made the 5 standard Sunnybrook facial expressions. Each subject was evaluated according to the Sunnybrook scale by 4 evaluators, 3 plastic surgeons, and 1 occupational therapist.

MAIN OUTCOME MEASURE

The percentage asymmetry values were calculated and compared between the facial palsy and control groups using both DIC and Sunnybrook.

RESULTS

Using DIC, facial palsy subjects had 32.99% asymmetry compared with 14.84% in controls (P < .01). Using Sunnybrook, facial palsy subjects had 24.11% asymmetry compared to 3.87% in controls (P < .01). The 2 metrics were positively correlated (P < .01). There was significant variability among the Sunnybrook evaluators (P = .02).

CONCLUSIONS

DIC is a novel technique of objectively quantifying facial motion of the animated face. As surgical and medical approaches toward facial palsy expand, it is essential to have a means to compare results and improve patient outcomes.

Clinical Efficacy of Electroneurography in Acute Facial Paralysis

The estimated incidence of acute facial paralysis is approximately 30 patients per 100000 populations annually. Facial paralysis is an extremely frightening situation and gives extreme stress to patients because obvious disfiguring face may cause significant functional, aesthetic, and psychological disturbances. For stressful patients with acute facial paralysis, it is very important for clinicians to answer the questions like whether or not their facial function will return to normal, how much of their facial function will be recovered, and how long this is going to take. It is also important for clinicians to treat the psychological aspects by adequately explaining the prognosis, in addition to providing the appropriate medical treatment. For decades, clinicians have used various electrophysiologic tests, including the nerve excitability test, the maximal stimulation test, electroneurography, and electromyography. In particular, electroneurography is the only objective measure that is useful in early stage of acute facial paralysis. In this review article, we first discuss the pathophysiology of injured peripheral nerve. And then, we describe about various electrophysiologic tests and discuss the electroneurography extensively.

A novel three-dimensional smile analysis based on dynamic evaluation of facial curve contour

The influence of three-dimensional facial contour and dynamic evaluation decoding on factors of smile esthetics is essential for facial beauty improvement. However, the kinematic features of the facial smile contour and the contribution from the soft tissue and underlying skeleton are uncharted. Here, the cheekbone-maxilla contour and nasolabial fold were combined into a “smile contour” delineating the overall facial topography emerges prominently in smiling. We screened out the stable and unstable points on the smile contour using facial motion capture and curve fitting, before analyzing the correlation between soft tissue coordinates and hard tissue counterparts of the screened points. Our finding suggests that the mouth corner region was the most mobile area characterizing smile expression, while the other areas remained relatively stable. Therefore, the perioral area should be evaluated dynamically while the static assessment outcome of other parts of the smile contour contribute partially to their dynamic esthetics. Moreover, different from the end piece, morphologies of the zygomatic area and the superior part of the nasolabial crease were determined largely by the skeleton in rest, implying the latter can be altered by orthopedic or orthodontic correction and the former better improved by cosmetic procedures to improve the beauty of smile.

Health-related quality of life in 794 patients with a peripheral facial palsy using the FaCE Scale: a retrospective cohort study

OBJECTIVES

To describe the health-related quality of life of patients visiting a tertiary referral centre for facial palsy, and to analyse factors associated with health-related quality of life, using the FaCE Scale instrument.

DESIGN

Retrospective cohort study.

SETTING

The Facial Nerve Center at the Massachusetts Eye and Ear Infirmary, a tertiary referral centre.

PARTICIPANTS

Patients with a peripheral facial palsy visiting the centre for an initial consultation between August 2007 and June 2012.

MAIN OUTCOME MEASURES

The total FaCE score and the FaCE social function subdomain. Multiple regression models were developed to identify factors associated with the total FaCE score and FaCE social function score.

RESULTS

A total of 794 patients with a mean age of 47.0 ± 16.0 years were analysed in this study, of which 59.9% were female. The mean House-Brackmann, Sunnybrook, total FaCE and FaCE social function scores were 3.6 ± 1.5, 48.2 ± 21.2, 47.3 ± 19.3 and 55.5 ± 19.2, respectively. Increasing age (r = -0.229, P < 0.001) was associated with a lower total FaCE score. Female gender (r = -4.422, P = 0.033) and increased duration of palsy (r = -0.018, P = 0.041) were associated with lower FaCE social function scores.

CONCLUSIONS

While counselling patients on what to expect during the recovery process after facial paralysis is an important part of any clinical visit, FaCE score correlations suggest that female patients with chronic facial palsy and increased age constitute a patient category that may require additional time and attention to prevent or mitigate psychosocial dysfunction.

Quantitative anatomical analysis of facial expression using a 3D motion capture system: Application to cosmetic surgery and facial recognition technology

The topography of the facial muscles differs between males and females and among individuals of the same gender. To explain the unique expressions that people can make, it is important to define the shapes of the muscle, their associations with the skin, and their relative functions. Three-dimensional (3D) motion-capture analysis, often used to study facial expression, was used in this study to identify characteristic skin movements in males and females when they made six representative basic expressions. The movements of 44 reflective markers (RMs) positioned on anatomical landmarks were measured. Their mean displacement was large in males [ranging from 14.31 mm (fear) to 41.15 mm (anger)], and 3.35-4.76 mm smaller in females [ranging from 9.55 mm (fear) to 37.80 mm (anger)]. The percentages of RMs involved in the ten highest mean maximum displacement values in making at least one expression were 47.6% in males and 61.9% in females. The movements of the RMs were larger in males than females but were more limited. Expanding our understanding of facial expression requires morphological studies of facial muscles and studies of related complex functionality. Conducting these together with quantitative analyses, as in the present study, will yield data valuable for medicine, dentistry, and engineering, for example, for surgical operations on facial regions, software for predicting changes in facial features and expressions after corrective surgery, and the development of face-mimicking robots.

A Pilot Study on the Influence of Facial Expression on Measurements in Three-Dimensional Digital Surfaces of the Face in Infants With Cleft Lip and Palate

OBJECTIVE

Three-dimensional surface imaging is an increasingly popular modality for face measurements in infants with cleft lip and palate. Infants are noncompliant toward producing specific facial expressions, and selecting the appropriate moment of acquisition is challenging. The objective was to estimate amount and spatial distribution of deformation of the face due to facial expression in infants with cleft lip and palate and provide recommendations for an improved acquisition protocol, including a method of quality control in terms of obtaining images with true neutral expression.

MATERIAL AND METHODS

Three-dimensional surface images of ten 4-month-old infants with unrepaired cleft lip and palate were obtained using a 3dMDface stereophotogrammetric system. For each subject, five surface images judged as representing a neutral expression were obtained during the same photo session. Mean and maximum deformations were calculated. A formalized review was performed, allowing the image exhibiting the "best" neutral expression to be selected, thus decreasing errors due to residual facial expression.

RESULTS

Deformation due to facial expression generally increased from forehead to chin. The amount of deformation in three selected regions were determined: nose (mean, 1 mm; maximum = 3 mm); cleft region (mean, 2 mm; maximum = 5 mm); chin region (mean, 5 mm; maximum = 12 mm). Analysis indicated that introduction of a formalized review of images could reduce these errors by a factor of 2.

CONCLUSIONS

The continuous change of facial expression in infants represents a substantial source of error; however, this may be reduced by incorporating a formalized review into the acquisition protocol.

Measurement and analysis of associated mimic muscle movements

OBJECTIVE

To measure movements of markers over the primary site and associated mimic muscles in certain facial expressions, for evaluating facial paresis and synkinesis.

METHODS

Participants included 22 normal subjects aged 45-66 years. Maximum shift (Smax) and velocity (Vmax) were measured using a custom-designed 3-D dynamic quantitative analysis system of facial motion (3-D ASFM) based on motion capture technology. Measures were taken from peri-oral muscles during forceful brow raising and tight eye closure, and from muscles around the eye during grinning, right/left/bilateral mouth corner raising and smiling.

RESULTS

1) During forceful brow raising, Smax was 3.65-4.46 mm for markers over perioral muscles, with the marker over the nasolabial fold showing a Vmax greater than others (60.60 mm/s on left and 62.70 mm/s on right). 2) In tight eye closure, Smax of perioral muscle markers was 1.58-1.92 mm, with Vmax being 11.40-14.76 mm/s. 3) In grinning, the largest eye muscle marker Smax was seen at the lower lid (3.93 mm on left and 4.15 mm on right) and the smallest at the inner canthus (1.59 mm on left and 1.53 mm on right), with the largest Vmax seen at the upper lid and smallest also at the inner canthus (11.71 mm/s on left and 11.09 mm/s on right). 4) In smiling, the largest non-oral Smax and Vmax were seen at the upper lid (3.05 mm and 36.14 mm/s on left and 2.53 mm and 28.90 mm/s on right) and the smallest also at the inner canthus (0.69 mm and 7.22 mm/s on left and 0.77 mm and 7.80 mm/s on right). 5) In right mouth corner raising, Smax and Vmax at lateral and medial canthus and at lower lid were greater on right than left, while those at upper lid and brow were slightly greater on left than right. 6) In left mouth corner raising, Smax and Vmax at lateral canthus and upper and lower lids were greater on left than right.

CONCLUSIONS

There are no absolute immobile points on the face when making facial expressions. In addition to the primary movement site, there are associated movements at other points on the face with consistent Smax and Vmax. In assessing facial paresis and synkinesis, physiological associated facial movements should be taken into consideration.

The clinical application of three-dimensional motion capture (4D): a novel approach to quantify the dynamics of facial animations

The aim of this pilot study was to evaluate the feasibility of measuring the change in magnitude, speed, and motion similarity of facial animations in head and neck oncology patients, before and after lip split mandibulotomy. Seven subjects (four males, three females) aged 42-80 years were recruited. The subjects were asked to perform four facial animations (maximal smile, lip purse, cheek puff, and grimace) from rest to maximal position. The animations were captured using a Di4D motion capture system, which recorded 60 frames/s. Nine facial soft tissue landmarks were manually digitized on the first frame of the three-dimensional image of each animation by the same operator and were tracked automatically for the sequential frames. The intra-operator digitization error was within 0.4mm. Lip purse and maximal smile animations showed the least amount of change in magnitude (0.2mm) following surgery; speed difference was least for smile animation (-0.1mm/s). Motion similarity was found to be highest for lip purse animation (0.78). This pilot study confirmed that surgery did influence the dynamics of facial animations, and the Di4D capture system can be regarded as a feasible objective tool for assessing the impact of surgical interventions on facial soft tissue movements.