Review of the Augmented Reality Systems for Shoulder Rehabilitation

Head-Mounted Displays in Context-Aware Systems for Open Surgery: A State-of-the-Art Review

Surgical context-aware systems (SCAS), which leverage real-time data and analysis from the operating room to inform surgical activities, can be enhanced through the integration of head-mounted displays (HMDs). Rather than user-agnostic data derived from conventional, and often static, external sensors, HMD-based SCAS relies on dynamic user-centric sensing of the surgical context. The analyzed context-aware information is then augmented directly into a user's field of view via augmented reality (AR) to directly improve their task and decision-making capability. This state-of-the-art review complements previous reviews by exploring the advancement of HMD- based SCAS, including their development and impact on enhancing situational awareness and surgical outcomes in the operating room. The survey demonstrates that this technology can mitigate risks associated with gaps in surgical expertise, increase procedural efficiency, and improve patient outcomes. We also highlight key limitations still to be addressed by the research community, including improving prediction accuracy, robustly handling data heterogeneity, and reducing system latency.

Mixed Reality-Based Smart Occupational Therapy Personalized Protocol for Cerebellar Ataxic Patients

BACKGROUND

Occupational therapy (OT) is an essential component of patient care, and it is especially beneficial if focused on meaningful activities. For ataxic patients, traditional procedures are currently the most efficient, although without specific guidelines and suggestions for virtual reality integration. In this context, this study proposes Hybrid Smart Rehabilitation (HSR) based on mixed reality (MR) as an aid in overcoming limitations of the traditional OT procedures.

METHODS

MR-HSR is designed specifically for ataxic patients and developed in Unity with the Holographic Remoting setting for run-time intervention on the scene. The subject reaches a book and grabs it with their hand inside a holographic guide with audio-visive feedback. Hand trajectories acquired from eight ataxic patients and eight healthy subjects were compared and new variables were analyzed to evaluate the performance. The Trust in Automation questionnaire was submitted to assess the opinion of the patients.

RESULTS

Patients confirmed their trust in the developer and in the improvement that this system can bring to their rehabilitation. The "total time" and "sway area" of the trajectory were statistically significant and, together with the deviation of the trajectory from the main axis of the guide, although not statistically significant, made it possible to build a classifier.

CONCLUSIONS

The patient-specific MR-HSR can be considered as an integrative tool for assessing the subject's condition by analyzing new quantitative variables which, if matched to the Scale for the Assessment and Rating of Ataxia (SARA), could be the basis of a new index to assess the progressiveness of ataxia.

Using Capability Maps Tailored to Arm Range of Motion in VR Exergames for Rehabilitation

Many neurological conditions, e.g., a stroke, can cause patients to experience upper limb (UL) motor impairments that hinder their daily activities. For such patients, while rehabilitation therapy is key for regaining autonomy and restoring mobility, its long-term nature entails ongoing time commitment and it is often not sufficiently engaging. Virtual reality (VR) can transform rehabilitation therapy into engaging game-like tasks that can be tailored to patient-specific activities, set goals, and provide rehabilitation assessment. Yet, most VR systems lack built-in methods to track progress over time and alter rehabilitation programs accordingly. We propose using arm kinematic modeling and capability maps to allow a VR system to understand a user's physical capability and limitation. Next, we suggest two use cases for the VR system to utilize the user's capability map for tailoring rehabilitation programs. Finally, for one use case, it is shown that the VR system can emphasize and assess the use of specific UL joints.Clinical relevance-This paper's VR-based system can tailor a rehabilitation tool to a user's capability and limit.

Visual Cue Based Corrective Feedback for Motor Skill Training in Mixed Reality: A Survey

When learning a motor skill it is helpful to get corrective feedback from an instructor. This will support the learner to execute the movement correctly. With modern technology, it is possible to provide this feedback via mixed reality. In most cases, this involves visual cues to help the user understand the corrective feedback. We analyzed recent research approaches utilizing visual cues for feedback in mixed reality. The scope of this article is visual feedback for motor skill learning, which involves physical therapy, exercise, rehabilitation etc. While some of the surveyed literature discusses therapeutic effects of the training, this article focuses on visualization techniques. We categorized the literature from a visualization standpoint, including visual cues, technology and characteristics of the feedback. This provided insights into how visual feedback in mixed reality is applied in the literature and how different aspects of the feedback are related. The insights obtained can help to better adjust future feedback systems to the target group and their needs. This article also provides a deeper understanding of the characteristics of the visual cues in general and promotes future, more detailed research on this topic.

A deep learning system to monitor and assess rehabilitation exercises in home-based remote and unsupervised conditions

In the domain of physical rehabilitation, the progress in machine learning and the availability of cost-effective motion capture technologies have paved the way for innovative systems capable of capturing human movements, automatically analyzing recorded data, and evaluating movement quality. This study introduces a novel, economically viable system designed for monitoring and assessing rehabilitation exercises. The system enables real-time evaluation of exercises, providing precise insights into deviations from correct execution. The evaluation comprises two significant components: range of motion (ROM) classification and compensatory pattern recognition. To develop and validate the effectiveness of the system, a unique dataset of 6 resistance training exercises was acquired. The proposed system demonstrated impressive capabilities in motion monitoring and evaluation. Notably, we achieved promising results, with mean accuracies of 89% for evaluating ROM-class and 98% for classifying compensatory patterns. By complementing conventional rehabilitation assessments conducted by skilled clinicians, this cutting-edge system has the potential to significantly improve rehabilitation practices. Additionally, its integration in home-based rehabilitation programs can greatly enhance patient outcomes and increase access to high-quality care.

Home-Based Rehabilitation of the Shoulder Using Auxiliary Systems and Artificial Intelligence: An Overview

Advancements in modern medicine have bolstered the usage of home-based rehabilitation services for patients, particularly those recovering from diseases or conditions that necessitate a structured rehabilitation process. Understanding the technological factors that can influence the efficacy of home-based rehabilitation is crucial for optimizing patient outcomes. As technologies continue to evolve rapidly, it is imperative to document the current state of the art and elucidate the key features of the hardware and software employed in these rehabilitation systems. This narrative review aims to provide a summary of the modern technological trends and advancements in home-based shoulder rehabilitation scenarios. It specifically focuses on wearable devices, robots, exoskeletons, machine learning, virtual and augmented reality, and serious games. Through an in-depth analysis of existing literature and research, this review presents the state of the art in home-based rehabilitation systems, highlighting their strengths and limitations. Furthermore, this review proposes hypotheses and potential directions for future upgrades and enhancements in these technologies. By exploring the integration of these technologies into home-based rehabilitation, this review aims to shed light on the current landscape and offer insights into the future possibilities for improving patient outcomes and optimizing the effectiveness of home-based rehabilitation programs.

Rehabilitation of Burn Victims: Improving Quality of Life in Victims With Face and Neck Burn Through an Augmented Reality Coupled Pamphlet

The present study aimed to investigate the effect of implementing rehabilitation programs using an augmented reality (AR) coupled pamphlet on the quality of life (QOL) of patients with face/neck burns. This randomized clinical trial was conducted on 60 patients (intervention = 30, control = 30) admitted to the burn center of Imam Reza Hospital, Mashhad, Iran. Patients in the intervention group performed their rehabilitation program using AR coupled pamphlet during 6 weeks, while control patients used simple pamphlet. The Burn-Specific Health Scale for Face and Neck (BSHS-FN) was completed at the beginning, and 2 and 6 weeks after the intervention. The results for 2 weeks after intervention measures in model 1 ANCOVA showed significant raise of measures only for Hand function (P = .035). However, for the total QOL score and other domains the differences were not significant (All P > .05). Adjusting for confounders, model 2 ANCOVA showed similar results for 2 weeks after intervention. However, in both models, the results indicated significant intervention effect for 6 weeks after intervention in QOL score and all domains (All P < .05), so that the amount of increase in measures were significantly higher in the intervention group. According to the findings, implementing rehabilitation program for patients with face and neck burns using AR coupled pamphlet can improve their QOL.

An augmented reality interface to control a collaborative robot in rehab: A preliminary usability evaluation

Human emotions can be seen as a valuable variable to explore in Human-Computer Interaction for effective, efficient, and satisfying interface development. The inclusion of appropriate emotional triggers in the design of interactive systems can play a decisive role in users' acceptance or rejection. It is well known that the major problem in motor rehabilitation is the high dropout rate resulting from the frustrated expectations given the typical slow recovery process and consequent lack of motivation to endure. This work proposes grouping a collaborative robot with one specific augmented reality equipment to create a rehabilitation system where some gamification levels might be added to provide a better and more motivating experience to patients. Such a system, as a whole, is customizable to adapt to each patient's needs on the rehabilitation exercises. By transforming a tedious exercise into a game, we expect to create an additional layer of enjoyment that can help in triggering positive emotions and stimulate users to continue the rehabilitation process. A pre-prototype was developed to validate this system's usability, and a cross-sectional study using a non-probabilistic sample of 31 individuals is presented and discussed. This study included the application of three standard questionnaires on usability and user experience. The analyses of these questionnaires show that the majority of the users found the system easy and enjoyable. The system was also analysed by a rehabilitation expert who gave a positive output regarding its usefulness, and positive impact on its use in the upper-limb rehabilitation processes. These results clearly encourage further development of the proposed system.

Digital rehabilitation for hand and wrist pain: a single-arm prospective longitudinal cohort study

Supplemental Digital Content is Available in the Text.

This study supports that a fully remote digital care program is feasible and able to promote high patient engagement in the telerehabilitation of patients with wrist and hand pain.

Introduction:

Wrist and hand represent the third most common body part in work-related injuries, being associated with long-term absenteeism. Telerehabilitation can promote access to treatment, patient adherence, and engagement, while reducing health care–related costs.

Objective:

Report the results of a fully remote digital care program (DCP) for wrist and hand pain (WP).

Methods:

A single-arm interventional study was conducted on individuals with WP applying for a DCP. Primary outcome was the mean change in the Numerical Pain Rating Scale after 8 weeks (considering a minimum clinically important change of 30%). Secondary outcomes were: disability (Quick Disabilities of the Arm, Shoulder, and Hand questionnaire), analgesic intake, surgery intention, mental health (patient health questionnaire [PHQ-9] and generalized anxiety disorder [GAD-7]), fear-avoidance beliefs (FABQ-PA), work productivity and activity impairment, and engagement.

Results:

From 189 individuals starting the DCP, 149 (78.8%) completed the intervention. A significant pain improvement was observed (51.3% reduction (2.26, 95% CI 1.73; 2.78)) and 70.4% of participants surpassing minimum clinically important change. This change correlated with improvements in disability (52.1%), FABQ-PA (32.2%), and activities impairment recovery (65.4%). Improvements were also observed in other domains: surgery intent (76.1%), mental health (67.0% in anxiety and 72.7% in depression), and overall productivity losses (68.2%). Analgesic intake decreased from 22.5% to 7.1%. Mean patient satisfaction score was 8.5/10.0 (SD 1.8).

Conclusions:

These findings support the feasibility and utility of a fully remote DCP for patients with WP. Clinically significant improvements were observed in all health-related and productivity-related outcomes, alongside very high patient adherence rates and satisfaction. This study strengthens that management of WP is possible through a remote DCP, decreasing access barriers and potentially easing health care expenditure.

Augmenting Performance: A Systematic Review of Optical See-Through Head-Mounted Displays in Surgery

We conducted a systematic review of recent literature to understand the current challenges in the use of optical see-through head-mounted displays (OST-HMDs) for augmented reality (AR) assisted surgery. Using Google Scholar, 57 relevant articles from 1 January 2021 through 18 March 2022 were identified. Selected articles were then categorized based on a taxonomy that described the required components of an effective AR-based navigation system: data, processing, overlay, view, and validation. Our findings indicated a focus on orthopedic (n=20) and maxillofacial surgeries (n=8). For preoperative input data, computed tomography (CT) (n=34), and surface rendered models (n=39) were most commonly used to represent image information. Virtual content was commonly directly superimposed with the target site (n=47); this was achieved by surface tracking of fiducials (n=30), external tracking (n=16), or manual placement (n=11). Microsoft HoloLens devices (n=24 in 2021, n=7 in 2022) were the most frequently used OST-HMDs; gestures and/or voice (n=32) served as the preferred interaction paradigm. Though promising system accuracy in the order of 2–5 mm has been demonstrated in phantom models, several human factors and technical challenges—perception, ease of use, context, interaction, and occlusion—remain to be addressed prior to widespread adoption of OST-HMD led surgical navigation.

Effectiveness of Augmented Reality for Lower Limb Rehabilitation: A Systematic Review

Augmented reality- (AR-) based interventions have shown potential benefits for lower limb rehabilitation. However, current literature has not revealed these benefits as a whole. The main purposes of this systematic review were to determine the efficacy of AR-based interventions on lower limb recovery of the larger population based on the current process that has been made in this regard. Relevant studies were retrieved from five electronic databases (Web of Science, PubMed, ScienceDirect, Scopus, and Cochrane Library) using “augmented reality” OR “AR” AND “lower limb” OR “lower extremity” AND “intervention” OR “treatment”. Sixteen studies that met the eligibility criteria were included in this review, and they were further grouped into three categories based on the participant types. Seven studies focused on the elderly adults, six on the stroke patients, and the last three on Parkinson patients. Based on the findings of these trials, the significant effects of AR-based interventions on lower limb rehabilitation (i.e., balance, gait, muscle, physical performance, and fall efficacy) have been initially confirmed. Favorable results were achieved at least the same as the interventions without AR except for the turning and timing in the freezing of gait of Parkinson patients. However, given the infancy of this technology in clinical practices, more robust trials with larger sample sizes and greater homogeneity in terms of devices and treatment settings are warranted for further verification.

“Sport and Anatomy”: Teaching, Research, and Assistance at the University of Pisa

Introduction: Over the last decades, the university system has experienced huge growth, facing several challenges. Accordingly, the University of Pisa recognizes the value and opportunities deriving from research and fully supports collaboration with the world of entrepreneurship and industry, as well as local communities. Study programs, teaching methods and technologies, learning environments, quality assurance, programmed student numbers, and research results are key features of the prestige of the scientific community. Aim: In this respect, “Sport and Anatomy”, a brand that includes an academic organization at the University of Pisa, holds two main goals: (i) to offer the top level in both educational and professional fields; and (ii) to optimize the fine-tuning among all these sections, thus becoming a reference point for sports management. Methods and results: Indispensable links between basic and specialist sciences through different Masters’ and schools were created. In addition to didactic activity, research activity, medical assistance, and rehabilitation were coordinated. Two main outcomes emerged from this experience: (i) improved stakeholder performances and (ii) optimized cooperation between university and local communities. Conclusions: “Sport and Anatomy” plays a key role in supervising and accomplishing in an innovative way all the three missions of the university (i.e., teaching, research, and dissemination of knowledge), thus strongly fulfilling the aims of modern university targets.

What is Significant in Modern Augmented Reality: A Systematic Analysis of Existing Reviews

Augmented reality (AR) is a field of technology that has evolved drastically during the last decades, due to its vast range of applications in everyday life. The aim of this paper is to provide researchers with an overview of what has been surveyed since 2010 in terms of AR application areas as well as in terms of its technical aspects, and to discuss the extent to which both application areas and technical aspects have been covered, as well as to examine whether one can extract useful evidence of what aspects have not been covered adequately and whether it is possible to define common taxonomy criteria for performing AR reviews in the future. To this end, a search with inclusion and exclusion criteria has been performed in the Scopus database, producing a representative set of 47 reviews, covering the years from 2010 onwards. A proper taxonomy of the results is introduced, and the findings reveal, among others, the lack of AR application reviews covering all suggested criteria.

Effectiveness of Augmented Reality in Stroke Rehabilitation: A Meta-Analysis

Augmented reality (AR)-based rehabilitation shows potential to improve upper and lower limb function after stroke. This study aims to review the effect of AR technology in the recovery of the upper and lower limb function in stroke patients. Published randomized controlled trials and observational investigations with adult stroke patients were retrieved from five electronic databases to analyze the effect of the AR systems in improving motor function and balance and gait function for stroke patients. The treatment effect was estimated by standardized mean difference (SMD) and 95% confidence interval (CI) using a random effect model for motor function outcomes at the body structure and function, body activity and participation level of the International Classification of Functioning, and balance and gait outcomes. In total, 13 investigations (9 for the upper limb and 4 for the lower limb) were identified. AR demonstrated a significant influence on the upper limb function (SMD = 0.657; 95% CI, 0.287 to 1.026; p = 0.000) and the lower limb function (SMD = 0.52; 95% CI, 0.039 to 1.001; p = 0.034). The present analysis suggests that AR applications could offer options for increasing treatment intensity and promoting motor recovery after a stroke. This approach can be used with the conventional rehabilitation methods as a new intervention for recovering upper and lower limb function.

Instrumental Validity of the Motion Detection Accuracy of a Smartphone-Based Training Game

Demographic changes associated with an expanding and aging population will lead to an increasing number of orthopedic surgeries, such as joint replacements. To support patients’ home exercise programs after total hip replacement and completing subsequent inpatient rehabilitation, a low-cost, smartphone-based augmented reality training game (TG) was developed. To evaluate its motion detection accuracy, data from 30 healthy participants were recorded while using the TG. A 3D motion analysis system served as reference. The TG showed differences of 18.03 mm to 24.98 mm along the anatomical axes. Surveying the main movement direction of the implemented exercises (squats, step-ups, side-steps), differences between 10.13 mm to 24.59 mm were measured. In summary, the accuracy of the TG’s motion detection is sufficient for use in exergames and to quantify progress in patients’ performance. Considering the findings of this study, the presented exer-game approach has potential as a low-cost, easily accessible support for patients in their home exercise program.

Monitoring Wound Healing With Contactless Measurements and Augmented Reality

Objective: This work presents a device for non-invasive wound parameters assessment, designed to overcome the drawbacks of traditional methods, which are mostly rough, inaccurate, and painful for the patient. The device estimates the morphological parameters of the wound and provides augmented reality (AR) visual feedback on the wound healing status by projecting the wound border acquired during the last examination, thus improving doctor-patient communication. Methods: An accurate 3D model of the wound is created by stereophotogrammetry and refined through self-organizing maps. The 3D model is used to estimate physical parameters for wound healing assessment and integrates AR functionalities based on a miniaturized projector. The physical parameter estimation functionalities are evaluated in terms of precision, accuracy, inter-operator variability, and repeatability, whereas AR wound border projection is evaluated in terms of accuracy on the same phantom. Results: The accuracy and precision of the device are respectively 2% and 1.2% for linear parameters, and 1.7% and 1.3% for area and volume. The AR projection shows an error distance <1 mm. No statistical difference was found between the measurements of different operators. Conclusion: The device has proven to be an objective and non-operator-dependent tool for assessing the morphological parameters of the wound. Comparison with non-contact devices shows improved accuracy, offering reliable and rigorous measurements. Clinical Impact: Chronic wounds represent a significant health problem with high recurrence rates due to the ageing of the population and diseases such as diabetes and obesity. The device presented in this work provides an easy-to-use non-invasive tool to obtain useful information for treatment.

Wearable Augmented Reality Application for Shoulder Rehabilitation

Augmented reality (AR) technology is gaining popularity and scholarly interest in the rehabilitation sector because of the possibility to generate controlled, user-specific environmental and perceptual stimuli which motivate the patient, while still preserving the possibility to interact with the real environment and other subjects, including the rehabilitation specialist. The paper presents the first wearable AR application for shoulder rehabilitation, based on Microsoft HoloLens, with real-time markerless tracking of the user’s hand. Potentialities and current limits of commercial head-mounted displays (HMDs) are described for the target medical field, and details of the proposed application are reported. A serious game was designed starting from the analysis of a traditional rehabilitation exercise, taking into account HoloLens specifications to maximize user comfort during the AR rehabilitation session. The AR application implemented consistently meets the recommended target frame rate for immersive applications with HoloLens device: 60 fps. Moreover, the ergonomics and the motivational value of the proposed application were positively evaluated by a group of five rehabilitation specialists and 20 healthy subjects. Even if a larger study, including real patients, is necessary for a clinical validation of the proposed application, the results obtained encourage further investigations and the integration of additional technical features for the proposed AR application.

Wrist Rehabilitation System Using Augmented Reality for Hemiplegic Stroke Patient Rehabilitation: A Feasibility Study

Objective: Our objective was to investigate the effect of the rehabilitation system using augmented reality (AR) on upper extremity motor performance of patients with stroke. Methods: The system using AR applying mirror therapy mechanism provides the intervention protocol for the patient with hemiplegia after stroke. The system consists of a patient positioning tool (a chair), a white surface table, an image acquisition unit, an image processing unit, an image displaying unit, an arm holder, a Velcro-strap, and two blue circle stickers. To assess the feasibility of our system in motor function recovery, a stroke patient was recruited to receive the AR intervention. The treatment was performed two times a day for ten minutes over two weeks (ten days of treating weeks), except for the time of installation, calibration, and three minute breaks. Jebsen Taylor hand function test and Arm Motor Fugl-Meyer assessment were used as primary and secondary outcome measures, respectively, to evaluate the effect of motor function recovery. Additionally, stroke impact scale, Korean version-Modified Barthel Index (K-MBI), active range of motion of wrist joint (ROM), and the grasp force in Newtons were measured. Participants’ feedback and adverse effects were recorded as well. Results: Motor function improvements were exhibited in wrist and hand subtest of Arm Motor Fugl-Meyer (baseline: 19; post-intervention: 23), proximal arm subtest of Fugl-Meyer (baseline: 31; post-intervention: 34), ROM (extending ROM: 10° and 3° for flexion and extension, repeatedly), stroke impact scale (baseline: 46; post-intervention: 54), K-MBI (baseline: 92; post-intervention: 95), nine-hole pegboard (baseline: 30 s; post-intervention: 25 s), and grasp force in Newtons (baseline: 12.7; post-intervention: 17.7). However, the adverse effects were reported after the intervention. Conclusion: The system using AR applying mirror therapy mechanism demonstrated the feasibility in motor function recovery for the stroke patient.