Wearable Augmented Reality Application for Shoulder Rehabilitation

Serious Gaming for Upper Limbs Rehabilitation-Game Controllers Features: A Scoping Review

The use of exergames in the rehabilitation of patients with upper limb dysfunctions has increased significantly. This scoping review aimed to investigate the game controllers (GCs) employed in exergame systems used for rehabilitation, offering insights into the platforms, sensors, and techniques used in their development, implementation, and utilization. We conducted a comprehensive search of Scopus and PubMed databases, encompassing articles published between February 2013 and February 2023. The eligibility criteria included studies on upper limbs (UL) rehabilitation using exergames published in English-language journals, resulting in the identification of 175 pertinent articles. Seven key categories were identified: pathology, participants' conditions, dosage of sessions, GCs, sensors, specific part of the UL rehabilitated, and ergonomics. Stroke (55.4%) and cerebral palsy (6.3%) were the most frequently addressed medical conditions in the exergame-based rehabilitation. The number of participants in the reviewed articles was from one to several hundred. Three types of participants were identified: patients, specialists, and volunteers. Randomized controlled trial (RCT) studies consistently featured a controlled number of sessions (ranging from 6 to 40) lasting an average of 20 minutes, while non-RCT studies displayed more variability. Commercial platforms were favored, accounting for 74.3% of GCs, with physical controllers (57.1%) surpassing virtual ones. Cameras were the predominant sensors (50.3%), although a wide array of sensor types including IMUs, push buttons, and force sensors were also used. Rehabilitation focuses 68% on general UL, 20.6% on hands, 4% on elbows, and 3.4% on arms and shoulders. Notably, only 26.3% of the studies considered ergonomics in the rehabilitation system. Although exergame systems are advancing rehabilitation treatments, there remains a need for further development and research on various aspects, such as ergonomics, controller design, and sensor integration, to enhance their suitability for patient use.

Synergy-Based Evaluation of Hand Motor Function in Object Handling Using Virtual and Mixed Realities

This study introduces a novel system for evaluating hand motor function through synergy-based analysis during object manipulation in virtual and mixed-reality environments. Conventional assessments of hand function are often subjective, relying on visual observation by therapists or patient-reported outcomes. To address these limitations, we developed a system that utilizes the leap motion controller (LMC) to capture finger motion data without the constraints of glove-type devices. Spatial synergies were extracted using principal component analysis (PCA) and Varimax rotation, providing insights into finger motor coordination with the sparse decomposition. Additionally, we incorporated the HoloLens 2 to create a mixed-reality object manipulation task that enhances spatial awareness for the user, improving natural interaction with virtual objects. Our results demonstrate that synergy-based analysis allows for the systematic detection of hand movement abnormalities that are not captured through traditional task performance metrics. This system demonstrates promise in advancing rehabilitation by enabling more objective and detailed evaluations of finger motor function, facilitating personalized therapy, and potentially contributing to the early detection of motor impairments in the future.

Emerging Applications of Augmented and Mixed Reality Technologies in Motor Rehabilitation: A Scoping Review

BACKGROUND

Augmented Reality (AR) and Mixed Reality (MR) are emerging technologies with notable potential for motor rehabilitation. Given the novelty and breadth of this field, this scoping review aims to identify how and to what extent AR and MR technologies are used in motor rehabilitation.

METHODS

We conducted a search in Scopus and PubMed (2010-2024), following PRISMA-ScR guidelines. In the analysis, we focused on four key aspects: (I) the AR/MR display technologies, (II) the sensors used to collect data to generate the augmented information, (III) the pathologies addressed, and (IV) the assessment of usability and acceptability.

RESULTS

Among 105 selected studies, 58% developed new prototypes, while 42% tested existing systems. Head-mounted displays were the most common device (56.2%), followed by monitors (34.3%) and video projectors (14.3%). The most commonly used sensors were RGB-D cameras (31.4%), sensors for localization and mapping (33.3%), normal cameras (17.1%), and electromyography sensors (14.3%). Regarding the target pathology, 34.2% of studies did not focus on a specific pathology, 26.7% were on stroke, 10.5% on limb loss, and 9.5% on Parkinson's disease. Over half (51.4%) of the studies investigated usability and acceptance.

CONCLUSIONS

AR/MR technologies hold promise for motor rehabilitation, but limited comparative studies and long-term investigations currently hinder a clear understanding of their benefits.

Integrated Quantitative Evaluation of Spatial Cognition and Motor Function with HoloLens Mixed Reality

The steady increase in the aging population worldwide is expected to cause a shortage of doctors and therapists for older people. This demographic shift requires more efficient and automated systems for rehabilitation and physical ability evaluations. Rehabilitation using mixed reality (MR) technology has attracted much attention in recent years. MR displays virtual objects on a head-mounted see-through display that overlies the user's field of vision and allows users to manipulate them as if they exist in reality. However, tasks in previous studies applying MR to rehabilitation have been limited to tasks in which the virtual objects are static and do not interact dynamically with the surrounding environment. Therefore, in this study, we developed an application to evaluate cognitive and motor functions with the aim of realizing a rehabilitation system that is dynamic and has interaction with the surrounding environment using MR technology. The developed application enabled effective evaluation of the user's spatial cognitive ability, task skillfulness, motor function, and decision-making ability. The results indicate the usefulness and feasibility of MR technology to quantify motor function and spatial cognition both for static and dynamic tasks in rehabilitation.

Home-Based Rehabilitation of the Shoulder Using Auxiliary Systems and Artificial Intelligence: An Overview

Advancements in modern medicine have bolstered the usage of home-based rehabilitation services for patients, particularly those recovering from diseases or conditions that necessitate a structured rehabilitation process. Understanding the technological factors that can influence the efficacy of home-based rehabilitation is crucial for optimizing patient outcomes. As technologies continue to evolve rapidly, it is imperative to document the current state of the art and elucidate the key features of the hardware and software employed in these rehabilitation systems. This narrative review aims to provide a summary of the modern technological trends and advancements in home-based shoulder rehabilitation scenarios. It specifically focuses on wearable devices, robots, exoskeletons, machine learning, virtual and augmented reality, and serious games. Through an in-depth analysis of existing literature and research, this review presents the state of the art in home-based rehabilitation systems, highlighting their strengths and limitations. Furthermore, this review proposes hypotheses and potential directions for future upgrades and enhancements in these technologies. By exploring the integration of these technologies into home-based rehabilitation, this review aims to shed light on the current landscape and offer insights into the future possibilities for improving patient outcomes and optimizing the effectiveness of home-based rehabilitation programs.

Towards Wearable Augmented Reality in Healthcare: A Comparative Survey and Analysis of Head-Mounted Displays

Head-mounted displays (HMDs) have the potential to greatly impact the surgical field by maintaining sterile conditions in healthcare environments. Google Glass (GG) and Microsoft HoloLens (MH) are examples of optical HMDs. In this comparative survey related to wearable augmented reality (AR) technology in the medical field, we examine the current developments in wearable AR technology, as well as the medical aspects, with a specific emphasis on smart glasses and HoloLens. The authors searched recent articles (between 2017 and 2022) in the PubMed, Web of Science, Scopus, and ScienceDirect databases and a total of 37 relevant studies were considered for this analysis. The selected studies were divided into two main groups; 15 of the studies (around 41%) focused on smart glasses (e.g., Google Glass) and 22 (59%) focused on Microsoft HoloLens. Google Glass was used in various surgical specialities and preoperative settings, namely dermatology visits and nursing skill training. Moreover, Microsoft HoloLens was used in telepresence applications and holographic navigation of shoulder and gait impairment rehabilitation, among others. However, some limitations were associated with their use, such as low battery life, limited memory size, and possible ocular pain. Promising results were obtained by different studies regarding the feasibility, usability, and acceptability of using both Google Glass and Microsoft HoloLens in patient-centric settings as well as medical education and training. Further work and development of rigorous research designs are required to evaluate the efficacy and cost-effectiveness of wearable AR devices in the future.

An augmented reality interface to control a collaborative robot in rehab: A preliminary usability evaluation

Human emotions can be seen as a valuable variable to explore in Human-Computer Interaction for effective, efficient, and satisfying interface development. The inclusion of appropriate emotional triggers in the design of interactive systems can play a decisive role in users' acceptance or rejection. It is well known that the major problem in motor rehabilitation is the high dropout rate resulting from the frustrated expectations given the typical slow recovery process and consequent lack of motivation to endure. This work proposes grouping a collaborative robot with one specific augmented reality equipment to create a rehabilitation system where some gamification levels might be added to provide a better and more motivating experience to patients. Such a system, as a whole, is customizable to adapt to each patient's needs on the rehabilitation exercises. By transforming a tedious exercise into a game, we expect to create an additional layer of enjoyment that can help in triggering positive emotions and stimulate users to continue the rehabilitation process. A pre-prototype was developed to validate this system's usability, and a cross-sectional study using a non-probabilistic sample of 31 individuals is presented and discussed. This study included the application of three standard questionnaires on usability and user experience. The analyses of these questionnaires show that the majority of the users found the system easy and enjoyable. The system was also analysed by a rehabilitation expert who gave a positive output regarding its usefulness, and positive impact on its use in the upper-limb rehabilitation processes. These results clearly encourage further development of the proposed system.

A Proof of Concept Combined Using Mixed Reality for Personalized Neurorehabilitation of Cerebellar Ataxic Patients

BACKGROUND

Guidelines for degenerative cerebellar ataxia neurorehabilitation suggest intensive coordinative training based on physiotherapeutic exercises. Scientific studies demonstrate virtual exergaming therapeutic value. However, patient-based personalization, post processing analyses and specific audio-visual feedbacks are not provided. This paper presents a wearable motion tracking system with recording and playback features. This system has been specifically designed for ataxic patients, for upper limbs coordination studies with the aim to retrain movement in a neurorehabilitation setting. Suggestions from neurologists and ataxia patients were considered to overcome the shortcomings of virtual systems and implement exergaming.

METHODS

The system consists of the mixed-reality headset Hololens2 and a proprietary exergaming implemented in Unity. Hololens2 can track and save upper limb parameters, head position and gaze direction in runtime.

RESULTS

Data collected from a healthy subject are reported to demonstrate features and outputs of the system.

CONCLUSIONS

Although further improvements and validations are needed, the system meets the needs of a dynamic patient-based exergaming for patients with cerebellar ataxia. Compared with existing solutions, the mixed-reality system is designed to provide an effective and safe therapeutic exergaming that supports both primary and secondary goals of an exergaming: what a patient should do and how patient actions should be performed.

Design and Validation of Vision-Based Exercise Biofeedback for Tele-Rehabilitation

Tele-rehabilitation has the potential to considerably change the way patients are monitored from their homes during the care process, by providing equitable access without the need to travel to rehab centers or shoulder the high cost of personal in-home services. Developing a tele-rehab platform with the capability of automating exercise guidance is likely to have a significant impact on rehabilitation outcomes. In this paper, a new vision-based biofeedback system is designed and validated to identify the quality of performed exercises. This new system will help patients to refine their movements to get the most out of their plan of care. An open dataset was used, which consisted of data from 30 participants performing nine different exercises. Each exercise was labeled as "Correctly" or "Incorrectly" executed by five clinicians. We used a pre-trained 3D Convolution Neural Network (3D-CNN) to design our biofeedback system. The proposed system achieved average accuracy values of 90.57% ± 9.17% and 83.78% ± 7.63% using 10-Fold and Leave-One-Subject-Out (LOSO) cross validation, respectively. In addition, we obtained average F1-scores of 71.78% ± 5.68% using 10-Fold and 60.64% ± 21.3% using LOSO validation. The proposed 3D-CNN was able to classify the rehabilitation videos and feedback on the quality of exercises to help users modify their movement patterns.

Bibliometric and visualized analysis of scientific publications on rehabilitation of rotator cuff injury based on web of science

Background

Since the discovery of rehabilitation as an intervention for rotator cuff injury, its impact on the recovery of rotator cuff injury has attracted crucial attention, and the number of related studies is increasing worldwide. There were no bibliometric and visualized analysis studies in this field. This study aimed to investigate the research hotpots and trends in the rehabilitation of rotator cuff injury via bibliometric and visualized analysis and to identify the future development of clinical practice.

Method

The publications regarding rehabilitation of rotator cuff injury from inception to December 2021 were obtained from the Web of Science Core Collection database. The trends of publications, co-authorship and co-occurrence analysis and visualized analysis were carried out using Citespace, VOSviewer, Scimago Graphica software, and R Project.

Results

A total of 795 publications were included in this study. The number of publications significantly increased yearly. The United States published the highest number of related papers and the papers published by the United States had the highest citations. The University of Laval, the University of Montreal and Keele University were the top 3 most contributive institutions. Additionally, the Journal of Shoulder and Elbow Surgery was the journal with the highest number of publications. The most common keywords were "rotator cuff", "rehabilitation", "physical therapy", "management", and "telerehabilitation".

Conclusion

The total number of publications has shown a steady upward trend. The cooperation between countries globally was still relatively lacking, and therefore it is necessary to strengthen cooperation between different countries and regions to provide conditions for multi-center, large sample, and high-quality research. In addition to the relatively mature rehabilitation of rotator cuff injury such as passive motion or exercise therapy, telerehabilitation has also attracted much attention with the progress of science.

“Sport and Anatomy”: Teaching, Research, and Assistance at the University of Pisa

Introduction: Over the last decades, the university system has experienced huge growth, facing several challenges. Accordingly, the University of Pisa recognizes the value and opportunities deriving from research and fully supports collaboration with the world of entrepreneurship and industry, as well as local communities. Study programs, teaching methods and technologies, learning environments, quality assurance, programmed student numbers, and research results are key features of the prestige of the scientific community. Aim: In this respect, “Sport and Anatomy”, a brand that includes an academic organization at the University of Pisa, holds two main goals: (i) to offer the top level in both educational and professional fields; and (ii) to optimize the fine-tuning among all these sections, thus becoming a reference point for sports management. Methods and results: Indispensable links between basic and specialist sciences through different Masters’ and schools were created. In addition to didactic activity, research activity, medical assistance, and rehabilitation were coordinated. Two main outcomes emerged from this experience: (i) improved stakeholder performances and (ii) optimized cooperation between university and local communities. Conclusions: “Sport and Anatomy” plays a key role in supervising and accomplishing in an innovative way all the three missions of the university (i.e., teaching, research, and dissemination of knowledge), thus strongly fulfilling the aims of modern university targets.

ARMIA: A Sensorized Arm Wearable for Motor Rehabilitation

In this paper, we present ARMIA: a sensorized arm wearable that includes a combination of inertial and sEMG sensors to interact with serious games in telerehabilitation setups. This device reduces the cost of robotic assistance technologies to be affordable for end-users at home and at rehabilitation centers. Hardware and acquisition software specifications are described together with potential applications of ARMIA in real-life rehabilitation scenarios. A detailed comparison with similar medical technologies is provided, with a specific focus on wearable devices and virtual and augmented reality approaches. The potential advantages of the proposed device are also described showing that ARMIA could provide similar, if not better, the effectivity of physical therapy as well as giving the possibility of home-based rehabilitation.

Augmented Reality: Mapping Methods and Tools for Enhancing the Human Role in Healthcare HMI

Background: Augmented Reality (AR) represents an innovative technology to improve data visualization and strengthen the human perception. Among Human–Machine Interaction (HMI), medicine can benefit most from the adoption of these digital technologies. In this perspective, the literature on orthopedic surgery techniques based on AR was evaluated, focusing on identifying the limitations and challenges of AR-based healthcare applications, to support the research and the development of further studies. Methods: Studies published from January 2018 to December 2021 were analyzed after a comprehensive search on PubMed, Google Scholar, Scopus, IEEE Xplore, Science Direct, and Wiley Online Library databases. In order to improve the review reporting, the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines were used. Results: Authors selected sixty-two articles meeting the inclusion criteria, which were categorized according to the purpose of the study (intraoperative, training, rehabilitation) and according to the surgical procedure used. Conclusions: AR has the potential to improve orthopedic training and practice by providing an increasingly human-centered clinical approach. Further research can be addressed by this review to cover problems related to hardware limitations, lack of accurate registration and tracking systems, and absence of security protocols.

Architecture of a Hybrid Video/Optical See-through Head-Mounted Display-Based Augmented Reality Surgical Navigation Platform

In the context of image-guided surgery, augmented reality (AR) represents a ground-breaking enticing improvement, mostly when paired with wearability in the case of open surgery. Commercially available AR head-mounted displays (HMDs), designed for general purposes, are increasingly used outside their indications to develop surgical guidance applications with the ambition to demonstrate the potential of AR in surgery. The applications proposed in the literature underline the hunger for AR-guidance in the surgical room together with the limitations that hinder commercial HMDs from being the answer to such a need. The medical domain demands specifically developed devices that address, together with ergonomics, the achievement of surgical accuracy objectives and compliance with medical device regulations. In the framework of an EU Horizon2020 project, a hybrid video and optical see-through augmented reality headset paired with a software architecture, both specifically designed to be seamlessly integrated into the surgical workflow, has been developed. In this paper, the overall architecture of the system is described. The developed AR HMD surgical navigation platform was positively tested on seven patients to aid the surgeon while performing Le Fort 1 osteotomy in cranio-maxillofacial surgery, demonstrating the value of the hybrid approach and the safety and usability of the navigation platform.

EFFECTS OF A DEEP LEARNING-BASED SMARTPHONE APPLICATION ON SHOULDER ABDUCTION KINEMATICS AND BRAIN ACTIVATION IN ADHESIVE CAPSULITIS: A RANDOMIZED CONTROLLED TRIAL

Patients with adhesive capsulitis (AC) demonstrate limited shoulder movement, often accompanied by pain. Common treatment methods include pain medication, and continuous passive movement (CPM). However, it is sometimes difficult to improve the reduction of pain and movement using a CPM intervention because the patient’s interest is diminished. In this study, we developed an innovative deep learning-based smartphone application (Funrehab exercise game (FEG)) to provide accurate kinematics movement and motivation as well as high-intensity and repetitive movements using deep learning. We compared the effects of CPM and FEG on brain activity and shoulder range of motion in patients with AC. Sixteen patients (males, [Formula: see text]; females, [Formula: see text]; mean age, [Formula: see text] years) with acute AC were randomized into either CPM group or FEG group 4 days/week for 2 weeks. The outcome measures were shoulder abduction kinematics movement and electroencephalography (EEG) brain activity (bilateral prefrontal, bilateral sensorimotor cortex, and somatosensory association cortex) during the intervention. The analysis of variance (ANOVA) test was performed at [Formula: see text], and the analysis demonstrated that FEG showed superior effects on shoulder abduction kinematics and brain [Formula: see text] and [Formula: see text]-wave activations compared to CPM. Our results provide a novel and promising clinical evidence that FEG can more effectively improve neurophysiological EEG data and shoulder abduction movements than CPM in patients with AC.

Review of Microsoft HoloLens Applications over the Past Five Years

Since Microsoft HoloLens first appeared in 2016, HoloLens has been used in various industries, over the past five years. This study aims to review academic papers on the applications of HoloLens in several industries. A review was performed to summarize the results of 44 papers (dated between January 2016 and December 2020) and to outline the research trends of applying HoloLens to different industries. This study determined that HoloLens is employed in medical and surgical aids and systems, medical education and simulation, industrial engineering, architecture, civil engineering and other engineering fields. The findings of this study contribute towards classifying the current uses of HoloLens in various industries and identifying the types of visualization techniques and functions.

Upper Limb Rehabilitation Tools in Virtual Reality Based on Haptic and 3D Spatial Recognition Analysis: A Pilot Study

With aging, cerebrovascular diseases can occur more often. Stroke cases involve hemiplegia, which causes difficulties in performing activities of daily living. Existing rehabilitation treatments are based on the subjective evaluation of the therapist as the need for non-contact care arises; it is necessary to develop a system that can self-rehabilitate and offer objective analysis. Therefore, we developed rehabilitation tools that enable self-rehabilitation exercises in a virtual space based on haptics. Thirty adults without neurological damage were trained five times in a virtual environment, and the time, number of collisions, and coordinates were digitized and stored in real time. An analysis of variance (ANOVA) of the time and distance similarity changes revealed that as the number of rounds increased, no changes or increases occurred (p ≥ 0.05), and the collisions and paths were stable as the training progressed (p < 0.05). ANOVA showed a high correlation (0.90) with a decrease in the number of crashes and time required. It was meaningful to users when performing rehabilitation training more than four times and significantly impacted the analysis. This study analyzed the upper limb and cognitive rehabilitation of able-boded people in three-dimensional space in a virtual environment; the performance difficulty could be controlled through variations in rehabilitation models.

Upper Limb Physical Rehabilitation Using Serious Videogames and Motion Capture Systems: A Systematic Review

The use of videogames and motion capture systems in rehabilitation contributes to the recovery of the patient. This systematic review aimed to explore the works related to these technologies. The PRISMA method (Preferred Reporting Items for Systematic reviews and Meta-Analyses) was used to search the databases Scopus, PubMed, IEEE Xplore, and Web of Science, taking into consideration four aspects: physical rehabilitation, the use of videogames, motion capture technologies, and upper limb rehabilitation. The literature selection was limited to open access works published between 2015 and 2020, obtaining 19 articles that met the inclusion criteria. The works reported the use of inertial measurement units (37%), a Kinect sensor (48%), and other technologies (15%). It was identified that 26% used commercial products, while 74% were developed independently. Another finding was that 47% of the works focus on post-stroke motor recovery. Finally, diverse studies sought to support physical rehabilitation using motion capture systems incorporating inertial units, which offer precision and accessibility at a low cost. There is a clear need to continue generating proposals that confront the challenges of rehabilitation with technologies which offer precision and healthcare coverage, and which, additionally, integrate elements that foster the patient's motivation and participation.

Augmented-Reality-Based 3D Emotional Messenger for Dynamic User Communication with Smart Devices

With the development of Internet technologies, chat environments have migrated from PCs to mobile devices. Conversations have moved from phone calls and text messages to mobile messaging services or “messengers,” which has led to a significant surge in the use of mobile messengers such as Line and WhatsApp. However, because these messengers mainly use text as the communication medium, they have the inherent disadvantage of not effectively representing the user’s nonverbal expressions. In this context, we propose a new emotional communication messenger that improves upon the limitations of existing static expressions in current messenger applications. We develop a chat messenger based on augmented reality (AR) technology using smartglasses, which are a type of a wearable device. To this end, we select a server model that is suitable for AR, and we apply an effective emotional expression method based on 16 different basic emotions classified as per Russell’s model. In our app, these emotions can be expressed via emojis, animations, particle effects, and sound clips. Finally, we verify the efficacy of our messenger by conducting a user study to compare it with current 2D-based messenger services. Our messenger service can serve as a prototype for future AR-based messenger apps.

Interactive serious game for shoulder rehabilitation based on real-time hand tracking

BACKGROUND

Virtual reality is becoming popular in the rehabilitation field thanks to the several advantages it can offer to patients and physicians. Indeed, serious games can: motivate and engage the patient; offer different levels of challenge and difficulty based on the patient baseline, and integrate objective measures of the patient's performance during each rehabilitation session.

OBJECTIVE

We designed and implemented a serious game for shoulder rehabilitation based on real-time hand tracking. The aim was to maintain the medical benefits of traditional rehabilitation, while reducing human resources and costs and facilitating active patient participation.

METHODS

Our software application provides the user with a shoulder horizontal adduction exercise. This exercise takes place in a 2D interactive game environment, controlled by hand movements on a desk pad. The hardware includes a standard desktop computer and screen, and the Leap Motion Controller: a hand tracking system. Changing the desk pad material allows the physiotherapist to vary the friction between the user hand and the supporting surface.

RESULTS

Fourteen healthy volunteers and six rehabilitation experts tested our serious game. The results showed that the application is attractive, ergonomic and clinically useful.

CONCLUSION

Despite promising results, clinical validation is necessary to demonstrate the efficacy of the serious game.