How Do People with Limited Movement Personalize Upper-Body Gestures? Considerations for the Design of Personalized and Accessible Gesture Interfaces

Always-on, upper-body input from sensors like accelerometers, infrared cameras, and electromyography hold promise to enable accessible gesture input for people with upper-body motor impairments. When these sensors are distributed across the person's body, they can enable the use of varied body parts and gestures for device interaction. Personalized upper-body gestures that enable input from diverse body parts including the head, neck, shoulders, arms, hands and fingers and match the abilities of each user, could be useful for ensuring that gesture systems are accessible. In this work, we characterize the personalized gesture sets designed by 25 participants with upper-body motor impairments and develop design recommendations for upper-body personalized gesture interfaces. We found that the personalized gesture sets that participants designed were highly ability-specific. Even within a specific type of disability, there were significant differences in what muscles participants used to perform upper-body gestures, with some pre-dominantly using shoulder and upper-arm muscles, and others solely using their finger muscles. Eight percent of gestures that participants designed were with their head, neck, and shoulders, rather than their hands and fingers, demonstrating the importance of tracking the whole upper-body. To combat fatigue, participants performed 51% of gestures with their hands resting on or barely coming off of their armrest, highlighting the importance of using sensing mechanisms that are agnostic to the location and orientation of the body. Lastly, participants activated their muscles but did not visibly move during 10% of the gestures, demonstrating the need for using sensors that can sense muscle activations without movement. Both inertial measurement unit (IMU) and electromyography (EMG) wearable sensors proved to be promising sensors to differentiate between personalized gestures. Personalized upper-body gesture interfaces that take advantage of each person's abilities are critical for enabling accessible upper-body gestures for people with upper-body motor impairments.

Implementing Ability-Based Design: A Systematic Approach to Conceptual User Modeling

The notion of Ability-Based Design , put forth by Wobbrock et al. [80, 82] as a solution to the challenge of creating accessible technology, has been discussed in human-computer interaction research now for over a decade. However, despite being cited as influential on various projects, the concept still lacks a general characterization of how to implement its required focus on abilities. In particular, it lacks a formulation of how to perceive and model users within an articulated design process. To address this shortcoming, we rely on conceptual user modeling to examine Ability-Based Design and propose a characterization of it that is not dependent upon a specific project or research effort, but that enables the ability-based design of new technologies in a systematic manner. Our findings show that Ability-Based Design’s focus on abilities requires important changes in typical user modeling approaches that cannot be met with current techniques. Based on the challenges identified through our analysis, we propose a first modification not only of current user modeling, but of current requirements analysis approaches to address abilities and their intertwined dependencies with tasks and contexts as core elements of conceptual models in Ability-Based Design. We thereby demonstrate not only the complexity of modeling users’ abilities, but also draw out promising ideas and perspectives for future research, emphasizing the need for future evaluative work on our approach.

An Epidemiology-inspired Large-scale Analysis of Android App Accessibility

Accessibility barriers in mobile applications (apps) can make it challenging for people who have impairments or use assistive technology to use those apps. Ross et al.’s epidemiology-inspired framework emphasizes that a wide variety of factors may influence an app's accessibility and presents large-scale analysis as a powerful tool for understanding the prevalence of accessibility barriers (i.e., inaccessibility diseases ). Drawing on this framework, we performed a large-scale analysis of free Android apps, exploring the frequency of accessibility barriers and factors that may have contributed to barrier prevalence. We tested a population of 9,999 apps for seven accessibility barriers: few TalkBack-focusable elements, missing labels, duplicate labels, uninformative labels, editable TextViews with contentDescription , fully overlapping clickable elements, and undersized elements. We began by measuring the prevalence of each accessibility barrier across all relevant element classes and apps. Missing labels and undersized elements were the most prevalent barriers. As a measure of the spread of barriers across apps, we assessed the five most reused classes of elements for missing labels and undersized elements. The Image Button class was among the most barrier-prone of the high reuse element classes; 53% of Image Button elements were missing labels and 40% were undersized. We also investigated factors that may have contributed to the high barrier prevalence in certain classes of elements, selecting examples based on prior knowledge, our analyses, and metrics of reuse and barrier-proneness. These case studies explore: (1) how the few TalkBack-focusable elements accessibility barrier relates to app category (e.g., Education, Entertainment) and the tools used to implement an app, (2) the prevalence of label-based barriers in image-based buttons, (3) design patterns that affect the labeling and size of Radio Buttons and Checkboxes, and (4) accessibility implications of the sizing of third-party plug-in elements. Our work characterizes the current state of Android accessibility, suggests improvements to the app ecosystem, and demonstrates analysis techniques that can be applied in further app accessibility assessments.

Design for Social Accessibility Method Cards

This article is an extended version of our 2018 ASSETS paper entitled, “Incorporating Social Factors in Accessible Design.” In our ASSETS paper, we demonstrated the viability of the Design for Social Accessibility perspective through a series of user-centered workshops with professional designers. With this expanded article, we conducted a follow-up research study with a user-centered design course that examined the use of Design for Social Accessibility Method Cards over a longer design cycle, specifically as the method and cards contributed to a term-long project, rather than just a workshop. We also offer a new analysis on work leading to the development of Design for Social Accessibility, with a focus on how practical considerations in the design process influence how designers engage accessible design. We found that the concrete and real-life scenarios in the Design for Social Accessibility Method Cards helped mediate useful interactions between student designers and deaf and hard-of-hearing users. In addition, we identified how practical choices in investigating strategies for socially accessible design enabled designers to center disabled perspectives. The contributions of this work—when added to the findings of our ASSETS 2018 paper on incorporating social factors—demonstrate the viability of Design for Social Accessibility by providing: (1) empirical data showing that designers can use the Design for Social Accessibility perspective and method cards to generate accessible designs and appropriately engage deaf and hard-of-hearing users to incorporate social considerations; and (2) a summative analysis highlighting practical steps for how designers can use the Design for Social Accessibility perspective and methods cards to create accessible designs.

Tenets for Social Accessibility

Despite years of addressing disability in technology design and advocating user-centered design practices, popular mainstream technologies remain largely inaccessible for people with disabilities. We conducted a design course study investigating how student designers regard disability and explored how designing for multiple disabled and nondisabled users encouraged students to think about accessibility in the design process. Across two university course offerings one year apart, we examined how students focused on a design project while learning user-centered design concepts and techniques, working with people with and without disabilities throughout the project. In addition, we compared how students incorporated disability-focused design approaches within a classroom setting. We found that designing for multiple stakeholders with and without disabilities expanded student understanding of accessible design by demonstrating that people with the same disability could have diverse needs and by aligning such needs with those of nondisabled users. We also found that using approaches targeted toward designing for people with disabilities complemented interactions with users, particularly with regard to managing varying abilities across users, or incorporating social aspects. Our findings contribute to an understanding about how we might incur change in design practice by working with multiple stakeholders with and without disabilities whenever possible. We refined Design for Social Accessibility by incorporating these findings into three tenets emphasizing: (1) design for disability ought to incorporate users with and without disabilities, (2) design should address functional and social factors simultaneously, and (3) design should include tools to spur consideration of social factors in accessible design.

Comparing Speech and Keyboard Text Entry for Short Messages in Two Languages on Touchscreen Phones

With the ubiquity of mobile touchscreen devices like smartphones, two widely used text entry methods have emerged: small touch-based keyboards and speech recognition. Although speech recognition has been available on desktop computers for years, it has continued to improve at a rapid pace, and it is currently unknown how today's modern speech recognizers compare to state-of-the-art mobile touch keyboards, which also have improved considerably since their inception. To discover both methods' “upper-bound performance,” we evaluated them in English and Mandarin Chinese on an Apple iPhone 6 Plus in a laboratory setting. Our experiment was carried out using Baidu's Deep Speech 2, a deep learning-based speech recognition system, and the built-in Qwerty (English) or Pinyin (Mandarin) Apple iOS keyboards. We found that with speech recognition, the English input rate was 2.93 times faster (153 vs. 52 WPM), and the Mandarin Chinese input rate was 2.87 times faster (123 vs. 43 WPM) than the keyboard for short message transcription under laboratory conditions for both methods. Furthermore, although speech made fewer errors during entry (5.30% vs. 11.22% corrected error rate), it left slightly more errors in the final transcribed text (1.30% vs. 0.79% uncorrected error rate). Our results show that comparatively, under ideal conditions for both methods, upper-bound speech recognition performance has greatly improved compared to prior systems, and might see greater uptake in the future, although further study is required to quantify performance in non-laboratory settings for both methods.

Self-Conscious or Self-Confident? A Diary Study Conceptualizing the Social Accessibility of Assistive Technology

With the recent influx of smartphones, tablets, and wearables such as watches and glasses, personal interactive device use is increasingly visible and commonplace in public and social spaces. Assistive Technologies (ATs) used by people with disabilities are observable to others and, as a result, can affect how AT users are perceived. This raises the possibility that what we call “social accessibility” may be as important as “functional accessibility” when considering ATs. But, to date, ATs have almost exclusively been regarded as functional aids. For example, ATs are defined by the Technical Assistance to the States Act as technologies that are “used to increase, maintain or improve functional capabilities of individuals with disabilities.” To investigate perceptions and self-perceptions of AT users, we conducted a diary study of two groups of participants: people with disabilities and people without disabilities. Our goal was to explore the types of interactions and perceptions that arise around AT use in social and public spaces. During our 4-week study, participants with sensory disabilities wrote about feeling either self-conscious or self-confident when using an assistive device in a social or public situation. Meanwhile, participants without disabilities were prompted to record their reactions and feelings whenever they saw ATs used in social or public situations. We found that AT form and function does influence social interactions by impacting self-efficacy and self-confidence. When the design of form or function is poor, or when inequality between technological accessibility exists, social inclusion is negatively affected, as are perceptions of ability. We contribute a definition for the “social accessibility” of ATs and subsequently offer Design for Social Accessibility (DSA) as a holistic design stance focused on balancing an AT user's sociotechnical identity with functional requirements.

Evaluating Intelligibility and Battery Drain of Mobile Sign Language Video Transmitted at Low Frame Rates and Bit Rates

Mobile sign language video conversations can become unintelligible if high video transmission rates cause network congestion and delayed video. In an effort to understand the perceived lower limits of intelligible sign language video intended for mobile communication, we evaluated sign language video transmitted at four low frame rates (1, 5, 10, and 15 frames per second [fps]) and four low fixed bit rates (15, 30, 60, and 120 kilobits per second [kbps]) at a constant spatial resolution of 320 × 240 pixels. We discovered an “intelligibility ceiling effect,” in which increasing the frame rate above 10fps did not improve perceived intelligibility, and increasing the bit rate above 60kbps produced diminishing returns. Given the study parameters, our findings suggest that relaxing the recommended frame rate and bit rate to 10fps at 60kbps will provide intelligible video conversations while reducing total bandwidth consumption to 25% of the ITU-T standard (at least 25fps and 100kbps). As part of this work, we developed the Human Signal Intelligibility Model , a new conceptual model useful for informing evaluations of video intelligibility and our methodology for creating linguistically accessible web surveys for deaf people. We also conducted a battery-savings experiment quantifying battery drain when sign language video is transmitted at the lower frame rates and bit rates. Results confirmed that increasing the transmission rates monotonically decreased the battery life.

Ability-Based Design

Current approaches to accessible computing share a common goal of making technology accessible to users with disabilities. Perhaps because of this goal, they may also share a tendency to centralize disability rather than ability. We present a refinement to these approaches called ability-based design that consists of focusing on ability throughout the design process in an effort to create systems that leverage the full range of human potential. Just as user-centered design shifted the focus of interactive system design from systems to users, ability-based design attempts to shift the focus of accessible design from disability to ability. Although prior approaches to accessible computing may consider users’ abilities to some extent, ability-based design makes ability its central focus. We offer seven ability-based design principles and describe the projects that inspired their formulation. We also present a research agenda for ability-based design.

Goal Crossing with Mice and Trackballs for People with Motor Impairments

Prior research shows that people with motor impairments face considerable challenges when using conventional mice and trackballs. One challenge is positioning the mouse cursor within confined target areas; another is executing a precise click without slipping. These problems can make mouse pointing in graphical user interfaces very difficult for some people. This article explores goal crossing as an alternative strategy for more accessible target acquisition. In goal crossing, targets are boundaries that are simply crossed by the mouse cursor. Thus, goal crossing avoids the two aforementioned problems. To date, however, researchers have not examined the feasibility of goal crossing for people with motor difficulties. We therefore present a study comparing area pointing and goal crossing. Our performance results indicate that although Fitts' throughput for able-bodied users is higher for area pointing than for goal crossing (4.72 vs. 3.61 bits/s), the opposite is true for users with motor impairments (2.34 vs. 2.88 bits/s). However, error rates are higher for goal crossing than for area pointing under a strict definition of crossing errors (6.23% vs. 1.94%). We also present path analyses and an examination of submovement velocity, acceleration, and jerk (the change in acceleration over time). These results show marked differences between crossing and pointing and almost categorically favor crossing. An important finding is that crossing reduces jerk for both participant groups, indicating more fluid, stable motion. To help realize the potential of goal crossing for computer access, we offer design concepts for crossing widgets that address the occlusion problem, which occurs when one crossing goal obscures another in persistent mouse-cursor interfaces. This work provides the motivation and initial steps for further exploration of goal crossing on the desktop, and may help researchers and designers to radically reshape user interfaces to provide accessible goal crossing, thereby lowering barriers to access.