Capturing Situational Information with Smartphones and Mobile Sensing Methods

Laughter in everyday life: an event-based experience sampling method study using wrist-worn wearables

Laughter is a universal, nonverbal vocal expression of broad significance for humans. Interestingly, rather little is known about how often we laugh and how laughter is associated with our personality. In a large, event-based, experience sampling method study (N = 52; k = 9,261 assessments) using wrist-worn wearables and a physical analogue scale, we analyzed belly laughs and fit of laughter events in participants' everyday life for 4 weeks. Additionally, we assessed associations with laughter frequency such as personality, happiness, life satisfaction, gelotophobia (i.e., fear of being laughed at), and cheerfulness. Validating our new measurement approach (i.e., wearables, physical analogue scale), laughter events elicited higher happiness ratings compared to reference assessments, as expected. On average, participants reported 2.5 belly laughs per day and on every fourth day a fit of laughter. As expected, participants who were happier and more satisfied with their life laughed more frequently than unhappier, unsatisfied participants. Women and younger participants laughed significantly more than men and older participants. Regarding personality, laughter frequency was positively associated with openness and conscientiousness. No significant association was found for gelotophobia, and results for cheerfulness and related concepts were mixed. By using state-of-the-art statistical methods (i.e., recurrent event regression) for the event-based, multi-level data on laughter, we could replicate past results on laughing.

ESMira: A decentralized open-source application for collecting experience sampling data

This paper introduces ESMira, a server and mobile app (Android, iOS) developed for research projects using experience sampling method (ESM) designs. ESMira offers a very simple setup process and ease of use, while being free, decentralized, and open-source (source code is available on GitHub). The ongoing development of ESMira started in early 2019, with a focus on scientific requirements (e.g., informed consent, ethical considerations), data security (e.g., encryption), and data anonymity (e.g., completely anonymous data workflow). ESMira sets itself apart from other platforms by both being free of charge and providing study administrators with full control over study data without the need for specific technological skills (e.g., programming). This means that study administrators can have ESMira running on their own webspace without needing much technical knowledge, allowing them to remain independent from any third-party service. Furthermore, ESMira offers an extensive list of features (e.g., an anonymous built-in chat to contact participants; a reward system that allows participant incentivization without breaching anonymity; live graphical feedback for participants) and can deal with complex study designs (e.g., nested time-based sampling). In this paper, we illustrate the basic structure of ESMira, explain how to set up a new server and create studies, and introduce the platform's basic functionalities.

Social-Ecological Measurement of Daily Life: How Relationally Focused Ambulatory Assessment can Advance Clinical Intervention Science

Individuals’ daily behaviors and social interactions play a central role in the diagnosis and treatment of psychological disorders. Despite this, observational ambulatory assessment methods—research methods that allow for direct and passive assessment of individuals’ momentary activities and interactions—have a remarkably scant history in the clinical science field. Prior discussions of ambulatory assessment methods in clinical science have focused on subjective methods (e.g., ecological momentary assessment) and physiological methods (e.g., wearable heart rate monitoring). Comparatively less attention has been dedicated to ambulatory assessment methods that collect objective, relational data about individuals’ social behaviors and their interactions with their momentary environmental contexts. Drawing on extant social-ecological measurement frameworks, this article first provides a conceptual and psychometric rationale for the integration of daily relational data into clinical science research. Next, the nascent research applying such methods to clinical science is reviewed, and priorities for further research organized by the NIH Stage Model for Clinical Science Research are recommended. These data can provide unique information about the social contexts of diverse patient populations; identify social-ecological targets for transdiagnostic, precision, and culturally responsive interventions; and contribute novel data about the effectiveness of established interventions at creating behavioral and relational change.

Machine learning-based classification of the movements of children with profound or severe intellectual or multiple disabilities using environment data features

Communication interventions have broadened from dialogical meaning-making, assessment approaches, to remote-controlled interactive objects. Yet, interpretation of the mostly pre-or protosymbolic, distinctive, and idiosyncratic movements of children with intellectual disabilities (IDs) or profound intellectual and multiple disabilities (PIMD) using computer-based assistive technology (AT), machine learning (ML), and environment data (ED: location, weather indices and time) remain insufficiently unexplored. We introduce a novel behavior inference computer-based communication-aid AT system structured on machine learning (ML) framework to interpret the movements of children with PIMD/IDs using ED. To establish a stable system, our study aimed to train, cross-validate (10-fold), test and compare the classification accuracy performance of ML classifiers (eXtreme gradient boosting [XGB], support vector machine [SVM], random forest [RF], and neural network [NN]) on classifying the 676 movements to 2, 3, or 7 behavior outcome classes using our proposed dataset recalibration (adding ED to movement datasets) with or without Boruta feature selection (53 child characteristics and movements, and ED-related features). Natural-child-caregiver-dyadic interactions observed in 105 single-dyad video-recorded (30-hour) sessions targeted caregiver-interpreted facial, body, and limb movements of 20 8-to 16-year-old children with PIMD/IDs and simultaneously app-and-sensor-collected ED. Classification accuracy variances and the influences of and the interaction among recalibrated dataset, feature selection, classifiers, and classes on the pooled classification accuracy rates were evaluated using three-way ANOVA. Results revealed that Boruta and NN-trained dataset in class 2 and the non-Boruta SVM-trained dataset in class 3 had >76% accuracy rates. Statistically significant effects indicating high classification rates (>60%) were found among movement datasets: with ED, non-Boruta, class 3, SVM, RF, and NN. Similar trends (>69%) were found in class 2, NN, Boruta-trained movement dataset with ED, and SVM and RF, and non-Boruta-trained movement dataset with ED in class 3. These results support our hypotheses that adding environment data to movement datasets, selecting important features using Boruta, using NN, SVM and RF classifiers, and classifying movements to 2 and 3 behavior outcomes can provide >73.3% accuracy rates, a promising performance for a stable ML-based behavior inference communication-aid AT system for children with PIMD/IDs.

Grouped feature importance and combined features effect plot

Interpretable machine learning has become a very active area of research due to the rising popularity of machine learning algorithms and their inherently challenging interpretability. Most work in this area has been focused on the interpretation of single features in a model. However, for researchers and practitioners, it is often equally important to quantify the importance or visualize the effect of feature groups. To address this research gap, we provide a comprehensive overview of how existing model-agnostic techniques can be defined for feature groups to assess the grouped feature importance, focusing on permutation-based, refitting, and Shapley-based methods. We also introduce an importance-based sequential procedure that identifies a stable and well-performing combination of features in the grouped feature space. Furthermore, we introduce the combined features effect plot, which is a technique to visualize the effect of a group of features based on a sparse, interpretable linear combination of features. We used simulation studies and real data examples to analyze, compare, and discuss these methods.

Integrating Behavior of Children with Profound Intellectual, Multiple, or Severe Motor Disabilities With Location and Environment Data Sensors for Independent Communication and Mobility: App Development and Pilot Testing

Background

Children with profound intellectual and multiple disabilities (PIMD) or severe motor and intellectual disabilities (SMID) only communicate through movements, vocalizations, body postures, muscle tensions, or facial expressions on a pre- or protosymbolic level. Yet, to the best of our knowledge, there are few systems developed to specifically aid in categorizing and interpreting behaviors of children with PIMD or SMID to facilitate independent communication and mobility. Further, environmental data such as weather variables were found to have associations with human affects and behaviors among typically developing children; however, studies involving children with neurological functioning impairments that affect communication or those who have physical and/or motor disabilities are unexpectedly scarce.

Objective

This paper describes the design and development of the ChildSIDE app, which collects and transmits data associated with children’s behaviors, and linked location and environment information collected from data sources (GPS, iBeacon device, ALPS Sensor, and OpenWeatherMap application programming interface [API]) to the database. The aims of this study were to measure and compare the server/API performance of the app in detecting and transmitting environment data from the data sources to the database, and to categorize the movements associated with each behavior data as the basis for future development and analyses.

Methods

This study utilized a cross-sectional observational design by performing multiple single-subject face-to-face and video-recorded sessions among purposively sampled child-caregiver dyads (children diagnosed with PIMD/SMID, or severe or profound intellectual disability and their primary caregivers) from September 2019 to February 2020. To measure the server/API performance of the app in detecting and transmitting data from data sources to the database, frequency distribution and percentages of 31 location and environment data parameters were computed and compared. To categorize which body parts or movements were involved in each behavior, the interrater agreement κ statistic was used.

Results

The study comprised 150 sessions involving 20 child-caregiver dyads. The app collected 371 individual behavior data, 327 of which had associated location and environment data from data collection sources. The analyses revealed that ChildSIDE had a server/API performance >93% in detecting and transmitting outdoor location (GPS) and environment data (ALPS sensors, OpenWeatherMap API), whereas the performance with iBeacon data was lower (82.3%). Behaviors were manifested mainly through hand (22.8%) and body movements (27.7%), and vocalizations (21.6%).

Conclusions

The ChildSIDE app is an effective tool in collecting the behavior data of children with PIMD/SMID. The app showed high server/API performance in detecting outdoor location and environment data from sensors and an online API to the database with a performance rate above 93%. The results of the analysis and categorization of behaviors suggest a need for a system that uses motion capture and trajectory analyses for developing machine- or deep-learning algorithms to predict the needs of children with PIMD/SMID in the future.

‘Personality in its Natural Habitat’ Revisited: A Pooled, Multi–sample Examination of the Relationships between the Big Five Personality Traits and Daily Behaviour and Language Use

Past research using the Electronically Activated Recorder (EAR), an observational ambulatory assessment method for the real–world measurement of daily behaviour, has identified several behavioural manifestations of the Big Five domains in a small college sample ( N = 96). With the use of a larger and more diverse sample of pooled data from N = 462 participants from a total of four community samples who wore the EAR from 2 to 6 days, the primary purpose of the present study was to obtain more precise and generalizable effect estimates of the Big Five–behaviour relationships and to re–examine the degree to which these relationships are gender specific. In an extension of the original article, the secondary purpose of the present study was to examine if the Big Five–behaviour relationships differed across two facets of each Big Five domain. Overall, while several of the behavioural manifestations of the Big Five were generally consistent with the trait definitions (replicating some findings from the original article), we found little evidence of gender differences (not replicating a basic finding from the original article). Unique to the present study, the Big Five–behaviour relationships were not always comparable across the two facets of each Big Five domain. © 2020 European Association of Personality Psychology

Motivated behavior in intimate relationships: Comparing the predictive value of motivational variables

Motivational variables are considered fundamental factors influencing the occurrence of behavior. The current study compared different types of motivational variables (implicit and explicit motive dispositions, motivation as states and as aggregated person-level variables) in their ability to predict communal and agentic behavior reports in intimate relationships. 510 individuals completed measures of dispositional communion and agency motives and participated in a dyadic experience sampling study with five assessments per day across four weeks. They reported on their momentary communal and agentic motivation, as well as on their own and their partner’s behaviors. All examined types of motivational variables predicted certain behavior reports on the between-person or within-person level and had incremental effects beyond the other motivational variables in at least one motive domain. Directly replicating and conceptually extending prior research, the effects of motivational states and their aggregates were consistently found across behavioral outcomes, across self- and partner-reports and across the motive domains of communion and agency. Using the example of motivational states, the general value of assessing within-person variables for psychological phenomena in ESM-designs is discussed.

Predicting personality from patterns of behavior collected with smartphones

Smartphones are sensor-rich computers that can easily be used to collect extensive records of behaviors, posing serious threats to individuals’ privacy. This study examines the extent to which individuals’ personality dimensions (assessed at broad domain and narrow facet levels) can be predicted from six classes of behavior: 1) communication and social behavior, 2) music consumption, 3) app usage, 4) mobility, 5) overall phone activity, and 6) day- and night-time activity, in a large sample. The cross-validated results show which Big Five personality dimensions are predictable and which specific patterns of behavior are indicative of which dimensions, revealing communication and social behavior as most predictive overall. Our results highlight the benefits and dangers posed by the widespread collection of smartphone data.

Smartphones enjoy high adoption rates around the globe. Rarely more than an arm’s length away, these sensor-rich devices can easily be repurposed to collect rich and extensive records of their users’ behaviors (e.g., location, communication, media consumption), posing serious threats to individual privacy. Here we examine the extent to which individuals’ Big Five personality dimensions can be predicted on the basis of six different classes of behavioral information collected via sensor and log data harvested from smartphones. Taking a machine-learning approach, we predict personality at broad domain (rmedian = 0.37) and narrow facet levels (rmedian = 0.40) based on behavioral data collected from 624 volunteers over 30 consecutive days (25,347,089 logging events). Our cross-validated results reveal that specific patterns in behaviors in the domains of 1) communication and social behavior, 2) music consumption, 3) app usage, 4) mobility, 5) overall phone activity, and 6) day- and night-time activity are distinctively predictive of the Big Five personality traits. The accuracy of these predictions is similar to that found for predictions based on digital footprints from social media platforms and demonstrates the possibility of obtaining information about individuals’ private traits from behavioral patterns passively collected from their smartphones. Overall, our results point to both the benefits (e.g., in research settings) and dangers (e.g., privacy implications, psychological targeting) presented by the widespread collection and modeling of behavioral data obtained from smartphones.

Social Support and Common Dyadic Coping in Couples' Dyadic Management of Type II Diabetes: Protocol for an Ambulatory Assessment Application

Background

Type II diabetes mellitus (T2DM) is a common chronic disease. To manage blood glucose levels, patients need to follow medical recommendations for healthy eating, physical activity, and medication adherence in their everyday life. Illness management is mainly shared with partners and involves social support and common dyadic coping (CDC). Social support and CDC have been identified as having implications for people’s health behavior and well-being. Visible support, however, may also be negatively related to people’s well-being. Thus, the concept of invisible support was introduced. It is unknown which of these concepts (ie, visible support, invisible support, and CDC) displays the most beneficial associations with health behavior and well-being when considered together in the context of illness management in couple’s everyday life. Therefore, a novel ambulatory assessment application for the open-source behavioral intervention platform MobileCoach (AAMC) was developed. It uses objective sensor data in combination with self-reports in couple’s everyday life.

Objective

The aim of this paper is to describe the design of the Dyadic Management of Diabetes (DyMand) study, funded by the Swiss National Science Foundation (CR12I1_166348/1). The study was approved by the cantonal ethics committee of the Canton of Zurich, Switzerland (Req-2017_00430).

Methods

This study follows an intensive longitudinal design with 2 phases of data collection. The first phase is a naturalistic observation phase of couples’ conversations in combination with experience sampling in their daily lives, with plans to follow 180 T2DM patients and their partners using sensor data from smartwatches, mobile phones, and accelerometers for 7 consecutive days. The second phase is an observational study in the laboratory, where couples discuss topics related to their diabetes management. The second phase complements the first phase by focusing on the assessment of a full discussion about diabetes-related concerns. Participants are heterosexual couples with 1 partner having a diagnosis of T2DM.

Results

The AAMC was designed and built until the end of 2018 and internally tested in March 2019. In May 2019, the enrollment of the pilot phase began. The data collection of the DyMand study will begin in September 2019, and analysis and presentation of results will be available in 2021.

Conclusions

For further research and practice, it is crucial to identify the impact of social support and CDC on couples’ dyadic management of T2DM and their well-being in daily life. Using AAMC will make a key contribution with regard to objective operationalizations of visible and invisible support, CDC, physical activity, and well-being. Findings will provide a sound basis for theory- and evidence-based development of dyadic interventions to change health behavior in the context of couple’s dyadic illness management. Challenges to this multimodal sensor approach and its feasibility aspects are discussed.

International Registered Report Identifier (IRRID)

PRR1-10.2196/13685

Concept, Possibilities and Pilot-Testing of a New Smartphone Application for the Social and Life Sciences to Study Human Behavior Including Validation Data from Personality Psychology

With the advent of the World Wide Web, the smartphone and the Internet of Things, not only society but also the sciences are rapidly changing. In particular, the social sciences can profit from these digital developments, because now scientists have the power to study real-life human behavior via smartphones and other devices connected to the Internet of Things on a large-scale level. Although this sounds easy, scientists often face the problem that no practicable solution exists to participate in such a new scientific movement, due to a lack of an interdisciplinary network. If so, the development time of a new product, such as a smartphone application to get insights into human behavior takes an enormous amount of time and resources. Given this problem, the present work presents an easy way to use a smartphone application, which can be applied by social scientists to study a large range of scientific questions. The application provides measurements of variables via tracking smartphone–use patterns, such as call behavior, application use (e.g., social media), GPS and many others. In addition, the presented Android-based smartphone application, called Insights, can also be used to administer self-report questionnaires for conducting experience sampling and to search for co-variations between smartphone usage/smartphone data and self-report data. Of importance, the present work gives a detailed overview on how to conduct a study using an application such as Insights, starting from designing the study, installing the application to analyzing the data. In the present work, server requirements and privacy issues are also discussed. Furthermore, first validation data from personality psychology are presented. Such validation data are important in establishing trust in the applied technology to track behavior. In sum, the aim of the present work is (i) to provide interested scientists a short overview on how to conduct a study with smartphone app tracking technology, (ii) to present the features of the designed smartphone application and (iii) to demonstrate its validity with a proof of concept study, hence correlating smartphone usage with personality measures.

Personality Traits Predict Smartphone Usage

The present study investigates to what degree individual differences can predict frequency and duration of actual behaviour, manifested in mobile application (app) usage on smartphones. In particular, this work focuses on the identification of stable associations between personality on the factor and facet level, fluid intelligence, demography and app usage in 16 distinct categories. A total of 137 subjects (87 women and 50 men), with an average age of 24 ( SD = 4.72), participated in a 90–min psychometric lab session as well as in a subsequent 60–day data logging study in the field. Our data suggest that personality traits predict mobile application usage in several specific categories such as communication, photography, gaming, transportation and entertainment. Extraversion, conscientiousness and agreeableness are better predictors of mobile application usage than basic demographic variables in several distinct categories. Furthermore, predictive performance is slightly higher for single factor—in comparison with facet–level personality scores. Fluid intelligence and demographics additionally show stable associations with categorical app usage. In sum, this study demonstrates how individual differences can be effectively related to actual behaviour and how this can assist in understanding the behavioural underpinnings of personality. Copyright © 2017 European Association of Personality Psychology

Using Smartphones to Collect Behavioral Data in Psychological Science: Opportunities, Practical Considerations, and Challenges

Smartphones now offer the promise of collecting behavioral data unobtrusively, in situ, as it unfolds in the course of daily life. Data can be collected from the onboard sensors and other phone logs embedded in today's off-the-shelf smartphone devices. These data permit fine-grained, continuous collection of people's social interactions (e.g., speaking rates in conversation, size of social groups, calls, and text messages), daily activities (e.g., physical activity and sleep), and mobility patterns (e.g., frequency and duration of time spent at various locations). In this article, we have drawn on the lessons from the first wave of smartphone-sensing research to highlight areas of opportunity for psychological research, present practical considerations for designing smartphone studies, and discuss the ongoing methodological and ethical challenges associated with research in this domain. It is our hope that these practical guidelines will facilitate the use of smartphones as a behavioral observation tool in psychological science.

Restoring Agency to the Human Actor

A cursory read of the social psychological literature suggests that when people find themselves in strong situations, they fail to display agency. The early classic studies of conformity, obedience, and bystander intervention, for example, are renowned for showing that when challenged by strong situational pressures, participants acquiesced—even if it meant abandoning their moral principles or disregarding their own sensory data. Later studies of learned helplessness, ego depletion, and stereotype threat echoed this “power of the situation” theme, demonstrating that exposure to (or the expectation of) a frustrating or unpleasant experience suppressed subsequent efforts to actualize goals and abilities. Although this work has provided many valuable insights into the influence of situational pressures, it has been used to buttress an unbalanced and misleading portrait of human agency. This portrait fails to recognize that situations are not invariably enemies of agency. Instead, strong situational forces often allow for, and may even encourage, expressions of human agency. We examine the nature, causes, and consequences of this phenomenon. We endorse a broader approach that emphasizes how responding to situational pressure can coexist with agency. This new emphasis should create greater convergence between social psychological models and the experience of agency in everyday life.