Individual differences in the calibration of trust in automation

Using voice recognition to measure trust during interactions with automated vehicles

Trust in an automated vehicle system (AVs) can impact the experience and safety of drivers and passengers. This work investigates the effects of speech to measure drivers' trust in the AVs. Seventy-five participants were randomly assigned to high-trust (the AVs with 100% correctness, 0 crash, and 4 system messages with visual-auditory TORs) and low-trust group (the AVs with a correctness of 60%, a crash rate of 40%, 2 system messages with visual-only TORs). Voice interaction tasks were used to collect speech information during the driving process. The results revealed that our settings successfully induced trust and distrust states. The corresponding extracted speech feature data of the two trust groups were used for back-propagation neural network training and evaluated for its ability to accurately predict the trust classification. The highest classification accuracy of trust was 90.80%. This study proposes a method for accurately measuring trust in automated vehicles using voice recognition.

Designing explainable AI to improve human-AI team performance: A medical stakeholder-driven scoping review

The rise of complex AI systems in healthcare and other sectors has led to a growing area of research called Explainable AI (XAI) designed to increase transparency. In this area, quantitative and qualitative studies focus on improving user trust and task performance by providing system- and prediction-level XAI features. We analyze stakeholder engagement events (interviews and workshops) on the use of AI for kidney transplantation. From this we identify themes which we use to frame a scoping literature review on current XAI features. The stakeholder engagement process lasted over nine months covering three stakeholder group's workflows, determining where AI could intervene and assessing a mock XAI decision support system. Based on the stakeholder engagement, we identify four major themes relevant to designing XAI systems - 1) use of AI predictions, 2) information included in AI predictions, 3) personalization of AI predictions for individual differences, and 4) customizing AI predictions for specific cases. Using these themes, our scoping literature review finds that providing AI predictions before, during, or after decision-making could be beneficial depending on the complexity of the stakeholder's task. Additionally, expert stakeholders like surgeons prefer minimal to no XAI features, AI prediction, and uncertainty estimates for easy use cases. However, almost all stakeholders prefer to have optional XAI features to review when needed, especially in hard-to-predict cases. The literature also suggests that providing both system- and prediction-level information is necessary to build the user's mental model of the system appropriately. Although XAI features improve users' trust in the system, human-AI team performance is not always enhanced. Overall, stakeholders prefer to have agency over the XAI interface to control the level of information based on their needs and task complexity. We conclude with suggestions for future research, especially on customizing XAI features based on preferences and tasks.

How do humans learn about the reliability of automation?

In a range of settings, human operators make decisions with the assistance of automation, the reliability of which can vary depending upon context. Currently, the processes by which humans track the level of reliability of automation are unclear. In the current study, we test cognitive models of learning that could potentially explain how humans track automation reliability. We fitted several alternative cognitive models to a series of participants' judgements of automation reliability observed in a maritime classification task in which participants were provided with automated advice. We examined three experiments including eight between-subjects conditions and 240 participants in total. Our results favoured a two-kernel delta-rule model of learning, which specifies that humans learn by prediction error, and respond according to a learning rate that is sensitive to environmental volatility. However, we found substantial heterogeneity in learning processes across participants. These outcomes speak to the learning processes underlying how humans estimate automation reliability and thus have implications for practice.

Calibration of Trust in Automated Driving: A Matter of Initial Level of Trust and Automated Driving Style?

OBJECTIVE

Automated driving is becoming a reality, and such technology raises new concerns about human-machine interaction on road. This paper aims to investigate factors influencing trust calibration and evolution over time.

BACKGROUND

Numerous studies showed trust was a determinant in automation use and misuse, particularly in the automated driving context.

METHOD

Sixty-one drivers participated in an experiment aiming to better understand the influence of initial level of trust (Trustful vs. Distrustful) on drivers' behaviors and trust calibration during two sessions of simulated automated driving. The automated driving style was manipulated as positive (smooth) or negative (abrupt) to investigate human-machine early interactions. Trust was assessed over time through questionnaires. Drivers' visual behaviors and take-over performances during an unplanned take-over request were also investigated.

RESULTS

Results showed an increase of trust over time, for both Trustful and Distrustful drivers regardless the automated driving style. Trust was also found to fluctuate over time depending on the specific events handled by the automated vehicle. Take-over performances were not influenced by the initial level of trust nor automated driving style.

CONCLUSION

Trust in automated driving increases rapidly when drivers' experience such a system. Initial level of trust seems to be crucial in further trust calibration and modulate the effect of automation performance. Long-term trust evolutions suggest that experience modify drivers' mental model about automated driving systems.

APPLICATION

In the automated driving context, trust calibration is a decisive question to guide such systems' proper utilization, and road safety.

The Perception of Automation Reliability and Acceptance of Automated Advice

OBJECTIVE

Examine (1) the extent to which humans can accurately estimate automation reliability and calibrate to changes in reliability, and how this is impacted by the recent accuracy of automation; and (2) factors that impact the acceptance of automated advice, including true automation reliability, reliability perception, and the difference between an operator's perception of automation reliability and perception of their own reliability.

BACKGROUND

Existing evidence suggests humans can adapt to changes in automation reliability but generally underestimate reliability. Cognitive science indicates that humans heavily weight evidence from more recent experiences.

METHOD

Participants monitored the behavior of maritime vessels (contacts) in order to classify them, and then received advice from automation regarding classification. Participants were assigned to either an initially high (90%) or low (60%) automation reliability condition. After some time, reliability switched to 75% in both conditions.

RESULTS

Participants initially underestimated automation reliability. After the change in true reliability, estimates in both conditions moved towards the common true reliability, but did not reach it. There were recency effects, with lower future reliability estimates immediately following incorrect automation advice. With lower initial reliability, automation acceptance rates tracked true reliability more closely than perceived reliability. A positive difference between participant assessments of the reliability of automation and their own reliability predicted greater automation acceptance.

CONCLUSION

Humans underestimate the reliability of automation, and we have demonstrated several critical factors that impact the perception of automation reliability and automation use.

APPLICATION

The findings have potential implications for training and adaptive human-automation teaming.

Understanding trust calibration in automated driving: the effect of time, personality, and system warning design

Under the human-automation codriving future, dynamic trust should be considered. This paper explored how trust changes over time and how multiple factors (time, trust propensity, neuroticism, and takeover warning design) calibrate trust together. We launched two driving simulator experiments to measure drivers' trust before, during, and after the experiment under takeover scenarios. The results showed that trust in automation increased during short-term interactions and dropped after four months, which is still higher than pre-experiment trust. Initial trust and trust propensity had a stable impact on trust. Drivers trusted the system more with the two-stage (MR + TOR) warning design than the one-stage (TOR). Neuroticism had a significant effect on the countdown compared with the content warning.Practitioner summary: The results provide new data and knowledge for trust calibration in the takeover scenario. The findings can help design a more reasonable automated driving system in long-term human-automation interactions.

Should my automated car drive as I do? Investigating speed preferences of drivengers in various driving conditions

Studies investigating the question of how automated cars (ACs) should drive converge to show that a personalized automated driving-style, i.e., mimicking the driving-style of the human behind the wheel, has a positive influence on various aspects of his experience (e.g., comfort). However, few studies have investigated the fact that these benefits might vary with respect to driver-related variables, such as trust in ACs, and contextual variables of the driving activity, such as weather conditions. Additionally, the context of intermediate levels of automation, such as SAE level 3, remains largely unexplored. The objective of this study was to investigate these points. In a scenario-based experimental protocol, participants were exposed to written scenarios in which a character is driven by a SAE level 3 AC in different combinations of conditions (i.e., types of roads, weather conditions and traffic congestion levels). For each condition, participants were asked to indicate how fast they would prefer their AC to drive and how fast they would manually drive in the same situation. Through analyses of variance and equivalence tests, results showed a tendency for participants to overall prefer a slightly lower AC speed than their own. However, a linear regression analysis showed that while participants with the lowest levels of trust preferred an AC speed lower than theirs, those with the highest levels preferred an AC speed nearly identical to theirs. Overall, the results of this study suggest that it would be more beneficial to implement a personalization approach for the design of automated driving-styles rather than a one for all approach.

Measures for explainable AI: Explanation goodness, user satisfaction, mental models, curiosity, trust, and human-AI performance

If a user is presented an AI system that portends to explain how it works, how do we know whether the explanation works and the user has achieved a pragmatic understanding of the AI? This question entails some key concepts of measurement such as explanation goodness and trust. We present methods for enabling developers and researchers to: (1) Assess the a priori goodness of explanations, (2) Assess users' satisfaction with explanations, (3) Reveal user's mental model of an AI system, (4) Assess user's curiosity or need for explanations, (5) Assess whether the user's trust and reliance on the AI are appropriate, and finally, (6) Assess how the human-XAI work system performs. The methods we present derive from our integration of extensive research literatures and our own psychometric evaluations. We point to the previous research that led to the measurement scales which we aggregated and tailored specifically for the XAI context. Scales are presented in sufficient detail to enable their use by XAI researchers. For Mental Model assessment and Work System Performance, XAI researchers have choices. We point to a number of methods, expressed in terms of methods' strengths and weaknesses, and pertinent measurement issues.

Human behavioral response to fluctuating automation reliability

Human perception of automation reliability and automation acceptance behaviours are key to effective human-automation teaming. This study examined factors that impact perceptions of automation reliability over time and the acceptance of automated advice. Participants completed a maritime vessel classification task in which they classified vessels (contacts) with the assistance of automation. In Experiment 1 automation reliability successively switched from high to low (or vice versa). In Experiment 2 automation reliability decreased by varying magnitudes before returning to high. Participants did not initially calibrate to true reliability and experiencing low automation reliability reduced future reliability estimates when experiencing subsequent high reliability. Automation acceptance was predicted by positive differences between participant perception of automation reliability and confidence in their own manual classification reliability. Experiencing low automation reliability caused perceptions of reliability and automation acceptance rates to diverge. These findings have important implications for training and adaptive human-automation teaming in complex work environments.

Evaluating Effective Connectivity of Trust in Human-Automation Interaction: A Dynamic Causal Modeling (DCM) Study

OBJECTIVE

Using dynamic causal modeling (DCM), we examined how credibility and reliability affected the way brain regions exert causal influence over each other-effective connectivity (EC)-in the context of trust in automation.

BACKGROUND

Multiple brain regions of the central executive network (CEN) and default mode network (DMN) have been implicated in trust judgment. However, the neural correlates of trust judgment are still relatively unexplored in terms of the directed information flow between brain regions.

METHOD

Sixteen participants observed the performance of four computer algorithms, which differed in credibility and reliability, of the system monitoring subtask of the Air Force Multi-Attribute Task Battery (AF-MATB). Using six brain regions of the CEN and DMN commonly identified to be activated in human trust, a total of 30 (forward, backward, and lateral) connection models were developed. Bayesian model averaging (BMA) was used to quantify the connectivity strength among the brain regions.

RESULTS

Relative to the high trust condition, low trust showed unique presence of specific connections, greater connectivity strengths from the prefrontal cortex, and greater network complexity. High trust condition showed no backward connections.

CONCLUSION

Results indicated that trust and distrust can be two distinctive neural processes in human-automation interaction-distrust being a more complex network than trust, possibly due to the increased cognitive load.

APPLICATION

The causal architecture of distributed brain regions inferred using DCM can help not only in the design of a balanced human-automation interface design but also in the proper use of automation in real-life situations.

Trust in the Danger Zone: Individual Differences in Confidence in Robot Threat Assessments

Effective human–robot teaming (HRT) increasingly requires humans to work with intelligent, autonomous machines. However, novel features of intelligent autonomous systems such as social agency and incomprehensibility may influence the human’s trust in the machine. The human operator’s mental model for machine functioning is critical for trust. People may consider an intelligent machine partner as either an advanced tool or as a human-like teammate. This article reports a study that explored the role of individual differences in the mental model in a simulated environment. Multiple dispositional factors that may influence the dominant mental model were assessed. These included the Robot Threat Assessment (RoTA), which measures the person’s propensity to apply tool and teammate models in security contexts. Participants (N = 118) were paired with an intelligent robot tasked with making threat assessments in an urban setting. A transparency manipulation was used to influence the dominant mental model. For half of the participants, threat assessment was described as physics-based (e.g., weapons sensed by sensors); the remainder received transparency information that described psychological cues (e.g., facial expression). We expected that the physics-based transparency messages would guide the participant toward treating the robot as an advanced machine (advanced tool mental model activation), while psychological messaging would encourage perceptions of the robot as acting like a human partner (teammate mental model). We also manipulated situational danger cues present in the simulated environment. Participants rated their trust in the robot’s decision as well as threat and anxiety, for each of 24 urban scenes. They also completed the RoTA and additional individual-difference measures. Findings showed that trust assessments reflected the degree of congruence between the robot’s decision and situational danger cues, consistent with participants acting as Bayesian decision makers. Several scales, including the RoTA, were more predictive of trust when the robot was making psychology-based decisions, implying that trust reflected individual differences in the mental model of the robot as a teammate. These findings suggest scope for designing training that uncovers and mitigates the individual’s biases toward intelligent machines.

Influences on User Trust in Healthcare Artificial Intelligence: A Systematic Review

Background: Artificial Intelligence (AI) is becoming increasingly prominent in domains such as healthcare. It is argued to be transformative through altering the way in which healthcare data is used. The realisation and success of AI depend heavily on people’s trust in its applications. Yet, influences on trust in healthcare AI (HAI) applications so far have been underexplored. The objective of this study was to identify aspects related to users, AI applications and the wider context influencing trust in HAI. Methods: We performed a systematic review to map out influences on user trust in HAI. To identify relevant studies, we searched seven electronic databases in November 2019 (ACM digital library, IEEE Explore, NHS Evidence, ProQuest Dissertations & Thesis Global, PsycINFO, PubMed, Web of Science Core Collection). Searches were restricted to publications available in English and German. To be included studies had to be empirical; focus on an AI application (excluding robotics) in a health-related setting; and evaluate applications with regards to users. Results: Three studies, one mixed-method and two qualitative studies in English were included. Influences on trust fell into three broad categories: human-related (knowledge, expectation, mental model, self-efficacy, type of user, age, gender), AI-related (data privacy and safety, operational safety, transparency, design, customizability, trialability, explainability, understandability, power-control-balance, benevolence) and context-related (AI company, media, users’ social network). The factors resulted in an updated logic model illustrating the relationship between these aspects. Conclusion: Trust in HAI depends on a variety of factors, both external and internal to AI applications. This study contributes to our understanding of what influences trust in HAI by highlighting key influences, as well as pointing to gaps and issues in existing research on trust and AI. In so doing, it offers a starting point for further investigation of trust environments as well as trustworthy AI applications.

Real-Time Estimation of Drivers’ Trust in Automated Driving Systems

Trust miscalibration issues, represented by undertrust and overtrust, hinder the interaction between drivers and self-driving vehicles. A modern challenge for automotive engineers is to avoid these trust miscalibration issues through the development of techniques for measuring drivers’ trust in the automated driving system during real-time applications execution. One possible approach for measuring trust is through modeling its dynamics and subsequently applying classical state estimation methods. This paper proposes a framework for modeling the dynamics of drivers’ trust in automated driving systems and also for estimating these varying trust levels. The estimation method integrates sensed behaviors (from the driver) through a Kalman filter-based approach. The sensed behaviors include eye-tracking signals, the usage time of the system, and drivers’ performance on a non-driving-related task. We conducted a study ($$n=80$$ n = 80 ) with a simulated SAE level 3 automated driving system, and analyzed the factors that impacted drivers’ trust in the system. Data from the user study were also used for the identification of the trust model parameters. Results show that the proposed approach was successful in computing trust estimates over successive interactions between the driver and the automated driving system. These results encourage the use of strategies for modeling and estimating trust in automated driving systems. Such trust measurement technique paves a path for the design of trust-aware automated driving systems capable of changing their behaviors to control drivers’ trust levels to mitigate both undertrust and overtrust.

Exploring Personalised Autonomous Vehicles to Influence User Trust

Trust is a major determinant of acceptance of an autonomous vehicle (AV), and a lack of appropriate trust could prevent drivers and society in general from taking advantage of such technology. This paper makes a new attempt to explore the effects of personalised AVs as a novel approach to the cognitive underpinnings of drivers’ trust in AVs. The personalised AV system is able to identify the driving behaviours of users and thus adapt the driving style of the AV accordingly. A prototype of a personalised AV was designed and evaluated in a lab-based experimental study of 36 human drivers, which investigated the impact of the personalised AV on user trust when compared with manual human driving and non-personalised AVs. The findings show that a personalised AV appears to be significantly more reliable through accepting and understanding each driver’s behaviour, which could thereby increase a user’s willingness to trust the system. Furthermore, a personalised AV brings a sense of familiarity by making the system more recognisable and easier for users to estimate the quality of the automated system. Personalisation parameters were also explored and discussed to support the design of AV systems to be more socially acceptable and trustworthy.

Automation-Induced Complacency Potential: Development and Validation of a New Scale

Complacency, or sub-optimal monitoring of automation performance, has been cited as a contributing factor in numerous major transportation and medical incidents. Researchers are working to identify individual differences that correlate with complacency as one strategy for preventing complacency-related accidents. Automation-induced complacency potential is an individual difference reflecting a general tendency to be complacent across a wide variety of situations which is similar to, but distinct from trust. Accurately assessing complacency potential may improve our ability to predict and prevent complacency in safety-critical occupations. Much past research has employed an existing measure of complacency potential. However, in the 25 years since that scale was published, our conceptual understanding of complacency itself has evolved, and we propose that an updated scale of complacency potential is needed. The goal of the present study was to develop, and provide initial validation evidence for, a new measure of automation-induced complacency potential that parallels the current conceptualization of complacency. In a sample of 475 online respondents, we tested 10 new items and found that they clustered into two separate scales: Alleviating Workload (which focuses on attitudes about the use of automation to ease workloads) and Monitoring (which focuses on attitudes toward monitoring of automation). Alleviating workload correlated moderately with the existing complacency potential rating scale, while monitoring did not. Further, both the alleviating workload and monitoring scales showed discriminant validity from the previous complacency potential scale and from similar constructs, such as propensity to trust. In an initial examination of criterion-related validity, only the monitoring-focused scale had a significant relationship with hypothetical complacency (r = -0.42, p < 0.01), and it had significant incremental validity over and above all other individual difference measures in the study. These results suggest that our new monitoring-related items have potential for use as a measure of automation-induced complacency potential and, compared with similar scales, this new measure may have unique value.

Learning From the Slips of Others: Neural Correlates of Trust in Automated Agents

With the rise of increasingly complex artificial intelligence (AI), there is a need to design new methods to monitor AI in a transparent, human-aware manner. Decades of research have demonstrated that people, who are not aware of the exact performance levels of automated algorithms, often experience a mismatch in expectations. Consequently, they will often provide either too little or too much trust in an algorithm. Detecting such a mismatch in expectations, or trust calibration, remains a fundamental challenge in research investigating the use of automation. Due to the context-dependent nature of trust, universal measures of trust have not been established. Trust is a difficult construct to investigate because even the act of reflecting on how much a person trusts a certain agent can change the perception of that agent. We hypothesized that electroencephalograms (EEGs) would be able to provide such a universal index of trust without the need of self-report. In this work, EEGs were recorded for 21 participants (mean age = 22.1; 13 females) while they observed a series of algorithms perform a modified version of a flanker task. Each algorithm's degree of credibility and reliability were manipulated. We hypothesized that neural markers of action monitoring, such as the observational error-related negativity (oERN) and observational error positivity (oPe), are potential candidates for monitoring computer algorithm performance. Our findings demonstrate that (1) it is possible to reliably elicit both the oERN and oPe while participants monitored these computer algorithms, (2) the oPe, as opposed to the oERN, significantly distinguished between high and low reliability algorithms, and (3) the oPe significantly correlated with subjective measures of trust. This work provides the first evidence for the utility of neural correlates of error monitoring for examining trust in computer algorithms.

Introduction matters: Manipulating trust in automation and reliance in automated driving

Trust in automation is a key determinant for the adoption of automated systems and their appropriate use. Therefore, it constitutes an essential research area for the introduction of automated vehicles to road traffic. In this study, we investigated the influence of trust promoting (Trust promoted group) and trust lowering (Trust lowered group) introductory information on reported trust, reliance behavior and take-over performance. Forty participants encountered three situations in a 17-min highway drive in a conditionally automated vehicle (SAE Level 3). Situation 1 and Situation 3 were non-critical situations where a take-over was optional. Situation 2 represented a critical situation where a take-over was necessary to avoid a collision. A non-driving-related task (NDRT) was presented between the situations to record the allocation of visual attention. Participants reporting a higher trust level spent less time looking at the road or instrument cluster and more time looking at the NDRT. The manipulation of introductory information resulted in medium differences in reported trust and influenced participants' reliance behavior. Participants of the Trust promoted group looked less at the road or instrument cluster and more at the NDRT. The odds of participants of the Trust promoted group to overrule the automated driving system in the non-critical situations were 3.65 times (Situation 1) to 5 times (Situation 3) higher. In Situation 2, the Trust promoted group's mean take-over time was extended by 1154 ms and the mean minimum time-to-collision was 933 ms shorter. Six participants from the Trust promoted group compared to no participant of the Trust lowered group collided with the obstacle. The results demonstrate that the individual trust level influences how much drivers monitor the environment while performing an NDRT. Introductory information influences this trust level, reliance on an automated driving system, and if a critical take-over situation can be successfully solved.

The Effect of Incorrect Reliability Information on Expectations, Perceptions, and Use of Automation

OBJECTIVE

We examined how providing artificially high or low statements about automation reliability affected expectations, perceptions, and use of automation over time.

BACKGROUND

One common method of introducing automation is providing explicit statements about the automation's capabilities. Research is needed to understand how expectations from such introductions affect perceptions and use of automation.

METHOD

Explicit-statement introductions were manipulated to set higher-than (90%), same-as (75%), or lower-than (60%) levels of expectations in a dual-task scenario with 75% reliable automation. Two experiments were conducted to assess expectations, perceptions, compliance, reliance, and task performance over (a) 2 days and (b) 4 days.

RESULTS

The baseline assessments showed initial expectations of automation reliability matched introduced levels of expectation. For the duration of each experiment, the lower-than groups' perceptions were lower than the actual automation reliability. However, the higher-than groups' perceptions were no different from actual automation reliability after Day 1 in either study. There were few differences between groups for automation use, which generally stayed the same or increased with experience using the system.

CONCLUSION

Introductory statements describing artificially low automation reliability have a long-lasting impact on perceptions about automation performance. Statements including incorrect automation reliability do not appear to affect use of automation.

APPLICATION

Introductions should be designed according to desired outcomes for expectations, perceptions, and use of the automation. Low expectations have long-lasting effects.