1
|
McGregor M, Azzopardi L, Halvey M. A Systematic Review of Cost, Effort, and Load Research in Information Search and Retrieval, 1972-2020. ACM T INFORM SYST 2023. [DOI: 10.1145/3583069] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/11/2023]
Abstract
During the
Information Search and Retrieval
(ISR) process, user-system interactions such as submitting queries, examining results, and engaging with information, impose some degree of demand on the user’s resources. Within ISR, these demands are well recognised, and numerous studies have demonstrated that the Cost, Effort, and Load (CEL) experienced during the search process are affected by a variety of factors. Despite this recognition, there is no universally accepted definition of the constructs of CEL within the field of ISR. Ultimately this has led to problems with how these constructs have been interpreted and subsequently measured. This systematic review contributes a synthesis of literature, summarising key findings relating to how researchers have been defining and measuring CEL within ISR over the past 50 years. After manually screening 1,109 articles, we detail and analyse 91 articles which examine CEL within ISR. The discussion focuses on comparing the similarities and differences between CEL definitions and measures before identifying the limitations of the current state of the nomenclature. Opportunities for future research are also identified. Going forward, we propose a CEL taxonomy that integrates the relationships between CEL and their related constructs, which will help focus and disambiguate future research in this important area.
Collapse
Affiliation(s)
- Molly McGregor
- Department of Computer and Information Sciences, University of Strathclyde, UK
| | - Leif Azzopardi
- Department of Computer and Information Sciences, University of Strathclyde, UK
| | - Martin Halvey
- Department of Computer and Information Sciences, University of Strathclyde, UK
| |
Collapse
|
2
|
Daronnat S, Azzopardi L, Halvey M, Dubiel M. Inferring Trust From Users' Behaviours; Agents' Predictability Positively Affects Trust, Task Performance and Cognitive Load in Human-Agent Real-Time Collaboration. Front Robot AI 2021; 8:642201. [PMID: 34307467 PMCID: PMC8295498 DOI: 10.3389/frobt.2021.642201] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2020] [Accepted: 06/10/2021] [Indexed: 11/23/2022] Open
Abstract
Collaborative virtual agents help human operators to perform tasks in real-time. For this collaboration to be effective, human operators must appropriately trust the agent(s) they are interacting with. Multiple factors influence trust, such as the context of interaction, prior experiences with automated systems and the quality of the help offered by agents in terms of its transparency and performance. Most of the literature on trust in automation identified the performance of the agent as a key factor influencing trust. However, other work has shown that the behavior of the agent, type of the agent’s errors, and predictability of the agent’s actions can influence the likelihood of the user’s reliance on the agent and efficiency of tasks completion. Our work focuses on how agents’ predictability affects cognitive load, performance and users’ trust in a real-time human-agent collaborative task. We used an interactive aiming task where participants had to collaborate with different agents that varied in terms of their predictability and performance. This setup uses behavioral information (such as task performance and reliance on the agent) as well as standardized survey instruments to estimate participants’ reported trust in the agent, cognitive load and perception of task difficulty. Thirty participants took part in our lab-based study. Our results showed that agents with more predictable behaviors have a more positive impact on task performance, reliance and trust while reducing cognitive workload. In addition, we investigated the human-agent trust relationship by creating models that could predict participants’ trust ratings using interaction data. We found that we could reliably estimate participants’ reported trust in the agents using information related to performance, task difficulty and reliance. This study provides insights on behavioral factors that are the most meaningful to anticipate complacent or distrusting attitudes toward automation. With this work, we seek to pave the way for the development of trust-aware agents capable of responding more appropriately to users by being able to monitor components of the human-agent relationships that are the most salient for trust calibration.
Collapse
Affiliation(s)
- Sylvain Daronnat
- Department of Computer and Information Sciences, University of Strathclyde, Glasgow, United Kingdom
| | - Leif Azzopardi
- Department of Computer and Information Sciences, University of Strathclyde, Glasgow, United Kingdom
| | - Martin Halvey
- Department of Computer and Information Sciences, University of Strathclyde, Glasgow, United Kingdom
| | - Mateusz Dubiel
- Department of Computer and Information Sciences, University of Strathclyde, Glasgow, United Kingdom
| |
Collapse
|
7
|
Travers B, Henderson S, Vasireddy S, SeQueira EJ, Cornell PJ, Richards S, Khan A, Hasan S, Withrington R, Leak A, Sandhu J, Joseph A, Packham JC, Lyle S, Martin JC, Goodfellow RM, Rhys-Dillon C, Morgan JT, Mogford S, Rowan-Phillips J, Moss D, Wilson H, McEntegart A, Morgan JT, Martin JC, Rhys Dillon C, Goodfellow R, Gould L, Bukhari M, Hassan S, Butt S, Deighton C, Gadsby K, Love V, Kara N, Gohery M, Keat A, Lewis A, Robinson R, Bastawrous S, Roychowdhury B, Roskell S, Douglas B, Keating H, Giles S, McPeake J, Molloy C, Chalam V, Mulherin D, Price T, Sheeran T, Benjamin SR, Thompson PW, Cornell P, Siddle HJ, Backhouse MR, Monkhouse RA, Harris NJ, Helliwell PS, Azzopardi L, Hudson S, Mallia C, Cassar K, Coleiro B, Cassar PJ, Aquilina D, Camilleri F, Serracino Inglott A, Azzopardi LM, Robinson S, Peta H, Margot L, David W, Mann C, Gooberman-Hill R, Jagannath D, Healey E, Goddard C, Pugh MT, Gilham L, Bawa S, Barlow JH, MacFarland L, Tindall L, Leddington Wright S, Tooby J, Ravindran J, Perkins P, McGregor L, Mabon E, Bawa S, Bond U, Swan J, O'Connor MB, Rathi J, Regan MJ, Phelan MJ, Doherty T, Martin K, Ruth C, Panthakalam S, Bondin D, Castelino M, Evin S, Gooden A, Peacock C, Teh LS, Ryan SJ, Bryant E, Carter A, Cox S, Moore AP, Jackson A, Kuisma R, Pattman J, Juarez M, Quilter A, Williamson L, Collins D, Price E, Chao Y, Mooney J, Watts R, Graham K, Birrell F, Reed M, Croyle S, Stell J, Vasireddy S, Storrs P, McLoughlin YM, Scott G, McKenna F, Papou A, Rahmeh FH, Richards SC, Westlake SL, Birrell F, Morgan L, Baqir W, Walsh NE, Ward L, Caine R, Williams M, Breslin A, Owen C, Ahmad Y, Morgan L, Blair A, Birrell F, Ramachandran Nair J, Zia A, Mewar D, Peffers GM, Larder R, Dockrell D, Wilson S, Cummings J, Bansal J, Barlow J. BHPR: Audit/Service Delivery [239-277]: 239. Arma-Based Audit of Rheumatology Service Delivered Predominantly Outside the Traditional Hospital Setting. Rheumatology (Oxford) 2010. [DOI: 10.1093/rheumatology/keq730] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/14/2022] Open
|
10
|
Abstract
Although the seminal proposal to introduce language modeling in information retrieval was based on a multivariate Bernoulli model, the predominant modeling approach is now centered on multinomial models. Language modeling for retrieval based on multivariate Bernoulli distributions is seen inefficient and believed less effective than the multinomial model. In this article, we examine the multivariate Bernoulli model with respect to its successor and examine its role in future retrieval systems. In the context of Bayesian learning, these two modeling approaches are described, contrasted, and compared both theoretically and computationally. We show that the query likelihood following a multivariate Bernoulli distribution introduces interesting retrieval features which may be useful for specific retrieval tasks such as sentence retrieval. Then, we address the efficiency aspect and show that algorithms can be designed to perform retrieval efficiently for multivariate Bernoulli models, before performing an empirical comparison to study the behaviorial aspects of the models. A series of comparisons is then conducted on a number of test collections and retrieval tasks to determine the empirical and practical differences between the different models. Our results indicate that for sentence retrieval the multivariate Bernoulli model can significantly outperform the multinomial model. However, for the other tasks the multinomial model provides consistently better performance (and in most cases significantly so). An analysis of the various retrieval characteristics reveals that the multivariate Bernoulli model tends to promote long documents whose nonquery terms are informative. While this is detrimental to the task of document retrieval (documents tend to contain considerable nonquery content), it is valuable for other tasks such as sentence retrieval, where the retrieved elements are very short and focused.
Collapse
|