Information retrieval evaluation in practice: A case study approach

Towards standardisation of evidence-based clinical care process specifications

There is a strong push towards standardisation of treatment approaches, care processes and documentation of clinical practice. However, confusion persists regarding terminology and description of many clinical care process specifications which this research seeks to resolve by developing a taxonomic characterisation of clinical care process specifications. Literature on clinical care process specifications was analysed, creating the starting point for identifying common characteristics and how each is constructed and used in the clinical setting. A taxonomy for clinical care process specifications is presented. The De Bleser approach to limited clinical care process specifications characterisation was extended and each clinical care process specification is successfully characterised in terms of purpose, core elements and relationship to the other clinical care process specification types. A case study on the diagnosis and treatment of Type 2 Diabetes in the United Kingdom was used to evaluate the taxonomy and demonstrate how the characterisation framework applies. Standardising clinical care process specifications ensures that the format and content are consistent with expectations, can be read more quickly and high-quality information can be recorded about the patient. Standardisation also enables computer interpretability, which is important in integrating Learning Health Systems into the modern clinical environment. The approach presented allows terminologies for clinical care process specifications that were widely used interchangeably to be easily distinguished, thus, eliminating the existing confusion.

Testing the stability of “wisdom of crowds” judgments of search results over time and their similarity with the search engine rankings

Purpose

– One of the under-explored aspects in the process of user information seeking behaviour is influence of time on relevance evaluation. It has been shown in previous studies that individual users might change their assessment of search results over time. It is also known that aggregated judgements of multiple individual users can lead to correct and reliable decisions; this phenomenon is known as the “wisdom of crowds”. The purpose of this paper is to examine whether aggregated judgements will be more stable and thus more reliable over time than individual user judgements.

Design/methodology/approach

– In this study two simple measures are proposed to calculate the aggregated judgements of search results and compare their reliability and stability to individual user judgements. In addition, the aggregated “wisdom of crowds” judgements were used as a means to compare the differences between human assessments of search results and search engine’s rankings. A large-scale user study was conducted with 87 participants who evaluated two different queries and four diverse result sets twice, with an interval of two months. Two types of judgements were considered in this study: relevance on a four-point scale, and ranking on a ten-point scale without ties.

Findings

– It was found that aggregated judgements are much more stable than individual user judgements, yet they are quite different from search engine rankings.

Practical implications

– The proposed “wisdom of crowds”-based approach provides a reliable reference point for the evaluation of search engines. This is also important for exploring the need of personalisation and adapting search engine’s ranking over time to changes in users preferences.

Originality/value

– This is a first study that applies the notion of “wisdom of crowds” to examine an under-explored in the literature phenomenon of “change in time” in user evaluation of relevance.

A Markov Chain Model for Changes in Users' Assessment of Search Results

Previous research shows that users tend to change their assessment of search results over time. This is a first study that investigates the factors and reasons for these changes, and describes a stochastic model of user behaviour that may explain these changes. In particular, we hypothesise that most of the changes are local, i.e. between results with similar or close relevance to the query, and thus belong to the same"coarse" relevance category. According to the theory of coarse beliefs and categorical thinking, humans tend to divide the range of values under consideration into coarse categories, and are thus able to distinguish only between cross-category values but not within them. To test this hypothesis we conducted five experiments with about 120 subjects divided into 3 groups. Each student in every group was asked to rank and assign relevance scores to the same set of search results over two or three rounds, with a period of three to nine weeks between each round. The subjects of the last three-round experiment were then exposed to the differences in their judgements and were asked to explain them. We make use of a Markov chain model to measure change in users' judgments between the different rounds. The Markov chain demonstrates that the changes converge, and that a majority of the changes are local to a neighbouring relevance category. We found that most of the subjects were satisfied with their changes, and did not perceive them as mistakes but rather as a legitimate phenomenon, since they believe that time has influenced their relevance assessment. Both our quantitative analysis and user comments support the hypothesis of the existence of coarse relevance categories resulting from categorical thinking in the context of user evaluation of search results.

Designing Information Technology Architectures: A Cost-Oriented Methodology

This paper proposes a design methodology of information technology architectures tying organizational requirements to technical choices and costs. The primary goal is to provide a structured support for the selection of the minimum-cost architecture satisfying given organizational requirements. Previous empirical studies have attempted absolute cost comparisons of different architectural solutions, primarily relying on the expertise of practitioners and a priori beliefs, but have rarely taken into account the impact of organizational requirements on costs. Requirements are modelled as information processes, composed of tasks exchanging information and characterized by varying levels of computational complexity. Different architectural distributions of presentation, computation and data management applications are compared. The cost implications of organizational requirements for processing intensity, communication intensity and networking are analysed. The results show a relationship between structural features of information processes and architectural costs and indicate how architectural design should be based on organizational as well as technology considerations.