Elicitability of Instance and Object Ranking

Assessing the quality of a forecasting model crucially depends on a proper scoring rule or suitable loss function. As for point forecasts, the existence of a strictly consistent loss function that allows for a fair comparison of competing forecast models has to be guaranteed, which means that the corresponding statistical functional has to be elicitable. We consider instance and object ranking problems that intend to correctly predict the ordering of instances in a data set. A ranking prediction is naturally identified with a point forecast in the respective symmetric group, that is, the forecaster predicts one single permutation of the row indices. We show that, in the presence of ties, this strategy does not allow for strictly consistent scoring functions because of multiple true permutations. Those multiple optima cannot be entirely covered by a single point forecast, which causes all corresponding optima to be minimizers of standard scoring functions that operate on symmetric groups, so these scoring functions are not strictly consistent. As a remedy, we consider accurately accounting for ties. This is done by treating each configuration of clear orderings and ties as an additional category, which induces extended decision spaces with a clearly defined single optimum. Because these decision spaces are still finite, each type of instance ranking problem that we consider in this work and corresponding ranking functional, mapping into a symmetric group, can be identified with a certain classification problem and corresponding classification functional, mapping into one of our extended decision spaces, which is elicitable.

Learning the Preferences of Physicians for the Organization of Result Lists of Medical Evidence Articles

Background: Online medical knowledge repositories such as MEDLINE and The Cochrane Library are increasingly used by physicians to retrieve articles to aid with clinical decision making. The prevailing approach for organizing retrieved articles is in the form of a rank-ordered list, with the assumption that the higher an article is presented on a list, the more relevant it is.

Objectives: Despite this common list-based organization, it is seldom studied how physicians perceive the association between the relevance of articles and the order in which articles are presented. In this paper we describe a case study that captured physician preferences for 3-element lists of medical articles in order to learn how to organize medical knowledge for decision-making.

Methods: Comprehensive relevance evaluations were developed to represent 3-element lists of hypothetical articles that may be retrieved from an online medical knowledge source such as MEDLINE or The Cochrane Library. Comprehensive relevance evalua tions asses not only an article’s relevance for a query, but also whether it has been placed on the correct list position. In other words an article may be relevant and correctly placed on a result list (e.g. the most relevant article appears first in the result list), an article may be relevant for a query but placed on an incorrect list position (e.g. the most relevant article appears second in a result list), or an article may be irrelevant for a query yet still appear in the result list. The relevance evaluations were presented to six senior physi cians who were asked to express their preferences for an article’s relevance and its position on a list by pairwise comparisons representing different combinations of 3-element lists. The elicited preferences were assessed using a novel GRIP (Generalized Regression with Intensities of Preference) method and represented as an additive value function. Value functions were derived for individual physicians as well as the group of physicians.

Results: The results show that physicians assign significant value to the 1st position on a list and they expect that the most relevant article is presented first. Whilst physicians still prefer obtaining a correctly placed article on position 2, they are also quite satisfied with misplaced relevant article. Low consideration of the 3rd position was uniformly confirmed.

Conclusions: Our findings confirm the importance of placing the most relevant article on the 1st position on a list and the importance paid to position on a list significantly diminishes after the 2nd position. The derived value functions may be used by developers of clinical decision support applications to decide how best to organize medical knowledge for decision making and to create personalized evaluation measures that can augment typical measures used to evaluate information retrieval systems.