Determinants of Difficulty and Discriminating Power of Image-based Test Items in Postgraduate Radiological Examinations

Scoring Single-Response Multiple-Choice Items: Scoping Review and Comparison of Different Scoring Methods

BACKGROUND

Single-choice items (eg, best-answer items, alternate-choice items, single true-false items) are 1 type of multiple-choice items and have been used in examinations for over 100 years. At the end of every examination, the examinees' responses have to be analyzed and scored to derive information about examinees' true knowledge.

OBJECTIVE

The aim of this paper is to compile scoring methods for individual single-choice items described in the literature. Furthermore, the metric expected chance score and the relation between examinees' true knowledge and expected scoring results (averaged percentage score) are analyzed. Besides, implications for potential pass marks to be used in examinations to test examinees for a predefined level of true knowledge are derived.

METHODS

Scoring methods for individual single-choice items were extracted from various databases (ERIC, PsycInfo, Embase via Ovid, MEDLINE via PubMed) in September 2020. Eligible sources reported on scoring methods for individual single-choice items in written examinations including but not limited to medical education. Separately for items with n=2 answer options (eg, alternate-choice items, single true-false items) and best-answer items with n=5 answer options (eg, Type A items) and for each identified scoring method, the metric expected chance score and the expected scoring results as a function of examinees' true knowledge using fictitious examinations with 100 single-choice items were calculated.

RESULTS

A total of 21 different scoring methods were identified from the 258 included sources, with varying consideration of correctly marked, omitted, and incorrectly marked items. Resulting credit varied between -3 and +1 credit points per item. For items with n=2 answer options, expected chance scores from random guessing ranged between -1 and +0.75 credit points. For items with n=5 answer options, expected chance scores ranged between -2.2 and +0.84 credit points. All scoring methods showed a linear relation between examinees' true knowledge and the expected scoring results. Depending on the scoring method used, examination results differed considerably: Expected scoring results from examinees with 50% true knowledge ranged between 0.0% (95% CI 0% to 0%) and 87.5% (95% CI 81.0% to 94.0%) for items with n=2 and between -60.0% (95% CI -60% to -60%) and 92.0% (95% CI 86.7% to 97.3%) for items with n=5.

CONCLUSIONS

In examinations with single-choice items, the scoring result is not always equivalent to examinees' true knowledge. When interpreting examination scores and setting pass marks, the number of answer options per item must usually be taken into account in addition to the scoring method used.

Recognition of affective prosody in bipolar and depressive conditions: A systematic review and meta-analysis

BACKGROUND

Inconsistent results have been reported about the affective prosody recognition (APR) ability in patients with bipolar (BD) and depressive (DD) disorders. We aimed to (i) evaluate the magnitude of APR dysfunction in BD and DD patients, (ii) identify moderators for heterogeneous results, and (iii) highlight research trends in this field.

METHODS

A computerized literature search was conducted in five electronic databases from the inception to May 9th, 2022 to identify behavioural experiments that studied APR in BD or DD patients. Effect sizes were calculated using a random-effect model and recalculated after removing outliers and adjusting publication bias.

RESULTS

Twelve eligible articles totalling 16 studies were included in the meta-analysis, aggregating 612 patients and 809 healthy controls. Individual r² ranged from 0.008 to 0.355, six of which reached a medium-to-large association strength. A medium-to-large pooled effect size (Hedges g = -0.58, 95 % CI -0.75 to -0.40, p < 0.001) for overall APR impairment in BD and DD patients was obtained. The Beck Depression Inventory score and answer option number were significant moderators. Neuropsychological mechanisms, multi-modal interaction and comorbidity effects have become primary research concerns.

LIMITATIONS

Extant statistics were insufficient for disorder-specific analysis.

CONCLUSIONS

Current findings demonstrate deficits of overall APR in BD and DD patients at a medium-to-large magnitude. APR can clinically serve for early screening and prognosis, but the depression severity, task complexity and confounding variables influence patients' APR performance. Future studies should incorporate neuroimaging approaches and investigate the effects of tonal language stimuli and clinical interventions.

Written-Based Progress Testing: A Scoping Review

PURPOSE

Progress testing is an increasingly popular form of assessment in which a comprehensive test is administered to learners repeatedly over time. To inform potential users, this scoping review aimed to document barriers, facilitators, and potential outcomes of the use of written progress tests in higher education.

METHOD

The authors followed Arksey and O'Malley's scoping review methodology to identify and summarize the literature on progress testing. They searched 6 databases (Academic Search Complete, CINAHL, ERIC, Education Source, MEDLINE, and PsycINFO) on 2 occasions (May 22, 2018, and April 21, 2020) and included articles written in English or French and pertaining to written progress tests in higher education. Two authors screened articles for the inclusion criteria (90% agreement), then data extraction was performed by pairs of authors. Using a snowball approach, the authors also screened additional articles identified from the included reference lists. They completed a thematic analysis through an iterative process.

RESULTS

A total of 104 articles were included. The majority of progress tests used a multiple-choice and/or true-or-false question format (95, 91.3%) and were administered 4 times a year (38, 36.5%). The most documented source of validity evidence was internal consistency (38, 36.5%). Four major themes were identified: (1) barriers and challenges to the implementation of progress testing (e.g., need for additional resources); (2) established collaboration as a facilitator of progress testing implementation; (3) factors that increase the acceptance of progress testing (e.g., formative use); and (4) outcomes and consequences of progress test use (e.g., progress testing contributes to an increase in knowledge).

CONCLUSIONS

Progress testing appears to have a positive impact on learning, and there is significant validity evidence to support its use. Although progress testing is resource- and time-intensive, strategies such as collaboration with other institutions may facilitate its use.

Recognition of affective prosody in autism spectrum conditions: A systematic review and meta-analysis

LAY ABSTRACT

Differences in understanding others' emotions and attitudes through features in speech (e.g. intonation) have been observed in individuals with autism spectrum conditions, which contribute greatly to their social communication challenges. However, some studies reported that individuals with autism spectrum condition performed comparably to typically developing individuals on affective prosody recognition. Here, we provide a comprehensive review with statistical analysis of 23 existing studies on this topic to examine potential factors that could explain the discrepancies. Compared with typically developing individuals, autism spectrum condition participants generally appeared to encounter more difficulties in affective prosody recognition. But this finding was likely due to the tendency of the existing research to overly focus on deficits in autism. The affective prosody recognition performance in individuals with autism spectrum condition was closely related to the number of answer options offered to them. Moreover, the degree of difficulty in affective prosody recognition encountered by individuals with autism spectrum condition varied across emotions. The findings of this systematic review highlighted the need for further research on affective prosody recognition in autism (e.g. studies that include tonal language speakers and autism spectrum condition individuals with lower cognitive or verbal abilities).

Introducing Summative Progress Testing in Radiology Residency: Little Change in Residents' Test Results After Transitioning from Formative Progress Testing

INTRODUCTION

Educational effects of transitioning from formative to summative progress testing are unclear. Our purpose was to investigate whether such transitioning in radiology residency is associated with a change in progress test results.

METHODS

We investigated a national cohort of radiology residents (N > 300) who were semi-annually assessed through a mandatory progress test. Until 2014, this test was purely formative for all residents, but in 2014/2015, it was transitioned (as part of a national radiology residency program revision) to include a summative pass requirement for new residents. In 7 posttransitioning tests in 2015-2019, including summatively and formatively tested residents who followed the revised and pre-transitioning residency program, respectively, we assessed residents' relative test scores and percentage of residents that reached pass standards.

RESULTS

Due to our educational setting, most posttransitioning tests had no residents in the summative condition in postgraduate year 4-5, nor residents in the formative condition in year 0.5-2. Across the 7 tests, relative test scores in postgraduate year 1-3 of the summative resident group and year 3.5-4.5 of the formative group differed significantly (p < 0.01 and p < 0.05, respectively, Kruskal-Wallis test). However, scores fluctuated without consistent time trends and without consistent differences between both resident groups. Percentage of residents reaching the pass standard did not differ significantly across tests or between groups.

DISCUSSION

Transitioning from formative to summative progress testing was associated with overall steady test results of the whole resident group in 4 post-transitioning years. We do not exclude that transitioning may have positive educational effects for resident subgroups.