A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment

Autism Knowledge Assessments: A Closer Examination of Validity by Autism Experts

PURPOSE

The goal of the current study was to conduct a substantive validity review of four autism knowledge assessments with prior psychometric support (Gillespie-Lynch in J Autism and Dev Disord 45(8):2553-2566, 2015; Harrison in J Autism and Dev Disord 47(10):3281-3295, 2017; McClain in J Autism and Dev Disord 50(3):998-1006, 2020; McMahon in Res Autism Spectr Disord 71:101499, 2020). 69 autism experts who served on the editorial board of one or more peer-reviewed autism journals evaluated the accuracy and ambiguity of autism knowledge questions. 34% of the questions were flagged as "potentially problematic" for accuracy, and 17% of the questions were flagged as "potentially problematic" for ambiguity. Autism expert feedback revealed three themes across ambiguous questions: (1) an oversimplification of mixed or still-evolving research literature, (2) an insufficient recognition of the heterogeneity of the autism spectrum, and (3) a lack of clarity in the question/answer prompt. Substantive validity of future autism knowledge assessments should be carefully evaluated via feedback from a diverse group of autism experts and/or potential respondents. Potentially problematic questions can be removed or modified to improve the validity of autism knowledge assessments.

Boosting wisdom of the crowd for medical image annotation using training performance and task features

A crucial bottleneck in medical artificial intelligence (AI) is high-quality labeled medical datasets. In this paper, we test a large variety of wisdom of the crowd algorithms to label medical images that were initially classified by individuals recruited through an app-based platform. Individuals classified skin lesions from the International Skin Lesion Challenge 2018 into 7 different categories. There was a large dispersion in the geographical location, experience, training, and performance of the recruited individuals. We tested several wisdom of the crowd algorithms of varying complexity from a simple unweighted average to more complex Bayesian models that account for individual patterns of errors. Using a switchboard analysis, we observe that the best-performing algorithms rely on selecting top performers, weighting decisions by training accuracy, and take into account the task environment. These algorithms far exceed expert performance. We conclude by discussing the implications of these approaches for the development of medical AI.