Analysis of Collective Human Intelligence for Diagnosis of Pigmented Skin Lesions Harnessed by Gamification Via a Web-Based Training Platform: Simulation Reader Study

Crowdsourcing a diagnosis? Exploring the accuracy of the size and type of group diagnosis: an experimental study

BACKGROUND

The consultation process, where a clinician seeks an opinion from another clinician, is foundational in medicine. However, the effectiveness of group diagnosis has not been studied.

OBJECTIVE

To compare individual diagnosis to group diagnosis on two dimensions: group size (n=3 or 6) and group process (interactive or artificial groups).

METHODOLOGY

Thirty-six internal or emergency medicine residents participated in the study. Initially, each resident worked through four written cases on their own, providing a primary diagnosis and a differential diagnosis. Next, participants formed into groups of three. Using a videoconferencing platform, they worked through four additional cases, collectively providing a single primary diagnosis and differential diagnosis. The process was repeated using a group of six with four new cases. Cases were all counterbalanced. Retrospectively, nominal (ie, artificial) groups were formed by aggregating individual participant data into subgroups of three and six and analytically computing scores. Presence of the correct diagnosis as primary diagnosis or included in the differential diagnosis, as well as the number of diagnoses mentioned, was calculated for all conditions. Means were compared using analysis of variance.

RESULTS

For both authentic and nominal groups, the diagnostic accuracy of group diagnosis was superior to individual for both the primary diagnosis and differential diagnosis. However, there was no improvement in diagnostic accuracy when comparing a group of three to a group of six. Interactive and nominal groups were equivalent; however, this may be an artefact of the method used to combine data.

CONCLUSIONS

Group diagnosis improves diagnostic accuracy. However, a larger group is not necessarily superior to a smaller group. In this study, interactive group discussion does not result in improved diagnostic accuracy.

Gamified learning in dermatology and dermoscopy education: a paradigm shift

Teaching methods in medical education have been changing. More recent teaching modalities have gone beyond the traditional delivery of knowledge, promoting learning motivation, and improving teaching and learning outcomes. 'Gamification' and 'serious games' are methodologies that use the principles of games to facilitate learning processes and the acquisition of skills and knowledge, thereby improving attitudes towards learning when compared with traditional teaching methods. As dermatology is a visual field, images are a key component of different teaching strategies. Likewise, dermoscopy, a noninvasive diagnostic technique that allows the visualization of structures within the epidermis and upper dermis, also uses images and pattern recognition strategies. A series of Apps using game-based strategy have been created to teach and facilitate dermoscopy learning; however, studies are required to demonstrate their effectiveness. This review summarizes the current evidence of game-based learning strategies in medical education, including dermatology and dermoscopy.

A reinforcement learning model for AI-based decision support in skin cancer

We investigated whether human preferences hold the potential to improve diagnostic artificial intelligence (AI)-based decision support using skin cancer diagnosis as a use case. We utilized nonuniform rewards and penalties based on expert-generated tables, balancing the benefits and harms of various diagnostic errors, which were applied using reinforcement learning. Compared with supervised learning, the reinforcement learning model improved the sensitivity for melanoma from 61.4% to 79.5% (95% confidence interval (CI): 73.5-85.6%) and for basal cell carcinoma from 79.4% to 87.1% (95% CI: 80.3-93.9%). AI overconfidence was also reduced while simultaneously maintaining accuracy. Reinforcement learning increased the rate of correct diagnoses made by dermatologists by 12.0% (95% CI: 8.8-15.1%) and improved the rate of optimal management decisions from 57.4% to 65.3% (95% CI: 61.7-68.9%). We further demonstrated that the reward-adjusted reinforcement learning model and a threshold-based model outperformed naïve supervised learning in various clinical scenarios. Our findings suggest the potential for incorporating human preferences into image-based diagnostic algorithms.

[Artificial intelligence-based classification for the diagnostics of skin cancer]

Convolutional neural networks (CNN) achieve a level of performance comparable or even superior to dermatologists in the assessment of pigmented and nonpigmented skin lesions. In the analysis of images by artificial neural networks, images on a pixel level pass through various layers of the network with different graphic filters. Based on excellent study results, a first deep learning network (Moleanalyzer pro, Fotofinder Systems GmBH, Bad Birnbach, Germany) received market approval in Europe. However, such neural networks also reveal relevant limitations, whereby rare entities with insufficient training images are classified less adequately and image artifacts can lead to false diagnoses. Best results can ultimately be achieved in a cooperation of "man with machine". For future skin cancer screening, automated total body mapping is evaluated, which combines total body photography, automated data extraction and assessment of all relevant skin lesions.

Assessment of melanoma thickness based on dermoscopy images: an open, web-based, international, diagnostic study

BACKGROUND

Preoperative assessment of whether a melanoma is invasive or in situ (MIS) is a common task that might have important implications for triage, prognosis and the selection of surgical margins. Several dermoscopic features suggestive of melanoma have been described, but only a few of these are useful in differentiating MIS from invasive melanoma.

OBJECTIVE

The primary aim of this study was to evaluate how accurately a large number of international readers, individually as well as collectively, were able to discriminate between MIS and invasive melanomas as well as estimate the Breslow thickness of invasive melanomas based on dermoscopy images. The secondary aim was to compare the accuracy of two machine learning convolutional neural networks (CNNs) and the collective reader response.

METHODS

We conducted an open, web-based, international, diagnostic reader study using an online platform. The online challenge opened on 10 May 2021 and closed on 19 July 2021 (71 days) and was advertised through several social media channels. The investigation included, 1456 dermoscopy images of melanomas (788 MIS; 474 melanomas ≤1.0 mm and 194 >1.0 mm). A test set comprising 277 MIS and 246 invasive melanomas was used to compare readers and CNNs.

RESULTS

We analysed 22 314 readings by 438 international readers. The overall accuracy (95% confidence interval) for melanoma thickness was 56.4% (55.7%-57.0%), 63.4% (62.5%-64.2%) for MIS and 71.0% (70.3%-72.1%) for invasive melanoma. Readers accurately predicted the thickness in 85.9% (85.4%-86.4%) of melanomas ≤1.0 mm (including MIS) and in 70.8% (69.2%-72.5%) of melanomas >1.0 mm. The reader collective outperformed a de novo CNN but not a pretrained CNN in differentiating MIS from invasive melanoma.

CONCLUSIONS

Using dermoscopy images, readers and CNNs predict melanoma thickness with fair to moderate accuracy. Readers most accurately discriminated between thin (≤1.0 mm including MIS) and thick melanomas (>1.0 mm).

Benefits, drawbacks, and challenges of social media use in dermatology: A systematic review

The presence of dermatological information on social media has grown exponentially over the last two decades. Consequently, the recent literature on this topic is abundant. Many authors have highlighted that social media constitutes a unique opportunity for patient education. Additionally, numerous other benefits of these platforms have been reported. However, other authors have focused on the potential risks that these networks involve. The main concerns are patient confidentiality, legal considerations and ethical issues. Therefore, we stand at a crossroads where the many advantages of social media use in dermatology seem to be underestimated due to the presence of potential drawbacks. At this point, we propose that a systematic review of the positive and negative aspects of using social media in dermatology is necessary. We carried out a comprehensive systematic review dating from inception to July 2021. Finally, 161 articles were included. Fifteen benefits, 11 drawbacks and 10 challenges of social media use in dermatology were identified and discussed. Suggested strategies to address the identified drawbacks were provided. Overall, while there are risks to using social media, they are outnumbered by their benefits. Therefore, dermatologists should embrace this opportunity to educate patients and aim to create rigorous and engaging content.

Flap failure prediction in microvascular tissue reconstruction using machine learning algorithms

BACKGROUND

Microvascular tissue reconstruction is a well-established, commonly used technique for a wide variety of the tissue defects. However, flap failure is associated with an additional hospital stay, medical cost burden, and mental stress. Therefore, understanding of the risk factors associated with this event is of utmost importance.

AIM

To develop machine learning-based predictive models for flap failure to identify the potential factors and screen out high-risk patients.

METHODS

Using the data set of 946 consecutive patients, who underwent microvascular tissue reconstruction of free flap reconstruction for head and neck, breast, back, and extremity, we established three machine learning models including random forest classifier, support vector machine, and gradient boosting. Model performances were evaluated by the indicators such as area under the curve of receiver operating characteristic curve, accuracy, precision, recall, and F1 score. A multivariable regression analysis was performed for the most critical variables in the random forest model.

RESULTS

Post-surgery, the flap failure event occurred in 34 patients (3.6%). The machine learning models based on various preoperative and intraoperative variables were successfully developed. Among them, the random forest classifier reached the best performance in receiver operating characteristic curve, with an area under the curve score of 0.770 in the test set. The top 10 variables in the random forest were age, body mass index, ischemia time, smoking, diabetes, experience, prior chemotherapy, hypertension, insulin, and obesity. Interestingly, only age, body mass index, and ischemic time were statistically associated with the outcomes.

CONCLUSION

Machine learning-based algorithms, especially the random forest classifier, were very important in categorizing patients at high risk of flap failure. The occurrence of flap failure was a multifactor-driven event and was identified with numerous factors that warrant further investigation. Importantly, the successful application of machine learning models may help the clinician in decision-making, understanding the underlying pathologic mechanisms of the disease, and improving the long-term outcome of patients.

Collective human intelligence outperforms artificial intelligence in a skin lesion classification task

BACKGROUND AND OBJECTIVES

Convolutional neural networks (CNN) enable accurate diagnosis of medical images and perform on or above the level of individual physicians. Recently, collective human intelligence (CoHI) was shown to exceed the diagnostic accuracy of individuals. Thus, diagnostic performance of CoHI (120 dermatologists) versus individual dermatologists versus two state-of-the-art CNN was investigated.

PATIENTS AND METHODS

Cross-sectional reader study with presentation of 30 clinical cases to 120 dermatologists. Six diagnoses were offered and votes collected via remote voting devices (quizzbox®, Quizzbox Solutions GmbH, Stuttgart, Germany). Dermatoscopic images were classified by a binary and multiclass CNN (FotoFinder Systems GmbH, Bad Birnbach, Germany). Three sets of diagnostic classifications were scored against ground truth: (1) CoHI, (2) individual dermatologists, and (3) CNN.

RESULTS

CoHI attained a significantly higher accuracy [95 % confidence interval] (80.0 % [62.7 %-90.5 %]) than individual dermatologists (75.7 % [73.8 %-77.5 %]) and CNN (70.0 % [52.1 %-83.3 %]; all P < 0.001) in binary classifications. Moreover, CoHI achieved a higher sensitivity (82.4 % [59.0 %-93.8 %]) and specificity (76.9 % [49.7 %-91.8 %]) than individual dermatologists (sensitivity 77.8 % [75.3 %-80.2 %], specificity 73.0 % [70.6 %-75.4 %]) and CNN (sensitivity 70.6 % [46.9 %-86.7 %], specificity 69.2 % [42.4 %-87.3 %]). The diagnostic accuracy of CoHI was superior to that of individual dermatologists (P < 0.001) in multiclass evaluation, with the accuracy of the latter comparable to multiclass CNN.

CONCLUSIONS

Our analysis revealed that the majority vote of an interconnected group of dermatologists (CoHI) outperformed individuals and CNN in a demanding skin lesion classification task.

Unity Is Intelligence: A Collective Intelligence Experiment on ECG Reading to Improve Diagnostic Performance in Cardiology

Medical errors have a huge impact on clinical practice in terms of economic and human costs. As a result, technology-based solutions, such as those grounded in artificial intelligence (AI) or collective intelligence (CI), have attracted increasing interest as a means of reducing error rates and their impacts. Previous studies have shown that a combination of individual opinions based on rules, weighting mechanisms, or other CI solutions could improve diagnostic accuracy with respect to individual doctors. We conducted a study to investigate the potential of this approach in cardiology and, more precisely, in electrocardiogram (ECG) reading. To achieve this aim, we designed and conducted an experiment involving medical students, recent graduates, and residents, who were asked to annotate a collection of 10 ECGs of various complexity and difficulty. For each ECG, we considered groups of increasing size (from three to 30 members) and applied three different CI protocols. In all cases, the results showed a statistically significant improvement (ranging from 9% to 88%) in terms of diagnostic accuracy when compared to the performance of individual readers; this difference held for not only large groups, but also smaller ones. In light of these results, we conclude that CI approaches can support the tasks mentioned above, and possibly other similar ones as well. We discuss the implications of applying CI solutions to clinical settings, such as cases of augmented ‘second opinions’ and decision-making.

Human-computer collaboration for skin cancer recognition

The rapid increase in telemedicine coupled with recent advances in diagnostic artificial intelligence (AI) create the imperative to consider the opportunities and risks of inserting AI-based support into new paradigms of care. Here we build on recent achievements in the accuracy of image-based AI for skin cancer diagnosis to address the effects of varied representations of AI-based support across different levels of clinical expertise and multiple clinical workflows. We find that good quality AI-based support of clinical decision-making improves diagnostic accuracy over that of either AI or physicians alone, and that the least experienced clinicians gain the most from AI-based support. We further find that AI-based multiclass probabilities outperformed content-based image retrieval (CBIR) representations of AI in the mobile technology environment, and AI-based support had utility in simulations of second opinions and of telemedicine triage. In addition to demonstrating the potential benefits associated with good quality AI in the hands of non-expert clinicians, we find that faulty AI can mislead the entire spectrum of clinicians, including experts. Lastly, we show that insights derived from AI class-activation maps can inform improvements in human diagnosis. Together, our approach and findings offer a framework for future studies across the spectrum of image-based diagnostics to improve human-computer collaboration in clinical practice.