Crowdsourcing to delineate skin affected by chronic graft-vs-host disease

Crowdsourcing Skin Demarcations of Chronic Graft-Versus-Host Disease in Patient Photographs: Training Versus Performance Study

BACKGROUND

Chronic graft-versus-host disease (cGVHD) is a significant cause of long-term morbidity and mortality in patients after allogeneic hematopoietic cell transplantation. Skin is the most commonly affected organ, and visual assessment of cGVHD can have low reliability. Crowdsourcing data from nonexpert participants has been used for numerous medical applications, including image labeling and segmentation tasks.

OBJECTIVE

This study aimed to assess the ability of crowds of nonexpert raters-individuals without any prior training for identifying or marking cGHVD-to demarcate photos of cGVHD-affected skin. We also studied the effect of training and feedback on crowd performance.

METHODS

Using a Canfield Vectra H1 3D camera, 360 photographs of the skin of 36 patients with cGVHD were taken. Ground truth demarcations were provided in 3D by a trained expert and reviewed by a board-certified dermatologist. In total, 3000 2D images (projections from various angles) were created for crowd demarcation through the DiagnosUs mobile app. Raters were split into high and low feedback groups. The performances of 4 different crowds of nonexperts were analyzed, including 17 raters per image for the low and high feedback groups, 32-35 raters per image for the low feedback group, and the top 5 performers for each image from the low feedback group.

RESULTS

Across 8 demarcation competitions, 130 raters were recruited to the high feedback group and 161 to the low feedback group. This resulted in a total of 54,887 individual demarcations from the high feedback group and 78,967 from the low feedback group. The nonexpert crowds achieved good overall performance for segmenting cGVHD-affected skin with minimal training, achieving a median surface area error of less than 12% of skin pixels for all crowds in both the high and low feedback groups. The low feedback crowds performed slightly poorer than the high feedback crowd, even when a larger crowd was used. Tracking the 5 most reliable raters from the low feedback group for each image recovered a performance similar to that of the high feedback crowd. Higher variability between raters for a given image was not found to correlate with lower performance of the crowd consensus demarcation and cannot therefore be used as a measure of reliability. No significant learning was observed during the task as more photos and feedback were seen.

CONCLUSIONS

Crowds of nonexpert raters can demarcate cGVHD images with good overall performance. Tracking the top 5 most reliable raters provided optimal results, obtaining the best performance with the lowest number of expert demarcations required for adequate training. However, the agreement amongst individual nonexperts does not help predict whether the crowd has provided an accurate result. Future work should explore the performance of crowdsourcing in standard clinical photos and further methods to estimate the reliability of consensus demarcations.

Updates in SJS/TEN: collaboration, innovation, and community

Stevens-Johnson Syndrome/Toxic Epidermal Necrolysis (SJS/TEN) is a predominantly drug-induced disease, with a mortality rate of 15-20%, that engages the expertise of multiple disciplines: dermatology, allergy, immunology, clinical pharmacology, burn surgery, ophthalmology, urogynecology, and psychiatry. SJS/TEN has an incidence of 1-5/million persons per year in the United States, with even higher rates globally. One of the challenges of SJS/TEN has been developing the research infrastructure and coordination to answer questions capable of transforming clinical care and leading to improved patient outcomes. SJS/TEN 2021, the third research meeting of its kind, was held as a virtual meeting on August 28-29, 2021. The meeting brought together 428 international scientists, in addition to a community of 140 SJS/TEN survivors and family members. The goal of the meeting was to brainstorm strategies to support the continued growth of an international SJS/TEN research network, bridging science and the community. The community workshop section of the meeting focused on eight primary themes: mental health, eye care, SJS/TEN in children, non-drug induced SJS/TEN, long-term health complications, new advances in mechanisms and basic science, managing long-term scarring, considerations for skin of color, and COVID-19 vaccines. The meeting featured several important updates and identified areas of unmet research and clinical need that will be highlighted in this white paper.

Hyperspectral imaging to accurately segment skin erythema and hyperpigmentation in cutaneous chronic graft-versus-host disease

In 51 lesions from 15 patients with the inflammatory skin condition chronic graft-versus-host-disease, hyperspectral imaging accurately delineated active erythema and post-inflammatory hyperpigmentation. The method was validated by dermatologist-approved confident delineations of only definitely affected and definitely unaffected areas in photographs. A prototype hyperspectral imaging system acquired a 2.5 × 3.5 cm2 area of skin at 120 wavelengths in the 450-850 nm range. Unsupervised extraction of unknown absorbers by endmember analysis achieved a comparable accuracy to that of supervised extraction of known absorbers (melanin, hemoglobin) by chromophore mapping: 0.78 (IQR: 0.39-0.85) vs. 0.83 (0.53-0.91) to delineate erythema and 0.74 (0.57-0.87) vs. 0.73 (0.52-0.84) to delineate hyperpigmentation. Both algorithms achieved higher specificity than sensitivity. Whereas a trained human confidently marked a median of 7% of image pixels, unsupervised and supervised algorithms delineated a median of 14% and 27% pixels. Hyperspectral imaging could overcome a fundamental practice gap of distinguishing active from inactive manifestations of inflammatory skin disease.

Agreement Between Experts and an Untrained Crowd for Identifying Dermoscopic Features Using a Gamified App: Reader Feasibility Study

BACKGROUND

Dermoscopy is commonly used for the evaluation of pigmented lesions, but agreement between experts for identification of dermoscopic structures is known to be relatively poor. Expert labeling of medical data is a bottleneck in the development of machine learning (ML) tools, and crowdsourcing has been demonstrated as a cost- and time-efficient method for the annotation of medical images.

OBJECTIVE

The aim of this study is to demonstrate that crowdsourcing can be used to label basic dermoscopic structures from images of pigmented lesions with similar reliability to a group of experts.

METHODS

First, we obtained labels of 248 images of melanocytic lesions with 31 dermoscopic "subfeatures" labeled by 20 dermoscopy experts. These were then collapsed into 6 dermoscopic "superfeatures" based on structural similarity, due to low interrater reliability (IRR): dots, globules, lines, network structures, regression structures, and vessels. These images were then used as the gold standard for the crowd study. The commercial platform DiagnosUs was used to obtain annotations from a nonexpert crowd for the presence or absence of the 6 superfeatures in each of the 248 images. We replicated this methodology with a group of 7 dermatologists to allow direct comparison with the nonexpert crowd. The Cohen κ value was used to measure agreement across raters.

RESULTS

In total, we obtained 139,731 ratings of the 6 dermoscopic superfeatures from the crowd. There was relatively lower agreement for the identification of dots and globules (the median κ values were 0.526 and 0.395, respectively), whereas network structures and vessels showed the highest agreement (the median κ values were 0.581 and 0.798, respectively). This pattern was also seen among the expert raters, who had median κ values of 0.483 and 0.517 for dots and globules, respectively, and 0.758 and 0.790 for network structures and vessels. The median κ values between nonexperts and thresholded average-expert readers were 0.709 for dots, 0.719 for globules, 0.714 for lines, 0.838 for network structures, 0.818 for regression structures, and 0.728 for vessels.

CONCLUSIONS

This study confirmed that IRR for different dermoscopic features varied among a group of experts; a similar pattern was observed in a nonexpert crowd. There was good or excellent agreement for each of the 6 superfeatures between the crowd and the experts, highlighting the similar reliability of the crowd for labeling dermoscopic images. This confirms the feasibility and dependability of using crowdsourcing as a scalable solution to annotate large sets of dermoscopic images, with several potential clinical and educational applications, including the development of novel, explainable ML tools.

Methods to Assess Disease Activity and Severity in Cutaneous Chronic Graft-versus-Host Disease: A Critical Literature Review

Chronic graft-versus-host disease (cGVHD), a potentially debilitating complication of hematopoietic cell transplantation, confers increased risk for mortality. Whereas treatment decisions rely on an accurate assessment of disease activity/severity, validated methods of assessing cutaneous cGVHD activity/severity appear to be limited. In this study, we aimed to identify and evaluate current data on the assessment of disease activity/severity in cutaneous cGVHD. Using modified PRISMA methods, we performed a critical literature review for relevant articles. Our literature search identified 1741 articles, of which 1635 were excluded as duplicates or failure to meet inclusion criteria. Of the included studies (n = 106), 39 (37%) addressed clinical and/or histopathologic parameters, 53 (50%) addressed serologic parameters, 8 (7.5%) addressed imaging parameters, and 6 (5.5%) addressed computer-based technologies. The only formally validated metric of disease activity/severity assessment in cutaneous cGVHD is the National Institutes of Health consensus scoring system, which is founded on clinical assessment alone. The lack of an objective marker for cGVHD necessitates further studies. An evaluation of the potential contributions of serologic, imaging, and/or computer-based technologies is warranted.