Does your dermatology classifier know what it doesn't know? Detecting the long-tail of unseen conditions

Supervised deep learning models have proven to be highly effective in classification of dermatological conditions. These models rely on the availability of abundant labeled training examples. However, in the real-world, many dermatological conditions are individually too infrequent for per-condition classification with supervised learning. Although individually infrequent, these conditions may collectively be common and therefore are clinically significant in aggregate. To prevent models from generating erroneous outputs on such examples, there remains a considerable unmet need for deep learning systems that can better detect such infrequent conditions. These infrequent 'outlier' conditions are seen very rarely (or not at all) during training. In this paper, we frame this task as an out-of-distribution (OOD) detection problem. We set up a benchmark ensuring that outlier conditions are disjoint between the model training, validation, and test sets. Unlike traditional OOD detection benchmarks where the task is to detect dataset distribution shift, we aim at the more challenging task of detecting subtle differences resulting from a different pathology or condition. We propose a novel hierarchical outlier detection (HOD) loss, which assigns multiple abstention classes corresponding to each training outlier class and jointly performs a coarse classification of inliers vs. outliers, along with fine-grained classification of the individual classes. We demonstrate that the proposed HOD loss based approach outperforms leading methods that leverage outlier data during training. Further, performance is significantly boosted by using recent representation learning methods (BiT, SimCLR, MICLe). Further, we explore ensembling strategies for OOD detection and propose a diverse ensemble selection process for the best result. We also perform a subgroup analysis over conditions of varying risk levels and different skin types to investigate how OOD performance changes over each subgroup and demonstrate the gains of our framework in comparison to baseline. Furthermore, we go beyond traditional performance metrics and introduce a cost matrix for model trust analysis to approximate downstream clinical impact. We use this cost matrix to compare the proposed method against the baseline, thereby making a stronger case for its effectiveness in real-world scenarios.

Validation and Clinical Applicability of Whole-Volume Automated Segmentation of Optical Coherence Tomography in Retinal Disease Using Deep Learning

Question

Is deep learning–based segmentation of macular disease in optical coherence tomography (OCT) suitable for clinical use?

Findings

In this diagnostic study of OCT data from 173 patients with age-related macular degeneration or diabetic macular edema, model segmentations qualitatively ranked better or comparable for clinical applicability to 1 or more expert grader segmentations in 127 scans (73%) by a panel of 3 retinal specialists. Scans with high quantitative accuracy scores were not reliably associated with higher rankings.

Meaning

These findings suggest that qualitative evaluation adds to quantitative approaches when assessing clinical applicability of segmentation tools and clinician satisfaction in practice.

Importance

Quantitative volumetric measures of retinal disease in optical coherence tomography (OCT) scans are infeasible to perform owing to the time required for manual grading. Expert-level deep learning systems for automatic OCT segmentation have recently been developed. However, the potential clinical applicability of these systems is largely unknown.

Objective

To evaluate a deep learning model for whole-volume segmentation of 4 clinically important pathological features and assess clinical applicability.

Design, Setting, Participants

This diagnostic study used OCT data from 173 patients with a total of 15 558 B-scans, treated at Moorfields Eye Hospital. The data set included 2 common OCT devices and 2 macular conditions: wet age-related macular degeneration (107 scans) and diabetic macular edema (66 scans), covering the full range of severity, and from 3 points during treatment. Two expert graders performed pixel-level segmentations of intraretinal fluid, subretinal fluid, subretinal hyperreflective material, and pigment epithelial detachment, including all B-scans in each OCT volume, taking as long as 50 hours per scan. Quantitative evaluation of whole-volume model segmentations was performed. Qualitative evaluation of clinical applicability by 3 retinal experts was also conducted. Data were collected from June 1, 2012, to January 31, 2017, for set 1 and from January 1 to December 31, 2017, for set 2; graded between November 2018 and January 2020; and analyzed from February 2020 to November 2020.

Main Outcomes and Measures

Rating and stack ranking for clinical applicability by retinal specialists, model-grader agreement for voxelwise segmentations, and total volume evaluated using Dice similarity coefficients, Bland-Altman plots, and intraclass correlation coefficients.

Results

Among the 173 patients included in the analysis (92 [53%] women), qualitative assessment found that automated whole-volume segmentation ranked better than or comparable to at least 1 expert grader in 127 scans (73%; 95% CI, 66%-79%). A neutral or positive rating was given to 135 model segmentations (78%; 95% CI, 71%-84%) and 309 expert gradings (2 per scan) (89%; 95% CI, 86%-92%). The model was rated neutrally or positively in 86% to 92% of diabetic macular edema scans and 53% to 87% of age-related macular degeneration scans. Intraclass correlations ranged from 0.33 (95% CI, 0.08-0.96) to 0.96 (95% CI, 0.90-0.99). Dice similarity coefficients ranged from 0.43 (95% CI, 0.29-0.66) to 0.78 (95% CI, 0.57-0.85).

Conclusions and Relevance

This deep learning–based segmentation tool provided clinically useful measures of retinal disease that would otherwise be infeasible to obtain. Qualitative evaluation was additionally important to reveal clinical applicability for both care management and research.

Effects of hemolysis on plasma electrolyte and chemistry values in the common green iguana (Iguana iguana)

Hemolysis of serum and plasma samples is a common problem in veterinary diagnostic laboratories. We measured the effects of hemolysis on nine plasma analytes in 10 clinically normal common green iguanas (Iguana iguana). Blood samples with moderate and marked hemolysis were produced from each iguana by freezing, centrifuging, and decanting plasma from a portion of each blood sample, and combining the nonhemolyzed plasma with different amounts of hemolyzed plasma from the same individual. Moderate hemolysis significantly increased plasma phosphorus levels. Marked hemolysis significantly increased plasma values of potassium, phosphorus, total protein, and aspartate aminotransferase. The severity of hemolysis must be considered when interpreting values for these analytes in iguana plasma.

Hematology and chemistry reference values for free-ranging harbor seals (Phoca vitulina) and the effects of hemolysis on chemistry values of captive harbor seals

Most reported laboratory reference values for harbor seals (Phoca vitulina) are derived from captive seals, or stranded seals that have recovered from disease in marine mammal centers. This study established hematology and serum chemistry reference values for free-ranging harbor seals, using methods and that are current and readily available, and determined the effects of hemolysis on serum chemistry values of captive harbor seals. Blood samples were collected for hematologic and serum chemistry measurements from 14 clinically normal, adult male and female harbor seals and two juvenile harbor seals (approximate age 6 mo) captured in saltwater sloughs and estuaries near Moss Landing, California, USA. Values for amylase, globulin, and differential leukocyte count, not previously reported, were determined. In general, hematology and chemistry values in adults were similar to those reported for free-ranging and captive harbor seals, except for glucose, urea nitrogen, and lactate dehydrogenase (LDH) values, which were higher than those reported previously. Red blood cell counts in the two juveniles were higher than in adults and in young harbor seals studied previously. To determine the effects of hemolysis on serum chemistry values, two intensities of hemolysis were generated experimentally in blood collected from 11 harbor seals recovering from injuries or stranding at the Marine Mammal Center (Sausalito, California 94965, USA). Moderate hemolysis (++, 1 g/L hemoglobin, red-tinged) significantly increased LDH activity, whereas severe hemolysis ( , 2 g/L hemoglobin, cherry red) significantly increased total protein, albumin, calculated globulin, LDH, and total bilirubin and significantly decreased creatinine. The effects of hemolysis must be considered when chemistry results of harbor seals are interpreted.

Peripartal changes in serum alkaline phosphatase activity and lactate dehydrogenase activity in dairy cows

Peripartal serum alkaline phosphatase activity and lactate dehydrogenase activity were measured in 30 dairy cows in order to examine the association between retained fetal membranes and enzyme activity. Daily blood samples were obtained from pregnant cows, starting 15 days before the expected day of calving until eight days after parturition. Sera from 15 cows which retained fetal membranes longer than 24 hours and 15 cows which shed fetal membranes within six hours after parturition were analyzed for alkaline phosphatase and lactate dehydrogenase enzyme activities. Mean alkaline phosphatase enzyme activities ranged from 15.93 to 32.6 U/L in retained and nonretained placenta cows. There was a trend towards higher serum alkaline phosphatase activities in retained placenta cows but the differences were not significant among the groups (P greater than 0.05). Mean lactate dehydrogenase activities ranged from 307.2 to 438.86 U/L in nonretained and retained placenta cows. Lactate dehydrogenase enzyme activities in nonretained and retained placenta cows were similar (P greater than 0.05). The alkaline phosphatase and lactate dehydrogenase enzyme activities peaked at the time of parturition in both groups. However, the differences in alkaline phosphatase and lactate dehydrogenase activities on different days within non-retained and retained placenta cows were significant (P less than 0.05). Results indicate that prepartal changes in alkaline phosphatase and lactate dehydrogenase enzyme activities are not predictive of placental retention postpartum.

Babesiosis in the Greyhound

Babesiosis was diagnosed in five 11- to 18-day old Greyhound pups. In 3 pups, Babesia canis organisms were identified by examination of a Wright's-Giemsa-stained smear of blood. In 2 pups, the diagnosis was established by examination of a splenic impression smear obtained at necropsy. Findings in the 3 clinical cases included depression, weakness, anorexia, pallor, anemia, and thrombocytopenia. Subcutaneous administration of diminazene aceturate resulted in rapid clinical recovery in these cases. In the 2 pups that were necropsied, variable numbers of erythrocytes containing Babesia organisms were observed in the microvasculature of tissues. Subinoculation of blood into an intact dog and a splenectomized dog resulted in parasitemia and B canis serum titers, as determined by indirect fluorescent antibody testing. A site visit to the kennel from which the pups had originated led to identification of numerous Rhipicephalus sanguineus in small buildings and pens. Of 107 dogs from this kennel that were tested, 63 had an indirect fluorescent antibody titer for B canis. A limited serologic survey of Greyhound kennels in West Virginia, Oklahoma, Texas, Mississippi, and Florida identified a large number of dogs with indirect fluorescent antibody titers for B canis.

Haemobartonellosis in a dog in Saskatchewan

Images