Crowdsourcing for translational research: analysis of biomarker expression using cancer microarrays

Task design for crowdsourced glioma cell annotation in microscopy images

Crowdsourcing has been used in computational pathology to generate cell and cell nuclei annotations for machine learning. Herein, we broaden its scope to the previously unsolved challenging task of glioma cell detection. This requires multiplexed immunofluorescence microscopy due to diffuse invasiveness and exceptional similarity between glioma cells and reactive astrocytes. In four pilot experiments, we iteratively developed a task design enabling high-quality annotations by crowdworkers on Amazon Mechanical Turk. We applied majority or weighted vote and validated them against ground truth in the final setting. On the base of a YOLO convolutional neural network architecture, we used these consensus labels for training with different image representations regarding colors, intensities, and immmunohistochemical marker combinations. A crowd of 712 workers defined aggregated point annotations in 235 images with an average [Formula: see text] score of 0.627 for majority vote. The networks resulted in acceptable [Formula: see text] scores up to 0.69 for YOLOv8 on average and indicated first evidence for transferability to images lacking tumor markers, especially in IDH-wildtype glioblastoma. Our work confirms feasibility of crowdsourcing to generate labels suitable for training of machine learning tools in the challenging and clinically relevant use case of glioma microenvironment.

Toward a generalizable machine learning workflow for neurodegenerative disease staging with focus on neurofibrillary tangles

Machine learning (ML) has increasingly been used to assist and expand current practices in neuropathology. However, generating large imaging datasets with quality labels is challenging in fields which demand high levels of expertise. Further complicating matters is the often seen disagreement between experts in neuropathology-related tasks, both at the case level and at a more granular level. Neurofibrillary tangles (NFTs) are a hallmark pathological feature of Alzheimer disease, and are associated with disease progression which warrants further investigation and granular quantification at a scale not currently accessible in routine human assessment. In this work, we first provide a baseline of annotator/rater agreement for the tasks of Braak NFT staging between experts and NFT detection using both experts and novices in neuropathology. We use a whole-slide-image (WSI) cohort of neuropathology cases from Emory University Hospital immunohistochemically stained for Tau. We develop a workflow for gathering annotations of the early stage formation of NFTs (Pre-NFTs) and mature intracellular (iNFTs) and show ML models can be trained to learn annotator nuances for the task of NFT detection in WSIs. We utilize a model-assisted-labeling approach and demonstrate ML models can be used to aid in labeling large datasets efficiently. We also show these models can be used to extract case-level features, which predict Braak NFT stages comparable to expert human raters, and do so at scale. This study provides a generalizable workflow for various pathology and related fields, and also provides a technique for accomplishing a high-level neuropathology task with limited human annotations.

Annotation protocol and crowdsourcing multiple instance learning classification of skin histological images: The CR-AI4SkIN dataset

Digital Pathology (DP) has experienced a significant growth in recent years and has become an essential tool for diagnosing and prognosis of tumors. The availability of Whole Slide Images (WSIs) and the implementation of Deep Learning (DL) algorithms have paved the way for the appearance of Artificial Intelligence (AI) systems that support the diagnosis process. These systems require extensive and varied data for their training to be successful. However, creating labeled datasets in histopathology is laborious and time-consuming. We have developed a crowdsourcing-multiple instance labeling/learning protocol that is applied to the creation and use of the CR-AI4SkIN dataset.2 CR-AI4SkIN contains 271 WSIs of 7 Cutaneous Spindle Cell (CSC) neoplasms with expert and non-expert labels at region and WSI levels. It is the first dataset of these types of neoplasms made available. The regions selected by the experts are used to learn an automatic extractor of Regions of Interest (ROIs) from WSIs. To produce the embedding of each WSI, the representations of patches within the ROIs are obtained using a contrastive learning method, and then combined. Finally, they are fed to a Gaussian process-based crowdsourcing classifier, which utilizes the noisy non-expert WSI labels. We validate our crowdsourcing-multiple instance learning method in the CR-AI4SkIN dataset, addressing a binary classification problem (malign vs. benign). The proposed method obtains an F1 score of 0.7911 on the test set, outperforming three widely used aggregation methods for crowdsourcing tasks. Furthermore, our crowdsourcing method also outperforms the supervised model with expert labels on the test set (F1-score = 0.6035). The promising results support the proposed crowdsourcing multiple instance learning annotation protocol. It also validates the automatic extraction of interest regions and the use of contrastive embedding and Gaussian process classification to perform crowdsourcing classification tasks.

Research data management in health and biomedical citizen science: practices and prospects

Background

Public engagement in health and biomedical research is being influenced by the paradigm of citizen science. However, conventional health and biomedical research relies on sophisticated research data management tools and methods. Considering these, what contribution can citizen science make in this field of research? How can it follow research protocols and produce reliable results?

Objective

The aim of this article is to analyze research data management practices in existing biomedical citizen science studies, so as to provide insights for members of the public and of the research community considering this approach to research.

Methods

A scoping review was conducted on this topic to determine data management characteristics of health and bio medical citizen science research. From this review and related web searching, we chose five online platforms and a specific research project associated with each, to understand their research data management approaches and enablers.

Results

Health and biomedical citizen science platforms and projects are diverse in terms of types of work with data and data management activities that in themselves may have scientific merit. However, consistent approaches in the use of research data management models or practices seem lacking, or at least are not prevalent in the review.

Conclusions

There is potential for important data collection and analysis activities to be opaque or irreproducible in health and biomedical citizen science initiatives without the implementation of a research data management model that is transparent and accessible to team members and to external audiences. This situation might be improved with participatory development of standards that can be applied to diverse projects and platforms, across the research data life cycle.

Breast cancer in patients with germline TP53 pathogenic variants have typical tumour characteristics: the Cohort study of TP53 carrier early onset breast cancer (COPE study)

Germline TP53 pathogenic variants are rare but associated with a high risk of cancer; they are often identified in the context of clinically diagnosed Li-Fraumeni syndrome predisposing to a range of young onset cancers including sarcomas and breast cancer. The study aim was to conduct a detailed morphological review and immuno-phenotyping of breast cancer arising in carriers of a germline TP53 pathogenic variant. We compared breast cancers from five defined groups: (1) TP53 carriers with breast cancer (n = 59), (2) early onset HER2-amplified breast cancer, no germline pathogenic variant in BRCA1/2 or TP53 (n = 55), (3) BRCA1 pathogenic variant carriers (n = 60); (4) BRCA2 pathogenic variant carriers (n = 61) and (5) young onset breast cancer with no known germline pathogenic variant (n = 98). Pathologists assessed a pre-agreed set of morphological characteristics using light microscopy. Immunohistochemistry (IHC) for HER2, ER, PR, p53, integrin alpha v beta 6 (αvβ6) integrin, α-smooth muscle actin (α-SMA) and pSMAD2/3 was performed on tissue microarrays of invasive carcinoma. We confirmed a previously reported high prevalence of HER2-amplified, ductal no special type invasive breast carcinoma amongst known TP53 germline pathogenic variant carriers 20 of 36 (56%). Furthermore we observed a high frequency of densely sclerotic tumour stroma in cancers from TP53 carriers (29/36, 80.6%) when compared with non-carriers, 50.9% (28/55), 34.7% (50/144), 41.4% (65/157), 43.8% (95/217) in groups 2-5 respectively. The majority of germline TP53 gene carrier breast tumours had a high intensity of integrin αvβ6, α-SMA and pSMAD2/3 expression in the majority of cancer cells. In conclusion, aggressive HER2 positive breast cancers with densely sclerotic stroma are common in germline TP53 carriers. High levels of αvβ6 integrin, α-SMA and pSMAD2/3 expression suggest that the dense stromal phenotype may be driven by upregulated transforming growth factor beta signalling.

Scientific Discovery Games for Biomedical Research

Over the past decade, scientific discovery games (SDGs) have emerged as a viable approach for biomedical research, engaging hundreds of thousands of volunteer players and resulting in numerous scientific publications. After describing the origins of this novel research approach, we review the scientific output of SDGs across molecular modeling, sequence alignment, neuroscience, pathology, cellular biology, genomics, and human cognition. We find compelling results and technical innovations arising in problem-oriented games such as Foldit and Eterna and in data-oriented games such as EyeWire and Project Discovery. We discuss emergent properties of player communities shared across different projects, including the diversity of communities and the extraordinary contributions of some volunteers, such as paper writing. Finally, we highlight connections to artificial intelligence, biological cloud laboratories, new game genres, science education, and open science that may drive the next generation of SDGs.

Crowdsourcing of Histological Image Labeling and Object Delineation by Medical Students

Crowdsourcing in pathology has been performed on tasks that are assumed to be manageable by nonexperts. Demand remains high for annotations of more complex elements in digital microscopic images, such as anatomical structures. Therefore, this paper investigates conditions to enable crowdsourced annotations of high-level image objects, a complex task considered to require expert knowledge. Seventy six medical students without specific domain knowledge who voluntarily participated in three experiments solved two relevant annotation tasks on histopathological images: 1) labeling of images showing tissue regions and 2) delineation of morphologically defined image objects. We focus on methods to ensure sufficient annotation quality including several tests on the required number of participants and on the correlation of participants' performance between tasks. In a set up simulating annotation of images with limited ground truth, we validated the feasibility of a confidence score using full ground truth. For this, we computed a majority vote using weighting factors based on individual assessment of contributors against scattered gold standard annotated by pathologists. In conclusion, we provide guidance for task design and quality control to enable a crowdsourced approach to obtain accurate annotations required in the era of digital pathology.

Harnessing citizen science through mobile phone technology to screen for immunohistochemical biomarkers in bladder cancer

BACKGROUND

Immunohistochemistry (IHC) is often used in personalisation of cancer treatments. Analysis of large data sets to uncover predictive biomarkers by specialists can be enormously time-consuming. Here we investigated crowdsourcing as a means of reliably analysing immunostained cancer samples to discover biomarkers predictive of cancer survival.

METHODS

We crowdsourced the analysis of bladder cancer TMA core samples through the smartphone app 'Reverse the Odds'. Scores from members of the public were pooled and compared to a gold standard set scored by appropriate specialists. We also used crowdsourced scores to assess associations with disease-specific survival.

RESULTS

Data were collected over 721 days, with 4,744,339 classifications performed. The average time per classification was approximately 15 s, with approximately 20,000 h total non-gaming time contributed. The correlation between crowdsourced and expert H-scores (staining intensity × proportion) varied from 0.65 to 0.92 across the markers tested, with six of 10 correlation coefficients at least 0.80. At least two markers (MRE11 and CK20) were significantly associated with survival in patients with bladder cancer, and a further three markers showed results warranting expert follow-up.

CONCLUSIONS

Crowdsourcing through a smartphone app has the potential to accurately screen IHC data and greatly increase the speed of biomarker discovery.

A Review on the Applications of Crowdsourcing in Human Pathology

The advent of the digital pathology has introduced new avenues of diagnostic medicine. Among them, crowdsourcing has attracted researchers' attention in the recent years, allowing them to engage thousands of untrained individuals in research and diagnosis. While there exist several articles in this regard, prior works have not collectively documented them. We, therefore, aim to review the applications of crowdsourcing in human pathology in a semi-systematic manner. We first, introduce a novel method to do a systematic search of the literature. Utilizing this method, we, then, collect hundreds of articles and screen them against a predefined set of criteria. Furthermore, we crowdsource part of the screening process, to examine another potential application of crowdsourcing. Finally, we review the selected articles and characterize the prior uses of crowdsourcing in pathology.