Interpretable multimodal deep learning for real-time pan-tissue pan-disease pathology search on social media

Identification of histopathological classification and establishment of prognostic indicators of gastric adenocarcinoma based on deep learning algorithm

The aim of this study is to establish a deep learning (DL) model to predict the pathological type of gastric adenocarcinoma cancer based on whole-slide images(WSIs). We downloaded 356 histopathological images of gastric adenocarcinoma (STAD) patients from The Cancer Genome Atlas database and randomly divided them into the training set, validation set and test set (8:1:1). Additionally, 80 H&E-stained WSIs of STAD were collected for external validation. The CLAM tool was used to cut the WSIs and further construct the model by DL algorithm, achieving an accuracy of over 90% in identifying and predicting histopathological subtypes. External validation results demonstrated the model had a certain generalization ability. Moreover, DL features were extracted from the model to further investigate the differences in immune infiltration and patient prognosis between the two subtypes. The DL model can accurately predict the pathological classification of STAD patients, and provide certain reference value for clinical diagnosis. The nomogram combining DL-signature, gene-signature and clinical features can be used as a prognostic classifier for clinical decision-making and treatment.

A multimodal generative AI copilot for human pathology

Computational pathology1,2 has witnessed considerable progress in the development of both task-specific predictive models and task-agnostic self-supervised vision encoders3,4. However, despite the explosive growth of generative artificial intelligence (AI), there have been few studies on building general-purpose multimodal AI assistants and copilots5 tailored to pathology. Here we present PathChat, a vision-language generalist AI assistant for human pathology. We built PathChat by adapting a foundational vision encoder for pathology, combining it with a pretrained large language model and fine-tuning the whole system on over 456,000 diverse visual-language instructions consisting of 999,202 question and answer turns. We compare PathChat with several multimodal vision-language AI assistants and GPT-4V, which powers the commercially available multimodal general-purpose AI assistant ChatGPT-4 (ref. 6). PathChat achieved state-of-the-art performance on multiple-choice diagnostic questions from cases with diverse tissue origins and disease models. Furthermore, using open-ended questions and human expert evaluation, we found that overall PathChat produced more accurate and pathologist-preferable responses to diverse queries related to pathology. As an interactive vision-language AI copilot that can flexibly handle both visual and natural language inputs, PathChat may potentially find impactful applications in pathology education, research and human-in-the-loop clinical decision-making.

A visual-language foundation model for computational pathology

The accelerated adoption of digital pathology and advances in deep learning have enabled the development of robust models for various pathology tasks across a diverse array of diseases and patient cohorts. However, model training is often difficult due to label scarcity in the medical domain, and a model's usage is limited by the specific task and disease for which it is trained. Additionally, most models in histopathology leverage only image data, a stark contrast to how humans teach each other and reason about histopathologic entities. We introduce CONtrastive learning from Captions for Histopathology (CONCH), a visual-language foundation model developed using diverse sources of histopathology images, biomedical text and, notably, over 1.17 million image-caption pairs through task-agnostic pretraining. Evaluated on a suite of 14 diverse benchmarks, CONCH can be transferred to a wide range of downstream tasks involving histopathology images and/or text, achieving state-of-the-art performance on histology image classification, segmentation, captioning, and text-to-image and image-to-text retrieval. CONCH represents a substantial leap over concurrent visual-language pretrained systems for histopathology, with the potential to directly facilitate a wide array of machine learning-based workflows requiring minimal or no further supervised fine-tuning.

Artifact Detection and Restoration in Histology Images With Stain-Style and Structural Preservation

The artifacts in histology images may encumber the accurate interpretation of medical information and cause misdiagnosis. Accordingly, prepending manual quality control of artifacts considerably decreases the degree of automation. To close this gap, we propose a methodical pre-processing framework to detect and restore artifacts, which minimizes their impact on downstream AI diagnostic tasks. First, the artifact recognition network AR-Classifier first differentiates common artifacts from normal tissues, e.g., tissue folds, marking dye, tattoo pigment, spot, and out-of-focus, and also catalogs artifact patches by their restorability. Then, the succeeding artifact restoration network AR-CycleGAN performs de-artifact processing where stain styles and tissue structures can be maximally retained. We construct a benchmark for performance evaluation, curated from both clinically collected WSIs and public datasets of colorectal and breast cancer. The functional structures are compared with state-of-the-art methods, and also comprehensively evaluated by multiple metrics across multiple tasks, including artifact classification, artifact restoration, downstream diagnostic tasks of tumor classification and nuclei segmentation. The proposed system allows full automation of deep learning based histology image analysis without human intervention. Moreover, the structure-independent characteristic enables its processing with various artifact subtypes. The source code and data in this research are available at https://github.com/yunboer/AR-classifier-and-AR-CycleGAN.

A visual-language foundation model for pathology image analysis using medical Twitter

The lack of annotated publicly available medical images is a major barrier for computational research and education innovations. At the same time, many de-identified images and much knowledge are shared by clinicians on public forums such as medical Twitter. Here we harness these crowd platforms to curate OpenPath, a large dataset of 208,414 pathology images paired with natural language descriptions. We demonstrate the value of this resource by developing pathology language-image pretraining (PLIP), a multimodal artificial intelligence with both image and text understanding, which is trained on OpenPath. PLIP achieves state-of-the-art performances for classifying new pathology images across four external datasets: for zero-shot classification, PLIP achieves F1 scores of 0.565-0.832 compared to F1 scores of 0.030-0.481 for previous contrastive language-image pretrained model. Training a simple supervised classifier on top of PLIP embeddings also achieves 2.5% improvement in F1 scores compared to using other supervised model embeddings. Moreover, PLIP enables users to retrieve similar cases by either image or natural language search, greatly facilitating knowledge sharing. Our approach demonstrates that publicly shared medical information is a tremendous resource that can be harnessed to develop medical artificial intelligence for enhancing diagnosis, knowledge sharing and education.

Biased data, biased AI: deep networks predict the acquisition site of TCGA images

BACKGROUND

Deep learning models applied to healthcare applications including digital pathology have been increasing their scope and importance in recent years. Many of these models have been trained on The Cancer Genome Atlas (TCGA) atlas of digital images, or use it as a validation source. One crucial factor that seems to have been widely ignored is the internal bias that originates from the institutions that contributed WSIs to the TCGA dataset, and its effects on models trained on this dataset.

METHODS

8,579 paraffin-embedded, hematoxylin and eosin stained, digital slides were selected from the TCGA dataset. More than 140 medical institutions (acquisition sites) contributed to this dataset. Two deep neural networks (DenseNet121 and KimiaNet were used to extract deep features at 20× magnification. DenseNet was pre-trained on non-medical objects. KimiaNet has the same structure but trained for cancer type classification on TCGA images. The extracted deep features were later used to detect each slide's acquisition site, and also for slide representation in image search.

RESULTS

DenseNet's deep features could distinguish acquisition sites with 70% accuracy whereas KimiaNet's deep features could reveal acquisition sites with more than 86% accuracy. These findings suggest that there are acquisition site specific patterns that could be picked up by deep neural networks. It has also been shown that these medically irrelevant patterns can interfere with other applications of deep learning in digital pathology, namely image search. This study shows that there are acquisition site specific patterns that can be used to identify tissue acquisition sites without any explicit training. Furthermore, it was observed that a model trained for cancer subtype classification has exploited such medically irrelevant patterns to classify cancer types. Digital scanner configuration and noise, tissue stain variation and artifacts, and source site patient demographics are among factors that likely account for the observed bias. Therefore, researchers should be cautious of such bias when using histopathology datasets for developing and training deep networks.

Multimodal data fusion for cancer biomarker discovery with deep learning

Technological advances now make it possible to study a patient from multiple angles with high-dimensional, high-throughput multi-scale biomedical data. In oncology, massive amounts of data are being generated ranging from molecular, histopathology, radiology to clinical records. The introduction of deep learning has significantly advanced the analysis of biomedical data. However, most approaches focus on single data modalities leading to slow progress in methods to integrate complementary data types. Development of effective multimodal fusion approaches is becoming increasingly important as a single modality might not be consistent and sufficient to capture the heterogeneity of complex diseases to tailor medical care and improve personalised medicine. Many initiatives now focus on integrating these disparate modalities to unravel the biological processes involved in multifactorial diseases such as cancer. However, many obstacles remain, including lack of usable data as well as methods for clinical validation and interpretation. Here, we cover these current challenges and reflect on opportunities through deep learning to tackle data sparsity and scarcity, multimodal interpretability, and standardisation of datasets.

Artificial intelligence in Dermatopathology

INTRODUCTION

Ever evolving research in medical field has reached an exciting stage with advent of newer technologies. With the introduction of digital microscopy, pathology has transitioned to become more digitally oriented speciality. The potential of artificial intelligence (AI) in dermatopathology is to aid the diagnosis, and it requires dermatopathologists' guidance for efficient functioning of artificial intelligence.

METHOD

Comprehensive literature search was performed using electronic online databases "PubMed" and "Google Scholar." Articles published in English language were considered for the review.

RESULTS

Convolutional neural network, a type of deep neural network, is considered as an ideal tool in image recognition, processing, classification, and segmentation. Implementation of AI in tumor pathology is involved in the diagnosis, grading, staging, and prognostic prediction as well as in identification of genetic or pathological features. In this review, we attempt to discuss the use of AI in dermatopathology, the attitude of patients and clinicians, its challenges, limitation, and potential opportunities in future implementation.

Artificial intelligence: Its role in dermatopathology

Artificial intelligence (AI), a major frontier in the field of medical research, can potentially lead to a paradigm shift in clinical practice. A type of artificial intelligence system known as convolutional neural network points to the possible utility of deep learning in dermatopathology. Though pathology has been traditionally restricted to microscopes and glass slides, recent advancement in digital pathological imaging has led to a transition making it a potential branch for the implementation of artificial intelligence. The current application of artificial intelligence in dermatopathology is to complement the diagnosis and requires a well-trained dermatopathologist's guidance for better designing and development of deep learning algorithms. Here we review the recent advances of artificial intelligence in dermatopathology, its applications in disease diagnosis and in research, along with its limitations and future potential.

Computational Analysis of Pathological Image Enables Interpretable Prediction for Microsatellite Instability

Background

Microsatellite instability (MSI) is associated with several tumor types and has become increasingly vital in guiding patient treatment decisions; however, reasonably distinguishing MSI from its counterpart is challenging in clinical practice.

Methods

In this study, interpretable pathological image analysis strategies are established to help medical experts to identify MSI. The strategies only require ubiquitous hematoxylin and eosin–stained whole-slide images and perform well in the three cohorts collected from The Cancer Genome Atlas. Equipped with machine learning and image processing technique, intelligent models are established to diagnose MSI based on pathological images, providing the rationale of the decision in both image level and pathological feature level.

Findings

The strategies achieve two levels of interpretability. First, the image-level interpretability is achieved by generating localization heat maps of important regions based on deep learning. Second, the feature-level interpretability is attained through feature importance and pathological feature interaction analysis. Interestingly, from both the image-level and feature-level interpretability, color and texture characteristics, as well as their interaction, are shown to be mostly contributed to the MSI prediction.

Interpretation

The developed transparent machine learning pipeline is able to detect MSI efficiently and provide comprehensive clinical insights to pathologists. The comprehensible heat maps and features in the intelligent pipeline reflect extra- and intra-cellular acid–base balance shift in MSI tumor.

The Influence of Social Media in Promoting Knowledge Acquisition and Pathology Excellence in Nigeria

The use of social media has evolved from platforms designed primarily for social connection and news sharing to include vibrant virtual academic environments. These platforms allow pathologists from across the globe to interact, exchange knowledge, and collaborate. Pathology in Nigeria, as in much of Africa, faces severe knowledge and practice gaps, with a lack of supporting modern laboratory infrastructure. Social media represents a potentially highly valuable avenue to help address some of these deficiencies. In this Perspective piece, we highlight our experience with the increasing role of social media in providing quality medical education in pathology globally, with an emphasis on how it bridges many of these gaps in Nigeria. Social media sites serve as sources of readily accessible, free, high-quality information to pathologists and trainees through academic discussions, quizzes, journal clubs, and informal consultations. They also provide opportunities for professional networking and research collaborations. Despite the availability and wide reach of these platforms, social media as a tool for advancement of knowledge in pathology is still undersubscribed in this part of the world. Improving awareness of and support for these tools will ideally help mitigate some of the challenges of practicing pathology in low and middle-income settings.

Communicator-Driven Data Preprocessing Improves Deep Transfer Learning of Histopathological Prediction of Pancreatic Ductal Adenocarcinoma

Simple Summary

Pancreatic cancer has a dismal prognosis and its diagnosis can be challenging. Histopathological slides can be digitalized and their analysis can then be supported by computer algorithms. For this purpose, computer algorithms (neural networks) need to be trained to detect the desired tissue type (e.g., pancreatic cancer). However, raw training data often contain many different tissue types. Here we show a preprocessing step using two communicators that sort unfitting tissue tiles into a new dataset class. Using the improved dataset neural networks distinguished pancreatic cancer from other tissue types on digitalized histopathological slides including lymph node metastases.

Abstract

Pancreatic cancer is a fatal malignancy with poor prognosis and limited treatment options. Early detection in primary and secondary locations is critical, but fraught with challenges. While digital pathology can assist with the classification of histopathological images, the training of such networks always relies on a ground truth, which is frequently compromised as tissue sections contain several types of tissue entities. Here we show that pancreatic cancer can be detected on hematoxylin and eosin (H&E) sections by convolutional neural networks using deep transfer learning. To improve the ground truth, we describe a preprocessing data clean-up process using two communicators that were generated through existing and new datasets. Specifically, the communicators moved image tiles containing adipose tissue and background to a new data class. Hence, the original dataset exhibited improved labeling and, consequently, a higher ground truth accuracy. Deep transfer learning of a ResNet18 network resulted in a five-class accuracy of about 94% on test data images. The network was validated with independent tissue sections composed of healthy pancreatic tissue, pancreatic ductal adenocarcinoma, and pancreatic cancer lymph node metastases. The screening of different models and hyperparameter fine tuning were performed to optimize the performance with the independent tissue sections. Taken together, we introduce a step of data preprocessing via communicators as a means of improving the ground truth during deep transfer learning and hyperparameter tuning to identify pancreatic ductal adenocarcinoma primary tumors and metastases in histological tissue sections.

Fast and scalable search of whole-slide images via self-supervised deep learning

The adoption of digital pathology has enabled the curation of large repositories of gigapixel whole-slide images (WSIs). Computationally identifying WSIs with similar morphologic features within large repositories without requiring supervised training can have significant applications. However, the retrieval speeds of algorithms for searching similar WSIs often scale with the repository size, which limits their clinical and research potential. Here we show that self-supervised deep learning can be leveraged to search for and retrieve WSIs at speeds that are independent of repository size. The algorithm, which we named SISH (for self-supervised image search for histology) and provide as an open-source package, requires only slide-level annotations for training, encodes WSIs into meaningful discrete latent representations and leverages a tree data structure for fast searching followed by an uncertainty-based ranking algorithm for WSI retrieval. We evaluated SISH on multiple tasks (including retrieval tasks based on tissue-patch queries) and on datasets spanning over 22,000 patient cases and 56 disease subtypes. SISH can also be used to aid the diagnosis of rare cancer types for which the number of available WSIs is often insufficient to train supervised deep-learning models.

Deep Learning Approaches and Applications in Toxicologic Histopathology: Current Status and Future Perspectives

Whole slide imaging enables the use of a wide array of digital image analysis tools that are revolutionizing pathology. Recent advances in digital pathology and deep convolutional neural networks have created an enormous opportunity to improve workflow efficiency, provide more quantitative, objective, and consistent assessments of pathology datasets, and develop decision support systems. Such innovations are already making their way into clinical practice. However, the progress of machine learning - in particular, deep learning (DL) - has been rather slower in nonclinical toxicology studies. Histopathology data from toxicology studies are critical during the drug development process that is required by regulatory bodies to assess drug-related toxicity in laboratory animals and its impact on human safety in clinical trials. Due to the high volume of slides routinely evaluated, low-throughput, or narrowly performing DL methods that may work well in small-scale diagnostic studies or for the identification of a single abnormality are tedious and impractical for toxicologic pathology. Furthermore, regulatory requirements around good laboratory practice are a major hurdle for the adoption of DL in toxicologic pathology. This paper reviews the major DL concepts, emerging applications, and examples of DL in toxicologic pathology image analysis. We end with a discussion of specific challenges and directions for future research.

Selection, Visualization, and Interpretation of Deep Features in Lung Adenocarcinoma and Squamous Cell Carcinoma

Although deep learning networks applied to digital images have shown impressive results for many pathology-related tasks, their black-box approach and limitation in terms of interpretability are significant obstacles for their widespread clinical utility. This study investigates the visualization of deep features (DFs) to characterize two lung cancer subtypes, adenocarcinoma and squamous cell carcinoma. This study demonstrates that a subset of DFs exist that can accurately distinguish these two cancer subtypes, prominent DFs. Visualization of such individual DFs allows us to understand better histopathologic patterns at both the whole-slide and patch levels, allowing discrimination of these cancer types. These DFs were visualized at the whole slide image level through DF-specific heatmaps and at tissue patch level through generating activation maps. In addition, we show that these prominent DFs contain information that can distinguish carcinomas of organs other than the lung. This framework may serve as a platform for evaluating the interpretability of any deep network for diagnostic decision making.

Artificial intelligence for advance requesting of immunohistochemistry in diagnostically uncertain prostate biopsies

The use of immunohistochemistry in the reporting of prostate biopsies is an important adjunct when the diagnosis is not definite on haematoxylin and eosin (H&E) morphology alone. The process is however inherently inefficient with delays while waiting for pathologist review to make the request and duplicated effort reviewing a case more than once. In this study, we aimed to capture the workflow implications of immunohistochemistry requests and demonstrate a novel artificial intelligence tool to identify cases in which immunohistochemistry (IHC) is required and generate an automated request. We conducted audits of the workflow for prostate biopsies in order to understand the potential implications of automated immunohistochemistry requesting and collected prospective cases to train a deep neural network algorithm to detect tissue regions that presented ambiguous morphology on whole slide images. These ambiguous foci were selected on the basis of the pathologist requesting immunohistochemistry to aid diagnosis. A gradient boosted trees classifier was then used to make a slide-level prediction based on the outputs of the neural network prediction. The algorithm was trained on annotations of 219 immunohistochemistry-requested and 80 control images, and tested by threefold cross-validation. Validation was conducted on a separate validation dataset of 222 images. Non IHC-requested cases were diagnosed in 17.9 min on average, while IHC-requested cases took 33.4 min over multiple reporting sessions. We estimated 11 min could be saved on average per case by automated IHC requesting, by removing duplication of effort. The tool attained 99% accuracy and 0.99 Area Under the Curve (AUC) on the test data. In the validation, the average agreement with pathologists was 0.81, with a mean AUC of 0.80. We demonstrate the proof-of-principle that an AI tool making automated immunohistochemistry requests could create a significantly leaner workflow and result in pathologist time savings.

Data-efficient and weakly supervised computational pathology on whole-slide images

Deep-learning methods for computational pathology require either manual annotation of gigapixel whole-slide images (WSIs) or large datasets of WSIs with slide-level labels and typically suffer from poor domain adaptation and interpretability. Here we report an interpretable weakly supervised deep-learning method for data-efficient WSI processing and learning that only requires slide-level labels. The method, which we named clustering-constrained-attention multiple-instance learning (CLAM), uses attention-based learning to identify subregions of high diagnostic value to accurately classify whole slides and instance-level clustering over the identified representative regions to constrain and refine the feature space. By applying CLAM to the subtyping of renal cell carcinoma and non-small-cell lung cancer as well as the detection of lymph node metastasis, we show that it can be used to localize well-known morphological features on WSIs without the need for spatial labels, that it overperforms standard weakly supervised classification algorithms and that it is adaptable to independent test cohorts, smartphone microscopy and varying tissue content.

Next-Generation Scholarship: Rebranding Hematopathology Using Twitter: The MD Anderson Experience

Hematopathologists are witnessing very exciting times, as a new era of unsurpassed technological advances is unfolding exponentially, enhancing our understanding of diseases at the genomic and molecular levels. In the evolving field of precision medicine, our contributions as hematopathologists to medical practice are of paramount importance. Social media platforms such as Twitter have helped facilitate and enrich our professional  interactions and collaborations with others in our field and in other medical disciplines leading to a more holistic approach to patient care. These platforms also have created a novel means for instantaneous dissemination of new findings and recent publications, and are proving to be increasingly useful tools that can be harnessed to expand our knowledge and amplify our presence in the medical community. In this Editorial, we share our experience as hematopathologists with Twitter, and how we leveraged this platform to boost scholarly activities within and beyond our subspecialty, and as a powerful medium for worldwide dissemination of educational material and to promote our remote teaching activities during the COVID-19 pandemic.

The Network That Never Sleeps

This review describes how Twitter is currently used by laboratory professionals for education, research, and networking. This platform has a global audience. It enables users to post information publicly, easily, rapidly, and free of charge. The absence of hierarchies enables interactions that may not be feasible offline. Laboratory professionals teach thousands of people using text, images, polls, and videos. Academic discussion flourishes without paywalls. Published research is shared faster than ever before, articles are discussed in online journal clubs, and research collaborations are facilitated. Pathologists network globally and make new friends within and beyond their specialty. Pathology departments and residency programs showcase trainees and faculty and celebrate graduations. As users in one time zone go to bed, others who are just waking up begin to read and tweet, creating a 24/7/365 live global online conference. We encourage others to plug into the power of Twitter, the network that never sleeps.

Attitudes Toward Artificial Intelligence Within Dermatopathology: An International Online Survey

Background: Artificial intelligence (AI) has recently surfaced as a research topic in dermatology and dermatopathology. In a recent survey, dermatologists were overall positive toward a development with an increased use of AI, but little is known about the corresponding attitudes among pathologists working with dermatopathology. The objective of this investigation was to make an inventory of these attitudes.

Participants and Methods: An anonymous and voluntary online survey was prepared and distributed to pathologists who regularly analyzed dermatopathology slides/images. The survey consisted of 39 question divided in five sections; (1) AI as a topic in pathology; (2) previous exposure to AI as a topic in general; (3) applications for AI in dermatopathology; (4) feelings and attitudes toward AI and (5) self-reported tech-savviness and demographics. The survey opened on March 13, 2020 and closed on May 5, 2020.

Results: Overall, 718 responders (64.1% females) representing 91 countries were analyzed. While 81.5% of responders were aware of AI as an emerging topic in pathology, only 18.8% had either good or excellent knowledge about AI. In terms of diagnosis classification, 42.6% saw strong or very strong potential for automated suggestion of skin tumor diagnoses. The corresponding figure for inflammatory skin diseases was 23.0% (P_adj < 0.0001). For specific applications, the highest potential was considered for automated detection of mitosis (79.2%), automated suggestion of tumor margins (62.1%) and immunostaining evaluation (62.7%). The potential for automated suggestion of immunostaining (37.6%) and genetic panels (48.3%) were lower. Age did not impact the overall attitudes toward AI. Only 6.0% of the responders agreed or strongly agreed that the human pathologist will be replaced by AI in the foreseeable future. For the entire group, 72.3% agreed or strongly agreed that AI will improve dermatopathology and 84.1% thought that AI should be a part of medical training.

Conclusions: Pathologists are generally optimistic about the impact and potential benefit of AI in dermatopathology. The highest potential is expected for narrow specified tasks rather than a global automated suggestion of diagnoses. There is a strong need for education about AI and its use within dermatopathology.

Big Data, Natural Language Processing, and Deep Learning to Detect and Characterize Illicit COVID-19 Product Sales: Infoveillance Study on Twitter and Instagram

BACKGROUND

The coronavirus disease (COVID-19) pandemic is perhaps the greatest global health challenge of the last century. Accompanying this pandemic is a parallel "infodemic," including the online marketing and sale of unapproved, illegal, and counterfeit COVID-19 health products including testing kits, treatments, and other questionable "cures." Enabling the proliferation of this content is the growing ubiquity of internet-based technologies, including popular social media platforms that now have billions of global users.

OBJECTIVE

This study aims to collect, analyze, identify, and enable reporting of suspected fake, counterfeit, and unapproved COVID-19-related health care products from Twitter and Instagram.

METHODS

This study is conducted in two phases beginning with the collection of COVID-19-related Twitter and Instagram posts using a combination of web scraping on Instagram and filtering the public streaming Twitter application programming interface for keywords associated with suspect marketing and sale of COVID-19 products. The second phase involved data analysis using natural language processing (NLP) and deep learning to identify potential sellers that were then manually annotated for characteristics of interest. We also visualized illegal selling posts on a customized data dashboard to enable public health intelligence.

RESULTS

We collected a total of 6,029,323 tweets and 204,597 Instagram posts filtered for terms associated with suspect marketing and sale of COVID-19 health products from March to April for Twitter and February to May for Instagram. After applying our NLP and deep learning approaches, we identified 1271 tweets and 596 Instagram posts associated with questionable sales of COVID-19-related products. Generally, product introduction came in two waves, with the first consisting of questionable immunity-boosting treatments and a second involving suspect testing kits. We also detected a low volume of pharmaceuticals that have not been approved for COVID-19 treatment. Other major themes detected included products offered in different languages, various claims of product credibility, completely unsubstantiated products, unapproved testing modalities, and different payment and seller contact methods.

CONCLUSIONS

Results from this study provide initial insight into one front of the "infodemic" fight against COVID-19 by characterizing what types of health products, selling claims, and types of sellers were active on two popular social media platforms at earlier stages of the pandemic. This cybercrime challenge is likely to continue as the pandemic progresses and more people seek access to COVID-19 testing and treatment. This data intelligence can help public health agencies, regulatory authorities, legitimate manufacturers, and technology platforms better remove and prevent this content from harming the public.