AMMU: A survey of transformer-based biomedical pretrained language models

Transformer-based pretrained language models (PLMs) have started a new era in modern natural language processing (NLP). These models combine the power of transformers, transfer learning, and self-supervised learning (SSL). Following the success of these models in the general domain, the biomedical research community has developed various in-domain PLMs starting from BioBERT to the latest BioELECTRA and BioALBERT models. We strongly believe there is a need for a survey paper that can provide a comprehensive survey of various transformer-based biomedical pretrained language models (BPLMs). In this survey, we start with a brief overview of foundational concepts like self-supervised learning, embedding layer and transformer encoder layers. We discuss core concepts of transformer-based PLMs like pretraining methods, pretraining tasks, fine-tuning methods, and various embedding types specific to biomedical domain. We introduce a taxonomy for transformer-based BPLMs and then discuss all the models. We discuss various challenges and present possible solutions. We conclude by highlighting some of the open issues which will drive the research community to further improve transformer-based BPLMs. The list of all the publicly available transformer-based BPLMs along with their links is provided at https://mr-nlp.github.io/posts/2021/05/transformer-based-biomedical-pretrained-language-models-list/.

Real-time analysis of a mass vaccination effort confirms the safety of FDA-authorized mRNA COVID-19 vaccines

Background

As the coronavirus disease 2019 (COVID-19) vaccination campaign unfolds, it is important to continuously assess the real-world safety of Food and Drug Administration (FDA)-authorized vaccines. Curation of large-scale electronic health records (EHRs) enables near-real-time safety evaluations that were not previously possible.

Methods

In this retrospective study, we deployed deep neural networks over a large EHR system to automatically curate the adverse effects mentioned by physicians in over 1.2 million clinical notes between December 1, 2020 and April 20, 2021. We compared notes from 68,266 individuals who received at least one dose of BNT162b2 (n = 51,795) or mRNA-1273 (n = 16,471) to notes from 68,266 unvaccinated individuals who were matched by demographic, geographic, and clinical features.

Findings

Individuals vaccinated with BNT162b2 or mRNA-1273 had a higher rate of return to the clinic, but not the emergency department, after both doses compared to unvaccinated controls. The most frequently documented adverse effects within 7 days of each vaccine dose included myalgia, headache, and fatigue, but the rates of EHR documentation for each side effect were remarkably low compared to those derived from active solicitation during clinical trials. Severe events, including anaphylaxis, facial paralysis, and cerebral venous sinus thrombosis, were rare and occurred at similar frequencies in vaccinated and unvaccinated individuals.

Conclusions

This analysis of vaccine-related adverse effects from over 1.2 million EHR notes of more than 130,000 individuals reaffirms the safety and tolerability of the FDA-authorized mRNA COVID-19 vaccines in practice.

Funding

This study was funded by nference.

This is a study of the mRNA COVID-19 vaccines developed by Pfizer/BioNTech and Moderna. Although these vaccines have been shown to be safe and tolerated in clinical trials, it is important to confirm their safety profiles in practice. The results from this study show that individuals receiving these vaccines are likely to experience muscle and joint soreness, but they are not more likely to seek out emergent clinical care or experience severe medical events than unvaccinated individuals. As one of the largest real-world safety studies of COVID-19 vaccines to date, these data reinforce that we should continue expanding efforts to deliver more vaccines with high confidence in their safety.

Building a best-in-class automated de-identification tool for electronic health records through ensemble learning

The presence of personally identifiable information (PII) in natural language portions of electronic health records (EHRs) constrains their broad reuse. Despite continuous improvements in automated detection of PII, residual identifiers require manual validation and correction. Here, we describe an automated de-identification system that employs an ensemble architecture, incorporating attention-based deep-learning models and rule-based methods, supported by heuristics for detecting PII in EHR data. Detected identifiers are then transformed into plausible, though fictional, surrogates to further obfuscate any leaked identifier. Our approach outperforms existing tools, with a recall of 0.992 and precision of 0.979 on the i2b2 2014 dataset and a recall of 0.994 and precision of 0.967 on a dataset of 10,000 notes from the Mayo Clinic. The de-identification system presented here enables the generation of de-identified patient data at the scale required for modern machine-learning applications to help accelerate medical discoveries.

Synthesis, SAR Study and Evaluation of Mannich and Schiff Bases of Pyrazol-5(4H)-one Moiety Containing 3-(Hydrazinyl)-2-phenylquinazolin-4(3H)-one

In the present investigation, a series of 12 Mannich bases (QP1-12) and 5 Schiff bases (QSP1-5) of pyrazol-5(4H)-one moiety containing 3-(hydrazinyl)-2-phenylquinazolin-4(3H)-one has been synthesized and characterized by physicochemical as well as spectral means. The synthesized Mannich and Schiff bases were screened for their preliminary antimicrobial activity against Gram-positive and Gram-negative bacterial as well as fungal strains by the determination of zone of inhibition. Mannich bases (QP1-12) were found to be more potent antibacterial agents against Gram-positive bacteria, whereas Schiff bases (QSP1-5) were more potent against Gram-negative bacteria and fungi. Minimum inhibitory concentration result demonstrated that Mannich base compound (QP7) having ortho -OH and para -COOH group showed some improvement in antibacterial activity (minimum inhibitory concentration of 48.88×10⁻³ μM/ml) among the tested Gram-positive organisms and it also exhibit minimum inhibitory concentration of value of 12.22×10⁻³ μM/ml for Klebsiella pneumoniae. The antitubercular activity of synthesized compounds against Mycobacterium tuberculosis (H₃₇Rv) was determined using microplate alamar blue assay. Compound QP11 showed appreciable antitubercular activity (minimum inhibitory concentration of 6.49×10⁻³ μM/ml) which was more active than the standard drugs, ethambutol (minimum inhibitory concentration of 7.60×10⁻³ μM/ml) and ciprofloxacin (9.4×10⁻³ μM/ml). Compounds QP11, QP9, QSP1, QSP2, and QSP5 have good selective index and may be selected as a lead compound for the development of novel antitubercular agents.

Long-term persistence of cellular hyporesponsiveness to filarial antigens after clearance of microfilaremia

The persistence of parasite-specific cellular hyporesponsiveness after clearance of blood microfilariae (mf) was studied in 18 individuals who had been treated with a single dose of ivermectin, diethylcarbamazine, or a combination 2-3 years previously and who had initially cleared their parasitemia. At recruitment into the present study, 50% were again mf+ and 50% remained mf-. There were no significant differences between the mf+ and mf- groups in the amount of interferon-gamma (IFN-gamma) produced by peripheral blood mononuclear cells in response to adult or microfilarial antigens, although IFN-gamma production in response to purified protein derivative was greater in the mf+ group (geometric mean [gm] = 3,791 pg/ml; P = 0.02) than in the mf- group (gm = 600 pg/ml). These data suggest that although microfilaremic individuals may temporarily regain the ability to produce IFN-gamma to parasite antigens post-treatment, they subsequently revert to a state of hyporesponsiveness to mf-containing antigens that appears to be independent of the recurrence of microfilaremia and the response to nonparasite antigens.

'AMRITHAPALA' (Janakia arayalpatra, Joseph & Chandrasekharan), A NEW DRUG FROM THE KANI TRIBE OF KERALA

Amrithapala (Janakia arayalpatra), a rare and endemic plant species found in the Southern forests of Western Ghat region of kerala, is used by the local 'Kani' tribe as an effective remedy for peptic ulcer, cancer-like afflictions and as a rejuvenating tonic. Search made in Ayurvedic literature indicates that the plant may be the divine drug named variously as MRITHA SANJEEVINI (the drug that can revive unconscious or dead) or SANJEEVINI, THAMPRA RASAYANI in the Oushadha Nighantu (Dictionary of Medicinal Drugs) of Tayyil Kumaran Krishnan (1906).