Multitask Learning Strategy with Pseudo-Labeling: Face Recognition, Facial Landmark Detection, and Head Pose Estimation

Most facial analysis methods perform well in standardized testing but not in real-world testing. The main reason is that training models cannot easily learn various human features and background noise, especially for facial landmark detection and head pose estimation tasks with limited and noisy training datasets. To alleviate the gap between standardized and real-world testing, we propose a pseudo-labeling technique using a face recognition dataset consisting of various people and background noise. The use of our pseudo-labeled training dataset can help to overcome the lack of diversity among the people in the dataset. Our integrated framework is constructed using complementary multitask learning methods to extract robust features for each task. Furthermore, introducing pseudo-labeling and multitask learning improves the face recognition performance by enabling the learning of pose-invariant features. Our method achieves state-of-the-art (SOTA) or near-SOTA performance on the AFLW2000-3D and BIWI datasets for facial landmark detection and head pose estimation, with competitive face verification performance on the IJB-C test dataset for face recognition. We demonstrate this through a novel testing methodology that categorizes cases as soft, medium, and hard based on the pose values of IJB-C. The proposed method achieves stable performance even when the dataset lacks diverse face identifications.

A comparison of visual and auditory EEG interfaces for robot multi-stage task control

Shared autonomy holds promise for assistive robotics, whereby physically-impaired people can direct robots to perform various tasks for them. However, a robot that is capable of many tasks also introduces many choices for the user, such as which object or location should be the target of interaction. In the context of non-invasive brain-computer interfaces for shared autonomy-most commonly electroencephalography-based-the two most common choices are to provide either auditory or visual stimuli to the user-each with their respective pros and cons. Using the oddball paradigm, we designed comparable auditory and visual interfaces to speak/display the choices to the user, and had users complete a multi-stage robotic manipulation task involving location and object selection. Users displayed differing competencies-and preferences-for the different interfaces, highlighting the importance of considering modalities outside of vision when constructing human-robot interfaces.

Enhanced Multitask Learning for Hash Code Generation of Palmprint Biometrics

This paper presents a novel multitask learning framework for palmprint biometrics, which optimizes classification and hashing branches jointly. The classification branch within our framework facilitates the concurrent execution of three distinct tasks: identity recognition and classification of soft biometrics, encompassing gender and chirality. On the other hand, the hashing branch enables the generation of palmprint hash codes, optimizing for minimal storage as templates and efficient matching. The hashing branch derives the complementary information from these tasks by amalgamating knowledge acquired from the classification branch. This approach leads to superior overall performance compared to individual tasks in isolation. To enhance the effectiveness of multitask learning, two additional modules, an attention mechanism module and a customized gate control module, are introduced. These modules are vital in allocating higher weights to crucial channels and facilitating task-specific expert knowledge integration. Furthermore, an automatic weight adjustment module is incorporated to optimize the learning process further. This module fine-tunes the weights assigned to different tasks, improving performance. Integrating the three modules above has shown promising accuracies across various classification tasks and has notably improved authentication accuracy. The extensive experimental results validate the efficacy of our proposed framework.

Transfer learning empowers accurate pharmacokinetics prediction of small samples

Accurate assessment of pharmacokinetic (PK) properties is crucial for selecting optimal candidates and avoiding downstream failures. Transfer learning is an innovative machine learning approach enabling high-throughput prediction with limited data. Recently, transfer learning methods showed promise in predicting ADME/PK parameters. Given the prolific growth of research on transfer learning for PK prediction, a comprehensive review of its advantages and challenges is imperative. This study explores the fundamentals, classifications, toolkits and applications of various transfer learning techniques for PK prediction, demonstrating their utility through three practical case studies. This work will serve as a reference for drug design researchers.

Adapting the Number of Questions Based on Detected Psychological Distress for Cognitive Behavioral Therapy With an Embodied Conversational Agent: Comparative Study

BACKGROUND

The high prevalence of mental illness is a critical social problem. The limited availability of mental health services is a major factor that exacerbates this problem. One solution is to deliver cognitive behavioral therapy (CBT) using an embodied conversational agent (ECA). ECAs make it possible to provide health care without location or time constraints. One of the techniques used in CBT is Socratic questioning, which guides users to correct negative thoughts. The effectiveness of this approach depends on a therapist's skill to adapt to the user's mood or distress level. However, current ECAs do not possess this skill. Therefore, it is essential to implement this adaptation ability to the ECAs.

OBJECTIVE

This study aims to develop and evaluate a method that automatically adapts the number of Socratic questions based on the level of detected psychological distress during a CBT session with an ECA. We hypothesize that this adaptive approach to selecting the number of questions will lower psychological distress, reduce negative emotional states, and produce more substantial cognitive changes compared with a random number of questions.

METHODS

In this study, which envisions health care support in daily life, we recruited participants aged from 18 to 65 years for an experiment that involved 2 different conditions: an ECA that adapts a number of questions based on psychological distress detection or an ECA that only asked a random number of questions. The participants were assigned to 1 of the 2 conditions, experienced a single CBT session with an ECA, and completed questionnaires before and after the session.

RESULTS

The participants completed the experiment. There were slight differences in sex, age, and preexperimental psychological distress levels between the 2 conditions. The adapted number of questions condition showed significantly lower psychological distress than the random number of questions condition after the session. We also found a significant difference in the cognitive change when the number of questions was adapted based on the detected distress level, compared with when the number of questions was fewer than what was appropriate for the level of distress detected.

CONCLUSIONS

The results show that an ECA adapting the number of Socratic questions based on detected distress levels increases the effectiveness of CBT. Participants who received an adaptive number of questions experienced greater reductions in distress than those who received a random number of questions. In addition, the participants showed a greater amount of cognitive change when the number of questions matched the detected distress level. This suggests that adapting the question quantity based on distress level detection can improve the results of CBT delivered by an ECA. These results illustrate the advantages of ECAs, paving the way for mental health care that is more tailored and effective.

Joint micrograph denoising and protein localization in cryo-electron microscopy

Cryo-electron microscopy (cryo-EM) is an imaging technique that allows the visualization of proteins and macromolecular complexes at near-atomic resolution. The low electron doses used to prevent radiation damage to the biological samples result in images where the power of noise is 100 times stronger than that of the signal. Accurate identification of proteins from these low signal-to-noise ratio (SNR) images is a critical task, as the detected positions serve as inputs for the downstream 3D structure determination process. Current methods either fail to identify all true positives or result in many false positives, especially when analyzing images from smaller-sized proteins that exhibit extremely low contrast, or require manual labeling that can take days to complete. Acknowledging the fact that accurate protein identification is dependent upon the visual interpretability of micrographs, we propose a framework that can perform denoising and detection in a joint manner and enable particle localization under extremely low SNR conditions using self-supervised denoising and particle identification from sparsely annotated data. We validate our approach on three challenging single-particle cryo-EM datasets and projection images from one cryo-electron tomography dataset with extremely low SNR, showing that it outperforms existing state-of-the-art methods used for cryo-EM image analysis by a significant margin. We also evaluate the performance of our algorithm under decreasing SNR conditions and show that our method is more robust to noise than competing methods.

Multitask Learning for Mental Health: Depression, Anxiety, Stress (DAS) Using Wearables

This study investigates the prediction of mental well-being factors-depression, stress, and anxiety-using the NetHealth dataset from college students. The research addresses four key questions, exploring the impact of digital biomarkers on these factors, their alignment with conventional psychology literature, the time-based performance of applied methods, and potential enhancements through multitask learning. The findings reveal modality rankings aligned with psychology literature, validated against paper-based studies. Improved predictions are noted with temporal considerations, and further enhanced by multitasking. Mental health multitask prediction results show aligned baseline and multitask performances, with notable enhancements using temporal aspects, particularly with the random forest (RF) classifier. Multitask learning improves outcomes for depression and stress but not anxiety using RF and XGBoost.

Multitask Deep Learning for Joint Detection of Necrotizing Viral and Noninfectious Retinitis From Common Blood and Serology Test Data

Purpose

Necrotizing viral retinitis is a serious eye infection that requires immediate treatment to prevent permanent vision loss. Uncertain clinical suspicion can result in delayed diagnosis, inappropriate administration of corticosteroids, or repeated intraocular sampling. To quickly and accurately distinguish between viral and noninfectious retinitis, we aimed to develop deep learning (DL) models solely using noninvasive blood test data.

Methods

This cross-sectional study trained DL models using common blood and serology test data from 3080 patients (noninfectious uveitis of the posterior segment [NIU-PS] = 2858, acute retinal necrosis [ARN] = 66, cytomegalovirus [CMV], retinitis = 156). Following the development of separate base DL models for ARN and CMV retinitis, multitask learning (MTL) was employed to enable simultaneous discrimination. Advanced MTL models incorporating adversarial training were used to enhance DL feature extraction from the small, imbalanced data. We evaluated model performance, disease-specific important features, and the causal relationship between DL features and detection results.

Results

The presented models all achieved excellent detection performances, with the adversarial MTL model achieving the highest receiver operating characteristic curves (0.932 for ARN and 0.982 for CMV retinitis). Significant features for ARN detection included varicella-zoster virus (VZV) immunoglobulin M (IgM), herpes simplex virus immunoglobulin G, and neutrophil count, while for CMV retinitis, they encompassed VZV IgM, CMV IgM, and lymphocyte count. The adversarial MTL model exhibited substantial changes in detection outcomes when the key features were contaminated, indicating stronger causality between DL features and detection results.

Conclusions

The adversarial MTL model, using blood test data, may serve as a reliable adjunct for the expedited diagnosis of ARN, CMV retinitis, and NIU-PS simultaneously in real clinical settings.

ThyroidNet: A Deep Learning Network for Localization and Classification of Thyroid Nodules

Aim

This study aims to establish an artificial intelligence model, ThyroidNet, to diagnose thyroid nodules using deep learning techniques accurately.

Methods

A novel method, ThyroidNet, is introduced and evaluated based on deep learning for the localization and classification of thyroid nodules. First, we propose the multitask TransUnet, which combines the TransUnet encoder and decoder with multitask learning. Second, we propose the DualLoss function, tailored to the thyroid nodule localization and classification tasks. It balances the learning of the localization and classification tasks to help improve the model's generalization ability. Third, we introduce strategies for augmenting the data. Finally, we submit a novel deep learning model, ThyroidNet, to accurately detect thyroid nodules.

Results

ThyroidNet was evaluated on private datasets and was comparable to other existing methods, including U-Net and TransUnet. Experimental results show that ThyroidNet outperformed these methods in localizing and classifying thyroid nodules. It achieved improved accuracy of 3.9% and 1.5%, respectively.

Conclusion

ThyroidNet significantly improves the clinical diagnosis of thyroid nodules and supports medical image analysis tasks. Future research directions include optimization of the model structure, expansion of the dataset size, reduction of computational complexity and memory requirements, and exploration of additional applications of ThyroidNet in medical image analysis.

Multitask Learning-Based Deep Signal Identification for Advanced Spectrum Sensing

The explosive demand for wireless communications has intensified the complexity of spectrum dynamics, particularly within unlicensed bands. To promote efficient spectrum utilization and minimize interference during communication, spectrum sensing needs to evolve to a stage capable of detecting multidimensional spectrum states. Signal identification, which identifies each device's signal source, is a potent method for deriving the spectrum usage characteristics of wireless devices. However, most existing signal identification methods mainly focus on signal classification or modulation classification, thus offering limited spectrum information. In this paper, we propose DSINet, a multitask learning-based deep signal identification network for advanced spectrum sensing systems. DSINet addresses the deep signal identification problem, which involves not only classifying signals but also deriving the spectrum usage characteristics of signals across various spectrum dimensions, including time, frequency, power, and code. Comparative analyses reveal that DSINet outperforms existing shallow signal identification models, with performance improvements of 3.3% for signal classification, 3.3% for hall detection, and 5.7% for modulation classification. In addition, DSINet solves four different tasks with a 65.5% smaller model size and 230% improved computational performance compared to single-task learning model sets, providing meaningful results in terms of practical use.

Multitask Learning with Convolutional Neural Networks and Vision Transformers Can Improve Outcome Prediction for Head and Neck Cancer Patients

Neural-network-based outcome predictions may enable further treatment personalization of patients with head and neck cancer. The development of neural networks can prove challenging when a limited number of cases is available. Therefore, we investigated whether multitask learning strategies, implemented through the simultaneous optimization of two distinct outcome objectives (multi-outcome) and combined with a tumor segmentation task, can lead to improved performance of convolutional neural networks (CNNs) and vision transformers (ViTs). Model training was conducted on two distinct multicenter datasets for the endpoints loco-regional control (LRC) and progression-free survival (PFS), respectively. The first dataset consisted of pre-treatment computed tomography (CT) imaging for 290 patients and the second dataset contained combined positron emission tomography (PET)/CT data of 224 patients. Discriminative performance was assessed by the concordance index (C-index). Risk stratification was evaluated using log-rank tests. Across both datasets, CNN and ViT model ensembles achieved similar results. Multitask approaches showed favorable performance in most investigations. Multi-outcome CNN models trained with segmentation loss were identified as the optimal strategy across cohorts. On the PET/CT dataset, an ensemble of multi-outcome CNNs trained with segmentation loss achieved the best discrimination (C-index: 0.29, 95% confidence interval (CI): 0.22-0.36) and successfully stratified patients into groups with low and high risk of disease progression (p=0.003). On the CT dataset, ensembles of multi-outcome CNNs and of single-outcome ViTs trained with segmentation loss performed best (C-index: 0.26 and 0.26, CI: 0.18-0.34 and 0.18-0.35, respectively), both with significant risk stratification for LRC in independent validation (p=0.002 and p=0.011). Further validation of the developed multitask-learning models is planned based on a prospective validation study, which has recently completed recruitment.

FFMAVP: a new classifier based on feature fusion and multitask learning for identifying antiviral peptides and their subclasses

Antiviral peptides (AVPs) are widely found in animals and plants, with high specificity and strong sensitivity to drug-resistant viruses. However, due to the great heterogeneity of different viruses, most of the AVPs have specific antiviral activities. Therefore, it is necessary to identify the specific activities of AVPs on virus types. Most existing studies only identify AVPs, with only a few studies identifying subclasses by training multiple binary classifiers. We develop a two-stage prediction tool named FFMAVP that can simultaneously predict AVPs and their subclasses. In the first stage, we identify whether a peptide is AVP or not. In the second stage, we predict the six virus families and eight species specifically targeted by AVPs based on two multiclass tasks. Specifically, the feature extraction module in the two-stage task of FFMAVP adopts the same neural network structure, in which one branch extracts features based on amino acid feature descriptors and the other branch extracts sequence features. Then, the two types of features are fused for the following task. Considering the correlation between the two tasks of the second stage, a multitask learning model is constructed to improve the effectiveness of the two multiclass tasks. In addition, to improve the effectiveness of the second stage, the network parameters trained through the first-stage data are used to initialize the network parameters in the second stage. As a demonstration, the cross-validation results, independent test results and visualization results show that FFMAVP achieves great advantages in both stages.

NMDA-driven dendritic modulation enables multitask representation learning in hierarchical sensory processing pathways

While sensory representations in the brain depend on context, it remains unclear how such modulations are implemented at the biophysical level, and how processing layers further in the hierarchy can extract useful features for each possible contextual state. Here, we demonstrate that dendritic N-Methyl-D-Aspartate spikes can, within physiological constraints, implement contextual modulation of feedforward processing. Such neuron-specific modulations exploit prior knowledge, encoded in stable feedforward weights, to achieve transfer learning across contexts. In a network of biophysically realistic neuron models with context-independent feedforward weights, we show that modulatory inputs to dendritic branches can solve linearly nonseparable learning problems with a Hebbian, error-modulated learning rule. We also demonstrate that local prediction of whether representations originate either from different inputs, or from different contextual modulations of the same input, results in representation learning of hierarchical feedforward weights across processing layers that accommodate a multitude of contexts.

Automatic International Classification of Diseases Coding via Note-Code Interaction Network with Denoising Mechanism

Clinical notes are comprehensive files containing explicit information about a patient's visit. However, accurately assigning medical codes from clinical documents can be a persistent challenge due to the complexity of clinical data and the vast range of medical codes. Moreover, the large volume of medical records, the noisy medical records, and the uneven quality of coders all negatively impact the quality of the final codes. Deep learning technology has recently been integrated into automatic International Classification of Diseases (ICD) coding tasks to improve accuracy. Nevertheless, the imbalanced class problem, the complexness of code associations, and the noise in lengthy records still restrict the advancement of ICD coding tasks in deep learning. Thus, we present the Note-code Interaction Denoising Network (NIDN) that employs the self-attention mechanism to pull critical semantic features in electronic medical records (EMRs). Our model utilizes the label attention mechanism for retaining code-specific text expression. We introduce Clinical Classifications Software coding for multitask learning, capturing the functional relationships of medical coding to oblige in model prediction. To minimize the impact of noise on model prediction and improve the label distribution imbalance, a denoising module is introduced to filter noise. Our practical consequences indicate that the model NIDN exceeds competitive models on a third version of Medical Information Mart for Intensive Care data set.

Orchestrating information across tissues via a novel multitask GAT framework to improve quantitative gene regulation relation modeling for survival analysis

Survival analysis is critical to cancer prognosis estimation. High-throughput technologies facilitate the increase in the dimension of genic features, but the number of clinical samples in cohorts is relatively small due to various reasons, including difficulties in participant recruitment and high data-generation costs. Transcriptome is one of the most abundantly available OMIC (referring to the high-throughput data, including genomic, transcriptomic, proteomic and epigenomic) data types. This study introduced a multitask graph attention network (GAT) framework DQSurv for the survival analysis task. We first used a large dataset of healthy tissue samples to pretrain the GAT-based HealthModel for the quantitative measurement of the gene regulatory relations. The multitask survival analysis framework DQSurv used the idea of transfer learning to initiate the GAT model with the pretrained HealthModel and further fine-tuned this model using two tasks i.e. the main task of survival analysis and the auxiliary task of gene expression prediction. This refined GAT was denoted as DiseaseModel. We fused the original transcriptomic features with the difference vector between the latent features encoded by the HealthModel and DiseaseModel for the final task of survival analysis. The proposed DQSurv model stably outperformed the existing models for the survival analysis of 10 benchmark cancer types and an independent dataset. The ablation study also supported the necessity of the main modules. We released the codes and the pretrained HealthModel to facilitate the feature encodings and survival analysis of transcriptome-based future studies, especially on small datasets. The model and the code are available at http://www.healthinformaticslab.org/supp/.

AMSCN: A Novel Dual-Task Model for Automatic Modulation Classification and Specific Emitter Identification

Specific emitter identification (SEI) and automatic modulation classification (AMC) are generally two separate tasks in the field of radio monitoring. Both tasks have similarities in terms of their application scenarios, signal modeling, feature engineering, and classifier design. It is feasible and promising to integrate these two tasks, with the benefit of reducing the overall computational complexity and improving the classification accuracy of each task. In this paper, we propose a dual-task neural network named AMSCN that simultaneously classifies the modulation and the transmitter of the received signal. In the AMSCN, we first use a combination of DenseNet and Transformer as the backbone network to extract the distinguishable features; then, we design a mask-based dual-head classifier (MDHC) to reinforce the joint learning of the two tasks. To train the AMSCN, a multitask cross-entropy loss is proposed, which is the sum of the cross-entropy loss of the AMC and the cross-entropy loss of the SEI. Experimental results show that our method achieves performance gains for the SEI task with the aid of additional information from the AMC task. Compared with the traditional single-task model, our classification accuracy of the AMC is generally consistent with the state-of-the-art performance, while the classification accuracy of the SEI is improved from 52.2% to 54.7%, which demonstrates the effectiveness of the AMSCN.

A social theory-enhanced graph representation learning framework for multitask prediction of drug-drug interactions

Current machine learning-based methods have achieved inspiring predictions in the scenarios of mono-type and multi-type drug-drug interactions (DDIs), but they all ignore enhancive and depressive pharmacological changes triggered by DDIs. In addition, these pharmacological changes are asymmetric since the roles of two drugs in an interaction are different. More importantly, these pharmacological changes imply significant topological patterns among DDIs. To address the above issues, we first leverage Balance theory and Status theory in social networks to reveal the topological patterns among directed pharmacological DDIs, which are modeled as a signed and directed network. Then, we design a novel graph representation learning model named SGRL-DDI (social theory-enhanced graph representation learning for DDI) to realize the multitask prediction of DDIs. SGRL-DDI model can capture the task-joint information by integrating relation graph convolutional networks with Balance and Status patterns. Moreover, we utilize task-specific deep neural networks to perform two tasks, including the prediction of enhancive/depressive DDIs and the prediction of directed DDIs. Based on DDI entries collected from DrugBank, the superiority of our model is demonstrated by the comparison with other state-of-the-art methods. Furthermore, the ablation study verifies that Balance and Status patterns help characterize directed pharmacological DDIs, and that the joint of two tasks provides better DDI representations than individual tasks. Last, we demonstrate the practical effectiveness of our model by a version-dependent test, where 88.47 and 81.38% DDI out of newly added entries provided by the latest release of DrugBank are validated in two predicting tasks respectively.

RAMP: response-aware multi-task learning with contrastive regularization for cancer drug response prediction

The accurate prediction of cancer drug sensitivity according to the multiomics profiles of individual patients is crucial for precision cancer medicine. However, the development of prediction models has been challenged by the complex crosstalk of input features and the resistance-dominant drug response information contained in public databases. In this study, we propose a novel multidrug response prediction framework, response-aware multitask prediction (RAMP), via a Bayesian neural network and restrict it by soft-supervised contrastive regularization. To utilize network embedding vectors as representation learning features for heterogeneous networks, we harness response-aware negative sampling, which applies cell line-drug response information to the training of network embeddings. RAMP overcomes the prediction accuracy limitation induced by the imbalance of trained response data based on the comprehensive selection and utilization of drug response features. When trained on the Genomics of Drug Sensitivity in Cancer dataset, RAMP achieved an area under the receiver operating characteristic curve > 89%, an area under the precision-recall curve > 59% and an $\textrm{F}_1$ score > 52% and outperformed previously developed methods on both balanced and imbalanced datasets. Furthermore, RAMP predicted many missing drug responses that were not included in the public databases. Our results showed that RAMP will be suitable for the high-throughput prediction of cancer drug sensitivity and will be useful for guiding cancer drug selection processes. The Python implementation for RAMP is available at https://github.com/hvcl/RAMP.

Multispecies Machine Learning Predictions of In Vitro Intrinsic Clearance with Uncertainty Quantification Analyses

In pharmaceutical research, compounds are optimized for metabolic stability to avoid a too fast elimination of the drug. Intrinsic clearance (CL_int) measured in liver microsomes or hepatocytes is an important parameter during lead optimization. In this work, machine learning models were developed to relate the compound structure to microsomal metabolic stability and predict CL_int for new compounds. A multitask (MT) learning architecture was introduced to model the CL_int of six species simultaneously, giving as a result a multispecies machine learning model. MT graph neural network (MT-GNN) regression was identified as the top-performing method, and an ensemble of 10 MT-GNN models was evaluated prospectively. Geometric mean fold errors were consistently smaller than 2-fold. Moreover, high precision values were obtained in the prediction of "high" (>300 μL/min/mg) and "low" (<100 μL/min/mg) CL_int compounds. Precision values ranged from 80 to 94% for low CL_int predictions and from 75 to 97% for high CL_int predictions, depending on the species. Uncertainty on experimental values and model predictions was systematically quantified. Experimental variability (aleatoric uncertainty) of all historical Novartis in vitro clearance experiments was analyzed. Interestingly, MT-GNN models' performance approached assays' experimental variability. Moreover, uncertainty estimation in predictions (epistemic uncertainty) enabled identifying predictions associated with lower and higher error. Taken together, our manuscript combines a multispecies deep learning model and large-scale uncertainty analyses to improve CL_int predictions and facilitate early informed decisions for compound prioritization.

Automated measurement of spine indices on axial MR images for lumbar spinal stenosis diagnosis using segmentation-guided regression network

PURPOSE

Automated measurement of spine indices on axial magnetic resonance (MR) images plays a significant role in lumbar spinal stenosis diagnosis. Existing direct spine indices measurement approaches fail to explicitly focus on the task-specific region or feature channel with the additional information for guiding. We aim to achieve accurate spine indices measurement by introducing the guidance of the segmentation task.

METHODS

In this paper, we propose a segmentation-guided regression network (SGRNet) to achieve automated spine indices measurement. SGRNet consists of a segmentation path for generating the spine segmentation prediction and a regression path for producing spine indices estimation. The segmentation path is a U-Net-like network which includes a segmentation encoder and a decoder which generates multilevel segmentation features and segmentation prediction. The proposed segmentation-guided attention module (SGAM) in the regression encoder extracts the attention-aware regression feature under the guidance of the segmentation feature. Based on the attention-aware regression feature, a fully connected layer is utilized to output the accurate spine indices estimation.

RESULTS

Experiments on the open-access Lumbar Spine MRI data set show that SGRNet achieves state-of-the-art performance with a mean absolute error of 0.49 mm and mean Pearson correlation coefficient of 0.956 for four indices estimation.

CONCLUSIONS

The proposed SGAM in SGRNet is capable of improving the performance of spine indices measurement by focusing on the task-specific region and feature channel under the guidance of the segmentation task.

PUNDIT: Pulmonary nodule detection with image category transformation

BACKGROUND

Convolutional neural networks (CNNs) have achieved great success in pulmonary nodules detection, which plays an important role in lung cancer screening.

PURPOSE

In this paper, we proposed a novel strategy for pulmonary nodule detection by learning it from a harder task, which was to transform nodule images into normal images. We named this strategy as pulmonary nodule detection with image category transformation (PUNDIT).

METHODS

There were two steps for nodules detection, nodule candidate detection and false positive (FP) reduction. In nodule candidate detection step, a segmentation-based framework was built for detection. We designed an image category transformation (ICT) task to translate nodule images into pixel-to-pixel normal images and share the information of detection and transformation tasks by multitask learning. As for references of transformation tasks, we proposed background consistency losses into standard cycle-consistent adversarial networks, which can solve the problem of background uncontrolled changing. A three-dimensional network was used in FP reduction step.

RESULTS

PUNDIT was evaluated in two datasets, cancer screening dataset (CSD) with 1186 nodules for cross-validation and (CTD) with 3668 nodules for external test. Results were mainly evaluated by competition performance metric (CPM), the average sensitivity at seven predefined FP rates. The CPM was improved from 0.906 to 0.931 in CSD, and from 0.835 to 0.848 in CTD.

CONCLUSIONS

Experimental results showed that PUNDIT can improve the performance of pulmonary nodules detection effectively.

SDC-Net: End-to-End Multitask Self-Driving Car Camera Cocoon IoT-Based System

Currently, deep learning and IoT collaboration is heavily invading automotive applications especially in autonomous driving throughout successful assistance functionalities. Crash avoidance, path planning, and automatic emergency braking are essential functionalities for autonomous driving. Trigger-action-based IoT platforms are widely used due to its simplicity and ability of doing receptive tasks accurately. In this work, we propose SDC-Net system: an end-to-end deep learning IoT hybrid system in which a multitask neural network is trained based on different input representations from a camera-cocoon setup installed in CARLA simulator. We build our benchmark dataset covering different scenarios and corner cases that the vehicle may expose in order to navigate safely and robustly while testing. The proposed system aims to output relevant control actions for crash avoidance, path planning and automatic emergency braking. Multitask learning with a bird's eye view input representation outperforms the nearest representation in precision, recall, f1-score, accuracy, and average MSE by more than 11.62%, 9.43%, 10.53%, 6%, and 25.84%, respectively.

A deep learning-based automatic system for intracranial aneurysms diagnosis on three-dimensional digital subtraction angiographic images

BACKGROUND

Intracranial aneurysms (IAs) are a life-threatening disease. Their rupture can lead to hemorrhagic stroke. Most studies applying deep learning for the detection of aneurysms are based on angiographic images. However, critical diagnostic information such as morphology and aneurysm location are not captured by deep learning algorithms and still require manual assessments.

PURPOSE

Digital subtraction angiography (DSA) is the gold standard for aneurysm diagnosis. To facilitate the fully automatic diagnosis of aneurysms, we proposed a comprehensive system for the detection, morphology measurement, and location classification of aneurysms on three-dimensional DSA images, allowing automatic diagnosis without further human input.

METHODS

The system comprised three neural networks: a network for aneurysm detection, a network for morphology measurement, and a network for aneurysm location identification. A cross-scale dual-path transformer module was proposed to effectively fuse local and global information to capture aneurysms of varying sizes. A multitask learning approach was also proposed to allow an accurate localization of aneurysm neck for morphology measurement.

RESULTS

The cross-scale dual-path transformer module was shown to outperform other state-of-the-art network architectures, improving segmentation, and classification accuracy. The detection network in our system achieved an F2 score of 0.946 (recall 93%, precision 100%), better than the winning team in the Cerebral Aneurysm Detection and Analysis challenge. The measurement network achieved a relative error of less than 10% for morphology measurement, at the same level as human operators. Perfect accuracy (100%) was achieved on aneurysm location classification.

CONCLUSIONS

We have demonstrated that a comprehensive system can automatically detect, measure morphology and report the aneurysm location of aneurysms without human intervention. This can be a potential tool for the diagnosis of IAs, improving radiologists' performance and reducing their workload.

An Adaptive Task-Related Component Analysis Method for SSVEP Recognition

Steady-State Visual Evoked Potential (SSVEP) recognition methods use a subject's calibration data to differentiate between brain responses, hence, providing the SSVEP-based brain-computer interfaces (BCIs) with high performance. However, they require sufficient calibration EEG trials to achieve that. This study develops a new method to learn from limited calibration EEG trials, and it proposes and evaluates a novel adaptive data-driven spatial filtering approach for enhancing SSVEP detection. The spatial filter learned from each stimulus utilizes temporal information from the corresponding EEG trials. To introduce the temporal information into the overall procedure, a multitask learning approach, based on the Bayesian framework, is adopted. The performance of the proposed method was evaluated into two publicly available benchmark datasets, and the results demonstrated that our method outperformed competing methods by a significant margin.

Identifying Tissue- and Cohort-Specific RNA Regulatory Modules in Cancer Cells Using Multitask Learning

Simple Summary

Understanding the underlying biological mechanisms of primary tumors is crucial for predicting how tumors respond to therapies and exploring accurate treatment strategies. miRNA–mRNA interactions have a major effect on many biological processes that are important in the formation and progression of cancer. In this study, we introduced a computational pipeline to extract tissue- and cohort-specific miRNA–mRNA regulatory modules of multiple cancer types from the same origin using miRNA and mRNA expression profiles of primary tumors. Our model identified regulatory modules of underlying cancer types (i.e., cohort-specific) and shared regulatory modules between cohorts (i.e., tissue-specific).

Abstract

MicroRNA (miRNA) alterations significantly impact the formation and progression of human cancers. miRNAs interact with messenger RNAs (mRNAs) to facilitate degradation or translational repression. Thus, identifying miRNA–mRNA regulatory modules in cohorts of primary tumor tissues are fundamental for understanding the biology of tumor heterogeneity and precise diagnosis and treatment. We established a multitask learning sparse regularized factor regression (MSRFR) method to determine key tissue- and cohort-specific miRNA–mRNA regulatory modules from expression profiles of tumors. MSRFR simultaneously models the sparse relationship between miRNAs and mRNAs and extracts tissue- and cohort-specific miRNA–mRNA regulatory modules separately. We tested the model’s ability to determine cohort-specific regulatory modules of multiple cancer cohorts from the same tissue and their underlying tissue-specific regulatory modules by extracting similarities between cancer cohorts (i.e., blood, kidney, and lung). We also detected tissue-specific and cohort-specific signatures in the corresponding regulatory modules by comparing our findings from various other tissues. We show that MSRFR effectively determines cancer-related miRNAs in cohort-specific regulatory modules, distinguishes tissue- and cohort-specific regulatory modules from each other, and extracts tissue-specific information from different cohorts of disease-related tissue. Our findings indicate that the MSRFR model can support current efforts in precision medicine to define tumor-specific miRNA–mRNA signatures.

Salient deformable network for abdominal multiorgan registration

BACKGROUND

Image registration has long been an active research area in the society of medical image computing, which is to perform spatial transformation between a pair of images and establish a point-wise correspondence to achieve spatial consistency.

PURPOSE

Previous work mainly focused on learning complicated deformation fields by maximizing the global-level (i.e., foreground plus background) image similarity. We argue that taking the background similarity into account may not be a good solution, if we only seek the accurate alignment of target organs/regions in real clinical practice.

METHODS

We, therefore, propose a novel concept of S a l i e n t $Salient$ R e g i s t r a t i o n $Registration$ and introduce a novel deformable network equipped with a saliency module. Specifically, a multitask learning-based saliency module is proposed to discriminate the salient regions-of-registration in a semisupervised manner. Then, our deformable network analyzes the intensity and anatomical similarity of salient regions, and finally conducts the salient deformable registration.

RESULTS

We evaluate the efficacy of the proposed network on challenging abdominal multiorgan CT scans. The experimental results demonstrate that the proposed registration network outperforms other state-of-the-art methods, achieving a mean Dice similarity coefficient (DSC) of 40.2%, Hausdorff distance (95 HD) of 20.8 mm, and average symmetric surface distance (ASSD) of 4.58 mm. Moreover, even by training using one labeled data, our network can still attain satisfactory registration performance, with a mean DSC of 39.2%, 95 HD of 21.2 mm, and ASSD of 4.78 mm.

CONCLUSIONS

The proposed network provides an accurate solution for multiorgan registration and has the potential to be used for improving other registration applications. The code is publicly available at https://github.com/Rrrfrr/Salient-Deformable-Network.

A Deep Learning Framework for Automatic Meal Detection and Estimation in Artificial Pancreas Systems

Current artificial pancreas (AP) systems are hybrid closed-loop systems that require manual meal announcements to manage postprandial glucose control effectively. This poses a cognitive burden and challenge to users with T1D since this relies on frequent user engagement to maintain tight glucose control. In order to move towards fully automated closed-loop glucose control, we propose an algorithm based on a deep learning framework that performs multitask quantile regression, for both meal detection and carbohydrate estimation. Our proposed method is evaluated in silico on 10 adult subjects from the UVa/Padova simulator with a Bio-inspired Artificial Pancreas (BiAP) control algorithm over a 2 month period. Three different configurations of the AP are evaluated -BiAP without meal announcement (BiAP-NMA), BiAP with meal announcement (BiAP-MA), and BiAP with meal detection (BiAP-MD). We present results showing an improvement of BiAP-MD over BiAP-NMA, demonstrating 144.5 ± 6.8 mg/dL mean blood glucose level (-4.4 mg/dL, p< 0.01) and 77.8 ± 6.3% mean time between 70 and 180 mg/dL (+3.9%, p< 0.001). This improvement in control is realised without a significant increase in mean in hypoglycaemia (+0.1%, p= 0.4). In terms of detection of meals and snacks, the proposed method on average achieves 93% precision and 76% recall with a detection delay time of 38 ± 15 min (92% precision, 92% recall, and 37 min detection time for meals only). Furthermore, BiAP-MD handles hypoglycaemia better than BiAP-MA based on CVGA assessment with fewer control errors (10% vs. 20%). This study suggests that multitask quantile regression can improve the capability of AP systems for postprandial glucose control without increasing hypoglycaemia.

Machine Learning-Assisted QSAR Models on Contaminant Reactivity Toward Four Oxidants: Combining Small Data Sets and Knowledge Transfer

To develop predictive models for the reactivity of organic contaminants toward four oxidants─SO₄^•-, HClO, O₃, and ClO₂─all with small sample sizes, we proposed two approaches: combining small data sets and transferring knowledge between them. We first merged these data sets and developed a unified model using machine learning (ML), which showed better predictive performance than the individual models for HClO (RMSE_test: 2.1 to 2.04), O₃ (2.06 to 1.94), ClO₂ (1.77 to 1.49), and SO₄^•- (0.75 to 0.70) because the model "corrected" the wrongly learned effects of several atom groups. We further developed knowledge transfer models for three pairs of the data sets and observed different predictive performances: improved for O₃ (RMSE_test: 2.06 to 2.01)/HClO (2.10 to 1.98), mixed for O₃ (2.06 to 2.01)/ClO₂ (1.77 to 1.95), and unchanged for ClO₂ (1.77 to 1.77)/HClO (2.1 to 2.1). The effectiveness of the latter approach depended on whether there was consistent knowledge shared between the data sets and on the performance of the individual models. We also compared our approaches with multitask learning and image-based transfer learning and found that our approaches consistently improved the predictive performance for all data sets while the other two did not. This study demonstrated the effectiveness of combining small, similar data sets and transferring knowledge between them to improve ML model performance.

Clustering by Errors: A Self-Organized Multitask Learning Method for Acoustic Scene Classification

Acoustic scene classification (ASC) tries to inference information about the environment using audio segments. The inter-class similarity is a significant issue in ASC as acoustic scenes with different labels may sound quite similar. In this paper, the similarity relations amongst scenes are correlated with the classification error. A class hierarchy construction method by using classification error is then proposed and integrated into a multitask learning framework. The experiments have shown that the proposed multitask learning method improves the performance of ASC. On the TUT Acoustic Scene 2017 dataset, we obtain the ensemble fine-grained accuracy of 81.4%, which is better than the state-of-the-art. By using multitask learning, the basic Convolutional Neural Network (CNN) model can be improved by about 2.0 to 3.5 percent according to different spectrograms. The coarse category accuracies (for two to six super-classes) range from 77.0% to 96.2% by single models. On the revised version of the LITIS Rouen dataset, we achieve the ensemble fine-grained accuracy of 83.9%. The multitask learning models obtain an improvement of 1.6% to 1.8% compared to their basic models. The coarse category accuracies range from 94.9% to 97.9% for two to six super-classes with single models.

Transformational machine learning: Learning how to learn from many related scientific problems

Machine learning (ML) is the branch of artificial intelligence (AI) that develops computational systems that learn from experience. In supervised ML, the ML system generalizes from labelled examples to learn a model that can predict the labels of unseen examples. Examples are generally represented using features that directly describe the examples. For instance, in drug design, ML uses features that describe molecular shape and so on. In cases where there are multiple related ML problems, it is possible to use a different type of feature: predictions made about the examples by ML models learned on other problems. We call this transformational ML. We show that this results in better predictions and improved understanding when applied to scientific problems.

Almost all machine learning (ML) is based on representing examples using intrinsic features. When there are multiple related ML problems (tasks), it is possible to transform these features into extrinsic features by first training ML models on other tasks and letting them each make predictions for each example of the new task, yielding a novel representation. We call this transformational ML (TML). TML is very closely related to, and synergistic with, transfer learning, multitask learning, and stacking. TML is applicable to improving any nonlinear ML method. We tested TML using the most important classes of nonlinear ML: random forests, gradient boosting machines, support vector machines, k-nearest neighbors, and neural networks. To ensure the generality and robustness of the evaluation, we utilized thousands of ML problems from three scientific domains: drug design, predicting gene expression, and ML algorithm selection. We found that TML significantly improved the predictive performance of all the ML methods in all the domains (4 to 50% average improvements) and that TML features generally outperformed intrinsic features. Use of TML also enhances scientific understanding through explainable ML. In drug design, we found that TML provided insight into drug target specificity, the relationships between drugs, and the relationships between target proteins. TML leads to an ecosystem-based approach to ML, where new tasks, examples, predictions, and so on synergistically interact to improve performance. To contribute to this ecosystem, all our data, code, and our ∼50,000 ML models have been fully annotated with metadata, linked, and openly published using Findability, Accessibility, Interoperability, and Reusability principles (∼100 Gbytes).

BI-RADS-NET: AN EXPLAINABLE MULTITASK LEARNING APPROACH FOR CANCER DIAGNOSIS IN BREAST ULTRASOUND IMAGES

In healthcare, it is essential to explain the decision-making process of machine learning models to establish the trustworthiness of clinicians. This paper introduces BI-RADS-Net, a novel explainable deep learning approach for cancer detection in breast ultrasound images. The proposed approach incorporates tasks for explaining and classifying breast tumors, by learning feature representations relevant to clinical diagnosis. Explanations of the predictions (benign or malignant) are provided in terms of morphological features that are used by clinicians for diagnosis and reporting in medical practice. The employed features include the BI-RADS descriptors of shape, orientation, margin, echo pattern, and posterior features. Additionally, our approach predicts the likelihood of malignancy of the findings, which relates to the BI-RADS assessment category reported by clinicians. Experimental validation on a dataset consisting of 1,192 images indicates improved model accuracy, supported by explanations in clinical terms using the BI-RADS lexicon.

Simultaneous image reconstruction and lesion segmentation in accelerated MRI using multitasking learning

PURPOSE

Magnetic resonance imaging (MRI) serves as an important medical imaging modality for a variety of clinical applications. However, the problem of long imaging time limited its wide usage. In addition, prolonged scan time will cause discomfort to the patient, leading to severe image artifacts. On the other hand, manually lesion segmentation is time consuming. Algorithm-based automatic lesion segmentation is still challenging, especially for accelerated imaging with low quality.

METHODS

In this paper, we proposed a multitask learning-based method to perform image reconstruction and lesion segmentation simultaneously, called "RecSeg". Our hypothesis is that both tasks can benefit from the usage of the proposed combined model. In the experiment, we validated the proposed multitask model on MR k-space data with different acceleration factors (2×, 4×, and 6×). Two connected U-nets were used for the tasks of liver and renal image reconstruction and segmentation. A total of 50 healthy subjects and 100 patients with hepatocellular carcinoma were included for training and testing. For the segmentation part, we use healthy subjects to verify organ segmentation, and hepatocellular carcinoma patients to verify lesion segmentation. The organs and lesions were manually contoured by an experienced radiologist.

RESULTS

Experimental results show that the proposed RecSeg yielded the highest PSNR (RecSeg: 32.39 ± 1.64 vs. KSVD: 29.53 ± 2.74 and single U-net: 31.18 ± 1.68, respectively, p < 0.05) and highest structural similarity index measure (SSIM) (RecSeg: 0.93 ± 0.01 vs. KSVD: 0.88 ± 0.02 and single U-net: 0.90 ± 0.01, respectively, p < 0.05) under 6× acceleration. Moreover, in the task of lesion segmentation, it is proposed that RecSeg produced the highest Dice score (RecSeg: 0.86 ± 0.01 vs. KSVD: 0.82 ± 0.01 and single U-net: 0.84 ± 0.01, respectively, p < 0.05).

CONCLUSIONS

This study focused on the simultaneous reconstruction of medical images and the segmentation of organs and lesions. It is observed that the multitask learning-based method can improve performances of both image reconstruction and lesion segmentation.

MT-clinical BERT: scaling clinical information extraction with multitask learning

OBJECTIVE

Clinical notes contain an abundance of important, but not-readily accessible, information about patients. Systems that automatically extract this information rely on large amounts of training data of which there exists limited resources to create. Furthermore, they are developed disjointly, meaning that no information can be shared among task-specific systems. This bottleneck unnecessarily complicates practical application, reduces the performance capabilities of each individual solution, and associates the engineering debt of managing multiple information extraction systems.

MATERIALS AND METHODS

We address these challenges by developing Multitask-Clinical BERT: a single deep learning model that simultaneously performs 8 clinical tasks spanning entity extraction, personal health information identification, language entailment, and similarity by sharing representations among tasks.

RESULTS

We compare the performance of our multitasking information extraction system to state-of-the-art BERT sequential fine-tuning baselines. We observe a slight but consistent performance degradation in MT-Clinical BERT relative to sequential fine-tuning.

DISCUSSION

These results intuitively suggest that learning a general clinical text representation capable of supporting multiple tasks has the downside of losing the ability to exploit dataset or clinical note-specific properties when compared to a single, task-specific model.

CONCLUSIONS

We find our single system performs competitively with all state-the-art task-specific systems while also benefiting from massive computational benefits at inference.

Deep Large-Scale Multitask Learning Network for Gene Expression Inference

Gene expression profiling makes it possible to conduct many biological studies in a variety of fields due to its thorough characterization of cellular states under various experimental conditions. Despite recent advances in high-throughput technology, profiling an entire set of genomes is still difficult and expensive. Due to the high correlation between expression patterns of different genes, the aforementioned problem can be solved with a cost-effective approach that collects only a small subset of genes, called landmark genes, representing the entire set of genes, and infer the remaining genes, called target genes, using a computational model. There are several shallow and deep regression models in literature to estimate the expressions of target genes from the landmark genes. However, the shallow mostly have limited capacity in learning the nonlinear and complex gene expression data and are prone to underfitting, and the deep models generally do not take advantage of correlation among target genes in the learning process and suffer from overfitting. Considering the gene expression inference as a multitask learning problem, we propose a new deep multitask learning algorithm to tackle these issues. Our learning framework automatically learns the correlation between target genes and uses this knowledge to improve its generalization. Specifically, we utilize a subnetwork with low-dimensional latent variables to discover the relationships between target genes and enforce a seamless and easy to implement regularization to our deep regression model. Unlike the existing multitask learning methods that can only deal with dozens or hundreds of tasks, our algorithm is able to efficiently learn the relationships between ∼10,000 target genes and, thus, is scalable to a large number of tasks. Our proposed method outperforms the shallow and deep regression models for gene expression inference and alternative multitask learning algorithms on two large-scale datasets regardless of the network architecture.

Multitask Learning Based Three-Dimensional Striatal Segmentation of MRI: fMRI and PET Objective Assessments

BACKGROUND

Recent studies have established a clear topographical and functional organization of projections to and from complex subdivisions of the striatum. Manual segmentation of these functional subdivisions is labor-intensive and time-consuming, and automated methods are not as reliable as manual segmentation.

PURPOSE

To utilize multitask learning (MTL) as a method to segment subregions of the striatum consisting of pre-commissural putamen (prePU), pre-commissural caudate (preCA), post-commissural putamen (postPU), post-commissural caudate (postCA), and ventral striatum (VST).

STUDY TYPE

Retrospective.

POPULATION

Eighty-seven total data sets from patients with schizophrenia and matched controls.

FIELD STRENGTH/SEQUENCE

1.5 T and 3.0 T, T₁ -weighted (SPGR SENSE, 3D BRAVO).

ASSESSMENT

MTL-generated segmentations were compared to the Imperial College London Clinical Imaging Center (CIC) atlas. Dice similarity coefficient (DSC) was used to compare the automated methods to manual segmentations. Positron emission tomography (PET) imaging: 60 minutes of emission data were acquired using [¹¹ C]raclopride. Data were reconstructed by filtered back projection (FBP) with computed tomography (CT) used for attenuation correction. Binding potential values, BP_ND , and region of interest (ROI) time series and whole-brain connectivity using functional magnetic resonance imaging (fMRI) images were compared between manual and both automated segmentations.

STATISTICAL TESTS

Pearson correlation and paired t-test.

RESULTS

MTL-generated segmentations showed excellent spatial agreement with manual (DSC ≥0.72 across all striatal subregions). BP_ND values from MTL-generated segmentations were shown to correlate well with manual segmentations with R²  ≥ 0.91 in all caudate and putamen subregions, and R²  = 0.69 in VST. Mean Pearson correlation coefficients of the fMRI data between MTL-generated and manual segmentations were also high in time series (≥0.86) and whole-brain connectivity (≥0.89) across all subregions.

DATA CONCLUSION

Across both PET and fMRI task-based assessments, results from MTL-generated segmentations more closely corresponded to results from manually drawn ROIs than CIC-generated segmentations did. Therefore, the proposed MTL approach is a fast and reliable method for three-dimensional striatal subregion segmentation with results comparable to manually segmented ROIs.

LEVEL OF EVIDENCE

2 TECHNICAL EFFICACY STAGE: 1.

The challenges of generalizability in artificial intelligence for ADME/Tox endpoint and activity prediction

INTRODUCTION

Artificial intelligence (AI) has seen a massive resurgence in recent years with wide successes in computer vision, natural language processing, and games. The similar creation of robust and accurate AI models for ADME/Tox endpoint and activity prediction would be revolutionary to drug discovery pipelines. There have been numerous demonstrations of successful applications, but a key challenge remains: how generalizable are these predictive models?

AREAS COVERED

The authors present a summary of current promising components of AI models in the context of early drug discovery where ADME/Tox endpoint and activity prediction is the main driver of the iterative drug design process. Following that is a review of applicability domains and dataset construction considerations which determine generalizability bottlenecks for AI deployment. Further reviewed is the role of promising learning frameworks - multitask, transfer, and meta learning - which leverage auxiliary data to overcome issues of generalizability.

EXPERT OPINION

The authors conclude that the most promising direction toward integrating reliable and informative AI models into the drug discovery pipeline is a conjunction of learned feature representations, deep learning, and novel learning frameworks. Such a solution would address the sparse and incomplete datasets that are available for key endpoints related to drug discovery.

Automatic segmentation of hippocampus in hippocampal sparing whole brain radiotherapy: A multitask edge-aware learning

PURPOSE

This study aimed to improve the accuracy of the hippocampus segmentation through multitask edge-aware learning.

METHOD

We developed a multitask framework for computerized hippocampus segmentation. We used three-dimensional (3D) U-net as our backbone model with two training objectives: (a) to minimize the difference between the targeted binary mask and the model prediction; and (b) to optimize an auxiliary edge-prediction task which is designed to guide the model detection of the weak boundary of the hippocampus in model optimization. To balance the multiple task objectives, we proposed an improved gradient normalization by adaptively adjusting the weight of losses from different tasks. A total of 247 T1-weighted MRIs including 131 without contrast and 116 with contrast were collected from 247 patients to train and validate the proposed method. Segmentation was quantitatively evaluated with the dice coefficient (Dice), Hausdorff distance (HD), and average Hausdorff distance (AVD). The 3D U-net was used for baseline comparison. We used a Wilcoxon signed-rank test to compare repeated measurements (Dice, HD, and AVD) by different segmentations.

RESULTS

Through fivefold cross-validation, our multitask edge-aware learning achieved Dice of 0.8483 ± 0.0036, HD of 7.5706 ± 1.2330 mm, and AVD of 0.1522 ± 0.0165 mm, respectively. Conversely, the baseline results were 0.8340 ± 0.0072, 10.4631 ± 2.3736 mm, and 0.1884 ± 0.0286 mm, respectively. With a Wilcoxon signed-rank test, we found that the differences between our method and the baseline were statistically significant (P < 0.05).

CONCLUSION

Our results demonstrated the efficiency of multitask edge-aware learning in hippocampus segmentation for hippocampal sparing whole-brain radiotherapy. The proposed framework may also be useful for other low-contrast small organ segmentations on medical imaging modalities.

Automatic extraction of cancer registry reportable information from free-text pathology reports using multitask convolutional neural networks

OBJECTIVE

We implement 2 different multitask learning (MTL) techniques, hard parameter sharing and cross-stitch, to train a word-level convolutional neural network (CNN) specifically designed for automatic extraction of cancer data from unstructured text in pathology reports. We show the importance of learning related information extraction (IE) tasks leveraging shared representations across the tasks to achieve state-of-the-art performance in classification accuracy and computational efficiency.

MATERIALS AND METHODS

Multitask CNN (MTCNN) attempts to tackle document information extraction by learning to extract multiple key cancer characteristics simultaneously. We trained our MTCNN to perform 5 information extraction tasks: (1) primary cancer site (65 classes), (2) laterality (4 classes), (3) behavior (3 classes), (4) histological type (63 classes), and (5) histological grade (5 classes). We evaluated the performance on a corpus of 95 231 pathology documents (71 223 unique tumors) obtained from the Louisiana Tumor Registry. We compared the performance of the MTCNN models against single-task CNN models and 2 traditional machine learning approaches, namely support vector machine (SVM) and random forest classifier (RFC).

RESULTS

MTCNNs offered superior performance across all 5 tasks in terms of classification accuracy as compared with the other machine learning models. Based on retrospective evaluation, the hard parameter sharing and cross-stitch MTCNN models correctly classified 59.04% and 57.93% of the pathology reports respectively across all 5 tasks. The baseline models achieved 53.68% (CNN), 46.37% (RFC), and 36.75% (SVM). Based on prospective evaluation, the percentages of correctly classified cases across the 5 tasks were 60.11% (hard parameter sharing), 58.13% (cross-stitch), 51.30% (single-task CNN), 42.07% (RFC), and 35.16% (SVM). Moreover, hard parameter sharing MTCNNs outperformed the other models in computational efficiency by using about the same number of trainable parameters as a single-task CNN.

CONCLUSIONS

The hard parameter sharing MTCNN offers superior classification accuracy for automated coding support of pathology documents across a wide range of cancers and multiple information extraction tasks while maintaining similar training and inference time as those of a single task-specific model.

Biomarker-Based Classification and Localization of Renal Lesions Using Learned Representations of Histology-A Machine Learning Approach to Histopathology

Several deep learning approaches have been proposed to address the challenges in computational pathology by learning structural details in an unbiased way. Transfer learning allows starting from a learned representation of a pretrained model to be directly used or fine-tuned for a new domain. However, in histopathology, the problem domain is tissue-specific and putting together a labelled data set is challenging. On the other hand, whole slide-level annotations, such as biomarker levels, are much easier to obtain. We compare two pretrained models, one histology-specific and one from ImageNet on various computational pathology tasks. We show that a domain-specific model (HistoNet) contains richer information for biomarker classification, localization of biomarker-relevant morphology within a slide, and the prediction of expert-graded features. We use a weakly supervised approach to discriminate slides based on biomarker level and simultaneously predict which regions contribute to that prediction. We employ multitask learning to show that learned representations correlate with morphological features graded by expert pathologists. All of these results are demonstrated in the context of renal toxicity in a mechanistic study of compound toxicity in rat models. Our results emphasize the importance of histology-specific models and their knowledge representations for solving a wide range of computational pathology tasks.

Data-Driven Modeling of Pregnancy-Related Complications

A healthy pregnancy depends on complex interrelated biological adaptations involving placentation, maternal immune responses, and hormonal homeostasis. Recent advances in high-throughput technologies have provided access to multiomics biological data that, combined with clinical and social data, can provide a deeper understanding of normal and abnormal pregnancies. Integration of these heterogeneous datasets using state-of-the-art machine-learning methods can enable the prediction of short- and long-term health trajectories for a mother and offspring and the development of treatments to prevent or minimize complications. We review advanced machine-learning methods that could: provide deeper biological insights into a pregnancy not yet unveiled by current methodologies; clarify the etiologies and heterogeneity of pathologies that affect a pregnancy; and suggest the best approaches to address disparities in outcomes affecting vulnerable populations.

Automatic quality assessment for 2D fetal sonographic standard plane based on multitask learning

ABSTRACT

The quality control of fetal sonographic (FS) images is essential for the correct biometric measurements and fetal anomaly diagnosis. However, quality control requires professional sonographers to perform and is often labor-intensive. To solve this problem, we propose an automatic image quality assessment scheme based on multitask learning to assist in FS image quality control. An essential criterion for FS image quality control is that all the essential anatomical structures in the section should appear full and remarkable with a clear boundary. Therefore, our scheme aims to identify those essential anatomical structures to judge whether an FS image is the standard image, which is achieved by 3 convolutional neural networks. The Feature Extraction Network aims to extract deep level features of FS images. Based on the extracted features, the Class Prediction Network determines whether the structure meets the standard and Region Proposal Network identifies its position. The scheme has been applied to 3 types of fetal sections, which are the head, abdominal, and heart. The experimental results show that our method can make a quality assessment of an FS image within less a second. Also, our method achieves competitive performance in both the segmentation and diagnosis compared with state-of-the-art methods.

Anticancer drug synergy prediction in understudied tissues using transfer learning

OBJECTIVE

Drug combination screening has advantages in identifying cancer treatment options with higher efficacy without degradation in terms of safety. A key challenge is that the accumulated number of observations in in-vitro drug responses varies greatly among different cancer types, where some tissues are more understudied than the others. Thus, we aim to develop a drug synergy prediction model for understudied tissues as a way of overcoming data scarcity problems.

MATERIALS AND METHODS

We collected a comprehensive set of genetic, molecular, phenotypic features for cancer cell lines. We developed a drug synergy prediction model based on multitask deep neural networks to integrate multimodal input and multiple output. We also utilized transfer learning from data-rich tissues to data-poor tissues.

RESULTS

We showed improved accuracy in predicting synergy in both data-rich tissues and understudied tissues. In data-rich tissue, the prediction model accuracy was 0.9577 AUROC for binarized classification task and 174.3 mean squared error for regression task. We observed that an adequate transfer learning strategy significantly increases accuracy in the understudied tissues.

CONCLUSIONS

Our synergy prediction model can be used to rank synergistic drug combinations in understudied tissues and thus help to prioritize future in-vitro experiments. Code is available at https://github.com/yejinjkim/synergy-transfer.

A Multitask-Aided Transfer Learning-Based Diagnostic Framework for Bearings under Inconsistent Working Conditions

Rolling element bearings are a vital part of rotating machines and their sudden failure can result in huge economic losses as well as physical causalities. Popular bearing fault diagnosis techniques include statistical feature analysis of time, frequency, or time-frequency domain data. These engineered features are susceptible to variations under inconsistent machine operation due to the non-stationary, non-linear, and complex nature of the recorded vibration signals. To address these issues, numerous deep learning-based frameworks have been proposed in the literature. However, the logical reasoning behind crack severities and the longer training times needed to identify multiple health characteristics at the same time still pose challenges. Therefore, in this work, a diagnosis framework is proposed that uses higher-order spectral analysis and multitask learning (MTL), while also incorporating transfer learning (TL). The idea is to first preprocess the vibration signals recorded from a bearing to look for distinct patterns for a given fault type under inconsistent working conditions, e.g., variable motor speeds and loads, multiple crack severities, compound faults, and ample noise. Later, these bispectra are provided as an input to the proposed MTL-based convolutional neural network (CNN) to identify the speed and the health conditions, simultaneously. Finally, the TL-based approach is adopted to identify bearing faults in the presence of multiple crack severities. The proposed diagnostic framework is evaluated on several datasets and the experimental results are compared with several state-of-the-art diagnostic techniques to validate the superiority of the proposed model under inconsistent working conditions.

Crowd Counting with Semantic Scene Segmentation in Helicopter Footage

Continually improving crowd counting neural networks have been developed in recent years. The accuracy of these networks has reached such high levels that further improvement is becoming very difficult. However, this high accuracy lacks deeper semantic information, such as social roles (e.g., student, company worker, or police officer) or location-based roles (e.g., pedestrian, tenant, or construction worker). Some of these can be learned from the same set of features as the human nature of an entity, whereas others require wider contextual information from the human surroundings. The primary end-goal of developing recognition software is to involve them in autonomous decision-making systems. Therefore, it must be foolproof, which is, it must have good semantic understanding of the input. In this study, we focus on counting pedestrians in helicopter footage and introduce a dataset created from helicopter videos for this purpose. We use semantic segmentation to extract the required additional contextual information from the surroundings of an entity. We demonstrate that it is possible to increase the pedestrian counting accuracy in this manner. Furthermore, we show that crowd counting and semantic segmentation can be simultaneously achieved, with comparable or even improved accuracy, by using the same crowd counting neural network for both tasks through hard parameter sharing. The presented method is generic and it can be applied to arbitrary crowd density estimation methods. A link to the dataset is available at the end of the paper.

Discovering Temporal Patterns in Longitudinal Nontargeted Metabolomics Data via Group and Nuclear Norm Regularized Multivariate Regression

Temporal associations in longitudinal nontargeted metabolomics data are generally ignored by common pattern recognition methods such as partial least squares discriminant analysis (PLS-DA) and orthogonal partial least squares discriminant analysis (OPLS-DA). To discover temporal patterns in longitudinal metabolomics, a multitask learning (MTL) method employing structural regularization was proposed. The group regularization term of the proposed MTL method enables the selection of a small number of tentative biomarkers while maintaining high prediction accuracy. Meanwhile, the nuclear norm imposed into the regression coefficient accounts for the interrelationship of the metabolomics data obtained on consecutive time points. The effectiveness of the proposed method was demonstrated by comparison study performed on a metabolomics dataset and a simulating dataset. The results showed that a compact set of tentative biomarkers charactering the whole antipyretic process of Qingkailing injection were selected with the proposed method. In addition, the nuclear norm introduced in the new method could help the group norm to improve the method’s recovery ability.

Modeling Physico-Chemical ADMET Endpoints with Multitask Graph Convolutional Networks

Simple physico-chemical properties, like logD, solubility, or melting point, can reveal a great deal about how a compound under development might later behave. These data are typically measured for most compounds in drug discovery projects in a medium throughput fashion. Collecting and assembling all the Bayer in-house data related to these properties allowed us to apply powerful machine learning techniques to predict the outcome of those assays for new compounds. In this paper, we report our finding that, especially for predicting physicochemical ADMET endpoints, a multitask graph convolutional approach appears a highly competitive choice. For seven endpoints of interest, we compared the performance of that approach to fully connected neural networks and different single task models. The new model shows increased predictive performance compared to previous modeling methods and will allow early prioritization of compounds even before they are synthesized. In addition, our model follows the generalized solubility equation without being explicitly trained under this constraint.

Simultaneous Parameter Learning and Bi-clustering for Multi-Response Models

We consider multi-response and multi-task regression models, where the parameter matrix to be estimated is expected to have an unknown grouping structure. The groupings can be along tasks, or features, or both, the last one indicating a bi-cluster or “checkerboard” structure. Discovering this grouping structure along with parameter inference makes sense in several applications, such as multi-response Genome-Wide Association Studies (GWAS). By inferring this additional structure we can obtain valuable information on the underlying data mechanisms (e.g., relationships among genotypes and phenotypes in GWAS). In this paper, we propose two formulations to simultaneously learn the parameter matrix and its group structures, based on convex regularization penalties. We present optimization approaches to solve the resulting problems and provide numerical convergence guarantees. Extensive experiments demonstrate much better clustering quality compared to other methods, and our approaches are also validated on real datasets concerning phenotypes and genotypes of plant varieties.

Quantifying risk factors in medical reports with a context-aware linear model

OBJECTIVE

We seek to quantify the mortality risk associated with mentions of medical concepts in textual electronic health records (EHRs). Recognizing mentions of named entities of relevant types (eg, conditions, symptoms, laboratory tests or behaviors) in text is a well-researched task. However, determining the level of risk associated with them is partly dependent on the textual context in which they appear, which may describe severity, temporal aspects, quantity, etc.

METHODS

To take into account that a given word appearing in the context of different risk factors (medical concepts) can make different contributions toward risk level, we propose a multitask approach, called context-aware linear modeling, which can be applied using appropriately regularized linear regression. To improve the performance for risk factors unseen in training data (eg, rare diseases), we take into account their distributional similarity to other concepts.

RESULTS

The evaluation is based on a corpus of 531 reports from EHRs with 99 376 risk factors rated manually by experts. While context-aware linear modeling significantly outperforms single-task models, taking into account concept similarity further improves performance, reaching the level of human annotators' agreements.

CONCLUSION

Our results show that automatic quantification of risk factors in EHRs can achieve performance comparable to human assessment, and taking into account the multitask structure of the problem and the ability to handle rare concepts is crucial for its accuracy.

Dynamic Field Monitoring Based on Multitask Learning in Sensor Networks

Field monitoring serves as an important supervision tool in a variety of engineering domains. An efficient monitoring would trigger an alarm timely once it detects an out-of-control event by learning the state change from the collected sensor data. However, in practice, multiple sensor data may not be gathered appropriately into a database for some unexpected reasons, such as sensor aging, wireless communication failures, and data reading errors, which leads to a large number of missing data as well as inaccurate or delayed detection, and poses a great challenge for field monitoring in sensor networks. This fact motivates us to develop a multitask-learning based field monitoring method in order to achieve an efficient detection when considerable missing data exist during data acquisition. Specifically, we adopt a log likelihood ratio (LR)-based multivariate cumulative sum (MCUSUM) control chart given spatial correlation among neighboring regions within the monitored field. To deal with the missing data problem, we integrate a multitask learning model into the LR-based MCUSUM control chart in the sensor network. Both simulation and real case studies are conducted to validate our proposed approach and the results show that our approach can achieve an accurate and timely detection for an out-of-control state when a large number of missing data exist in the sensor database. Our model provides an effective field monitoring strategy for engineering applications to accurately and timely detect the products with abnormal quality during production and reduce product losses.