Integrating transformer and autoencoder techniques with spectral graph algorithms for the prediction of scarcely labeled molecular data

Synergistic Integration of Deep Neural Networks and Finite Element Method with Applications of Nonlinear Large Deformation Biomechanics

Patient-specific finite element analysis (FEA) holds great promise in advancing the prognosis of cardiovascular diseases by providing detailed biomechanical insights such as high-fidelity stress and deformation on a patient-specific basis. Albeit feasible, FEA that incorporates three-dimensional, complex patient-specific geometry can be time-consuming and unsuitable for time-sensitive clinical applications. To mitigate this challenge, machine learning (ML) models, e.g., deep neural networks (DNNs), have been increasingly utilized as potential alternatives to finite element method (FEM) for biomechanical analysis. So far, efforts have been made in two main directions: (1) learning the input-to-output mapping of traditional FEM solvers and replacing FEM with data-driven ML surrogate models; (2) solving equilibrium equations using physics-informed loss functions of neural networks. While these two existing strategies have shown improved performance in terms of speed or scalability, ML models have not yet provided practical advantages over traditional FEM due to generalization issues. This has led us to the question: instead of abandoning or replacing the traditional FEM framework that can reliably solve biomechanical problems, can we integrate FEM and DNNs to enhance performance? In this study, we propose a synergistic integration of DNNs and FEM to overcome their individual limitations. Using biomechanical analysis of the human aorta as the test bed, we demonstrated two novel integrative strategies in forward and inverse problems. For the forward problem, we developed DNNs with state-of-the-art architectures to predict a nodal displacement field, and this initial DNN solution was then updated by a FEM-based refinement process, yielding a fast and accurate computing framework. For the inverse problem of heterogeneous material parameter identification, our method employs DNN as a regularizer of the spatial distribution of material parameters, aiding the optimizer in locating the optimal solution. In our demonstrative examples, despite that the DNN-only forward models yielded small displacement errors in most test cases; stress errors were considerably large, and for some test cases, the peak stress errors were greater than 50%. Our DNN-FEM integration eliminated these non-negligible errors in DNN-only models and was magnitudes faster than the FEM-only approach. Additionally, compared to FEM-only inverse method with errors greater than 50%, our DNN-FEM inverse approach significantly improved the parameter identification accuracy and reduced the errors to less than 1%.

Machine Learning Methods for Small Data Challenges in Molecular Science

Small data are often used in scientific and engineering research due to the presence of various constraints, such as time, cost, ethics, privacy, security, and technical limitations in data acquisition. However, big data have been the focus for the past decade, small data and their challenges have received little attention, even though they are technically more severe in machine learning (ML) and deep learning (DL) studies. Overall, the small data challenge is often compounded by issues, such as data diversity, imputation, noise, imbalance, and high-dimensionality. Fortunately, the current big data era is characterized by technological breakthroughs in ML, DL, and artificial intelligence (AI), which enable data-driven scientific discovery, and many advanced ML and DL technologies developed for big data have inadvertently provided solutions for small data problems. As a result, significant progress has been made in ML and DL for small data challenges in the past decade. In this review, we summarize and analyze several emerging potential solutions to small data challenges in molecular science, including chemical and biological sciences. We review both basic machine learning algorithms, such as linear regression, logistic regression (LR), k-nearest neighbor (KNN), support vector machine (SVM), kernel learning (KL), random forest (RF), and gradient boosting trees (GBT), and more advanced techniques, including artificial neural network (ANN), convolutional neural network (CNN), U-Net, graph neural network (GNN), Generative Adversarial Network (GAN), long short-term memory (LSTM), autoencoder, transformer, transfer learning, active learning, graph-based semi-supervised learning, combining deep learning with traditional machine learning, and physical model-based data augmentation. We also briefly discuss the latest advances in these methods. Finally, we conclude the survey with a discussion of promising trends in small data challenges in molecular science.

Synergistic Integration of Deep Neural Networks and Finite Element Method with Applications for Biomechanical Analysis of Human Aorta

Motivation: Patient-specific finite element analysis (FEA) has the potential to aid in the prognosis of cardiovascular diseases by providing accurate stress and deformation analysis in various scenarios. It is known that patient-specific FEA is time-consuming and unsuitable for time-sensitive clinical applications. To mitigate this challenge, machine learning (ML) techniques, including deep neural networks (DNNs), have been developed to construct fast FEA surrogates. However, due to the data-driven nature of these ML models, they may not generalize well on new data, leading to unacceptable errors.

Methods

We propose a synergistic integration of DNNs and finite element method (FEM) to overcome each other’s limitations. We demonstrated this novel integrative strategy in forward and inverse problems. For the forward problem, we developed DNNs using state-of-the-art architectures, and DNN outputs were then refined by FEM to ensure accuracy. For the inverse problem of heterogeneous material parameter identification, our method employs a DNN as regularization for the inverse analysis process to avoid erroneous material parameter distribution.

Results

We tested our methods on biomechanical analysis of the human aorta. For the forward problem, the DNN-only models yielded acceptable stress errors in majority of test cases; yet, for some test cases that could be out of the training distribution (OOD), the peak stress errors were larger than 50%. The DNN-FEM integration eliminated the large errors for these OOD cases. Moreover, the DNN-FEM integration was magnitudes faster than the FEM-only approach. For the inverse problem, the FEM-only inverse method led to errors larger than 50%, and our DNN-FEM integration significantly improved performance on the inverse problem with errors less than 1%.