-
Physics-Informed Neural Networks with Adaptive Localized Artificial Viscosity
Authors:
E. J. R. Coutinho,
M. Dall'Aqua,
L. McClenny,
M. Zhong,
U. Braga-Neto,
E. Gildin
Abstract:
Physics-informed Neural Network (PINN) is a promising tool that has been applied in a variety of physical phenomena described by partial differential equations (PDE). However, it has been observed that PINNs are difficult to train in certain "stiff" problems, which include various nonlinear hyperbolic PDEs that display shocks in their solutions. Recent studies added a diffusion term to the PDE, an…
▽ More
Physics-informed Neural Network (PINN) is a promising tool that has been applied in a variety of physical phenomena described by partial differential equations (PDE). However, it has been observed that PINNs are difficult to train in certain "stiff" problems, which include various nonlinear hyperbolic PDEs that display shocks in their solutions. Recent studies added a diffusion term to the PDE, and an artificial viscosity (AV) value was manually tuned to allow PINNs to solve these problems. In this paper, we propose three approaches to address this problem, none of which rely on an a priori definition of the artificial viscosity value. The first method learns a global AV value, whereas the other two learn localized AV values around the shocks, by means of a parametrized AV map or a residual-based AV map. We applied the proposed methods to the inviscid Burgers equation and the Buckley-Leverett equation, the latter being a classical problem in Petroleum Engineering. The results show that the proposed methods are able to learn both a small AV value and the accurate shock location and improve the approximation error over a nonadaptive global AV alternative method.
△ Less
Submitted 15 March, 2022;
originally announced March 2022.
-
TensorDiffEq: Scalable Multi-GPU Forward and Inverse Solvers for Physics Informed Neural Networks
Authors:
Levi D. McClenny,
Mulugeta A. Haile,
Ulisses M. Braga-Neto
Abstract:
Physics-Informed Neural Networks promise to revolutionize science and engineering practice, by introducing domain-aware deep machine learning models into scientific computation. Several software suites have emerged to make the implementation and usage of these architectures available to the research and industry communities. Here we introduce\linebreak TensorDiffEq, built on Tensorflow 2.x, which…
▽ More
Physics-Informed Neural Networks promise to revolutionize science and engineering practice, by introducing domain-aware deep machine learning models into scientific computation. Several software suites have emerged to make the implementation and usage of these architectures available to the research and industry communities. Here we introduce\linebreak TensorDiffEq, built on Tensorflow 2.x, which presents an intuitive Keras-like interface for problem domain definition, model definition, and solution of forward and inverse problems using physics-aware deep learning methods. TensorDiffEq takes full advantage of Tensorflow 2.x infrastructure for deployment on multiple GPUs, allowing the implementation of large high-dimensional and complex models. Simultaneously, TensorDiffEq supports the Keras API for custom neural network architecture definitions. In the case of smaller or simpler models, the package allows for rapid deployment on smaller-scale CPU platforms with negligible changes to the implementation scripts. We demonstrate the basic usage and capabilities of TensorDiffEq in solving forward, inverse, and data assimilation problems of varying sizes and levels of complexity. The source code is available at https://github.com/tensordiffeq.
△ Less
Submitted 29 March, 2021;
originally announced March 2021.
-
Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism
Authors:
Levi McClenny,
Ulisses Braga-Neto
Abstract:
Physics-Informed Neural Networks (PINNs) have emerged recently as a promising application of deep neural networks to the numerical solution of nonlinear partial differential equations (PDEs). However, it has been recognized that adaptive procedures are needed to force the neural network to fit accurately the stubborn spots in the solution of "stiff" PDEs. In this paper, we propose a fundamentally…
▽ More
Physics-Informed Neural Networks (PINNs) have emerged recently as a promising application of deep neural networks to the numerical solution of nonlinear partial differential equations (PDEs). However, it has been recognized that adaptive procedures are needed to force the neural network to fit accurately the stubborn spots in the solution of "stiff" PDEs. In this paper, we propose a fundamentally new way to train PINNs adaptively, where the adaptation weights are fully trainable and applied to each training point individually, so the neural network learns autonomously which regions of the solution are difficult and is forced to focus on them. The self-adaptation weights specify a soft multiplicative soft attention mask, which is reminiscent of similar mechanisms used in computer vision. The basic idea behind these SA-PINNs is to make the weights increase as the corresponding losses increase, which is accomplished by training the network to simultaneously minimize the losses and maximize the weights. In addition, we show how to build a continuous map of self-adaptive weights using Gaussian Process regression, which allows the use of stochastic gradient descent in problems where conventional gradient descent is not enough to produce accurate solutions. Finally, we derive the Neural Tangent Kernel matrix for SA-PINNs and use it to obtain a heuristic understanding of the effect of the self-adaptive weights on the dynamics of training in the limiting case of infinitely-wide PINNs, which suggests that SA-PINNs work by producing a smooth equalization of the eigenvalues of the NTK matrix corresponding to the different loss terms. In numerical experiments with several linear and nonlinear benchmark problems, the SA-PINN outperformed other state-of-the-art PINN algorithm in L2 error, while using a smaller number of training epochs.
△ Less
Submitted 18 June, 2024; v1 submitted 7 September, 2020;
originally announced September 2020.
-
Deep Multimodal Transfer-Learned Regression in Data-Poor Domains
Authors:
Levi McClenny,
Mulugeta Haile,
Vahid Attari,
Brian Sadler,
Ulisses Braga-Neto,
Raymundo Arroyave
Abstract:
In many real-world applications of deep learning, estimation of a target may rely on various types of input data modes, such as audio-video, image-text, etc. This task can be further complicated by a lack of sufficient data. Here we propose a Deep Multimodal Transfer-Learned Regressor (DMTL-R) for multimodal learning of image and feature data in a deep regression architecture effective at predicti…
▽ More
In many real-world applications of deep learning, estimation of a target may rely on various types of input data modes, such as audio-video, image-text, etc. This task can be further complicated by a lack of sufficient data. Here we propose a Deep Multimodal Transfer-Learned Regressor (DMTL-R) for multimodal learning of image and feature data in a deep regression architecture effective at predicting target parameters in data-poor domains. Our model is capable of fine-tuning a given set of pre-trained CNN weights on a small amount of training image data, while simultaneously conditioning on feature information from a complimentary data mode during network training, yielding more accurate single-target or multi-target regression than can be achieved using the images or the features alone. We present results using phase-field simulation microstructure images with an accompanying set of physical features, using pre-trained weights from various well-known CNN architectures, which demonstrate the efficacy of the proposed multimodal approach.
△ Less
Submitted 16 June, 2020;
originally announced June 2020.
