-
Highly-scalable, physics-informed GANs for learning solutions of stochastic PDEs
Authors:
Liu Yang,
Sean Treichler,
Thorsten Kurth,
Keno Fischer,
David Barajas-Solano,
Josh Romero,
Valentin Churavy,
Alexandre Tartakovsky,
Michael Houston,
Prabhat,
George Karniadakis
Abstract:
Uncertainty quantification for forward and inverse problems is a central challenge across physical and biomedical disciplines. We address this challenge for the problem of modeling subsurface flow at the Hanford Site by combining stochastic computational models with observational data using physics-informed GAN models. The geographic extent, spatial heterogeneity, and multiple correlation length s…
▽ More
Uncertainty quantification for forward and inverse problems is a central challenge across physical and biomedical disciplines. We address this challenge for the problem of modeling subsurface flow at the Hanford Site by combining stochastic computational models with observational data using physics-informed GAN models. The geographic extent, spatial heterogeneity, and multiple correlation length scales of the Hanford Site require training a computationally intensive GAN model to thousands of dimensions. We develop a hierarchical scheme for exploiting domain parallelism, map discriminators and generators to multiple GPUs, and employ efficient communication schemes to ensure training stability and convergence. We developed a highly optimized implementation of this scheme that scales to 27,500 NVIDIA Volta GPUs and 4584 nodes on the Summit supercomputer with a 93.1% scaling efficiency, achieving peak and sustained half-precision rates of 1228 PF/s and 1207 PF/s.
△ Less
Submitted 28 October, 2019;
originally announced October 2019.
-
Exascale Deep Learning for Scientific Inverse Problems
Authors:
Nouamane Laanait,
Joshua Romero,
Junqi Yin,
M. Todd Young,
Sean Treichler,
Vitalii Starchenko,
Albina Borisevich,
Alex Sergeev,
Michael Matheson
Abstract:
We introduce novel communication strategies in synchronous distributed Deep Learning consisting of decentralized gradient reduction orchestration and computational graph-aware grouping of gradient tensors. These new techniques produce an optimal overlap between computation and communication and result in near-linear scaling (0.93) of distributed training up to 27,600 NVIDIA V100 GPUs on the Summit…
▽ More
We introduce novel communication strategies in synchronous distributed Deep Learning consisting of decentralized gradient reduction orchestration and computational graph-aware grouping of gradient tensors. These new techniques produce an optimal overlap between computation and communication and result in near-linear scaling (0.93) of distributed training up to 27,600 NVIDIA V100 GPUs on the Summit Supercomputer. We demonstrate our gradient reduction techniques in the context of training a Fully Convolutional Neural Network to approximate the solution of a longstanding scientific inverse problem in materials imaging. The efficient distributed training on a dataset size of 0.5 PB, produces a model capable of an atomically-accurate reconstruction of materials, and in the process reaching a peak performance of 2.15(4) EFLOPS$_{16}$.
△ Less
Submitted 24 September, 2019;
originally announced September 2019.
-
Task Bench: A Parameterized Benchmark for Evaluating Parallel Runtime Performance
Authors:
Elliott Slaughter,
Wei Wu,
Yuankun Fu,
Legend Brandenburg,
Nicolai Garcia,
Wilhem Kautz,
Emily Marx,
Kaleb S. Morris,
Wonchan Lee,
Qinglei Cao,
George Bosilca,
Seema Mirchandaney,
Sean Treichler,
Patrick McCormick,
Alex Aiken
Abstract:
We present Task Bench, a parameterized benchmark designed to explore the performance of parallel and distributed programming systems under a variety of application scenarios. Task Bench lowers the barrier to benchmarking multiple programming systems by making the implementation for a given system orthogonal to the benchmarks themselves: every benchmark constructed with Task Bench runs on every Tas…
▽ More
We present Task Bench, a parameterized benchmark designed to explore the performance of parallel and distributed programming systems under a variety of application scenarios. Task Bench lowers the barrier to benchmarking multiple programming systems by making the implementation for a given system orthogonal to the benchmarks themselves: every benchmark constructed with Task Bench runs on every Task Bench implementation. Furthermore, Task Bench's parameterization enables a wide variety of benchmark scenarios that distill the key characteristics of larger applications.
We conduct a comprehensive study with implementations of Task Bench in 15 programming systems on up to 256 Haswell nodes of the Cori supercomputer. We introduce a novel metric, minimum effective task granularity to study the baseline runtime overhead of each system. We show that when running at scale, 100 μs is the smallest granularity that even the most efficient systems can reliably support with current technologies. We also study each system's scalability, ability to hide communication and mitigate load imbalance.
△ Less
Submitted 17 November, 2020; v1 submitted 15 August, 2019;
originally announced August 2019.
-
Exascale Deep Learning for Climate Analytics
Authors:
Thorsten Kurth,
Sean Treichler,
Joshua Romero,
Mayur Mudigonda,
Nathan Luehr,
Everett Phillips,
Ankur Mahesh,
Michael Matheson,
Jack Deslippe,
Massimiliano Fatica,
Prabhat,
Michael Houston
Abstract:
We extract pixel-level masks of extreme weather patterns using variants of Tiramisu and DeepLabv3+ neural networks. We describe improvements to the software frameworks, input pipeline, and the network training algorithms necessary to efficiently scale deep learning on the Piz Daint and Summit systems. The Tiramisu network scales to 5300 P100 GPUs with a sustained throughput of 21.0 PF/s and parall…
▽ More
We extract pixel-level masks of extreme weather patterns using variants of Tiramisu and DeepLabv3+ neural networks. We describe improvements to the software frameworks, input pipeline, and the network training algorithms necessary to efficiently scale deep learning on the Piz Daint and Summit systems. The Tiramisu network scales to 5300 P100 GPUs with a sustained throughput of 21.0 PF/s and parallel efficiency of 79.0%. DeepLabv3+ scales up to 27360 V100 GPUs with a sustained throughput of 325.8 PF/s and a parallel efficiency of 90.7% in single precision. By taking advantage of the FP16 Tensor Cores, a half-precision version of the DeepLabv3+ network achieves a peak and sustained throughput of 1.13 EF/s and 999.0 PF/s respectively.
△ Less
Submitted 3 October, 2018;
originally announced October 2018.
