Showing 1–50 of 64 results for author: Kurth, T

Huge Ensembles Part I: Design of Ensemble Weather Forecasts using Spherical Fourier Neural Operators

Authors: Ankur Mahesh, William Collins, Boris Bonev, Noah Brenowitz, Yair Cohen, Joshua Elms, Peter Harrington, Karthik Kashinath, Thorsten Kurth, Joshua North, Travis OBrien, Michael Pritchard, David Pruitt, Mark Risser, Shashank Subramanian, Jared Willard

Abstract: Studying low-likelihood high-impact extreme weather events in a warming world is a significant and challenging task for current ensemble forecasting systems. While these systems presently use up to 100 members, larger ensembles could enrich the sampling of internal variability. They may capture the long tails associated with climate hazards better than traditional ensemble sizes. Due to computatio… ▽ More Studying low-likelihood high-impact extreme weather events in a warming world is a significant and challenging task for current ensemble forecasting systems. While these systems presently use up to 100 members, larger ensembles could enrich the sampling of internal variability. They may capture the long tails associated with climate hazards better than traditional ensemble sizes. Due to computational constraints, it is infeasible to generate huge ensembles (comprised of 1,000-10,000 members) with traditional, physics-based numerical models. In this two-part paper, we replace traditional numerical simulations with machine learning (ML) to generate hindcasts of huge ensembles. In Part I, we construct an ensemble weather forecasting system based on Spherical Fourier Neural Operators (SFNO), and we discuss important design decisions for constructing such an ensemble. The ensemble represents model uncertainty through perturbed-parameter techniques, and it represents initial condition uncertainty through bred vectors, which sample the fastest growing modes of the forecast. Using the European Centre for Medium-Range Weather Forecasts Integrated Forecasting System (IFS) as a baseline, we develop an evaluation pipeline composed of mean, spectral, and extreme diagnostics. Using large-scale, distributed SFNOs with 1.1 billion learned parameters, we achieve calibrated probabilistic forecasts. As the trajectories of the individual members diverge, the ML ensemble mean spectra degrade with lead time, consistent with physical expectations. However, the individual ensemble members' spectra stay constant with lead time. Therefore, these members simulate realistic weather states, and the ML ensemble thus passes a crucial spectral test in the literature. The IFS and ML ensembles have similar Extreme Forecast Indices, and we show that the ML extreme weather forecasts are reliable and discriminating. △ Less

Submitted 6 August, 2024; originally announced August 2024.

Huge Ensembles Part II: Properties of a Huge Ensemble of Hindcasts Generated with Spherical Fourier Neural Operators

Authors: Ankur Mahesh, William Collins, Boris Bonev, Noah Brenowitz, Yair Cohen, Peter Harrington, Karthik Kashinath, Thorsten Kurth, Joshua North, Travis OBrien, Michael Pritchard, David Pruitt, Mark Risser, Shashank Subramanian, Jared Willard

Abstract: In Part I, we created an ensemble based on Spherical Fourier Neural Operators. As initial condition perturbations, we used bred vectors, and as model perturbations, we used multiple checkpoints trained independently from scratch. Based on diagnostics that assess the ensemble's physical fidelity, our ensemble has comparable performance to operational weather forecasting systems. However, it require… ▽ More In Part I, we created an ensemble based on Spherical Fourier Neural Operators. As initial condition perturbations, we used bred vectors, and as model perturbations, we used multiple checkpoints trained independently from scratch. Based on diagnostics that assess the ensemble's physical fidelity, our ensemble has comparable performance to operational weather forecasting systems. However, it requires several orders of magnitude fewer computational resources. Here in Part II, we generate a huge ensemble (HENS), with 7,424 members initialized each day of summer 2023. We enumerate the technical requirements for running huge ensembles at this scale. HENS precisely samples the tails of the forecast distribution and presents a detailed sampling of internal variability. For extreme climate statistics, HENS samples events 4$σ$ away from the ensemble mean. At each grid cell, HENS improves the skill of the most accurate ensemble member and enhances coverage of possible future trajectories. As a weather forecasting model, HENS issues extreme weather forecasts with better uncertainty quantification. It also reduces the probability of outlier events, in which the verification value lies outside the ensemble forecast distribution. △ Less

Submitted 2 August, 2024; originally announced August 2024.

Coupled Ocean-Atmosphere Dynamics in a Machine Learning Earth System Model

Authors: Chenggong Wang, Michael S. Pritchard, Noah Brenowitz, Yair Cohen, Boris Bonev, Thorsten Kurth, Dale Durran, Jaideep Pathak

Abstract: Seasonal climate forecasts are socioeconomically important for managing the impacts of extreme weather events and for planning in sectors like agriculture and energy. Climate predictability on seasonal timescales is tied to boundary effects of the ocean on the atmosphere and coupled interactions in the ocean-atmosphere system. We present the Ocean-linked-atmosphere (Ola) model, a high-resolution (… ▽ More Seasonal climate forecasts are socioeconomically important for managing the impacts of extreme weather events and for planning in sectors like agriculture and energy. Climate predictability on seasonal timescales is tied to boundary effects of the ocean on the atmosphere and coupled interactions in the ocean-atmosphere system. We present the Ocean-linked-atmosphere (Ola) model, a high-resolution (0.25°) Artificial Intelligence/ Machine Learning (AI/ML) coupled earth-system model which separately models the ocean and atmosphere dynamics using an autoregressive Spherical Fourier Neural Operator architecture, with a view towards enabling fast, accurate, large ensemble forecasts on the seasonal timescale. We find that Ola exhibits learned characteristics of ocean-atmosphere coupled dynamics including tropical oceanic waves with appropriate phase speeds, and an internally generated El Niño/Southern Oscillation (ENSO) having realistic amplitude, geographic structure, and vertical structure within the ocean mixed layer. We present initial evidence of skill in forecasting the ENSO which compares favorably to the SPEAR model of the Geophysical Fluid Dynamics Laboratory. △ Less

Submitted 12 June, 2024; originally announced June 2024.

Neural Operators with Localized Integral and Differential Kernels

Authors: Miguel Liu-Schiaffini, Julius Berner, Boris Bonev, Thorsten Kurth, Kamyar Azizzadenesheli, Anima Anandkumar

Abstract: Neural operators learn mappings between function spaces, which is practical for learning solution operators of PDEs and other scientific modeling applications. Among them, the Fourier neural operator (FNO) is a popular architecture that performs global convolutions in the Fourier space. However, such global operations are often prone to over-smoothing and may fail to capture local details. In cont… ▽ More Neural operators learn mappings between function spaces, which is practical for learning solution operators of PDEs and other scientific modeling applications. Among them, the Fourier neural operator (FNO) is a popular architecture that performs global convolutions in the Fourier space. However, such global operations are often prone to over-smoothing and may fail to capture local details. In contrast, convolutional neural networks (CNN) can capture local features but are limited to training and inference at a single resolution. In this work, we present a principled approach to operator learning that can capture local features under two frameworks by learning differential operators and integral operators with locally supported kernels. Specifically, inspired by stencil methods, we prove that we obtain differential operators under an appropriate scaling of the kernel values of CNNs. To obtain local integral operators, we utilize suitable basis representations for the kernels based on discrete-continuous convolutions. Both these approaches preserve the properties of operator learning and, hence, the ability to predict at any resolution. Adding our layers to FNOs significantly improves their performance, reducing the relative L2-error by 34-72% in our experiments, which include a turbulent 2D Navier-Stokes and the spherical shallow water equations. △ Less

Submitted 8 June, 2024; v1 submitted 26 February, 2024; originally announced February 2024.

Comments: Accepted at 2024 International Conference on Machine Learning

How causal inference concepts can guide research into the effects of climate on infectious diseases

Authors: Laura Andrea Barrero Guevara, Sarah C Kramer, Tobias Kurth, Matthieu Domenech de Cellès

Abstract: A pressing question resulting from global warming is how infectious diseases will be affected by climate change. Answering this question requires research into the effects of weather on the population dynamics of transmission and infection; elucidating these effects, however, has proven difficult due to the challenges of assessing causality from the predominantly observational data available in ep… ▽ More A pressing question resulting from global warming is how infectious diseases will be affected by climate change. Answering this question requires research into the effects of weather on the population dynamics of transmission and infection; elucidating these effects, however, has proven difficult due to the challenges of assessing causality from the predominantly observational data available in epidemiological research. Here, we show how concepts from causal inference -- the sub-field of statistics aiming at inferring causality from data -- can guide that research. Through a series of case studies, we illustrate how such concepts can help assess study design and strategically choose a study's location, evaluate and reduce the risk of bias, and interpret the multifaceted effects of meteorological variables on transmission. More broadly, we argue that interdisciplinary approaches based on explicit causal frameworks are crucial for reliably estimating the effect of weather and accurately predicting the consequences of climate change. △ Less

Submitted 19 February, 2024; originally announced February 2024.

A Practical Probabilistic Benchmark for AI Weather Models

Authors: Noah D. Brenowitz, Yair Cohen, Jaideep Pathak, Ankur Mahesh, Boris Bonev, Thorsten Kurth, Dale R. Durran, Peter Harrington, Michael S. Pritchard

Abstract: Since the weather is chaotic, forecasts aim to predict the distribution of future states rather than make a single prediction. Recently, multiple data driven weather models have emerged claiming breakthroughs in skill. However, these have mostly been benchmarked using deterministic skill scores, and little is known about their probabilistic skill. Unfortunately, it is hard to fairly compare AI wea… ▽ More Since the weather is chaotic, forecasts aim to predict the distribution of future states rather than make a single prediction. Recently, multiple data driven weather models have emerged claiming breakthroughs in skill. However, these have mostly been benchmarked using deterministic skill scores, and little is known about their probabilistic skill. Unfortunately, it is hard to fairly compare AI weather models in a probabilistic sense, since variations in choice of ensemble initialization, definition of state, and noise injection methodology become confounding. Moreover, even obtaining ensemble forecast baselines is a substantial engineering challenge given the data volumes involved. We sidestep both problems by applying a decades-old idea -- lagged ensembles -- whereby an ensemble can be constructed from a moderately-sized library of deterministic forecasts. This allows the first parameter-free intercomparison of leading AI weather models' probabilistic skill against an operational baseline. The results reveal that two leading AI weather models, i.e. GraphCast and Pangu, are tied on the probabilistic CRPS metric even though the former outperforms the latter in deterministic scoring. We also reveal how multiple time-step loss functions, which many data-driven weather models have employed, are counter-productive: they improve deterministic metrics at the cost of increased dissipation, deteriorating probabilistic skill. This is confirmed through ablations applied to a spherical Fourier Neural Operator (SFNO) approach to AI weather forecasting. Separate SFNO ablations modulating effective resolution reveal it has a useful effect on ensemble dispersion relevant to achieving good ensemble calibration. We hope these and forthcoming insights from lagged ensembles can help guide the development of AI weather forecasts and have thus shared the diagnostic code. △ Less

Submitted 27 January, 2024; originally announced January 2024.

Comments: 15 pages, 5 figures

Advancing Parsimonious Deep Learning Weather Prediction using the HEALPix Mesh

Authors: Matthias Karlbauer, Nathaniel Cresswell-Clay, Dale R. Durran, Raul A. Moreno, Thorsten Kurth, Boris Bonev, Noah Brenowitz, Martin V. Butz

Abstract: We present a parsimonious deep learning weather prediction model to forecast seven atmospheric variables with 3-h time resolution for up to one-year lead times on a 110-km global mesh using the Hierarchical Equal Area isoLatitude Pixelization (HEALPix). In comparison to state-of-the-art (SOTA) machine learning (ML) weather forecast models, such as Pangu-Weather and GraphCast, our DLWP-HPX model us… ▽ More We present a parsimonious deep learning weather prediction model to forecast seven atmospheric variables with 3-h time resolution for up to one-year lead times on a 110-km global mesh using the Hierarchical Equal Area isoLatitude Pixelization (HEALPix). In comparison to state-of-the-art (SOTA) machine learning (ML) weather forecast models, such as Pangu-Weather and GraphCast, our DLWP-HPX model uses coarser resolution and far fewer prognostic variables. Yet, at one-week lead times, its skill is only about one day behind both SOTA ML forecast models and the SOTA numerical weather prediction model from the European Centre for Medium-Range Weather Forecasts. We report several improvements in model design, including switching from the cubed sphere to the HEALPix mesh, inverting the channel depth of the U-Net, and introducing gated recurrent units (GRU) on each level of the U-Net hierarchy. The consistent east-west orientation of all cells on the HEALPix mesh facilitates the development of location-invariant convolution kernels that successfully propagate weather patterns across the globe without requiring separate kernels for the polar and equatorial faces of the cube sphere. Without any loss of spectral power after the first two days, the model can be unrolled autoregressively for hundreds of steps into the future to generate realistic states of the atmosphere that respect seasonal trends, as showcased in one-year simulations. △ Less

Submitted 19 June, 2024; v1 submitted 11 September, 2023; originally announced November 2023.

Comments: Submitted to Journal of Advances in Modeling Earth Systems (JAMES)

Journal ref: Journal of Advances in Modeling Earth Systems, 16, e2023MS004021

Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere

Authors: Boris Bonev, Thorsten Kurth, Christian Hundt, Jaideep Pathak, Maximilian Baust, Karthik Kashinath, Anima Anandkumar

Abstract: Fourier Neural Operators (FNOs) have proven to be an efficient and effective method for resolution-independent operator learning in a broad variety of application areas across scientific machine learning. A key reason for their success is their ability to accurately model long-range dependencies in spatio-temporal data by learning global convolutions in a computationally efficient manner. To this… ▽ More Fourier Neural Operators (FNOs) have proven to be an efficient and effective method for resolution-independent operator learning in a broad variety of application areas across scientific machine learning. A key reason for their success is their ability to accurately model long-range dependencies in spatio-temporal data by learning global convolutions in a computationally efficient manner. To this end, FNOs rely on the discrete Fourier transform (DFT), however, DFTs cause visual and spectral artifacts as well as pronounced dissipation when learning operators in spherical coordinates since they incorrectly assume a flat geometry. To overcome this limitation, we generalize FNOs on the sphere, introducing Spherical FNOs (SFNOs) for learning operators on spherical geometries. We apply SFNOs to forecasting atmospheric dynamics, and demonstrate stable auto\-regressive rollouts for a year of simulated time (1,460 steps), while retaining physically plausible dynamics. The SFNO has important implications for machine learning-based simulation of climate dynamics that could eventually help accelerate our response to climate change. △ Less

Submitted 6 June, 2023; originally announced June 2023.

FourCastNet: Accelerating Global High-Resolution Weather Forecasting using Adaptive Fourier Neural Operators

Authors: Thorsten Kurth, Shashank Subramanian, Peter Harrington, Jaideep Pathak, Morteza Mardani, David Hall, Andrea Miele, Karthik Kashinath, Animashree Anandkumar

Abstract: Extreme weather amplified by climate change is causing increasingly devastating impacts across the globe. The current use of physics-based numerical weather prediction (NWP) limits accuracy due to high computational cost and strict time-to-solution limits. We report that a data-driven deep learning Earth system emulator, FourCastNet, can predict global weather and generate medium-range forecasts f… ▽ More Extreme weather amplified by climate change is causing increasingly devastating impacts across the globe. The current use of physics-based numerical weather prediction (NWP) limits accuracy due to high computational cost and strict time-to-solution limits. We report that a data-driven deep learning Earth system emulator, FourCastNet, can predict global weather and generate medium-range forecasts five orders-of-magnitude faster than NWP while approaching state-of-the-art accuracy. FourCast-Net is optimized and scales efficiently on three supercomputing systems: Selene, Perlmutter, and JUWELS Booster up to 3,808 NVIDIA A100 GPUs, attaining 140.8 petaFLOPS in mixed precision (11.9%of peak at that scale). The time-to-solution for training FourCastNet measured on JUWELS Booster on 3,072GPUs is 67.4minutes, resulting in an 80,000times faster time-to-solution relative to state-of-the-art NWP, in inference. FourCastNet produces accurate instantaneous weather predictions for a week in advance, enables enormous ensembles that better capture weather extremes, and supports higher global forecast resolutions. △ Less

Submitted 8 August, 2022; originally announced August 2022.

FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators

Authors: Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, Pedram Hassanzadeh, Karthik Kashinath, Animashree Anandkumar

Abstract: FourCastNet, short for Fourier Forecasting Neural Network, is a global data-driven weather forecasting model that provides accurate short to medium-range global predictions at $0.25^{\circ}$ resolution. FourCastNet accurately forecasts high-resolution, fast-timescale variables such as the surface wind speed, precipitation, and atmospheric water vapor. It has important implications for planning win… ▽ More FourCastNet, short for Fourier Forecasting Neural Network, is a global data-driven weather forecasting model that provides accurate short to medium-range global predictions at $0.25^{\circ}$ resolution. FourCastNet accurately forecasts high-resolution, fast-timescale variables such as the surface wind speed, precipitation, and atmospheric water vapor. It has important implications for planning wind energy resources, predicting extreme weather events such as tropical cyclones, extra-tropical cyclones, and atmospheric rivers. FourCastNet matches the forecasting accuracy of the ECMWF Integrated Forecasting System (IFS), a state-of-the-art Numerical Weather Prediction (NWP) model, at short lead times for large-scale variables, while outperforming IFS for variables with complex fine-scale structure, including precipitation. FourCastNet generates a week-long forecast in less than 2 seconds, orders of magnitude faster than IFS. The speed of FourCastNet enables the creation of rapid and inexpensive large-ensemble forecasts with thousands of ensemble-members for improving probabilistic forecasting. We discuss how data-driven deep learning models such as FourCastNet are a valuable addition to the meteorology toolkit to aid and augment NWP models. △ Less

Submitted 22 February, 2022; originally announced February 2022.

MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systems

Authors: Steven Farrell, Murali Emani, Jacob Balma, Lukas Drescher, Aleksandr Drozd, Andreas Fink, Geoffrey Fox, David Kanter, Thorsten Kurth, Peter Mattson, Dawei Mu, Amit Ruhela, Kento Sato, Koichi Shirahata, Tsuguchika Tabaru, Aristeidis Tsaris, Jan Balewski, Ben Cumming, Takumi Danjo, Jens Domke, Takaaki Fukai, Naoto Fukumoto, Tatsuya Fukushi, Balazs Gerofi, Takumi Honda , et al. (18 additional authors not shown)

Abstract: Scientific communities are increasingly adopting machine learning and deep learning models in their applications to accelerate scientific insights. High performance computing systems are pushing the frontiers of performance with a rich diversity of hardware resources and massive scale-out capabilities. There is a critical need to understand fair and effective benchmarking of machine learning appli… ▽ More Scientific communities are increasingly adopting machine learning and deep learning models in their applications to accelerate scientific insights. High performance computing systems are pushing the frontiers of performance with a rich diversity of hardware resources and massive scale-out capabilities. There is a critical need to understand fair and effective benchmarking of machine learning applications that are representative of real-world scientific use cases. MLPerf is a community-driven standard to benchmark machine learning workloads, focusing on end-to-end performance metrics. In this paper, we introduce MLPerf HPC, a benchmark suite of large-scale scientific machine learning training applications driven by the MLCommons Association. We present the results from the first submission round, including a diverse set of some of the world's largest HPC systems. We develop a systematic framework for their joint analysis and compare them in terms of data staging, algorithmic convergence, and compute performance. As a result, we gain a quantitative understanding of optimizations on different subsystems such as staging and on-node loading of data, compute-unit utilization, and communication scheduling, enabling overall $>10 \times$ (end-to-end) performance improvements through system scaling. Notably, our analysis shows a scale-dependent interplay between the dataset size, a system's memory hierarchy, and training convergence that underlines the importance of near-compute storage. To overcome the data-parallel scalability challenge at large batch sizes, we discuss specific learning techniques and hybrid data-and-model parallelism that are effective on large systems. We conclude by characterizing each benchmark with respect to low-level memory, I/O, and network behavior to parameterize extended roofline performance models in future rounds. △ Less

Submitted 26 October, 2021; v1 submitted 21 October, 2021; originally announced October 2021.

IMPECCABLE: Integrated Modeling PipelinE for COVID Cure by Assessing Better LEads

Authors: Aymen Al Saadi, Dario Alfe, Yadu Babuji, Agastya Bhati, Ben Blaiszik, Thomas Brettin, Kyle Chard, Ryan Chard, Peter Coveney, Anda Trifan, Alex Brace, Austin Clyde, Ian Foster, Tom Gibbs, Shantenu Jha, Kristopher Keipert, Thorsten Kurth, Dieter Kranzlmüller, Hyungro Lee, Zhuozhao Li, Heng Ma, Andre Merzky, Gerald Mathias, Alexander Partin, Junqi Yin , et al. (11 additional authors not shown)

Abstract: The drug discovery process currently employed in the pharmaceutical industry typically requires about 10 years and $2-3 billion to deliver one new drug. This is both too expensive and too slow, especially in emergencies like the COVID-19 pandemic. In silicomethodologies need to be improved to better select lead compounds that can proceed to later stages of the drug discovery protocol accelerating… ▽ More The drug discovery process currently employed in the pharmaceutical industry typically requires about 10 years and $2-3 billion to deliver one new drug. This is both too expensive and too slow, especially in emergencies like the COVID-19 pandemic. In silicomethodologies need to be improved to better select lead compounds that can proceed to later stages of the drug discovery protocol accelerating the entire process. No single methodological approach can achieve the necessary accuracy with required efficiency. Here we describe multiple algorithmic innovations to overcome this fundamental limitation, development and deployment of computational infrastructure at scale integrates multiple artificial intelligence and simulation-based approaches. Three measures of performance are:(i) throughput, the number of ligands per unit time; (ii) scientific performance, the number of effective ligands sampled per unit time and (iii) peak performance, in flop/s. The capabilities outlined here have been used in production for several months as the workhorse of the computational infrastructure to support the capabilities of the US-DOE National Virtual Biotechnology Laboratory in combination with resources from the EU Centre of Excellence in Computational Biomedicine. △ Less

Submitted 13 October, 2020; originally announced October 2020.

Using Machine Learning to Augment Coarse-Grid Computational Fluid Dynamics Simulations

Authors: Jaideep Pathak, Mustafa Mustafa, Karthik Kashinath, Emmanuel Motheau, Thorsten Kurth, Marcus Day

Abstract: Simulation of turbulent flows at high Reynolds number is a computationally challenging task relevant to a large number of engineering and scientific applications in diverse fields such as climate science, aerodynamics, and combustion. Turbulent flows are typically modeled by the Navier-Stokes equations. Direct Numerical Simulation (DNS) of the Navier-Stokes equations with sufficient numerical reso… ▽ More Simulation of turbulent flows at high Reynolds number is a computationally challenging task relevant to a large number of engineering and scientific applications in diverse fields such as climate science, aerodynamics, and combustion. Turbulent flows are typically modeled by the Navier-Stokes equations. Direct Numerical Simulation (DNS) of the Navier-Stokes equations with sufficient numerical resolution to capture all the relevant scales of the turbulent motions can be prohibitively expensive. Simulation at lower-resolution on a coarse-grid introduces significant errors. We introduce a machine learning (ML) technique based on a deep neural network architecture that corrects the numerical errors induced by a coarse-grid simulation of turbulent flows at high-Reynolds numbers, while simultaneously recovering an estimate of the high-resolution fields. Our proposed simulation strategy is a hybrid ML-PDE solver that is capable of obtaining a meaningful high-resolution solution trajectory while solving the system PDE at a lower resolution. The approach has the potential to dramatically reduce the expense of turbulent flow simulations. As a proof-of-concept, we demonstrate our ML-PDE strategy on a two-dimensional turbulent (Rayleigh Number $Ra=10^9$) Rayleigh-Bénard Convection (RBC) problem. △ Less

Submitted 3 October, 2020; v1 submitted 30 September, 2020; originally announced October 2020.

Comments: Corrected typographical errors in the previous version related to the incorrectly formatted accented character "é" appearing in various places in the manuscript

Hierarchical Roofline Performance Analysis for Deep Learning Applications

Authors: Charlene Yang, Yunsong Wang, Steven Farrell, Thorsten Kurth, Samuel Williams

Abstract: This paper presents a practical methodology for collecting performance data necessary to conduct hierarchical Roofline analysis on NVIDIA GPUs. It discusses the extension of the Empirical Roofline Toolkit for broader support of a range of data precisions and Tensor Core support and introduces a Nsight Compute based method to accurately collect application performance information. This methodology… ▽ More This paper presents a practical methodology for collecting performance data necessary to conduct hierarchical Roofline analysis on NVIDIA GPUs. It discusses the extension of the Empirical Roofline Toolkit for broader support of a range of data precisions and Tensor Core support and introduces a Nsight Compute based method to accurately collect application performance information. This methodology allows for automated machine characterization and application characterization for Roofline analysis across the entire memory hierarchy on NVIDIA GPUs, and it is validated by a complex deep learning application used for climate image segmentation. We use two versions of the code, in TensorFlow and PyTorch respectively, to demonstrate the use and effectiveness of this methodology. We highlight how the application utilizes the compute and memory capabilities on the GPU and how the implementation and performance differ in two deep learning frameworks. △ Less

Submitted 24 November, 2020; v1 submitted 11 September, 2020; originally announced September 2020.

Comments: 9 pages

Time-Based Roofline for Deep Learning Performance Analysis

Authors: Yunsong Wang, Charlene Yang, Steven Farrell, Yan Zhang, Thorsten Kurth, Samuel Williams

Abstract: Deep learning applications are usually very compute-intensive and require a long run time for training and inference. This has been tackled by researchers from both hardware and software sides, and in this paper, we propose a Roofline-based approach to performance analysis to facilitate the optimization of these applications. This approach is an extension of the Roofline model widely used in tradi… ▽ More Deep learning applications are usually very compute-intensive and require a long run time for training and inference. This has been tackled by researchers from both hardware and software sides, and in this paper, we propose a Roofline-based approach to performance analysis to facilitate the optimization of these applications. This approach is an extension of the Roofline model widely used in traditional high-performance computing applications, and it incorporates both compute/bandwidth complexity and run time in its formulae to provide insights into deep learning-specific characteristics. We take two sets of representative kernels, 2D convolution and long short-term memory, to validate and demonstrate the use of this new approach, and investigate how arithmetic intensity, cache locality, auto-tuning, kernel launch overhead, and Tensor Core usage can affect performance. Compared to the common ad-hoc approach, this study helps form a more systematic way to analyze code performance and identify optimization opportunities for deep learning applications. △ Less

Submitted 22 September, 2020; v1 submitted 9 September, 2020; originally announced September 2020.

Comments: 9 pages

$F_K / F_π$ from Möbius domain-wall fermions solved on gradient-flowed HISQ ensembles

Authors: Nolan Miller, Henry Monge-Camacho, Chia Cheng Chang, Ben Hörz, Enrico Rinaldi, Dean Howarth, Evan Berkowitz, David A. Brantley, Arjun Singh Gambhir, Christopher Körber, Christopher J. Monahan, M. A. Clark, Bálint Joó, Thorsten Kurth, Amy Nicholson, Kostas Orginos, Pavlos Vranas, André Walker-Loud

Abstract: We report the results of a lattice quantum chromodynamics calculation of $F_K/F_π$ using Möbius domain-wall fermions computed on gradient-flowed $N_f=2+1+1$ highly-improved staggered quark (HISQ) ensembles. The calculation is performed with five values of the pion mass ranging from $130 \lesssim m_π\lesssim 400$ MeV, four lattice spacings of $a\sim 0.15, 0.12, 0.09$ and $0.06$ fm and multiple valu… ▽ More We report the results of a lattice quantum chromodynamics calculation of $F_K/F_π$ using Möbius domain-wall fermions computed on gradient-flowed $N_f=2+1+1$ highly-improved staggered quark (HISQ) ensembles. The calculation is performed with five values of the pion mass ranging from $130 \lesssim m_π\lesssim 400$ MeV, four lattice spacings of $a\sim 0.15, 0.12, 0.09$ and $0.06$ fm and multiple values of the lattice volume. The interpolation/extrapolation to the physical pion and kaon mass point, the continuum, and infinite volume limits are performed with a variety of different extrapolation functions utilizing both the relevant mixed-action effective field theory expressions as well as discretization-enhanced continuum chiral perturbation theory formulas. We find that the $a\sim0.06$ fm ensemble is helpful, but not necessary to achieve a subpercent determination of $F_K/F_π$. We also include an estimate of the strong isospin breaking corrections and arrive at a final result of $F_{K^\pm}/F_{π^\pm} = 1.1942(45)$ with all sources of statistical and systematic uncertainty included. This is consistent with the Flavour Lattice Averaging Group average value, providing an important benchmark for our lattice action. Combining our result with experimental measurements of the pion and kaon leptonic decays leads to a determination of $|V_{us}|/|V_{ud}| = 0.2311(10)$. △ Less

Submitted 3 September, 2020; v1 submitted 10 May, 2020; originally announced May 2020.

Comments: v3: published version; v2: version submitted to journal; v1: 26 pages including 13 figures, appendices, and references. See https://github.com/callat-qcd/project_fkfpi for the analysis and data

Report number: LLNL-JRNL-809712, RIKEN-iTHEMS-Report-20, JLAB-THY-20-3192

Journal ref: Phys. Rev. D 102, 034507 (2020)

Directed Acyclic Graphs and causal thinking in clinical risk prediction modeling

Authors: Marco Piccininni, Stefan Konigorski, Jessica L Rohmann, Tobias Kurth

Abstract: Background: In epidemiology, causal inference and prediction modeling methodologies have been historically distinct. Directed Acyclic Graphs (DAGs) are used to model a priori causal assumptions and inform variable selection strategies for causal questions. Although tools originally designed for prediction are finding applications in causal inference, the counterpart has remained largely unexplored… ▽ More Background: In epidemiology, causal inference and prediction modeling methodologies have been historically distinct. Directed Acyclic Graphs (DAGs) are used to model a priori causal assumptions and inform variable selection strategies for causal questions. Although tools originally designed for prediction are finding applications in causal inference, the counterpart has remained largely unexplored. The aim of this theoretical and simulation-based study is to assess the potential benefit of using DAGs in clinical risk prediction modeling. Methods and Findings: We explore how incorporating knowledge about the underlying causal structure can provide insights about the transportability of diagnostic clinical risk prediction models to different settings. A single-predictor model in the causal direction is likely to have better transportability than one in the anticausal direction. We further probe whether causal knowledge can be used to improve predictor selection. We empirically show that the Markov Blanket, the set of variables including the parents, children, and parents of the children of the outcome node in a DAG, is the optimal set of predictors for that outcome. Conclusions: Our findings challenge the generally accepted notion that a change in the distribution of the predictors does not affect diagnostic clinical risk prediction model calibration if the predictors are properly included in the model. Furthermore, using DAGs to identify Markov Blanket variables may be a useful, efficient strategy to select predictors in clinical risk prediction models if strong knowledge of the underlying causal structure exists or can be learned. △ Less

Submitted 21 February, 2020; originally announced February 2020.

Comments: 22 pages, 3 figures, 1 table

Journal ref: BMC Med Res Methodol 20, 179 (2020)

Lattice QCD Determination of $g_A$

Authors: André Walker-Loud, Evan Berkowitz, David A. Brantley, Arjun Gambhir, Pavlos Vranas, Chris Bouchard, Chia Cheng Chang, M. A. Clark, Nicolas Garron, Bálint Joó, Thorsten Kurth, Henry Monge-Camacho, Amy Nicholson, Christopher J. Monahan, Kostas Orginos, Enrico Rinaldi

Abstract: The nucleon axial coupling, $g_A$, is a fundamental property of protons and neutrons, dictating the strength with which the weak axial current of the Standard Model couples to nucleons, and hence, the lifetime of a free neutron. The prominence of $g_A$ in nuclear physics has made it a benchmark quantity with which to calibrate lattice QCD calculations of nucleon structure and more complex calculat… ▽ More The nucleon axial coupling, $g_A$, is a fundamental property of protons and neutrons, dictating the strength with which the weak axial current of the Standard Model couples to nucleons, and hence, the lifetime of a free neutron. The prominence of $g_A$ in nuclear physics has made it a benchmark quantity with which to calibrate lattice QCD calculations of nucleon structure and more complex calculations of electroweak matrix elements in one and few nucleon systems. There were a number of significant challenges in determining $g_A$, notably the notorious exponentially-bad signal-to-noise problem and the requirement for hundreds of thousands of stochastic samples, that rendered this goal more difficult to obtain than originally thought. I will describe the use of an unconventional computation method, coupled with "ludicrously'" fast GPU code, access to publicly available lattice QCD configurations from MILC and access to leadership computing that have allowed these challenges to be overcome resulting in a determination of $g_A$ with 1% precision and all sources of systematic uncertainty controlled. I will discuss the implications of these results for the convergence of $SU(2)$ Chiral Perturbation theory for nucleons, as well as prospects for further improvements to $g_A$ (sub-percent precision, for which we have preliminary results) which is part of a more comprehensive application of lattice QCD to nuclear physics. This is particularly exciting in light of the new CORAL supercomputers coming online, Sierra and Summit, for which our lattice QCD codes achieve a machine-to-machine speed up over Titan of an order of magnitude. △ Less

Submitted 17 December, 2019; originally announced December 2019.

Comments: Plenary presentation at The 9th International workshop on Chiral Dynamics

Report number: RIKEN-iTHEMS-Report-19, LLNL-PROC-800060

Journal ref: POS(CD2018)020

Highly-scalable, physics-informed GANs for learning solutions of stochastic PDEs

Authors: Liu Yang, Sean Treichler, Thorsten Kurth, Keno Fischer, David Barajas-Solano, Josh Romero, Valentin Churavy, Alexandre Tartakovsky, Michael Houston, Prabhat, George Karniadakis

Abstract: Uncertainty quantification for forward and inverse problems is a central challenge across physical and biomedical disciplines. We address this challenge for the problem of modeling subsurface flow at the Hanford Site by combining stochastic computational models with observational data using physics-informed GAN models. The geographic extent, spatial heterogeneity, and multiple correlation length s… ▽ More Uncertainty quantification for forward and inverse problems is a central challenge across physical and biomedical disciplines. We address this challenge for the problem of modeling subsurface flow at the Hanford Site by combining stochastic computational models with observational data using physics-informed GAN models. The geographic extent, spatial heterogeneity, and multiple correlation length scales of the Hanford Site require training a computationally intensive GAN model to thousands of dimensions. We develop a hierarchical scheme for exploiting domain parallelism, map discriminators and generators to multiple GPUs, and employ efficient communication schemes to ensure training stability and convergence. We developed a highly optimized implementation of this scheme that scales to 27,500 NVIDIA Volta GPUs and 4584 nodes on the Summit supercomputer with a 93.1% scaling efficiency, achieving peak and sustained half-precision rates of 1228 PF/s and 1207 PF/s. △ Less

Submitted 28 October, 2019; originally announced October 2019.

Comments: 3rd Deep Learning on Supercomputers Workshop (DLS) at SC19

The Stochastic Feynman-Hellmann Method

Authors: Arjun Singh Gambhir, Evan Berkowitz, David Brantley, Chia Cheng Chang, M. A. Clark, Thorsten Kurth, Chris Monahan, Amy Nicholson, Pavlos Vranas, André Walker-Loud

Abstract: The Feynman-Hellmann method, as implemented by Bouchard et al. [1612.06963], was recently employed successfully to determine the nucleon axial charge. A limitation of the method was the restriction to a single operator and a single momentum during the computation of each "Feynman- Hellmann" propagator. By using stochastic techniques to estimate the all-to-all propagator, we relax this constraint a… ▽ More The Feynman-Hellmann method, as implemented by Bouchard et al. [1612.06963], was recently employed successfully to determine the nucleon axial charge. A limitation of the method was the restriction to a single operator and a single momentum during the computation of each "Feynman- Hellmann" propagator. By using stochastic techniques to estimate the all-to-all propagator, we relax this constraint and demonstrate the successful implementation of this new method. We show reproduction of the axial charge on a test ensemble and non-zero momentum transfer points of the axial and vector form factors. △ Less

Submitted 8 May, 2019; originally announced May 2019.

Comments: 7 pages, 5 figures, Proceedings for The 36th International Symposium on Lattice Field Theory

Report number: INT-PUB-19-003

Journal ref: PoS(LATTICE2018)126

Short Range Operator Contributions to $0νββ$ decay from LQCD

Authors: Henry Monge-Camacho, Evan Berkowitz, David Brantley, Chia Cheng Chang, M. A. Clark, Arjun Gambhir, Nicolas Garrón, Bálint Joó, Thorsten Kurth, Amy Nicholson, Enrico Rinaldi, Brian Tiburzi, Pavlos Vranas, André Walker-Loud

Abstract: The search for neutrinoless double beta decay of nuclei is believed to be one of the most promising means to search for new physics. Observation of this very rare nuclear process, which violates Lepton Number conservation, would imply the neutrino sector has a Majorana mass component and may also provide an explanation for the universe matter-antimatter asymmetry of the universe. In the case where… ▽ More The search for neutrinoless double beta decay of nuclei is believed to be one of the most promising means to search for new physics. Observation of this very rare nuclear process, which violates Lepton Number conservation, would imply the neutrino sector has a Majorana mass component and may also provide an explanation for the universe matter-antimatter asymmetry of the universe. In the case where a heavy intermediate particle is exchanged in this process, QCD contributions from short range interactions become relevant and the calculation of matrix elements with four-quark operators becomes necessary. In these proceedings we will discuss our current progress in the calculation of these four-quark operators from LQCD. △ Less

Submitted 26 April, 2019; originally announced April 2019.

Comments: Proceedings of the 36th Annual International Symposium on Lattice Field Theory, 22-28 July, 2018, Michigan State University, East Lansing, Michigan

Report number: RBRC-1297,LLNL-PROC-772237

Progress in Multibaryon Spectroscopy

Authors: Evan Berkowitz, David Brantley, Kenneth McElvain, André Walker-Loud, Chia Cheng Chang, M. A. Clark, Thorsten Kurth, Bálint Joó, Henry Monge-Camacho, Amy Nicholson, Enrico Rinaldi, Pavlos Vranas

Abstract: Anchoring the nuclear interaction in QCD is a long-outstanding problem in nuclear physics. While the lattice community has made enormous progress in mesonic physics and single nucleon physics, continuum-limit physical-point multi-nucleon physics has remained out of reach. I will review CalLat's strategy for multi-nucleon spectroscopy and our latest results. Anchoring the nuclear interaction in QCD is a long-outstanding problem in nuclear physics. While the lattice community has made enormous progress in mesonic physics and single nucleon physics, continuum-limit physical-point multi-nucleon physics has remained out of reach. I will review CalLat's strategy for multi-nucleon spectroscopy and our latest results. △ Less

Submitted 25 February, 2019; originally announced February 2019.

Comments: Proceedings of the 36th Annual International Symposium on Lattice Field Theory, 22-28 July, 2018, Michigan State University, East Lansing, Michigan

Report number: RBRC-1299, RIKEN-iTHEMS-Report-19

Symmetries and Interactions from Lattice QCD

Authors: A. Nicholson, E. Berkowitz, H. Monge-Camacho, D. Brantley, N. Garron, C. C. Chang, E. Rinaldi, C. Monahan, C. Bouchard, M. A. Clark, B. Joo, T. Kurth, B. C. Tiburzi, P. Vranas, A. Walker-Loud

Abstract: Precision experimental tests of the Standard Model of particle physics (SM) are one of our best hopes for discovering what new physics lies beyond the SM (BSM). Key in the search for new physics is the connection between theory and experiment. Forging this connection for searches involving low-energy hadronic or nuclear environments requires the use of a non-perturbative theoretical tool, lattice… ▽ More Precision experimental tests of the Standard Model of particle physics (SM) are one of our best hopes for discovering what new physics lies beyond the SM (BSM). Key in the search for new physics is the connection between theory and experiment. Forging this connection for searches involving low-energy hadronic or nuclear environments requires the use of a non-perturbative theoretical tool, lattice QCD. We present two recent lattice QCD calculations by the CalLat collaboration relevant for new physics searches: the nucleon axial coupling, $g_A$, whose precise value as predicted by the SM could help point to new physics contributions to the so-called "neutron lifetime puzzle", and hadronic matrix elements of short-ranged operators relevant for neutrinoless double beta decay searches. △ Less

Submitted 28 December, 2018; originally announced December 2018.

Comments: Plenary talk presented CIPANP2018. 11 pages, 3 figures

Report number: CIPANP2018-Nicholson, LLNL-CONF-764382, RBRC-1296, RIKEN-iTHEMS-Report-18, INT-PUB-18-063

Exascale Deep Learning for Climate Analytics

Authors: Thorsten Kurth, Sean Treichler, Joshua Romero, Mayur Mudigonda, Nathan Luehr, Everett Phillips, Ankur Mahesh, Michael Matheson, Jack Deslippe, Massimiliano Fatica, Prabhat, Michael Houston

Abstract: We extract pixel-level masks of extreme weather patterns using variants of Tiramisu and DeepLabv3+ neural networks. We describe improvements to the software frameworks, input pipeline, and the network training algorithms necessary to efficiently scale deep learning on the Piz Daint and Summit systems. The Tiramisu network scales to 5300 P100 GPUs with a sustained throughput of 21.0 PF/s and parall… ▽ More We extract pixel-level masks of extreme weather patterns using variants of Tiramisu and DeepLabv3+ neural networks. We describe improvements to the software frameworks, input pipeline, and the network training algorithms necessary to efficiently scale deep learning on the Piz Daint and Summit systems. The Tiramisu network scales to 5300 P100 GPUs with a sustained throughput of 21.0 PF/s and parallel efficiency of 79.0%. DeepLabv3+ scales up to 27360 V100 GPUs with a sustained throughput of 325.8 PF/s and a parallel efficiency of 90.7% in single precision. By taking advantage of the FP16 Tensor Cores, a half-precision version of the DeepLabv3+ network achieves a peak and sustained throughput of 1.13 EF/s and 999.0 PF/s respectively. △ Less

Submitted 3 October, 2018; originally announced October 2018.

Comments: 12 pages, 5 tables, 4, figures, Super Computing Conference November 11-16, 2018, Dallas, TX, USA

Simulating the weak death of the neutron in a femtoscale universe with near-Exascale computing

Authors: Evan Berkowitz, M. A. Clark, Arjun Gambhir, Ken McElvain, Amy Nicholson, Enrico Rinaldi, Pavlos Vranas, André Walker-Loud, Chia Cheng Chang, Bálint Joó, Thorsten Kurth, Kostas Orginos

Abstract: The fundamental particle theory called Quantum Chromodynamics (QCD) dictates everything about protons and neutrons, from their intrinsic properties to interactions that bind them into atomic nuclei. Quantities that cannot be fully resolved through experiment, such as the neutron lifetime (whose precise value is important for the existence of light-atomic elements that make the sun shine and life p… ▽ More The fundamental particle theory called Quantum Chromodynamics (QCD) dictates everything about protons and neutrons, from their intrinsic properties to interactions that bind them into atomic nuclei. Quantities that cannot be fully resolved through experiment, such as the neutron lifetime (whose precise value is important for the existence of light-atomic elements that make the sun shine and life possible), may be understood through numerical solutions to QCD. We directly solve QCD using Lattice Gauge Theory and calculate nuclear observables such as neutron lifetime. We have developed an improved algorithm that exponentially decreases the time-to solution and applied it on the new CORAL supercomputers, Sierra and Summit. We use run-time autotuning to distribute GPU resources, achieving 20% performance at low node count. We also developed optimal application mapping through a job manager, which allows CPU and GPU jobs to be interleaved, yielding 15% of peak performance when deployed across large fractions of CORAL. △ Less

Submitted 10 October, 2018; v1 submitted 3 October, 2018; originally announced October 2018.

Comments: 2018 Gordon Bell Finalist: 9 pages, 9 figures; v2: fixed 2 typos and appended acknowledgements

Report number: LLNL-JRNL-749850, RIKEN-iTHEMS-Report-18 ACM Class: C.1.4; D.1.3

Journal ref: Supercomputing 2018, pp. 697-705

A percent-level determination of the nucleon axial coupling from Quantum Chromodynamics

Authors: Chia Cheng Chang, Amy Nicholson, Enrico Rinaldi, Evan Berkowitz, Nicolas Garron, David A. Brantley, Henry Monge-Camacho, Christopher J. Monahan, Chris Bouchard, M. A. Clark, Bálint Joó, Thorsten Kurth, Kostas Orginos, Pavlos Vranas, André Walker-Loud

Abstract: The $\textit{axial coupling of the nucleon}$, $g_A$, is the strength of its coupling to the $\textit{weak}$ axial current of the Standard Model of particle physics, in much the same way as the electric charge is the strength of the coupling to the electromagnetic current. This axial coupling dictates the rate at which neutrons decay to protons, the strength of the attractive long-range force betwe… ▽ More The $\textit{axial coupling of the nucleon}$, $g_A$, is the strength of its coupling to the $\textit{weak}$ axial current of the Standard Model of particle physics, in much the same way as the electric charge is the strength of the coupling to the electromagnetic current. This axial coupling dictates the rate at which neutrons decay to protons, the strength of the attractive long-range force between nucleons and other features of nuclear physics. Precision tests of the Standard Model in nuclear environments require a quantitative understanding of nuclear physics rooted in Quantum Chromodynamics, a pillar of the Standard Model. The prominence of $g_A$ makes it a benchmark quantity to determine theoretically - a difficult task because quantum chromodynamics is non-perturbative, precluding known analytical methods. Lattice Quantum Chromodynamics provides a rigorous, non-perturbative definition of quantum chromodynamics that can be implemented numerically. It has been estimated that a precision of two percent would be possible by 2020 if two challenges are overcome: contamination of $g_A$ from excited states must be controlled in the calculations and statistical precision must be improved markedly. Here we report a calculation of $g_A^{QCD} = 1.271\pm0.013$, using an unconventional method inspired by the Feynman-Hellmann theorem that overcomes these challenges. △ Less

Submitted 30 May, 2018; originally announced May 2018.

Comments: Published in Nature. 46 pages total: Main text 4 pages, Extended Data 8 pages, Supplemental 34 pages. Supporting data and code at https://github.com/callat-qcd/project_gA or https://zenodo.org/record/1241374

Report number: BNL-203631-2018-JAAM, INT-PUB-18-021, LLNL-JRNL-747003, RBRC-1283, RIKEN-iTHEMS-Report-18, LTH 1166

Journal ref: Nature 558, 91-94 (2018)

Heavy physics contributions to neutrinoless double beta decay from QCD

Authors: A. Nicholson, E. Berkowitz, H. Monge-Camacho, D. Brantley, N. Garron, C. C. Chang, E. Rinaldi, M. A. Clark, B. Joo, T. Kurth, B. Tiburzi, P. Vranas, A. Walker-Loud

Abstract: Observation of neutrinoless double beta decay, a lepton number violating process that has been proposed to clarify the nature of neutrino masses, has spawned an enormous world-wide experimental effort. Relating nuclear decay rates to high-energy, beyond the Standard Model (BSM) physics requires detailed knowledge of non-perturbative QCD effects. Using lattice QCD, we compute the necessary matrix e… ▽ More Observation of neutrinoless double beta decay, a lepton number violating process that has been proposed to clarify the nature of neutrino masses, has spawned an enormous world-wide experimental effort. Relating nuclear decay rates to high-energy, beyond the Standard Model (BSM) physics requires detailed knowledge of non-perturbative QCD effects. Using lattice QCD, we compute the necessary matrix elements of short-range operators, which arise due to heavy BSM mediators, that contribute to this decay via the leading order $π^- \to π^+$ exchange diagrams. Utilizing our result and taking advantage of effective field theory methods will allow for model-independent calculations of the relevant two-nucleon decay, which may then be used as input for nuclear many-body calculations of the relevant experimental decays. Contributions from short-range operators may prove to be equally important to, or even more important than, those from long-range Majorana neutrino exchange. △ Less

Submitted 1 November, 2018; v1 submitted 7 May, 2018; originally announced May 2018.

Comments: Published version. Corrected missing term in chiral expansion, added supplemental material, updated references. The Jupyter notebook (DOI:10.5281/zenodo.1243313) accompanying this work can be found on github https://github.com/callat-qcd/project_0vbb

Report number: LLNL-JRNL-751220, RBRC-1266, RIKEN-iTHEMS-Report-18, BNL-209118-2018-JAAM

Journal ref: Phys. Rev. Lett. 121, 172501 (2018)

Scaling GRPC Tensorflow on 512 nodes of Cori Supercomputer

Authors: Amrita Mathuriya, Thorsten Kurth, Vivek Rane, Mustafa Mustafa, Lei Shao, Debbie Bard, Prabhat, Victor W Lee

Abstract: We explore scaling of the standard distributed Tensorflow with GRPC primitives on up to 512 Intel Xeon Phi (KNL) nodes of Cori supercomputer with synchronous stochastic gradient descent (SGD), and identify causes of scaling inefficiency at higher node counts. To our knowledge, this is the first exploration of distributed GRPC Tensorflow scalability on a HPC supercomputer at such large scale with s… ▽ More We explore scaling of the standard distributed Tensorflow with GRPC primitives on up to 512 Intel Xeon Phi (KNL) nodes of Cori supercomputer with synchronous stochastic gradient descent (SGD), and identify causes of scaling inefficiency at higher node counts. To our knowledge, this is the first exploration of distributed GRPC Tensorflow scalability on a HPC supercomputer at such large scale with synchronous SGD. We studied scaling of two convolution neural networks - ResNet-50, a state-of-the-art deep network for classification with roughly 25.5 million parameters, and HEP-CNN, a shallow topology with less than 1 million parameters for common scientific usages. For ResNet-50, we achieve >80% scaling efficiency on up to 128 workers, using 32 parameter servers (PS tasks) with a steep decline down to 23% for 512 workers using 64 PS tasks. Our analysis of the efficiency drop points to low network bandwidth utilization due to combined effect of three factors. (a) Heterogeneous distributed parallelization algorithm which uses PS tasks as centralized servers for gradient averaging is suboptimal for utilizing interconnect bandwidth. (b) Load imbalance among PS tasks hinders their efficient scaling. (c) Underlying communication primitive GRPC is currently inefficient on Cori high-speed interconnect. The HEP-CNN demands less interconnect bandwidth, and shows >80% weak scaling efficiency for up to 256 nodes with only 1 PS task. Our findings are applicable to other deep learning networks. Big networks with millions of parameters stumble upon the issues discussed here. Shallower networks like HEP-CNN with relatively lower number of parameters can efficiently enjoy weak scaling even with a single parameter server. △ Less

Submitted 26 December, 2017; originally announced December 2017.

Comments: Published as a poster in NIPS 2017 Workshop: Deep Learning At Supercomputer Scale

Deep Neural Networks for Physics Analysis on low-level whole-detector data at the LHC

Authors: Wahid Bhimji, Steven Andrew Farrell, Thorsten Kurth, Michela Paganini, Prabhat, Evan Racah

Abstract: There has been considerable recent activity applying deep convolutional neural nets (CNNs) to data from particle physics experiments. Current approaches on ATLAS/CMS have largely focussed on a subset of the calorimeter, and for identifying objects or particular particle types. We explore approaches that use the entire calorimeter, combined with track information, for directly conducting physics an… ▽ More There has been considerable recent activity applying deep convolutional neural nets (CNNs) to data from particle physics experiments. Current approaches on ATLAS/CMS have largely focussed on a subset of the calorimeter, and for identifying objects or particular particle types. We explore approaches that use the entire calorimeter, combined with track information, for directly conducting physics analyses: i.e. classifying events as known-physics background or new-physics signals. We use an existing RPV-Supersymmetry analysis as a case study and explore CNNs on multi-channel, high-resolution sparse images: applied on GPU and multi-node CPU architectures (including Knights Landing (KNL) Xeon Phi nodes) on the Cori supercomputer at NERSC. △ Less

Submitted 29 November, 2017; v1 submitted 9 November, 2017; originally announced November 2017.

Comments: Presented at ACAT 2017 Conference, Submitted to J. Phys. Conf. Ser

Nucleon axial coupling from Lattice QCD

Authors: Chia Cheng Chang, Amy Nicholson, Enrico Rinaldi, Evan Berkowitz, Nicolas Garron, David Brantley, Henry Monge-Camacho, Chris Monahan, Chris Bouchard, M. A. Clark, Balint Joo, Thorsten Kurth, Kostas Orginos, Pavlos Vranas, Andre Walker-Loud

Abstract: We present state-of-the-art results from a lattice QCD calculation of the nucleon axial coupling, $g_A$, using Möbius Domain-Wall fermions solved on the dynamical $N_f = 2 + 1 + 1$ HISQ ensembles after they are smeared using the gradient-flow algorithm. Relevant three-point correlation functions are calculated using a method inspired by the Feynman-Hellmann theorem, and demonstrate significant imp… ▽ More We present state-of-the-art results from a lattice QCD calculation of the nucleon axial coupling, $g_A$, using Möbius Domain-Wall fermions solved on the dynamical $N_f = 2 + 1 + 1$ HISQ ensembles after they are smeared using the gradient-flow algorithm. Relevant three-point correlation functions are calculated using a method inspired by the Feynman-Hellmann theorem, and demonstrate significant improvement in signal for fixed stochastic samples. The calculation is performed at five pion masses of $m_π\sim \{400, 350, 310, 220, 130\}$~MeV, three lattice spacings of $a\sim\{0.15, 0.12, 0.09\}$~fm, and we do a dedicated volume study with $m_πL\sim\{3.22, 4.29, 5.36\}$. Control over all relevant sources of systematic uncertainty are demonstrated and quantified. We achieve a preliminary value of $g_A = 1.285(17)$, with a relative uncertainty of 1.33\%. △ Less

Submitted 17 October, 2017; originally announced October 2017.

Comments: 18 pages, 8 figures, Lattice 2017 Proceedings

Report number: INT-PUB-18-022

Calm Multi-Baryon Operators

Authors: Evan Berkowitz, Amy Nicholson, Chia Cheng Chang, Enrico Rinaldi, M. A. Clark, Bálint Joó, Thorsten Kurth, Pavlos Vranas, André Walker-Loud

Abstract: Outstanding problems in nuclear physics require input and guidance from lattice QCD calculations of few baryons systems. However, these calculations suffer from an exponentially bad signal-to-noise problem which has prevented a controlled extrapolation to the physical point. The variational method has been applied very successfully to two-meson systems, allowing for the extraction of the two-meson… ▽ More Outstanding problems in nuclear physics require input and guidance from lattice QCD calculations of few baryons systems. However, these calculations suffer from an exponentially bad signal-to-noise problem which has prevented a controlled extrapolation to the physical point. The variational method has been applied very successfully to two-meson systems, allowing for the extraction of the two-meson states very early in Euclidean time through the use of improved single hadron operators. The sheer numerical cost of using the same techniques in two-baryon systems has been prohibitive. We present an alternate strategy which offers some of the same advantages as the variational method while being significantly less numerically expensive. We first use the Matrix Prony method to form an optimal linear combination of single baryon interpolating fields generated from the same source and different sink interpolators. Very early in Euclidean time this linear combination is numerically free of excited state contamination, so we coin it a calm baryon. This calm baryon operator is then used in the construction of the two-baryon correlation functions. To test this method, we perform calculations on the WM/JLab iso-clover gauge configurations at the SU(3) flavor symmetric point with mπ $\sim$ 800 MeV --- the same configurations we have previously used for the calculation of two-nucleon correlation functions. We observe the calm baryon removes the excited state contamination from the two-nucleon correlation function to as early a time as the single-nucleon is improved, provided non-local (displaced nucleon) sources are used. For the local two-nucleon correlation function (where both nucleons are created from the same space-time location) there is still improvement, but there is significant excited state contamination in the region the single calm baryon displays no excited state contamination. △ Less

Submitted 16 October, 2017; originally announced October 2017.

Comments: 8 pages, 3 figures, proceedings for LATTICE 2017

Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data

Authors: Thorsten Kurth, Jian Zhang, Nadathur Satish, Ioannis Mitliagkas, Evan Racah, Mostofa Ali Patwary, Tareq Malas, Narayanan Sundaram, Wahid Bhimji, Mikhail Smorkalov, Jack Deslippe, Mikhail Shiryaev, Srinivas Sridharan, Prabhat, Pradeep Dubey

Abstract: This paper presents the first, 15-PetaFLOP Deep Learning system for solving scientific pattern classification problems on contemporary HPC architectures. We develop supervised convolutional architectures for discriminating signals in high-energy physics data as well as semi-supervised architectures for localizing and classifying extreme weather in climate data. Our Intelcaffe-based implementation… ▽ More This paper presents the first, 15-PetaFLOP Deep Learning system for solving scientific pattern classification problems on contemporary HPC architectures. We develop supervised convolutional architectures for discriminating signals in high-energy physics data as well as semi-supervised architectures for localizing and classifying extreme weather in climate data. Our Intelcaffe-based implementation obtains $\sim$2TFLOP/s on a single Cori Phase-II Xeon-Phi node. We use a hybrid strategy employing synchronous node-groups, while using asynchronous communication across groups. We use this strategy to scale training of a single model to $\sim$9600 Xeon-Phi nodes; obtaining peak performance of 11.73-15.07 PFLOP/s and sustained performance of 11.41-13.27 PFLOP/s. At scale, our HEP architecture produces state-of-the-art classification accuracy on a dataset with 10M images, exceeding that achieved by selections on high-level physics-motivated features. Our semi-supervised architecture successfully extracts weather patterns in a 15TB climate dataset. Our results demonstrate that Deep Learning can be optimized and scaled effectively on many-core, HPC systems. △ Less

Submitted 17 August, 2017; originally announced August 2017.

Comments: 12 pages, 9 figures

An accurate calculation of the nucleon axial charge with lattice QCD

Authors: Evan Berkowitz, David Brantley, Chris Bouchard, Chia Cheng Chang, M. A. Clark, Nicholas Garron, Balint Joo, Thorsten Kurth, Chris Monahan, Henry Monge-Camacho, Amy Nicholson, Kostas Orginos, Enrico Rinaldi, Pavlos Vranas, Andre Walker-Loud

Abstract: We report on a lattice QCD calculation of the nucleon axial charge, $g_A$, using Möbius Domain-Wall fermions solved on the dynamical $N_f=2+1+1$ HISQ ensembles after they are smeared using the gradient-flow algorithm. The calculation is performed with three pion masses, $m_π\sim\{310,220,130\}$ MeV. Three lattice spacings ($a\sim\{0.15,0.12,0.09\}$ fm) are used with the heaviest pion mass, while t… ▽ More We report on a lattice QCD calculation of the nucleon axial charge, $g_A$, using Möbius Domain-Wall fermions solved on the dynamical $N_f=2+1+1$ HISQ ensembles after they are smeared using the gradient-flow algorithm. The calculation is performed with three pion masses, $m_π\sim\{310,220,130\}$ MeV. Three lattice spacings ($a\sim\{0.15,0.12,0.09\}$ fm) are used with the heaviest pion mass, while the coarsest two spacings are used on the middle pion mass and only the coarsest spacing is used with the near physical pion mass. On the $m_π\sim220$ MeV, $a\sim0.12$ fm point, a dedicated volume study is performed with $m_πL \sim \{3.22,4.29,5.36\}$. Using a new strategy motivated by the Feynman-Hellmann Theorem, we achieve a precise determination of $g_A$ with relatively low statistics, and demonstrable control over the excited state, continuum, infinite volume and chiral extrapolation systematic uncertainties, the latter of which remains the dominant uncertainty. Our final determination at 2.6\% total uncertainty is $g_A = 1.278(21)(26)$, with the first uncertainty including statistical and systematic uncertainties from fitting and the second including model selection systematics related to the chiral and continuum extrapolation. The largest reduction of the second uncertainty will come from a greater number of pion mass points as well as more precise lattice QCD results near the physical pion mass. △ Less

Submitted 4 April, 2017; originally announced April 2017.

Comments: 17 pages + 11 pages of references and appendices. 15 figures. Interested readers can download the Python analysis scripts and an hdf5 data file at https://github.com/callat-qcd/project_gA_v0

Möbius domain-wall fermions on gradient-flowed dynamical HISQ ensembles

Authors: Evan Berkowitz, Chris Bouchard, Chia Cheng Chang, M. A. Clark, Balint Joo, Thorsten Kurth, Christopher Monahan, Amy Nicholson, Kostas Orginos, Enrico Rinaldi, Pavlos Vranas, Andre Walker-Loud

Abstract: We report on salient features of a mixed lattice QCD action using valence Möbius domain-wall fermions solved on the dynamical $N_f=2+1+1$ HISQ ensembles generated by the MILC Collaboration. The approximate chiral symmetry properties of the valence fermions are shown to be significantly improved by utilizing the gradient-flow scheme to first smear the HISQ configurations. The greater numerical cost… ▽ More We report on salient features of a mixed lattice QCD action using valence Möbius domain-wall fermions solved on the dynamical $N_f=2+1+1$ HISQ ensembles generated by the MILC Collaboration. The approximate chiral symmetry properties of the valence fermions are shown to be significantly improved by utilizing the gradient-flow scheme to first smear the HISQ configurations. The greater numerical cost of the Möbius domain-wall inversions is mitigated by the highly efficient QUDA library optimized for NVIDIA GPU accelerated compute nodes. We have created an interface to this optimized QUDA solver in Chroma. We provide tuned parameters of the action and performance of QUDA using ensembles with the lattice spacings $a \simeq \{0.15, 0.12, 0.09\}$ fm and pion masses $m_π\simeq \{310, 220,130\}$ MeV. We have additionally generated two new ensembles with $a\sim0.12$ fm and $m_π\sim\{400, 350\}$ MeV. With a fixed flow-time of $t_{gf}=1$ in lattice units, the residual chiral symmetry breaking of the valence fermions is kept below 10\% of the light quark mass on all ensembles, $m_{res} \lesssim 0.1\times m_l$, with moderate values of the fifth dimension $L_5$ and a domain-wall height $M_5 \leq 1.3$. As a benchmark calculation, we perform a continuum, infinite volume, physical pion and kaon mass extrapolation of $F_{K^\pm}/F_{π^\pm}$ and demonstrate our results are independent of flow-time, and consistent with the FLAG determination of this quantity at the level of less than one standard deviation. △ Less

Submitted 21 September, 2017; v1 submitted 25 January, 2017; originally announced January 2017.

Comments: 14 pages + refs, 5 figures; v3 is version accepter for publication; v2 includes continuum chiral extrapolation analysis of FK/Fpi and details of two new HISQ ensembles generated

Report number: LLNL-JRNL-719521, RBRC-1227

Journal ref: Phys. Rev. D 96, 054513 (2017)

On the Feynman-Hellmann Theorem in Quantum Field Theory and the Calculation of Matrix Elements

Authors: Chris Bouchard, Chia Cheng Chang, Thorsten Kurth, Kostas Orginos, Andre Walker-Loud

Abstract: The Feynman-Hellmann theorem can be derived from the long Euclidean-time limit of correlation functions determined with functional derivatives of the partition function. Using this insight, we fully develop an improved method for computing matrix elements of external currents utilizing only two-point correlation functions. Our method applies to matrix elements of any external bilinear current, inc… ▽ More The Feynman-Hellmann theorem can be derived from the long Euclidean-time limit of correlation functions determined with functional derivatives of the partition function. Using this insight, we fully develop an improved method for computing matrix elements of external currents utilizing only two-point correlation functions. Our method applies to matrix elements of any external bilinear current, including nonzero momentum transfer, flavor-changing, and two or more current insertion matrix elements. The ability to identify and control all the systematic uncertainties in the analysis of the correlation functions stems from the unique time dependence of the ground-state matrix elements and the fact that all excited states and contact terms are Euclidean-time dependent. We demonstrate the utility of our method with a calculation of the nucleon axial charge using gradient-flowed domain-wall valence quarks on the $N_f=2+1+1$ MILC highly improved staggered quark ensemble with lattice spacing and pion mass of approximately 0.15 fm and 310 MeV respectively. We show full control over excited-state systematics with the new method and obtain a value of $g_A = 1.213(26)$ with a quark-mass-dependent renormalization coefficient. △ Less

Submitted 5 July, 2017; v1 submitted 20 December, 2016; originally announced December 2016.

Comments: 12 pages, 13 figures, version published in PRD

Journal ref: Phys. Rev. D 96, 014504 (2017)

Neutrinoless double beta decay from lattice QCD

Authors: Amy Nicholson, Evan Berkowitz, Chia Cheng Chang, M. A. Clark, Balint Joo, Thorsten Kurth, Enrico Rinaldi, Brian Tiburzi, Pavlos Vranas, Andre Walker-Loud

Abstract: While the discovery of non-zero neutrino masses is one of the most important accomplishments by physicists in the past century, it is still unknown how and in what form these masses arise. Lepton number-violating neutrinoless double beta decay is a natural consequence of Majorana neutrinos and many BSM theories, and many experimental efforts are involved in the search for these processes. Understa… ▽ More While the discovery of non-zero neutrino masses is one of the most important accomplishments by physicists in the past century, it is still unknown how and in what form these masses arise. Lepton number-violating neutrinoless double beta decay is a natural consequence of Majorana neutrinos and many BSM theories, and many experimental efforts are involved in the search for these processes. Understanding how neutrinoless double beta decay would manifest in nuclear environments is key for understanding any observed signals. In these proceedings we present an overview of a set of one- and two-body matrix elements relevant for experimental searches for neutrinoless double beta decay, describe the role of lattice QCD calculations, and present preliminary lattice QCD results. △ Less

Submitted 16 August, 2016; originally announced August 2016.

Comments: Plenary talk given at the 34th International Symposium on Lattice Field Theory, Southampton, UK, 24-30 July 2016

Report number: LLNL-PROC-700398

Two-nucleon scattering in multiple partial waves

Authors: Amy Nicholson, Evan Berkowitz, Enrico Rinaldi, Pavlos Vranas, Thorsten Kurth, Balint Joo, Mark Strother, Andre Walker-Loud

Abstract: We determine scattering phase shifts for S,P,D, and F partial wave channels in two-nucleon systems using lattice QCD methods. We use a generalization of Luscher's finite volume method to determine infinite volume phase shifts from a set of finite volume ground- and excited-state energy levels on two volumes, V=(3.4 fm)^3 and V=(4.5 fm)^3. The calculations are performed in the SU(3)-flavor limit, c… ▽ More We determine scattering phase shifts for S,P,D, and F partial wave channels in two-nucleon systems using lattice QCD methods. We use a generalization of Luscher's finite volume method to determine infinite volume phase shifts from a set of finite volume ground- and excited-state energy levels on two volumes, V=(3.4 fm)^3 and V=(4.5 fm)^3. The calculations are performed in the SU(3)-flavor limit, corresponding to a pion mass of approximately 800 MeV. From the energy dependence of the phase shifts we are able to extract scattering parameters corresponding to an effective range expansion. △ Less

Submitted 6 November, 2015; originally announced November 2015.

Comments: 7 pages, 11 figures. Proceedings of the 33rd International Symposium on Lattice Field Theory, July 14-18, 2015, Kobe, Japan

Report number: LLNL-PROC-678486

Nuclear Parity Violation from Lattice QCD

Authors: Thorsten Kurth, Evan Berkowitz, Enrico Rinaldi, Pavlos Vranas, Amy Nicholson, Mark Strother, Andre Walker-Loud

Abstract: The electroweak interaction at the level of quarks and gluons are well understood from precision measurements in high energy collider experiments. Relating these fundamental parameters to Hadronic Parity Violation in nuclei however remains an outstanding theoretical challenge. One of the most interesting observables in this respect is the parity violating hadronic neutral current: it is hard to me… ▽ More The electroweak interaction at the level of quarks and gluons are well understood from precision measurements in high energy collider experiments. Relating these fundamental parameters to Hadronic Parity Violation in nuclei however remains an outstanding theoretical challenge. One of the most interesting observables in this respect is the parity violating hadronic neutral current: it is hard to measure in collider experiments and is thus the least constrained observable of the Standard Model. Precision measurements of parity violating transitions in nuclei can help to improve these constraints. In these systems however, the weak interaction is masked by effects of the seven orders of magnitude stronger non-perturbative strong interaction. Therefore, in order to relate experimental measurements of the parity violating pion-nucleon couplings to the fundamental Lagrangian of the SM, these non-perturbative effects have to be well understood. In this paper, we are going to present a Lattice QCD approach for computing the $ΔI{=}2$ parity violating matrix element in proton proton scattering. This process does not involve disconnected diagrams in the isospin symmetric limit and is thus a perfect testbed for studying the feasibility of the more involved calculation of the parity violating pion-nucleon coupling. △ Less

Submitted 6 November, 2015; originally announced November 2015.

Comments: PoS Lattice 2015

Report number: LLNL-PROC-678488

Two-Nucleon Higher Partial-Wave Scattering from Lattice QCD

Authors: Evan Berkowitz, Thorsten Kurth, Amy Nicholson, Balint Joo, Enrico Rinaldi, Mark Strother, Pavlos M. Vranas, Andre Walker-Loud

Abstract: We present a determination of nucleon-nucleon scattering phase shifts for l >= 0. The S, P, D and F phase shifts for both the spin-triplet and spin-singlet channels are computed with lattice Quantum ChromoDynamics. For l > 0, this is the first lattice QCD calculation using the Luscher finite-volume formalism. This required the design and implementation of novel lattice methods involving displaced… ▽ More We present a determination of nucleon-nucleon scattering phase shifts for l >= 0. The S, P, D and F phase shifts for both the spin-triplet and spin-singlet channels are computed with lattice Quantum ChromoDynamics. For l > 0, this is the first lattice QCD calculation using the Luscher finite-volume formalism. This required the design and implementation of novel lattice methods involving displaced sources and momentum-space cubic sinks. To demonstrate the utility of our approach, the calculations were performed in the SU(3)-flavor limit where the light quark masses have been tuned to the physical strange quark mass, corresponding to m_pi = m_K ~ 800 MeV. In this work, we have assumed that only the lowest partial waves contribute to each channel, ignoring the unphysical partial wave mixing that arises within the finite-volume formalism. This assumption is only valid for sufficiently low energies; we present evidence that it holds for our study using two different channels. Two spatial volumes of V ~ (3.5 fm)^3 and V ~ (4.6 fm)^3 were used. The finite-volume spectrum is extracted from the exponential falloff of the correlation functions. Said spectrum is mapped onto the infinite volume phase shifts using the generalization of the Luscher formalism for two-nucleon systems. △ Less

Submitted 24 December, 2016; v1 submitted 4 August, 2015; originally announced August 2015.

Comments: Added plots and expanded text to clarify discussions, particularly for extraction of the energies. Published version

Report number: LLNL-JRNL-674381, NT@WM-15-13, JLAB-THY-15-2116

Journal ref: Physics Letters B, Volume 765, 10 February 2017, Pages 285 to 292

High-Performance I/O: HDF5 for Lattice QCD

Authors: Thorsten Kurth, Andrew Pochinsky, Abhinav Sarje, Sergey Syritsyn, Andre Walker-Loud

Abstract: Practitioners of lattice QCD/QFT have been some of the primary pioneer users of the state-of-the-art high-performance-computing systems, and contribute towards the stress tests of such new machines as soon as they become available. As with all aspects of high-performance-computing, I/O is becoming an increasingly specialized component of these systems. In order to take advantage of the latest avai… ▽ More Practitioners of lattice QCD/QFT have been some of the primary pioneer users of the state-of-the-art high-performance-computing systems, and contribute towards the stress tests of such new machines as soon as they become available. As with all aspects of high-performance-computing, I/O is becoming an increasingly specialized component of these systems. In order to take advantage of the latest available high-performance I/O infrastructure, to ensure reliability and backwards compatibility of data files, and to help unify the data structures used in lattice codes, we have incorporated parallel HDF5 I/O into the SciDAC supported USQCD software stack. Here we present the design and implementation of this I/O framework. Our HDF5 implementation outperforms optimized QIO at the 10-20% level and leaves room for further improvement by utilizing appropriate dataset chunking. △ Less

Submitted 28 January, 2015; originally announced January 2015.

Comments: Contribution to the 32nd International Symposium on Lattice Field Theory (Lattice 2014), 23-28 June 2014, Columbia University, New York, NY, USA

Report number: NT@WM-15-02, JLAB-THY-15-2007

Lattice QCD at the physical point meets SU(2) chiral perturbation theory

Authors: Stephan Durr, Zoltan Fodor, Christian Hoelbling, Stefan Krieg, Thorsten Kurth, Laurent Lellouch, Thomas Lippert, Rehan Malak, Thibaut Metivet, Antonin Portelli, Alfonso Sastre, Kalman Szabo

Abstract: We perform a detailed, fully-correlated study of the chiral behavior of the pion mass and decay constant, based on 2+1 flavor lattice QCD simulations. These calculations are implemented using tree-level, O(a)-improved Wilson fermions, at four values of the lattice spacing down to 0.054 fm and all the way down to below the physical value of the pion mass. They allow a sharp comparison with the pred… ▽ More We perform a detailed, fully-correlated study of the chiral behavior of the pion mass and decay constant, based on 2+1 flavor lattice QCD simulations. These calculations are implemented using tree-level, O(a)-improved Wilson fermions, at four values of the lattice spacing down to 0.054 fm and all the way down to below the physical value of the pion mass. They allow a sharp comparison with the predictions of SU(2) chiral perturbation theory (χPT) and a determination of some of its low energy constants. In particular, we systematically explore the range of applicability of NLO SU(2) χPT in two different expansions: the first in quark mass (x-expansion), and the second in pion mass (ξ-expansion). We find that these expansions begin showing signs of failure around M_π=300 MeV for the typical percent-level precision of our N_f=2+1 lattice results. We further determine the LO low energy constants (LECs), F=88.0 \pm 1.3\pm 0.3 and B^\msbar(2 GeV)=2.58 \pm 0.07 \pm 0.02 GeV, and the related quark condensate, Σ^\msbar(2 GeV)=(271\pm 4\pm 1 MeV)^3, as well as the NLO ones, l_3=2.5 \pm 0.5 \pm 0.4 and l_4=3.8 \pm 0.4 \pm 0.2, with fully controlled uncertainties. We also explore the NNLO expansions and the values of NNLO LECs. In addition, we show that the lattice results favor the presence of chiral logarithms. We further demonstrate how the absence of lattice results with pion masses below 200 MeV can lead to misleading results and conclusions. Our calculations allow a fully controlled, ab initio determination of the pion decay constant with a total 1% error, which is in excellent agreement with experiment. △ Less

Submitted 14 October, 2013; originally announced October 2013.

Report number: CPT-P005-2013, WUB/13-14

Journal ref: Phys. Rev. D 90, 114504 (2014)

Isospin splittings in the light baryon octet from lattice QCD and QED

Authors: Sz. Borsanyi, S. Dürr, Z. Fodor, J. Frison, C. Hoelbling, S. D. Katz, S. Krieg, Th. Kurth, L. Lellouch, Th. Lippert, A. Portelli, A. Ramos, A. Sastre, K. Szabo

Abstract: While electromagnetic and up-down quark mass difference effects on octet baryon masses are very small, they have important consequences. The stability of the hydrogen atom against beta decay is a prominent example. Here we include these effects by adding them to valence quarks in a lattice QCD calculation based on $N_f=2+1$ simulations with 5 lattice spacings down to 0.054 fm, lattice sizes up to… ▽ More While electromagnetic and up-down quark mass difference effects on octet baryon masses are very small, they have important consequences. The stability of the hydrogen atom against beta decay is a prominent example. Here we include these effects by adding them to valence quarks in a lattice QCD calculation based on $N_f=2+1$ simulations with 5 lattice spacings down to 0.054 fm, lattice sizes up to 6 fm and average up-down quark masses all the way down to their physical value. This allows us to gain control over all systematic errors, except for the one associated with neglecting electromagnetism in the sea. We compute the octet baryon isomultiplet mass splittings, as well as the individual contributions from electromagnetism and the up-down quark mass difference. Our results for the total splittings are in good agreement with experiment. △ Less

Submitted 6 January, 2014; v1 submitted 10 June, 2013; originally announced June 2013.

Comments: 5 pages, 3 figures. Version accepted for publication by Phys. Rev. Lett

Report number: CPT-P002-2013

Journal ref: Phys. Rev. Lett. 111, 252001 (2013)

Phase shifts in I=2 ππ-scattering from two lattice approaches

Authors: T. Kurth, N. Ishii, T. Doi, S. Aoki, T. Hatsuda

Abstract: We present a lattice QCD study of the phase shift of $I{=}2$ $ππ$ scattering on the basis of two different approaches: the standard finite volume approach by Luscher and the recently introduced HAL QCD potential method. Quenched QCD simulations are performed on lattices with extents $N_s{=}16,24,32,48$ and $N_t{=}128$ as well as lattice spacing $a{\sim}0.115\,\mathrm{fm}$ and a pion mass of… ▽ More We present a lattice QCD study of the phase shift of $I{=}2$ $ππ$ scattering on the basis of two different approaches: the standard finite volume approach by Luscher and the recently introduced HAL QCD potential method. Quenched QCD simulations are performed on lattices with extents $N_s{=}16,24,32,48$ and $N_t{=}128$ as well as lattice spacing $a{\sim}0.115\,\mathrm{fm}$ and a pion mass of $m_π{\sim}940\,\mathrm{MeV}$. The phase shift and the scattering length are calculated in these two methods. In the potential method, the error is dominated by the systematic uncertainty associated with the violation of rotational symmetry due to finite lattice spacing. In Luscher's approach, such systematic uncertainty is difficult to be evaluated and thus is not included in this work. A systematic uncertainty attributed to the quenched approximation, however, is not evaluated in both methods. In case of the potential method, the phase shift can be calculated for arbitrary energies below the inelastic threshold. The energy dependence of the phase shift is also obtained from Luscher's method using different volumes and/or nonrest-frame extension of it. The results are found to agree well with the potential method. △ Less

Submitted 5 December, 2013; v1 submitted 20 May, 2013; originally announced May 2013.

Comments: Version 2, consistent with published version

Report number: WUB/13-06, UTCCS-P-73, RIKEN-QHP-84, YITP-13-27

Journal ref: JHEP12(2013)015

Anisotropy tuning with the Wilson flow

Authors: S. Borsanyi, S. Durr, Z. Fodor, S. D. Katz, S. Krieg, T. Kurth, S. Mages, A. Schafer, K. K. Szabo

Abstract: We use the Wilson flow to define the gauge anisotropy at a given physical scale. We demonstrate the use of the anisotropic flow by performing the tuning of the bare gauge anisotropy in the tree-level Symanzik action for several lattice spacings and target anisotropies. We use this method to tune the anisotropy parameters in full QCD, where we also exploit the diminishing effect of a well chosen sm… ▽ More We use the Wilson flow to define the gauge anisotropy at a given physical scale. We demonstrate the use of the anisotropic flow by performing the tuning of the bare gauge anisotropy in the tree-level Symanzik action for several lattice spacings and target anisotropies. We use this method to tune the anisotropy parameters in full QCD, where we also exploit the diminishing effect of a well chosen smearing on the renormalization of the fermion anisotropy. △ Less

Submitted 3 May, 2012; originally announced May 2012.

Comments: 21 pages, 7 figures, 2 tables

High-precision scale setting in lattice QCD

Authors: S. Borsanyi, S. Durr, Z. Fodor, C. Hoelbling, S. D. Katz, S. Krieg, T. Kurth, L. Lellouch, T. Lippert, C. McNeile, K. K. Szabo

Abstract: Scale setting is of central importance in lattice QCD. It is required to predict dimensional quantities in physical units. Moreover, it determines the relative lattice spacings of computations performed at different values of the bare coupling, and this is needed for extrapolating results into the continuum. Thus, we calculate a new quantity, $w_0$, for setting the scale in lattice QCD, which is b… ▽ More Scale setting is of central importance in lattice QCD. It is required to predict dimensional quantities in physical units. Moreover, it determines the relative lattice spacings of computations performed at different values of the bare coupling, and this is needed for extrapolating results into the continuum. Thus, we calculate a new quantity, $w_0$, for setting the scale in lattice QCD, which is based on the Wilson flow like the scale $t_0$ (M. Luscher, JHEP 1008 (2010) 071). It is cheap and straightforward to implement and compute. In particular, it does not involve the delicate fitting of correlation functions at asymptotic times. It typically can be determined on the few per-mil level. We compute its continuum extrapolated value in 2+1-flavor QCD for physical and non-physical pion and kaon masses, to allow for mass-independent scale setting even away from the physical mass point. We demonstrate its robustness by computing it with two very different actions (one of them with staggered, the other with Wilson fermions) and by showing that the results agree for physical quark masses in the continuum limit. △ Less

Submitted 5 November, 2012; v1 submitted 20 March, 2012; originally announced March 2012.

Comments: 15 pages, 7 figures, 2 tables; Version published in JHEP

Report number: ITP-Budapest 657; CPT-P004-2012; WUB/12-02

Journal ref: JHEP 1209 (2012) 010

Systematic errors in partially-quenched QCD plus QED lattice simulations

Authors: Antonin Portelli, Stephan Dürr, Zoltán Fodor, Julien Frison, Christian Hoelbling, Sandor D. Katz, Stefan Krieg, Thorsten Kurth, Laurent Lellouch, Thomas Lippert, Alberto Ramos, Kálmán K. Szabó

Abstract: At the precision reached in current lattice QCD calculations, electromagnetic effects are becoming numerically relevant. Here, electromagnetic effects are included by superimposing $\mathrm{U}(1)$ degrees of freedom on $N_f = 2+1$ QCD configurations from the Budapest-Marseille-Wuppertal Collaboration. We present preliminary results for the electromagnetic corrections to light pseudoscalars mesons… ▽ More At the precision reached in current lattice QCD calculations, electromagnetic effects are becoming numerically relevant. Here, electromagnetic effects are included by superimposing $\mathrm{U}(1)$ degrees of freedom on $N_f = 2+1$ QCD configurations from the Budapest-Marseille-Wuppertal Collaboration. We present preliminary results for the electromagnetic corrections to light pseudoscalars mesons masses and discuss some of the associated systematic errors. △ Less

Submitted 13 January, 2012; originally announced January 2012.

Comments: 7 pages, 2 figures, The XXIX International Symposium on Lattice Field Theory, July 10-16, 2011, Squaw Valley, Lake Tahoe, California, USA

Report number: CPT-P001-2012

Sigma term and strangeness content of octet baryons

Authors: S. Durr, Z. Fodor, T. Hemmert, C. Hoelbling, J. Frison, S. D. Katz, S. Krieg, T. Kurth, L. Lellouch, T. Lippert, A. Portelli, A. Ramos, A. Schafer, K. K. Szabo

Abstract: By using lattice QCD computations we determine the sigma terms and strangeness content of all octet baryons by means of an application of the Hellmann-Feynman theorem. In addition to polynomial and rational expressions for the quark mass dependence of octet members, we use SU(3) covariant baryon chiral perturbation theory to perform the extrapolation to the physical up and down quark masses. Our N… ▽ More By using lattice QCD computations we determine the sigma terms and strangeness content of all octet baryons by means of an application of the Hellmann-Feynman theorem. In addition to polynomial and rational expressions for the quark mass dependence of octet members, we use SU(3) covariant baryon chiral perturbation theory to perform the extrapolation to the physical up and down quark masses. Our N_f=2+1 lattice ensembles include pion masses down to about 190 MeV in large volumes (M_πL > 4), and three values of the lattice spacing. Our main results are the nucleon sigma term σ_{πN} = 39(4)(^{+18}_{-7}) and the strangeness content y_{N} = 0.20(7)(^{+13}_{-17}). Under the assumption of validity of covariant baryon χPT in our range of masses one finds y_{N} = 0.276(77)(^{+90}_{-62}). △ Less

Submitted 27 February, 2016; v1 submitted 20 September, 2011; originally announced September 2011.

Comments: LaTeX. 15 pages, 5 figures. Text and results corrected according to Erratum [Phys. Rev. D 93, 039905(E) (2016)]

Journal ref: Phys. Rev. D 85, 014509 (2012)

Spectra of heavy-light and heavy-heavy mesons containing charm quarks, including higher spin states for $N_f=2+ 1$

Authors: G. Bali, S. Collins, S. Durr, Z. Fodor, R. Horsley, C. Hoelbling, S. D. Katz, I. Kanamori, S. Krieg, T. Kurth, L. Lellouch, T. Lippert, C. McNeile, Y. Nakamura, D. Pleiter, P. Perez-Rubio, P. Rakow, A. Schafer, G. Schierholz K. K. Szabo, F. Winter, J. Zanotti

Abstract: We study the spectra of heavy-light and heavy-heavy mesons containing charm quarks, including higher spin states. We use two sets of $N_f = 2 + 1$ gauge configurations, one set from QCDSF using the SLiNC action, and the other configurations from the Budapest-Marseille-Wuppertal collaboration, using the HEX smeared clover action. To extract information about the excited states, we choose a suitable… ▽ More We study the spectra of heavy-light and heavy-heavy mesons containing charm quarks, including higher spin states. We use two sets of $N_f = 2 + 1$ gauge configurations, one set from QCDSF using the SLiNC action, and the other configurations from the Budapest-Marseille-Wuppertal collaboration, using the HEX smeared clover action. To extract information about the excited states, we choose a suitable basis of operators to implement the variational method. △ Less

Submitted 31 August, 2011; originally announced August 2011.

Comments: 7 pages, 5 figures, Talk presented at the XXIX International Symposium on Lattice Field Theory, Lattice2011, July 11-16, 2011, The Village at Squaw Valley, California, USA

Precision computation of the kaon bag parameter

Authors: S. Durr, Z. Fodor, C. Hoelbling, S. D. Katz, S. Krieg, T. Kurth, L. Lellouch, T. Lippert, C. McNeile, A. Portelli, K. K. Szabo

Abstract: Indirect CP violation in K \rightarrow ππ decays plays a central role in constraining the flavor structure of the Standard Model (SM) and in the search for new physics. For many years the leading uncertainty in the SM prediction of this phenomenon was the one associated with the nonperturbative strong interaction dynamics in this process. Here we present a fully controlled lattice QCD calculation… ▽ More Indirect CP violation in K \rightarrow ππ decays plays a central role in constraining the flavor structure of the Standard Model (SM) and in the search for new physics. For many years the leading uncertainty in the SM prediction of this phenomenon was the one associated with the nonperturbative strong interaction dynamics in this process. Here we present a fully controlled lattice QCD calculation of these effects, which are described by the neutral kaon mixing parameter B_K . We use a two step HEX smeared clover-improved Wilson action, with four lattice spacings from a\approx0.054 fm to a\approx0.093 fm and pion masses at and even below the physical value. Nonperturbative renormalization is performed in the RI-MOM scheme, where we find that operator mixing induced by chiral symmetry breaking is very small. Using fully nonperturbative continuum running, we obtain our main result B_K^{RI}(3.5GeV)=0.531(6)_{stat}(2)_{sys}. A perturbative 2-loop conversion yields B_K^{MSbar-NDR}(2GeV)=0.564(6)_{stat}(3)_{sys}(6)_{PT}, which is in good agreement with current results from fits to experimental data. △ Less

Submitted 9 November, 2011; v1 submitted 16 June, 2011; originally announced June 2011.

Comments: 10 pages, 7 figures. v2: Added one reference and one figure, replaced 2 figures for better readability and updated ensembles, conclusions unchanged. Final, published version

Report number: WUB/11-03; CPT-P008-2011

Journal ref: Physics Letters B 705 (2011), pp. 477-481

FK/Fpi from the Budapest-Marseille-Wuppertal Collaboration

Authors: A. Ramos, S. Durr, Z. Fodor, C. Hoelbling, S. D. Katz, S. Krieg, T. Kurth, L. Lellouch, T. Lippert, K. K. Szabo

Abstract: Based on a series of lattice calculations we determine the ratio FK/Fpi in QCD. With experimental data from kaon decay and nuclear double beta decay, we obtain a precise determination of |Vus|. Our simulation includes 2+1 flavours of sea quarks, with three lattice spacings, large volumes and a simulated pion mass reaching down to about 190 MeV for a full control over the systematic uncertainties. Based on a series of lattice calculations we determine the ratio FK/Fpi in QCD. With experimental data from kaon decay and nuclear double beta decay, we obtain a precise determination of |Vus|. Our simulation includes 2+1 flavours of sea quarks, with three lattice spacings, large volumes and a simulated pion mass reaching down to about 190 MeV for a full control over the systematic uncertainties. △ Less

Submitted 20 January, 2011; originally announced January 2011.

Comments: 6 pages, 2 figures. Proceedings of CKM2010, the 6th International Workshop on the CKM Unitarity Triangle, University of Warwick, UK, 6-10 September 2010