-
GALÆXI: Solving complex compressible flows with high-order discontinuous Galerkin methods on accelerator-based systems
Authors:
Daniel Kempf,
Marius Kurz,
Marcel Blind,
Patrick Kopper,
Philipp Offenhäuser,
Anna Schwarz,
Spencer Starr,
Jens Keim,
Andrea Beck
Abstract:
This work presents GALÆXI as a novel, energy-efficient flow solver for the simulation of compressible flows on unstructured meshes leveraging the parallel computing power of modern Graphics Processing Units (GPUs). GALÆXI implements the high-order Discontinuous Galerkin Spectral Element Method (DGSEM) using shock capturing with a finite-volume subcell approach to ensure the stability of the high-o…
▽ More
This work presents GALÆXI as a novel, energy-efficient flow solver for the simulation of compressible flows on unstructured meshes leveraging the parallel computing power of modern Graphics Processing Units (GPUs). GALÆXI implements the high-order Discontinuous Galerkin Spectral Element Method (DGSEM) using shock capturing with a finite-volume subcell approach to ensure the stability of the high-order scheme near shocks. This work provides details on the general code design, the parallelization strategy, and the implementation approach for the compute kernels with a focus on the element local mappings between volume and surface data due to the unstructured mesh. GALÆXI exhibits excellent strong scaling properties up to 1024 GPUs if each GPU is assigned a minimum of one million degrees of freedom degrees of freedom. To verify its implementation, a convergence study is performed that recovers the theoretical order of convergence of the implemented numerical schemes. Moreover, the solver is validated using both the incompressible and compressible formulation of the Taylor-Green-Vortex at a Mach number of 0.1 and 1.25, respectively. A mesh convergence study shows that the results converge to the high-fidelity reference solution and that the results match the original CPU implementation. Finally, GALÆXI is applied to a large-scale wall-resolved large eddy simulation of a linear cascade of the NASA Rotor 37. Here, the supersonic region and shocks at the leading edge are captured accurately and robustly by the implemented shock-capturing approach. It is demonstrated that GALÆXI requires less than half of the energy to carry out this simulation in comparison to the reference CPU implementation. This renders GALÆXI as a potent tool for accurate and efficient simulations of compressible flows in the realm of exascale computing and the associated new HPC architectures.
△ Less
Submitted 19 April, 2024;
originally announced April 2024.
-
Foundational Competencies and Responsibilities of a Research Software Engineer
Authors:
Florian Goth,
Renato Alves,
Matthias Braun,
Leyla Jael Castro,
Gerasimos Chourdakis,
Simon Christ,
Jeremy Cohen,
Stephan Druskat,
Fredo Erxleben,
Jean-Noël Grad,
Magnus Hagdorn,
Toby Hodges,
Guido Juckeland,
Dominic Kempf,
Anna-Lena Lamprecht,
Jan Linxweiler,
Frank Löffler,
Michele Martone,
Moritz Schwarzmeier,
Heidi Seibold,
Jan Philipp Thiele,
Harald von Waldow,
Samantha Wittke
Abstract:
The term Research Software Engineer, or RSE, emerged a little over 10 years ago as a way to represent individuals working in the research community but focusing on software development. The term has been widely adopted and there are a number of high-level definitions of what an RSE is. However, the roles of RSEs vary depending on the institutional context they work in. At one end of the spectrum,…
▽ More
The term Research Software Engineer, or RSE, emerged a little over 10 years ago as a way to represent individuals working in the research community but focusing on software development. The term has been widely adopted and there are a number of high-level definitions of what an RSE is. However, the roles of RSEs vary depending on the institutional context they work in. At one end of the spectrum, RSE roles may look similar to a traditional research role. At the other extreme, they resemble that of a software engineer in industry. Most RSE roles inhabit the space between these two extremes. Therefore, providing a straightforward, comprehensive definition of what an RSE does and what experience, skills and competencies are required to become one is challenging. In this community paper we define the broad notion of what an RSE is, explore the different types of work they undertake, and define a list of fundamental competencies as well as values that define the general profile of an RSE. On this basis, we elaborate on the progression of these skills along different dimensions, looking at specific types of RSE roles, proposing recommendations for organisations, and giving examples of future specialisations. An appendix details how existing curricula fit into this framework.
△ Less
Submitted 12 August, 2024; v1 submitted 19 November, 2023;
originally announced November 2023.
-
Towards Exascale CFD Simulations Using the Discontinuous Galerkin Solver FLEXI
Authors:
Marcel Blind,
Min Gao,
Daniel Kempf,
Patrick Kopper,
Marius Kurz,
Anna Schwarz,
Andrea Beck
Abstract:
Modern high-order discretizations bear considerable potential for the exascale era due to their high fidelity and the high, local computational load that allows for computational efficiency in massively parallel simulations. To this end, the discontinuous Galerkin (DG) framework FLEXI was selected to demonstrate exascale readiness within the Center of Excellence for Exascale CFD (CEEC) by simulati…
▽ More
Modern high-order discretizations bear considerable potential for the exascale era due to their high fidelity and the high, local computational load that allows for computational efficiency in massively parallel simulations. To this end, the discontinuous Galerkin (DG) framework FLEXI was selected to demonstrate exascale readiness within the Center of Excellence for Exascale CFD (CEEC) by simulating shock buffet on a three-dimensional wing segment at transsonic flight conditions. This paper summarizes the recent progress made to enable the simulation of this challenging exascale problem. For this, it is first demonstrated that FLEXI scales excellently to over 500 000 CPU cores on HAWK at the HLRS. To tackle the considerable resolution requirements near the wall, a novel wall model is proposed that takes compressibility effects into account and yields decent results for the simulation of a NACA 64A-110 airfoil. To address the shocks in the domain, a finite-volume-based shock capturing method was implemented in FLEXI, which is validated here using the simulation of a linear compressor cascade at supersonic flow conditions, where the method is demonstrated to yield efficient, robust and accurate results. Lastly, we present the TensorFlow-Fortran-Binding (TFFB) as an easy-to-use library to deploy trained machine learning models in Fortran solvers such as FLEXI.
△ Less
Submitted 22 June, 2023;
originally announced June 2023.
-
An Efficient Sliding Mesh Interface Method for High-Order Discontinuous Galerkin Schemes
Authors:
Jakob Dürrwächter,
Marius Kurz,
Patrick Kopper,
Daniel Kempf,
Claus-Dieter Munz,
Andrea Beck
Abstract:
Sliding meshes are a powerful method to treat deformed domains in computational fluid dynamics, where different parts of the domain are in relative motion. In this paper, we present an efficient implementation of a sliding mesh method into a discontinuous Galerkin compressible Navier-Stokes solver and its application to a large eddy simulation of a 1-1/2 stage turbine. The method is based on the m…
▽ More
Sliding meshes are a powerful method to treat deformed domains in computational fluid dynamics, where different parts of the domain are in relative motion. In this paper, we present an efficient implementation of a sliding mesh method into a discontinuous Galerkin compressible Navier-Stokes solver and its application to a large eddy simulation of a 1-1/2 stage turbine. The method is based on the mortar method and is high-order accurate. It can handle three-dimensional sliding mesh interfaces with various interface shapes. For plane interfaces, which are the most common case, conservativity and free-stream preservation are ensured. We put an emphasis on efficient parallel implementation. Our implementation generates little computational and storage overhead. Inter-node communication via MPI in a dynamically changing mesh topology is reduced to a bare minimum by ensuring a priori information about communication partners and data sorting. We provide performance and scaling results showing the capability of the implementation strategy. Apart from analytical validation computations and convergence results, we present a wall-resolved implicit LES of the 1-1/2 stage Aachen turbine test case as a large scale practical application example.
△ Less
Submitted 10 August, 2020;
originally announced August 2020.
-
The DUNE Framework: Basic Concepts and Recent Developments
Authors:
Peter Bastian,
Markus Blatt,
Andreas Dedner,
Nils-Arne Dreier,
Christian Engwer,
René Fritze,
Carsten Gräser,
Christoph Grüninger,
Dominic Kempf,
Robert Klöfkorn,
Mario Ohlberger,
Oliver Sander
Abstract:
This paper presents the basic concepts and the module structure of the Distributed and Unified Numerics Environment and reflects on recent developments and general changes that happened since the release of the first Dune version in 2007 and the main papers describing that state [1, 2]. This discussion is accompanied with a description of various advanced features, such as coupling of domains and…
▽ More
This paper presents the basic concepts and the module structure of the Distributed and Unified Numerics Environment and reflects on recent developments and general changes that happened since the release of the first Dune version in 2007 and the main papers describing that state [1, 2]. This discussion is accompanied with a description of various advanced features, such as coupling of domains and cut cells, grid modifications such as adaptation and moving domains, high order discretizations and node level performance, non-smooth multigrid methods, and multiscale methods. A brief discussion on current and future development directions of the framework concludes the paper.
△ Less
Submitted 22 June, 2020; v1 submitted 30 September, 2019;
originally announced September 2019.
