A cast of thousands: How the IDEAS Productivity project has advanced software productivity and sustainability
Authors:
Lois Curfman McInnes,
Michael Heroux,
David E. Bernholdt,
Anshu Dubey,
Elsa Gonsiorowski,
Rinku Gupta,
Osni Marques,
J. David Moulton,
Hai Ah Nam,
Boyana Norris,
Elaine M. Raybourn,
Jim Willenbring,
Ann Almgren,
Ross Bartlett,
Kita Cranfill,
Stephen Fickas,
Don Frederick,
William Godoy,
Patricia Grubel,
Rebecca Hartman-Baker,
Axel Huebl,
Rose Lynch,
Addi Malviya Thakur,
Reed Milewicz,
Mark C. Miller
, et al. (9 additional authors not shown)
Abstract:
Computational and data-enabled science and engineering are revolutionizing advances throughout science and society, at all scales of computing. For example, teams in the U.S. DOE Exascale Computing Project have been tackling new frontiers in modeling, simulation, and analysis by exploiting unprecedented exascale computing capabilities-building an advanced software ecosystem that supports next-gene…
▽ More
Computational and data-enabled science and engineering are revolutionizing advances throughout science and society, at all scales of computing. For example, teams in the U.S. DOE Exascale Computing Project have been tackling new frontiers in modeling, simulation, and analysis by exploiting unprecedented exascale computing capabilities-building an advanced software ecosystem that supports next-generation applications and addresses disruptive changes in computer architectures. However, concerns are growing about the productivity of the developers of scientific software, its sustainability, and the trustworthiness of the results that it produces. Members of the IDEAS project serve as catalysts to address these challenges through fostering software communities, incubating and curating methodologies and resources, and disseminating knowledge to advance developer productivity and software sustainability. This paper discusses how these synergistic activities are advancing scientific discovery-mitigating technical risks by building a firmer foundation for reproducible, sustainable science at all scales of computing, from laptops to clusters to exascale and beyond.
△ Less
Submitted 16 February, 2024; v1 submitted 3 November, 2023;
originally announced November 2023.
The Petascale DTN Project: High Performance Data Transfer for HPC Facilities
Authors:
Eli Dart,
William Allcock,
Wahid Bhimji,
Tim Boerner,
Ravinderjeet Cheema,
Andrew Cherry,
Brent Draney,
Salman Habib,
Damian Hazen,
Jason Hill,
Matt Kollross,
Suzanne Parete-Koon,
Daniel Pelfrey,
Adrian Pope,
Jeff Porter,
David Wheeler
Abstract:
The movement of large-scale (tens of Terabytes and larger) data sets between high performance computing (HPC) facilities is an important and increasingly critical capability. A growing number of scientific collaborations rely on HPC facilities for tasks which either require large-scale data sets as input or produce large-scale data sets as output. In order to enable the transfer of these data sets…
▽ More
The movement of large-scale (tens of Terabytes and larger) data sets between high performance computing (HPC) facilities is an important and increasingly critical capability. A growing number of scientific collaborations rely on HPC facilities for tasks which either require large-scale data sets as input or produce large-scale data sets as output. In order to enable the transfer of these data sets as needed by the scientific community, HPC facilities must design and deploy the appropriate data transfer capabilities to allow users to do data placement at scale.
This paper describes the Petascale DTN Project, an effort undertaken by four HPC facilities, which succeeded in achieving routine data transfer rates of over 1PB/week between the facilities. We describe the design and configuration of the Data Transfer Node (DTN) clusters used for large-scale data transfers at these facilities, the software tools used, and the performance tuning that enabled this capability.
△ Less
Submitted 8 September, 2021; v1 submitted 26 May, 2021;
originally announced May 2021.
