-
The Service Analysis and Network Diagnosis DataPipeline
Authors:
Derek Weitzel,
Shawn McKee,
Brian Paul Bockelman,
John Thiltges,
Marian Babik,
Ilija Vukotic
Abstract:
Modern network performance monitoring toolkits, such as perfSONAR, take a remarkable number of measurements about the local network environment. To gain a complete picture of network performance, however, one needs to aggregate data across a large number of endpoints. The Service Analysis and Network Diagnosis (SAND) data pipeline collects data from diverse sources and ingests these measurements i…
▽ More
Modern network performance monitoring toolkits, such as perfSONAR, take a remarkable number of measurements about the local network environment. To gain a complete picture of network performance, however, one needs to aggregate data across a large number of endpoints. The Service Analysis and Network Diagnosis (SAND) data pipeline collects data from diverse sources and ingests these measurements into a message bus. The message bus allows the project to send the data to multiple consumers, including a tape archive, an Elasticsearch database, and a peer infrastructure at CERN. In this paper, we explain the architecture and evolution of the SAND data pipeline, the scale of the resulting dataset, and how it supports a wide variety of network analysis applications.
△ Less
Submitted 6 December, 2021;
originally announced December 2021.
-
WLCG Networks: Update on Monitoring and Analytics
Authors:
Marian Babik,
Shawn McKee,
Pedro Andrade,
Brian Paul Bockelman,
Robert Gardner,
Edgar Mauricio Fajardo Hernandez,
Edoardo Martelli,
Ilija Vukotic,
Derek Weitzel,
Marian Zvada
Abstract:
WLCG relies on the network as a critical part of its infrastructure and therefore needs to guarantee effective network usage and prompt detection and resolution of any network issues including connection failures, congestion and traffic routing. The OSG Networking Area, in partnership with WLCG, is focused on being the primary source of networking information for its partners and constituents. It…
▽ More
WLCG relies on the network as a critical part of its infrastructure and therefore needs to guarantee effective network usage and prompt detection and resolution of any network issues including connection failures, congestion and traffic routing. The OSG Networking Area, in partnership with WLCG, is focused on being the primary source of networking information for its partners and constituents. It was established to ensure sites and experiments can better understand and fix networking issues, while providing an analytics platform that aggregates network monitoring data with higher level workload and data trans-fer services. This has been facilitated by the global network of the perfSONAR instances that have been commissioned and are operated in collaboration with WLCG Network Throughput Working Group. An additional important updateis the inclusion of the newly funded NSF project SAND (Service Analytics and Network Diagnosis) which is focusing on network analytics. This paper describes the current state of the network measurement and analytics platform and summarizes the activities taken by the working group and our collaborators. This includes the progress being made in providing higher level analytics,alerting and alarming from the rich set of network metrics we are gathering.
△ Less
Submitted 1 July, 2020;
originally announced July 2020.
-
Network Capabilities for the HL-LHC Era
Authors:
Marian Babik,
Shawn McKee
Abstract:
High Energy Physics (HEP) experiments rely on the networks as one of the critical parts of their infrastructure both within the participating laboratories and sites as well as globally to interconnect the sites, data centers and experiments instrumentation. Network virtualisation and programmable networks are two key enablers that facilitate agile, fast and more economical network infrastructures…
▽ More
High Energy Physics (HEP) experiments rely on the networks as one of the critical parts of their infrastructure both within the participating laboratories and sites as well as globally to interconnect the sites, data centers and experiments instrumentation. Network virtualisation and programmable networks are two key enablers that facilitate agile, fast and more economical network infrastructures as well as service development, deployment and provisioning.Adoption of these technologies by HEP sites and experiments will allow them to design more scalable and robust networks while decreasing the overall cost and improving the effectiveness of the resource utilization.The primary challenge we currently face is ensuring that WLCG and its constituent collaborations will have the networking capabilities required to most effectively exploit LHC data for the lifetime of the LHC. In this paper we provide a high level summary of the HEPiX NFV Working Group report that explored some of the novel network capabilities that could potentially be deployment in time for HL-LHC.
△ Less
Submitted 1 July, 2020;
originally announced July 2020.
-
A Roadmap for HEP Software and Computing R&D for the 2020s
Authors:
Johannes Albrecht,
Antonio Augusto Alves Jr,
Guilherme Amadio,
Giuseppe Andronico,
Nguyen Anh-Ky,
Laurent Aphecetche,
John Apostolakis,
Makoto Asai,
Luca Atzori,
Marian Babik,
Giuseppe Bagliesi,
Marilena Bandieramonte,
Sunanda Banerjee,
Martin Barisits,
Lothar A. T. Bauerdick,
Stefano Belforte,
Douglas Benjamin,
Catrin Bernius,
Wahid Bhimji,
Riccardo Maria Bianchi,
Ian Bird,
Catherine Biscarat,
Jakob Blomer,
Kenneth Bloom,
Tommaso Boccali
, et al. (285 additional authors not shown)
Abstract:
Particle physics has an ambitious and broad experimental programme for the coming decades. This programme requires large investments in detector hardware, either to build new facilities and experiments, or to upgrade existing ones. Similarly, it requires commensurate investment in the R&D of software to acquire, manage, process, and analyse the shear amounts of data to be recorded. In planning for…
▽ More
Particle physics has an ambitious and broad experimental programme for the coming decades. This programme requires large investments in detector hardware, either to build new facilities and experiments, or to upgrade existing ones. Similarly, it requires commensurate investment in the R&D of software to acquire, manage, process, and analyse the shear amounts of data to be recorded. In planning for the HL-LHC in particular, it is critical that all of the collaborating stakeholders agree on the software goals and priorities, and that the efforts complement each other. In this spirit, this white paper describes the R&D activities required to prepare for this software upgrade.
△ Less
Submitted 19 December, 2018; v1 submitted 18 December, 2017;
originally announced December 2017.
-
Data processing model for the CDF experiment
Authors:
J. Antos,
M. Babik,
D. Benjamin,
S. Cabrera,
A. W. Chan,
Y. C. Chen,
M. Coca,
B. Cooper,
S. Farrington,
K. Genser,
K. Hatakeyama,
S. Hou,
T. L. Hsieh,
B. Jayatilaka,
S. Y. Jun,
A. V. Kotwal,
A. C. Kraan,
R. Lysak,
I. V. Mandrichenko,
P. Murat,
A. Robson,
P. Savard,
M. Siket,
B. Stelzer,
J. Syu
, et al. (5 additional authors not shown)
Abstract:
The data processing model for the CDF experiment is described. Data processing reconstructs events from parallel data streams taken with different combinations of physics event triggers and further splits the events into datasets of specialized physics datasets. The design of the processing control system faces strict requirements on bookkeeping records, which trace the status of data files and…
▽ More
The data processing model for the CDF experiment is described. Data processing reconstructs events from parallel data streams taken with different combinations of physics event triggers and further splits the events into datasets of specialized physics datasets. The design of the processing control system faces strict requirements on bookkeeping records, which trace the status of data files and event contents during processing and storage. The computing architecture was updated to meet the mass data flow of the Run II data collection, recently upgraded to a maximum rate of 40 MByte/sec. The data processing facility consists of a large cluster of Linux computers with data movement managed by the CDF data handling system to a multi-petaByte Enstore tape library. The latest processing cycle has achieved a stable speed of 35 MByte/sec (3 TByte/day). It can be readily scaled by increasing CPU and data-handling capacity as required.
△ Less
Submitted 9 June, 2006; v1 submitted 5 June, 2006;
originally announced June 2006.
-
Data production models for the CDF experiment
Authors:
J. Antos,
M. Babik,
D. Benjamin,
S. Cabrera,
A. W. Chan,
Y. C. Chen,
M. Coca,
B. Cooper,
K. Genser,
K. Hatakeyama,
S. Hou,
T. L. Hsieh,
B. Jayatilaka,
A. C. Kraan,
R. Lysak,
I. V. Mandrichenko,
A. Robson,
M. Siket,
B. Stelzer,
J. Syu,
P. K. Teng,
S. C. Timm,
T. Tomura,
E. Vataga,
S. A. Wolbers
, et al. (1 additional authors not shown)
Abstract:
The data production for the CDF experiment is conducted on a large Linux PC farm designed to meet the needs of data collection at a maximum rate of 40 MByte/sec. We present two data production models that exploits advances in computing and communication technology. The first production farm is a centralized system that has achieved a stable data processing rate of approximately 2 TByte per day.…
▽ More
The data production for the CDF experiment is conducted on a large Linux PC farm designed to meet the needs of data collection at a maximum rate of 40 MByte/sec. We present two data production models that exploits advances in computing and communication technology. The first production farm is a centralized system that has achieved a stable data processing rate of approximately 2 TByte per day. The recently upgraded farm is migrated to the SAM (Sequential Access to data via Metadata) data handling system. The software and hardware of the CDF production farms has been successful in providing large computing and data throughput capacity to the experiment.
△ Less
Submitted 5 June, 2006;
originally announced June 2006.
-
Data production of a large Linux PC Farm for the CDF experiment
Authors:
J. Antos,
M. Babik,
A. W. Chan,
Y. C. Chen,
S. Hou,
T. L. Hsieh,
R. Lysak,
I. V. Mandrichenko,
M. Siket,
J. Syu,
P. K. Teng,
S. C. Timm,
S. A. Wolbers,
P. Yeh
Abstract:
The data production farm for the CDF experiment is designed and constructed to meet the needs of the Run II data collection at a maximum rate of 20 MByte/sec during the run. The system is composed of a large cluster of personal computers (PCs) with a high-speed network interconnect and a custom design control system for the flow of data and the scheduling of tasks on this PC farm. The farm explo…
▽ More
The data production farm for the CDF experiment is designed and constructed to meet the needs of the Run II data collection at a maximum rate of 20 MByte/sec during the run. The system is composed of a large cluster of personal computers (PCs) with a high-speed network interconnect and a custom design control system for the flow of data and the scheduling of tasks on this PC farm. The farm explores and exploits advances in computing and communication technology. The data processing has achieved a stable production rate of approximately 2 TByte per day. The software and hardware of the CDF production farms has been successful in providing large computing and data throughput capacity to the experiment.
△ Less
Submitted 4 March, 2006;
originally announced March 2006.
