-
Deployment of Elastic Virtual Hybrid Clusters Across Cloud Sites
Authors:
Miguel Caballer,
Marica Antonacci,
Zdeněk Šustr,
Michele Perniola,
Germán Moltó
Abstract:
Virtual clusters are widely used computing platforms than can be deployed in multiple cloud platforms. The ability to dynamically grow and shrink the number of nodes has paved the way for customised elastic computing both for High Performance Computing and High Throughput Computing workloads. However, elasticity is typically restricted to a single cloud site, thus hindering the ability to provisio…
▽ More
Virtual clusters are widely used computing platforms than can be deployed in multiple cloud platforms. The ability to dynamically grow and shrink the number of nodes has paved the way for customised elastic computing both for High Performance Computing and High Throughput Computing workloads. However, elasticity is typically restricted to a single cloud site, thus hindering the ability to provision computational resources from multiple geographically distributed cloud sites. To this aim, this paper introduces an architecture of open-source components that coherently deploy a virtual elastic cluster across multiple cloud sites to perform large-scale computing. These hybrid virtual elastic clusters are automatically deployed and configured using an Infrastructure as Code (IaC) approach on a distributed hybrid testbed that spans different organizations, including on-premises and public clouds, supporting automated tunneling of communications across the cluster nodes with advanced VPN topologies. The results indicate that cluster-based computing of embarrassingly parallel jobs can benefit from hybrid virtual clusters that aggregate computing resources from multiple cloud back-ends and bring them together into a dedicated, albeit virtual network.
The work presented in this article has been partially funded by the European Union's (EU) Horizon 2020 research project DEEP Hybrid-DataCloud (grant agreement No 777435).
△ Less
Submitted 17 February, 2021;
originally announced February 2021.
-
Orchestrating Complex Application Architectures in Heterogeneous Clouds
Authors:
Miguel Caballer,
Sahdev Zala,
Álvaro López García,
Germán Moltó,
Pablo Orviz Fernández,
Mathieu Velten
Abstract:
Private cloud infrastructures are now widely deployed and adopted across technology industries and research institutions. Although cloud computing has emerged as a reality, it is now known that a single cloud provider cannot fully satisfy complex user requirements. This has resulted in a growing interest in developing hybrid cloud solutions that bind together distinct and heterogeneous cloud infra…
▽ More
Private cloud infrastructures are now widely deployed and adopted across technology industries and research institutions. Although cloud computing has emerged as a reality, it is now known that a single cloud provider cannot fully satisfy complex user requirements. This has resulted in a growing interest in developing hybrid cloud solutions that bind together distinct and heterogeneous cloud infrastructures. In this paper we describe the orchestration approach for heterogeneous clouds that has been implemented and used within the INDIGO-DataCloud project. This orchestration model uses existing open-source software like OpenStack and leverages the OASIS Topology and Specification for Cloud Applications (TOSCA) open standard as the modeling language. Our approach uses virtual machines and Docker containers in an homogeneous and transparent way providing consistent application deployment for the users. This approach is illustrated by means of two different use cases in different scientific communities, implemented using the INDIGO-DataCloud solutions.
△ Less
Submitted 9 November, 2017;
originally announced November 2017.
-
INDIGO-DataCloud:A data and computing platform to facilitate seamless access to e-infrastructures
Authors:
INDIGO-DataCloud Collaboration,
:,
Davide Salomoni,
Isabel Campos,
Luciano Gaido,
Jesus Marco de Lucas,
Peter Solagna,
Jorge Gomes,
Ludek Matyska,
Patrick Fuhrman,
Marcus Hardt,
Giacinto Donvito,
Lukasz Dutka,
Marcin Plociennik,
Roberto Barbera,
Ignacio Blanquer,
Andrea Ceccanti,
Mario David,
Cristina Duma,
Alvaro López-García,
Germán Moltó,
Pablo Orviz,
Zdenek Sustr,
Matthew Viljoen,
Fernando Aguilar
, et al. (40 additional authors not shown)
Abstract:
This paper describes the achievements of the H2020 project INDIGO-DataCloud. The project has provided e-infrastructures with tools, applications and cloud framework enhancements to manage the demanding requirements of scientific communities, either locally or through enhanced interfaces. The middleware developed allows to federate hybrid resources, to easily write, port and run scientific applicat…
▽ More
This paper describes the achievements of the H2020 project INDIGO-DataCloud. The project has provided e-infrastructures with tools, applications and cloud framework enhancements to manage the demanding requirements of scientific communities, either locally or through enhanced interfaces. The middleware developed allows to federate hybrid resources, to easily write, port and run scientific applications to the cloud. In particular, we have extended existing PaaS (Platform as a Service) solutions, allowing public and private e-infrastructures, including those provided by EGI, EUDAT, and Helix Nebula, to integrate their existing services and make them available through AAI services compliant with GEANT interfederation policies, thus guaranteeing transparency and trust in the provisioning of such services. Our middleware facilitates the execution of applications using containers on Cloud and Grid based infrastructures, as well as on HPC clusters. Our developments are freely downloadable as open source components, and are already being integrated into many scientific applications.
△ Less
Submitted 5 February, 2019; v1 submitted 6 November, 2017;
originally announced November 2017.
-
INDIGO-Datacloud: foundations and architectural description of a Platform as a Service oriented to scientific computing
Authors:
D. Salomoni,
I. Campos,
L. Gaido,
G. Donvito,
M. Antonacci,
P. Fuhrman,
J. Marco,
A. Lopez-Garcia,
P. Orviz,
I. Blanquer,
M. Caballer,
G. Molto,
M. Plociennik,
M. Owsiak,
M. Urbaniak,
M. Hardt,
A. Ceccanti,
B. Wegh,
J. Gomes,
M. David,
C. Aiftimiei,
L. Dutka,
B. Kryza,
T. Szepieniec,
S. Fiore
, et al. (10 additional authors not shown)
Abstract:
In this paper we describe the architecture of a Platform as a Service (PaaS) oriented to computing and data analysis. In order to clarify the choices we made, we explain the features using practical examples, applied to several known usage patterns in the area of HEP computing. The proposed architecture is devised to provide researchers with a unified view of distributed computing infrastructures,…
▽ More
In this paper we describe the architecture of a Platform as a Service (PaaS) oriented to computing and data analysis. In order to clarify the choices we made, we explain the features using practical examples, applied to several known usage patterns in the area of HEP computing. The proposed architecture is devised to provide researchers with a unified view of distributed computing infrastructures, focusing in facilitating seamless access. In this respect the Platform is able to profit from the most recent developments for computing and processing large amounts of data, and to exploit current storage and preservation technologies, with the appropriate mechanisms to ensure security and privacy.
△ Less
Submitted 22 April, 2016; v1 submitted 31 March, 2016;
originally announced March 2016.
