Info: Zenodo’s user support line is staffed on regular business days between Dec 23 and Jan 5. Response times may be slightly longer than normal.

ETP4HPC
Published April 11, 2022 | Version v1
Report Open

Federated HPC, Cloud and Data infrastructures

Creators

  • 1. HPE
  • 2. Seagate
  • 3. UPM
  • 4. SICOS

Description

An increasing interest is observed in making a diversity of compute and storage resources, which are geographic spread, available in a federated manner. A common services layer can facilitate easier access, more elasticity as well as lower response times, and improved utilisation of the underlying resources. In this white paper, current trends are analysed both from an infrastructure provider as well as an end-user perspective. Here the focus is on federated e-infrastructures that among others include high-performance computing (HPC) systems as compute resources. Two initiatives, namely Fenix and GAIA-X, are presented as illustrative examples. Based on a more detailed exploration of selected topical areas related to federated e-infrastructures, various R&I challenges are identified and recommendations for further efforts formulated.

Files

ETP4HPC_WP_Federated-HPC-cloud-date-infra_final.pdf

Files (1.9 MB)

Additional details

References

  • [AARC2019] Nicolas Liampotis et al., "AARC Blueprint Architecture 2019", 2019 (DOI: 10.5281/zenodo.3672785)
  • [Akram2021] Ayaz Akram et al., "Performance Analysis of Scientific Computing Workloads on General Purpose TEEs", IPDPS, 2021 (DOI: 10.1109/IPDPS49936.2021.00115)
  • [Al-Mamun2019] Abdullah Al-Mamun et al., "HPChain: An MPI-Based Blockchain Framework for Data Fidelity in High-Performance Computing Systems", Supercomputing, 2019 (https://sc19.supercomputing.org/proceedings/tech_poster/poster_files/rpost106s2-file3.pdf)
  • [Alam2020] Sadaf Alam et al., "Archival Data Repository Services to Enable HPC and Cloud Workflows in a Federated Research e-Infrastructure", IEEE/ACM International Workshop on Interoperability of Supercomputing and Cloud Technologies, 2020 (DOI: 10.1109/SuperCompCloud51944.2020.00012)
  • [Boissonneault2016] Maxime Boissonneault et al., "Providing a Unified Software Environment for Canada's National Advanced Computing Centers", PEARC'19, 2019 (DOI 10.1145/3332186.3332210)
  • [Catlett2005] Charles Catlett et al., "TeraGrid: Analysis of Organization, System Architecture, and Middleware Enabling New Types of Applications", in: L. Grandinetti (Ed.), "High-Performance Computing and Grids in Action", IOS Press, 2008.
  • [Coombs2020] C. Coombs, D. Hislop, S. K. Taneva and S. Barnard, "The strategic impacts of Intelligent Automation for knowledge and service work: An interdisciplinary review", The Journal of Strategic Information Systems, 2020 (DOI: 10.1016/j.jsis.2020.101600)
  • [Dang2019] Y. Dang, Q. Lin and P. Huang, "AIOps: Real-World Challenges and Research Innovations", IEEE/ACM 41st International Conference on Software Engineering, 2019 (DOI: 10.1109/ICSE-Companion.2019.00023)
  • [Desbiens2021] Frédéric Desbiens, "EdgeOps for IoT: A Vision For Edge", Connected World, 2021 (https://connectedworld.com/edgeops-for-iot-a-vision-for-edge/)
  • [Dona2021] Rizart Dona, Riccardo Di Maria (ESCAPE project), "The ESCAPE Data Lake: The machinery behind testing, monitoring and supporting a unified federated storage infrastructure of the exabyte-scale", CHEP 2021, EPJ Web of Conferences, 2021 (DOI: 10.1051/epjconf/202125102060)
  • [ELIXIR] ELIXIR Compute Platform, https://elixir-europe.org/platforms/compute (access 27.03.2022).
  • [EOSC2021] European Commission, Directorate-General for Research and Innovation, Vaghetti, D., Kanellopoulos, C., Johansson, L., et al., "EOSC Authentication and Authorization Infrastructure (AAI): report from the EOSC Executive Board Working Group (WG) Architecture AAI Task Force (TF)", Publications Office, 2021 (https://data.europa.eu/doi/10.2777/8702)
  • [EU2020] European Commission, "Equipping Europe for world-class High-Performance Computing in the next decade", SWD(2020) 179 final, https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:52020SC0179&rid=9.
  • [Fenix2021] Fenix, "Fenix Strategy Document", 2021 (https://fenix-ri.eu/sites/default/files/public/file-uploads/Fenix%20Strategy%20Document%20202111111-public.pdf)
  • [Gaia-X2021] Gaia-X, "Gaia-X Architecture Document", 21.09 release, 2021 (https://www.gaia-x.eu/sites/default/files/2021-10/Gaia-X_Architecture_Document_2109.pdf)
  • [Gentzsch2011] Wolfgang Gentzsch et al., "DEISA - Distributed European Infrastructure for Supercomputing Applications", J Grid Computing, 2011 (DOI: 10.1007/s10723-011-9183-2)
  • [Gesing2018] S. Gesing, "Science Gateways in HPC: Usability Meets Efficiency and Effectiveness" In: Modelling and Simulation in HPC and Cloud Systems, Book Series "Studies in Big Data" Print ISBN: 978-3-319-73766-9, Electronic ISBN: 978-3-319-73767-6, Springer International Publishing, 2018
  • [Henze2017] Martin Henze et al., "Practical Data Compliance for Cloud Storage", IEEE International Conference on Cloud Engineering (IC2E), 2017 (DOI: 10.1109/IC2E.2017.32)
  • [Matsuoka2005] Satoshi Matsuoka et al., "Japanese Computational Grid Research Project: NAREGI", Proceedings of the IEEE, Vol. 93, Issue 3, 2005 (DOI: 10.1109/JPROC.2004.842748)
  • [Marazakis2022] Manolis Marazakis et al., "HPC for urgent decision making", ETP4HPC White Paper, 2022 (DOI: 10.5281/zenodo.6107362)
  • [Netto2019] Marco A. S. Netto et al., "HPC cloud for scientific and business applications: taxonomy, vision, and research challenges", ACM Computing Surveys (CSUR) 51 (1), 1-29, 2019 (DOI: 10.1145/3150224)
  • [Peisert2021] Sean Peisert, "Security Trustworthy Scientific Computing", Communications of the ACM, 2021 (DOI: 10.1145/3457191)
  • [Souppaya2017] Murugiah Souppaya, John Morello, Karen Scarfone, "Application Container Security Guide", NIST Special Publication 800-190, 2017 (DOI: 10.6028/NIST.SP.800-190)
  • [Swinbank2021] John D. Swinbank et al. (ESCAPE), "D5.3 - Performance Assessment of Initial Science Platform Prototype", 2021 (https://projectescape.eu/sites/default/files/ESCAPE-D5_3.pdf)