Planned intervention: On Wednesday April 3rd 05:30 UTC Zenodo will be unavailable for up to 2-10 minutes to perform a storage cluster upgrade.
Published May 1, 2013 | Version v1
Working paper Open

Lessons Learned about Metadata Performance in the PRACE File System Prototype

  • 1. Barcelona Supercomputing Center, Barcelona, Spain

Contributors

  • 1. Barcelona Supercomputing Center, Barcelona, Spain ; Universitat Politècnica de Catalunya, Barcelona, Spain

Description

The purpose of this white paper is to document the measurements obtained in PRACE-2IP regarding file system metadata performance, and to assess mechanisms to further improve such performance. The final goal of the task is to identify the open issues related to file systems for multi-petascale and exascale facilities, and propose novel solutions that can be applied to Lustre, enabling it to manage a huge number of files on a system with many heterogeneous devices while efficiently delivering huge data bandwidth and low latency, minimizing the response time.
The performed tasks being reported included the observation, measurement and study of a large scale system currently in production, in order to identify the key metadata-related issues; the development of a prototype aimed to improve the metadata behaviour in such system and also to provide a framework to easily deploy novel metadata management techniques on top of other systems; the measurement and study of specially deployed Lustre and GPFS prototypes to validate the presence of the metadata issues observed in current in-production systems; and finally the porting of the framework prototype to test novel metadata management techniques on both a production system using GPFS and the PRACE Lustre prototype facility at CEA.

Files

WP66.pdf

Files (661.1 kB)

Name Size Download all
md5:a10875f3600f6ebf1b26f8acbde85eac
661.1 kB Preview Download

Additional details

Funding

PRACE-2IP – PRACE - Second Implementation Phase Project 283493
European Commission