Lessons Learned about Metadata Performance in the PRACE File System Prototype
Contributors
Other:
- 1. Barcelona Supercomputing Center, Barcelona, Spain ; Universitat Politècnica de Catalunya, Barcelona, Spain
Description
The purpose of this white paper is to document the measurements obtained in PRACE-2IP regarding file system metadata performance, and to assess mechanisms to further improve such performance. The final goal of the task is to identify the open issues related to file systems for multi-petascale and exascale facilities, and propose novel solutions that can be applied to Lustre, enabling it to manage a huge number of files on a system with many heterogeneous devices while efficiently delivering huge data bandwidth and low latency, minimizing the response time.
The performed tasks being reported included the observation, measurement and study of a large scale system currently in production, in order to identify the key metadata-related issues; the development of a prototype aimed to improve the metadata behaviour in such system and also to provide a framework to easily deploy novel metadata management techniques on top of other systems; the measurement and study of specially deployed Lustre and GPFS prototypes to validate the presence of the metadata issues observed in current in-production systems; and finally the porting of the framework prototype to test novel metadata management techniques on both a production system using GPFS and the PRACE Lustre prototype facility at CEA.
Files
WP66.pdf
Files
(661.1 kB)
Name | Size | Download all |
---|---|---|
md5:a10875f3600f6ebf1b26f8acbde85eac
|
661.1 kB | Preview Download |