SHAPE Project Open Ocean: High Performance Processing Chain - faster on‐line statistics calculation

Open Ocean is a French SME company which develops innovative on-line solutions to help plan and manage offshore developments. They conceived an oceanographic data study tool which computes and formats data (Pre-Processing and Processing) and which provides relevant oceanographic information to industrial marine companies (Post-Processing) through a web interface. But the “time-to-solution” of this post-processing step is too long and hence not compatible with industrial use. Therefore, the goal of this SHAPE project is to improve post-processing by optimising a parallelized Python program of Open Ocean which processes and computes statistics (e.g. wind speed) on big datasets. To carry this out, engineers of Open Ocean and IDRIS worked together to optimise this program by using resources available in a national supercomputing centre: high performance parallel machine and parallel file system (GPFS, 100 GB/S bandwidth).
This paper describes the parallelisation process implemented by Open Ocean and its porting on the Ada machine (IBM cluster of Intel E5-4650 processors, 332 compute nodes) at IDRIS. It also covers performance testing and identification of the bottleneck in the execution.