D9.3.3: Report on prototypes evaluation

The objective of Work Package 9 task 3 is to assess and make recommendations to the PRACE RI for joint developments with industrial partners to develop highly energy efficient HPC components and systems, as well as power and cooling technologies. WP9 has carried out this task through evaluation of a number of prototypes targeting novel approaches to HPC server and system design with many prototypes having some degree of direct industry involvement or support.
Prototype efforts assessed the use of FPGAs for function acceleration, the use of CPUs for the mobile market and with a TDP about two orders of magnitude less than typical x86 CPUs for the HPC market, DSPs common for embedded systems and with a TDP about one order of magnitude less than x86 CPUs, the emerging heterogeneous CPUs integrating x86 and GPU cores, and traditional GPUs with a novel direct communication between GPUs via Infiniband between nodes. Two prototypes focused on novel approaches to scalability of I/O systems in support of Exascale systems and their energy efficiency. Technologies assessed included integration of I/O nodes into the MPP or cluster interconnect fabric, the use of flash technology, scalable disk systems and virtual tape libraries based on disk systems with spun down idle disks. Data management in file systems, in particular the management of large numbers of small files, was also addressed with the I/O-prototypes. One prototype evaluation assessed the issues and benefits of integrated cooling solutions for hot water cooling.
The findings of the evaluations of prototypes looking at HPC server architectures is that 1) an optimized FPGA implementation of matrix-multiplication can offer a 5 – 10 times higher energy efficiency than an x86 software solution, 2) an optimized implementation of matrix multiplication on the DSP can yield about half the energy efficiency gain of an FPGA implementation, 3) the first and second generation x86+GPU CPUs are not competitive in regards to energy efficiency even with standard x86 CPUs for the functions investigated, and 4) that good scalability can be achieved for clusters based on nodes using mobile CPUs though the floating-point capabilities of the current generation mobile CPUs is insufficient to be competitive in terms of energy efficiency.
Software optimization can make an order of magnitude or more difference in efficiency and energy efficiency for kernels frequently used for HPC benchmarking even for HPC established architectures, such as the x86. Further efforts are required to better understand the energy advantages of novel approaches to HPC server designs.
The evaluation of the I/O prototypes largely gave the expected outcomes, but did expose the dependence on good software implementations and the efficiency and scalability issues related to large file systems in which a large fraction of the files are small.
The integrated cooling with hot water prototype demonstrated that the technology is viable, but that further study is necessary to assess the economic benefits, and that those likely depend on the local situation. That could however change if technology would become available to convert heat into electricity in a cost competitive way.