A System for Energy Measurement on Accelerators (SEMA)

The fastest supercomputer in 2016 consumed over 17MW for a mere 33 Petaflops of performance. This makes energy efficiency a crucial obstacle to overcome in order to make Exascale computing a reality. Over 50% of the performance of this supercomputer is due to the presence of accelerators or coprocessors that are designed to do certain kind of mathematical operations in an energy efficient manner. Currently, accelerators such as NVIDIA GPGPUs and Intel Xeon Phi dominate this market. However, with the addition of FPGAs from Altera and Xilinx in HPC clusters, more options for accelerators are becoming available. Furthermore, several EU funded projects, including the Centre of Excellence and FETHPC projects such as ESCAPE, READEX, ESiWACE investigate HPC systems consisting of accelerators for Exascale-related applications. The dominance of such devices for energy efficient computing makes it crucial to understand the relationship between power consumption, performance and application code in order to exploit them in the most effective manner.
The measurement of power consumption is supported at different levels of accuracy by different vendors through their platforms. However, they lack any standardized metrics between them making it difficult to compare them directly. Further, the accuracy they support is at a coarse level making it impossible to use the measurements to profile the application code and to extract insights that help the programmer.
At the Irish Center for High-End Computing (ICHEC), we have developed a System for Energy Measurement on Accelerators (SEMA) that allows measurement of any accelerator or any number of them to an accuracy of milliwatt and a resolution of millisecond. SEMA works based on the standard current shunt-based power measurement technique. Such a methodology is not new and there has been prior work done within a lab environment. SEMA embarks to be different and we focus equally on usability as much as technical feasibility. To meet this endeavor, we have come up with set of novel ideas that abstract the technical details and provide the users with a very simple interface to measure energy and power. This interface can also be used to profile very short regions of code within the application. This allows extraction of insights into the performance and power consumption of different pieces of code at a level not possible before, thereby leading to improvements in understanding the code behavior and optimizations. These can also be fed back to the design of better energy efficient accelerator architectures in the future.
The current SEMA system is capable of performing power measurements on three different accelerators within a single system. However, we need to expand this system to work in a cluster environment in order to test large parallel applications. This requires improving the SEMA hardware with better integration into the host system and better programming infrastructure to manage the myriad of sensors. In the future, we hope such a system could be integrated as a standard interface in supercomputing clusters.