Preprint Open Access
Efficient and Eventually Consistent Collective Operations
Iakymchuk, Roman;
Faustino, Amandio;
Emerson, Andrew;
Barreto, Joao;
Bartsch, Valeria;
Rodrigues, Rodrigo;
Monteiro, Jose
Collective operations are common features of parallel programming models that are frequently used in High-Performance (HPC) and machine/ deep learning (ML/ DL) applications. In strong scaling scenarios, collective operations can negatively impact the overall application performance: with the increase in core count, the load per rank decreases, while the time spent in collective operations increases logarithmically.
In this article, we propose a design of eventually consistent collectives suitable for ML/ DL computations by reducing communication in Broadcast and Reduce, as well as by exploring the Stale Synchronous Parallel (SSP) synchronization model for the Allreduce collective. Moreover, we also enrich the GASPI ecosystem with frequently used classic/ consistent collective operations -- such as Allreduce for large messages and AlltoAll used in an HPC code. Our implementations show promising preliminary results with significant improvements, especially for Allreduce and AlltoAll, compared to the vendor-provided MPI alternatives.
| Name | Size | |
|---|---|---|
|
iPDPSw__collectives-16.pdf
md5:c46e03010eefb07d87e1baca909e8604 |
549.8 kB | Download |
| All versions | This version | |
|---|---|---|
| Views | 249 | 249 |
| Downloads | 155 | 155 |
| Data volume | 85.2 MB | 85.2 MB |
| Unique views | 226 | 226 |
| Unique downloads | 151 | 151 |
- Publication date:
- March 8, 2021
- DOI:
-
- Keyword(s):
- Collectives Allreduce AlltoAll Stale Synchronous Parallel GASPI
- Grants:
-
European Commission:
- EPEEC - European joint Effort toward a Highly Productive Programming Environment for Heterogeneous Exascale Computing (EPEEC) (801051)
- License (for files):
- Creative Commons Attribution 4.0 International