HPL(Linpack) and Other Popular Benchmarks on Heterogeneous HPC Clusters

This talk presents a practitioner's guide to running HPL and related benchmarks on heterogeneous campus HPC clusters, drawing on ASU's November 2025 Top500 (#484, 2.62 PFlops) and Green500 (#77) submissions on the Sol cluster. We walk through the full Top500/Green500 submission workflow using NVIDIA's HPC-Benchmarks container — covering EEHPCWG power measurement levels, single-node validation, the UCX/NVSHMEM/NCCL environment-variable soup, GPU clock pinning, and iterative N/NB/P×Q parameter sweeps — and report a counterintuitive negative result: adding three H100 nodes to a 60-node A100 partition dropped Rmax by roughly one petaflop, a loss that simple load-imbalance models cannot fully explain and that no parameter tuning recovered. We then survey the broader benchmark landscape (HPCG, STREAM, NCCL tests, MLPerf HPC, IO500) and introduce a pairwise OSU micro-benchmark methodology using random disjoint node pairs, which provides O(N)-cost per-iteration fabric health coverage that catches intermittent cable and NIC failures invisible to cluster-aggregate NCCL tests or static node-health checks. The talk closes with operational how-tos for LBNL NHC and IO500, and recommends homogeneous-partition submissions plus SLATE evaluation as the near-term path for mixed-generation clusters.