Published October 11, 2026
| Version v1
Preprint
Open
Training an Entire 21B-Parameter Mixture-of-Experts Model on Each GPU
Description
This paper reports reaching about 49% MFU when pretraining a 21B-parameter MoE LLM on a single GH200 GPU
This paper does not claim to present a new algorithm or the best MFU. It is based on actual measurement data obtained using existing methods on the GH200.
Cited papers and equivalent methods are detailed in the paper.
Files
gh200_mfu_en.pdf
Additional details
Additional titles
- Subtitle (En)
- A Case Study of the Optimization Path to 49% MFU per GH200
Software
- Repository URL
- https://github.com/DW-dev-UE/LLM-from-scratch
- Programming language
- Python
- Development Status
- Active