OpenGPT-X: Novel Architecture Exploration

The OpenGPT-X project is a German initiative with ten collaborators to build, train, and deploy a multilingual open- source language model. Models trained within the project will be used for pilot cases by industry partners and commercialized through the Gaia-X Federation. Due to the substantial memory and compute resources required for efficiently training large language models, high-performance computing systems such as JUWELS Booster1 are essential. This paper presents the results of the exploration of novel hardware architecture conducted within the scope of the project.