Einsum Benchmark: Enabling the Development of Next-Generation Tensor Execution Engines

Modern artificial intelligence and machine learning workflows rely on efficient tensor libraries. However, tuning tensor libraries without considering the actual problems they are meant to execute can lead to a mismatch between expected performance and the actual performance. Einsum libraries are tuned to efficiently execute tensor expressions with only a few, relatively large, dense, floating-point tensors. But, practical applications of einsum cover a much broader range of tensor expressions than those that can currently be executed efficiently. For this reason, we have created a benchmark dataset that encompasses this broad range of tensor expressions, allowing future implementations of einsum to build upon and be evaluated against. In addition, we also provide generators for einsum expression and converters to einsum expressions in our repository, so that additional data can be generated as needed. The benchmark dataset, the generators and converters are released openly and are publicly available at https://benchmark.einsum.org.

The broader data collection process included contributions from individuals whose data was transformed. We duly acknowledge the following for making their data publicly available:

• Fichte, Johannes; Hecher, Markus; Florim Hamiti: Model Counting Competition 2020
• Fichte, Johannes; Hecher, Markus: Model Counting Competition 2021 2022 2023
• Fichte, Johannes; Hecher, Markus; Woltran, Stefan; Zisser, Markus: A Benchmark Collection of #SAT Instances and Tree Decompositions
• Meel, Kuldeep S.: Model Counting and Uniform Sampling Instances
• Automated Reasoning Group at the University of California, Irvine: UAI Competitions
• Martinis, John M. et al.: Quantum supremacy using a programmable superconducting processor Dataset. Dryad.

Moreover, we thank the following authors of open source software used to generated instances:

• Gray, Johnnie: quimb, cotengra
• Soos, Mate, Meel, Kuldeep S: Arjun
• Stoian, Mihail: Netzwerk
• Liu, Jinguo; Lua, Xiuzhe; Wang, Lei: Yao.jl
• Liu, Jinguo: YaoToEinsum.jl