Planned intervention: On Wednesday April 3rd 05:30 UTC Zenodo will be unavailable for up to 2-10 minutes to perform a storage cluster upgrade.

There is a newer version of the record available.

Published March 21, 2021 | Version 1.0
Software Open

GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow

Description

GPT-Neo is an implementation of model & data-parallel GPT-2 and GPT-3-like models, utilizing Mesh Tensorflow for distributed support. This codebase is designed for TPUs. It should also work on GPUs, though we do not recommend this hardware configuration.

Files

EleutherAI/gpt-neo-v1.1.zip

Files (86.6 kB)

Name Size Download all
md5:3b301a003caf2da94ae1c786c90d41d5
86.6 kB Preview Download

Additional details

Related works

Is cited by
Preprint: arXiv:2105.09938 (arXiv)
Preprint: arXiv:2107.13586 (arXiv)
Preprint: arXiv:2107.03374 (arXiv)
Preprint: arXiv:2107.06499 (arXiv)
Is supplement to
https://github.com/EleutherAI/gpt-neo/tree/v1.1 (URL)