Published March 10, 2023
| Version v2.0
Software
Open
GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch
Description
With GPT-NeoX 2.0, we now support upstream DeepSpeed. This enables the use of new DeepSpeed features such as Curriculum Learning, Communication Logging, and Autotuning.
For any changes in upstream DeepSpeed that are fundamentally incompatible with GPT-NeoX 2.0, we do the following:
- Attempt to create a PR to upstream DeepSpeed
- Stage the PR on DeeperSpeed 2.x, so that there's always a DeepSpeed version that's guaranteed to work with GPT-Neox 2.x.
Therefore, we recommend using DeeperSpeed 2.x unless your use-case relies on a specific upstream DeepSpeed feature that we haven't merged into DeeperSpeed 2.x yet.
What's Changed- Mup Support in https://github.com/EleutherAI/gpt-neox/pull/704
- Bring deepspeed_main up-to-date in https://github.com/EleutherAI/gpt-neox/pull/746
- Latest DeepSpeed Support in https://github.com/EleutherAI/gpt-neox/pull/663
- Curriculum Learning Support in https://github.com/EleutherAI/gpt-neox/pull/695
- Autotuning Support in https://github.com/EleutherAI/gpt-neox/pull/739
Full Changelog: https://github.com/EleutherAI/gpt-neox/compare/v1.0...v2.0
Notes
If you use this software, please cite it using these metadata.
Files
EleutherAI/gpt-neox-v2.0.zip
Files
(302.5 kB)
| Name | Size | Download all |
|---|---|---|
|
md5:207e9a18b9547858f31b0cb734d3a675
|
302.5 kB | Preview Download |
Additional details
Related works
- Is supplement to
- https://github.com/EleutherAI/gpt-neox/tree/v2.0 (URL)