There is a newer version of the record available.

Published October 10, 2023 | Version v0.3.2
Software Open

stanford-futuredata/megablocks: v0.3.2

Creators

  • 1. MosaicML
  • 2. Databricks
  • 3. Stanford '22
  • 4. NVIDIA

Description

What's Changed
  • Support for bfloat16
  • Optimizations for top_k > 1
  • Support for fully-sharded data parallelism
  • Support tensor model parallelism when expert_parallel_world_size > num_experts
  • Optimizations for activation memory
  • Support activation quantization (thanks @dblalock!)
  • Optimizations for SM90 (Hopper)
  • Lots of bug fixes, cleanup and small optimizations
New Contributors

Full Changelog: https://github.com/stanford-futuredata/megablocks/compare/v0.1...v0.3.2

Files

stanford-futuredata/megablocks-v0.3.2.zip

Files (1.1 MB)

Name Size Download all
md5:c6f223a848edb0b4359c4e4ca31a6781
1.1 MB Preview Download

Additional details

Related works

Is supplement to
https://github.com/stanford-futuredata/megablocks/tree/v0.3.2 (URL)