There is a newer version of the record available.

Published October 15, 2024 | Version v1.3.0
Software Open

PKU-YuanGroup/Open-Sora-Plan: Release v1.3.0

Authors/Creators

  • 1. @PKU-YuanGroup
  • 2. 🤡 School
  • 3. PKU-Yuan-Lab
  • 4. Huazhong University of Science and Technology
  • 5. Peking University
  • 6. Tencent
  • 7. @huawei-noah
  • 8. Sun Yat-sen University
  • 9. The Chinese University of Hong Kong
  • 10. flutterflow.io
  • 11. University of Science and Technology of China

Description

In version 1.3.0, Open-Sora-Plan introduced the following five key features:

  1. A more powerful and cost-efficient WFVAE. We decompose video into several sub-bands using wavelet transforms, naturally capturing information across different frequency domains, leading to more efficient and robust VAE learning.
  2. Prompt Refiner. A large language model designed to refine short text inputs.
  3. High-quality data cleaning strategy. The cleaned panda70m dataset retains only 27% of the original data.
  4. DiT with new sparse attention. A more cost-effective and efficient learning approach.
  5. Dynamic resolution and dynamic duration. This enables more efficient utilization of videos with varying lengths (treating a single frame as an image).

For further details, please refer to our report.

  • COMING SOON ⚡️⚡️⚡️ For large model parallelisation training, TP & SP and more strategies are coming...

    近期将新增华为昇腾多模态MindSpeed-MM分支,借助华为MindSpeed-MM套件的能力支撑Open-Sora Plan参数的扩增,为更大参数规模的模型训练提供TP、SP等分布式训练能力。

Files

PKU-YuanGroup/Open-Sora-Plan-v1.3.0.zip

Files (617.6 kB)

Name Size Download all
md5:2bc35ab1895930b1a20edf1559f063d4
617.6 kB Preview Download

Additional details

Related works

Is supplement to
Software: https://github.com/PKU-YuanGroup/Open-Sora-Plan/tree/v1.3.0 (URL)