MineDojo Internet Knowledge Base (YouTube)

Project website: minedojo.org

Paper: arxiv.org/abs/2206.08853

GitHub: github.com/MineDojo/MineDojo

Minecraft is among the most streamed games on YouTube. Human players have demonstrated a stunning range of creative activities and sophisticated missions that take hours to complete. We collect 730K+ narrated Minecraft videos, which add up to 33 years of duration and 2.2B words in English transcripts. The time-aligned transcripts enable the agent to ground free-form natural language in video pixels and learn the semantics of diverse activities without laborious human labeling.

There are two files in our YouTube knowledge base.

• youtube_tutorial.json (tutorial videos): 

  Minecraft tutorial videos include step-by-step demonstrations and sometimes detailed verbal explanations. They also serve as a rich source of creative missions that humans find interesting. We harvest thousands of tasks from these videos in our benchmarking suite. 

• youtube_full.json (general gameplay videos):

  Unlike tutorials, general gameplay videos do not necessarily provide guidance on particular tasks. Instead, they capture the “in-the-wild” human experiences that are much larger in quantity, diverse in contents, and rich in learning signals.

Data Structure

list[
    {
        "id": str,         # video id
        "title": str,      # video title
        "link": str,       # video link
        "view_count": int  # number of times the video has been viewed
        "like_count": int  # number of users who have indicated that they liked the video
        "duration": float  # video duration in seconds
        "fps": float,      # video FPS
    }
]

Check out our paper!

@article{fan2022minedojo,
  title = {MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge},
  author = {Linxi Fan and Guanzhi Wang and Yunfan Jiang and Ajay Mandlekar and Yuncong Yang and Haoyi Zhu and Andrew Tang and De-An Huang and Yuke Zhu and Anima Anandkumar},
  year = {2022},
  journal = {arXiv preprint arXiv: Arxiv-2206.08853}
}