Webis Rank-DistiLLM

This dataset contains run files for training queries from MS MARCO passage re-ranked by RankGPT-4 Turbo or RankZephyr. These run files can be used to distill smaller and more efficient models while upholding effectiveness.

The files __colbert__msmarco-passage-train-judged.run and __bm25__msmarco-passage-train-judged.run contain the top 500 passages for all queries that have at least one relevance judgement in the MS MARCO training query set retrieved by ColBERTv2 and BM25 respectively.

All other files are sub-sampled from these run files and re-ranked by either RankGPT-4 Turbo or RankZephyr. A file's name reveals which LLM was used for re-ranking, which first-stage retrieval model was used, how many queries were re-ranked, and to which depth the rankings were sampled. For example, the file __rankzephyr-colbert-10000-sampled-100__msmarco-passage-train-judged.run contains the top 100 passages retrieved by ColBERTv2 for 10,000 queries.