BIRCO Dataset
Authors/Creators
Description
BIRCO is a collection of existing Information Retrieval datasets after carefull curation to make it suitable for Large Language Model (LLM) based systems evaluation. Here are the references for each of the 5 datasets used in BIRCO:
1. DORIS-MAE: Wang, Jianyou Andre, et al. "Scientific document retrieval using multi-level aspect-based queries." Advances in Neural Information Processing Systems 36 (2024). (https://proceedings.neurips.cc/paper_files/paper/2023/hash/78f9c04bdcb06f1ada3902912d8b64ba-Abstract-Datasets_and_Benchmarks.html)
2. ArguAna: Wachsmuth, Henning, Shahbaz Syed, and Benno Stein. "Retrieval of the best counterargument without prior topic knowledge." Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018. (https://aclanthology.org/P18-1023/)
3. WhatThatBook: Lin, Kevin, et al. "Decomposing Complex Queries for Tip-of-the-tongue Retrieval." arXiv preprint arXiv:2305.15053 (2023). (https://arxiv.org/abs/2305.15053)
4. Clinical-Trial: Koopman, Bevan, and Guido Zuccon. "A test collection for matching patients to clinical trials." Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 2016. (https://dl.acm.org/doi/abs/10.1145/2911451.2914672)
5. RELIC: Thai, Katherine, et al. "RELiC: Retrieving Evidence for Literary Claims." Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022. (https://aclanthology.org/2022.acl-long.517/)
The dataset is stored as a json format. The structure of the file is as follows in python dict:
├── ada_embedding_for_datasets_v1.pickle
│
├── "doris-mae"
│ ├── "query" (60 queries)
│ │ ├── query_id_1: "query text 1"
│ │ ├── query_id_2: "query text 2"
│ │ └── query_id_3: "query text 3"
│ │ ...
│ ├── "corpus" (5543 paper abstracts)
│ │ ├── corpus_id_1: "corpus text 1"
│ │ ├── corpus_id_2: "corpus text 2"
│ │ └── corpus_id_3: "corpus text 3"
│ │ ...
│ └── "qrel" (avg. candidate pool size: 110.55)
│ ├── query_id_1
│ │ ├── corpus_id_1: relevance_score (rational number between 0-2)
│ │ ├── corpus_id_2: relevance_score
│ │ └── corpus_id_3: relevance_score
│ │ ...
│ ├── query_id_2
│ │ ├── corpus_id_1: relevance_score
│ │ ├── corpus_id_2: relevance_score
│ │ └── corpus_id_3: relevance_score
│ │ ...
│ └── query_id_3
│ ├── corpus_id_1: relevance_score
│ ├── corpus_id_2: relevance_score
│ └── corpus_id_3: relevance_score
│ ...
│
├── "arguana"
│ ├── "query" (100 queries)
│ │ ├── query_id_1: "query text 1"
│ │ ├── query_id_2: "query text 2"
│ │ └── query_id_3: "query text 3"
│ │ ...
│ ├── "corpus" (3148 arguments)
│ │ ├── corpus_id_1: "corpus text 1"
│ │ ├── corpus_id_2: "corpus text 2"
│ │ └── corpus_id_3: "corpus text 3"
│ │ ...
│ └── "qrel" (avg. candidate pool size: 50.01)
│ ├── query_id_1
│ │ ├── corpus_id_1: relevance_score (either 0 or 1)
│ │ ├── corpus_id_2: relevance_score
│ │ └── corpus_id_3: relevance_score
│ │ ...
│ ├── query_id_2
│ │ ├── corpus_id_1: relevance_score
│ │ ├── corpus_id_2: relevance_score
│ │ └── corpus_id_3: relevance_score
│ │ ...
│ └── query_id_3
│ ├── corpus_id_1: relevance_score
│ ├── corpus_id_2: relevance_score
│ └── corpus_id_3: relevance_score
│ ...
│
├── "wtb"
│ ├── "query" (100 queries)
│ │ ├── query_id_1: "query text 1"
│ │ ├── query_id_2: "query text 2"
│ │ └── query_id_3: "query text 3"
│ │ ...
│ ├── "corpus" (1767 book descriptions)
│ │ ├── corpus_id_1: "corpus text 1"
│ │ ├── corpus_id_2: "corpus text 2"
│ │ └── corpus_id_3: "corpus text 3"
│ │ ...
│ └── "qrel" (avg. candidate pool size: 50.43)
│ ├── query_id_1
│ │ ├── corpus_id_1: relevance_score (either 0 or 1)
│ │ ├── corpus_id_2: relevance_score
│ │ └── corpus_id_3: relevance_score
│ │ ...
│ ├── query_id_2
│ │ ├── corpus_id_1: relevance_score
│ │ ├── corpus_id_2: relevance_score
│ │ └── corpus_id_3: relevance_score
│ │ ...
│ └── query_id_3
│ ├── corpus_id_1: relevance_score
│ ├── corpus_id_2: relevance_score
│ └── corpus_id_3: relevance_score
│ ...
│
├── "clinical-trial" (avg. candidate pool size )
│ ├── "query" (50 queries)
│ │ ├── query_id_1: "query text 1"
│ │ ├── query_id_2: "query text 2"
│ │ └── query_id_3: "query text 3"
│ │ ...
│ ├── "corpus" (3256 clinical trial descriptions)
│ │ ├── corpus_id_1: "corpus text 1"
│ │ ├── corpus_id_2: "corpus text 2"
│ │ └── corpus_id_3: "corpus text 3"
│ │ ...
│ └── "qrel" (avg. candidate pool size: 68.40)
│ ├── query_id_1
│ │ ├── corpus_id_1: relevance_score (0, 1, or 2)
│ │ ├── corpus_id_2: relevance_score
│ │ └── corpus_id_3: relevance_score
│ │ ...
│ ├── query_id_2
│ │ ├── corpus_id_1: relevance_score
│ │ ├── corpus_id_2: relevance_score
│ │ └── corpus_id_3: relevance_score
│ │ ...
│ └── query_id_3
│ ├── corpus_id_1: relevance_score
│ ├── corpus_id_2: relevance_score
│ └── corpus_id_3: relevance_score
│ ...
│
└── "relic"
├── "query" (100 queries)
│ ├── query_id_1: "query text 1"
│ ├── query_id_2: "query text 2"
│ └── query_id_3: "query text 3"
│ ...
├── "corpus" (5017 quotations from books)
│ ├── corpus_id_1: "corpus text 1"
│ ├── corpus_id_2: "corpus text 2"
│ └── corpus_id_3: "corpus text 3"
│ ...
└── "qrel" (avg. candidate pool size: 50.59)
├── query_id_1
│ ├── corpus_id_1: relevance_score (either 0 or 1)
│ ├── corpus_id_2: relevance_score
│ └── corpus_id_3: relevance_score
│ ...
├── query_id_2
│ ├── corpus_id_1: relevance_score
│ ├── corpus_id_2: relevance_score
│ └── corpus_id_3: relevance_score
│ ...
└── query_id_3
├── corpus_id_1: relevance_score
├── corpus_id_2: relevance_score
└── corpus_id_3: relevance_score
...
Files
BIRCO_dataset.json
Files
(20.1 MB)
| Name | Size | Download all |
|---|---|---|
|
md5:548cad5d25ce8c0714274ba0ec17fa78
|
20.1 MB | Preview Download |