Pairwise Multi-Class Document Classification for Semantic Relations between Wikipedia Articles (Dataset, Models & Code)

Many digital libraries recommend literature to their users considering the similarity between a query document and their repository. However, they often fail to distinguish what is the relationship that makes two documents alike. In this paper, we model the problem of finding the relationship between two documents as a pairwise document classification task. To find the semantic relation between documents, we apply a series of techniques, such as GloVe, Paragraph-Vectors, BERT, and XLNet under different configurations (e.g., sequence length, vector concatenation scheme), including a Siamese architecture for the Transformer-based systems. We perform our experiments on a newly proposed dataset of 32,168 Wikipedia article pairs and Wikidata properties that define the semantic document relations. Our results show vanilla BERT as the best performing system with an F1-score of 0.93,
which we manually examine to better understand its applicability to other domains. Our findings suggest that classifying semantic relations between documents is a solvable task and motivates the development of recommender systems based on the evaluated techniques. The discussions in this paper serve as first steps in the exploration of documents through SPARQL-like queries such that one could find documents that are similar in one aspect but dissimilar in another.

Additional information can be found on GitHub.

The following data is supplemental to the experiments described in our research paper. The data consists of:

• Datasets (articles, class labels, cross-validation splits)
• Pretrained models (Transformers, GloVe, Doc2vec)
• Model output (prediction) for the best performing models

Dataset

The Wikipedia article corpus is available in enwiki-20191101-pages-articles.weighted.10k.jsonl.bz2. The original data have been downloaded as XML dump, and the corresponding articles were extracted as plain-text with gensim.scripts.segment_wiki. The archive contains only articles that are available in training or test data.

The actual dataset is provided as used in the stratified k-fold with k=4 in train_testdata__4folds.tar.gz.

├── 1
│   ├── test.csv
│   └── train.csv
├── 2
│   ├── test.csv
│   └── train.csv
├── 3
│   ├── test.csv
│   └── train.csv
└── 4
    ├── test.csv
    └── train.csv

4 directories, 8 files

Pretrained models

PyTorch: vanilla and Siamese BERT + XLNet

Pretrained model for each fold is available in the corresponding model archives:

# Vanilla
model_wiki.bert_base__joint__seq512.tar.gz
model_wiki.xlnet_base__joint__seq512.tar.gz

# Siamese
model_wiki.bert_base__siamese__seq512__4d.tar.gz
model_wiki.xlnet_base__siamese__seq512__4d.tar.gz