Published July 31, 2026 | Version 1
Dataset Restricted

A Corpus of Asante Twi

Authors/Creators

  • 1. ROR icon Mangalam Research Center for Buddhist Languages
  • 2. Luminos Fund

Description

This is a ~14 million tokens preprocessed corpus of Asante Twi, realized for the Luminos Fund's Bilingual Boost initiative thanks to funding from the Gates Foundation.

An interactive, searchable and visualization-rich version of this corpus is vailable at https://asante-twi-corpus.fly.dev/app/

Corpus composition

Subcorpus Documents Tokens Share
Doctrinal 3,214 12,125,333 86.5%
General 2,246 1,353,049 9.7%
Children 190 535,293 3.8%
Total 5,650 14,013,675 100%

The corpus also includes 34,647 Twi-English aligned sentences

Data Quatlity

Preprocessing accuracy has been evalauted against a single-annotator triple-proofread set of 13 documents for a total of 12,162 proofed tokens (10,372 words + punctuation; 9,460 development / 2,702 sealed held-out, expert-proofed and re-tokenized to the delivered corpus).  Accuracy, words-only (punctuation excluded — it is tagged trivially and inflates scores):

Metric · words-only (excl. punctuation) Dev · 8,028 words Held-out · 2,344 words
Lemma accuracy (row · aligned micro-F1) 0.915 0.902
— correct / missed 7,342 / 686 2,115 / 229
Lemma F1 (bag-of-tokens) 0.926 0.908
PoS accuracy (token) 0.873 0.827
PoS macro-F1 (per-tag) 0.799 0.788
— all-token (incl. punctuation): lemma / PoS acc 0.927 / 0.892 0.914 / 0.846

Lemma accuracy (~0.91) and PoS macro-F1 (~0.79) hold across development and the sealed held-out set, so the combined 12,162-token gold standard confirms the pipeline generalizes rather than overfits. 5-fold spread (words-only): dev lemma 0.915 ± 0.005 / PoS 0.873 ± 0.008; held-out lemma 0.902 ± 0.011 / PoS 0.827 ± 0.024.

 

Resources Included

  1. 5,650 corpus files
  2. one metadata file detailing genre, title, author... of each corpus document
  3. 34,647 Twi-English aligned sentences 

 

 

 

Files

Restricted

The record is publicly accessible, but files are restricted. Log in to check if you have access.

Additional details

Software

Repository URL
https://github.com/ligeialugli/AsanteTwi_CorpusPreprocessingPipeline
Development Status
Active