Published November 4, 2026
| Version v1.0.0
Software
Open
Sindhi Tokenizer: SentencePiece Unigram tokenizer for Sindhi with a draft 14-baseline benchmark study
Authors/Creators
- 1. Department of Computer Science, Shaikh Ayaz University, Shikarpur, Pakistan
Description
A SentencePiece Unigram tokenizer for Sindhi with byte fallback, evaluated against 14 baseline tokenizers across four test sets. The associated manuscript is a draft for future publication and has not yet been published.
Notes
Files
alinawazcode/sindhi-tokenizer-v1.0.0.zip
Additional details
Related works
- Is supplement to
- Software: https://github.com/alinawazcode/sindhi-tokenizer/tree/v1.0.0 (URL)
- Preprint: 10.20944/preprints202611.1598.v1 (DOI)
- Preprint: arXiv:2611.20126 (arXiv)
Software
- Repository URL
- https://github.com/alinawazcode/sindhi-tokenizer