There is a newer version of the record available.

Published November 4, 2026 | Version v1.0.0

Sindhi Tokenizer: SentencePiece Unigram tokenizer for Sindhi with a draft 14-baseline benchmark study

  • 1. Department of Computer Science, Shaikh Ayaz University, Shikarpur, Pakistan

Description

A SentencePiece Unigram tokenizer for Sindhi with byte fallback, evaluated against 14 baseline tokenizers across four test sets. The associated manuscript is a draft for future publication and has not yet been published.

Notes

If you use this tokenizer or code, please cite it as below.

Files

alinawazcode/sindhi-tokenizer-v1.0.0.zip

Files (1.7 MB)

Name Size
md5:3d50497f8dbf6e6cb0d275ff40839567
1.7 MB Preview Download

Additional details

Related works