The definition of tokens in relation to words and annotation tasks

Tokens are the basic units of annotations. When working with corpora of non-standardized texts, tokenization is often problematic, as the usage of whitespace can vary. We show examples of how decisions in the tokenization process can influence an annotation and argue that the principles underlying the tokenization should be grounded in theoretical concepts selected on the basis of the annotation task. We present a corpus of Early New High German texts in which the annotation layers reference two different concepts of words: syntactic words and graphematic words. Consequently, we use two kinds of tokens: graphic tokens and syntactic tokens.