Building the first comprehensive machine-readable Turkish sign language resource: methods, challenges and solutions

This article describes the procedures employed during the development of the first comprehensive machine-readable Turkish Sign Language (TiD) resource: a bilingual lexical database and a parallel corpus between Turkish and TiD. In addition to sign language specific annotations (such as non-manual markers, classifiers and buoys) following the recently introduced TiD knowledge representation (Eryiğit et al. 2016), the parallel corpus contains also annotations of dependency relations, which makes it the first parallel treebank between a sign language and an auditory-vocal language.


Introduction
Parallel texts are crucial resources for statistical machine translation (MT) studies, the vast majority of which have been conducted between auditory-vocal language pa irs having a writing system.The parallel data sets compiled from such languages are mostly obtained by the automatic alignment of written translated text materials.Even though written materials do not represent spoken language accurately, most MT systems are based on pairs of written language corpora since developing machine-readable speech data, which would heavily rely on speech recognition systems, is more costly and more prone to errors (Koehn et al. 2007).MT studies focusing on sign languages face an additional challenge: using written text for sign languages is troublesome since sign languages do not have conventionalized writing systems that would lead to large sets of written language materials.Thus, researchers developing an MT system where sign languages are involved must first take on the task of developing a data set which contains the written representations of the sign language data.One way of doing this is to annotate the video recordings of sign language data manually, the other is to employ a sign language recognition system that converts the visual form of the sign language to a written form.Manual annotation is inevitably a costly and slow procedure, and thus does not result in the creation of large data sets that can serve as the input for the training of statistical MT systems.Sign language recognition systems, on the other hand, do not yet yield high performances, again mostly due to the lack of training data.Therefore, despite their drawbacks, manually created parallel data sets are still very valuable resources for MT systems for the following reasons: • As test data sets, they serve as important resources for intrinsic evaluation of MT systems using only written representations, leaving out errors resulting from speech or sign language recognition/generation components of an end-to-end system.• Together with formal sign language representation definitions, they serve as prototypes for end-to-end systems, exemplifying how sign language recognizers should produce machine-readable outputs to feed MT modules or how MT modules should produce their outputs to successfully feed sign animation environments (e.g.avatars, humanoid robots).
Most of the sign language corpus studies mentioned in the literature have been conducted for the purpose of linguistic research [e.g.Leeson et al. (2006) for Irish Sign Language, Schembri et al. (2013) for British Sign Language, Johnston (2016) for Australian Sign Language, Wallin and Mesch (2015) for Swedish Sign Language, Koizumi et al. (2002) for Japanese Sign Language, Prillwitz et al. (2008) for German Sign Language, Crasborn andZwitserlood (2008), De Vos et al. (2015) for Sign Language of the Netherlands and O ¨zsoy et al. (2013) for Turkish Sign Language], and the annotation schemes developed for those corpora are not easily transferable to machine translation studies.The annotations in such corpora are mostly designed to take human-readability and understandability into consideration and, as a result, are most of the time not machine-readable.Johnston (2008) is the first study to introduce ID-glosses and one of the preliminary studies towards machine readability of sign language corpora.In the few corpus studies which focus on machine translation, the compiled parallel data sets are generally small and domain specific, such as aeronautics and weather news [e.g.Su and Wu (2009) between Chinese and Taiwanese Sign Languages, Bungeroth and Ney (2004) and Bungeroth et al. (2006) for weather reports in German and German Sign Language, Bungeroth et al. (2008)'s ATIS corpus for air travel domain available in English, German, Irish Sign Language, German Sign Language and South African Sign Language].Othman et al. (2012) proposes a rule-based translation method from English to American Sign Language in order to produce a large parallel corpus to be used in statistical studies.
The advantages of using syntactic information in rule-based machine translation have been reported (Galley et al. 2004(Galley et al. , 2006;;DeNeefe et al. 2007).Hence, an important resource for rule-based MT is parallel treebanks, in which parallel data sets collected from the languages in focus are syntactically annotated (Cur ˇı ´n et al. 2004;Cmejrek et al. 2005;Megyesi et al. 2008;Ahrenberg 2007;Uchimoto et al. 2004;Tinsley et al. 2009).Since the 90s, dependency analysis (Tesnie `re 1959) has become a widely used approach for natural language parsing and as a result, dependency treebanks have become crucial resources.The most important reason for this popularity is that dependency trees provide reasonable approximation to the semantic layer and are directly usable in higher level NLP tasks. 1ign languages used in different countries/communities differ substantially from each other (and also from the spoken languages used in these countries) at the lexical, morphological and syntactic levels.Due to recent developments in natural language technologies, the need for computerized processing for TiD has increased.The first machine-readable knowledge representation for TiD introduced in Eryig ˘it et al. ( 2016) makes available the necessary infrastructure for its utilization in resource creation: an interactive online dictionary platform and an ELAN2 (Wittenburg et al. 2006;Crasborn and Sloetjes 2008) add-on for TiD.This article introduces a bilingual TiD lexicon and the first machine-readable parallel treebank for the TiD -Turkish language pair using the above-mentioned knowledge representation and infrastructure.This resource has advantages over other TiD corpora intended for use in computerized studies due to its comprehensive annotation scheme, which includes sign language specific features (such as nonmanual markers, classifiers and buoys). 3Our study not only introduces a comprehensively annotated machine-readable corpus for TiD but also the first parallel dependency treebank between a sign language and an auditory-vocal language.
The article is structured as follows: Sects. 2 and 3 provide the methodology of data collection and annotation respectively, Sect. 4 discusses the challenges faced and the solutions provided, Sect. 5 gives the evaluation of the developed resource, and discussions and Sect.6 concludes the article.

Methodology of data collection
In this section, we describe the data sources of the parallel corpus (between Turkish and TiD) that are the foundations of the introduced language resource (TiDLaR).The linguistic data of TiDLaR is restricted to the vocabulary items and sentences in the first grade primary school coursebooks (Dalkılıc ¸and Go ¨lge 2013;Boz et al. 2013; Demirog ˘lu and Go ¨kahmetog ˘lu 2013) published by the Ministry of Education in Turkey, since this study was part of a project which was the first attempt to develop an MT system as a supplementary tool for primary school education of deaf and hard of hearing students. 4iDLaR consists of two components that interact with each other: a lexical database of 1530 signs (collected under 990 entries) and a parallel corpus of 420 annotated Turkish-TiD sentence pairs.The lexical database (henceforth, Lexicon with a capital L) has been designed in such a way that it can both function as an independent dictionary and also serve as the lexicon of the automated translation system.In this article, we describe some relevant properties of this database as it functions as the lexicon of the annotated language resource.For a detailed description of the properties of the used online dictionary platform, the reader is referred to Eryig ˘it et al. ( 2016) which gives a survey of the available Turkish-TiD dictionaries and the reasons why a more comprehensive dictionary is needed.From now on in this article, we refer to this component as the (TiDLaR) Lexicon.

Turkish-TiD parallel corpus (TiDLaR)
In order to make the collected dataset more suitable for machine translation studies (especially for the one focused on our above-mentioned project), we opted for eliciting the TI ˙D utterances as translations of written Turkish sentences instead of alternative data collection approaches such as via visual stimuli.The chosen approach is very similar to the data collection processes employed for machine translation between auditory-vocal languages (see details in Sect.1).This approach is preferred due to the facts that (1) not every sentence can be elicited with visual stimuli and (2) elicitation of sentences with a method other than translation would easily lead to TiD sentences whose meanings are completely different from the meanings of the Turkish counterparts, which would make the dataset hard to use in machine translation.
A total of 306 Turkish sentences in TiDLaR are from the first grade Social Studies (Dalkılıc ¸and Go ¨lge 2013) and Mathematics (Boz et al. 2013) coursebooks.The TiD sentences are created as translations of these written Turkish sentences.In some cases more than one possible translation for a Turkish sentence was offered by the TiD consultant.In those cases, all the proposed alternatives in TiD were recorded and annotated.The corpus thus contains the videos and the annotated files of 420 TiD sentences.Translation was carried out by a number of fluent signers of TiD.Two of these signers were fluent, non-native signers who became deaf in early childhood and were exposed to Turkish Sign Language before the age of 12.Both are active members of the TiD signing community in Istanbul.The third fluent signer is a professional interpreter, trained and employed by the Turkey Ministry of Family and Social Policies.She is a hearing child of Deaf5 adults (CODA)6 and thus, learned TiD as her mother tongue.Due to budget restrictions only one signing language consultant could be employed at a time.Therefore, the Deaf signers contributed to this study in a sequential fashion: the second one checked the translations of the first one and modified them when necessary.The hearing interpreter, on the other hand, both checked the translations of the first and the second one and also discussed with the second one the cases in which the signed translations were not clear.
Two hearing linguistics researchers fluent in Turkish Sign Language worked with the Deaf consultants in order to translate the written Turkish sentences to TiD.First, the consultant was asked to read the book on her own, and then the linguistics researchers went over the sentences with her in TiD to clarify the meanings of the sentences.It is crucial to note that Turkish is a second language for these Deaf consultants, and they are not very fluent readers of Turkish.Thus, in a number of cases, the researchers had to explain the meanings of the sentences to the consultants in order to elicit the best translations.Other challenges of translation will be discussed in Sect.4.1.When the discussion and clarification of the meanings of the Turkish sentences were completed, the Deaf consultants were asked to translate these sentences into TiD as naturally as possible.They were asked to make sure that the translations did not look like signed Turkish, i.e. word-to-sign translations.In fact, when the second Deaf consultant and the professional interpreter watched the videos of the translated TiD sentences, they were also asked to judge these in terms of naturalness.Once the translation phase was completed, the TiD data were annotated using ELAN.The details of the annotation procedures are explained in Sect.3.

TiDLaR lexicon
The TiD lexical items within the Lexicon come in the following categories: (1) one or more counterparts of the Turkish vocabulary items or multi-word expressions found in the sentences from the Social Studies (Dalkılıc ¸and Go ¨lge 2013), Mathematics (Boz et al. 2013) and Turkish (Demirog ˘lu and Go ¨kahmetog ˘lu 2013) coursebooks, and (2) those signs that were uttered during the translation process from Turkish to TiD but do not have counterparts in the books.For instance, if the consultant reported that there are two signs for the color 'green' in TiD, both signs were recorded and entered into the Lexicon.Furthermore, if the consultant reported that there are two different signs for different (but related) meanings of a Turkish lexical item, then both of these signs were again recorded and entered.One such example is c ¸alıs ¸mak 'to work' in Turkish.This verb is the counterpart of both 'to study' and 'to work' and these two different meanings are expressed with different signs in TiD (as in English).
Since sign languages do not have commonly used writing systems, it is common practice to use ''glosses'' in the local spoken language to represent the meanings of signs (Miller 2001).Thus, ID-glosses (Johnston 2008(Johnston , 2016) ) in Turkish, following universal conventions in linguistic studies, are used to uniquely identify the signs in the TiDLaR Lexicon and Corpus.For each lexical entry, one or more of the following are provided7 : (1) a video in which the lexical item is signed by a Deaf consultant, (2) if the sign is a noun and can be inflected for plurality, the video of its plural form, and (3) if there is another sign with the same meaning, the video of that sign.8

Methodology of annotation
The annotation of TiDLaR required different levels of expertise.Similar to the data collection stage, several annotators with different backgrounds played a role during the annotations.Our two Deaf annotators, who worked at the translation stage, also worked to transcribe the TiD utterances (provided in the TiD Tier under ELAN tiers Fig. 2).Then, two of our linguistics researchers fluent in TiD annotated the TiD utterances at the morphology layer.Since the dependency annotations also require a different level of expertise, two other annotators experienced in Turkish morphology and syntax annotations (but with only a basic understanding of TiD) worked for the dependency annotation of both the TiD utterances and the Turkish sentences.
The annotation (knowledge-representation) scheme that we followed for TiD was previously introduced in Eryig ˘it et al. (2016).Similarly, for the annotation of the Turkish sentences, we followed the scheme introduced in Sulubacak et al. (2016b).Thus, for a detailed description of features and categories that constitute these annotation schemes, we refer the reader to these previous studies.In this article, after providing brief summaries of these annotation schemes, we focus on how we created this resource, our annotation procedure, the challenges we encountered during the work, and the places where we had to extend the TiD scheme.
The sub-sections below describe the annotation procedure used in this study to annotate the TiD sentences using ELAN extended with a TiD add-on described in Eryig ˘it et al. (2016) (Fig. 1) and the annotation procedure used to annotate the Turkish sentences using the ITU Treebank Annotation Tool (Eryig ˘it 2007b).The TiD add-on produces an extra XML file for the machine-readable TiD transcriptions (to be used in machine translation studies) while leaving the original ELAN XML file intact.By doing so, all the annotations made on ELAN tiers are still readable by classical ELAN versions whereas the TiD annotation XML files together with the TiD schema file may be parsed by XML parsers and used in machine translation.

Annotation of the TiD utterances
Although there exist environments such as ELAN for the manual annotation of sign language video resources, these have been generally used in linguistic research with the aim of creating human-readable annotations.Recently, Eryig ˘it et al. ( 2016) has proposed a machine-readable representation scheme for TiD which is linked to ELAN manual annotation layers via a new add-on that enables manually-annotated TiD corpora to serve as sources for machine translation research (henceforth TiD add-on).This add-on mainly provides the following advantages: (1) it automatically extracts the sign annotations available from the Lexicon; (2) it provides the tools for annotating additional context-dependent features for signs (e.g.annotating the agreement information of agreeing verbs), and (3) it enables the annotator to enter detailed information about the signs that are context-dependent and not present in the Lexicon (e.g.classifiers, buoys).This add-on integrates a new TiD tab (cf. the top right of Fig. 2) that makes the TiD add-on features available.
A typical annotated ELAN file in our study contains six tiers: ''Discourse'', ''Utterance'', ''Turkish'', ''TiD'', ''MainFlow'' and ''SupportFlow''.9''Discourse'' and ''Utterance'' tiers contain information related to the current annotation (e.g. an identification number in our case).Although the Discourse tier may contain one or more utterances, due to the structure of our course books and also for simplicity of annotation, we had only one utterance annotated per discourse in TiDLaR.The ''Turkish'' tier contains the Turkish sentence taken from the course book.The ''TiD'' tier contains the glosses of the TiD signs in the TiD sentence.This tier was annotated by a Deaf annotator. 10The ''MainFlow'' and ''SupportFlow'' tiers contain the ID-glosses of the signs (specified with SMALL CAPS henceforth), which are most of the time retrieved from the Lexicon.Exceptional cases to this are explained in later sections.The ID-glosses were entered into and retrieved from the Lexicon by the linguistics researchers.
The MainFlow is the sequential realization of sign instances and the SupportFlow is the flow of the extra signs made by the non-dominant hand (Eryig ˘it et al. 2016).In other words, the MainFlow tier contains the representation of the signs which are produced either by only the dominant hand (the preferred hand of a signer while articulating one-handed signs) or by both hands.For example, if a sign is twohanded as in HOUSE , it is annotated only in the MainFlow.The SupportFlow tier, on the other hand, mainly contains the representations of signs produced only by the non-dominant hand.These are usually buoys and classifiers (see Sect. 3.1.1). 11;12  The content of the TiD tab depends on (i.e.changes dynamically according to) the selected ELAN tier.The annotation features that become editable in the TiD tab when either the Discourse or the Utterance tier is selected are those that affect the entire sentence.When the MainFlow or the SupportFlow tier is active, the annotations at sign-level become available under three TiD sub-tabs: ''Basic'', ''Arguments'' and ''Advanced'' as was shown in Fig. 2 (emphasized by red rectangles on the figure).The remainder of this section focuses on our annotations at sign-level, utterance-level, and syntactic-level.

Sign-level annotations
The annotation procedure of sign-level annotations starts with retrieving the IDgloss of each sign from TiDLaR Lexicon (see Fig. 2 Gloss field under the Basic Tab).Next, the main category of the sign is selected (e.g.lexical, classifier, or buoy) and annotations start at different levels.According to the selected category information, the related arguments are dynamically enabled for human annotation (under the Arguments sub-tab).If a sign in an utterance that is being annotated displays properties different from or additional to the ones already specified for it in its Lexicon entry, these are also specified (under the Advanced sub-tab).These properties, labeled as ''non-manual markers'', ''plural'', ''incorporation'', ''incorporation type'', and ''hand usage'', will be discussed below.

Lexical signs:
If the sign is a lexical sign, then its available meanings and parts of speech (e.g.noun, verb, or pronoun) are retrieved again from the Lexicon and the relevant tags in the context of the given TiD utterance are selected by our annotators (see Fig. 2, Parts of speech and Meaning fields under the Basic Tab).As specified above, the related arguments are dynamically enabled for human annotation according to the selected POS and category information.For instance, if the sign is defined as an agreement verb, information on its inflectional features (i.e.what it agrees with) is provided at this stage.

Classifiers:
In addition to lexical signs, the utterances in the corpus also contain contextdependent signs such as classifiers and buoys.Classifiers are manual signs that do not have specific lexical meanings but represent entities by denoting their most salient characteristics, such as a two-legged animal or a flat surface.They may represent entities in a static position as in ''A cup stands on the table'', or in motion as in ''A person is walking''. 13ince classifiers are mostly context-dependent signs (Zwitserlood 2012), it is not possible to enter all possible classifier variations in the Lexicon.That is why classifiers are annotated in the arguments tab by choosing their handshape, orientation, and movement, if any.To be able to track more than one classifier construction in an utterance, their glosses are given based on their semantic reference, such as PICTURE.CL, which has CL to distinguish it from the lexical PICTURE and signals its dependency relation (for later stage annotations, to be discussed below), differently than the lexical sign.
Following the TiD representation scheme, the classifiers are annotated under four classifier sub-categories: handling, body-part, entity, and size and shape.Handling classifiers represent entities that are being held and/or moved by an agent, such as representing the hand of an agent opening a door (Zwitserlood 2012).Body-part classifiers represent the body itself, such as eyes, hands or feet.Entity classifiers represent entities by denoting particular semantic and/or shape features such as using the index finger handshape to represent an upright human being.Size and shape specifiers denote nouns according to the visual-geometric features of their referents.An example would be tracing the edges of a rectangular surface.
The classifier arguments (handshapes and movements) are taken from Kubus (2008)'s exhaustive list.For instance, during the annotation of an entity-movement classifier, the annotator can choose among different movement patterns such as swimming, walking, sliding, going upstairs etc.In the case of an entity-location classifier, for instance, its handshape (e.g.flat), its locus in the signing space (e.g.contralateral-up) and its orientation (e.g.lateral) are specified, as shown in Fig. 3.

Buoys:
Buoys are signs that help guide the discourse by serving as conceptual landmarks as the discourse proceeds (Liddell 2003).That is, a buoy signals to the addressee that the conversation is related to the topic it represents.These signs are produced frequently by the non-dominant hand in a stationary configuration as the dominant hand continues producing the other signs.Some buoys appear only briefly, whereas others may be maintained during a significant stretch of signing.They can also be referred to (e.g.pointed at) by the dominant hand.Buoys are annotated under two types: list buoys and fragment buoys.A list buoy is used as enumeration when referring back to a list of items.The fingers of the non-dominant hand represent an ordered set of topics in the discourse.The signer may start with expressing the entire list, for instance, holding open four fingers and referring back to the first, second, third and the fourth finger in an order, or he/she may enumerate the list without first expressing the total number of the items in the list.A sign is called a fragment buoy when it associates the meaning of a sign with all or part of its final state of production.Fragment buoys are not lexicalized; they are created spontaneously.During the annotation in the Basic tab, one of these types is specified for a buoy sign.
The following example illustrates the annotation of fragment buoys.The first sign in the MainFlow in Fig. 4 is EV.KONUT 'house' . After signing it, the signer keeps a part of the sign HOUSE on the non-dominant hand while signing DIS ¸.YO ¨N 'outside' with the dominant hand.DIS ¸.YO ¨N is annotated in the MainFlow and the buoy is annotated in the SupportFlow.When the category of a sign is selected as ''fragment buoy'', the annotator also indicates the fragment of which lexical sign functions as a buoy.In the example, the fragment buoy is linked to the lexical sign EV.KONUT 'house' via ''Fragment Buoy Ref.'' field.A fragment buoy can also be referred back to by a pointing sign.In those cases, the category of the pointing sign is selected as POINTING_BUOY_FRAGMENT.Which fragment buoy it points to is also specified in ''Pointing Fragment Buoy'' field as an argument.Non-manual markers: Non-manual markers (NMMs) are gestures that have linguistic functions (e.g.facial expressions, eye brow position, head position, head nod or head shake, body lean, body tilt, mouth gestures and mouthings) (Pfau and Quer 2010).NMMs can be lexical features of individual lexical items and can thus occur only during the articulation of the manual sign.They can also have syntactic and pragmatic functions and thus can spread over larger constituents such as the verb phrase or the entire sentence.These functions can be marking topics, yes/no questions, negation, etc.In the TiDLaR annotation scheme, if a sign has a lexical non-manual marker, this is specified in its lexical entry when it is entered into the Lexicon.However, if that sign is signed with a different non-manual marker in the annotated utterance (e.g. with ''topic''-marking non-manuals), then this additional non-manual marker is specified under the ''Non-manual markers'' feature in the Advanced tab.

Plurality:
Fig. 4 The selection of Fragment Buoy reference for the sign EV.KONUT 'house' Eng.: HOUSE OUTSIDE WHICH ANIMAL THERE.ARE -> ''Which animals are there outside the house?''Kubus ¸(2008) and Zwitserlood et al. (2012) report that TiD does not have a single systematic plural marker but employs different strategies such as reduplication and use of quantors.Thus, it is not possible to have a morphological rule that can predict the plural form of a given singular noun.The annotators choose from among the 26 plural formation strategies (such as ''locative reduplication'') reported in Kubus ¸(2008) when annotating the pluralization strategy of a plural noun.

Incorporation:
The feature ''Incorporation'' refers to cases where a sign co-occurs with another sign that is phonologically dependent on it. 14The types of incorporation choices available here are ''numerical incorporation'', ''negative incorporation'' and ''general incorporation''.It is commonly observed in sign languages that number signs (1-5 or 1-9) incorporate into nouns, pronouns and temporal expressions (Costello et al. 2017).This is illustrated below.Figure 5a shows the number sign ONE in isolation.Figure 5b, on the other hand, shows a complex sign, meaning 'first grade', where ONE is incorporated into the sign GRADE.ONE in isolation is signed vertically in the neutral signing space with the index handshape.When incorporated, however, it is signed in the location of GRADE, which is the upper part of the arm of the non-dominant hand.

Hand usage:
The articulation of a sign may display different handedness properties than those specified for it in the Lexicon.Such deviations are annotated: • if a sign is articulated with the non-dominant hand in the annotated utterance, whereas its entry in the Lexicon specifies its articulation with dominant hand; • if a sign is articulated with both hands in the annotated utterance, whereas its entry in the Lexicon specifies its articulation with only the dominant hand; • if a sign is articulated with a single hand in the annotated utterance, whereas its articulation is specified as two-handed in the Lexicon.

Utterance-level annotations
The properties annotated at the utterance-level are ''Features'', ''Utterance Non-Manuals'' and ''Non Manuals Suppressed''.''Features'' refers to different clause types such as declarative, interrogative and exclamative, tense types such as past tense, aspect types such as perfective, and modality types such as obligation.''Utterance Non-manuals'' refers to the non-manual markers that spread over the entire utterance, such as those marking a sentence as a yes/no question (see 3.1.3for an explanation of NMMs).The information on the lexical non-manuals of the sign, if there is any, is available in the Lexicon and is retrieved together with other kinds of information of the sign.However, these lexical NMMs can sometimes be suppressed by the utterance NMMs.In such cases, the suppression is annotated by selecting the suppressed non-manual under the ''Non Manuals Suppressed'' feature.
The annotators annotate each utterance by first selecting the Utterance tier and then choosing from the relevant property values that become available on the TiD tab.

Dependency annotations
According to the dependency formalism, each syntactic token within a sentence should have been connected to one head token, and head tokens may have many dependent tokens linked to them.Following this rule, the head sign of each sign within the syntax tree of the parent utterance is annotated together with its dependency relationship type.An example dependency tree is provided in Fig. 6 for a TiD utterance (meaning 'You should close your eyes.') and its Turkish counterpart (Gözlerini kapatman gerekiyor.).The annotator selects one of the sign glosses from a combobox as the dependency head of the current sign (e.g. 'eye closing' is the dependency head of the sign 'your') and a dependency relation (e.g.POSSESSOR) from a predetermined list of possible relation types (to be detailed in Sect.4.2.2).A dummy ROOT node is added to each utterance in order to act as the root of the dependency tree depicted with an empty dashed rectangle in Fig. 6.

Annotation of the Turkish sentences
The studies for Turkish treebanking started in the early 2000s (Atalay et al. 2003;Oflazer et al. 2003) and have continued since then (Eryig ˘it 2007a; Sulubacak and Eryig ˘it 2013;Pamay et al. 2015;Eryig ˘it et al. 2015;Sulubacak et al. 2016a, b;Sulubacak and Eryig ˘it 2018).Following earlier studies, the Turkish part of the TiDLaR Corpus was annotated in two main layers: morphology and syntax.The new web version of the ITU Treebank annotation tool (Eryig ˘it 2007b; Pamay et al. 2015), used in recent studies, was selected as the annotation environment, and two experienced annotators worked during the annotation process.In the morphology layer, the annotators were supposed to choose a correct morphological analysis for each input Turkish token from an automatically produced list of possible analyses (S ¸ahin et al. 2013;Eryig ˘it 2014).In the syntax layer, the dependency annotations were carried out according to Sulubacak et al. (2016b). 15Each of the two annotators annotated half of the sentences and controlled the other half.In case of conflict, the annotators worked together and corrected the analysis upon their agreement.
Table 1 provides the main parts-of-speech (POS) tags and dependency relation distributions within Turkish and TiD sections of the TiDLaR corpus.The minor POS-tag and different non-manual type distributions in the TiD corpus are provided in the Appendix.As stated previously, during the annotations of the TiD section, the representation scheme provided in Eryig ˘it et al. ( 2016) was adopted.However, during the syntactic annotations, we needed new dependency relation types in addition to the basic ones given in this scheme.The challenges we faced and the newly added relation types are discussed in Sect.4.2.

Challenges and solutions
Creation of the parallel corpus raised a number of challenges.The problems that emerged during the translation phase and the methods to ensure accuracy and reliability in translation that the researchers developed are described in the first sub-section below.The challenges in ELAN annotations and identification of dependency relations are more analytic in nature, and this issue is discussed in the following sub-section.

Challenges in translation: ensuring reliability
Needless to say, translating sentences of a spoken language into a sign language is very different from translating between two spoken languages that are used in educational settings, and the task thus comes with its own challenges.As explained in 2.2, TiD lexical items in the Lexicon of our Resource mostly consist of translations of words found in the first-grade course books.The task of translating these words into TiD was far from trivial.This is because TiD is not formally used in educational settings (Dikyuva et al. 2017) and the TiD lexicon lacks many signs that express the concepts found in school course books.This issue has been addressed in our study in the following ways: (1) if there was a chapter in the book with a large number of concepts whose counterparts are not in the TiD lexicon, the sentences in those chapters were not translated and left out of the sentential database, (2) the second Deaf signer consulted other signers in the community and the professional interpreter of the project to see whether there were signs used for these concepts, however rarely, and (3) in some cases, the lexical items of Turkish were finger-spelled.
As mentioned in Sect.3.1, once the initial translation of the sentences into TiD and the recording of the citation forms of the individual lexical items was completed by the first Deaf consultant, the second Deaf consultant watched each TiD sentence and the lexical item video and evaluated them for naturalness and accuracy.In a number of cases, she suggested a modification, and the new form was recorded and the initial version replaced by the new form.In another set of cases, when the second consultant was not sure what the best translation would be, more than one translation was recorded for future consultation.
In the next phase, a professional interpreter joined the study temporarily.Initially, she was not shown the Turkish sentences in the course books.Thus, when she watched the videos of the TiD sentences, she did not know what they were translations of.She was asked to translate these TiD sentences into Turkish orally, and her translations were recorded and transcribed.The researchers compared her translations with the original Turkish sentences and marked these pairs as (1) ''complete match'', (2) ''semantic match but syntax is very different'', and (3) ''problematic''.The last category refers to cases where the interpreter either didn't understand the TiD sentence or provided a translation very different from what the original Turkish sentence expressed.The interpreter and the second Deaf consultant discussed these problematic cases, and when they decided on a form they agreed on, the Deaf consultant was recorded signing the new TiD sentence.A similar procedure was carried out for the lexical entries in the TiDLaR Lexicon.

Challenges in ELAN annotations and in identifying the dependency relations
The challenges faced during the annotation of the data depended mainly on the design decisions and the nature of the corpus.As explained in Sect.3, ELAN annotations contain dependent tiers and a dynamic lexicon.Moreover, the syntactic annotations are based on the representation of the dependency relations between constituents.Both of these properties of the corpus design forced the researchers to make certain decisions regarding annotation and linguistic analysis.Some of these are discussed below.

Annotating simultaneity
One commonly encountered challenge in annotating sign language utterances is related to the simultaneity property of sign languages (Johnston 2016).Non-manual markers co-occuring with manual signs and use of classifiers and buoys are among the most common examples of simultaneity and were discussed in Sect.3. The challenge faced during the annotation of the non-manual markers is related to the hierarchical dependency of tiers of the annotation scheme adopted in this study (Sect.3.1).When a non-lexical non-manual marker spreads over more than one sign, this has to be indicated by entering that non-manual marker for each sign the marker spreads over.
Another challenge was annotating the relations of the manual signs that were articulated with different hands simultaneously.Most of these cases involve a classifier or a buoy on the non-dominant hand.If it is a classifier, then the sign is glossed as a classifier with information on what it represents, for instance, as PICTURE.CL (Sect.3).If it is a buoy, then it is glossed in a way that shows its type, for instance, in the case of a fragment buoy, as F_BUOY.These glosses are entered in the SupportFlow, and to reflect the simultaneity, their time intervals are aligned with the time intervals of the signs whose glosses are in the MainFlow.
Figure 7 below exemplifies the annotation of a fragment buoy.The signer signs the lexical sign EV.KONUT 'house' and its gloss is entered in the MainFlow.As she further signs DIS ¸.YO ¨N 'out(side)' with the dominant hand, she holds a fragment of the sign EV.KONUT on the non-dominant hand as a fragment buoy.Since the fragment buoy continues to be held until the rest of the utterance, its gloss is aligned with all of the glosses in the MainFlow.Note that since the SupportFlow tier is dependent on the MainFlow tier (see Sect. 3), a separate gloss has to be entered in the SupportFlow for each gloss in the MainFlow even though all these separate glosses refer to the same sign (in this example, the fragment buoy of the sign EV.KONUT 'house').
Many utterances in the Mathematics course book express spatial relations such as ''The cat is under the dog.'', ''Which animal is above the rooster?''etc.The expression of these relations in TiD does not involve prepositions such as under and above with fixed pronunciations.Rather, the signing space is used to represent these spatial relations topographically, and classifiers represent the entities involved.Expressing the spatial relations may require simultaneous use of the hands and the movement of the hands.For instance, in the case of 'The cat is under the dog.', the non-dominant hand represents the dog and is kept steady while the dominant hand represents the cat and moves downwards below the non-dominant hand.The annotation scheme enables the annotator to identify the signs as classifiers representing specific entities as, for instance, and their movement using the feature ''Classifier_Movement'' provided by the TiD add-on.

Identification of dependency relations
One example of TiD data that required linguistic (re)analysis and the modification of glossing for the purposes of identification of dependency relations is the use of a sign formerly glossed simply as POSS, as in POSS DRESS 'her dress'.The data contain straightforward cases such as POSS DRESS where POSS can be analyzed as a possessive pronominal determiner such as her and linked to DRESS via a possession relation.However, the same sign also occurs together with an overt possessor as in ANIMAL POSS FOOT 'the animal's foot'.Since the sign POSS here does not express the possessor of the foot in this example, but ANIMAL does, in this utterance POSS cannot be analyzed as a possessive pronominal determiner.To circumvent the problem, we proposed two homophonous POSS signs: POSS-PRONOUN and POSS-ADPOSITION.The latter one is analyzed as the counterpart of a preposition such as 'of' in English.Since TiD is a head-final language, this sign is analyzed as a postposition mediating the possession relation between the possessor ANIMAL and the possessee FOOT.Thus, in the dependency relation annotations, in an example such as ANIMAL POSS-ADP FOOT, ANIMAL is linked to POSS-ADPOSITION via the CASE relation.A similar situation holds for the annotation of negation in TiD, which is expressed at the morphological level in Turkish.A new dependency relation (NEGATIVE) has been added in order to express the relations emerging from the negation markers.
Another challenge was annotating holds (with classifiers or buoys) in the SupportFlow with different signs accompanying them in the MainFlow.As explained in the previous section, since the SupportFlow tier is dependent on the MainFlow tier, a continuous buoy or classifier sign co-occuring with more than one sign in the MainFlow was annotated as a separate token (Fig. 7).In terms of dependency, these separate signs were linked to their first token in the tier via the CONTINUE relation.The relation between a lexical item and its classifier is specified with the COREFERENCE relation.This article has introduced our approach to generating the first parallel treebank between a sign language and an auditory-vocal language for use in machine translation studies.Although Turkish (being from a different language family than English and many European languages) poses interesting challenges for NLP studies, we believe the approach introduced here may be used for developing similar resources for other language pairs by making the necessary adaptations.Similar to the unification studies for POS tag and dependency representations [i.e.Universal Dependencies Project, Nivre et al. ( 2016)], we believe future studies on developing a universal machine readable knowledge representation scheme for sign languages will speed up machine translation studies in this field.
The resource described in this article has been recently used and evaluated within a rule-based machine translation system (Eryig ˘it 2017) (from Turkish to TiD).In this study, a transfer-based machine translation approach relying both on syntactic and semantic transfer has been adopted.The input to the component of translation rules is the automatic morphological and syntactic analysis of the source language (produced via a Turkish NLP pipeline) and the output which goes to feed the animation layer is the generated machine readable representation of the target language (TiD).In cases where it was not possible to find an equivalent sign entry with the same lexical sense of a Turkish input, the senses were mapped to concepts for semantic transfer adapted from the ''Lexicon Model for Ontologies (LEMON), McCrae et al. (2011)''. Table 2 [from Eryig ˘it (2017)] provides the success rates (based on WER, PER and BLEU metrics) of the above mentioned MT sytem on the developed resource.The BLEU score over all sentences was measured as 0.47 and as 0.64 on short sentences.

Conclusion
This article has introduced the first comprehensive machine-readable Turkish Sign Language resource (TiDLaR) built up as a result of an interdisciplinary endeavor.Scientists from linguistics, natural language processing, robotics and 3D animation fields designed TiDLaR both from a linguistic and computer science perspective hoping that it could serve as a valuable resource for both communities.For instance, linguists studying the grammar of TiD would easily search for different types of agreement verbs and different types of classifiers (cf.Appendix, Table 3 on the distribution of the parts-of-speech of the TiD data), or the distribution of 28 different non-manual markers (Table 4).Researchers working on Turkish-TiD machinetranslation systems would find a prototype model both to test their MT systems and to feed the animation environments, which would obtain the required information about what should be included or suppressed over the basic motion data in order to move the avatar naturally and smoothly to produce the necessary visual sign utterance.
TiDLaR is composed of two components that interact with each other: a lexical database (available as an online interactive dictionary platform http://www.tid.itu.edu.tr/TidGlossary/) and a parallel corpus of annotated Turkish-TiD sentence pairs, which will be available to researchers for academic purposes upon the publication of this manuscript.In addition to sign language specific annotations following the recently introduced TiD knowledge representation (Eryig ˘it et al. 2016), the parallel corpus contains also annotations at the morphological and syntactic (based on dependency formalism) levels, which makes it the first parallel treebank between a sign language and an auditory-vocal language.

Fig. 2
Fig. 2 Tiers and Tabs in ELAN TiD Add-on English glosses and translation: I THIN -> ''I'm thin''

Fig. 3
Fig. 3 Arguments tab when a classifier is annotated Eng.: ''Explain to me what you see in the picture''

Fig. 5 Fig. 6
Fig. 5 Annotation of number incorporations a ONE b

Fig. 7
Fig. 7 Annotating simultaneity: a fragment buoy on the non-dominant hand Eng.: YOU HOUSE OUTSIDE AROUND WHAT SEE -> ''What do you see outside/around the house?''

Table 1
TiDLaR parts-of-speech tag and dependency relation distributions

Table 2
Evaluation of an MT system on the developed resource (Eryig ˘it 2017)

Table 4
Building the first comprehensive machine-readable... Distribution of the non-manual markers in the TiD corpus