Jauhiainen, Tommi
Jauhiainen, Heidi
2022-01-15
<p>HeLI off-the-shelf language identifier with language models for 200 languages.</p>
<p> </p>
<p>Usage:</p>
<p>java -jar HeLI.jar -r <infile> -w <outfile></p>
<p> </p>
<p>The program will read the <infile> and classify the language of each line as one of the 200 languages it knows</p>
<p>and writes the results, one ISO 639-3 code per line, into file <outfile>.</p>
<p> </p>
<p>You can use the -c option to make the program print a confidence score for the identification after each language code.</p>
<p> </p>
<p>Usage:</p>
<p>java -jar HeLI.jar -c <infile> <outfile></p>
<p> </p>
<p>You can give the list of comma-separated ISO 639-3 identifiers for relevant languages after -l option.</p>
<p> </p>
<p>Usage:</p>
<p>java -jar HeLI.jar -r <infile> -w <outfile> -l fin,swe,eng</p>
<p> </p>
<p>You can give the number of top-scored languages to print after the -t option. (overrides confidence)</p>
<p> </p>
<p>Usage:</p>
<p>java -jar HeLI.jar -r <infile> -w <outfile> -l fin,swe,eng -t 2</p>
<p> </p>
<p>If you omit both of the filenames, the program will read the standard input one line at a time and write the result to standard output.</p>
<p> </p>
<p>It can identify c. 3000 sentences per second using one core on a 2021 laptop and around 3 gigabytes of memory.</p>
<p> </p>
<p>If you use this program in producing scientific publications, please refer to: </p>
<p> @inproceedings{jauhiainen-etal-2017-evaluation,</p>
<p> title = "Evaluation of language identification methods using 285 languages",</p>
<p> author = "Jauhiainen, Tommi and</p>
<p> Lind{\'e}n, Krister and</p>
<p> Jauhiainen, Heidi",</p>
<p> booktitle = "Proceedings of the 21st Nordic Conference on Computational Linguistics",</p>
<p> month = may,</p>
<p> year = "2017",</p>
<p> address = "Gothenburg, Sweden",</p>
<p> publisher = "Association for Computational Linguistics",</p>
<p> url = "https://www.aclweb.org/anthology/W17-0221",</p>
<p> pages = "183--191",</p>
<p> }</p>
<p>Producing and publishing this software has been partly supported by The Finnish Research Impact Foundation Tandem Industry Academia -funding in cooperation with Lingsoft.</p>
https://doi.org/10.5281/zenodo.5853071
oai:zenodo.org:5853071
eng
Zenodo
https://doi.org/10.5281/zenodo.4780897
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
language identification
HeLI-OTS 1.2
info:eu-repo/semantics/other