There is a newer version of this record available.

Software Open Access

HeLI-OTS 1.2

Jauhiainen, Tommi; Jauhiainen, Heidi

HeLI off-the-shelf language identifier with language models for 200 languages.

 

Usage:

java -jar HeLI.jar -r <infile> -w <outfile>

 

The program will read the <infile> and classify the language of each line as one of the 200 languages it knows

and writes the results, one ISO 639-3 code per line, into file <outfile>.

 

You can use the -c option to make the program print a confidence score for the identification after each language code.

 

Usage:

java -jar HeLI.jar -c <infile> <outfile>

 

You can give the list of comma-separated ISO 639-3 identifiers for relevant languages after -l option.

 

Usage:

java -jar HeLI.jar -r <infile> -w <outfile> -l fin,swe,eng

 

You can give the number of top-scored languages to print after the -t option. (overrides confidence)

 

Usage:

java -jar HeLI.jar -r <infile> -w <outfile> -l fin,swe,eng -t 2

 

If you omit both of the filenames, the program will read the standard input one line at a time and write the result to standard output.

 

It can identify c. 3000 sentences per second using one core on a 2021 laptop and around 3 gigabytes of memory.

 

If you use this program in producing scientific publications, please refer to: 

 @inproceedings{jauhiainen-etal-2017-evaluation,

     title = "Evaluation of language identification methods using 285 languages",

     author = "Jauhiainen, Tommi  and

       Lind{\'e}n, Krister  and

       Jauhiainen, Heidi",

     booktitle = "Proceedings of the 21st Nordic Conference on Computational Linguistics",

     month = may,

     year = "2017",

     address = "Gothenburg, Sweden",

     publisher = "Association for Computational Linguistics",

     url = "https://www.aclweb.org/anthology/W17-0221",

     pages = "183--191",

 }

Producing and publishing this software has been partly supported by The Finnish Research Impact Foundation Tandem Industry Academia -funding in cooperation with Lingsoft.

Files (88.2 MB)
Name Size
HeLI.class
md5:95657280ee492a6ab4844eeb4454a5c0
13.7 kB Download
HeLI.jar
md5:8537531e1e6f74f67a58fcfc0ac302e3
44.1 MB Download
HeLI.java
md5:c71b0f3cd044bf424908d905fa0a7a97
22.5 kB Download
HeLI.mf
md5:bb91c0c41fd40f3fb8a7c4f98c9a7c87
39 Bytes Download
languagelist
md5:f44bcfe8a8a8108095b6bc35cea8e31d
884 Bytes Download
LanguageModels.zip
md5:efd3371472a6b3a93133773a6c09d87b
44.1 MB Download
LICENSE
md5:bb0ae3b700049fd806e2a043e01265d6
11.4 kB Download
README.md
md5:e6f06930e25726624e53eb7a901e0874
2.7 kB Download
  • Jauhiainen, Tommi et al. (2017). Evaluation of language identification methods using 285 languages. https://www.aclweb.org/anthology/W17-0221

977
236
views
downloads
All versions This version
Views 97746
Downloads 23610
Data volume 3.0 GB88.2 MB
Unique views 72640
Unique downloads 1143

Share

Cite as