January 15, 2022 Software Open Access

Jauhiainen, Tommi; Jauhiainen, Heidi

Dublin Core Export

<?xml version='1.0' encoding='utf-8'?>
<oai_dc:dc xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
  <dc:creator>Jauhiainen, Tommi</dc:creator>
  <dc:creator>Jauhiainen, Heidi</dc:creator>
  <dc:date>2022-01-15</dc:date>
  <dc:description>HeLI off-the-shelf language identifier with language models for 200 languages.

Usage:
java -jar HeLI.jar -r &lt;infile&gt; -w &lt;outfile&gt;

The program will read the &lt;infile&gt; and classify the language of each line as one of the 200 languages it knows
and writes the results, one ISO 639-3 code per line, into file &lt;outfile&gt;.

You can use the -c option to make the program print a confidence score for the identification after each language code.

Usage:
java -jar HeLI.jar -c -r &lt;infile&gt; -w &lt;outfile&gt;

You can give the list of comma-separated ISO 639-3 identifiers for relevant languages after -l option.

Usage:
java -jar HeLI.jar -r &lt;infile&gt; -w &lt;outfile&gt; -l fin,swe,eng

You can give the number of top-scored languages to print after the -t option. (overrides confidence)

Usage:
java -jar HeLI.jar -r &lt;infile&gt; -w &lt;outfile&gt; -l fin,swe,eng -t 2

If you omit both of the filenames, the program will read the standard input one line at a time and write the result to standard output.

It can identify c. 3000 sentences per second using one core on a 2021 laptop and around 3 gigabytes of memory.

If you use this program in producing scientific publications, please refer to: 
 @inproceedings{jauhiainen-etal-2017-evaluation,
     title = "Evaluation of language identification methods using 285 languages",
     author = "Jauhiainen, Tommi  and
       Lind{\'e}n, Krister  and
       Jauhiainen, Heidi",
     booktitle = "Proceedings of the 21st Nordic Conference on Computational Linguistics",
     month = may,
     year = "2017",
     address = "Gothenburg, Sweden",
     publisher = "Association for Computational Linguistics",
     url = "https://www.aclweb.org/anthology/W17-0221",
     pages = "183--191",
 }

Producing and publishing this software has been partly supported by The Finnish Research Impact Foundation Tandem Industry Academia -funding in cooperation with Lingsoft.</dc:description>
  <dc:identifier>https://zenodo.org/record/5853116</dc:identifier>
  <dc:identifier>10.5281/zenodo.5853116</dc:identifier>
  <dc:identifier>oai:zenodo.org:5853116</dc:identifier>
  <dc:language>eng</dc:language>
  <dc:relation>doi:10.5281/zenodo.4780897</dc:relation>
  <dc:rights>info:eu-repo/semantics/openAccess</dc:rights>
  <dc:rights>https://creativecommons.org/licenses/by/4.0/legalcode</dc:rights>
  <dc:subject>language identification</dc:subject>
  <dc:title>HeLI-OTS 1.2 with Python examples</dc:title>
  <dc:type>info:eu-repo/semantics/other</dc:type>
  <dc:type>software</dc:type>
</oai_dc:dc>