A Korean raw text collection for creating a language model
Description
A very large Korean raw text collection for creating a language model
We collected a very large monolingual dataset for Korean, which contains over 9.6M sentences and 130.6M eojeols, to create a language model: Korean Wikipedida (https://dumps.wikimedia.org/kowiki/20201101/, 5.3M sentences and 71.8M eojeols, respectively), the Sejong morphologically analyzed corpus (3.0M and 40.0M), and articles from The Hankyoreh daily newspaper during 2016 (1.2M and 18.6M).
We preprocessed raw text into morpheme-segmented text using the POS tagging system (park-tyers:2019:LAW). We also attached the POS label to the morpheme-segmented lexicon, and explicitly include a + symbol for consecutive morphemes.
시인/NNG 윤동주/NNP +,/SP 이준익/NNP 감독/NNG 영화/NNG +로/JKB 부활/NNG
See https://github.com/jungyeul/sjmorph for the POS tagging system described in park-tyers:2019:LAW.
Files
Files
(2.3 GB)
| Name | Size | Download all |
|---|---|---|
|
md5:7c74af5ef7fa9ac5f1e13aa2e1e7e9f2
|
334.8 MB | Download |
|
md5:733150c7dcba6a3834cb0c850206b5df
|
1.2 GB | Download |
|
md5:17d63a4c7d84b6a727f8d5ffcc838adb
|
716.4 MB | Download |