Dataset Open Access
James P. Diprose;
Cameron Neylon
The COKI Language Dataset contains predictions for 122 million academic publications. The dataset consists of DOI, title, ISO language code and the fastText language prediction probability score.
Methodology
A subset of the COKI Academic Observatory Dataset, which is produced by the Academic Observatory Workflows codebase [1], was extracted and converted to CSV with Bigquery and downloaded to a virtual machine. The subset consists of all publications with DOIs in our dataset, including each publication’s title and abstract from both Crossref Metadata and Microsoft Academic Graph. The CSV files were then processed with a Python script. The titles and abstracts for each record were pre-processed, concatenated together and analysed with fastText. The titles and abstracts from Crossref Metadata were used first, with the MAG titles and abstracts serving as a fallback when the Crossref Metadata information was empty. Language was predicted for each publication using the fastText lid.176.bin language identification model [2]. fastText was chosen because of its high accuracy and fast runtime speed [3]. The final output dataset consists of DOI, title, ISO language code and the fastText language prediction probability score.
Query or Download
The data is publicly accessible in BigQuery in the following two tables:
When you make queries on these tables, make sure that you are in your own Google Cloud project, otherwise the queries will fail.
See the COKI Language Detection README for instructions on how to download the data from Zenodo and load it into BigQuery.
Code
The code that generated this dataset, the BigQuery schemas and instructions for loading the data into BigQuery can be found here: https://github.com/The-Academic-Observatory/coki-language
License
COKI Language Dataset © 2022 by Curtin University is licenced under CC BY 4.0.
Attributions
This work contains information from:
References
[1] https://doi.org/10.5281/zenodo.6366695
[2] https://fasttext.cc/docs/en/language-identification.html
[3] https://modelpredict.com/language-identification-survey
Name | Size | |
---|---|---|
000000000000.csv.gz
md5:59b13c0133269c201b07b5d29ec67c2a |
43.9 MB | Download |
000000000001.csv.gz
md5:77daae006ccc214ce594bd29ea3bd0b1 |
43.9 MB | Download |
000000000002.csv.gz
md5:603f0e4bbfe8e0f6d51e27a06ed38f6e |
43.8 MB | Download |
000000000003.csv.gz
md5:a5a6e300db241ab8dc0fae49405b8bf3 |
43.9 MB | Download |
000000000004.csv.gz
md5:9e78838a9aa91a8ec0dfeaed85145135 |
43.8 MB | Download |
000000000005.csv.gz
md5:c706665076b0f29e23a71b587e9a28eb |
43.9 MB | Download |
000000000006.csv.gz
md5:44283390324e8611cc427c1d9fab43ae |
44.0 MB | Download |
000000000007.csv.gz
md5:cbf9366aa664bc283935ecd1db7ba1fb |
43.9 MB | Download |
000000000008.csv.gz
md5:4c0e715e3212259ee646a79e99c3c8fd |
43.9 MB | Download |
000000000009.csv.gz
md5:5cb1808438468367c8157ec79495f4be |
43.8 MB | Download |
000000000010.csv.gz
md5:2c2d0ec0593753872f433e23a621b851 |
43.9 MB | Download |
000000000011.csv.gz
md5:468a4f0308a6a4d187aea83a4f4f5fba |
43.9 MB | Download |
000000000012.csv.gz
md5:91db4607480a59d1ad838074a36caa1e |
43.9 MB | Download |
000000000013.csv.gz
md5:67df441ce9c16fb8ce07632af2f2b91d |
43.9 MB | Download |
000000000014.csv.gz
md5:2e92fc32905634fd26518203755c21e7 |
43.9 MB | Download |
000000000015.csv.gz
md5:9dba8397b2d20959f50663158b90385c |
43.9 MB | Download |
000000000016.csv.gz
md5:ddc50487ea757c03aaae4d3b52fccc15 |
44.0 MB | Download |
000000000017.csv.gz
md5:4a17850a1b9444309867d32c2b47b997 |
43.9 MB | Download |
000000000018.csv.gz
md5:663944f7ce2bebe1b8762a9749a527e4 |
44.0 MB | Download |
000000000019.csv.gz
md5:26bb451c76e7022558929e5f3244a600 |
43.9 MB | Download |
000000000020.csv.gz
md5:1d9f121871ea37c17aaadcbfc6800be9 |
43.8 MB | Download |
000000000021.csv.gz
md5:d316a3a3a3ca6ec62b72d6b65c24b034 |
43.8 MB | Download |
000000000022.csv.gz
md5:835f68495b0fe1d51fdf55df25e7df8f |
43.9 MB | Download |
000000000023.csv.gz
md5:3939c683daa7e9c1062f2d72e21ac9c9 |
44.0 MB | Download |
000000000024.csv.gz
md5:94141b7187083b1efd59b531831a142e |
43.9 MB | Download |
000000000025.csv.gz
md5:acc028382061dd7ee133eb7c2aaebb9f |
43.9 MB | Download |
000000000026.csv.gz
md5:867135ec21dd9d9076d89f09c100e34f |
43.9 MB | Download |
000000000027.csv.gz
md5:990694cdb71d792913559586b96b8bbf |
43.8 MB | Download |
000000000028.csv.gz
md5:1016e7eafbc1afb271b22f21f5f0f1a1 |
43.9 MB | Download |
000000000029.csv.gz
md5:663a2034c2fec9d196643eb8e0889b61 |
43.8 MB | Download |
000000000030.csv.gz
md5:f91be8d4f840ab39d5f402363dd828aa |
43.9 MB | Download |
000000000031.csv.gz
md5:d2ef89cec2e4b92ef586f9a48703044a |
43.9 MB | Download |
000000000032.csv.gz
md5:2c6a50b9e85594e9d5b789a20a5a2b09 |
43.9 MB | Download |
000000000033.csv.gz
md5:8b008ccf072de2a2438a4c22cb3989ee |
43.9 MB | Download |
000000000034.csv.gz
md5:64cf20d7fe78cdb6edb92a55ea05dbf5 |
43.9 MB | Download |
000000000035.csv.gz
md5:f36c5ef654d8db5abf4ed6a7a4e79fc7 |
43.9 MB | Download |
000000000036.csv.gz
md5:28e9fc6895906b263a6accf004d63a28 |
43.9 MB | Download |
000000000037.csv.gz
md5:5150e907c743538c449a771f3b9aa9db |
43.9 MB | Download |
000000000038.csv.gz
md5:cc18589f03b5a3879e3eb79e783860cd |
43.9 MB | Download |
000000000039.csv.gz
md5:c0cdccd5433edeaba901c4f91dc27ee9 |
43.9 MB | Download |
000000000040.csv.gz
md5:d31ae86b4d14986a436ede1427f9736e |
43.9 MB | Download |
000000000041.csv.gz
md5:9d12c454adba763816f1ceb7f058e4b7 |
43.9 MB | Download |
000000000042.csv.gz
md5:bbd0cb4f97416a1cc89fe94d1edad980 |
43.8 MB | Download |
000000000043.csv.gz
md5:908dab1ff394d334a63db15d92b38d99 |
43.8 MB | Download |
000000000044.csv.gz
md5:0cf776f36da49f6a9b68e6bb1d73a308 |
43.8 MB | Download |
000000000045.csv.gz
md5:d6ee87b596e999ab8bf40bdc263a65c0 |
43.9 MB | Download |
000000000046.csv.gz
md5:f66078fb6900046b1db39debc3a83c06 |
43.9 MB | Download |
000000000047.csv.gz
md5:f66288942c31621acbb5d85eac107065 |
43.8 MB | Download |
000000000048.csv.gz
md5:47cbf2e52035db18cb7d7e6d78fbbb38 |
43.9 MB | Download |
000000000049.csv.gz
md5:00d2afc6aa950c6e4df90ef044a08713 |
43.8 MB | Download |
000000000050.csv.gz
md5:5218bd0bc93ce5205d15044deaf53b9d |
43.9 MB | Download |
000000000051.csv.gz
md5:8708c6959b99c422db83ceef4e0cabce |
43.9 MB | Download |
000000000052.csv.gz
md5:2a20004c01c18c721016649092ec5f8d |
43.9 MB | Download |
000000000053.csv.gz
md5:a7ba885126020b6072ea33489495c059 |
43.9 MB | Download |
000000000054.csv.gz
md5:39e9488a8e8821a4ad5c6f5935e7fbee |
43.9 MB | Download |
000000000055.csv.gz
md5:4280d04bded918d9cb984a8f14a0e836 |
43.8 MB | Download |
000000000056.csv.gz
md5:debf36f88e737123418b46d6276d0de4 |
43.9 MB | Download |
000000000057.csv.gz
md5:b2c70ffb6391c46c928e236f850276f8 |
43.9 MB | Download |
000000000058.csv.gz
md5:188b24a1e9475128d0a97667775894d4 |
43.8 MB | Download |
000000000059.csv.gz
md5:c82ae20ab080cc17d2dd9e52764c156d |
43.9 MB | Download |
000000000060.csv.gz
md5:12b37d40f657c3980c19addf20cf4031 |
43.8 MB | Download |
000000000061.csv.gz
md5:478928022302a37792782a0d80af045e |
43.9 MB | Download |
000000000062.csv.gz
md5:8f66deb68525ab8f026c08a3f639e5d1 |
44.0 MB | Download |
000000000063.csv.gz
md5:adc66d3209d6ea60feb3347424e7e86f |
43.9 MB | Download |
000000000064.csv.gz
md5:3eb9c11a74d950668045423602635a0b |
43.9 MB | Download |
000000000065.csv.gz
md5:be57838cb39c9757038af289d62e8853 |
43.9 MB | Download |
000000000066.csv.gz
md5:14adbff5b302b3bf928d40cc156ce3bb |
43.9 MB | Download |
000000000067.csv.gz
md5:3d14c239ec7bf15ae97aad15a590d1e4 |
43.9 MB | Download |
000000000068.csv.gz
md5:395d1b414e42b87a4bd263bafe2eb2f5 |
43.9 MB | Download |
000000000069.csv.gz
md5:06563d44e1915c8b4ebc3ad86aabc1dc |
43.9 MB | Download |
000000000070.csv.gz
md5:18297845e2ba7fe3be2ff903e2583b24 |
43.9 MB | Download |
000000000071.csv.gz
md5:5d25514fbe06b9b67c5be9548595c972 |
44.0 MB | Download |
000000000072.csv.gz
md5:46de4a6a524d190a9d7b1680555ccc6e |
43.9 MB | Download |
000000000073.csv.gz
md5:0fdd357fd423c24b59d0595157bb6e0d |
43.8 MB | Download |
000000000074.csv.gz
md5:23f6a62cd7c5224c69c1f465075978c6 |
44.0 MB | Download |
000000000075.csv.gz
md5:43732062730e485f1f3bfc1229e3528e |
43.9 MB | Download |
000000000076.csv.gz
md5:a874f1946222b84c542cbb1b918a9072 |
43.9 MB | Download |
000000000077.csv.gz
md5:b1a75efc51be803d355077aa52316a2a |
43.9 MB | Download |
000000000078.csv.gz
md5:d433cef7d0e6a38383bf2372ceb54e01 |
43.9 MB | Download |
000000000079.csv.gz
md5:10b3140be6bbf79a4fe5b289fa2154fb |
43.9 MB | Download |
000000000080.csv.gz
md5:0c952d65aa7505b6cfa843f55466bf07 |
43.9 MB | Download |
000000000081.csv.gz
md5:9890e28ebf10fae58bfbaef271a00aef |
43.9 MB | Download |
000000000082.csv.gz
md5:baf099bde0292b80c751868ca0d4c3c5 |
43.9 MB | Download |
000000000083.csv.gz
md5:5f99a79bacfdd9c90542c7a907614292 |
43.9 MB | Download |
000000000084.csv.gz
md5:1384da0fa6aa760a7ce82f630aa871da |
43.9 MB | Download |
000000000085.csv.gz
md5:4797159de2080bd932d204b90559e3ce |
43.9 MB | Download |
000000000086.csv.gz
md5:f90eb1202c46577ac0dc03991fbcd1ec |
43.8 MB | Download |
000000000087.csv.gz
md5:00f1cba4a2401c467b2f01c8dc14433f |
43.9 MB | Download |
000000000088.csv.gz
md5:a800ca5083f1e6aa2905d16516ce5ae1 |
43.9 MB | Download |
000000000089.csv.gz
md5:1f6bacfcb42cf9c119eec3f2c632afa9 |
43.9 MB | Download |
000000000090.csv.gz
md5:53f6f58a9ec8648ffff27b14acad3ae4 |
43.8 MB | Download |
000000000091.csv.gz
md5:58c81183b2fbfbf517f28cfb43ad0b26 |
44.0 MB | Download |
000000000092.csv.gz
md5:90d2144630d0df3a3761b000fef07431 |
43.9 MB | Download |
000000000093.csv.gz
md5:f1971d1e846b216500ce3129d2cb2678 |
43.9 MB | Download |
000000000094.csv.gz
md5:e84a83de8947cf23d7c547682e14e40e |
43.8 MB | Download |
000000000095.csv.gz
md5:6fb3c5cc06792ed65d8bcf8c0d3d0ef9 |
43.9 MB | Download |
000000000096.csv.gz
md5:ba7024b5d9fac128dff5eaff67c5211c |
43.8 MB | Download |
000000000097.csv.gz
md5:4df7bd07cc45ef4a7ff7d73921cf06cb |
43.9 MB | Download |
000000000098.csv.gz
md5:a4770efe788ae92ead36006524d14919 |
43.8 MB | Download |
000000000099.csv.gz
md5:43f745b56bc4cfdc32730764e72d245f |
44.0 MB | Download |
000000000100.csv.gz
md5:761d6c6ad4a455f27c72668c950c5132 |
43.9 MB | Download |
000000000101.csv.gz
md5:4edc20298403f8e830d425e52e5013ad |
44.0 MB | Download |
000000000102.csv.gz
md5:d9ea1a7768ab95d13e4cd40989bbb9f1 |
43.9 MB | Download |
000000000103.csv.gz
md5:446b2c449e85ea9f6d271891ac5ae9a3 |
43.9 MB | Download |
000000000104.csv.gz
md5:9864c4c5d8bdbe276f2b54a16a0e07cf |
44.0 MB | Download |
000000000105.csv.gz
md5:70e3202e6db8b33117157aa1b4825806 |
43.8 MB | Download |
000000000106.csv.gz
md5:d5db6407f7e50c4bcca430ade6032c81 |
43.8 MB | Download |
000000000107.csv.gz
md5:dcc7977bba516cb114d610ecc2669971 |
44.0 MB | Download |
000000000108.csv.gz
md5:2d3e9a2b46b48ae1b2109a650b724973 |
44.0 MB | Download |
000000000109.csv.gz
md5:4afe954d262fbddd01936e505446d825 |
43.8 MB | Download |
000000000110.csv.gz
md5:9ac3c90d21f2b565866f9f8e9d212d50 |
43.9 MB | Download |
000000000111.csv.gz
md5:262105159ca8e1a629921ec42a228028 |
44.0 MB | Download |
000000000112.csv.gz
md5:8bb40e2f2f2fefbc77c74766c6b7ff30 |
43.9 MB | Download |
000000000113.csv.gz
md5:4196d8fb7fe98c317482a85f1e61c130 |
43.8 MB | Download |
000000000114.csv.gz
md5:55824386ab02b9f4f566f0e59db2d90b |
43.9 MB | Download |
000000000115.csv.gz
md5:fe983fd29788b34426fff5375a9a9f17 |
44.0 MB | Download |
000000000116.csv.gz
md5:cbf3ea10e17f506d6ead5bbc849ae307 |
44.0 MB | Download |
000000000117.csv.gz
md5:ab5446023a95c038a886aeaff03cfe1b |
43.8 MB | Download |
000000000118.csv.gz
md5:2f6e05e8ed14f6fb6cfdb7c7feb8ea29 |
43.9 MB | Download |
000000000119.csv.gz
md5:c39e551c8d62eb6a07cc2518de447b45 |
43.9 MB | Download |
000000000120.csv.gz
md5:2dd8ecaba6a20b4c1f4c2f471433df28 |
43.9 MB | Download |
000000000121.csv.gz
md5:4acce80bafe4b59c875568b7463c1d5f |
43.9 MB | Download |
000000000122.csv.gz
md5:0106fed8fb38167cd0f7c42f7e015f02 |
44.0 MB | Download |
000000000123.csv.gz
md5:a3f0e933a83306194e584f3395515211 |
43.9 MB | Download |
000000000124.csv.gz
md5:8faa7b8c4290e6db2dd2a49f8a4bf8cc |
43.8 MB | Download |
000000000125.csv.gz
md5:eb82ce2878e52d916e72cc5e7946f05f |
43.9 MB | Download |
000000000126.csv.gz
md5:0264e0d4c357f63844f7b1c0feb2fdb8 |
43.9 MB | Download |
000000000127.csv.gz
md5:50f22a24ec512e5520f43537fbfadefc |
43.8 MB | Download |
000000000128.csv.gz
md5:62fb4bd37ffb66d357dbe95879998b5a |
44.0 MB | Download |
000000000129.csv.gz
md5:fa9183e6f67419188c722dcc2363f55e |
43.7 MB | Download |
000000000130.csv.gz
md5:65bbe8c98d32333021b3d1c172786b30 |
43.9 MB | Download |
000000000131.csv.gz
md5:a5c52c99168a9245ab39c2d5023809d5 |
43.9 MB | Download |
000000000132.csv.gz
md5:f47c0a767b73540aefccf130d974552c |
43.8 MB | Download |
000000000133.csv.gz
md5:8982392c8ba11fdd95402d1603ee18d2 |
44.0 MB | Download |
000000000134.csv.gz
md5:ec96f22db7c26e8bf23d743cb79fc5df |
43.8 MB | Download |
000000000135.csv.gz
md5:72857604ba2a87b623ce3bcb9ab9ae31 |
44.0 MB | Download |
000000000136.csv.gz
md5:266552d9b0ee35face2973e95a751030 |
43.9 MB | Download |
000000000137.csv.gz
md5:aa46aad209e81347ddbab8df92316fb4 |
43.8 MB | Download |
000000000138.csv.gz
md5:3206d63c4bb0ceaaffd6063b066dfb1a |
43.9 MB | Download |
000000000139.csv.gz
md5:af034116fe6e801d14983a55ae01ad19 |
44.1 MB | Download |
000000000140.csv.gz
md5:857c026be1a4cd290315666128549632 |
44.0 MB | Download |
000000000141.csv.gz
md5:3d0192634963e1e25b98bc75545d406f |
43.9 MB | Download |
000000000142.csv.gz
md5:19ba4570ab17ed764d22b813df6da960 |
43.8 MB | Download |
000000000143.csv.gz
md5:9ce173d1073d0f0b812cdc897a55b7b7 |
44.0 MB | Download |
000000000144.csv.gz
md5:308c8a9eb20510ce524155f6b1b17967 |
43.9 MB | Download |
000000000145.csv.gz
md5:3f0bac27419eaef26d1a74e81bfa4472 |
43.9 MB | Download |
000000000146.csv.gz
md5:abd6fb853fc751c7ebdafcb91d76e7b1 |
43.8 MB | Download |
000000000147.csv.gz
md5:01849d34c30497a0b8633d07f65cae28 |
43.9 MB | Download |
000000000148.csv.gz
md5:cc2159535dacbdf36f6cc36246b75452 |
43.9 MB | Download |
000000000149.csv.gz
md5:71f1731745e35a737937eb1ef5652b72 |
43.9 MB | Download |
000000000150.csv.gz
md5:55113f2781c12d0e1307e7e62dcfea1e |
43.9 MB | Download |
000000000151.csv.gz
md5:c7c3e49ae3a9fc71c638a91778dd37a7 |
43.9 MB | Download |
000000000152.csv.gz
md5:2fb75f9d2b081c911af113514d45a9a8 |
44.0 MB | Download |
000000000153.csv.gz
md5:5e56149b11d32eafcc531da433969944 |
43.8 MB | Download |
000000000154.csv.gz
md5:cbdc1e6b8474e29f3b079c8701224af6 |
43.9 MB | Download |
000000000155.csv.gz
md5:ff26161770824aa425960526e6a929bb |
43.9 MB | Download |
000000000156.csv.gz
md5:b299173a7bbd1bf3c393a5e3a3da8ff7 |
43.9 MB | Download |
000000000157.csv.gz
md5:89b3c881405ac26f9d23cda3fef1bea8 |
43.9 MB | Download |
000000000158.csv.gz
md5:74ecd3398c7828a5ed7f3930246d54c5 |
43.9 MB | Download |
000000000159.csv.gz
md5:b84ffa2be38dd3413375e98294301d9b |
44.0 MB | Download |
000000000160.csv.gz
md5:bb8efc24229e369820a866a6b334b63a |
43.9 MB | Download |
000000000161.csv.gz
md5:f360b87cf117db36ed857e5ceeba0604 |
43.9 MB | Download |
000000000162.csv.gz
md5:19b11af5892be538d2b229268964d5f2 |
43.9 MB | Download |
000000000163.csv.gz
md5:862cfca4933d2c2f9b37e4438ea8eeca |
43.8 MB | Download |
000000000164.csv.gz
md5:d5da9990af177caa6e97f03388da1368 |
44.0 MB | Download |
000000000165.csv.gz
md5:82b9631295396a750d595ec428671683 |
43.9 MB | Download |
000000000166.csv.gz
md5:ea0f6903da27f73c73b3e516358ee893 |
43.9 MB | Download |
000000000167.csv.gz
md5:150cc9643f2f606ae2246fed9684ee78 |
43.9 MB | Download |
000000000168.csv.gz
md5:503a30efc343e40a0d9682ebb3a0d1f5 |
43.9 MB | Download |
000000000169.csv.gz
md5:5e9faf63d0a49ba6ef53d5e60490434a |
43.8 MB | Download |
000000000170.csv.gz
md5:c1561cd99c7656bf573cb844f653c330 |
43.9 MB | Download |
000000000171.csv.gz
md5:ba60a8cd688eec0cab6a87417d93b9fd |
43.9 MB | Download |
000000000172.csv.gz
md5:75e462823e88a662d861805e953e8c9c |
43.9 MB | Download |
000000000173.csv.gz
md5:21b84d7e990516f3a1532a6021d3144c |
43.8 MB | Download |
000000000174.csv.gz
md5:019f0c2459afefb017a18e95dce85348 |
43.8 MB | Download |
000000000175.csv.gz
md5:fc2f985898ab97688962daa9810d8fa4 |
43.9 MB | Download |
000000000176.csv.gz
md5:a8287c572c6a43fb4ec1db8a1d53e522 |
43.9 MB | Download |
iso_language.csv
md5:3923f6de6df2e130c3283f9665c9b68f |
2.2 kB | Download |
All versions | This version | |
---|---|---|
Views | 138 | 138 |
Downloads | 419 | 419 |
Data volume | 16.8 GB | 16.8 GB |
Unique views | 106 | 106 |
Unique downloads | 28 | 28 |