Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects
Authors/Creators
- 1. Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS
- 2. University of Bonn
- 3. Lamarr Institute for Machine Learning and Artificial Intelligence
Description
This preprint evaluates 67 systems on 43 Arabic-focused tasks across nine task families using a shared zero-shot protocol. It compares Jev with other decision models and large language models, and examines content-matched differences between Modern Standard Arabic and Arabic dialects within Jev 1.14.0.
Jev 1.14.0 achieves a mean task-primary metric of 69.24 on a scale multiplied by 100, compared with 68.77 for Jev 1.13.0 and 71.81 for Gemma-4-31B. Relative performance varies across task families. The analyses are descriptive; aggregate scores should be interpreted alongside task coverage, task-family results and matched language comparisons.
The record includes the manuscript and a companion archive containing LaTeX source, complete primary scores and coverage, sensitivity analyses, matched correctness summaries, system and dataset identifiers, templates, recorded per-request processing rates and offline validation scripts. Timing definitions distinguish request wall time from amortized LLM batch processing. Underlying dataset texts and full model responses are not included. Historical hypothesis outputs are retained with their validation limitations explicitly documented.
Files
Files
(13.7 GB)
| Name | Size | |
|---|---|---|
|
md5:f7f14a58b5aafba395257556585f4745
|
110.5 MB | Download |
|
md5:403578000580bb870962050b8f961fc2
|
204.2 MB | Download |
|
md5:e6cb7ad82d8107b134a006155b054e44
|
202.5 MB | Download |
|
md5:11ac087a53e1231ea0196c9e7eb3eade
|
208.4 MB | Download |
|
md5:9b0668c4d00bda41bd00d28a32fffe71
|
210.5 MB | Download |
|
md5:a85bfe6972266067f7fedbc24d7e2c35
|
201.0 MB | Download |
|
md5:8862f55b4c7d9165bc92bc6727a9df30
|
218.8 MB | Download |
|
md5:a1dea69048879a08b65e6a18a482d415
|
188.2 MB | Download |
|
md5:5adf16883c474612fb98d5a067e8930f
|
188.0 MB | Download |
|
md5:973af7976303a103714e61bd5b0ed0d7
|
192.5 MB | Download |
|
md5:6240b74034734b768f58eeb84f6f56b3
|
206.2 MB | Download |
|
md5:93c714247518fb77de9a19ce22d7638a
|
209.1 MB | Download |
|
md5:bd39f582850114ce6232880b98340547
|
207.4 MB | Download |
|
md5:6a8899d8333d2ce53a79922cb565b191
|
206.8 MB | Download |
|
md5:ba38a0b3c44c43c6caf6cf9f0a525ab5
|
206.1 MB | Download |
|
md5:41b26e3e2ad88ce3d37ff1f464220792
|
230.3 MB | Download |
|
md5:0527319fa6124effd7f3138c94be9d9d
|
233.4 MB | Download |
|
md5:6c7cab15c5ded47f054e6879c07c9a96
|
210.2 MB | Download |
|
md5:5e39941e56c862a2b6a9bfe2591513ce
|
106.1 MB | Download |
|
md5:3d1c6a29878dfd3865257e5346a66eb4
|
89.5 MB | Download |
|
md5:4f6140283ac1c50d43351660443137bd
|
213.7 MB | Download |
|
md5:6856979d844cfb0f691f8bb7960b7257
|
213.7 MB | Download |
|
md5:c1b8a7958195342d1eea74ee89d6e15d
|
213.6 MB | Download |
|
md5:997f122d71a0145725e7edff64e6a589
|
210.0 MB | Download |
|
md5:f05e1fd74704bb60be97c56ac9de41c6
|
208.8 MB | Download |
|
md5:de3b057e817b14688390104b95e7f1a6
|
211.7 MB | Download |
|
md5:695cf1f6ee60d77834b33f99dacc3150
|
217.0 MB | Download |
|
md5:676a3e4e0df04e3a61a782872835f347
|
138.8 MB | Download |
|
md5:3a93e1546c24d6205ebfc44d127a6b84
|
137.3 MB | Download |
|
md5:d7355c74c5e088bb3dde62ce1984a3c3
|
137.6 MB | Download |
|
md5:c78afd2b1dddfe203066f627ed32fe89
|
204.5 MB | Download |
|
md5:15c23399c0be118b968e46b94655ff38
|
203.4 MB | Download |
|
md5:4b8ee7130f6ba24548a79b1d386509af
|
152.7 MB | Download |
|
md5:987f69f3db9904a94df7f2de57a8f71b
|
152.9 MB | Download |
|
md5:372f15377a7255c63a8b9bfddcf0920a
|
204.8 MB | Download |
|
md5:a5d6a625ad4da7c1651b5cbe39537fb7
|
208.0 MB | Download |
|
md5:82242c693304190782c9e3484f86d9ea
|
210.6 MB | Download |
|
md5:9159b2b8ca37fdc27ed95275ade7e1e0
|
211.8 MB | Download |
|
md5:306373feb6c05fe6d8bcfe271ff76ab0
|
227.1 MB | Download |
|
md5:ecdc7049c534184147cb3fba86b5def3
|
217.8 MB | Download |
|
md5:1d435f6328c11afcb8e003b48c9936ce
|
96.7 MB | Download |
|
md5:1bec87fe78873264cb8bf5968b219a19
|
190.5 MB | Download |
|
md5:82bf4e3637a87e288946a89697c6f9ef
|
219.3 MB | Download |
|
md5:e4db34143af16e4bab13f3957170e1c7
|
218.8 MB | Download |
|
md5:6e8b3aa687da1eb7fb46c800a7a1af0a
|
220.9 MB | Download |
|
md5:102feedaf6f063dd1b5248d1c9810075
|
220.4 MB | Download |
|
md5:ef62c7c2099f5e4d7179462a4a860614
|
214.9 MB | Download |
|
md5:b8c4a24f3bc0bbe4f0e6b8141a2a9ff9
|
192.3 MB | Download |
|
md5:f1e01b56d1f215d60e487933d67d7c65
|
204.2 MB | Download |
|
md5:4673c1cd539dccaf7a08c86dece5074c
|
193.8 MB | Download |
|
md5:243963895424548954dc6d95ba758688
|
207.1 MB | Download |
|
md5:6d22a5d2478d108c1ddd9854f7716289
|
207.5 MB | Download |
|
md5:c9688b730280134a55fe78503ee627ad
|
206.4 MB | Download |
|
md5:012f6502d7bba82aa4ed40d07cf7b171
|
209.9 MB | Download |
|
md5:f6e6b6c7854ca931ca4a1947d1d253fc
|
210.9 MB | Download |
|
md5:9fbcadf8b2ed4f210a46be77f10340c1
|
217.6 MB | Download |
|
md5:08d9adec772d1c2585dd56ac82c29815
|
209.1 MB | Download |
|
md5:81105c1c0570a6932e74d7899e5c7ff1
|
209.6 MB | Download |
|
md5:8f8e77c1c0dbd9ab709a132952452fda
|
193.2 MB | Download |
|
md5:311e22c62131602660f7d77853f57309
|
216.3 MB | Download |
|
md5:01aa254e0e1f275ccc5bd038310e29b6
|
216.5 MB | Download |
|
md5:a3bb53ffaace8f4206f4e31c30e1596e
|
216.4 MB | Download |
|
md5:5c513366532053d549209826934db44f
|
216.4 MB | Download |
|
md5:4eb79cdf5290336759d2cc812ff7b37f
|
216.3 MB | Download |
|
md5:06fb73c2a71578c26bca880d3531b575
|
215.3 MB | Download |
|
md5:1c7da8cfd445411e097d88c23df78e22
|
218.5 MB | Download |
|
md5:f64f27e51d62653bc98956164c134b1f
|
217.9 MB | Download |
|
md5:23c872bcabee00c3804ae3dc005a38df
|
217.6 MB | Download |
|
md5:e84afa49133d812af933501f8717ca78
|
217.9 MB | Download |