Published November 4, 2026 | Version 1

Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects

  • 1. Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS
  • 2. University of Bonn
  • 3. Lamarr Institute for Machine Learning and Artificial Intelligence

Description

This preprint evaluates 67 systems on 43 Arabic-focused tasks across nine task families using a shared zero-shot protocol. It compares Jev with other decision models and large language models, and examines content-matched differences between Modern Standard Arabic and Arabic dialects within Jev 1.14.0.

 

Jev 1.14.0 achieves a mean task-primary metric of 69.24 on a scale multiplied by 100, compared with 68.77 for Jev 1.13.0 and 71.81 for Gemma-4-31B. Relative performance varies across task families. The analyses are descriptive; aggregate scores should be interpreted alongside task coverage, task-family results and matched language comparisons.

 

The record includes the manuscript and a companion archive containing LaTeX source, complete primary scores and coverage, sensitivity analyses, matched correctness summaries, system and dataset identifiers, templates, recorded per-request processing rates and offline validation scripts. Timing definitions distinguish request wall time from amortized LLM batch processing. Underlying dataset texts and full model responses are not included. Historical hypothesis outputs are retained with their validation limitations explicitly documented.

 

Files

Files (13.7 GB)

Name Size
md5:f7f14a58b5aafba395257556585f4745
110.5 MB Download
md5:403578000580bb870962050b8f961fc2
204.2 MB Download
md5:e6cb7ad82d8107b134a006155b054e44
202.5 MB Download
md5:11ac087a53e1231ea0196c9e7eb3eade
208.4 MB Download
md5:9b0668c4d00bda41bd00d28a32fffe71
210.5 MB Download
md5:a85bfe6972266067f7fedbc24d7e2c35
201.0 MB Download
md5:8862f55b4c7d9165bc92bc6727a9df30
218.8 MB Download
md5:a1dea69048879a08b65e6a18a482d415
188.2 MB Download
md5:5adf16883c474612fb98d5a067e8930f
188.0 MB Download
md5:973af7976303a103714e61bd5b0ed0d7
192.5 MB Download
md5:6240b74034734b768f58eeb84f6f56b3
206.2 MB Download
md5:93c714247518fb77de9a19ce22d7638a
209.1 MB Download
md5:bd39f582850114ce6232880b98340547
207.4 MB Download
md5:6a8899d8333d2ce53a79922cb565b191
206.8 MB Download
md5:ba38a0b3c44c43c6caf6cf9f0a525ab5
206.1 MB Download
md5:41b26e3e2ad88ce3d37ff1f464220792
230.3 MB Download
md5:0527319fa6124effd7f3138c94be9d9d
233.4 MB Download
md5:6c7cab15c5ded47f054e6879c07c9a96
210.2 MB Download
md5:5e39941e56c862a2b6a9bfe2591513ce
106.1 MB Download
md5:3d1c6a29878dfd3865257e5346a66eb4
89.5 MB Download
md5:4f6140283ac1c50d43351660443137bd
213.7 MB Download
md5:6856979d844cfb0f691f8bb7960b7257
213.7 MB Download
md5:c1b8a7958195342d1eea74ee89d6e15d
213.6 MB Download
md5:997f122d71a0145725e7edff64e6a589
210.0 MB Download
md5:f05e1fd74704bb60be97c56ac9de41c6
208.8 MB Download
md5:de3b057e817b14688390104b95e7f1a6
211.7 MB Download
md5:695cf1f6ee60d77834b33f99dacc3150
217.0 MB Download
md5:676a3e4e0df04e3a61a782872835f347
138.8 MB Download
md5:3a93e1546c24d6205ebfc44d127a6b84
137.3 MB Download
md5:d7355c74c5e088bb3dde62ce1984a3c3
137.6 MB Download
md5:c78afd2b1dddfe203066f627ed32fe89
204.5 MB Download
md5:15c23399c0be118b968e46b94655ff38
203.4 MB Download
md5:4b8ee7130f6ba24548a79b1d386509af
152.7 MB Download
md5:987f69f3db9904a94df7f2de57a8f71b
152.9 MB Download
md5:372f15377a7255c63a8b9bfddcf0920a
204.8 MB Download
md5:a5d6a625ad4da7c1651b5cbe39537fb7
208.0 MB Download
md5:82242c693304190782c9e3484f86d9ea
210.6 MB Download
md5:9159b2b8ca37fdc27ed95275ade7e1e0
211.8 MB Download
md5:306373feb6c05fe6d8bcfe271ff76ab0
227.1 MB Download
md5:ecdc7049c534184147cb3fba86b5def3
217.8 MB Download
md5:1d435f6328c11afcb8e003b48c9936ce
96.7 MB Download
md5:1bec87fe78873264cb8bf5968b219a19
190.5 MB Download
md5:82bf4e3637a87e288946a89697c6f9ef
219.3 MB Download
md5:e4db34143af16e4bab13f3957170e1c7
218.8 MB Download
md5:6e8b3aa687da1eb7fb46c800a7a1af0a
220.9 MB Download
md5:102feedaf6f063dd1b5248d1c9810075
220.4 MB Download
md5:ef62c7c2099f5e4d7179462a4a860614
214.9 MB Download
md5:b8c4a24f3bc0bbe4f0e6b8141a2a9ff9
192.3 MB Download
md5:f1e01b56d1f215d60e487933d67d7c65
204.2 MB Download
md5:4673c1cd539dccaf7a08c86dece5074c
193.8 MB Download
md5:243963895424548954dc6d95ba758688
207.1 MB Download
md5:6d22a5d2478d108c1ddd9854f7716289
207.5 MB Download
md5:c9688b730280134a55fe78503ee627ad
206.4 MB Download
md5:012f6502d7bba82aa4ed40d07cf7b171
209.9 MB Download
md5:f6e6b6c7854ca931ca4a1947d1d253fc
210.9 MB Download
md5:9fbcadf8b2ed4f210a46be77f10340c1
217.6 MB Download
md5:08d9adec772d1c2585dd56ac82c29815
209.1 MB Download
md5:81105c1c0570a6932e74d7899e5c7ff1
209.6 MB Download
md5:8f8e77c1c0dbd9ab709a132952452fda
193.2 MB Download
md5:311e22c62131602660f7d77853f57309
216.3 MB Download
md5:01aa254e0e1f275ccc5bd038310e29b6
216.5 MB Download
md5:a3bb53ffaace8f4206f4e31c30e1596e
216.4 MB Download
md5:5c513366532053d549209826934db44f
216.4 MB Download
md5:4eb79cdf5290336759d2cc812ff7b37f
216.3 MB Download
md5:06fb73c2a71578c26bca880d3531b575
215.3 MB Download
md5:1c7da8cfd445411e097d88c23df78e22
218.5 MB Download
md5:f64f27e51d62653bc98956164c134b1f
217.9 MB Download
md5:23c872bcabee00c3804ae3dc005a38df
217.6 MB Download
md5:e84afa49133d812af933501f8717ca78
217.9 MB Download