Urdu-Punjabi Eval v0.1: open benchmark tooling and sample data for Urdu and Shahmukhi Punjabi (AI Excellence Centre, GIFT University)

# Urdu–Punjabi Eval — open benchmark & instruction data for low-resource South Asian languages

**AI Excellence Centre (AIXC), GIFT University — Gujranwala, Pakistan**

This repository is the open home of the AIXC language track: an evaluation benchmark and instruction dataset for **Urdu** and **Punjabi (Shahmukhi script)**, plus the tooling we use to build and validate them with native-speaker students and faculty.

- **Status:** early stage (v0.1). Schema, validator and evaluation harness are working; the dataset is being collected. Sample items are in `data/samples/`.
- **Licence:** code — Apache-2.0 (`LICENSE`); data — CC-BY-4.0 (`DATA_LICENSE`).
- **Contact:** aixc@gift.edu.pk · PI: Dr. Muhammad Faheem, Associate Dean (Computer Science), GIFT University.

## Why

Urdu has 230M+ speakers and Punjabi 100M+, yet there is no open, quality-controlled instruction or evaluation set for either. Models hallucinate, mix Shahmukhi/Gurmukhi scripts, or refuse. AIXC — established in July 2026 by GIFT University, the Gujranwala Business Alliance, NSART LLP (Kazakhstan) and ONIT Global — is building these datasets in the open.

## What is here

```
schema/item.schema.json JSON Schema for a benchmark/instruction item
src/aixc_eval/validate.py validator: schema + script checks (Arabic-script ranges, Gurmukhi leakage) + duplicates
src/aixc_eval/run_eval.py evaluation harness: exact-match / token-F1 / chrF-lite per task, pluggable model backends
src/aixc_eval/backends.py backends: `echo` (dry run), `openai_compat` (any OpenAI-compatible endpoint), `hf_pipeline`
data/samples/items.jsonl sample items (examples of the format; not the dataset)
CONTRIBUTING.md annotation guidelines and the human-in-the-loop process
```

## Item format

Every item is one JSON object per line:

```json
{"id": "ur-rc-000001", "lang": "ur", "script": "arab", "task": "reading_comprehension",
"context": "...", "prompt": "...", "reference": "...", "source": "original",
"validated_by": 2, "adjudicated": true, "domain": "education", "license": "CC-BY-4.0"}
```

`lang` ∈ {`ur`, `pa`}; `task` ∈ {`reading_comprehension`, `qa`, `translation`, `instruction`, `reasoning`}. See `schema/item.schema.json`.

## Quick start

```bash
pip install -e .
python -m aixc_eval.validate data/samples/items.jsonl
python -m aixc_eval.run_eval data/samples/items.jsonl --backend echo
# real model, any OpenAI-compatible server:
python -m aixc_eval.run_eval data/samples/items.jsonl --backend openai_compat --model gemma-2-9b-it --base-url http://localhost:8000/v1
```

## Process (human in the loop)

1. Seed prompts are drafted (by faculty or with LLM assistance) and translated.
2. Every item is validated by **two native speakers** (GIFT students); disagreements go to a faculty adjudicator.
3. Only items with `validated_by >= 2` and `adjudicated == true` enter the benchmark.
4. Error taxonomy and validation stats are published with each release.

## Roadmap (12 months)

- M1–3: guidelines, first 1,000 validated items
- M4–6: 3,000-item benchmark + 20,000-example instruction set, first open release
- M7–9: AI-tutoring pilot (first-year programming/maths, ~200 students), factory defect dataset (vision track)
- M10–12: evaluation report, two papers, v1.0

## Partners

GIFT University · Gujranwala Business Alliance · NSART LLP (Kazakhstan) · ONIT Global