FairEval: Human-Aligned Evaluation Framework for Generative Models

FairEval is an open-source evaluation framework designed to measure human alignment, bias, fairness and toxicity in generative models. It integrates rubric-based LLM-as-Judge scoring, category-wise toxicity detection, fairness auditing and human–model agreement metrics. The system provides dashboards for transparency and supports real-world model analysis across different tasks and demographic groups.

This technical report introduces the design, methodology and benchmarking results of FairEval, along with reproducible experiments and UI components. 

Project GitHub Repository: https://github.com/kritibehl/FairEval