Adaptive Benchmark Generation for Knowledge-Based AI Evaluation

This working paper proposes a knowledge-driven framework for adaptive benchmark generation in the evaluation of knowledge-based AI systems.

The approach treats benchmark design as an explicit methodological component of AI evaluation rather than as a preliminary manual task. It describes how representative organisational knowledge can be analysed to identify relevant concepts, relationships, and reasoning patterns, which can then inform the generation of evaluation benchmarks.

The framework is intended primarily for enterprise AI applications, including retrieval-augmented generation (RAG) systems and knowledge-based AI assistants, where underlying knowledge sources evolve over time and static evaluation datasets may gradually lose their representativeness.