Evaluating Language Models on Cross-Language Code Functional Equivalence

Official replication package for the paper "Evaluating Language Models on Cross-Language Code Functional Equivalence."

This repository investigates large language models' (LLMs) ability to judge code functional equivalence — determining whether two programs produce identical outputs for all valid inputs — both within the same programming language and across different programming languages (Java, Python, and C++). It contains the official implementation, datasets, and analysis scripts required to reproduce the experimental results reported in the paper.

Contents

• code/Manual_Analysis/ — Chain-of-Thought (CoT) manual analysis experiments on the o4-mini model, including 81 filtered examples, the analysis notebook, and the annotated spreadsheet (Manual Analysis.xlsx) recording model predictions, source code, human-coded error categories, and reasoning traces.
• code/Regression_Analysis/ — Regression analyses examining how Lines of Code (LOC), code similarity metrics (LexSim, CodeBLEU, CodeBERT, GraphCodeBERT, UniXcoder, BGE-Code), problem difficulty, and programming language relate to model performance.
• code/Statistic_Analysis/ — Raw query results from statistical evaluation experiments across 9 models (CodeLlama7B, CodeLlama13B, GPT-o4-mini, GPT-OSS-20B, Llama3.1-8B, Llama3.2-3B, Mistral7B, Qwen2.5-7B, Qwen2.5-14B) on 3 datasets.
• Dataset/ — All datasets used in the study:
  • PolyHuman — our own constructed multi-language dataset (5,000+ instances) built from Google DeepMind's CodeContests corpus, containing human-written Java, Python, and C++ solutions (2 passing, 1 failing per language) for each competitive-programming problem.
  • SeqcoBench — a benchmark of Python programs with semantic-preserving and semantic-altering transformations, used to test true code semantic understanding.
  • EquiBench — a benchmark of 2,400 program pairs for equivalence checking across four languages and six equivalence categories.

Requirements

Python 3.9+; GPU (e.g., Nvidia L4, 32GB memory, 8 vCPU); OpenAI API token; OpenRouter API token.

Citation

If you use this dataset or code, please cite our paper (citation details to be updated upon publication).