MBPP+ / EvalPlus audit
============================================================

Evaluated solutions: 1344
Tasks in overlap: 224

Original MBPP pass: 994/1344 (73.958%)
EvalPlus base pass: 985/1344 (73.289%)
EvalPlus plus pass: 855/1344 (63.616%)

Original/Base agreement:
1205/1344 (89.658%)

Original/Base discrepancies:
139/1344 (10.342%)

Direction:
PASS -> FAIL: 74
FAIL -> PASS: 65

Entry-point precheck:
Discrepancies with failed entry-point precheck: 12
Discrepancies with valid entry point: 127

Affected tasks:
28/224 tasks showed at least one discrepancy.

19 tasks differed in all 6 model/prompt conditions:
57, 61, 75, 87, 119, 125, 139, 166, 167, 233,
245, 270, 276, 296, 312, 433, 454, 468, 471

Tasks differing in 4/6 conditions:
130, 145, 285

Tasks differing in 3/6 conditions:
406

129/139 discrepancies (92.806%) occurred on tasks affected in
at least 3 of the 6 model/prompt conditions.

Interpretation:
The clustering of discrepancies by task, together with the similar
aggregate pass rates and the fact that most discrepancies occur despite
a valid expected entry point, is consistent with systematic differences
between the original MBPP test cases used in the study and the
sanitized/corrected MBPP base used by EvalPlus.
