[code] raw text -> extraction rule -> code column: {"claude-sonnet-4-6|equal_after_strip": 5, "claude-sonnet-4-6|exact": 995, "gemini-2.5-pro|equal_after_strip": 974, "gemini-2.5-pro|exact": 26, "gpt-5.4|exact": 1000}
[code] Gemini raw API JSONL: {"rows": 1000, "http200": 1000, "model_gemini-2.5-pro": 1000, "finish_stop": 1000, "content==raw_text": 1000}
[code] EvalPlus sample JSONL vs code column: {"rows": 1344, "solution==code": 1344}
[code] tasks whose prompt signature (first def of the reference) is not the function called by the tests: 20; passing generations on these tasks: 1/120
[C2] rows=1344; flag vs raw-status mismatches=0; pass_mbpp_original vs all_solutions mismatches=0
[C2] raw (base,plus) status combos: {"pass/pass": 855, "fail/fail": 356, "pass/fail": 130, "fail/pass": 3}
[C2] tasks with EvalPlus base test identical to original: 180; modified: 44
[C2] rows by (base test identical, discrepancy): {"False|False": 125, "False|True": 139, "True|False": 1080}
[C2] discrepancy rows: 139 on 28 tasks; direction: {"FAIL->PASS": 65, "PASS->FAIL": 74}
[C2]   FAIL->PASS  27  F->P: expected_output_type_changed
[C2]   FAIL->PASS  24  F->P: float_tolerance
[C2]   FAIL->PASS   6  F->P: inputs_changed
[C2]   FAIL->PASS   4  F->P: inputs_changed;expected_output_type_changed
[C2]   FAIL->PASS   4  F->P: set_equality
[C2]   PASS->FAIL   6  P->F: EvalPlus entry point 'find_even_pair' not defined
[C2]   PASS->FAIL   6  P->F: EvalPlus entry point 'next_power_of_2' not defined
[C2]   PASS->FAIL  60  P->F: TypeError
[C2]   PASS->FAIL   2  P->F: expected_output_type_changed
[C2] overlap == MBPP+ (HF, 378 tasks) restricted to 11-510: True; overlap subset of sanitized MBPP (257 tasks in 11-510): True; sanitized tasks absent from MBPP+: 33
[C1] rows=500; ids 11-510 complete: True; non-integer/blank metric cells: 0
[C1] reference/task_*.py byte-identical to MBPP 'code' field: 500/500
[C1] scanner logs: {"return_code_0": 500, "EXECUTION_SUCCESS": 500, "SonarScanner_CLI_7.2.0.5079": 500, "SonarQube_26.4.0.121862": 500, "profile_Sonar_way": 500, "project_key_matches": 500, "measure_error": 0}
[C1] generated solutions AST-identical to the reference: 74 ({"claude-sonnet-4-6": 45, "gpt-5.4": 25, "gemini-2.5-pro": 4}); all four Sonar metrics equal in 74/74
