[A1.1] AST vs Sonar (500 reference solutions): identical 455 (91.0%), Sonar higher 3, lower 42; Spearman 0.996; mean AST 2.776 vs Sonar 2.484, diff -0.292 [-0.440, -0.174]; LoA [-3.33, 2.74]
[A1.2] validation vs published (AST, correct = TRUE): max |d rho| = 0, max |d mean| = 0, max rel |d p_holm| = 0
[A1.2] ast   | Claude: rho 0.688, mean ref 2.65 vs gen 1.77 (diff -0.88 [-1.22, -0.56]), below/equal/above 32.9/52.0/15.1%, not higher 84.9%, r_rb -0.468, p_holm 1.60e-13 || Gemini: rho 0.620, mean ref 2.59 vs gen 2.06 (diff -0.53 [-0.90, -0.18]), below/equal/above 29.6/43.7/26.7%, not higher 73.3%, r_rb -0.232, p_holm 3.80e-04 || GPT: rho 0.587, mean ref 2.62 vs gen 2.61 (diff -0.00 [-0.42, +0.40]), below/equal/above 27.7/42.9/29.4%, not higher 70.6%, r_rb +0.014, p_holm 8.19e-01
[A1.2] sonar | Claude: rho 0.696, mean ref 2.31 vs gen 1.77 (diff -0.55 [-0.79, -0.29]), below/equal/above 31.1/52.8/16.1%, not higher 83.9%, r_rb -0.386, p_holm 2.37e-09 || Gemini: rho 0.626, mean ref 2.28 vs gen 2.06 (diff -0.22 [-0.49, +0.05]), below/equal/above 27.8/45.2/27.0%, not higher 73.0%, r_rb -0.151, p_holm 3.31e-02 || GPT: rho 0.592, mean ref 2.30 vs gen 2.61 (diff +0.32 [-0.02, +0.66]), below/equal/above 25.4/43.5/31.1%, not higher 68.9%, r_rb +0.121, p_holm 5.10e-02
[A1.3] validation vs published (AST, correct = TRUE), max abs/rel deviation: ff1 = 3.29e-05, pairwise_p_holm = 4.49e-15, desc_mean = 0.00048, validity_rho = 0, chi2 = 0, power = 5e-04
[A1.3] stratum assignment AST vs Sonar: same stratum 492/500 (98.4%), Cohen's kappa 0.976
[A1.3] ast   | boundaries 0 / 3 | tasks 203/162/135 | proxy rho -0.143 (p 1.33e-03) | pass rate 72.7/56.3/59.4, chi2 72.8 (p < 2.2e-16)
[A1.3] ast   | blocks 262/140/132 | identical 62.6%/38.6%/24.2% | W 0.1428/0.0215/0.1048 | Friedman p < 2.2e-16 / 4.91e-02 / 9.81e-07
[A1.3] ast   | Claude-GPT r_all 0.420/0.221/0.408, r_rb -0.943/-0.366/-0.576, p_holm 3.03e-11 / 2.66e-02 / 8.20e-06 | significant pairs per stratum 3/1/3 | power (n=120) 100.0%/90.2%/100.0%
[A1.3] ast   | mean CC by stratum: Claude 0.18/1.77/4.52, Gemini 0.55/1.94/5.12, GPT 0.98/2.16/6.16 | Claude zero share 91.8%/37.1%/15.9%
[A1.3] sonar | boundaries 0 / 3 | tasks 203/170/127 | proxy rho -0.152 (p 6.43e-04) | pass rate 72.7/56.6/59.2, chi2 72.3 (p < 2.2e-16)
[A1.3] sonar | blocks 262/148/124 | identical 62.6%/37.8%/24.2% | W 0.1428/0.0179/0.1174 | Friedman p < 2.2e-16 / 7.05e-02 / 4.75e-07
[A1.3] sonar | Claude-GPT r_all 0.420/0.220/0.415, r_rb -0.943/-0.363/-0.581, p_holm 3.03e-11 / 2.21e-02 / 1.12e-05 | significant pairs per stratum 3/1/2 | power (n=120) 100.0%/87.2%/100.0%
[A1.3] sonar | mean CC by stratum: Claude 0.18/1.77/4.73, Gemini 0.55/1.92/5.35, GPT 0.98/2.11/6.44 | Claude zero share 91.8%/36.3%/15.4%
[A2] FF1 (534 blocks, cluster bootstrap over 279 tasks, seed 20260809, R = 10000); V, Z, r_all, r_nonzero, p_holm identical to v2
[A2] FF1 Claude_Gemini    r_rb -0.396 [-0.555, -0.228] | mean diff -0.371 [-0.545, -0.191] | r_all 0.224, r_nonzero 0.349 | ties 314/534
[A2] FF1 Claude_GPT       r_rb -0.646 [-0.763, -0.512] | mean diff -0.906 [-1.178, -0.655] | r_all 0.363, r_nonzero 0.564 | ties 313/534
[A2] FF1 Gemini_GPT       r_rb -0.390 [-0.533, -0.232] | mean diff -0.536 [-0.772, -0.317] | r_all 0.221, r_nonzero 0.343 | ties 312/534
[A3] FF1 Claude vs Gemini: means 1.547 vs 1.918, relative difference -19.3% [-28.1%, -10.4%]; second model zero in 247/534 blocks (46.3%, per-pair ratio undefined), both zero in 234; medians 0 vs 1
[A3] FF1 Claude vs GPT: means 1.547 vs 2.453, relative difference -36.9% [-45.0%, -28.8%]; second model zero in 240/534 blocks (44.9%, per-pair ratio undefined), both zero in 233; medians 0 vs 1
[A3] FF1 Gemini vs GPT: means 1.918 vs 2.453, relative difference -21.8% [-29.0%, -14.2%]; second model zero in 240/534 blocks (44.9%, per-pair ratio undefined), both zero in 205; medians 1 vs 1
[A2] FF3 (readability minus standard; cluster = task, per-model seeds 20260909/20261009/20261109); V, Z, r_all, p_holm identical to v2
[A2] FF3 Claude  n = 339 | r_rb -0.366 [-0.601, -0.116] | mean diff -0.183 [-0.327, -0.041] | r_all 0.152, r_nonzero 0.324 | ties 265
[A2] FF3 Gemini  n = 291 | r_rb +0.311 [+0.096, +0.510] | mean diff +0.258 [+0.058, +0.454] | r_all 0.164, r_nonzero 0.273 | ties 186
[A2] FF3 GPT     n = 289 | r_rb +0.414 [+0.202, +0.616] | mean diff +0.304 [+0.083, +0.519] | r_all 0.214, r_nonzero 0.363 | ties 189
[A4] pairwise-complete blocks (both models correct on the same task and prompt); FF1 reference: 534 blocks, all three correct
[A4] Claude vs Gemini: 583 blocks (302 tasks; +49 beyond the 534) | means 1.599 vs 2.022, diff -0.424 [-0.609, -0.244] | ties 57.6% | r_all 0.233, r_nonzero 0.359, r_rb -0.407 [-0.561, -0.247] | p_holm 3.48e-08
[A4] Claude vs GPT: 583 blocks (304 tasks; +49 beyond the 534) | means 1.621 vs 2.556, diff -0.935 [-1.202, -0.685] | ties 58.1% | r_all 0.369, r_nonzero 0.570, r_rb -0.653 [-0.761, -0.527] | p_holm < 2.2e-16
[A4] Gemini vs GPT: 544 blocks (284 tasks; +10 beyond the 534) | means 1.941 vs 2.506, diff -0.564 [-0.793, -0.347] | ties 58.1% | r_all 0.230, r_nonzero 0.356, r_rb -0.405 [-0.543, -0.250] | p_holm 7.89e-08
[A5] 500 tasks per model; 'pass under both prompts' equals the FF3 paired sample (339/291/289)
[A5] Claude  both 339 | standard only 19 | readability only 11 | neither 131 | switching 30 (6.0%) | pass rate 71.6% -> 70.0% | exact McNemar p  0.2 (Holm 0.401), asymptotic chi2 1.63 p 0.201
[A5] Gemini  both 291 | standard only 15 | readability only 7 | neither 187 | switching 22 (4.4%) | pass rate 61.2% -> 59.6% | exact McNemar p 0.134 (Holm 0.401), asymptotic chi2 2.23 p 0.136
[A5] GPT     both 289 | standard only 14 | readability only 10 | neither 187 | switching 24 (4.8%) | pass rate 60.6% -> 59.8% | exact McNemar p 0.541 (Holm 0.541), asymptotic chi2 0.38 p 0.54
[A6] rmcorr via lm() with task_id as cluster; cluster bootstrap R = 10000, seeds as in FF2 v2
[A6] pooled            cyclomatic_complexity n = 1914 (383 tasks, 356 repeated) | pooled Spearman 0.948 [0.930, 0.963], Pearson 0.943 | rmcorr 0.887 [0.859, 0.913] (analytic [0.876, 0.897], df 1530, p < 2.2e-16) | rank rmcorr 0.919 [0.893, 0.942]
[A6] pooled            lines_of_code         n = 1914 (383 tasks, 356 repeated) | pooled Spearman 0.912 [0.894, 0.926], Pearson 0.879 | rmcorr 0.788 [0.726, 0.845] (analytic [0.769, 0.807], df 1530, p < 2.2e-16) | rank rmcorr 0.836 [0.803, 0.864]
[A6] pooled            code_smells           n = 1914 (383 tasks, 356 repeated) | pooled Spearman 0.168 [0.084, 0.251], Pearson 0.132 | rmcorr 0.056 [-0.092, 0.198] (analytic [0.006, 0.106], df 1530, p 2.76e-02) | rank rmcorr -0.062 [-0.131, 0.001]
[A6] Claude            cyclomatic_complexity n =  708 (369 tasks, 339 repeated) | pooled Spearman 0.928 [0.902, 0.951], Pearson 0.929 | rmcorr 0.883 [0.810, 0.936] (analytic [0.857, 0.904], df 338, p < 2.2e-16) | rank rmcorr 0.900 [0.822, 0.959]
[A6] Claude            lines_of_code         n =  708 (369 tasks, 339 repeated) | pooled Spearman 0.899 [0.876, 0.918], Pearson 0.884 | rmcorr 0.793 [0.657, 0.882] (analytic [0.750, 0.830], df 338, p < 2.2e-16) | rank rmcorr 0.758 [0.653, 0.834]
[A6] Claude            code_smells           n =  708 (369 tasks, 339 repeated) | pooled Spearman 0.170 [0.072, 0.266], Pearson 0.108 | rmcorr 0.237 [0.000, 0.445] (analytic [0.134, 0.335], df 338, p 1.00e-05) | rank rmcorr -0.009 [-0.138, 0.104]
[A6] Gemini            cyclomatic_complexity n =  604 (313 tasks, 291 repeated) | pooled Spearman 0.947 [0.925, 0.966], Pearson 0.943 | rmcorr 0.865 [0.807, 0.908] (analytic [0.833, 0.891], df 290, p < 2.2e-16) | rank rmcorr 0.878 [0.832, 0.918]
[A6] Gemini            lines_of_code         n =  604 (313 tasks, 291 repeated) | pooled Spearman 0.901 [0.879, 0.920], Pearson 0.869 | rmcorr 0.730 [0.594, 0.826] (analytic [0.671, 0.779], df 290, p < 2.2e-16) | rank rmcorr 0.788 [0.719, 0.842]
[A6] Gemini            code_smells           n =  604 (313 tasks, 291 repeated) | pooled Spearman 0.120 [0.017, 0.218], Pearson 0.081 | rmcorr 0.105 [-0.027, 0.247] (analytic [-0.010, 0.218], df 290, p 7.28e-02) | rank rmcorr 0.002 [-0.046, 0.046]
[A6] GPT               cyclomatic_complexity n =  602 (313 tasks, 289 repeated) | pooled Spearman 0.965 [0.950, 0.978], Pearson 0.951 | rmcorr 0.855 [0.760, 0.927] (analytic [0.820, 0.883], df 288, p < 2.2e-16) | rank rmcorr 0.937 [0.907, 0.966]
[A6] GPT               lines_of_code         n =  602 (313 tasks, 289 repeated) | pooled Spearman 0.930 [0.909, 0.947], Pearson 0.889 | rmcorr 0.781 [0.651, 0.886] (analytic [0.732, 0.823], df 288, p < 2.2e-16) | rank rmcorr 0.914 [0.873, 0.942]
[A6] GPT               code_smells           n =  602 (313 tasks, 289 repeated) | pooled Spearman 0.219 [0.118, 0.318], Pearson 0.210 | rmcorr 0.095 [-0.279, 0.478] (analytic [-0.020, 0.209], df 288, p 1.05e-01) | rank rmcorr -0.077 [-0.221, 0.018]
[A7] inclusion criterion (EvalPlus base test identical to the original test): 180 of 224 overlap tasks kept, 44 excluded (A7_excluded_tasks.csv)
[A7] S1 -> S2 drops 35 blocks; plus failures per model among S1 blocks: Claude 27, Gemini 30, GPT 26; blocks with >1 failing model: 29
[A7] S0_all_534             534 blocks (279 tasks) | Friedman chi2 84.48 p < 2.2e-16 | W 0.0791 | identical 46.8%, all zero 37.5% | order Claude < Gemini < GPT | mean ranks 1.795/2.043/2.162 | means 1.547/1.918/2.453
[A7] S1_unchanged_original  246 blocks (126 tasks) | Friedman chi2 54.26 p 1.65e-12 | W 0.1103 | identical 50.4%, all zero 42.3% | order Claude < Gemini < GPT | mean ranks 1.764/2.059/2.177 | means 1.240/1.585/2.256
[A7] S2_unchanged_plus      211 blocks (109 tasks) | Friedman chi2 34.83 p 2.74e-08 | W 0.0825 | identical 53.6%, all zero 44.5% | order Claude < Gemini < GPT | mean ranks 1.796/2.090/2.114 | means 1.114/1.412/1.882
[A7] S0_all_534             Claude vs Gemini: diff -0.371 [-0.545, -0.191] | ties 58.8% | r_all 0.224, r_nonzero 0.349, r_rb -0.396 [-0.555, -0.228] | p_holm 4.57e-07
[A7] S0_all_534             Claude vs GPT: diff -0.906 [-1.178, -0.655] | ties 58.6% | r_all 0.363, r_nonzero 0.564, r_rb -0.646 [-0.763, -0.512] | p_holm < 2.2e-16
[A7] S0_all_534             Gemini vs GPT: diff -0.536 [-0.772, -0.317] | ties 58.4% | r_all 0.221, r_nonzero 0.343, r_rb -0.390 [-0.533, -0.232] | p_holm 4.57e-07
[A7] S1_unchanged_original  Claude vs Gemini: diff -0.346 [-0.573, -0.115] | ties 61.8% | r_all 0.258, r_nonzero 0.417, r_rb -0.468 [-0.703, -0.223] | p_holm 1.04e-04
[A7] S1_unchanged_original  Claude vs GPT: diff -1.016 [-1.449, -0.641] | ties 64.2% | r_all 0.404, r_nonzero 0.676, r_rb -0.774 [-0.920, -0.595] | p_holm 6.72e-10
[A7] S1_unchanged_original  Gemini vs GPT: diff -0.671 [-1.086, -0.323] | ties 59.8% | r_all 0.253, r_nonzero 0.399, r_rb -0.453 [-0.653, -0.210] | p_holm 1.04e-04
[A7] S2_unchanged_plus      Claude vs Gemini: diff -0.299 [-0.519, -0.075] | ties 63.0% | r_all 0.260, r_nonzero 0.428, r_rb -0.477 [-0.734, -0.217] | p_holm 3.18e-04
[A7] S2_unchanged_plus      Claude vs GPT: diff -0.768 [-1.181, -0.407] | ties 68.7% | r_all 0.339, r_nonzero 0.606, r_rb -0.693 [-0.885, -0.453] | p_holm 2.54e-06
[A7] S2_unchanged_plus      Gemini vs GPT: diff -0.469 [-0.856, -0.131] | ties 63.0% | r_all 0.175, r_nonzero 0.288, r_rb -0.327 [-0.573, -0.034] | p_holm 1.09e-02
