Comparative Analysis of CodeT5 and JaCoText Robustness to Semantic and Syntactic Noise on MBPP Pro

Code generation models have achieved impressive performance. However, they tend to be brittle as slight edits to a prompt could lead to very different generations; these robustness properties, critical for user experience when deployed in real-life applications, are not well understood. Most existing works on robustness in text or code tasks have focused on classification, while robustness in generation tasks is an uncharted area and to date there is no comprehensive benchmark for robustness in code generation. In this paper, we propose ReCode, a comprehensive robustness evaluation benchmark f

Research goal: How does the pass@1 degradation of CodeT5 compare to JaCoText on MBPP Pro when subjected to semantic-preserving docstring perturbations versus syntactic code structure noise?

Autonomous synthesis report generated by SOVEREIGN Research Kernel. Tribunal consensus score: 8.3/10.