WEAM Pre-training Effects on Cross-lingual Model Robustness in MLQA Benchmarks

He et al. (2018) have called into question the utility of pre-training by showing that training from scratch can often yield similar performance to pre-training. We show that although pre-training may not improve performance on traditional classification metrics, it improves model robustness and uncertainty estimates. Through extensive experiments on adversarial examples, label corruption, class imbalance, out-of-distribution detection, and confidence calibration, we demonstrate large gains from pre-training and complementary effects with task-specific methods. We introduce adversarial pre-tra

Research goal: To what extent does WEAM pre-training improve the robustness (measured by accuracy on adversarial examples) of cross-lingual models compared to standard pre-training on MLQA benchmarks?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 8.2/10.