AdaGrad under heavy-tailed noise: symmetry, asymmetry and damping

Abstract

We ask how fast AdaGrad drives the average gradient norm to zero on smooth nonconvex problems when the stochastic gradient has only a finite p-th moment, 1 < p ≤ 2. With symmetric noise the rate is T^{−(p−1)/(2p)} for every p, and this is sharp at a fixed step size. With general noise and small damping, Liu's slower exponent (3p−4)/(4p) is sharp for every step size, and for p ≤ 4/3 no polynomial rate holds. The cause is a climb: asymmetric noise pushes AdaGrad uphill, and the height it gains pays for a long descent that heavy-tailed noise makes slow. Damping limits that height. For damping λ = σT^α and a fixed step size we find the exponent across the whole window: it grows linearly in α wherever it is positive, and becomes the symmetric one at α = (2−p)/p. The matching upper bounds come from Liu's own lemmas with the damping retained.

Highlights

• Symmetric noise is easy. AdaGrad reaches the rate T^{−(p−1)/(2p)} for every tail index p. Nothing breaks at p = 4/3.
• Lopsided noise is hard. With small damping, Liu's slower rate is the true worst case for every step size, and for p ≤ 4/3 there is no polynomial rate at all.
• Why. Noise with mean zero but a lopsided shape makes AdaGrad climb uphill. The height it gains pays for a long descent that heavy tails make slow.
• Damping repairs it gradually. With λ = σT^α the rate exponent rises linearly in α, wherever it is positive, and reaches the symmetric value at α = (2−p)/p.
• Credit. The matching upper bound is Liu's own argument with the damping kept.

Results at a glance

Rate exponents r with R_T ≈ T^−r up to logarithms, at a fixed step size unless stated (Table 1 of the paper).

For p < 4/3 and pα+2p−3 < 0 the lower bound has no polynomial decay and the magnitude is open.

Figures

Figure 1 (rates.png): the rate exponent under general noise at a fixed step size, against the damping exponent α, for p = 1.2, 1.5, 1.8. Each curve is flat, rises linearly, then stays at the symmetric exponent (p−1)/(2p) up to α = 1/p.

Figure 2 (ridge.png): the hard instance. Starting at x₁, tilt noise drives the iterate up the gentle slope; left of the point b, symmetric Pareto noise stalls it on the steep part.

What is in the paper (16 pages)

• Theorem 3.1 (symmetric noise): an upper bound of order T^{−(p−1)/(2p)} for coordinate-wise AdaGrad in any dimension with λ > 0, and a matching lower bound in dimension 1 at a fixed step size, which holds for every λ ≥ 0.
• Theorem 4.1 (the ridge): a damping-dependent lower bound that holds for every step size, under explicit finite-horizon conditions.
• Corollary 4.2 (small damping): Liu's exponent (3p−4)/(4p) is sharp for every step size; no polynomial rate for p ≤ 4/3.
• Corollary 4.3 (the damping window): the lower-bound exponent max{0, (pβ+2p−3)/(2p)} for λ ≤ σT^α, with β = max{α, (2−p)/(2p)}.
• Theorem 5.1 (the matching upper bound): from the argument for Liu's Lemma A.2 with the damping kept, and his Lemma A.3; no symmetry needed.
• Corollary 5.2: the rate exponents across the window.
• Tables: the rate exponents above (Table 1) and lower against upper bounds across the window (Table 2). Figures: the two above.

Not claimed

• The worst-case instance is chosen after the algorithm parameters and may depend on the horizon. The lower bounds do not show that AdaGrad fails on a fixed problem.
• The lower bounds are for the average gradient norm, not the best iterate.
• For (2−p)/p < α ≤ 1/p, the rates match for each fixed step size; the rate after tuning the step size with the horizon is open.
• Whether one common noise law for the whole run gives the same penalty is open (Section 6).

Files

• adagrad-heavy-tails.pdf: the paper.
• rates.png, ridge.png: Figures 1 and 2.
• adagrad-heavy-tails-v1.0.zip: the paper (PDF and LaTeX source), the figures and the script that draws them, a README, licences and SHA256SUMS.

Licence: CC BY 4.0; the figure script is additionally offered under the MIT License. The paper is AI-assisted; its section "Use of AI tools" says how. Not yet peer reviewed.