4314299
doi
10.5281/zenodo.4314299
oai:zenodo.org:4314299
user-ai_ml
AYADI, Imen
CEREMADE, Université Paris - Dauphine - PSL Research University
Stochastic Runge-Kutta methods and adaptive SGD-G2 stochastic gradient descent
TURINICI, Gabriel
CEREMADE, Université Paris - Dauphine - PSL Research University
info:eu-repo/semantics/openAccess
Creative Commons Attribution Non Commercial No Derivatives 4.0 International
https://creativecommons.org/licenses/by-nc-nd/4.0/legalcode
deep learning
neural network
stochastic gradient descent
machine learning
adaptive learning rate
<p>Presentation at the ICPR 2021 conference</p>
<p>The minimization of the loss function is of paramount importance in deep neural networks. Many popular optimization algorithms have been shown to correspond to some evolution equation of gradient flow type. Inspired by the numerical schemes used for general evolution equations, we introduce a second-order stochastic Runge Kutta method and show that it yields a consistent procedure for the minimization of the loss function. In addition, it can be coupled, in an adaptive framework, with the Stochastic Gradient Descent (SGD) to adjust automatically the learning rate of the SGD. The resulting adaptive SGD, called SGD-G2, shows good results in terms of convergence speed when tested on standard data-sets.</p>
Zenodo
2020-12-10
info:eu-repo/semantics/lecture
4314298
user-ai_ml
1607646430.897923
466713
md5:6c0999494ab95b613fcb977e8f1c4687
https://zenodo.org/records/4314299/files/ICPR_2021_Turinici_Ayadi_v1.pdf
public
10.5281/zenodo.4314298
isVersionOf
doi