A Deep Neural Network Optimization Method Via A Traffic Flow Model

We present, via the solution of nonlinear parabolic partial differential equations (PDEs), a continuous-time formulation for stochastic optimization algorithms used for training deep neural networks. Using continuous-time formulation of stochastic differential equations (SDEs), relaxation approaches like the stochastic gradient descent (SGD) method are interpreted as the solution of nonlinear PDEs that arise from modeling physical problems. We reinterpret, through homogenization of SDEs, the modified SGD algorithm as the solution of the viscous Burgers' equation that models a highway traffic flow.

Final report submitted in partial fulfillment of the African Masters in Machine Intelligence (AMMI) Master's Degree program at the African Institute for Mathematical Sciences in Rwanda. This report reflects studies and research conducted by the first author during the program from 2019 to 2020. The official submission deadline for this report was 31 March 2021. For more information about the program, please visit AIMS-AMMI.

Thanks to the program sponsors, Google and Meta Platforms (previously "Facebook").