A Geometric Understanding of Natural Gradient
Q Bai, S Rosenberg, W Xu - arXiv preprint arXiv:2202.06232, 2022 - arxiv.org
Q Bai, S Rosenberg, W Xu
arXiv preprint arXiv:2202.06232, 2022•arxiv.orgWhile natural gradients have been widely studied from both theoretical and empirical
perspectives, we argue that some fundamental theoretical issues regarding the existence of
gradients in infinite dimensional function spaces remain underexplored. We address these
issues by providing a geometric perspective and mathematical framework for studying
natural gradient that is more complete and rigorous than existing studies. Our results also
establish new connections between natural gradients and RKHS theory, and specifically to …
perspectives, we argue that some fundamental theoretical issues regarding the existence of
gradients in infinite dimensional function spaces remain underexplored. We address these
issues by providing a geometric perspective and mathematical framework for studying
natural gradient that is more complete and rigorous than existing studies. Our results also
establish new connections between natural gradients and RKHS theory, and specifically to …
While natural gradients have been widely studied from both theoretical and empirical perspectives, we argue that some fundamental theoretical issues regarding the existence of gradients in infinite dimensional function spaces remain underexplored. We address these issues by providing a geometric perspective and mathematical framework for studying natural gradient that is more complete and rigorous than existing studies. Our results also establish new connections between natural gradients and RKHS theory, and specifically to the Neural Tangent Kernel (NTK). Based on our theoretical framework, we derive a new family of natural gradients induced by Sobolev metrics and develop computational techniques for efficient approximation in practice. Preliminary experimental results reveal the potential of this new natural gradient variant.
arxiv.org