An Lyapunov Analysis of the Lion Optimizer

Abstract: Lion (Evolved Sign Momentum) is an optimizer discovered through program search (https://arxiv.org/abs/2302.06675). It was shown to perform comparably or favorably to AdamW but with higher memory efficiency by eliminating the need of updating and storing the second order moment. However, as the output of a random search program, it was unclear whether Lion is indeed a correct and theoretically-principled general purpose optimizer. In this talk, I will discuss this work (https://arxiv.org/abs/2310.05898) in which we show that the dynamics of Lion reveals intriguing mathematical structure, whose convergence is verified by a (more or less) sophostic Lyapunov function. We show that Lion in fact solves the bound constrained problem of min L(w) s.t. |w| <= bound, where the bound depends on the weight decay coefficient. We will also discuss the application of Lion to communication efficient distributed learning.  

 

Speaker Bio: Qiang Liu is an associate professor of Computer Science at UT Austin. He is interested in mathematical and computational techniques for AI and machine learning.