Google & UCLA Formulate Algorithm Discovery as Program Search, Yielding ‘Lion’ for SOTA DNN Optimization | Synced

In the new paper Symbolic Discovery of Optimization Algorithms, a research team from Google and UCLA presents a method for formulating algorithm discovery as program search and applies it to find EvoLved Sign Momentum (Lion), a simple and effective optimization algorithm that surpasses state-of-the-art methods while reducing computation costs.