AdEMAMix: A Deep Dive into a New Optimizer for Your Deep Neural Network | Towards Data Science

A better and faster option than the ADAM optimizer, from Apple Research