AdEMAMix: A Deep Dive into a New Optimizer for Your Deep Neural Network

A better and faster option than the ADAM optimizer, from Apple Research