When Transformers Sing: Adapting SpectralKD for Text-Based Knowledge Distillation | Towards Data Science

Exploring the frequency fingerprints of Transformers to guide smarter knowledge distillation