FlexLink: Boost GPU Bandwidth by 27% and Accelerate LLM Training by Unlocking Hidden Hardware Pathways

This is a Plain English Papers summary of a research paper called FlexLink: Boost GPU Bandwidth by 27% and Accelerate LLM Training by Unlocking Hidden Hardware Pathways. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter. The bandwidth bottleneck nobody talks about Training large language models across multiple GPUs seems like a compute problem. The GPUs finish their math so quickly that it feels like hardware abundance. But that intuition is backwards. As models scale to hundreds of billions of parameters, communication between GPUs becomes the actual ceiling on training speed. During a typical training step on distributed systems, GPUs need to synchronize gradients across machines, gather model parameters, and exchange intermediate activations. This happens thousands of times per second. The GPU itself finishes its calculations in microseconds, but waiting for data to arrive from another machine takes milliseconds. That waiting dominates everyth