Google's TurboQuant Cuts LLM KV Cache Memory by 6x, Enables 3-Bit Storage Without Accuracy Loss

Google released TurboQuant, a novel two-stage quantization algorithm that compresses the KV cache in long-context LLMs. It reduces memory by 6x, achieves 3-bit storage with no accuracy drop, and speeds up attention scoring by up to 8x on H100 GPUs. Google's TurboQuant Cuts LLM KV Cache Memory by 6x, Enables 3-Bit Storage Without Accuracy Loss Google has released details on TurboQuant, a new family of "theoretically grounded" quantization algorithms designed to tackle one of the most significant and growing costs in running large language models (LLMs) with long contexts: the Key-Value (KV) cache. The KV cache is a memory structure that stores intermediate representations (keys and values) for every token in a sequence. As a conversation or document grows longer, this cache expands linearly, consuming massive amounts of high-bandwidth memory (HBM). This often becomes the primary bottleneck for long-context inference, not raw compute, due to the cost of moving this data. Standard quantiz