At identical 2-bit precision, one decision about which axis you quantize along swings a benchmark score from 2.88 to 63.53. Keys and values need opposite treatment — and the reason is in the attention equation, not the hardware.
Take Llama-2-13B. Group its key-value cache by a quantization group size of 32 into two bits, while leaving everything else in place — same model, same bit budget, same group sizes, same benchmarks.
Depending on one single element of an implementation decision, CoQA accuracy results in either 2.88 or 63.53. The score using full precision is 66.37.
The decision is not about how many total bits are used. The question is simply what axis do you choose to group across when computing each scale factor? When you decide to use the channel as your grouping dimension (keys) and the token as your grouping dimension (values), you end up somewhere within four points of the full precision performance. If you flip either of those choices, you experience a loss in quality. If you flip both of these choices, the model no longer works.
Four ways to spend the same 2 bits on the same cache. Results from the KIVI ablation on Llama-2-13B at group size 32.
Quantization is usually thought of as just one dial: 8 bits, 4 bits, 2 bits, with smooth cost of accuracy attached. Inside KV cache, it is not like that. It is choosing coordinate systems, and different systems apply to keys and values. This piece explains why. Briefly: quantization error depends on the range of values within groups; keys and values have very different structure; and people often trip up because you can’t derive the right axis from value distribution at all. You have to look at how error changes after attention consumes it. That gives a general principle for compressing intermediate activations and a good reason to doubt reconstruction error as a proxy for quality.
Why the KV Cache Is Where This Bites
During the generation phase, a transformer stores all of the key and value projection (KV) data of tokens that it has previously processed into a cache so it does not have to recalculate this data again. That cache grows linearly with context length and batch size. Eventually, this will result in the cache growing larger than the model itself.
This increase in growth can be easily identified when looking at the memory consumption of different parts of the model. In the KVQuant analysis of LLaMA-7B, weights account for roughly 98 percent of memory at a sequence length of 512, with activations at 2 percent. At 128K context, the ratio inverts to about 16 percent weights and 84 percent KV cache. When we look at an analysis of OPT-175B cited by the KIVI authors, they found similar results. Specifically, at a batch size of 512 with a 512-token prompt, the KV cache reaches 1.2TB — several times the size of the model weights.
However, capacity is only half of the issue here. The GPU must read the entire KV cache from device memory for every single token it generates. This means that while the GPU is reading out the KV cache, the compute cores sit idle. As such, reducing the overall size of the cache both increases the available processing headroom and reduces the time spent waiting for transfers of data.
What Quantization Error Is Actually Made Of
Uniform integer quantization is straightforward mathematically. For a group of numbers, you record the smallest number as a zero point and then divide the range of that group by the number of levels that can be represented to get a step size. You then round each element to the nearest step. Two immediate results follow. First, error per element is bounded by half a step. Second, the step size is the group’s range divided by 2ᴮ − 1. At 2 bits, you have only 4 levels to cover whatever spread exists within that group. So an element that is a hundred times larger compared to its neighbors does not just perform badly. It inflates step size for all other elements sharing the same group, and all of them become coarser together. Group is the unit of damage. Choosing an axis means deciding which elements suffer together. Framing the question differently, it is no longer “how many bits can we afford?” but “where are extreme values and can we isolate them?”
Keys: The Outliers Live in Fixed Channels
Large language models contain activations that are unusually large compared to most activations. Sun and colleagues catalogued these very large activations across different families of models: in Mixtral 8x7B, the largest magnitude is near 7000 while the median feature magnitude is around 0.3 — roughly four orders of magnitude apart. These are very rare; they stay fixed in dimensions that rarely change with input, and they are not accidental. They act as implicit biases, and they are what focus attention on just a few tokens: attention sinks behavior. In key cache, this structure is very clear: specific channels carry very large magnitudes consistently across every token in a sequence. Group along tokens, and every group contains those outlier channels, so every group’s step size is set by the outliers, and all the ordinary channels pay for it. Group along channels, and the outlier channels form their own groups. Their internal range is large but self-contained; the ordinary channels are left alone. Results match. Averaged across layers and heads on Llama-2-13B, KIVI reports key reconstruction error of 13.67 under per-token grouping against 4.55 per-channel, and — more importantly — attention score error of 47.00 against 9.60. Quantizing keys per token produces roughly five times the score error. Scores then agree with meaningful metrics for keys; channel quantization excels on both fronts.
Values: Where the Intuition Breaks
The value cache does not show a channel-outlier pattern. It appears to be fairly flat. On its own, by the range argument, we could expect that either of these axes will produce a similar quality of compression.
They do not. Regardless of how key management is implemented (the 2.80 and 2.88 results), compressing per-channel values collapses the model.
And here’s the catch: if you measured this loss using the raw reconstruction error on the original tensor for which each value was compressed, per-channel value quantization actually looks slightly better, at 3.73 against 4.57. If you validated your compression the obvious way, you would pick the configuration that destroys the model.
Value cache quantization error on Llama-2-13B, measured two ways. The stored-tensor metric and the consumed-output metric disagree by more than an order of magnitude.
The resolution is that the value cache is never read directly. It is consumed by a matrix product: the attention output is a weighted sum of value vectors across tokens, with softmax attention scores as the weights. Because of this, the relevant error is the one introduced during this process and not within the tensors themselves. Measured in terms of the attention output, the order was completely reversed. Relative error reported by KIVI for the attention output due to per-token value vector quantization was 3.55 compared to 49.89 for per-channel quantization — over fourteen times higher for what seemed like the better choice based on how well it was compressed.
The explanation is attention sparsity, which they measured as 84.3 percent. The majority of the information contained in the output can be attributed to a small number of very important tokens. Per-token quantization confines each token’s error to that token, so errors on unimportant tokens get multiplied by near-zero attention weights and effectively vanish. Per-channel quantization smears every token’s error across a shared channel scale, so badly represented tokens contaminate the representation of the ones that matter. The sparsity that makes attention efficient is the same property that makes per-token quantization safe.
The transferable lesson is broader than the KV cache: measure compression error where the tensor is consumed, not where it is stored. An implicit assumption made by reconstruction error is that every component of a tensor has equal weight when contributing to the final output. Attention explicitly does not. Any downstream operation that weights, gates, or sparsifies its input breaks that assumption. Readers familiar with my previous article regarding blindspots in evaluation metrics in retrieval systems will recognize that these results are similar to previously described failures: easily computed metrics that report on something other than what was intended.
Rotary Embeddings Complicate the Keys
There are some issues with using Rotary Position Embeddings (RoPE). RoPE rotates pairs of channels based on the relative position of each token. That mixing partially dissolves the fixed-channel structure that made per-channel key quantization work in the first place — an outlier channel gets rotated into its neighbours, and the neighbours inherit the range. KVQuant’s answer is ordering: quantize keys before the rotation is applied, and apply RoPE after dequantization. Alongside per-channel key quantization, non-uniform datatypes, and isolating a small fraction of outliers, this gets them under 0.1 perplexity degradation at 3 bits, and enables serving LLaMA-7B at up to 1 million tokens of context on a single A100-80GB.
It is also important to understand the level of impact from RoPE. The authors of the paper “RotateKV” reported an increase of 145% in quantization errors once RoPE was added, and noted that outlier channels differ across attention heads — which is why applying one shared rotation matrix everywhere is insufficient, and head-adaptive rotations do better.
The Systems Tax, and Why It Is Not a Detail
Per-token quantization suits decoding well. Each token arrives; you quantize it, add it to the sequence (along the token dimension), nothing else moves.
However, per-channel quantization does not fit. As a channel’s statistics span tokens that have not been generated yet, you can’t compute a scale factor when a token comes in. KIVI’s workaround is to keep the most recent tokens — up to 128 — in full precision in a residual buffer, and quantize in groups once enough have accumulated.
As it happens, the residual buffer becomes load-bearing, rather than just an incidental thing. On GSM8K with Llama-2-7B, full precision scores 13.50. Fully quantized to 2 bits with the correct axes, it scores 5.76. The same axes and the same bits, plus the residual buffer of recently produced tokens at full precision, score 12.74. A sliding window of recently produced tokens at full precision will recover much of what was lost due to aggressive quantization on difficult multi-step problems — which would make sense if we consider which tokens were being attended to by a chain of arithmetic operations.
There is a significant benefit from doing all of these things correctly — as KIVI reports, 2.6 times less peak memory use for Llama-2-7B, allowing for batch sizes up to 4 times larger, as well as 2.35 to 3.47 times better throughput on a real-world service task.
What to Do with This
- Never use one quantizer for both. Use different quantizers for keys (per-channel) and for values (per token). A pipeline that applies a single quantizer to “the KV cache” has probably already sacrificed most of the possible quality when using a small number of bits to represent each value.
- Quantize keys before RoPE. This is a matter of correctness as opposed to a matter of preference.
- Store a full precision window of recently generated tokens. Although storing such a window takes very little memory compared to how large a cache can be, it is precisely this area that generates much of the accuracy for difficult tasks.
- Do not validate on reconstruction error. Always validate based upon the attention output or based upon end task performance. The storage metric is not merely noisy — for values it points the wrong way.
- Do not validate on short-context multiple-choice benchmarks. The KIVI authors deliberately avoid closed-ended tasks like MMLU for this evaluation, because a single decoding step reading output logits barely exercises the cache at all. Any evaluation that does not build a cache over time and then perform generation from it will never be able to observe the failures inherent in your system design.
Where the Work Is Headed
Although there is still some left to do regarding the geometric nature of the problem, many researchers continue to study ways that outlying channels are distributed among the various transformer heads, and how hardware limitations affect which groupings are cheapest: InnerQ folds channel-wise key normalization into the key and query weights during prefill. Therefore, no additional overhead is incurred at runtime. Furthermore, InnerQ stores high precision windows for both recently generated tokens and attention sink tokens. In doing so, InnerQ eliminates the opportunity for outliers in the sink channel to contaminate neighboring channels.
Others propose that instead of storing the entire cache, we should store only enough information to be able to rematerialize the key and/or value(s) on demand from a smaller cached representation.
Finally, it’s important to remember that accuracy is not the only parameter that quantization affects. Recently published research demonstrated alignment degradation resulting from quantizing KV caches. Moreover, this research documented alignment degradation even in production vLLM serving environments utilizing FP8 caches along with a training-free recovery protocol that restored up to 97% of what was lost in terms of alignment. As such, while a configuration may hold onto its benchmark results, it does not necessarily mean that it retains all other relevant parameters you care about.
The General Principle
The idea of quantization has been framed as a “precision budget”: how many bits can I afford to sacrifice? The KV cache shows that the more useful question is structural. Precision is allocated in groups; the group is the unit of damage, and the axis you group along determines which elements share their fate. The correct axis is the one on which your tensor is being consumed, i.e., the way you are using your tensor and NOT how your tensor appears when stored in memory. Keys are used via a dot-product computation against the query. A single corrupted channel will poison all scores. Values are consumed through a sparse-weighted-average computation across tokens. Therefore, a single corrupted token is simply weighted out.
Two tensors of identical dimensions and generated by two consecutive layers are treated differently. It is worth asking of any activation you plan to compress: what operation contracts this away, and does my grouping respect it?
