The Value of Memory in Language Agents
Resumo indisponível. Consulte o paper original.
PAPER / ARXIV:2609.19880
Yi Su, Hong Liu, Guanghua Yu, Jianchen Zhu
RESUMO
The KV cache has become a major bottleneck in deploying LLMs, as its memory footprint grows linearly with sequence length and batch size, imposing substantial pressure on both memory capacity and bandwidth. Among various cache compression techniques, quantization is particularly attractive due to its effectiveness and ease of deployment. However, most existing methods rely on fixed-width quantization, where a b bit representation is inherently limited to 2^b quantization levels. As the width decreases, the number of available levels shrinks exponentially, leading to severe information loss and rapid performance degradation. We further observe that fixed-width quantization fails to exploit the highly non-uniform distribution of KV cache. After rotation and normalization, KV values approximately follow a normal distribution, concentrated near the center and only a small fraction appearing in tails. Nevertheless, fixed-width quantization coding allocates the same number of bits to frequent and rare symbols. Entropy coding naturally exploits such non-uniformity by assigning shorter codewords to frequent symbols and longer codewords to rare symbols, substantially reducing the average number of bits required for representation. However, variable-length output is not suited to parallel attention kernels, where efficient dequantization and computation rely on regular memory layouts and fixed-stride accesses. To bridge this gap, we propose D-Quant, a flexible KV cache quantization framework that introduces a drift mechanism to convert entropy-coded representations of each token into fixed-size bitstreams, enabling regular memory access and parallel dequantization within attention kernels.
NO MESMO MAPA
Resumo indisponível. Consulte o paper original.
Resumo indisponível. Consulte o paper original.
Resumo indisponível. Consulte o paper original.
Resumo indisponível. Consulte o paper original.
Resumo indisponível. Consulte o paper original.