There is already a substantial body of research on KV cache compression and sparse attention. Yet deploying these methods in inference engines such as vLLM and SGLang remains challenging, and the papers often say little about these practical issues.
In our recent work, SparseEngine: Sparse-First Inference Engine, we redesigned the abstractions of an inference engine to make it possible to serve these sparse methods in practice.
Why do we need a sparse-first engine?
One major challenge is that different methods store, access, and update KV states in very different ways. For example:
SnapKV and H2O physically evict KV states, requiring irregular KV storage and GPU memory reclamation.
RetroInfer maintains a cluster-based KV index and uses a specialized attention computation workflow.
KIVI and Palu change the KV storage format, requiring additional management of quantization parameters or compressed representations.
This makes it difficult to accommodate all these methods within a conventional KV cache abstraction. Recent efforts such as Vortex [1] and Tangram [2] have started to support serving some sparse methods, but their fixed definitions and abstractions still make it difficult to support methods that fall outside those boundaries.
Making sparse computation a native engine capability
We therefore designed SparseEngine, a sparse-first inference engine, from the ground up. It currently supports 15 sparse methods across four categories, including SnapKV, H2O, Quest, KVzip, and OmniKV, while retaining essential serving capabilities such as continuous batching and prefix caching.
Figure 1. Method coverage and serving capabilities.
We support 14 model variants, including Qwen, Llama, GLM-4.7-Flash, Gemma 4, and MiniMax-M2.7. These span GQA, MLA, linear attention, and SWA architectures, as well as dense and MoE models. Our goal is to build an inference engine that is usable in practice.
Figure 2. The overall architecture of SparseEngine, with implementation examples for SnapKV and Quest.
Involving sparse methods in cross-request state management
We also let sparse methods participate in state management across requests. Prefix cache hit rate is an important consideration for agent workloads. In common agent scenarios, context compression can break prefix cache matching and incur substantial additional prefill cost. We bring sparse methods into state management through two mechanisms.
1. Chain Cache
Chain Cache allows historical caches that have undergone KV eviction to be reused across turns, avoiding repeated prefill. Static eviction methods such as SnapKV and KVzip remove tokens, disrupting conventional prefix cache matching. To make these methods efficient in agent workloads, we introduce a chain cache. Unlike a radix prefix tree, it maintains a single chain and matches logical token IDs directly, regardless of whether their corresponding KV states have been deleted. If the token IDs match, execution can continue from the retained cache. In principle, this allows a conversation to continue as long as it is not forked. Codex also appears to maintain a session ID, and forking a conversation seems to incur a prefix cache miss; it may use a related approach.
Figure 3. Radix Prefix Cache and Chain Cache. The logical token prefix is retained after physical KV eviction, allowing the remaining KV states to be reused.
2. Prefix-Cache Pruning
Prefix-Cache Pruning supports KV pruning over specified regions of the history, such as older tool outputs, to compress the context while preserving prefix matching. In coding agents, for example, tools may return very long logs or source files that are no longer needed later. We support pruning methods such as SnapKV and KVzip to trim these historical regions and reduce the overall context footprint.
Figure 4. Pruning historical KV states on demand while preserving prefix cache reuse.
Experimental results
For long-context workloads, KV eviction enables higher concurrency, delivering more than 10× the aggregate decoding throughput of vLLM.
At matched concurrency, the best decoding throughput speedup exceeds 2.5×.
For multi-turn agent workloads, the best end-to-end speedup reaches 2.2×.
Figure 5. Decoding throughput for Qwen3-30B-A3B and GLM-4.7-Flash. The top row compares matched concurrency; the bottom row compares throughput at each system’s largest successfully tested batch size.Figure 6. End-to-end speedups of SparseEngine on multi-turn agent workloads.
Paper and code
As KV cache compression and sparse attention are increasingly incorporated into foundation model training, we hope SparseEngine can both accelerate existing models and provide more flexible infrastructure for future natively sparse models and new KV cache optimization methods.
The paper and code are publicly available. We welcome discussions with anyone interested in inference engines, sparse attention, and agent infrastructure!
