HeyHomees Lab Notes · No. 7 · Tsinghua-NICS-EFC · Published October 22, 2026
By HeyHomees Newsroom ·
In 30 seconds
- Agents can drop selected cache entries and reposition retained tokens without rewriting the trajectory.
- A rolling-drop policy improves benchmark scores by 1.7 points and cuts average cost by 27.7%.
- AKV achieves up to 1.2× higher output throughput and 1.5× faster P95 latency.
What is it?
AKV is a cache-engineering system for large language model agents that adds drop and reposition operations to the KV cache, allowing agents to control active cache without modifying the conversation history or re-prefilling retained context.
How does it work?
Agents use a CachePolicy to specify which messages’ KV entries to drop (drop_message) and when to compact token positions (reposition). The AKV serving engine provides a stateless API that internally manages cache reuse, eviction, and restoration while remaining compatible with existing OpenAI-compatible serving APIs.
How well does it do?
A tool-response rolling-drop policy raises average benchmark scores by 1.7 points and lowers average cost by 27.7% versus threshold-triggered summarization. AKV also reaches up to 1.2× output throughput and a 1.5× speedup in P95 request latency, and recursive self-improvement of cache policies adds a further 7.1% cost reduction.
What does it need to run?
Running AKV requires Rust and Cargo, a Python virtual environment, the SGLang inference system, and an LLM such as GPT-OSS-120B served via an OpenAI-compatible Chat Completions API.
By the numbers
- Benchmark score improvement: 1.7 points
- Average cost reduction: 27.7%
- Output throughput increase: 1.2x
- P95 latency speedup: 1.5x
- Recursive policy cost reduction: 7.1%
The honest read
The reported improvements—1.7-point score gain, 27.7% cost cut, 1.2× throughput, and 1.5× latency speedup—are self-reported by the authors and measured on their internal benchmarks; the paper does not provide external validation or comparison to other cache-management methods. The figures reflect average results and a maximum throughput gain, not guaranteed performance across all workloads. Recursive policy gains add an extra 7.1% cost reduction, but this is described as “further reduces cost” without a baseline, so the absolute impact is unclear.
What we don't know yet
- Exact hardware specifications required for the reported throughput and latency numbers.
- Details of the benchmark datasets used for the score improvement.
Sources
- Paper (arXiv abstract) (primary)
- Code, GitHub
Researched and drafted with AI from the sources above; every figure is checked against them by code before it is published, and a line that fails is dropped.