Researchers from Tsinghua University and CUHK have open-sourced AKV, a serving system for KV cache engineering in LLM agents that cuts average cost by 27.7% across four agent benchmarks — while actually improving task scores by 1.7 points. The code is Apache-2.0 licensed and builds on SGLang.
AKV cache engineering for LLM agents (Credits: MarkTechPost)
The key idea is a shift in how agents handle long conversations. Instead of summarizing old history — a process that can quietly lose details from the middle of a run — AKV lets agents drop selected KV cache entries and reposition the rest, reusing what remains without recomputation.
AKV: Cache Engineering vs Context Engineering
Traditional context engineering trims or summarizes the text history itself. The problem is that editing that prefix invalidates every later KV state, forcing the retained suffix to be re-prefilled. Harnesses therefore summarize late and aggressively, which risks losing important details.
Cache engineering, by contrast, leaves the text trajectory untouched. It only changes which KV entries stay visible to attention. Retained entries are reused without recomputation, and they can still carry information from dropped messages. In the paper’s cost analysis, KV cache reads made up 82.4% of total API cost — so managing the cache directly attacks the biggest cost driver.
AKV defines three cache operations: Append, the standard update for new tokens; Drop, which removes selected entries from future attention without changing positions; and Repos, which rewrites retained keys to compacted positions using a RoPE rotation while leaving values unchanged, so no hidden states are recomputed. In ablations, adding Repos raised GPT-OSS-120B’s BrowseComp-Plus pass rate from 55% to 62%.
AKV Benchmark Results: 27.7% Cheaper, Higher Scores
Tested on GPT-OSS-120B, MiniMax-M2.7 and MiroThinker-1.7-Mini across BrowseComp-Plus, GAIA-Text, SWE-bench Verified and xbench-DeepSearch-2510, AKV beat both a standard summarization baseline and a Text Engineering baseline on cost. The standout result: GPT-OSS-120B on SWE-bench Verified dropped from 20.0 to 9.7 cents per task while its score rose from 40.0 to 43.2.
The accuracy gap opens on long runs. On BrowseComp-Plus at 96K tokens and above, GPT-OSS-120B scored 37% with AKV, versus 24% for the standard baseline and 32% for Text Engineering. The main policy is a tool-response rolling drop that keeps the latest 12 tool responses, and a CacheTool even lets the model call drop_context itself.
Serving this correctly is the hard part, since identical tokens can produce different cache states under different operation histories. AKV extends SGLang’s radix tree with drop and reposition nodes, frees dropped KV only when no active request still needs it, and restores evicted states in a single prefill pass. The interface is stateless and OpenAI-compatible, so existing clients can adopt it with minimal changes.
Also Read: tabsh: The Free Open-Source Tool That Puts All Your AI Coding Agents on One Browser Board and Nanbeige Releases Nanbeige4.3-33B, a 33B Open Agentic Coding Model With 512K Context
Source: MarkTechPost — read the full story there.