vLLM Cuts Multi-Turn Agent Latency with Bidirectional KV Cache Transfer

The Latency Challenge in Disaggregated Inference

Large Language Model (LLM) inference has become a critical bottleneck for agentic applications, where accumulated latency between turns directly impacts user experience. In disaggregated architectures, the prefill and decode phases run on separate nodes to optimize resource utilization, but this separation introduces a critical dependency: the transfer of the Key-Value (KV) cache between these nodes.

Traditionally, KV cache management was limited to the local memory of each instance. However, in multi-turn workloads, conversation context grows exponentially. Recalculating this context in every turn generates unnecessary computational overhead. vLLM's evolution toward a bidirectional transfer model aims to solve this by allowing prefill and decode nodes to share and reuse previously computed KV state, minimizing idle time and recomputation.

Push-Based and Bidirectional Transfer Mechanism

The core of this optimization lies in the use of a push-based connector based on NIXL (NVIDIA Inference Xfer Library). Unlike pull-based models, where the decoding node actively requests data, the push-based model allows the prefill node to proactively send the KV cache to the decoding node as soon as it is available. This eliminates the waiting latency associated with request and transfer negotiation.

The bidirectional nature of this transfer is crucial for multi-turn agents. Not only is the cache transferred from prefill to decode to initiate generation, but the update and return of intermediate states are also permitted. This means that at the end of a turn, the resulting KV cache can be stored and reused in the next turn, preventing the system from treating each interaction as an independent request from scratch.

Impact on Time To First Token (TTFT)

Time To First Token (TTFT) is the most sensitive metric for perceived speed in conversational interfaces. By moving transfer coordination off the critical path via the NIXL connector, vLLM drastically reduces the time between request reception and the generation of the first token.

In agentic workloads, where input context is extensive and partially reused, TTFT reduction is proportional to the amount of context reused. Bidirectional transfer allows the system to identify which parts of the cache already exist on the destination node and only transfer or recalculate what is strictly necessary, optimizing network bandwidth and compute latency.

Results on AWS EC2 P5en Instances

Benchmarks conducted on Amazon EC2 P5en instances, equipped with AWS EFA (Elastic Fabric Adapter) interconnect, demonstrate the efficacy of this architecture. The high-speed EFA network is a fundamental requirement to ensure KV cache transfer does not become a new bottleneck, allowing data movement to be nearly as fast as local memory access.

In tests with the Qwen3-32B model, a TTFT reduction of up to 1.6x was observed compared to standard disaggregated inference. For the GLM-5-FP8 model, the improvement was even more pronounced, reaching a 3.3x reduction in TTFT. These figures confirm that context reuse is more beneficial for models with long contexts and workloads with high prefix reuse.

Implications for AI Architects

For teams deploying AI agents in the cloud, this optimization has a dual impact: cost reduction and improved user experience. By reusing the KV cache, the number of compute operations (FLOPs) required per turn is reduced, translating to lower GPU consumption per request. This allows serving more simultaneous users with the same infrastructure or reducing the GPU fleet size for a given workload.

Furthermore, the latency improvement is critical for meeting Service Level Objectives (SLOs) in interactive applications. vLLM's architecture, combined with high-speed network infrastructure, positions disaggregated inference not just as a throughput optimization technique, but as a viable strategy for low-latency, high-interactivity applications such as code copilots and complex conversational assistants.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.