文档(金鹏): 2026-08-06 章节 48 篇文章摘要归档
- 46 篇原文+摘要双文件归档（按 来源/作者 分层，复用本地归档 20 篇+新抓取 26 篇） - 即梦生成 9 组主题配图（大图+列表缩略图）存入 知识/金鹏/20260806/ - 章节重组为 9 个主题分组并挂接摘要引用
This commit is contained in:
@@ -0,0 +1,143 @@
|
||||
# Disaggregated Prefilling (experimental) - vLLM
|
||||
|
||||
> **来源**：vLLM 官方文档
|
||||
> **作者**：未知
|
||||
> **发布日期**：2026-07-29
|
||||
> **原文链接**：https://docs.vllm.ai/en/latest/features/disagg_prefill/
|
||||
|
||||
---
|
||||
|
||||
# Disaggregated Prefilling (experimental)
|
||||
|
||||
This page introduces you to the disaggregated prefilling feature in vLLM.
|
||||
|
||||
Note
|
||||
|
||||
This feature is experimental and subject to change.
|
||||
|
||||
## Why disaggregated prefilling?
|
||||
|
||||
Two main reasons:
|
||||
|
||||
- __Tuning time-to-first-token (TTFT) and inter-token-latency (ITL) separately__. Disaggregated prefilling put prefill and decode phase of LLM inference inside different vLLM instances. This gives you the flexibility to assign different parallel strategies (e.g. `tp` and `pp`) to tune TTFT without affecting ITL, or to tune ITL without affecting TTFT.
|
||||
- __Controlling tail ITL__. Without disaggregated prefilling, vLLM may insert some prefill jobs during the decoding of one request. This results in higher tail latency. Disaggregated prefilling helps you solve this issue and control tail ITL. Chunked prefill with a proper chunk size also can achieve the same goal, but in practice it's hard to figure out the correct chunk size value. So disaggregated prefilling is a much more reliable way to control tail ITL.
|
||||
|
||||
Note
|
||||
|
||||
Disaggregated prefill DOES NOT improve throughput.
|
||||
|
||||
## Usage example
|
||||
|
||||
Now supports 9 types of connectors:
|
||||
|
||||
- __ExampleConnector__: refer to examples/disaggregated/example_connector/run.sh for the example usage of ExampleConnector disaggregated prefilling.
|
||||
- __LMCacheConnectorV1__: refer to examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh for the example usage of LMCacheConnectorV1 disaggregated prefilling which uses NIXL as the underlying KV transmission. LMCache also offers a multi-process (MP) mode via `LMCacheMPConnector`, where a standalone `lmcache server` holds the KV cache shared by one or more vLLM instances; see the LMCache examples and the LMCache docs for setup.
|
||||
- __NixlConnector__: refer to tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh for the example usage of NixlConnector disaggregated prefilling which support fully async send/recv. For detailed usage guide, see NixlConnector Usage Guide. For feature compatibility details, see NixlConnector Compatibility Matrix. You may specify one or multiple NIXL transfer backends, such as:
|
||||
|
||||
```
|
||||
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both", "kv_buffer_device":"cuda", "kv_connector_extra_config":{"backends":["UCX", "GDS"]}}'
|
||||
```
|
||||
|
||||
- __MooncakeConnector__: refer to examples/disaggregated/mooncake_connector/run_mooncake_connector.sh for the example usage of MooncakeConnector disaggregated prefilling. For detailed usage guide, see MooncakeConnector Usage Guide.
|
||||
- __MoRIIOConnector__ (ROCm only): see MoRI-IO Usage Guide for example usage and detailed documentation.
|
||||
- __MultiConnector__: take advantage of the kv_connector_extra_config: dict[str, Any] already present in KVTransferConfig to stash all the connectors we want in an ordered list of kwargs.such as:
|
||||
|
||||
```
|
||||
--kv-transfer-config '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both"},{"kv_connector":"ExampleConnector","kv_role":"kv_both","kv_connector_extra_config":{"shared_storage_path":"local_storage"}}]}}'
|
||||
```
|
||||
|
||||
- __OffloadingConnector__: enable offloading of KV data to CPU memory, customizing the CPU block size (in tokens) and total CPU memory bytes to allocate:
|
||||
|
||||
```
|
||||
--kv-transfer-config '{"kv_connector":"OffloadingConnector","kv_role":"kv_both","kv_connector_extra_config":{"block_size": 64, "cpu_bytes_to_use": 1000000000}}'
|
||||
```
|
||||
|
||||
For multi-tier offloading (e.g., CPU + filesystem tier) and the full configuration reference, see the KV Offloading Usage Guide.
|
||||
|
||||
- __FlexKVConnectorV1__: refer to examples/disaggregated/flexkv_connector/prefix_caching_flexkv.py for the example usage of FlexKVConnectorV1. FlexKV is a distributed KV Store and multi-level cache management system for ultra-large-scale LLM inference.
|
||||
|
||||
```
|
||||
--kv-transfer-config '{"kv_connector":"FlexKVConnectorV1","kv_role":"kv_both"}'
|
||||
```
|
||||
|
||||
## Reusing prefill token ids on decode
|
||||
|
||||
Note
|
||||
|
||||
This applies to disaggregated prefill and decode serving on the `/v1/chat/completions` endpoint, using a KV connector configured as in the Usage example above. It is experimental and subject to change.
|
||||
|
||||
In disaggregated serving, the prefill and decode stages both render the chat prompt from `messages` and tokenize it. Because the prefill stage has already produced the token ids, the decode stage can reuse them and skip its own templating and tokenization. The output is otherwise identical to a normal chat completion: it is detokenized to text, and tool and reasoning parsing, streaming, and structured output constraints all still apply.
|
||||
|
||||
The token ids are passed to the decode stage through `kv_transfer_params`, the dict already attached to the decode request to coordinate the transfer:
|
||||
|
||||
1. Send the prefill request with `return_token_ids` enabled, and read `prompt_token_ids` from the response.
|
||||
2. Set `kv_transfer_params["prompt_token_ids"]` to those ids on the decode request. `messages` is still required, but its content is not tokenized when the ids are present.
|
||||
|
||||
```
|
||||
prefill = client.chat.completions.create(
|
||||
model=model,
|
||||
messages=messages,
|
||||
extra_body={"return_token_ids": True, "kv_transfer_params": {"do_remote_decode": True}},
|
||||
)
|
||||
ids = prefill.prompt_token_ids
|
||||
|
||||
decode = client.chat.completions.create(
|
||||
model=model,
|
||||
messages=messages,
|
||||
stream=True,
|
||||
extra_body={"kv_transfer_params": {"do_remote_prefill": True, "prompt_token_ids": ids}},
|
||||
)
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
We implement disaggregated prefilling by running 2 vLLM instances. One for prefill (we call it prefill instance) and one for decode (we call it decode instance), and then use a connector to transfer the prefill KV caches and results from prefill instance to decode instance.
|
||||
|
||||
All disaggregated prefilling implementation is under `vllm/distributed/kv_transfer`.
|
||||
|
||||
Key abstractions for disaggregated prefilling:
|
||||
|
||||
- __Connector__: Connector allows __kv consumer__ to retrieve the KV caches of a batch of request from __kv producer__.
|
||||
- __LookupBuffer__: LookupBuffer provides two API: `insert` KV cache and `drop_select` KV cache. The semantics of `insert` and `drop_select` are similar to SQL, where `insert` inserts a KV cache into the buffer, and `drop_select` returns the KV cache that matches the given condition and drop it from the buffer.
|
||||
- __Pipe__: A single-direction FIFO pipe for tensor transmission. It supports `send_tensor` and `recv_tensor`.
|
||||
|
||||
Note
|
||||
|
||||
`insert` is non-blocking operation but `drop_select` is blocking operation.
|
||||
|
||||
Here is a figure illustrating how the above 3 abstractions are organized:
|
||||
|
||||
|
||||
|
||||
The workflow of disaggregated prefilling is as follows:
|
||||
|
||||
|
||||
|
||||
The `buffer` corresponds to `insert` API in LookupBuffer, and the `drop_select` corresponds to `drop_select` API in LookupBuffer.
|
||||
|
||||
Now every process in vLLM will have a corresponding connector. Specifically, we have:
|
||||
|
||||
- Scheduler connector: the connector that locates in the same process as the scheduler process. It schedules the KV cache transfer ops.
|
||||
- Worker connectors: the connectors that locate in the worker processes. They execute KV cache transfer ops.
|
||||
|
||||
Here is a figure illustrating how the above 2 connectors are organized:
|
||||
|
||||
|
||||
|
||||
The figure below shows how the worker connector works with the attention module to achieve layer-by-layer KV cache store and load:
|
||||
|
||||
|
||||
|
||||
## Third-party contributions
|
||||
|
||||
Disaggregated prefilling is highly related to infrastructure, so vLLM relies on third-party connectors for production-level disaggregated prefilling (and vLLM team will actively review and merge new PRs for third-party connectors).
|
||||
|
||||
We recommend three ways of implementations:
|
||||
|
||||
- __Fully-customized connector__: Implement your own `Connector`, and call third-party libraries to send and receive KV caches, and many many more (like editing vLLM's model input to perform customized prefilling, etc.). This approach gives you the most control, but at the risk of being incompatible with future vLLM versions.
|
||||
- __Database-like connector__: Implement your own `LookupBuffer` and support the `insert` and `drop_select` APIs just like SQL.
|
||||
- __Distributed P2P connector__: Implement your own `Pipe` and support the `send_tensor` and `recv_tensor` APIs, just like `torch.distributed`.
|
||||
|
||||
|
||||
Back to top
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
# 📊 文章摘要：Disaggregated Prefilling (experimental) - vLLM
|
||||
|
||||
> **原文**：[2026-07-29_Disaggregated_Prefilling_experimental_vLLM.md](./2026-07-29_Disaggregated_Prefilling_experimental_vLLM.md)
|
||||
> **原文链接**：https://docs.vllm.ai/en/latest/features/disagg_prefill/
|
||||
> **来源**：vLLM 官方文档
|
||||
> **作者**：未知
|
||||
> **发布日期**：2026-07-29
|
||||
> **摘要日期**：2026-08-06
|
||||
> **价值评级**：⭐⭐ 中
|
||||
|
||||
---
|
||||
|
||||
## 核心命题
|
||||
|
||||
> **预填充与解码分离** — 将 prefill 与 decode 阶段部署在不同 vLLM 实例、通过 KV 传输连接器协同，以独立调优 TTFT/ITL 并控制尾部延迟
|
||||
|
||||
---
|
||||
|
||||
## 文章概要
|
||||
|
||||
本文是 vLLM 官方文档，介绍实验性的分离式预填充（disaggregated prefilling）特性：将 LLM 推理的 prefill（预填充）与 decode（解码）阶段分别部署在两个 vLLM 实例中，通过连接器（Connector）把 KV cache 从 prefill 实例传输到 decode 实例，从而实现两个目标的独立调优——TTFT（首 token 延迟）与 ITL（token 间延迟），以及控制尾部 ITL（避免 decode 期间插入 prefill 任务导致尾延迟升高）。文档明确声明该特性不提升吞吐，并给出 9 种传输连接器（Nixl/Mooncake/FlexKV/LMCache/Offloading 等）、三大核心抽象（Connector/LookupBuffer/Pipe）与三方实现路径。其价值在于这是分离式推理架构的权威落地参考；局限是纯架构文档，无性能基准数据，生产级实现依赖第三方连接器。
|
||||
|
||||
---
|
||||
|
||||
## 关键要点
|
||||
|
||||
1. **分离的两个动机** — 一是独立调优 TTFT 与 ITL（可为 prefill 与 decode 实例配置不同并行策略如 tp/pp 而不互相影响）；二是控制尾部 ITL（chunked prefill 也能达到但 chunk size 难以调准，分离式更可靠）`[分类: 共识]`
|
||||
2. **明确边界：不提升吞吐** — 文档以警示形式声明 "Disaggregated prefill DOES NOT improve throughput"，避免用户误用；其收益在延迟控制与资源分配灵活性 `[分类: 范式突破]`（反直觉的官方边界声明，值得注意）
|
||||
3. **9 种 KV 传输连接器** — 覆盖多条技术路线：NixlConnector（异步 send/recv，UCX/GDS 后端）、MooncakeConnector、MoRIIOConnector（ROCm）、FlexKV（分布式 KV 存储）、LMCacheConnectorV1（含 MP 模式）、OffloadingConnector（KV 卸载至 CPU）、MultiConnector（组合）等 `[分类: 共识]`（生态处于快速演进中）
|
||||
4. **三大核心抽象** — Connector（kv producer/consumer 间的 KV 检索）、LookupBuffer（SQL 式 `insert`/`drop_select` 语义）、Pipe（单向 FIFO 张量管道 `send_tensor`/`recv_tensor`）；`insert` 非阻塞、`drop_select` 阻塞 `[分类: 范式突破]`（可复用的架构设计）
|
||||
5. **decode 阶段复用 prefill token ids** — 经 `kv_transfer_params` 传递 `prompt_token_ids`，decode 实例跳过模板化与 tokenization；输出与普通对话一致，工具解析/流式/结构化输出约束仍生效 `[分类: 共识]`
|
||||
6. **三种第三方实现路径** — 完全定制 Connector（控制力最强但可能与未来版本不兼容）/ 数据库式 LookupBuffer / 分布式 P2P Pipe；vLLM 团队声明生产级实现依赖第三方贡献并会积极合入 PR `[分类: 未探索]`（生态治理模式本身是开放性话题）
|
||||
|
||||
---
|
||||
|
||||
## 批判性分析
|
||||
|
||||
### 假设前提
|
||||
|
||||
文档默认读者接受以下前提：部署两套实例的额外基础设施与运维成本可接受；KV 跨机传输的带宽/延迟开销低于分离带来的收益；用户场景存在 TTFT 与 ITL 独立调优或尾部延迟控制的真实需求（如长上下文、交互式服务）。对单机或小规模部署，这些前提通常不成立。
|
||||
|
||||
### 论据与逻辑
|
||||
|
||||
作为工程文档，其结论（能控制尾部 ITL、不提升吞吐）来自架构层面的定性论证而非基准数据——全文没有任何性能对比数字，文档亦未声称做了 benchmark。"chunked prefill 的 chunk size 在实践中难以调准"这一论据是经验性陈述，缺乏数据支撑，但符合社区普遍观察。论点-实现（抽象设计）之间的对应关系清楚，逻辑链完整。
|
||||
|
||||
### 边界与局限
|
||||
|
||||
特性标注为 experimental and subject to change，接口可能变动；仅适用于配置了对应 KV 连接器的 `/v1/chat/completions` 端点（token 复用部分）；生产级连接器依赖第三方（vLLM 团队不提供内置生产实现）；对吞吐敏感而非延迟敏感的场景无收益，对部署复杂度敏感的小团队场景收益为负。
|
||||
|
||||
---
|
||||
|
||||
## 可引用金句
|
||||
|
||||
> "Disaggregated prefill DOES NOT improve throughput."
|
||||
|
||||
> "Chunked prefill with a proper chunk size also can achieve the same goal, but in practice it's hard to figure out the correct chunk size value. So disaggregated prefilling is a much more reliable way to control tail ITL."
|
||||
|
||||
---
|
||||
|
||||
## 总体评价
|
||||
|
||||
**亮点**：
|
||||
- 边界意识突出：开篇标注 experimental，并专门以警示形式澄清"不提升吞吐"，避免工程误用
|
||||
- 三大抽象（Connector/LookupBuffer/Pipe）简洁清晰，SQL 类比与 torch.distributed 类比降低了理解门槛
|
||||
- 给出 9 种连接器与三种实现路径，为选型和自研提供明确地图
|
||||
|
||||
**不足**：
|
||||
- 无任何性能基准或对比实验数据，收益大小需读者自行验证
|
||||
- 中文社区读者需注意：文档为英文，示例以命令行配置为主，缺乏端到端的部署架构图（仅有内部组件图）
|
||||
- 生态碎片化明显（9 种连接器并存），缺乏选型决策树
|
||||
|
||||
**适用场景**：大模型推理服务工程师、SRE 与架构师——尤其部署长上下文或交互式 LLM 服务、受尾部延迟困扰、或已在规划 prefill/decode 分离架构的团队；也适合需要为推理集群做资源隔离（prefill 与 decode 按需扩缩）的平台团队。
|
||||
|
||||
**关联建议**：结合 LMCache 博客（LMCacheConnector 的 MP 模式与 NVIDIA Dynamo 集成）阅读，了解 KV 缓存层在分离式推理中的生态位置；对照 Moonshot 的 Mooncake 项目（KVCache-centric 架构）理解分离式推理在超大集群中的完整形态；实践层面先跑通 examples/disaggregated/ 下的示例脚本再决定选型。
|
||||
|
||||
---
|
||||
|
||||
## 配图
|
||||
|
||||
|
||||
Reference in New Issue
Block a user