文档(金鹏): 2026-08-06 章节 48 篇文章摘要归档
- 46 篇原文+摘要双文件归档（按 来源/作者 分层，复用本地归档 20 篇+新抓取 26 篇） - 即梦生成 9 组主题配图（大图+列表缩略图）存入 知识/金鹏/20260806/ - 章节重组为 9 个主题分组并挂接摘要引用
This commit is contained in:
@@ -0,0 +1,263 @@
|
||||
# TetriInfer: Inference without Interference - Disaggregate LLM Inference for Mixed Downstream Workloads
|
||||
|
||||
> **来源**：arXiv
|
||||
> **作者**：Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, Yizhou Shan（等，共 12 位）
|
||||
> **发布日期**：2024-01-20
|
||||
> **原文链接**：https://arxiv.org/abs/2401.11181
|
||||
|
||||
---
|
||||
|
||||
## 论文元数据
|
||||
|
||||
- **arXiv ID**：2401.11181
|
||||
- **学科分类**：Distributed, Parallel, and Cluster Computing (cs.DC)
|
||||
- **作者机构**：中国科学院大学、中科院计算所（ICT, CAS）、华为云（Huawei Cloud）
|
||||
- **提交历史**：v1: 2024-01-20
|
||||
- **DOI**：https://doi.org/10.48550/arXiv.2401.11181
|
||||
|
||||
---
|
||||
|
||||
Cunchen Hu1,2111Work done while intern at Huawei Cloud.,
|
||||
Heyang Huang1,2,
|
||||
Liangliang Xu3,
|
||||
Xusheng Chen3,
|
||||
Jiang Xu3,
|
||||
Shuang Chen3,
|
||||
|
||||
Hao Feng3,
|
||||
Chenxi Wang1,2,
|
||||
Sa Wang1,2,
|
||||
Yungang Bao1,2,
|
||||
Ninghui Sun1,2,
|
||||
Yizhou Shan3
|
||||
|
||||
1University of Chinese Academy of Sciences, 2ICT, CAS
|
||||
3Huawei Cloud
|
||||
|
||||
## 摘要（Abstract）
|
||||
|
||||
Transformer-based large language model (LLM) inference serving is now the backbone of many cloud services.
|
||||
LLM inference consists of a prefill phase and a decode phase.
|
||||
However, existing LLM deployment practices often overlook the distinct characteristics of these phases, leading to significant interference.
|
||||
To mitigate interference, our insight is to carefully schedule and group inference requests based on their characteristics. We realize this idea in TetriInfer through three pillars. First, it partitions prompts into fixed-size chunks so that the accelerator always runs close to its computation-saturated limit. Second, it disaggregates prefill and decode instances so each can run independently. Finally, it uses a smart two-level scheduling algorithm augmented with predicted resource usage to avoid decode scheduling hotspots.
|
||||
Results show that TetriInfer improves time-to-first-token (TTFT), job completion time (JCT), and inference efficiency in turns of performance per dollar by a large margin, e.g., it uses 38% less resources all the while lowering average TTFT and average JCT by 97% and 47%, respectively.
|
||||
|
||||
## 1 引言（Introduction）
|
||||
|
||||
Since the boom of ChatGPT, large language model (LLM) based services have now played a vital role in our daily lives[4, 38, 20, 9, 34, 31].
|
||||
Behind the scenes, all use cases boil down to LLM inference serving. To run an inference request, the LLM model will first take the user inputs to generate the first token (known as the prefill phase), and then generate outputs token-by-token in an auto-regressive manner (known as the decode phase).
|
||||
Numerous works were proposed to improve the cost efficiency of LLM inference [21, 41].
|
||||
|
||||
There are various ways to interact with LLM, from simple chats to more complex downstream tasks such as document summarization, content creation, etc.
|
||||
As a result, LLM-empowered services serve inference requests with dramatically different properties that can be categorized across two dimensions: the input prompt length during the prefill phase and the generated token length during the decode phase.
|
||||
As shown in Figure 1, summarization tasks have long input prompts and short generated tokens, while context creation tasks are the opposite.
|
||||
Token lengths of different downstream tasks can differ by more than two orders of magnitude.
|
||||
Given the significant variation in LLM inference requests from various downstream tasks, the first research question we ask in this paper is how do these inference requests perform when running together?.
|
||||
|
||||
To answer this question, we run extensive tests that mix LLM prefill and decode requests of different lengths.
|
||||
Unfortunately, we have observed serious interference across all combinations. For example, mixing prefill requests could result in a 10x slowdown, combining prefill and decode requests could lead to a 5x slowdown, and mixing decode requests with different lengths could take a 16% throughput hit (see §2.2).
|
||||
A naive solution to avoid interference is to provision resources for each downstream task statically. Given the high cost of LLM serving infrastructure, this solution is impractical.
|
||||
To this end, the second research question we ask in this paper is how to build a distributed LLM inference serving system that minimizes interferences?
|
||||
|
||||
We take a step back to examine why interference exists. We find the fundamental issue lies in the fact that current LLM deployment practices do not account for the distinct characteristics exhibited by LLM prefill and decode phases.
|
||||
Specifically, the prefill phase resembles a computation-heavy batch job, with its computation scaling quadratically with the input prompt length.
|
||||
The decode phase resembles a memory-intensive, latency-critical task, with its resource usage scaling sublinearly with the generated token length [33].
|
||||
Interferences observed in our tests are classic system problems.
|
||||
Running prefill requests leads to a serious slowdown because we continue adding computation-heavy jobs to an already saturated hardware (§2.2.1).
|
||||
Combining prefill and decode requests hurts both because we co-run batch and latency-critical jobs simultaneously (§2.2.2).
|
||||
Mixing decode requests leads to a throughput drop because we are unaware of the memory bandwidth and capacity usage, thus leading to contention and head-of-line blocking (§2.2.3).
|
||||
|
||||
To solve these issues, our insight is to carefully schedule and group requests based on their characteristics.
|
||||
We realize this idea in TetriInfer222The name of our system, TetriInfer, implies that it can efficiently organize LLM inference requests, similar to how tetris blocks are stacked., a cloud-scale LLM inference
|
||||
serving system designed to battle interferences.
|
||||
|
||||
Our designs are three-fold.
|
||||
First, to avoid interference running prefill, we propose limiting the number of tokens processed in a single prefill iteration so that hardware is fully utilized without incurring extra penalties. TetriInfer partitions and pads input prompts into fixed-size chunks so that the accelerator always runs close to its computation-saturated limit (§3.3).
|
||||
Second, to avoid interference in co-running prefill and decode, we propose disaggregating prefill from decode phases.
|
||||
TetriInfer has dedicated prefill and decode instances.
|
||||
During runtime, prefill instances transfer prefilled KV cache to decode instances.
|
||||
The prefill and decode instances are virtual concepts in that
|
||||
each can scale independently and flip roles if load changes (§3.5).
|
||||
Third, to avoid interference running decode requests, we propose using a smart two-level scheduling algorithm augmented with predicted resource usage to avoid scheduling hotspots (§3.4). TetriInfer incorporates an LLM-based length prediction model to speculate the number of generated tokens of decode requests, and then schedule them accordingly.
|
||||
|
||||
We implement TetriInfer’s disaggregated prefill and decode instances based on vLLM [21]. Most of our modules are implemented in Python, except for the network stack module, which utilizes C++ to interface with low-level APIs for KV cache transfer. The fine-tuning part uses Trainer APIs offered by HuggingFace Transformer [16]. Since we cannot access high-end hardware, we implement a mock mechanism to emulate varying network bandwidth connecting prefill and decode instances, as illustrated in Figure 9.
|
||||
|
||||
We compare TetriInfer with vanilla vLLM using public dataset [35] in terms of time-to-first-token (TTFT), job completion time (JCT), and efficiency as in performance per dollar (perf/$).
|
||||
We run them atop a real testbed with emulated network bandwidth ranging from 200Gbps to 300GBps.
|
||||
For light prefill and heavy decode workload, TetriInfer improves perf/$ by 2.4x (Figure 16). For common mixed workload, TetriInfer improves average TTFT and average JCT by 85% and 50%, respectively (Figure 16).
|
||||
Nevertheless, we also find that TetriInfer’s design is not ideal for heavy prefill and heavy decode workloads since the room for improvement is marginal, and the overhead we introduce cannot be offset (Figure 16).
|
||||
Overall, our ideas mentioned above are effective.
|
||||
TetriInfer achieves effective LLM inference serving, outperforming vLLM by a large margin in TTFT, JCT, and perf/$ running most common workloads (§5.1).
|
||||
|
||||
|
||||
|
||||
Figure 1: Length Distribution. Prompt Tokens for Prefill and Generated Tokens during Decode. Data sources: conversation [35], summarization [17], writing [18].
|
||||
|
||||
## 2 Background and Motivation
|
||||
|
||||
We present a brief primer on LLM inference and study interferences while running various LLM inference requests to motivate our work. For model and testbed details, see §5.
|
||||
|
||||
### 2.1 Generative LLM Inference
|
||||
|
||||
|
||||
|
||||
Figure 2: Prefill and Decode’s Characteristics. Decode’s GPU utilization fluctuates because the task is faster than our monitoring granularity.
|
||||
|
||||
LLM inference is a process that involves generating a sequence of output tokens in response to an input prompt. This process consists of two phases: prefill and decode.
|
||||
The prefill phase outputs the first token and generates the key and value cache (KV cache) for future decoding [21]. The decode phase uses the previous KV cache to generate new tokens step-by-step in an auto-regressive manner.
|
||||
Generally, the prefill phase is computation-bound, and the decode phase is memory-bound [33].
|
||||
We report this in Figure 2.
|
||||
Results indicate that the prefill phase’s throughput stays flat once the accelerator is saturated at a certain number of tokens (which we name the accelerator-saturate threshold).
|
||||
The decode phase’s throughput continues increasing with a larger batch size but plateaus once the memory bandwidth is saturated.
|
||||
|
||||
### 2.2 Motivation: Interference Study
|
||||
|
||||
This section studies the impact of running different inference requests concurrently.
|
||||
Inspired by Figure 1,
|
||||
we classify inference requests across two dimensions (prefill and decode length) and one property (light or heavy), resulting in four distinct request types:
|
||||
heavy prefill,
|
||||
light prefill,
|
||||
heavy decode, and
|
||||
light decode.
|
||||
Here, heavy refers to a long token length, while light refers to a short token length.
|
||||
Below, we study mixing prefill
|
||||
|
||||
ong decoding tasks across different on-demand decode instances. Each timeline comprises four rounds (R1 to R4), with the length of prefill and decode boxes representing their sequence length and the width of the decode box indicating its resource usage. A wider decode box indicates the presence of lengthy generated tokens, resulting in larger resource usage and decoding latency.
|
||||
(b) shows TetriInfer’s architecture with four core modules highlighted.
|
||||
|
||||
## 3 设计（Design）
|
||||
|
||||
### 3.1 Overview
|
||||
|
||||
We realize the above insights in TetriInfer, an LLM inference serving system designed to battle interferences.
|
||||
First, we run prefill in a fixed-size computation unit by partition and pad input prompts into fixed-size chunks such that the accelerator always runs close to its computation-saturated limit (§3.3).
|
||||
Second, we design instances dedicated to running the prefill or decode phases. We schedule prefill requests to prefill instances only, and the same goes for decode requests. Prefill instances will transfer prefilled KV cache to decode instances.
|
||||
Our prefill and decode instances are virtual concepts in that each can scale independently and flip roles if load changes (§3.5).
|
||||
Finally, we design a two-level scheduling algorithm for both prefill and decode request scheduling. We incorporate a length-prediction model to speculate decode requests’ resource usage and then schedule them accordingly (§3.4).
|
||||
|
||||
We show TetriInfer’s architecture in Figure 6 (b) with four modules highlighted: centralized control plane, prefill instance, decode instance, and length prediction model.
|
||||
|
||||
Centralized control plane.
|
||||
It consists of a global scheduler and a cluster monitor.
|
||||
The global scheduler sends requests to prefill instances based on load and receives streaming outputs from decode instances.
|
||||
The cluster monitor collects statistics from prefill and decode instances and regularly broadcasts load information to prefill instances. It adds, removes, and flips prefill or decodes instances.
|
||||
|
||||
Prefill Instances.
|
||||
They only run the prefill phase of an LLM inference request.
|
||||
Each prefill instance has a local scheduler, a length predictor, the main LLM engine, and a dispatcher.
|
||||
All requests undergo four steps.
|
||||
First, the local prefill scheduler sorts requests based on pre-defined policies.
|
||||
Second, the length predictor runs a prediction model to speculate the requests’ decode lengths, which are then used to estimate resource usage during the decoding phase.
|
||||
Third, the main LLM engine partitions all requests into fixed chunks.
|
||||
Finally, for each request, the dispatcher runs an inter-decode load-balancing algorithm to select a decode instance and then forwards the generated KV cache to it.
|
||||
|
||||
Decode instances.
|
||||
They are virtually disaggregated from prefill instances and only run the decode phase of an LLM inference request.
|
||||
Each decode instance can receive requests from any prefill instance.
|
||||
It runs a local scheduler with three pre-defined policies for selecting decode requests to run in the main LLM engine.
|
||||
|
||||
Length Prediction Model.
|
||||
The prediction model is a small LLM model fine-tuned offline for predicting the generation length of LLM inference requests. TetriInfer’s prefill dispatcher and decode instance’s local scheduler utilize the speculated information to schedule decode instances and avoid hotspots measured in §2.2.3. The prediction model is small and deployed at all prefill instances.
|
||||
|
||||
### 3.2 Control Plane
|
||||
|
||||
TetriInfer has a centralized control plane to
|
||||
manage inference clusters at the cloud scale.
|
||||
It consists of a cluster monitor that manages the lifecycle of prefill and decode instances and a global scheduler that managesthe lifecycle of inference requests.
|
||||
The centralized control plane is a distributed system without a single point of failure or processing bottlenecks.
|
||||
|
||||
The cluster monitor is responsible for collecting and broadcasting statistics and scaling instances. Both prefill and decode instances regularly send their load information to the cluster monitor (e.g., every 100 ms). Since we run decentralized decode request scheduling at prefill instances, the cluster monitor will aggregate decode instances’ load information and broadcast it to all prefill instances.
|
||||
|
||||
The global scheduler is responsible for forwarding inference requests from external services to prefill instances and sending inference outputs from decode instances back to external services in a streaming fashion.
|
||||
The global scheduler maintains a request status table, which stores requests’ arrival time, current phase (e.g., prefill or decode), SLA requirement, etc.
|
||||
When a request arrives, the global scheduler will choose a prefill instance with the least load and then insert the request into the table.
|
||||
Following our insight to disaggregate prefill and decode instances, the global scheduler only decides which prefill instance will handle the request. It is up to the prefill instance’s dispatcher to decide which decode instances to use with a speculated resource usage.
|
||||
|
||||
### 3.3 Prefill Instance
|
||||
|
||||
The prefill instance runs the prefill phase of an inference request.
|
||||
To avoid interference among prefill requests, we use a prefill scheduler and chunked prefill to sort and partition all prompts into fixed-size chunks.
|
||||
To help avoid interference during the decode phase, we run a length predictor and a decentralized dispatcher to choose decode instances based on speculated resource usage.
|
||||
|
||||
#### 3.3.1 Prefill Scheduler
|
||||
|
||||
The prefill instance’s scheduler is crucial for improving the prefill phase’s latency and throughput.
|
||||
The scheduler maintains a raw request queue that stores requests from the global scheduler and a scheduled queue that stores sorted requests.
|
||||
In this work, we have designed and implemented three scheduler policies: first-come-first-serve (FCFS), shortest-job-first (SJF), and longest-job-first (LJF).
|
||||
We can use the latter two policies because we can accurately estimate a request’s prefill time based on the number of tokens in its prompt.
|
||||
We only explore non-preemptive policies, though chunked prefill (described soon) has opened the door to preemptive and out-of-order prefill scheduling, such as shortest-remaining-time-first, which we leave for future work.
|
||||
|
||||
The scheduled requests are sent to the length predictor which executes scheduled requests as-is using fixed-size batch (§3.3.2), and the main LLM which uses chunked prefill (§3.3.3).
|
||||
In Figure 7, we illustrate the above three scheduler policies and how scheduled requests are partitioned and merged into fixed-size chunks.
|
||||
Specifically, FCFS keeps the original request arrival order.
|
||||
Prompt tokens are partitioned and merged into chunks sequentially.
|
||||
This policy is the easiest to implement and works best for inference requests with similar prompt lengths.
|
||||
However, FCFS can lead to head-of-line blocking and high average job completion time (JCT) when requests have long prompts. This is problematic since the length differences among LLM inference requests are more than three orders of magnitude (see Figure 1).
|
||||
|
||||
In response, we add the shortest-job-first
|
||||
(SJF), and longest-job-first (LJF) to overcome these issues.
|
||||
These two policies schedule prefill requests based on prompt token lengths in ascending or descending order. By design, they can achieve lower JCT compared to FCFS. Nevertheless, they are no panacea. They introduce starvation for either long or short requests. To avoid starvation, we propose using a prefill scheduling batch (i.e., PrefillSchedBatch) variable to control how many inference requests can be scheduled at a time. For example, assume the raw request queue has twenty requests awaiting scheduling. If we set the batch size to ten, we will schedule twice, each with ten requests sorted and put into the scheduled queue. This simple mechanism prevents starvation during the prefill phase.
|
||||
|
||||
Our scheduler is effective. Results in Figure 16 show that SJF lowers average prefill waiting time by 7.8% compared to FCFS when the batch size is set to 16. Additionaly, the improvement is even more pronounced with larger batch sizes.
|
||||
|
||||
|
||||
|
||||
Figure 7: Prefill Scheduler Policies. The left shows four raw inference requests (R1 to R4). The right shows scheduled requests using FCFS, SJF, and LJF. We show the chunked version to illustrate slicing and merging (C1 to C4).
|
||||
|
||||
#### 3.3.2 Length Predictor
|
||||
|
||||
To address the interference cases measured in §2.2.3, it is essential to determine the number of tokens that a decode request is likely to generate. This information will enable us to schedule decode requests in a length-aware manner.
|
||||
As such, the prefill instance runs a length predictor to predict the length range of an inference request’s generated tokens.
|
||||
The prefill instance’s dispatcher utilizes this information for inter-decode instance scheduling (§3.3.4), while the decoding instance’s local scheduler employs this information for intra-decode instance scheduling (§3.4).
|
||||
|
||||
Our length predictor uses a small LLM-based classification model called a "predict model" to classify the length of generated tokens into fixed-size buckets if the request were executed by a specific target LLM model.
|
||||
The predict model is intentionally small, containing millions of parameters while the target model is much larger, with billions of parameters. As we run the length predictor at the prefill instance, we aim to minimize its cost and avoid impacting the main LLM model. Therefore, approaches like using a giant LLM to predict length are not feasible for us [48].
|
||||
Fortunately, a small LLM model is much faster than a giant LLM and uses much less resources.
|
||||
For example, we use OPT-125M as the predict model and OPT-13B as the target model, the small one is roughly ten times faster than the larger one.
|
||||
|
||||
We opt to predict the length range instead of an exact number of tokens because the latter is extremely difficult to predict.
|
||||
Various inference parameters, such as temperature and top-p [3], result in significant response variations from the same LLM model to the same question in practice. Since our primary goal is to use the estimated length to guide our request scheduling decisions, an exact length estimation is unnecessary; a length range suffices.
|
||||
For instance, if we estimate the length to be between ten to twenty tokens, we can deduce its resource usage’s lower and upper bounds.
|
||||
|
||||
In this work, we have tested two execution modes: a sequential mode, where we first execute the predict model followed by the target model, and a parallel mode, where both models are run simultaneously.
|
||||
The sequential mode adds extra latency for the target LLM model, while the parallel mode may reduce the target LLM model’s throughput.
|
||||
Based on our findings in Figure 17, we opted to use the parallel mode because the main LLM is not affected for most requests (more than 80%), though throughput take a 10% hit under extreme stress test.
|
||||
|
||||
Figure 8 outlines the offline fine-tuning and online prediction workflow. In this process, the predict model (depicted in red) is trained to speculate the decoding behavior of a specific target model (depicted in blue).
|
||||
The fine-tuning of the predict model involves three key steps.
|
||||
Firstly, we assemble a prompt-only training dataset inherited from public datasets, a large target LLM model (e.g., OPT-13B), and a classification model for our predict model (e.g., 125M OPTForSequenceClassification [16]).
|
||||
Secondly, we send training prompts to the target LLM model, which generates responses.
|
||||
Subsequently, we categorize the generated responses into fixed-size buckets with a chosen granularity.
|
||||
For instance, using a granularity of 100, responses with token lengths between 0 to 200 are labeled with 0, 200-400 are labeled with 1, and so on. These labels are paired with the training prompts to create a new dataset. Lastly, we partition the new dataset into a training section and an evaluation section and then proceed to train and evaluate the predict model using this dataset.
|
||||
|
||||
The length range granularity plays a crucial role. If set to one, we fall back to predicting an exact number of tokens, which is not practical. If set to target model’s context window size (e.g., 2K), we fall back to no prediction at all and could run into interferences reported in §2.2.1. Intuitively, a smaller granularity means more accurate resource and performance estimation but lower accuracy in practice. A larger granularity means higher accuracy but essentially makes scheduling harder. Regardless of granularity, it’s easy to calculate resource usage’s upper and lower bound but not performance.
|
||||
In this work, we can predict a granularity of 200 tokens with 74.9% accuracy.
|
||||
Since improving prediction accuracy is not the focus of this work, we leave it for future work.
|
||||
|
||||
|
||||
|
||||
Figure 8: Predict Model’s Fine-tuning and Prediction Flow. The target model is the one that we want to predict its decoding behavior. The predict model is the one we train. This work does not explore online fine-tuning.
|
||||
|
||||
Discussions.
|
||||
We run the length predictor at each prefill instance, hence prefill instances can make well-informed decisions on which decode instances should have enough resources to run certain decoding requests.
|
||||
Nevertheless, we identify two alternative designs. The first design is to run the length predictor at each decode instance. As a result, the prefill instance can only schedule requests based on the load of decoding instances. However, this design cannot avoid interference cases we measured in §2.2.3. Indeed, one could migrate interference requests among decoding instances at runtime based on predicted length. This would be an overly complex solution. The second design is to run the length predictor at the global scheduler before dispatching requests to refill instances. This design could make the global scheduler a bottleneck. We believe our current design is easier and simpler to reason about and deploy compared to alternatives.
|
||||
|
||||
#### 3.3.3 Chunked Prefill
|
||||
|
||||
After the prefill scheduler, we concurrently execute the prefill phase of the main LLM alongside the length predictor.
|
||||
We employ fixed-size chunks for the LLM prefill rather than using fixed batch sizes [21].
|
||||
|
||||
As demonstrated in §2.2.1, we observe that as the number of tokens in a prefill iteration increases, the accelerator’s throughput remains constant, while the latency continues to rise after reaching a certain threshold. We refer to this threshold as ChunkSize. Compared to the traditional fixed batch size approach, running prefill in ChunkSize allows for the optimal utilization of accelerators without incurring additional latency penalties. The accelerator and the LLM model architecture determine the ChunkSize. Models with larger hidden dimensions and accelerators with lower capabilities typically result in a smaller ChunkSize. For example, in our test environment, the value is 512 tokens for OPT 13B.
|
||||
|
||||
Figure 7 illustrates how chunked prefill wor
|
||||
|
||||
ded and two-sided, similar to RDMA’s classification.
|
||||
Accelerators like GPU or NPU can do one-sided memory access as they have low-level primitives such as direct memory copies between devices [26, 14].
|
||||
|
||||
To navigate the complicated physical data links and ensure that TetriInfer can always use the most performant link once deployed, we design a unified network transfer abstraction to utilize the different network stack options listed in Figure 9. The stack exposes APIs such as send, receive, read, write, etc. Our dispatcher calls these APIs to transmit the KV cache to remote decode instances.
|
||||
|
||||
Discussion.
|
||||
We identify two unexplored research questions.
|
||||
The first question pertains to whether it is beneficial to simultaneously utilize multiple data links for transmitting the KV cache. While this approach could enhance performance, it may also introduce complex control logic.
|
||||
The second question involves the sender accelerator accessing the memory of the receiver accelerator without involving the receiver’s CPU. This scenario raises typical challenges associated with building large-scale RDMA-based memory systems [10, 12].
|
||||
Unfortunately, we cannot explore either of these ideas in this wo
|
||||
@@ -0,0 +1,93 @@
|
||||
# 📊 文章摘要：TetriInfer: Inference without Interference - Disaggregate LLM Inference for Mixed Downstream Workloads
|
||||
|
||||
> **原文**：[2024-01-20_TetriInfer.md](./2024-01-20_TetriInfer.md)
|
||||
> **原文链接**：https://arxiv.org/abs/2401.11181
|
||||
> **来源**：arXiv
|
||||
> **作者**：Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, Yizhou Shan 等 12 位（中国科学院大学、中科院计算所、华为云）
|
||||
> **发布日期**：2024-01-20
|
||||
> **摘要日期**：2026-08-06
|
||||
> **价值评级**：⭐⭐⭐ 高
|
||||
|
||||
---
|
||||
|
||||
## 核心命题
|
||||
|
||||
> **消除干扰** — 混合下游任务请求的 prefill/decode 长度差异可达两个数量级，共跑必然相互干扰；按请求特征调度分组（定长 chunk 的 prefill + 阶段解耦 + 长度预测调度）是消除干扰、提升性能/美元比的系统化方案。
|
||||
|
||||
---
|
||||
|
||||
## 文章概要
|
||||
|
||||
TetriInfer 研究"不同下游任务（摘要、创作、对话等）的请求混跑"时的干扰问题：请求的输入 prompt 长度与生成 token 长度差异超两个数量级，混跑会引发严重性能恶化（实测：混跑 prefill 请求 10× 减速、prefill 与 decode 混跑 5× 减速、不同长度 decode 混跑吞吐损失 16%）。作者把干扰根源归结为现有部署忽略了两阶段的本质差异（prefill 是计算密集的批作业，decode 是内存密集的延迟敏感任务），提出三支柱设计：将 prompt 切分为固定大小 chunk 使加速器始终贴近计算饱和点运行；prefill/decode 实例解耦（虚拟实例，可独立扩缩容与角色翻转）；两级调度配合小 LLM 长度预测模型（200 token 粒度预测准确率 74.9%）避免 decode 调度热点。相比 vanilla vLLM：资源使用减少 38%，平均 TTFT 与 JCT 分别降低 97% 与 47%；轻 prefill 重 decode 负载下性能/美元比提升 2.4×。局限：重 prefill + 重 decode 负载下收益边际化且开销无法抵消，且评估受限于模拟网络带宽与 OPT-13B 等旧模型。
|
||||
|
||||
---
|
||||
|
||||
## 关键要点
|
||||
|
||||
1. **混合负载干扰是真实且严重的问题** — 实测数据：混跑 prefill 请求 10× 减速（向已饱和硬件持续添加计算密集作业）、prefill+decode 混跑 5× 减速（批作业与延迟敏感任务共跑）、不同长度 decode 混跑 16% 吞吐损失（内存带宽/容量竞争与队头阻塞）。`[分类: 范式突破]`
|
||||
2. **按特征分组而非统一处理** — 核心洞见：不应把所有请求当同类处理，而应按 prefill/decode 长度（重/轻两维）分类调度；系统名 TetriInfer 即喻意像俄罗斯方块一样组织请求。`[分类: 范式突破]`
|
||||
3. **定长 chunk 的 prefill** — 加速器在 token 数达到饱和阈值后吞吐不再增长、延迟却继续上升；把 prompt 切成固定大小 chunk（OPT-13B 上为 512 token）使硬件始终贴近计算饱和点，避免额外延迟惩罚。`[分类: 范式突破]`
|
||||
4. **虚拟解耦实例** — prefill/decode 实例是"虚拟概念"：各自独立扩缩容、负载变化时可角色翻转，兼顾解耦隔离与资源弹性。`[分类: 共识]`
|
||||
5. **小 LLM 预测生成长度** — 用百万参数级的预测模型（OPT-125M）对十亿级目标模型（OPT-13B）的生成长度做分桶分类预测（而非精确预测），200 token 粒度准确率 74.9%；并行执行模式下 80% 以上的请求主模型不受影响，极端压测下吞吐损失 10%。`[分类: 未探索]`
|
||||
6. **两级调度避免热点** — prefill 实例的调度器选择 decode 实例时用预测资源占用做负载均衡，decode 实例内部再用长度感知策略调度，避免 §2.2.3 测得的 decode 热点问题。`[分类: 未探索]`
|
||||
7. **量化收益** — 相比 vLLM：资源减少 38%、平均 TTFT 降低 97%、平均 JCT 降低 47%；轻 prefill 重 decode 负载 perf/$ 提升 2.4×，常见混合负载 TTFT/JCT 改善 85%/50%。`[分类: 共识]`
|
||||
8. **诚实标注不适场景** — 重 prefill + 重 decode 负载下设计不理想：改进空间边际化、引入的开销无法抵消——这是少见的对自身方案适用边界的明确声明。`[分类: 争议]`
|
||||
|
||||
---
|
||||
|
||||
## 批判性分析
|
||||
|
||||
### 假设前提
|
||||
|
||||
- 下游任务请求的输入/输出长度分布差异显著且可测量、可分类（重/轻两维），且这种分类足以指导调度。
|
||||
- 用一个小模型（百万参数）预测大模型（十亿参数）的生成长度范围是可行的（以固定粒度分桶），且并行运行小模型不显著影响主模型吞吐。
|
||||
- 干扰现象（10×/5×/16%）在目标生产环境复现，且模拟网络带宽（200Gbps-300GBps）能代表真实数据中心网络。
|
||||
- 集中式控制面（全局调度器 + 集群监控器）不会成为云规模瓶颈（论文称其为无单点的分布式系统）。
|
||||
|
||||
### 论据与逻辑
|
||||
|
||||
- 干扰测量的数据（10×/5×/16%）是本文论据的地基，直接驱动设计决策，链条清晰；三支柱设计各自针对一种干扰类型，映射关系明确。
|
||||
- 端到端收益（38% 资源、97% TTFT、47% JCT）与摘要结论一致，且明确区分了"轻 prefill 重 decode"（2.4× perf/$）与"常见混合负载"（85%/50%）等不同语境。
|
||||
- 弱点：论文基于 OPT-13B/OPT-125M 等较旧模型，未在 GPT-4 级别或 MoE 模型上验证；评估在模拟带宽与模拟环境下进行（作者明示无法访问高端硬件）；所下载原文缺失完整实验章节，部分收益数字无法在原文内交叉核对（以摘要与引言为准）。
|
||||
|
||||
### 边界与局限
|
||||
|
||||
- 明确不适用场景：重 prefill + 重 decode 负载（收益无法抵消开销）。
|
||||
- 长度预测粒度（200 token、74.9% 准确率）意味着对长度分布接近粒度边界的请求，调度决策可能偏差。
|
||||
- 依赖小模型与目标模型行为的一致性；换目标模型需重新微调预测模型。
|
||||
- 未探索在线微调预测模型、可抢占/乱序 prefill 调度（chunked prefill 已打开该可能，留作未来工作）。
|
||||
|
||||
---
|
||||
|
||||
## 可引用金句
|
||||
|
||||
> "We find the fundamental issue lies in the fact that current LLM deployment practices do not account for the distinct characteristics exhibited by LLM prefill and decode phases."
|
||||
> （我们发现根本问题在于：当前的 LLM 部署实践没有考虑 prefill 与 decode 阶段各自迥异的特性。）
|
||||
|
||||
> "We take a step back to examine why interference exists. We find the fundamental issue lies in the fact that current LLM deployment practices do not account for the distinct characteristics exhibited by LLM prefill and decode phases."
|
||||
> （我们退一步审视干扰为何存在，发现根本问题在于现有部署实践忽视了 prefill 与 decode 阶段的本质差异——prefill 像计算密集的批作业，decode 像内存密集的延迟敏感任务。）
|
||||
|
||||
---
|
||||
|
||||
## 总体评价
|
||||
|
||||
**亮点**：
|
||||
- 首个系统量化"混合下游任务干扰"的工作，10×/5×/16% 的干扰数据直观有力
|
||||
- 长度预测驱动的调度是独特贡献，用分桶分类规避了精确预测的不可能，工程务实
|
||||
- 定长 chunk prefill 与虚拟解耦实例的设计简洁可落地，且明确声明适用边界（重+重负载不适用），学术诚实度高
|
||||
- 与 DistServe、Splitwise 同期（2023.11-2024.1）独立验证了解耦思路，并补充了干扰测量与预测调度两个新维度
|
||||
|
||||
**不足**：
|
||||
- 评估环境受限（模拟带宽、旧模型 OPT-13B），无真实生产部署验证
|
||||
- 预测模型需要随目标模型重新微调，落地成本未量化
|
||||
- 下载版原文缺失实验与结论章节，部分细节只能依赖摘要与引言
|
||||
|
||||
**适用场景**：多任务混合流量（对话 + 摘要 + 创作等）的 serving 系统设计者；研究 LLM serving 干扰表征、长度预测调度的研究人员；云厂商推理平台团队。
|
||||
|
||||
**关联建议**：与 DistServe（阶段解耦的 goodput 理论）、Splitwise（异构硬件）、Mooncake（生产系统 KVCache 中心化）构成解耦 serving 四篇同期代表作，可联合精读；"输出长度预测"方向后续可关注 speculative decoding、MoE 路由预测等相关工作。
|
||||
|
||||
---
|
||||
|
||||
## 配图
|
||||
|
||||
|
||||
@@ -0,0 +1,219 @@
|
||||
# How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving
|
||||
|
||||
> **来源**：arXiv
|
||||
> **作者**：Hanjiang Wu, Abhimanyu Rajeshkumar Bambhaniya, Sarbartha Banerjee, Tuhin Khare, Sudarshan Srinivasan, Suvinay Subramanian（等，共 12 位）
|
||||
> **发布日期**：2026-05-27
|
||||
> **原文链接**：https://arxiv.org/abs/2605.28302
|
||||
|
||||
---
|
||||
|
||||
## 论文元数据
|
||||
|
||||
- **arXiv ID**：2605.28302
|
||||
- **学科分类**：Distributed, Parallel, and Cluster Computing (cs.DC)
|
||||
- **作者机构**：佐治亚理工学院（Georgia Institute of Technology）、Intel、Google、Google DeepMind、Infravana
|
||||
- **提交历史**：v1: 2026-05-27
|
||||
- **DOI**：https://doi.org/10.48550/arXiv.2605.28302
|
||||
|
||||
---
|
||||
|
||||
Hanjiang Wu1 Abhimanyu Rajeshkumar Bambhaniya1,5 Sarbartha Banerjee1
|
||||
|
||||
Tuhin Khare1 Sudarshan Srinivasan2 Suvinay Subramanian3
|
||||
|
||||
Souvik Kundu2 Madhu Kumar2 Midhilesh Elavazhagan2
|
||||
|
||||
William Won1 Amir Yazdanbakhsh4 Tushar Krishna1,5
|
||||
|
||||
1Georgia Institute of Technology 2Intel 3Google 4Google DeepMind 5Infravana
|
||||
|
||||
## 摘要（Abstract）
|
||||
|
||||
Modern large language model (LLM) inference has progressively disaggregated to keep pace with growing model sizes and tight TTFT and TPOT service-level objectives: from chunked-prefill aggregation, to prefill–decode (P/D) disaggregation, and most recently to operator-level Attention–FFN Disaggregation (AFD). This trend is especially important for mixture-of-experts (MoE) models, where memory-bound attention, compute-intensive expert FFNs, and MoE dispatch/combine communication create distinct resource demands across the serving pipeline. AFD further exposes this heterogeneity by placing attention and MoE-FFN execution on separate GPU groups. Each level of disaggregation deepens the scheduling design space across workload characteristics, resource allocation, and interconnect topology, leaving open the central question: When does each level of disaggregation actually pay off? We systematically characterize this trade-off for MoE inference across realistic workload use cases defined by input/output sequence lengths, prefix-KV reuse, and per-user latency constraints. Using chunked-prefill and P/D disaggregation as strong baselines, we study the benefits and limits of AFD at scale through a framework that fuses rich on-device kernel measurements with high-fidelity network simulation. Our findings deliver a practical map of when and where deeper disaggregation pays off for MoE serving at scale. Under strict TTFT/TPOT SLOs, AFD sustains around 4k tokens/s of system throughput on DeepSeek-V3.2 across chat, coding, and agentic-coding workloads, regimes in which non-AFD deployments are infeasible. Our design and analysis further distill concrete takeaways for jointly optimizing system throughput and user interactivity, including how to partition attention and FFN across GPUs as a function of workload and model architecture, providing design principles for current rack- and cluster-scale deployments as well as future disaggregated AI infrastructure.
|
||||
|
||||
## 1 引言（Introduction）
|
||||
|
||||
The rapid scaling of agentic large language models (LLMs) has enabled AI systems to perform increasingly complex tasks, including multi‑turn reasoning, code generation, and autonomous decision‑making. As these models grow to hundreds of billions of parameters and operate on long input contexts, their deployment places unprecedented pressure on inference infrastructure, particularly due to the expanding KV‑cache footprint and the need to scale across multiple compute nodes. At the same time, emerging agentic workloads and model architectures exhibit increasing compute characteristic heterogeneity, exposing limitations in existing LLM serving paradigms that struggle to simultaneously achieve high performance, efficiency, and scalability.
|
||||
|
||||
A central challenge stems from the heterogeneous execution characteristics of different components within modern LLM architectures, which impose conflicting demands on compute, memory bandwidth, and communication resources. Prior systems mitigate these effects through batching and scheduling techniques such as chunked prefill Agrawal et al. (2023) and continuous batching Yu et al. (2022), pipelining, or coarse‑grained prefill–decode (P/D) disaggregation Zhong et al. (2024); Patel et al. (2023). While effective at reducing phase‑level interference, these approaches implicitly treat the model as a monolithic execution unit, obscuring fine‑grained trade‑offs that become dominant at scale.
|
||||
|
||||
|
||||
|
||||
Figure 1: AIC++ Framework Overview. AIC++ takes model architecture, hardware configuration, and workload constraints as inputs and performs design-space exploration to identify optimal scheduling across token-level parallelism—data (DP), sequence (SP), tensor (TP), pipeline (PP), and expert parallelism (EP)—as well as phase-level prefill/decode (P/D) and operator-level attention–FFN disaggregation (AFD) [C1]. The framework leverages AIConfigurator to model heterogeneous GPU clusters (GPUA,GPUBGPU_{A},GPU_{B}) interconnected via scale-up (NVLink) and scale-out (InfiniBand) fabrics simulated with AstraSim [C2]. Based on this analysis, attention (PA,DA1,DA2P_{A},D_{A,D_{A) and FFN (PF,DF1,DF2P_{F},D_{F,D_{F) operators are placed on specific GPUs to maximize overall system throughput [C3].
|
||||
|
||||
Recent studies such as MegaScale-Infer Zhu et al. (2025) show that coarse-grained inference abstractions break down for large heterogeneous models, especially MoE architectures, where substantial compute heterogeneity exists within each Transformer block. Attention variants such as MHA Ashish (2017), GQA Hudson and Manning (2019), and MLA DeepSeek-AI et al. (2024) are largely memory-bound due to KV-cache access and data movement, whereas FFNs are compute-bound and dominated by dense GEMMs. Although MegaScale-Infer highlights inefficiencies from attention–FFN heterogeneity, it remains unclear how Attention–FFN Disaggregation (AFD) composes with existing parallelism strategies, prefill–decode (P/D) disaggregation, and different attention architectures.
|
||||
|
||||
Beyond architectural heterogeneity, AFD effectiveness also depends on workload characteristics such as input/output sequence length (ISL/OSL), prefix length, and system load (tokens/s/user). These factors directly affect scheduling decisions and determine when disaggregation is beneficial. Understanding these trade-offs is critical for current cluster-scale LLM serving and for future disaggregated inference platforms, including NVIDIA Groq 3 LPX NVIDIA (2026a) and Intel/SambaNova-style systems Intel (2026).
|
||||
|
||||
In this paper, we present a systematic study of AFD for LLM inference across diverse application domains. We analyze efficient deployment across three dimensions: distributed parallelism, including tensor, data, pipeline, sequence, and expert parallelism; phase-level P/D disaggregation; and operator-level attention–FFN disaggregation, as shown in Figure˜1(C1). We formulate scheduling selection as a design-space exploration (DSE) problem that captures operator-level compute heterogeneity and inter-node communication costs.
|
||||
|
||||
To support this study, we develop AIConfigurator++ (AIC++), a co-design framework that combines operator-level compute modeling from NVIDIA AIConfigurator Xu et al. (2026) with distributed communication modeling using AstraSim Rashidi et al. (2020). Grounded in a customized vLLM-based AFD prototype Kwon et al. (2023), AIC++ integrates kernel measurements with system-level simulation to evaluate compute–communication trade-offs across scheduling strategies, as illustrated in Figure˜1(C2).
|
||||
|
||||
We evaluate chatbot, coding, and agentic workloads using DeepSeek-V3.2, GPT-OSS-120B, Nemotron3-120B, and Qwen3-235B, covering MLA, GQA, sparse attention, and Mamba-based architectures. By varying ISL, OSL, prefix length, and system load, our framework identifies when AFD is beneficial, how micro-batching and operator placement improve compute–communication overlap, and which scheduling strategy maximizes throughput and interactivity under SLO constraints. Importantly, our analysis translates workload and model characteristics into concrete attention-to-FFN GPU ratios, providing actionable guidance for cluster-scale deployment.
|
||||
|
||||
More broadly, our findings suggest that as LLM workloads become increasingly heterogeneous, system optimization must move beyond coarse-grained placement toward operator-level disaggregation. Modeling-driven frameworks such as AIC++ can guide the co-design of future heterogeneous inference platforms. From Figure˜2, we observe that under stringent SLOs for TTFT and TPOT on DeepSeek-V3.2, AFD is able to achieve 4k tokens/s system throughput while the non-AFD deployment is infeasible to run. Through exhaustive design-space exploration to analyze the best deployment strategy on a cluster of 128 B200 GPUs (Section˜4), we see that AFD is not the best option when system throughput is the main target compared to the pure P/D disaggregation and chunked prefill. But our analysis that dynamically searches the attention-to-FFN ratios finds that it will always achieve the best latency and user interactivity with the AFD-specific microbatch overlapping technique that best utilizes the compute and communication resources. Finally, we present a case study highlighting AFD’s memory-segmentation benefit: by placing most model weights on FFN GPUs, AFD leaves more memory on attention GPUs for KV cache, enabling higher throughput under the same memory constraint.
|
||||
|
||||
Our key contributions are:
|
||||
|
||||
- C1:
|
||||
|
||||
Multi-dimensional DSE: We jointly optimize LLM serving across token-level parallelism, P/D disaggregation, and AFD, and translate workload characteristics into concrete attention-to-FFN GPU ratios.
|
||||
- C2:
|
||||
|
||||
System modeling for disaggregated architectures: We build AIC++, which models compute and data-transfer costs for scaled-out disaggregated inference systems.
|
||||
- C3:
|
||||
|
||||
Optimal AFD operator placement: We demonstrate that careful placement of attention and FFN operators is critical for effective AFD deployment.
|
||||
|
||||
|
||||
|
||||
Figure 2: System throughput at the best feasible deployment on DeepSeek-V3.2 (128× B200 with trtllm backend) under strict SLOs (TTFT<50/100/150 ms for Chat/Coding/Agentic Coding; TPOT caps 15 ms). Red cross marks infeasibility — the auto-con
|
||||
|
||||
advisory chatbots - typically exhibit large context length and small ISLs.
|
||||
These workloads favor aggregated deployments, as frequent cross‑node data transfers in disaggregated systems incur high communication overheads.
|
||||
In contrast, workloads with long ISLs, common in code‑completion and code‑generation tasks, benefit more from asynchronous function decomposition (AFD), which introduces an additional dimension to the scheduling design space.
|
||||
In this work, we characterize different workloads with different ISL, OSL and context length to find the Pareto-optimal scheduling that balances system throughput and user interactivity under SLO constraints (detailed in section 4).
|
||||
|
||||
### 2.2 Finding the optimal scheduling with disaggregated architecture
|
||||
|
||||
The emergence of disaggregated architectures with heterogeneous compute units within a node—such as NVIDIA Groq‑3 LPX, Rubin CPX Nvidia (2024), and Intel SambaNova—has made fine‑grained AFD an effective scheduling strategy despite the increased communication volume. High‑bandwidth scale‑up interconnects in these heterogeneous clusters enable memory‑bound attention operators to execute on memory‑rich devices, while compute‑intensive FFN blocks benefit from accelerators optimized for high arithmetic throughput.
|
||||
|
||||
System modeling requirements of disaggregated architecture:
|
||||
Exploring a large design space by evaluating hundreds of candidate configurations is prohibitively expensive for large‑scale disaggregated systems as observed by Miao et al. (2022); Zheng et al. (2022).
|
||||
Hence, accurate system modeling of compute and communication infrastructure is necessary for evaluation.
|
||||
AIConfigurator Xu et al. (2026) provides a compute modeling framework to estimate the compute and memory bandwidth of a wide range of modern GPUs.
|
||||
However, we additionally need accurate estimation of the data-transfer cost for fine-grained operator-level AFD.
|
||||
To address this, we augment AIConfigurator with the AstraSim Rashidi et al. (2020) network simulator to build AIC++,
|
||||
a disaggregated modeling framework that captures the compute-communication effects, maximizing their overlap for fine-grained AFD.
|
||||
Moreover, we evaluate the impact of AFD in heterogeneous scale-up AIC++ deployments.
|
||||
|
||||
|
||||
|
||||
Figure 3: (a) Runtime breakdown and (b) memory component breakdown for different model architectures running prompts with different context lengths.
|
||||
|
||||
Precise placement of attention and FFN operators:
|
||||
Fine‑grained data transfers introduced by AFD can lead to significant interconnect congestion if attention and FFN operators are placed arbitrarily across a disaggregated infrastructure. Consequently, effective scheduling of fine‑grained operator disaggregation requires jointly reasoning about compute affinity and data‑movement costs—a challenge explicitly addressed in our work. In particular, optimal operator placement must account for compute–communication overlap by co‑locating operators with high data‑exchange intensity on nearby or tightly coupled compute nodes. Going beyond prior approaches that primarily model heterogeneous architectures, our work jointly optimizes operator placement by explicitly considering interconnect congestion, identifying optimal mappings of attention and FFN operators that maximize overall system efficiency.
|
||||
|
||||
## 3 AIC++ 框架总览
|
||||
|
||||
In this section, we present AIC++, a framework for exploring the design space of AFD‑enabled MoE serving at scale. AIC++ combines AIConfigurator Xu et al. (2026) for accelerator‑level performance modeling with ASTRA‑Sim Rashidi et al. (2020) for high‑fidelity network simulation, enabling evaluation across attention architectures and application domains. While AIConfigurator accurately models modern LLM kernels, it does not capture AFD‑specific communication paths; we extend it by partitioning execution into attention and MoE‑FFN phases and binding each phase to an independent GPU backend, faithfully modeling runtime and memory behavior on disaggregated devices.
|
||||
|
||||
### 3.1 AFD design for MoE architectures
|
||||
|
||||
As shown in Figure˜4, the transition between the Attention and MoE–FFN phases is mediated by the MoE-Dispatch and MoE-Combine communication operators. In non-AFD deployments, these operators involve matched source and destination counts, as communication is confined to GPU ranks participating in expert parallelism (EP). Under AFD, attention and MoE–FFN phases execute on physically disaggregated GPUs, resulting in asymmetric fan-out or fan-in communication patterns depending on the scheduling strategy. For example, while an MoE model deployed on 8 GPUs with EP=8 exchanges tokens among all 8 GPUs, an AFD configuration with 2 attention GPUs and 6 FFN GPUs induces a fan-out pattern, requiring tokens generated by the attention GPUs to be distributed to a larger set of FFN GPUs.
|
||||
|
||||
AIC++ models these interactions using AstraSim to simulate scale-up and
|
||||
scale-out communication at packet granularity. By coupling AstraSim with
|
||||
AIConfigurator, AIC++ enforces communication-dependent execution, capturing
|
||||
contention, GPU utilization, and data-transfer efficiency under dynamic workloads.
|
||||
|
||||
Concretely, each transformer layer incurs two cross-AFD transfers. In the all-pairs mode used for the main results, the AFD worker establishes pairwise communicators between every attention rank and every FFN rank, so tokens generated by a single attention rank may be routed to any FFN rank hosting the selected experts.
|
||||
A2F/A2E (MoE-Dispatch) transfers post-attention hidden states, token ids, and per-expert routing metadata from attention ranks
|
||||
to FFN ranks hosting the selected experts, where each FFN rank filters tokens according to its hosted experts. Due to the top-kk routing and token dispatch, A2F exhibits a fan-out, making FFN-side ingress congestion dominant. F2A/E2A (MoE-Combine) aggregates expert outputs and
|
||||
returns a single reduced hidden state per token to the originating attention rank,
|
||||
forming a fan-in transfer in which attention-side ingress becomes the bottleneck. AIC++ expands both transfers into a full bipartite
|
||||
traffic matrix and feeds them into AstraSim’s tiered, congestion-aware network model,
|
||||
capturing contention when A2F egress and F2A ingress overlap on full-duplex
|
||||
interconnects. This part of the implementation with its communication pattern is based on our prototype implementation (refer to subsection 6.1).
|
||||
|
||||
### 3.2 Batch Overlap (BO) in AFD
|
||||
|
||||
As shown in Figure˜4 (left), AFD decomposes execution into four stages mapped to separate compute or communication resources: attention computation on the attention GPUs, dispatch of post-attention hidden states, token ids, and per-expert routing metadata to experts, MoE-FFN computation on the FFN GPUs, and aggregation/transfer of expert outputs back to the attention side for the next layer.
|
||||
|
||||
The return communication may share the same channel on half-duplex links, or proceed concurrently over a dedicated channel on full-duplex networks, enabling the pipelined execution shown in Figure˜4(B). Since modern datacenter GPU deployments commonly provide full-duplex interconnects such as NVLink and InfiniBand, AFD can exploit compute-communication overlap across GPUs and the network fabric. Accordingly, in the following discussion, we assume an AFD implementation with four-stage micro-batch overlap enabled. In contrast, under aggregated execution, all devices jointly execute all stages as a single group, as shown in Figure˜4(A).
|
||||
|
||||
To model batch overlap behavior in AIC++, we partition the per-step token budget
|
||||
Tbudget=batch_size×ISLT_{{budget}}=it{batch\_size}it{ISL} into MM microbatches.
|
||||
The number of microbatches MM is chosen to match the effective pipeline depth—three
|
||||
for half-duplex links and four for full-duplex links. For each microbatch,
|
||||
AIC++ queries AIConfigurator’s empirically measured GPU-cluster cost database
|
||||
to obtain the attention and FFN execution costs for a microbatch size of
|
||||
Tbudget/MT_{{budget}}/M, thereby preserving the nonlinear scaling behavior of small
|
||||
GEMMs and collective operations.
|
||||
|
||||
Let sis_{i} denote the measured per-microbatch execution cost of pipeline stage ii,
|
||||
aggregated across all LL transformer layers, and let
|
||||
smax=maxisis_{}=_{i}s_{i} be the bottleneck stage. Under steady-state cross-layer
|
||||
pipelining, the bottleneck stage processes all MM microbatches back-to-back,
|
||||
whereas each non-bottleneck stage incurs only a one-time pipeline fill and drain
|
||||
overhead of si/Ls_{i}/L. This cost is amortized over the full execution rather than
|
||||
charged per layer, consistent with the cross-layer scheduling strategies used by
|
||||
MegaScale-Infer and Step-Fun. The resulting end-to-end pipelined latency is shown in Equation˜1.
|
||||
|
||||
| | | | |
|
||||
| --- | --- | --- | --- |
|
||||
| | tpipe=M⋅smax+∑i:si≠smaxsiL.t_{{pipe}}=M s_{}+_{i:s_{i} s_{}}{s_{i}}{L}. | | (1) |
|
||||
|
||||
The first term captures the steady-state throughput dictated by the bottleneck
|
||||
resource, while the second term accounts for pipeline fill and drain bubbles
|
||||
introduced by non-bottleneck stages. We apply this formulation to both prefill and
|
||||
decode phases, enabling AIC++ to accurately reason about computation and
|
||||
communication costs under batch-overlapped execution.
|
||||
|
||||
|
||||
|
||||
Figure 4: Mapping AFD to 4 different stages in the model execution
|
||||
|
||||
### 3.3 Location-aware GPU placement
|
||||
|
||||
Because AstraSim labels each GPU with its physical position—node, scale-up domain, and link tier—and resolves congestion at packet granularity, AIC++ additionally enables a study of optimal GPU placement under combined AFD and P/D disaggregation. The policy is frequency-driven: the most frequent intra-layer A2F/F2A traffic (𝒪(layer){O}({layer}) per request) is bound to the highest-bandwidth scale-up domain (intra-node NVLink), while the less frequent inter-node KV-cache transfer (𝒪(1){O}(1) per request) is deferred to the scale-out domain (InfiniBand). This grouping co-locates GPUs with high communication affinity onto faster interconnects, avoiding contention on slower links. The detailed placement study—including segregated vs. paired P/D layouts and the resulting KV-transfer speed-up—is given in Appendix 6.2.
|
||||
|
||||
We also consider asymmetric prefill and decode configurations reflecting their distinct compute characteristics. AIC++ enables such asymmetry through independent prefill/decode scheduling and asymmetric attention and FFN worker allocation, while the network simulator co‑locates operators to reduce interconnect congestion and improve efficiency.
|
||||
|
||||
## 4 Evaluation
|
||||
|
||||
### 4.1 Design-Space Exploration (DSE) of different serving strategies
|
||||
|
||||
#### 4.1.1 Input workloads
|
||||
|
||||
To demonstrate the applicability of AFD, we evaluate a diverse set of model architectures spanning multiple application domains, as summarized in Table˜1. Specifically, we consider recent production‑deployed models including Qwen3‑235B, commonly used for chatbot workloads; GPT‑OSS‑120B, targeting medium‑scale reasoning tasks; DeepSeek‑V3.2, designed for reasoning‑intensive and agentic coding workflows; and Nemotron3‑120B, optimized for large‑context applications such as RAG serving Nvidia (2026). The corresponding prefix sizes, ISL, OSL, and architectural characteristics for each workload are detailed in Table˜1.
|
||||
|
||||
Table 1: Representative application workloads and model architectures.
|
||||
|
||||
Application Workload
|
||||
|
||||
| | | | |
|
||||
| --- | --- | --- | --- |
|
||||
| Use Case | Prefix | ISL | OSL |
|
||||
| Chat | 4096 | 512 | 256 |
|
||||
| Coding | 2048 | 4096 | 1024 |
|
||||
| Agentic Coding | 524k 111Models listed in this table may not natively support a 524k context window. We use 524k to model long-context agentic workloads and quantify the system impact of large KV-cache residency. | 256 | 8192 |
|
||||
|
||||
Model Architecture
|
||||
|
||||
| | | | |
|
||||
| --- | --- | --- | --- |
|
||||
| Model | Attention | # Experts | Precision |
|
||||
| GPT-OSS-120B | Full GQA + Window GQA | 128 | FP8 |
|
||||
| Qwen3-235B | GQA | 128 | FP8 |
|
||||
| Nemotron3-120B | Mamba 2 + GQA | 512 | FP8 |
|
||||
| DeepSeek V3.2 | MLA + Sparse Attention | 256 | FP8 |
|
||||
|
||||
#### 4.1.2 Cluster-scale DSE analysis
|
||||
|
||||
Figure˜5 reports the Pareto curve (tokens/s/user vs system tokens/s) produced by AIC++ on a 128 B200SXM cluster with TensorRT-LLM NVIDIA (2026b) as the performance backend, with each panel constrained by the workload’s SLO from Table˜1. To exhaustively probe the throughput frontier, our replica search enumerates every replica size from 2 to 128 GPUs and dynamically composes the per-replica parallelism plan across TP, DP, EP, and (for AFD) attention/FFN GPU groups. This lets the optimizer surface asymmetric layouts that only become feasible when prefill and decode have very different compute and memory profiles, including off-grid replica sizes that pack a long-context KV cache more efficiently than the obvious power-of-two choices. The cluster-scale results in this section are model-based DSE estimates that combine backend cost measurements with AstraSim communication modeling. Appendix 6.1 describes our vLLM-based AFD prototype, which we use to verify functional correctness of the all-pairs attention–FFN execution path and to ground the communication pattern modeled by AIC++.
|
||||
|
||||
|
||||
|
||||
Figure 5: Evaluation of a Cluster with 128 B200 GPUs for the representative model architectures and application workloads shown in Table 1, Y-axis denotes the system throughput (total tokens/s), X-axis denotes the interactivity (tokens/s/user)
|
||||
|
||||
No single strategy dominates the throughput frontier. Aggregated serving with chunked prefill, deployed as 16 single-node 8-GPU replicas that hold the expert FFNs through Expert Parallelism, wins most panels by amortizing the chunked-prefill bubble across the replica fleet and ingesting many disjoint token batches in parallel. Disaggregation takes the rest, but only after the wider search exposes asymmetric replica shapes: a single off-grid replica with many small 2-GPU prefill workers feeding a few large 8-GPU decode workers (Qwen3-235B-A22B chat, DeepSeek-V3.2 coding), or many tiny xPyD shards under _disagg+AFD_ that double aggregated’s throughput on Nemotron-3-Super coding. AFD on its own wins the throughput frontier on a single panel, GPT-OSS-120B chat, where the cheap sliding-window GQA layers let four 32-GPU replicas with a near-symmetric 16A+16F split outpace 16-replica agg. The wider TP-16 enumeration also unlocks a feasible _disagg-standard_ layout for DeepSeek-V3.2 agentic coding, where a narrower TP,≤,8 search returned no SLO-feasible configuration for the 524k-prefix workload.
|
||||
|
||||
On the latency axis, AFD wins every panel, with the optimal attention/FFN split tracking each model’s intrinsic attention/mixer cost relative to its FFN cost. DeepSeek-V3.2, whose MLA combined with sparse (DSA) attention shrinks both per-token attention compute and the KV-cache footprint, collapses the attention shard to its minimum on long-context workloads, dedicating almost the entire cluster to FFN (2A+126F on agentic, 16A+112F on chat). The 2A+126F layout is initially counter-intuitive given the 524k-token KV demand, but consistent with rate-matching: MLA compresses the latent KV cache so aggressively that the entire prefix fits in two GPUs’ HBM, and DSA keeps per-token attention cheap enou
|
||||
|
||||
manov.
|
||||
Vidur: A large-scale simulation framework for llm inference, 2024.
|
||||
URL https://arxiv.org/abs/2405.05465.
|
||||
- Ashish [2017]
|
||||
|
||||
Vaswani Ashish.
|
||||
Attention is all you need.
|
||||
_Advances in neural information processing systems_, 30:I, 2017.
|
||||
- Bambhaniya et al. [2026]
|
||||
|
||||
Abhimanyu Rajeshkumar Bambhaniya, Hanjiang Wu, Suvinay Subramanian, Sudarshan Srinivasan, Souvik Kundu, Amir Yazdanbakhsh, Midhilesh Elavazhagan, Madhu Kumar, Minlan Yu, Arijit Raychowdhury, and Tushar Krishna.
|
||||
Mist: A co-design framework for heterogeneous, multi-stage llm inference, 2026.
|
||||
URL https://arxiv.org/abs/2504.09775.
|
||||
- DeepSeek-AI et al. [2024]
|
||||
|
||||
DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng
|
||||
@@ -0,0 +1,94 @@
|
||||
# 📊 文章摘要：How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving
|
||||
|
||||
> **原文**：[2026-05-27_AFD_How_Far_Can_Disaggregation_Go.md](./2026-05-27_AFD_How_Far_Can_Disaggregation_Go.md)
|
||||
> **原文链接**：https://arxiv.org/abs/2605.28302
|
||||
> **来源**：arXiv
|
||||
> **作者**：Hanjiang Wu, Abhimanyu Rajeshkumar Bambhaniya, Sarbartha Banerjee, Tuhin Khare, Sudarshan Srinivasan, Suvinay Subramanian 等 12 位（佐治亚理工学院、Intel、Google、Google DeepMind、Infravana）
|
||||
> **发布日期**：2026-05-27
|
||||
> **摘要日期**：2026-08-06
|
||||
> **价值评级**：⭐⭐⭐ 高
|
||||
|
||||
---
|
||||
|
||||
## 核心命题
|
||||
|
||||
> **算子级解耦** — 解耦不止于阶段（prefill/decode），可下探到算子级：把内存受限的 attention 与计算密集的 MoE-FFN 拆分到不同 GPU 组（AFD），并以设计空间探索给出"何时、何处更深层解耦才划算"的实用地图。
|
||||
|
||||
---
|
||||
|
||||
## 文章概要
|
||||
|
||||
随着 MoE 模型（如 DeepSeek-V3.2）规模扩大与 TTFT/TPOT SLO 收紧，LLM serving 的解耦粒度不断加深：从 chunked-prefill 聚合，到 prefill/decode（P/D）解耦，再到算子级 attention-FFN 解耦（AFD）。本文系统回答"每一层解耦何时真正划算"：基于 vLLM 的 AFD 原型 + AIConfigurator（算子级计算建模）+ AstraSim（包粒度网络模拟）构建 AIC++ 框架，对 128 卡 B200 集群上 4 种模型（DeepSeek-V3.2、GPT-OSS-120B、Nemotron3-120B、Qwen3-235B）× 3 类负载（chat/coding/agentic coding）做全维设计空间探索。核心发现：严格 SLO 下 AFD 在 DeepSeek-V3.2 上可维持约 4k tokens/s 系统吞吐，而非 AFD 部署不可行；吞吐维度没有单一策略通吃（chunked prefill 聚合常胜），但延迟/交互性维度 AFD 全胜，且最优 attention/FFN GPU 比例可依模型与负载导出（如 DeepSeek-V3.2 长上下文下 2A+126F 的极端配比）。局限：结果基于建模 + 模拟的 DSE 估计，原型仅验证功能正确性。
|
||||
|
||||
---
|
||||
|
||||
## 关键要点
|
||||
|
||||
1. **解耦粒度演化谱系** — chunked-prefill 聚合 → P/D 阶段解耦 → AFD 算子级解耦，每一层解耦都加深调度设计空间（工作负载特性 × 资源分配 × 互连拓扑），本文首次系统回答"解耦能走多远"这一问题。`[分类: 范式突破]`
|
||||
2. **MoE 架构放大算子异构** — attention（MHA/GQA/MLA）内存受限、FFN 计算密集，加上 MoE dispatch/combine 通信，一个 transformer 块内就存在显著计算异构；粗粒度抽象（MegaScale-Infer 已指出的问题）在 MoE 上尤其失效。`[分类: 共识]`
|
||||
3. **严格 SLO 下 AFD 使不可能变为可能** — DeepSeek-V3.2 上以 TTFT<50/100/150ms、TPOT≤15ms 的严格约束（chat/coding/agentic coding），AFD 部署维持约 4k tokens/s 系统吞吐，而非 AFD 部署不可行（infeasible）。`[分类: 范式突破]`
|
||||
4. **吞吐前沿无通吃方案** — 128×B200 集群 DSE：chunked prefill 聚合部署（16 个单节点 8-GPU 副本）赢得多数面板；AFD 单独赢得一个面板（GPT-OSS-120B chat，16A+16F 近似对称拆分）；非对称副本形态（如 2-GPU prefill 工人喂 8-GPU decode 工人）是解耦方案胜出的关键。`[分类: 争议]`
|
||||
5. **延迟维度 AFD 全胜** — 每个面板的最优 attention/FFN 拆分都追踪模型内在的 attention/mixer 成本与 FFN 成本之比；AFD 特定微批重叠（four-stage pipeline，全双工链路）最大化计算-通信重叠，始终给出最佳延迟与用户交互性。`[分类: 未探索]`
|
||||
6. **反直觉的极端配比** — DeepSeek-V3.2 的 MLA + 稀疏注意力把每 token attention 计算与 KV cache 足迹压得极小，长上下文 agentic 负载（524k prefix）下最优配置为 2A+126F——整个 524k 前缀的 KV cache 可装入 2 个 GPU 的 HBM，几乎整个集群给 FFN。`[分类: 范式突破]`
|
||||
7. **内存分割是隐藏收益** — 把多数模型权重放在 FFN GPU 上，attention GPU 腾出显存装 KV cache，同一内存约束下可支撑更高吞吐——AFD 的收益不止调度层面。`[分类: 未探索]`
|
||||
8. **位置感知放置原则** — 最频繁的层内 A2F/F2A 流量（每请求 O(layer)）绑定最高带宽 scale-up 域（NVLink），低频的跨节点 KV cache 传输（每请求 O(1)）走 scale-out（InfiniBand）；A2F 扇形广播（fan-out）使 FFN 侧入口拥塞、F2A 扇形汇聚（fan-in）使 attention 侧入口成为瓶颈。`[分类: 未探索]`
|
||||
9. **AIC++ 方法论** — 内核级实测成本库（AIConfigurator）+ 包粒度拥塞感知网络模拟（AstraSim）联合建模，以 vLLM AFD 原型锚定通信模式与功能正确性；论文明示集群级结果均为"模型驱动 DSE 估计"。`[分类: 争议]`
|
||||
|
||||
---
|
||||
|
||||
## 批判性分析
|
||||
|
||||
### 假设前提
|
||||
|
||||
- 异构/解耦硬件时代将到来：节点内出现计算/内存不对称的加速器组（NVIDIA Groq-3 LPX、Rubin CPX、Intel/SambaNova 等），且存在 NVLink 级高带宽 scale-up 互连。
|
||||
- AFD 的通信模式（all-pairs 的 A2F/F2A 双向传输）在真实网络上可被微批重叠技术有效隐藏。
|
||||
- 内核级测量成本库 + AstraSim 网络模拟的组合足以逼近真实集群行为，从而替代全系统原型评估。
|
||||
- 以系统吞吐与用户交互性（tokens/s/user）为目标的 SLO 框架能代表生产诉求。
|
||||
|
||||
### 论据与逻辑
|
||||
|
||||
- 论据结构严谨：先以运行时分解与内存分解数据（图 3）确立算子异构事实，再用 AIC++ 在 128 卡规模做穷举式 DSE（副本规模 2-128 卡全枚举 + TP/DP/EP/SP/PP 与 P/D、AFD 联合搜索），结论（无通吃方案、AFD 延迟全胜、配比可推导）有模拟数据支撑。
|
||||
- 关键量化结论（约 4k tokens/s、2A+126F、2.35×/1.4× 类收益的可比性）均限定在各自严格语境中，未过度外推。
|
||||
- 弱点：集群级结论全部来自建模估计，AIC++ 的准确性未与真实 AFD 集群端到端对照；vLLM 原型仅验证功能正确性，无法排除建模误差对"吞吐前沿无通吃方案"等核心结论的影响；对非 AFD 部署"不可行"的判定依赖 SLO 设定与搜索范围的完整性（论文也承认窄 TP≤8 搜索曾漏掉可行配置）。
|
||||
|
||||
### 边界与局限
|
||||
|
||||
- AFD 的优势集中于延迟/交互性维度与严格 SLO 场景；纯吞吐目标下 chunked prefill 聚合常更优——结论不可泛化为"AFD 总是更好"。
|
||||
- 依赖高带宽 scale-up 互连；半双工/低带宽网络下微批重叠收益收缩（pipeline 深度由 4 降为 3）。
|
||||
- 配比指导依赖模型架构（MLA/GQA/Mamba 差异显著），换架构需重跑 DSE。
|
||||
- 结果基于模拟，需真实异构集群验证；524k prefix 属极端长上下文建模，超出部分模型原生窗口。
|
||||
|
||||
---
|
||||
|
||||
## 可引用金句
|
||||
|
||||
> "Our findings deliver a practical map of when and where deeper disaggregation pays off for MoE serving at scale."
|
||||
> （我们的发现提供了一张实用地图：在 MoE 规模化服务中，何时、何处更深层的解耦才真正划算。）
|
||||
|
||||
> "Under strict TTFT/TPOT SLOs, AFD sustains around 4k tokens/s of system throughput on DeepSeek-V3.2 across chat, coding, and agentic-coding workloads, regimes in which non-AFD deployments are infeasible."
|
||||
> （在严格的 TTFT/TPOT SLO 下，AFD 在 DeepSeek-V3.2 上于 chat、coding 与 agentic-coding 负载中维持约 4k tokens/s 的系统吞吐，而这是非 AFD 部署不可行的场景。）
|
||||
|
||||
---
|
||||
|
||||
## 总体评价
|
||||
|
||||
**亮点**：
|
||||
- 首次把解耦粒度推进到算子级（AFD）并系统回答"解耦能走多远"，把 2023-2024 年的阶段解耦共识延伸为新范式
|
||||
- "吞吐无通吃方案、延迟 AFD 全胜"的差异化结论打破了"解耦必然更好"的朴素预期，结论颗粒度精细
|
||||
- 给出可操作的 attention/FFN GPU 配比指导（含 2A+126F 这类反直觉结论背后的推理），并揭示内存分割这一隐藏收益
|
||||
- AIC++ 把内核实测与网络模拟结合的建模方法论，为异构推理基础设施设计提供了可复用工具
|
||||
|
||||
**不足**：
|
||||
- 集群级结论全部基于建模与模拟，缺少真实 AFD 集群的端到端验证
|
||||
- 部分结论（如"非 AFD 不可行"）对 SLO 设定与搜索范围敏感，泛化需谨慎
|
||||
- 评估硬件模型（B200 + TensorRT-LLM）与实际异构平台（Groq-3 LPX 等）仍有差距
|
||||
|
||||
**适用场景**：MoE 模型 serving 的系统架构师；面向异构加速器（Groq-3 LPX、Rubin CPX 等）与解耦集群的设计者；研究 LLM serving 设计空间探索的研究人员。
|
||||
|
||||
**关联建议**：向上游追溯 DistServe（P/D 解耦起点）、Splitwise（异构硬件）、Mooncake（生产实践），可看出"解耦粒度深化"的完整脉络；后续可关注 MegaScale-Infer（同方向的算子级优化）、Mist（同团队的异构多阶段 co-design 框架）以及 NVIDIA 解耦平台（LPX/CPX）的真实部署验证。
|
||||
|
||||
---
|
||||
|
||||
## 配图
|
||||
|
||||
|
||||
@@ -0,0 +1,388 @@
|
||||
# Splitwise: Efficient Generative LLM Inference Using Phase Splitting
|
||||
|
||||
> **来源**：arXiv
|
||||
> **作者**：Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini
|
||||
> **发布日期**：2023-11-30
|
||||
> **原文链接**：https://arxiv.org/abs/2311.18677
|
||||
|
||||
---
|
||||
|
||||
## 论文元数据
|
||||
|
||||
- **arXiv ID**：2311.18677
|
||||
- **学科分类**：Distributed, Parallel, and Cluster Computing (cs.DC)
|
||||
- **作者机构**：华盛顿大学（University of Washington）、微软（Microsoft）
|
||||
- **提交历史**：v1: 2023-11-30；v2: 2024-05-20
|
||||
- **DOI**：https://doi.org/10.48550/arXiv.2311.18677
|
||||
|
||||
---
|
||||
|
||||
Pratyush Patel1,
|
||||
Esha Choukse2,
|
||||
Chaojie Zhang2,
|
||||
|
||||
Aashaka Shah2,
|
||||
Íñigo Goiri2,
|
||||
Saeed Maleki2,
|
||||
Ricardo Bianchini2
|
||||
|
||||
1University of Washington 2Microsoft
|
||||
|
||||
## 摘要（Abstract）
|
||||
|
||||
Generative large language model (LLM) applications are growing rapidly, leading to large-scale deployments of expensive and power-hungry GPUs.
|
||||
Our characterization of LLM inference shows that each inference request undergoes two phases: a compute-intensive prompt computation phase and a memory-intensive token generation phase, each with distinct latency, throughput, memory, and power characteristics.
|
||||
Despite state-of-the-art batching and scheduling, the token generation phase underutilizes compute resources.
|
||||
Unlike prompt computation, token generation does not need the compute capability of the latest GPUs and can be run with lower power and cost.
|
||||
|
||||
Based on these insights, we propose Splitwise, a model deployment and scheduling technique that splits the two phases of LLM inference requests on to separate machines.
|
||||
Splitwise enables phase-specific resource management using hardware that is well suited for each phase.
|
||||
Request state is transferred efficiently between machines using optimized network libraries on the fast back-plane interconnects available in today’s GPU clusters.
|
||||
Using Splitwise, we design homogeneous and heterogeneous LLM inference clusters optimized for throughput, cost, and power.
|
||||
Compared to current designs, Splitwise clusters achieve up to 1.4×1.4 × higher throughput at 20% lower cost. Alternatively, they can deliver 2.35×2.35 × more throughput under the same power and cost budgets.
|
||||
|
||||
## I 引言（Introduction）
|
||||
|
||||
Recent advancements in generative large language models (LLMs) have significantly improved their response quality and accuracy [18, 71].
|
||||
These trends have led to the widespread adoption of LLMs across various domains [6, 21].
|
||||
Most modern LLMs are built using the transformer architecture [78, 77] and exhibit similar characteristics [63].
|
||||
Transformer model sizes have grown steadily, from the early BERT models [36] having 340 million parameters, to GPT-3 [28] with a staggering 175 billion parameters, and GPT-4 rumored to have even more.
|
||||
|
||||
LLMs typically run on expensive and power-hungry GPUs [16].
|
||||
The sudden and large-scale deployment of LLMs has led to a worldwide GPU capacity crunch [14].
|
||||
The computational demand for LLM inference far exceeds that of training due to the vast number of applications leveraging LLMs.
|
||||
Furthermore, since training LLMs requires expensive and dedicated supercomputers [60, 56], a large number of inferences are necessary to amortize the high training costs.
|
||||
LLM inference jobs, although orders of magnitude smaller than training, are still expensive given the compute involved.
|
||||
11脚注: Work partly done as an intern at Microsoft.
|
||||
|
||||
TABLE I: NVIDIA A100 vs. H100 specifications.
|
||||
|
||||
Generative LLM inference for a single request consists of several forward passes through the model, since the output tokens are generated one by one.
|
||||
This inherently has two contrasting phases of computation.
|
||||
First, the _prompt computation phase_, in which all the input prompt tokens run through the forward pass of the model in parallel to generate the first output token.
|
||||
This phase tends to be computationally intensive and requires the high FLOPs (floating point operations per second) of the latest GPUs today.
|
||||
Second, the _token generation phase_, in which subsequent output tokens are generated sequentially based on the forward pass of the last token and all the cached context from previous tokens in the sequence.
|
||||
Given the lack of compute parallelism, this phase tends to be more memory bandwidth and capacity bound, despite state-of-the-art batching.
|
||||
Running both phases on the same machine often leads to inconsistent end-to-end latencies due to the arbitrary batching of prompt and token phases.
|
||||
Due to these challenges, services need to over-provision expensive GPUs to meet tight inference service level objectives (SLOs) for interactive applications.
|
||||
At the same time, cloud service providers (CSPs) are having to build a lot of new datacenters to meet the GPU demand, and are running into a power wall [19].
|
||||
|
||||
The industry continues to release new computationally powerful GPUs, each much more power hungry and expensive than the last.
|
||||
However, as shown in Table I, the high-bandwidth memory (HBM) capacity and bandwidth on these GPUs has not scaled at the same rate recently.
|
||||
The latest NVIDIA H100 GPUs have 3.43×3.43 × more compute and 1.75×1.75 × more power compared to their predecessor A100 GPUs.
|
||||
However, their memory bandwidth only grew by 1.6×1.6 ×, with no increase in memory capacity.
|
||||
|
||||
Our work.
|
||||
Given the distinct properties of prompt computation and token generation phases, we propose splitting the inference request and running them on separate machines.
|
||||
Doing so allows us to separately manage hardware resources for each phase, thereby increasing the GPU utilization and the overall efficiency of the system.
|
||||
It also enables using different, better-suited hardware for each phase.
|
||||
To realize such a setup, the cached context from the prompt computation needs to be communicated over from the prompt processing machine to the token generation machine at low latency.
|
||||
We implement these transfers in an optimized manner over the back-end Infiniband interconnects avaialble in datacenters today, allowing us to increase efficiency without any perceived performance loss.
|
||||
|
||||
With Splitwise, we design clusters optimized for cost, throughput, and power, using production traces of LLM inference requests [4].
|
||||
Given the diverging memory and compute scaling rates across GPU generations, we also evaluate different GPUs and power caps for the different inference phases.
|
||||
This allows us to target better performance per dollar (Perf/$) for users, and better performance per watt (Perf/W) for CSPs.
|
||||
Additionally, users can target older GPUs, which are likely more readily available to them.
|
||||
|
||||
We show that Splitwise-based LLM inference clusters can achieve 1.4× higher throughput at 20% lower cost than existing clusters. Alternatively, they can deliver 2.35× more throughput with the same cost and power budgets.
|
||||
|
||||
Summary.
|
||||
We make the following contributions:
|
||||
|
||||
1. 1.
|
||||
|
||||
An extensive characterization of the differences in the execution and utilization patterns of the prompt and token generation phases in LLM inference on the NVIDIA A100 and H100 GPUs using production traces.
|
||||
2. 2.
|
||||
|
||||
Splitwise, our technique for optimized utilization of available hardware, which splits the prompt computation and token generation phases onto separate machines.
|
||||
3. 3.
|
||||
|
||||
A design exploration of homogeneous and heterogeneous cluster deployments with Splitwise to optimize the overall cost, request throughput, and provisioned power.
|
||||
4. 4.
|
||||
|
||||
An evaluation of the systems designed with Splitwise using production traces.
|
||||
|
||||
## II Background
|
||||
|
||||
### II-A Large Language Models
|
||||
|
||||
Modern LLMs are based on transformers.
|
||||
Transformer models use attention [77] and multi-layer-perceptron layers to understand the inputs and generate an output, respectively.
|
||||
Transformer-based LLMs include encoder-only [36, 54], decoder-only [67, 69, 71], and encoder-decoder [70] models.
|
||||
Generative LLMs, the focus of this paper, are usually either decoder-only, or encoder-decoder models.
|
||||
|
||||
### II-B Generative LLM inference phases
|
||||
|
||||
Figure 1 shows an example of generative LLM inference.
|
||||
Once the prompt query is received, all the input tokens are computed in parallel, within a single iteration, to generate the first token.
|
||||
We call this the prompt processing phase.
|
||||
The context generated from the attention layers during the prompt computation is saved in the key-value (KV) cache, since it is needed for all the future token generation iterations.
|
||||
After the first token is generated, the following tokens only use the last generated token and the KV-cache as inputs to the forward pass of the model.
|
||||
This makes the subsequent token generation more memory bandwidth and capacity intensive than the computationally heavy prompt phase.
|
||||
|
||||
Figure 1: An LLM inference example.
|
||||
|
||||
### II-C Performance metrics for LLMs
|
||||
|
||||
Prior work has proposed three main metrics for LLM inference: end-to-end (E2E) latency, time to first token (TTFT), and throughput.
|
||||
We add another latency metric: time between tokens (TBT), to track the online streaming throughput of the tokens as they are generated serially.
|
||||
Table II summarizes the key performance metrics that we consider in this work.
|
||||
|
||||
TABLE II: Performance metrics for LLMs.
|
||||
|
||||
Generative LLMs may be used for a variety of tasks with different kinds of SLOs.
|
||||
For batch tasks (_e.g._, summarization), TTFT or TBT latency metrics are less important than throughput.
|
||||
On the other hand, for latency-sensitive tasks (_e.g._, conversational APIs), TTFT and TBT are the more important metrics with tighter SLOs.
|
||||
|
||||
Figure 2: Batching mechanisms and their latency impact on the pro
|
||||
|
||||
ut keeps scaling up with the batch size until the machine runs out of memory.
|
||||
For this reason, the MLS tracks the memory and starts queueing tokens once the machine is close to running out of memory.
|
||||
|
||||
Mixed machines.
|
||||
To meet the TTFT SLO, the MLS must prioritize running prompts and schedule any new prompts in the pending queue immediately.
|
||||
If the machine is running token phases and has no additional capacity to run the prompt phase, the MLS will _preempt_ tokens.
|
||||
To avoid _starvation_ of the token phase due to preemption, we increase the priority of the token with age and limit the number of preemptions that each request can have.
|
||||
|
||||
### IV-C KV-cache transfer
|
||||
|
||||
As discussed in Section II, the KV-cache is generated during the prompt phase of the request, and it continuously grows during the token generation phase.
|
||||
In Splitwise, we need to transfer the KV-cache from the prompt machine to the token machine (shown in Figure 10) to complete the inference.
|
||||
This transfer delay is the main overhead associated with Splitwise.
|
||||
In this section, we discuss the impact of KV-cache transfer and how we optimize it.
|
||||
|
||||
(a)
|
||||
|
||||
(b)
|
||||
|
||||
Figure 11: Optimizing KV-cache transfer in Splitwise.
|
||||
|
||||
Figure 11(a) shows the Gantt chart for the prompt phase, the KV-cache transfer, and the token generation phase for a single batch of requests when naively transferring the KV cache in a serialized way.
|
||||
The KV-cache transfer starts only after the prompt phase has finished and the first token is generated.
|
||||
Further, it needs to complete before the next output token can be generated in the token generation phase.
|
||||
This directly impacts the maximum TBT and end-to-end latency of inference.
|
||||
|
||||
The time required for the transfer depends on the size of the KV cache (which is directly proportional to the number of prompt tokens) and on the bandwidth of the interconnect between the prompt and the token machines.
|
||||
Even when using fast InfiniBand links, the transfer overhead for large prompt sizes could become a significant fraction of the TBT.
|
||||
|
||||
In Splitwise, we optimize the KV-cache transfer by overlapping it with the computation in the prompt phase.
|
||||
As each layer in the LLM gets calculated in the prompt machine, the KV cache corresponding to that layer is also generated.
|
||||
At the end of each layer, we trigger an asynchronous transfer of the KV-cache for that layer while the prompt computation continues to the next layer.
|
||||
Figure 11(b) shows this asynchronous transfer which reduces the transfer overheads.
|
||||
Layer-wise transfer also enables other optimizations, such as earlier start of the token phase in the token machines, as well as earlier release of KV-cache memory on the prompt machines.
|
||||
|
||||
Layer-wise KV-cache transfer happens in parallel with the prompt computation for the next layer.
|
||||
This requires fine-grained synchronization per layer for correctness.
|
||||
Thus, it is possible to incur performance interference and increase the TTFT, especially for smaller prompts.
|
||||
However, for small prompts the total KV-cache size is small and does not need the layer-wise transfer to hide the latency.
|
||||
Since the number of tokens in a batch is already known at the start of computation, Splitwise picks the best technique for KV-cache transfer.
|
||||
It uses serialized KV-cache transfer for smaller prompts and layer-wise transfer and for larger prompts.
|
||||
We show that the overall transfer and interference overheads are relatively small in Section VI-A.
|
||||
|
||||
TABLE V: Evaluated Splitwise designs all normalized to DGX-A100
|
||||
|
||||
### IV-D Provisioning with Splitwise
|
||||
|
||||
We leverage Splitwise to optimize LLM inference cluster deployments for power, cost, and throughput.
|
||||
|
||||
Type of machines.
|
||||
We propose four main variants of Splitwise-based systems:
|
||||
_Splitwise-AA_,
|
||||
_Splitwise-HH_,
|
||||
_Splitwise-HA_,
|
||||
and _Splitwise-HHcap_.
|
||||
The nomenclature is simply drawn from the first letter representing the Prompt machine type, and the second letter representing the Token machine type.
|
||||
“A” represents a DGX-A100 machine, “H” represents a DGX-H100 machine,
|
||||
and “Hcap” represents a power-capped DGX-H100 machine.
|
||||
Table V shows a summary of the cost, power, and hardware in each of our evaluated systems.
|
||||
|
||||
Splitwise-AA uses DGX-A100 for both prompt and token pools, while Splitwise-HH uses DGX-H100 for both.
|
||||
These two variants represent the commonly available setups in providers where machines are homogeneous and interchangeable.
|
||||
|
||||
Splitwise-HA uses DGX-H100 for the prompt pool and DGX-A100 for the token pool.
|
||||
We choose this configuration based on Table IV, and the Insight VII (_i.e._, A100s can be more cost- and power-efficient for the token phase).
|
||||
|
||||
Splitwise-HHcap uses DGX-H100 machines for both prompt and token pools.
|
||||
However, we power cap the token machines down to 70% of their rated power, with each GPU capped by 50% of the power.
|
||||
We propose this design based on Figure 9 and Insight VII (_i.e._, the prompts phase is impacted by power caps while token has no performance impact with 50% lower power cap per GPU).
|
||||
|
||||
Number of machines.
|
||||
The LLM inference cluster deployment must be sized with the appropriate number of prompt and token machines.
|
||||
Our methodology involves searching the design space using our event-driven cluster simulator, which is described in detail in Section V.
|
||||
We need to provide as input:
|
||||
(1) the target cluster design (_e.g._, Splitwise-HA or Splitwise-HHcap),
|
||||
(2) an LLM-specific performance model that can estimate the TTFT and TBT at various input, output, and batch sizes,
|
||||
(3) a short trace derived from the target prompt and token size distributions for the service (_e.g._, Figure 3),
|
||||
(4) the SLOs (_e.g._, Table VI),
|
||||
(5) the constraints (_e.g._, throughput),
|
||||
and (6) the optimization goal (_e.g._, minimize cost).
|
||||
Using this information, our provisioning framework searches the space for the desired optimal point.
|
||||
For example, searching with a throughput constraint and a cost minimization goal gives us iso-throughput cost-optimized clusters across different designs.
|
||||
|
||||
|
||||
|
||||
Figure 12: Design space for provisioning a Splitwise-HH cluster.
|
||||
Cluster configurations targets a peak throughput of 70 RPS.
|
||||
The cost-optimal Splitwise-HH configuration is marked with ⋆⋆⋆ (27 prompt and 3 token machines).
|
||||
|
||||
Search space.
|
||||
Figure 12 shows an example of the two-dimensional search space for the number of prompt and token machines under Splitwise-HH for the coding workload (using a 2-minute trace).
|
||||
The simulator outputs the various percentiles for TTFT, TBT, and E2E latencies.
|
||||
Then, we select the clusters that meet the SLOs for each of these metrics and optimize our target function.
|
||||
For example, Figure 12 shows a ⋆⋆⋆ for the setup with 27 prompt and 3 token machines with the lowest cost that achieves 70 RPS.
|
||||
We call this setup _iso-throughput cost-optimized_.
|
||||
|
||||
Optimization.
|
||||
We can use three optimization goals:
|
||||
_throughput_, _cost_, and _power_.
|
||||
Throughput optimization is important for both, the cloud service provider (CSP) and the user.
|
||||
Cost optimization has different importance levels to the CSP and the user.
|
||||
For the CSP, a higher cost for the same throughput might be acceptable if there are gains in power and space requirements for the cluster.
|
||||
However, for the end-user, a higher cost at the same throughput is generally unacceptable.
|
||||
Finally, power optimization is attractive for a CSP, since it enables more GPUs to be deployed in the same datacenter [62, 63], but it may not be as important to the user.
|
||||
We only consider the provisioned power, and not the dynamic power utilization, in our study.
|
||||
|
||||
### IV-E Practical Considerations
|
||||
|
||||
Accuracy impact.
|
||||
Splitwise does not impact accuracy since it uses lossless KV-cache transfer and does not add any randomization.
|
||||
It executes inference with the same parameters and state as on a single machine.
|
||||
|
||||
Scalability.
|
||||
Since LLM requests are much longer than typical ML requests [37, 38], they incur lower scheduling overhead for similar cluster sizes.
|
||||
However, the CLS may become a scalability bottleneck for large clusters.
|
||||
Insights from prior work on partitioned or replicated scheduling could help improve scalability [61, 27, 72] and are orthogonal to Splitwise.
|
||||
|
||||
Reliability and fault tolerance.
|
||||
If the prompt or the token machine fail, Splitwise simply restarts requests from scratch, similar to today’s LLM serving systems [51, 44].
|
||||
Alternatively, Splitwise could checkpoint the KV-cache generated after prompt computation into an in-memory database.
|
||||
To recover, Splitwise can use this cache to skip prompt recomputation, and start right away with the token phase.
|
||||
The KV-cache could also be checkpointed periodically during the token phase.
|
||||
Designing safe and efficient failure recovery is out of scope for our paper.
|
||||
|
||||
## V Methodology
|
||||
|
||||
### V-A Experimental setup
|
||||
|
||||
To evaluate our proposal on real hardware, we implement Splitwise’s KV-cache transfer mechanism on top of vLLM [51]. Our implementation is open source [1].
|
||||
We run this modified vLLM on two DGX-A100 and two DGX-H10 virtual machines (VMs) on Microsoft Azure with specifications from Table I.
|
||||
These are the VMs used to collect the characterization data in Section III.
|
||||
These machines are connected with InfiniBand and the DGX-H100s have double the bandwidth (_i.e._, 400 Gbps).
|
||||
|
||||
Since vanilla vLLM only supports continuous batching with token preemption which can lead to much higher TBT, we implement state-of-the-art mixed continuous batching [81] as discussed earlier in Figure 2(c).
|
||||
|
||||
Our implementation of the Splitwise technique assigns machines either a prompt role, or a token role.
|
||||
As the prompt machine generates the first token, it transfers the KV-cache to the token machine using the technique described in Section IV-C.
|
||||
We use MSCCL++ [11], an optimized GPU-driven communication library, to implement the naive and layer-wise KV cache transfers.
|
||||
|
||||
In our implementation, the prompt machine uses the zero-copy one-sided put primitive of MSCCL++ to send KV-cache data over InfiniBand as soon as it is ready, without requiring the token machine to issue any receive instructions.
|
||||
Once we have issued a put for all layers, the prompt machine signals a semaphore that the token machine waits on.
|
||||
The synchronization done with the help of semaphores uses the same InfiniBand connection used to send KV-cache data.
|
||||
When processing a batch of prompts, each request is assigned a different semaphore since it may be routed to different token machines.
|
||||
We ship the KV-caches block-by-block in vLLM.
|
||||
To minimize the number of transfers, we also consider the contiguity of KV blocks as long as they use the same semaphore.
|
||||
|
||||
### V-B Simulator setup
|
||||
|
||||
We build a simulator to explore cluster designs and evaluate Splitwise at scale.
|
||||
The simulator code is open source [20].
|
||||
|
||||
|
||||
|
||||
Figure 13: Overview of the design of the Splitwise simulator.
|
||||
|
||||
Figure 13 shows the design of our simulator.
|
||||
The simulator is event-driven and faithfully models the Splitwise machine pools, schedulers, machine-level memory and queues, and KV-cache transfer.
|
||||
We first profile the LLM on the target hardware with various input/output sizes .
|
||||
Based on the characterization profiles, we build a performance model.
|
||||
The simulator takes as input the request traces, SLOs, the performance model, and the configurations for cluster and scheduler .
|
||||
For our evaluation, we use the prompt and token size distributions from the production traces in Section III.
|
||||
We tune the Poisson arrival rate to increase and decrease the load (requests per second) for cluster sizing.
|
||||
The simulator provides the achieved metrics per request (TTFT, TBT, E2E), and the machine utilization levels .
|
||||
We cross-validated the performance model with hardware experiments to ensure accuracy; we also validated the simulator end-to-end using production load with over 50K iterations to ensure fidelity .
|
||||
|
||||
Performance model.
|
||||
We build a piece-wise linear performance model using performance profiles at various batch sizes, input sizes, output sizes, in the required parallelism configuration on A100 and H100 machines from Section III.
|
||||
We validate that our performance model has high accuracy; it incurs a mean absolute percentage error (MAPE) of less than 3% when evaluated with a 80:20 train:test dataset split.
|
||||
|
||||
Communication model.
|
||||
In our evaluation, KV-cache transfers cause inter-machine communication, whereas tensor parallelism only causes intra-machine communication.
|
||||
We model inter-machine communication overheads by benchmarking our KV-cache transfer implementation over Infiniband in Section VI-A.
|
||||
|
||||
SLOs.
|
||||
To determine the maximum throughput that can be supported by a given cluster design, we use P50, P90, and P99 SLOs for TTFT, TBT, and E2E latency metrics.
|
||||
Table VI shows our SLO definition using DGX-A100 as a reference.
|
||||
We require all nine SLOs to be met.
|
||||
SLOs on TTFT are slightly looser, since it has a much smaller impact on the E2E latency.
|
||||
|
||||
TABLE VI: SLO expressed as slowdown compared to a request running on DGX-A100 under no contention.
|
||||
|
||||
Baselines.
|
||||
We compare our Splitwise designs against Baseline-A100 and Baseline-H100.
|
||||
The clusters in these baselines consist of just DGX-A100s and DGX-H100s, respectively.
|
||||
Both baselines use the same mixed continuous batching that Splitwise uses for mixed pool machines (described in Section IV-A).
|
||||
|
||||
## VI Evaluation
|
||||
|
||||
### VI-A Experimental results
|
||||
|
||||
KV-cache transfer latency.
|
||||
We first measure the latency to transfer the KV-cache as the prompt size grows.
|
||||
Figure 14 shows the visible transfer latency on both A100 and H100 setups with the naive and optimized transfer design as discussed in Figure 11.
|
||||
Compared to the prompt computation time, the overhead is minimal (<7%7<7\%< 7 %).
|
||||
The time for serialized transfers linearly increases with the prompt size since the size of the KV-cache also increases.
|
||||
The optimized per-layer transfer, on the other hand, hides much of the latency.
|
||||
For these transfers, we see a constant non-overlapped transfer time of around 8ms for the A100 and around 5ms for the H100 setup.
|
||||
The H100 setup has double the bandwidth of the A100 setup (_i.e._, 200 vs 400 Gbps), and the impact of this can be clearly seen with transfers in the H100 setup happening about twice as fast as those in the A100 setup.
|
||||
|
||||
As discussed in Section IV-C, for small prompt sizes (<512absent512<512< 512 in H100), Splitwise uses the serialized KV-cache transfer and for larger prompts, it uses per-layer transfers.
|
||||
|
||||
|
||||
|
||||
Figure 14: Overhead of the KV-cache transfer as the prompt size increases on A100s and H100s.
|
||||
|
||||
End-to-end impact.
|
||||
Next, we run the coding trace on the 2-machine Splitwise setups without batching, and compare the observed latency metrics to a 1-machine baseline setup with no batching.
|
||||
Figure 15 shows our results.
|
||||
The latency impact of serially transferring the KV-cache grows up to 3% of the E2E with large prompts.
|
||||
However, Splitwise only incurs 0.8% of E2E.
|
||||
In a user-facing inference, the only visible impact of KV-cache transfer overhead is the latency for the second token.
|
||||
Splitwise adds a 16.5% latency to the second token, as compared to the 64% overhead from a serialized transfer.
|
||||
Overall, the transfer impact in Splitwise is hardly perceivable even in a user-facing inference.
|
||||
|
||||
|
||||
|
||||
Figure 15: Overhead of KV cache transfer on TTFT, E2E latency for coding trace for A100 and H100.
|
||||
|
||||
### VI-B Iso-power throughput-optimized clusters
|
||||
|
||||
Cluster provisioning.
|
||||
We provision clusters using the methodology described in Section IV-D.
|
||||
We target a specific workload (_e.g._, conversation) at a peak load with the same power (_i.e._, iso-power) for each cluster design.
|
||||
For the baseline, we use the power for 40 DGX-H100 machines as our target peak power.
|
||||
For the A100 baseline, we can fit 70 DGX-A100 machines under the same power budget.
|
||||
We denote these two designs as 40P/T and 70P/T respectively, since they both use mixed batching in all machines.
|
||||
|
||||
For Splitwise cluster designs under the coding trace, Splitwise-AA provisions 55 prompt machines and 15 for the token pool, denoted as (55P, 15P).
|
||||
Note that like Baseline-A100, Splitwise-AA also provisions 75% more machines than Baseline-H100.
|
||||
The legends in Figure 16 show the different provisioning choices under coding and conversation workloads.
|
||||
Request size distributions reflect in the machine pool sizing.
|
||||
For example, we provision
|
||||
|
||||
[18].
|
||||
However, in the future, services may have enough GPU capacity to cache the context and avoid recomputation.
|
||||
This could sway the memory utilization pattern of the prompt phase from our characterization.
|
||||
Furthermore, it may require transferring the KV-cache back to a prompt machine to be ready for the next conversation request.
|
||||
|
||||
## VIII Related Work
|
||||
|
||||
Heterogeneous scheduling and dataflow systems.
|
||||
Prior work has studied heterogeneous scheduling for a variety of interactive services [83, 65, 68].
|
||||
These works exploit hardware heterogeneity to strike a balance between different objectives such as cost, energy, and performance.
|
||||
However, they run the entire workload on the same machine.
|
||||
Research on heterogeneous multiprocessor CPU scheduling attempts to match workload heterogeneity to hardware heterogeneity [40, 76, 41, 50, 80, 29].
|
||||
These works use profiling or online monitoring with metrics like request length or hardware performance counters to identify workload phases and allocate them appropriately on heterogeneous processors.
|
||||
However, they do not consider the complexities with batching.
|
||||
Distributed dataflow systems orchestrate large-scale computational graphs and aim to provide general-purpose programmability [34, 46, 75, 82].
|
||||
LLM inference under Splitwise can be viewed as a static computational graph with two stages, so it could be implemented using distributed frameworks that provide efficient GPU abstractions [59].
|
||||
Splitwise differs from these works since it uses a spe
|
||||
@@ -0,0 +1,93 @@
|
||||
# 📊 文章摘要：Splitwise: Efficient Generative LLM Inference Using Phase Splitting
|
||||
|
||||
> **原文**：[2023-11-30_Splitwise.md](./2023-11-30_Splitwise.md)
|
||||
> **原文链接**：https://arxiv.org/abs/2311.18677
|
||||
> **来源**：arXiv
|
||||
> **作者**：Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini（华盛顿大学、微软）
|
||||
> **发布日期**：2023-11-30
|
||||
> **摘要日期**：2026-08-06
|
||||
> **价值评级**：⭐⭐⭐ 高
|
||||
|
||||
---
|
||||
|
||||
## 核心命题
|
||||
|
||||
> **相位拆分** — LLM 推理的两阶段（计算密集的 prompt 处理、内存密集的 token 生成）对硬件的需求截然不同，把它们拆分到各自合适的机器（含异构、降配硬件），是突破"GPU 算力与内存带宽增长失衡"约束的成本与功耗优化路径。
|
||||
|
||||
---
|
||||
|
||||
## 文章概要
|
||||
|
||||
本文用生产 trace 对 A100/H100 上的 LLM 推理做表征分析，发现同一请求的两个阶段特性对立：prompt 计算阶段吃算力（FLOPs），token 生成阶段受内存带宽与容量约束、即使最优批处理也浪费算力。由此提出 Splitwise：把两阶段拆分到不同机器，用层级异步传输把 KV cache 的迁移开销与 prompt 计算重叠（小 prompt 走串行传输），并利用生产 trace + 事件驱动模拟器探索同构/异构集群设计（AA、HH、HA、HHcap 四类）。相比现有集群，Splitwise 可在成本降低 20% 的同时提升 1.4× 吞吐，或在同等成本与功耗预算下提供 2.35× 吞吐。局限：KV cache 传输是固有开销、依赖数据中心高速互连（InfiniBand/NVLink），集中式调度器在超大集群下可能成为扩展性瓶颈。
|
||||
|
||||
---
|
||||
|
||||
## 关键要点
|
||||
|
||||
1. **硬件失衡是动因** — H100 相对 A100 算力提升 3.43×、功耗提升 1.75×，但内存带宽仅增长 1.6×、容量零增长；最新 GPU 的算力优势在内存密集的 token 生成阶段被浪费。`[分类: 共识]`
|
||||
2. **两阶段资源画像对立** — prompt 阶段计算密集、受 FLOPs 约束；token 生成阶段内存带宽/容量受限，即使 state-of-the-art 批处理也显著低效利用算力，可降配硬件运行。`[分类: 范式突破]`
|
||||
3. **拆分而非解耦竞争** — 与 DistServe 同期提出阶段拆分思路，但出发点不同：DistServe 追求 goodput/SLO，Splitwise 强调异构硬件选型与成本/功耗优化（Perf/$ 与 Perf/W）。`[分类: 共识]`
|
||||
4. **KV cache 传输的工程化优化** — 逐层异步传输与 prompt 计算重叠，使 E2E 延迟影响仅 0.8%（串行传输为 3%）；第二 token 延迟增加 16.5%（串行方案 64%），用户几乎无感知；传输总开销 <7% 的 prompt 计算时间。`[分类: 范式突破]`
|
||||
5. **异构集群设计矩阵** — 四种设计：AA（A100 全同构）、HH（H100 全同构）、HA（H100 跑 prompt + A100 跑 token，token 阶段 A100 更划算）、HHcap（H100 双池但 token 机功耗上限 70%、单 GPU 限 50% 功耗——token 阶段对功耗限制不敏感）。`[分类: 范式突破]`
|
||||
6. **量化收益** — 同等功耗预算下：40 台 H100 baseline 被 55 prompt + 15 token 的 Splitwise-AA 方案超越；相比现有集群吞吐提升 1.4× 且成本降 20%，同成本同功耗下吞吐可达 2.35×。`[分类: 共识]`
|
||||
7. **模拟驱动的集群供给** — 事件驱动模拟器 + 分段线性性能模型（MAPE < 3%，与真实硬件实验交叉验证，端到端 5 万+ 迭代验证），在 TTFT/TBT/E2E 九个 SLO 约束下搜索异构机群配比（如 70 RPS 目标的成本最优配置为 27 prompt + 3 token 机器）。`[分类: 未探索]`
|
||||
8. **可靠性以重启为代价** — 节点故障时从头重启请求；论文提出可将 KV cache 检查点化到内存数据库以跳过 prompt 重算，但安全高效的故障恢复留作未来工作。`[分类: 未探索]`
|
||||
|
||||
---
|
||||
|
||||
## 批判性分析
|
||||
|
||||
### 假设前提
|
||||
|
||||
- GPU 集群存在（或未来将出现）供异构部署的多种机型，且异构机型间的价格/功耗/可用性差异足以驱动拆分决策。
|
||||
- 数据中心普遍具备高速后端互连（InfiniBand 等），KV cache 跨机传输可被有效隐藏。
|
||||
- token 生成阶段使用上一代或降配硬件不影响 SLO 达标（对交互式负载尤其依赖此假设）。
|
||||
- 生产请求的 prompt/token 长度分布可表征，且集群按峰值负载供给（论文只考虑供给功耗而非动态功耗）。
|
||||
|
||||
### 论据与逻辑
|
||||
|
||||
- 论据扎实：表征数据来自真实硬件（Azure 上的 DGX-A100/H100）+ 生产 trace；KV cache 传输开销有端到端实测（0.8% E2E、16.5% 第二 token）；模拟器性能模型 MAPE <3% 且经 5 万+ 迭代端到端验证，结论的量化支撑较强。
|
||||
- 逻辑链条完整：硬件失衡观测 → 阶段资源画像差异 → 拆分设计 → 传输优化 → 异构供给搜索 → 收益量化。
|
||||
- 弱点：1.4×/2.35× 等收益来自模拟器而非全系统真实部署；"第二 token 延迟"这类交互体验指标的受众感知评估主观；对大规模集群下集中式调度器（CLS）瓶颈仅以"正交于 Splitwise"带过，未量化。
|
||||
|
||||
### 边界与局限
|
||||
|
||||
- 结论适用于两阶段分离收益大于传输开销的场景：batch 处理类任务（摘要等）收益最大，交互式任务受第二 token 延迟影响。
|
||||
- 前提是高速互连数据中心；无 InfiniBand/NVLink 级带宽的环境下拆分收益会显著缩水。
|
||||
- 未来若 GPU 缓存技术（如长上下文缓存避免重算）改变 prompt 阶段的内存画像，表征结论可能失效（论文自身也承认此点）。
|
||||
- 容错、调度器扩展性、动态功耗等生产关键问题未解决。
|
||||
|
||||
---
|
||||
|
||||
## 可引用金句
|
||||
|
||||
> "Unlike prompt computation, token generation does not need the compute capability of the latest GPUs and can be run with lower power and cost."
|
||||
> （与 prompt 计算不同，token 生成并不需要最新 GPU 的计算能力，可以用更低的功耗与成本运行。）
|
||||
|
||||
> "Running both phases on the same machine often leads to inconsistent end-to-end latencies due to the arbitrary batching of prompt and token phases."
|
||||
> （在同一台机器上运行两个阶段，常因 prompt 与 token 阶段的随意混合批处理而导致端到端延迟不稳定。）
|
||||
|
||||
---
|
||||
|
||||
## 总体评价
|
||||
|
||||
**亮点**：
|
||||
- 系统化表征了 LLM 推理两阶段的硬件资源画像差异，数据一手且翔实（A100/H100 实测）
|
||||
- "异构降配 + 功耗上限"的集群设计思路（HA、HHcap）直接把成本/功耗优化落到集群拓扑层面，工程可操作性强
|
||||
- KV cache 逐层异步传输的工程方案精炼，与 Mooncake 的逐层 prefill 形成呼应
|
||||
- 模拟器 + 性能模型方法论严谨（MAPE <3%），可复用于集群供给规划
|
||||
|
||||
**不足**：
|
||||
- 收益数字主要来自模拟器评估，缺大规模真实部署验证
|
||||
- 对调度器扩展性、容错等生产约束讨论偏简略
|
||||
- 未考虑动态功耗、缓存普及对未来硬件画像的影响
|
||||
|
||||
**适用场景**：云厂商推理集群规划者、追求成本/功耗优化的 serving 基础设施团队；研究异构调度与数据中心级 LLM 部署的研究人员。
|
||||
|
||||
**关联建议**：与 DistServe（goodput 视角的阶段解耦）、TetriInfer（混合负载干扰）、Mooncake（生产系统 KVCache 中心化）对照，可形成"解耦 serving"的完整谱系；后续可关注微软相关后续工作与 vLLM 解耦功能的演进。
|
||||
|
||||
---
|
||||
|
||||
## 配图
|
||||
|
||||
|
||||
@@ -0,0 +1,276 @@
|
||||
# Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
|
||||
|
||||
> **来源**：arXiv
|
||||
> **作者**：Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu
|
||||
> **发布日期**：2024-06-24
|
||||
> **原文链接**：https://arxiv.org/abs/2407.00079
|
||||
|
||||
---
|
||||
|
||||
## 论文元数据
|
||||
|
||||
- **arXiv ID**：2407.00079
|
||||
- **学科分类**：Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Hardware Architecture (cs.AR)
|
||||
- **作者机构**：月之暗面（Moonshot AI）、清华大学
|
||||
- **提交历史**：v1: 2024-06-24；v2: 2024-07-02；v3: 2024-07-09；v4: 2025-09-03
|
||||
- **DOI**：https://doi.org/10.48550/arXiv.2407.00079
|
||||
|
||||
---
|
||||
|
||||
11脚注: Ruoyu Qin’s part of work done as an intern at Moonshot AI, contributed equally with Zheming Li.22脚注: Corresponding to zhang_mingxing@mail.tsinghua.edu.cn, xuxinran@moonshot.ai.
|
||||
|
||||
Ruoyu Qin♠♡1 Zheming Li♠1 Weiran He♠
|
||||
|
||||
&Mingxing Zhang♡2 Yongwei Wu♡ Weimin Zheng♡ Xinran Xu♠2
|
||||
|
||||
♠Moonshot AI ♡Tsinghua University
|
||||
|
||||
## 摘要（Abstract）
|
||||
|
||||
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI. It features a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters. It also leverages the underutilized CPU, DRAM, and SSD resources of the GPU cluster to implement a disaggregated cache of KVCache. The core of Mooncake is its KVCache-centric scheduler, which balances maximizing overall effective throughput while meeting latency-related Service Level Objectives (SLOs). Unlike traditional studies that assume all requests will be processed, Mooncake faces challenges due to highly overloaded scenarios. To mitigate these, we developed a prediction-based early rejection policy. Experiments show that Mooncake excels in long-context scenarios. Compared to the baseline method, Mooncake can achieve up to a 525% increase in throughput in certain simulated scenarios while adhering to SLOs. Under real workloads, Mooncake’s innovative architecture enables Kimi to handle 75% more requests.
|
||||
|
||||
## 1 引言（Introduction）
|
||||
|
||||
### 1.1 Motivation of Developing Mooncacke
|
||||
|
||||
With the rapid adoption of large language models (LLMs) in various scenarios [1, 2, 3, 4], the workloads for LLM serving have become significantly diversified. These workloads differ in input/output length, frequency and distribution of arrival, and, most importantly, demand different kinds of Service Level Objectives (SLOs). As a Model as a Service (MaaS) provider, one of the primary goals of Kimi [5] is to solve an optimization problem with multiple complex constraints. The optimization goal is to maximize overall effective throughput, which directly impacts revenue, while the constraints reflect varying levels of SLOs. These SLOs typically involve meeting latency-related requirements, mainly the time to first token (TTFT) and the time between tokens (TBT).
|
||||
|
||||
|
||||
|
||||
Figure 1: Mooncake Architecture.
|
||||
|
||||
To achieve this goal, a prerequisite is to make the best use of the various kinds of resources available in the GPU cluster.
|
||||
Specifically, although GPU servers are currently provided as highly integrated nodes (e.g., DGX/HGX supercomputers [6]), it is necessary to decouple and restructure them into several disaggregated resource pools, each optimized for different but collaborative goals. For example, many researchers [7, 8, 9] have suggested separating prefill servers from decoding servers because these two stages of LLM serving have very different computational characteristics, in which the KVCache shifts with requests moving from prefill to decoding servers.
|
||||
|
||||
Building on this idea, we found that the scheduling of KVCache is central to LLM serving scheduling. To improve overall throughput, there are typically two general approaches: 1) reuse KVCache as much as possible to reduce the required computation resources; and 2) maximize the number of tokens in each batch to improve the Model FLOPs Utilization (MFU). However, reusing KVCache from a remote location will prolong the TTFT, and a large batch size will lead to a larger TBT. Thus, the utilization of both these throughput-oriented optimizations may lead to violations of latency-related SLOs.
|
||||
|
||||
According to the above guidelines, we propose a disaggregated design that is centered around KVCache for scheduling and optimization. Figure 1 presents our current KVCache-centric disaggregated architecture for LLM serving, named Mooncake. For each request, the global scheduler (Conductor) needs to select a pair of prefill and decoding instances and schedule the request in the following steps: 1) transfer as much reusable KVCache as possible to the selected prefill instance; 2) complete the prefill stage in chunks/layers and continuously stream the output KVCache to the corresponding decoding instance; 3) load the KVCache and add the request to the continuous batching process at the decoding instance for generating request outputs.
|
||||
|
||||
Although this process seems straightforward, the selection policy is complex due to many restrictions. In the prefill stage, the main objective is to reuse the KVCache as much as possible to avoid redundant computation. However, waiting for KVCache stored on lower-tier storage may violate the TTFT SLO. Additionally, high demand on the KVCache server can lead to network congestion, prolonging the waiting time.
|
||||
Thus Conductor is also responsible for predicting the future usage of KVCache blocks and executing scheduling operations such as swapping and replication accordingly.
|
||||
The hottest blocks should be replicated to multiple nodes to avoid fetching congestion, while the coldest ones should be swapped out to reduce reserving costs.
|
||||
Prefill scheduling is also constrained by the availability of DRAM space in the prefill node, especially when much of the memory is reserved for the global KVCache pool.
|
||||
|
||||
In contrast, the decoding stage has different optimization goals and constraints. The aim is to aggregate as many tokens as possible in a decoding batch to improve MFU.
|
||||
However, this objective is restricted not only by the TBT SLO but also by the total size of the aggregated KVCache that can be contained in the VRAM.
|
||||
|
||||
More importantly, existing research on LLM serving assumes sufficient resources and focuses on improving resource utilization. In contrast, the current GPU/accelerator supply is limited, and many MaaS providers face severe overload problems, especially during peak times. Scheduling in such scenarios presents unique challenges that existing works have not explored. For example, we need to predict future loads and reject certain requests early if there will be no available decoding slots after the prefill stage, to save wasted computation resources.
|
||||
However, a straightforward implementation of such an early reject policy surprisingly leads to fluctuations in the overloads. This has led us to aim at predicting the generation length of specific queries and making overall load predictions in the short-term future to implement a better rejection policy. It is also necessary to classify different request priorities to implement priority-based scheduling.
|
||||
In this paper, we summarize these problems as overload-oriented scheduling and present our preliminary study results.
|
||||
|
||||
### 1.2 Design and Results of Mooncacke
|
||||
|
||||
In the following sections of this paper, we first present an overview of Mooncake’s architecture, including its main components and the typical workflow for processing a request (§3). Then, we describe the main design choices made during its implementation, especially those not covered in current research.
|
||||
|
||||
First, in §5, we discuss how to implement a separate prefill node pool that seamlessly handles the dynamic distribution of context length. We employ a chunked pipeline parallelism (CPP) mechanism to scale the processing of a single request across multiple nodes, which is necessary for reducing the TTFT of long-context inputs. Compared to traditional sequence parallelism (SP) based solutions, CPP reduces network consumption and simplifies the reliance on frequent elastic scaling. This mechanism is further supplemented with layer-wise prefill that enables stream transferring of KVCache to overlap latency.
|
||||
|
||||
Next, in §6, we detail our KVCache-centric request scheduling algorithm, which balances instance loads and user experience as measured by TTFT and TBT SLOs. This includes a heuristic-based automated hot-spot migration scheme that replicates hot KVCache blocks without requiring precise predictions of future KVCache usage. Experimental results show that our cache-aware scheduling can significantly lower TTFT in real-world scenarios. In end-to-end experiments using public datasets, simulated data, and real workloads, Mooncake excels in long-context scenarios. Compared to the baseline method, Mooncake can achieve up to a 525% increase in throughput while meeting SLOs. Under real workloads, Mooncake enables Kimi to handle 75% more requests.
|
||||
|
||||
Finally, unlike existing work on LLM serving that assumes all requests will be processed, Mooncake consistently faces overload due to Kimi’s rapid growth in user requests. Thus, Mooncake’s scheduling involves determining whether to accept or reject incoming requests based on the system load. In §7, we discuss our implementation of a unique early rejection policy that reduces wasted computational resources in overloaded scenarios. We further explore the load fluctuation problem caused by straightforward early rejection and how predicting future load can mitigate this issue.
|
||||
|
||||
Mooncake is currently the primary platform for serving Kimi and has successfully handled exponential workload growth, proving its effectiveness in scaling out to large and highly overloaded workloads. However, many more problems need to be explored, and these future directions are also included in the paper.
|
||||
|
||||
To protect proprietary information and facilitate reproducibility, all the experimental results reported in this paper are based on replayed traces of real workloads, but using a dummy model that follows the same architecture as LLaMA2-70B. The trace includes only the timing of request arrivals, the number of input tokens, and the number of output tokens, the remaped block hash, without any real user content. The trace is open-sourced at https://github.com/kvcache-ai/Mooncake.
|
||||
|
||||
## 2 Preliminary and Problem Definition
|
||||
|
||||
Modern large language models (LLMs) are based on the Transformer architecture, which utilizes attention mechanisms and multilayer perceptrons (MLPs) to process input. Popular Transformer-based models, such as GPT [10] and LLaMA [11], employ a decoder-only structure. Each inference request is logically divided into two stages: the prefill stage and the decoding stage.
|
||||
|
||||
|
||||
|
||||
Figure 2: Normalized throughput and latency of prefill and decoding stages with different sequence lengths or batch sizes for the dummy
|
||||
LLaMA2-70B model.
|
||||
|
||||
In the prefill stage, all input tokens are processed in parallel. This stage generates the first output token while storing intermediate results of computed keys and values, referred to as the KVCache. The decoding stage then uses this KVCache to autoregressively generate new tokens, adding new keys and values from the computation to the KVCache. The ability to process input tokens simultaneously in the prefill stage typically makes it computationally intensive, except for short requests. Since the computational complexity of attention networks scales quadratically with input length while the complexity of MLP scales linearly, computation time in the prefill stage generally increases superlinearly with input length, as shown in the left part of Figure 2.
|
||||
|
||||
In contrast, the decoding stage processes only one token at a time per batch due to the limitation of autoregressive generation. This makes it memory-constrained and causes computation time to increase sublinearly with batch size, as shown in the right part of Figure 2. A widely used optimization in the decoding stage is continuous batching [12, 13]. Before each iteration, the scheduler checks the status of all requests, adding newly arrived requests to the batch’s prefill stage while
|
||||
|
||||
ng contexts, bringing no significant overhead for short context prefill and avoiding frequent dynamic adjustment of node partitioning.
|
||||
This pipeline-based acceleration method has been explored in training systems [24], but to our knowledge, this is the first application in the inference stage, as long context inference has only recently emerged.
|
||||
|
||||
### 5.2 Layer-wise Prefill
|
||||
|
||||
Beyond computational power, the limited size of VRAM is also a precious resource, and we aim to minimize the VRAM occupation by states, primarily the KVCache.
|
||||
Theoretically, if the KVCache size of a request is SS and the processing time is TT, its occupation cost is S∗TS*T.
|
||||
If a request is chunked and the processing of each chunk is inlined with other decoding requests in chunked prefill, TT will increase, leading to a larger occupation cost.
|
||||
|
||||
|
||||
|
||||
Figure 7: Latency of storing KVCache of different request lengths (Layer-wise latency refers to the difference in latency between Layer-wise Prefill and Prefill without storing KVCache).
|
||||
|
||||
Moreover, since prefill is processed layer-by-layer and is computation-bound, it is possible to overlap the transferring and dumping of KVCache with computation, further reducing its occupation cost.
|
||||
In Mooncake, KVCache loading and storing are executed asynchronously via launch and wait operations. Before each layer’s attention computation begins, the model waits for the asynchronous loading of that layer’s KVCache to complete and triggers the next layer’s asynchronous KVCache loading. After the attention calculation is complete, asynchronous storage of that layer’s KVCache is launched. Once all layers’ computations are finished, the process waits for the completion of all asynchronous storage operations. Transfer overlapping allows the prefill instance’s execution time to be roughly equivalent to either the KVCache loading time or the standard prefilling time, depending on the prefix cache proportion relative to the input length. The experimental result of KVCache storing latency, as shown in Figure 7, demonstrates that the layer-wise prefill can effectively reduce the latency for long-context requests.
|
||||
|
||||
The main advantage of this overlap effectiveness is that it enables us to disregard the available VRAM size in prefill scheduling, as long as it can contain a single request.
|
||||
As shown in Figure 1, the scheduling of prefill nodes only considers the KVCache distribution and the available DRAM size.
|
||||
|
||||
In the future, we intend to explore more uses for this free VRAM. For example, OpenAI recently proposed the use of batch APIs [25], which enable users to send asynchronous groups of requests at 50% lower costs, but with only a clear 24-hour turnaround time. This service is ideal for processing jobs that do not require immediate responses. Since there is no stringent TBT for these batch requests, we can inline even the decoding stage of these requests into prefill processing for better MFU, if there is enough VRAM space to hold the corresponding KVCache.
|
||||
|
||||
## 6 以 KV Cache 为中心的调度
|
||||
|
||||
In this section, we mainly discuss how Conductor schedules the requests and KVCache blocks under normal conditions, leaving the discussion on overload scenarios for the next section.
|
||||
|
||||
Algorithm 1 KVCache-centric Scheduling Algorithm
|
||||
|
||||
1:prefill instance pool PP, decoding instance pool DD, request RR, cache block size BB.
|
||||
|
||||
2:the prefill and decoding instances (p,d)(p,d) to process RR.
|
||||
|
||||
3:𝑏𝑙𝑜𝑐𝑘_𝑘𝑒𝑦𝑠←PrefixHash(R.𝑝𝑟𝑜𝑚𝑝𝑡_𝑡𝑜𝑘𝑒𝑛𝑠,B){block\_keys}{PrefixHash}(R.{prompt\_tokens},B)
|
||||
|
||||
4:𝑇𝑇𝐹𝑇←inf{TTFT}
|
||||
|
||||
5:p←∅p
|
||||
|
||||
6:𝑏𝑒𝑠𝑡_𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛,𝑏𝑒𝑠𝑡_𝑚𝑎𝑡𝑐ℎ𝑒𝑑_𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒←FindBestPrefixMatch(P,𝑏𝑙𝑜𝑐𝑘_𝑘𝑒𝑦𝑠){best\_prefix\_len},{best\_matched\_instance}{FindBestPrefixMatch}(P,{block\_keys})
|
||||
|
||||
7:for 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒∈P{instance} P do
|
||||
|
||||
8: 𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛←𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒.𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛{prefix\_len}{instance.prefix\_len}
|
||||
|
||||
9: T𝑞𝑢𝑒𝑢𝑒←EstimatePrefillQueueTime(𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒){T_{queue}}{EstimatePrefillQueueTime}({instance})
|
||||
|
||||
10: if 𝑏𝑒𝑠𝑡_𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛<𝐤𝐯𝐜𝐚𝐜𝐡𝐞_𝐛𝐚𝐥𝐚𝐧𝐜𝐢𝐧𝐠_𝐭𝐡𝐫𝐞𝐬𝐡𝐨𝐥𝐝{{best\_prefix\_len}}{{prefix\_len}}<{ kvcache\_balancing\_threshold} then
|
||||
⊳ Cache-aware prefill scheduling
|
||||
|
||||
11: T𝑝𝑟𝑒𝑓𝑖𝑙𝑙←EstimatePrefillExecutionTime(len(R.𝑝𝑟𝑜𝑚𝑝𝑡_𝑡𝑜𝑘𝑒𝑛𝑠),𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛){T_{prefill}}{EstimatePrefillExecutionTime}({len}(R.{prompt\_tokens}),{prefix\_len})
|
||||
|
||||
12: if 𝑇𝑇𝐹𝑇>T𝑞𝑢𝑒𝑢𝑒+T𝑝𝑟𝑒𝑓𝑖𝑙𝑙{TTFT}>{T_{queue}}+{T_{prefill}} then
|
||||
|
||||
13: 𝑇𝑇𝐹𝑇←T𝑞𝑢𝑒𝑢𝑒+T𝑝𝑟𝑒𝑓𝑖𝑙𝑙{TTFT}{T_{queue}}+{T_{prefill}}
|
||||
|
||||
14: p←𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒p{instance}
|
||||
|
||||
15: end if
|
||||
|
||||
16: else⊳ Cache-aware and -balancing prefill scheduling
|
||||
|
||||
17: 𝑡𝑟𝑎𝑛𝑠𝑓𝑒𝑟_𝑙𝑒𝑛←𝑏𝑒𝑠𝑡_𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛−𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛{transfer\_len}{best\_prefix\_len}-{prefix\_len}
|
||||
|
||||
18: T𝑡𝑟𝑎𝑛𝑠𝑓𝑒𝑟←EstimateKVCacheTransferTime(𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒,𝑏𝑒𝑠𝑡_𝑚𝑎𝑡𝑐ℎ𝑒𝑑_𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒,𝑡𝑟𝑎𝑛𝑠𝑓𝑒𝑟_𝑙𝑒𝑛){T_{transfer}}{EstimateKVCacheTransferTime}({instance},{best\_matched\_instance},{transfer\_len})
|
||||
|
||||
19: T𝑝𝑟𝑒𝑓𝑖𝑙𝑙←EstimatePrefillExecutionTime(len(R.𝑝𝑟𝑜𝑚𝑝𝑡_𝑡𝑜𝑘𝑒𝑛𝑠),𝑏𝑒𝑠𝑡_𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛){T_{prefill}}{EstimatePrefillExecutionTime}({len}(R.{prompt\_tokens}),{best\_prefix\_len})
|
||||
|
||||
20: if 𝑇𝑇𝐹𝑇>T𝑡𝑟𝑎𝑛𝑠𝑓𝑒𝑟+T𝑞𝑢𝑒𝑢𝑒+T𝑝𝑟𝑒𝑓𝑖𝑙𝑙{TTFT}>{T_{transfer}}+{T_{queue}}+{T_{prefill}} then
|
||||
|
||||
21: 𝑇𝑇𝐹𝑇←T𝑡𝑟𝑎𝑛𝑠𝑓𝑒𝑟+T𝑞𝑢𝑒𝑢𝑒+T𝑝𝑟𝑒𝑓𝑖𝑙𝑙{TTFT}{T_{transfer}}+{T_{queue}}+{T_{prefill}}
|
||||
|
||||
22: p←𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒p{instance}
|
||||
|
||||
23: end if
|
||||
|
||||
24: end if
|
||||
|
||||
25:end for
|
||||
|
||||
26:d,𝑇𝐵𝑇←SelectDecodingInstance(D)d,{TBT}{SelectDecodingInstance}(D)
|
||||
⊳ Load-balancing decoding scheduling
|
||||
|
||||
27:if 𝑇𝑇𝐹𝑇>𝑇𝑇𝐹𝑇_𝑆𝐿𝑂{TTFT}>{TTFT\_SLO} or 𝑇𝐵𝑇>𝑇𝐵𝑇_𝑆𝐿𝑂{TBT}>{TBT\_SLO} then
|
||||
|
||||
28: reject RR; return
|
||||
|
||||
29:end if
|
||||
|
||||
30:if 𝑏𝑒𝑠𝑡_𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛p.𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛>𝐤𝐯𝐜𝐚𝐜𝐡𝐞_𝐛𝐚𝐥𝐚𝐧𝐜𝐢𝐧𝐠_𝐭𝐡𝐫𝐞𝐬𝐡𝐨𝐥𝐝{{best\_prefix\_len}}{p.{prefix\_len}}>{ kvcache\_balancing\_threshold} then
|
||||
|
||||
31: TransferKVCache(𝑏𝑒𝑠𝑡_𝑚𝑎𝑡𝑐ℎ𝑒𝑑_𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒,p){TransferKVCache}({best\_matched\_instance},p)
|
||||
⊳ KVCache hot-spot migration
|
||||
|
||||
32:end if
|
||||
|
||||
33:return (p,d)(p,d)
|
||||
|
||||
### 6.1 Prefill Global Scheduling
|
||||
|
||||
Previous research on LLM servi
|
||||
|
||||
ng typically uses a load-balancing strategy that evaluates the load on each instance based on the number of assigned requests. In Mooncake, however, the selection of prefill instances considers additional factors—not just load but also the prefix cache hit length and the distribution of reusable KVCache blocks. While there is a preference to route requests to prefill instances with longer prefix cache lengths to reduce computation costs, it may be beneficial to schedule them to other nodes to ensure overall system balance and meet TTFT SLOs. To address these complexities, we propose a cache-aware global scheduling algorithm that accounts for both the prefill time due to the prefix cache and the queuing time associated with the load on the instance.
|
||||
|
||||
Algorithm 1 details the mechanism for our cache-aware prefill scheduling. For every new request, its input tokens are divided into several blocks, and a hash key is computed for each block. This involves generating a hash key of tokens in a block concatenated with the hash key of the previous block (if available). The request’s block keys are then compared one by one against each prefill instance’s cache keys to identify the prefix match length (prefix_lenprefix\_len). Similar reuse logic is already implemented in vLLM, but the open-source version of vLLM only supports local KVCache caching.
|
||||
|
||||
With this matching information, Conductor estimates the corresponding execution time based on the request length and prefix_lenprefix\_len (which varies by instance). It then adds the estimated waiting time for that request to get the TTFT on that instance. Finally, Conductor assigns the request to the instance with the shortest TTFT and updates the cache and queue times for that instance accordingly. If the SLO is not achievable, Conductor directly returns the HTTP 429 Too Many Requests response status code to the upper layers.
|
||||
|
||||
The backbone of this scheduling framework is straightforward, but complexities are hidden in the engineering implementation of various components. For example, to predict the computation time of the prefill stage for a request, we employ a predictive model derived from offline test data. This model estimates the prefill duration based on the request’s length and prefix cache hit length. Thanks to the regular computation pattern of Transformers, the error bound of this prediction is small as long as enough offline data is available. The queuing time for a request is calculated by aggregating the prefill times of all queued requests. In practical implementations, TTFTs are computed in parallel, rendering the processing time negligible compared to the inference time.
|
||||
|
||||
More difficulty lies in predicting the transfer time because it is determined not only by the size of the transferred data but also by the current network status, especially whether the sending node is under congestion. This also necessitates the replication of hot KVCache blocks, which will be discussed in the next section.
|
||||
|
||||
### 6.2 Cache Load Balancing
|
||||
|
||||
In our Mooncake cluster, each prefill machine manages its own set of local prefix caches. The usage frequency of these caches varies significantly. For example, system prompts are accessed by almost every request, whereas caches storing content from a local long document may be used by only one user. As discussed in §6.1, Conductor’s role is crucial in achieving an optimal balance between cache matching and instance load. Thus, from the perspective of the distributed cache system, load balancing also plays an important role. Specifically, it involves strategizing on how to back up caches to ensure that global prefill scheduling can achieve both high cache hits and low load.
|
||||
|
||||
A straw-man solution to this KVCache scheduling problem could be collecting the global usages of each block, using a prediction model to forecast their future usages, and making scheduling decisions accordingly. However, unlike the estimation of prefill time, workloads are highly dynamic and change significantly over time. Especially for a MaaS provider experiencing rapid growth in its user base, it is impossible to accurately predict future usage. Thus, we propose a heuristic-based automated hot-spot migration scheme to enhance cache load balancing.
|
||||
|
||||
|
||||
|
||||
Figure 8: The prefill scheduling experiment in the Mooncake cluster.
|
||||
|
||||
As previously noted, requests may not always be directed to the prefill instance with the longest prefix cache length due to high instance load. In such cases, the conductor forwards the cache’s location and the request to an alternative instance if the estimated additional prefill time is shorter than the transfer time. This instance proactively retrieves the KVCache from the holder and stores it locally. More importantly, we prefer to compute the input tokens if the best remote prefix match length is no larger than the current local reusable prefix multiplied by a threshold111This threshold is currently adjusted manually, but can be adaptively adjusted by an algorithm in the future. Both strategies not only reduce the prefill time for requests but also facilitate the automatic replication of hot-spot caches, allowing for their broader distribution across multiple machines.
|
||||
|
||||
To validate the effectiveness of our strategy, we conducted a scheduling experiment that compares random scheduling and load-balancing scheduling with our strategy. We further compare the cache-aware scheduling described in §6.1 and the KVCache-centric scheduling described in this section that considers cache load balancing. In random scheduling, a prefill instance is selected arbitrarily for each request. In load-balancing scheduling, the instance with the lightest load is chosen. To evaluate, we built a Mooncake cluster consisting of 8 prefill instances and 8 decoding instances, using idle machines overnight, and replayed 23,000 real-world requests for the experiment. We assessed the performance of each scheduling algorithm using the average TTFT and the TTFT SLO attainment rate. The experimental results, depicted in Figure 8, demonstrate that both the cache-aware strategy and the cache load balancing strategy significantly reduce the TTFT of requests. Our KVCache-centric scheduling algorithm outperforms both random and load-balancing scheduling across both metrics. More experiment results can be found in §8.
|
||||
|
||||
## 7 面向过载的调度
|
||||
|
||||
Most existing work on LLM serving assumes that all requests will be processed, optimizing the throughput or the TTFT and TBT of requests accordingly. However, in real scenarios, processing every incoming request is neither economical nor realistic. For commercial inference services facing rapidly increasing volumes of user requests, the growth rate of the cluster’s inference resources is far slower than the increase in incoming requests. As a result, overload is a common issue in current LLM serving, especially during peak times.
|
||||
|
||||
To balance costs and user experience, the system should process as many requests as possible until the system load reaches a predefined threshold. After this point, remaining requests will be either directly rejected or deferred for later retry. Mooncake, implemented as a disaggregated inference system, allows for more flexible scheduling strategies but also confronts unique scheduling challenges not present in non-disaggregated systems and not mentioned in previous works[7, 8, 9].
|
||||
|
||||
In this section, we describe an early rejection policy designed specifically for a disaggregated architecture and address the load fluctuation caused by this approach. We then explore how predicting the generation length is necessary to mitigate these problems.
|
||||
|
||||
### 7.1 Scheduling in Overload Scenarios
|
||||
|
||||
In scenarios where system overload occurs, scheduling involves determining whether to accept or reject incoming requests based on the system load. A critical aspect of this process is defining what constitutes the “system load”, as this definition influences the threshold at which requests are rejected. In conventional coupled systems, the prediction of TTFT and TBT can be complicated by interference between the prefill and decoding stages. Therefore, the load is often measured simply by the ratio of the number of requests being processed to the system’s maximum capacity.
|
||||
|
||||
In contrast, Mooncake, with its disaggregated architecture, processes the prefill and decoding stages independently. Thus we use SLO satisfaction as a direct load measurement. Specifically, we define lttftl_{ttft} and ltbtl_{tbt} as the TTFT and TBT SLO constraints for requests, respectively. The load for prefill and decoding instances is then determined by comparing the predicted maximum TTFT and TBT on an instance against lttftl_{ttft} and ltbtl_{tbt}. With these two criteria, Mooncake’s scheduling requires two key decisions: first, whether to accept the prefill stage based on the prefill instance’s load, and second, whether to proceed with the decoding stage depending on the decoding instance’s load.
|
||||
|
||||
### 7.2 Early Rejection
|
||||
|
||||
In practice, the individual load on prefill or decoding instances does not accurately reflect the actual number of requests processed by the system. This discrepancy arises due to a time lag between scheduling prefill and decoding instances for a single request. If a request is rejected by the decoding instance due to high load after the prefill stage has been completed, the computational resources expended during the prefill stage are wasted. Consequently, the actual number of successfully processed requests during prefill is less than that indicated by the load metric.
|
||||
|
||||
To address this issue, it is natural to advance the load assessment of the decoding instance to precede the beginning of the prefill stage. We refer to this strategy as Early Rejection. Upon the arrival of a request, Conductor evaluates whether to accept the request based on the greater load between the prefill and decoding pools. Early Rejection significantly reduces ineffective computations from rejected requests and enhances load balancing.
|
||||
|
||||
|
||||
|
||||
Figure 9: The load of prefill and decoding instances over 20 minutes, before using the prediction-based early rejection.
|
||||
|
||||
### 7.3 Load Fluctuation Caused by Early Rejection
|
||||
|
||||
However, Early Rejection introduces new challenges. Figure 9 shows the observed real-world instance load over a 20-minute period in a cluster of 20 machines after using the Early Rejection strategy. It highlights significant anti-phase fluctuations between prefill and decoding machines. This phenomenon becomes more pronounced in clusters with fewer prefill machines and in scenarios where the prefill stage takes longer.
|
||||
|
||||
Upon further exploration, we found that this load fluctuation problem is rooted in the time lag between predicting the decoding load and its actual execution. Scheduling based on the current decoding load is inherently delayed. This delay causes fluctuations and phase staggering between the loads on prefill and decoding instances, as illustrated in the theoretical example described in Figure 10(a). The green curve represents the load of prefill instances (scaled from 0 to 1), and the yellow curve represents the load of decoding instances.
|
||||
|
||||
|
||||
|
||||
(a) Early Rejection.
|
||||
|
||||
|
||||
|
||||
(b) Early Rejection Based on Prediction.
|
||||
|
||||
Figure 10: Instance load when applying Early Rejection and Early Rejection Based on Prediction.
|
||||
|
||||
In Stage 1, the load on both prefill and decoding instances is low, so Conductor accepts a large number of requests until the load on prefill instances reaches its limit. In Stage 2, requests processed by prefill instances are scheduled to decoding instances, causing the load on decoding instances to be high. Consequently, Conductor rejects incoming requests, leading to a lower load on prefill instances. In Stage 3, no new requests enter the decoding stage, resulting in a decreased load. At this point, Conductor again accepts a large number of requests until the prefill instances are fully loaded. In Stage 4, as the load on decoding instances increases, Conductor rejects requests, causing a low load on prefill instances. This severe fluctuation in load between prefill and decoding instances results in poor resource utilization of the inference cluster.
|
||||
|
||||
### 7.4 Early Rejection Based on Prediction
|
||||
|
||||
To solve the load fluctuation problem, we propose a framework of Early Rejection Based on Prediction to address scheduling challenges in overload scenarios for disaggregated LLM serving systems like Mooncake. As illustrated in Figure 10(b), this framework predicts the decoding load after the prefill stage of incoming requests and uses this prediction to decide whether to accept the requests, which helps mitigate the fluctuation problem. The core component of this strategy is the accurate prediction of the decoding load for the subsequent period. We introduce two approaches for this:
|
||||
|
||||
Request level: Previous work highlights a significant challenge in predicting loads for LLM serving: the unknown output length of each request. If we could determine the output length in advance, it would be possible to estimate the TTFT and TBT much more accurately. This, in turn, would help predict the number of requests a decoding instance can complete and the number of new requests that will be added after a specified time, thereby obtaining the load at that time. However, predicting each request’s output length is challenging due to high costs [9] or low accuracy, especially under overload conditions where resources are scarce and accurate predictions are necessary, making request-level predictions particularly difficult.
|
||||
|
||||
System level: In contrast to request-level predictions, system-level predictions do not attempt to predict the completion time
|
||||
|
||||
e TTFT SLO. However, while approximately 100% of the requests for Mooncake-[10P+10D] satisfy the TBT SLO, only 57% of the requests for vLLM-[20M] meet this criterion, with some requests exhibiting extremely high TBTs. In this experiment, Mooncake can process approximately 75% more requests while adhering to the SLOs.
|
||||
|
||||
### 8.2 Performance in Overload Scenarios
|
||||
|
||||
In this section, we evaluate performance under overload scenarios, focusing on the maximum number of requests the system can handle, as discussed in §7. The baseline strategy, which rejects requests based on load before both stages start, leads to resource wastage by rejecting requests already processed in the prefill stage. In contrast, we propose the Early Rejection and Early Rejection based on Prediction strategies, detailed in §7.2 and §7.4, respectively. These strategies take the system’s load into comprehensive consideration, and hence reduce unnecessary request rejections.
|
||||
|
||||
Specifically, we built a Mooncake cluster with 8 prefill instances and 8 decoding instances and tested it using real traces from 23,000 requests. To simulate overload scenarios, we increased the replay speed to 2x.
|
||||
|
||||
Table 3: Number of requests rejected by the system under the overloaded-scenario experiment.
|
||||
|
||||
Table 3 shows Mooncake’s performance under different strategies. With the baseline strategy, the system rejects 4,183 requests. In contrast, under the Early Rejection and Early Rejection based on Prediction strategies, Mooncake rejects 3,771 and 3,589 requests, respectively. This demonstrates that by rejecting requests early, Mooncake can avoid unnecessary prefill computations, thereby improving the effective utilization of system resources. Furthermore, by predicting the load of decoding instances, Mooncake can mitigate load fluctuations, increasing the request handling capacity.
|
||||
|
||||
## 9 Related Work
|
||||
|
||||
Significant efforts have been dedicated to enhancing the efficiency of LLM serving systems through scheduling, memory management, a
|
||||
@@ -0,0 +1,92 @@
|
||||
# 📊 文章摘要：Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
|
||||
|
||||
> **原文**：[2024-06-24_Mooncake.md](./2024-06-24_Mooncake.md)
|
||||
> **原文链接**：https://arxiv.org/abs/2407.00079
|
||||
> **来源**：arXiv
|
||||
> **作者**：Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu（月之暗面 Moonshot AI、清华大学）
|
||||
> **发布日期**：2024-06-24
|
||||
> **摘要日期**：2026-08-06
|
||||
> **价值评级**：⭐⭐⭐ 高
|
||||
|
||||
---
|
||||
|
||||
## 核心命题
|
||||
|
||||
> **缓存为中心** — LLM serving 调度的核心不是 GPU 算力而是 KVCache：以缓存复用与分发为中心组织解耦架构，并面向真实过载场景设计预测式早期拒绝，是生产级 MaaS 平台的实践范式。
|
||||
|
||||
---
|
||||
|
||||
## 文章概要
|
||||
|
||||
Mooncake 是 Kimi 的生产 serving 平台，在 prefill/decoding 集群解耦的基础上，把 GPU 集群中闲置的 CPU、DRAM、SSD 组织成分层 KVCache 池，并以 Conductor 全局调度器为中心做"缓存感知"调度：复用尽可能多的前缀缓存、把热块复制到多节点、冷块换出到低层存储。与主流研究"假设所有请求都会被处理"不同，Mooncake 直面用户爆发式增长带来的过载问题，提出基于预测的早期拒绝策略（Early Rejection Based on Prediction），避免为注定被拒绝的请求浪费 prefill 算力，并缓解简单拒绝策略引起的 prefill/decoding 负载反相波动。模拟场景下相比 baseline 吞吐提升最高 525%（同时满足 SLO），真实负载下让 Kimi 多处理 75% 的请求。局限：实验基于 dummy 模型 + trace 重放，部分阈值靠人工调节，拒绝策略以牺牲低价值请求换取整体效率。
|
||||
|
||||
---
|
||||
|
||||
## 关键要点
|
||||
|
||||
1. **KVCache 是调度的中心矛盾** — 提升吞吐的两条路径（复用 KVCache 减少计算、增大 batch 提高 MFU）都会伤及延迟 SLO：远程取缓存拖长 TTFT，大 batch 放大 TBT；调度本质是在缓存复用与 SLO 之间做权衡。`[分类: 范式突破]`
|
||||
2. **分层解耦缓存池** — 利用 GPU 集群未充分利用的 CPU/DRAM/SSD 构建 KVCache 的分层存储与分发网络，让冷热块各得其所（热块复制防拥塞、冷块换出降成本）。`[分类: 未探索]`
|
||||
3. **缓存感知的全局调度** — Conductor 按前缀匹配长度、排队时间、传输时间估算各 prefill 实例的 TTFT，选择最优实例；不满足 SLO 时直接返回 HTTP 429。与仅看负载的调度相比，显著降低平均 TTFT。`[分类: 未探索]`
|
||||
4. **启发式热点迁移替代预测** — 工作负载高度动态、无法准确预测未来缓存使用，因此用"请求偏离最优缓存实例时按需迁移/复制缓存"的启发式方案实现热点块的自动复制，而非预测模型。`[分类: 未探索]`
|
||||
5. **长上下文利器：CPP + 逐层 prefill** — Chunked pipeline parallelism 把单个长请求跨节点流水处理（相对 sequence parallelism 降低网络消耗、简化弹性扩缩）；逐层异步加载/存储 KVCache 与计算重叠，使 prefill 实例调度几乎不再受 VRAM 大小约束。`[分类: 范式突破]`
|
||||
6. **面向过载的调度是全新问题域** — 用 SLO 满足度而非请求数/容量比衡量系统负载；把 decode 侧负载评估提前到 prefill 之前（早期拒绝），避免 prefill 算力浪费。`[分类: 范式突破]`
|
||||
7. **预测式早期拒绝抑制负载波动** — 简单早期拒绝会引发 prefill/decode 实例负载反相振荡（20 台机器实测），根因是预测与实际执行的时间差；预测未来 decode 负载可显著平抑波动：过载实验中拒绝请求数从 baseline 的 4183 降到 3771（早期拒绝）与 3589（预测式早期拒绝）。`[分类: 未探索]`
|
||||
8. **生产验证与信息保护的平衡** — 全部实验基于重放真实 trace（仅含到达时间、输入/输出 token 数、块哈希，不含用户内容）+ 与 LLaMA2-70B 同架构的 dummy 模型，兼顾可复现与商业机密保护。`[分类: 争议]`
|
||||
|
||||
---
|
||||
|
||||
## 批判性分析
|
||||
|
||||
### 假设前提
|
||||
|
||||
- 大规模 MaaS 提供商的 GPU 集群长期处于过载状态，且资源增长速度远慢于请求增长——拒绝部分请求是合理且必要的商业决策。
|
||||
- 前缀缓存命中是长上下文负载中的常见现象，值得为缓存复用投入全局协调开销。
|
||||
- 集群中存在大量闲置 CPU/DRAM/SSD 资源可供缓存池使用，且 GPU 服务器的集成形态可以重构为解耦资源池。
|
||||
- Transformer 计算模式规则，prefill 执行时间可通过离线数据建模准确预测。
|
||||
|
||||
### 论据与逻辑
|
||||
|
||||
- 论据结构合理：先以 23000 条真实请求在 8+8 实例集群上对比随机/负载均衡/缓存感知/缓存均衡四种调度，证明 KVCache-centric 调度全面占优；再以 20 台机器 20 分钟负载曲线揭示反相波动现象，并给出理论示例与预测方案的改进数据。
|
||||
- 525% 吞吐提升与 75% 请求承载提升均有明确实验来源（模拟场景与真实负载），且文中明确区分两者语境，未混淆。
|
||||
- 弱点：dummy 模型无法完全代表真实模型的 KV cache 分布与内存行为；"up to"表述下的最大提升场景条件不明；阈值（如 kvcache_balancing_threshold）依赖人工调整，可复现性打折。
|
||||
|
||||
### 边界与局限
|
||||
|
||||
- 结论面向"高过载 + 长上下文"的 MaaS 生产场景；负载未饱和、前缀命中率低的场景下，全局缓存协调的收益可能被协调开销抵消。
|
||||
- 拒绝策略会牺牲尾部用户的请求质量（以 429 拒绝），对用户体感与商业口碑的影响论文未量化。
|
||||
- 早期拒绝依赖输出长度预测，而过载条件下请求级长度预测本身困难（论文承认成本高、精度低），系统级预测是妥协方案。
|
||||
- 实验保护商业信息的手段（dummy 模型 + trace 重放）本身限制了结果的真实性边界。
|
||||
|
||||
---
|
||||
|
||||
## 可引用金句
|
||||
|
||||
> "Unlike traditional studies that assume all requests will be processed, Mooncake faces challenges due to highly overloaded scenarios."
|
||||
> （与传统研究假设所有请求都会被处理不同，Mooncake 面对的是严重过载场景的挑战。）
|
||||
|
||||
> "We found that the scheduling of KVCache is central to LLM serving scheduling."
|
||||
> （我们发现 KVCache 的调度是 LLM serving 调度的核心所在。）
|
||||
|
||||
---
|
||||
|
||||
## 总体评价
|
||||
|
||||
**亮点**：
|
||||
- 首个公开的、经受真实爆发式增长验证的"缓存为中心"解耦 serving 架构，工业界一手的工程视角稀缺
|
||||
- 把"过载调度"引入 LLM serving 问题域，早期拒绝与负载波动分析（反相振荡的根因剖析）极具启发性
|
||||
- 缓存感知调度 + 热点迁移 + 逐层 prefill 的组合拳，直接服务长上下文这一当前关键场景
|
||||
|
||||
**不足**：
|
||||
- 实验体系（dummy 模型、trace 重放、人工阈值）的科学严格性弱于学术性较强的对照工作
|
||||
- 关键阈值与工程细节披露有限，可复现性受限
|
||||
- 拒绝策略对用户体感、商业指标的长期影响未讨论
|
||||
|
||||
**适用场景**：LLM serving 系统设计者、MaaS 平台（Kimi 类长上下文助手）基础设施团队；研究过载调度与缓存管理的研究人员。
|
||||
|
||||
**关联建议**：与 DistServe（解耦的 goodput 优化理论框架）、Splitwise（异构硬件部署）、TetriInfer（长度预测调度）对照阅读，可拼出解耦 serving 的全景；后续可关注 Mooncake 开源仓库（github.com/kvcache-ai/Mooncake）的演进与 vLLM 社区对前缀缓存调度的采纳。
|
||||
|
||||
---
|
||||
|
||||
## 配图
|
||||
|
||||
|
||||
@@ -0,0 +1,237 @@
|
||||
# DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
|
||||
|
||||
> **来源**：arXiv
|
||||
> **作者**：Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang
|
||||
> **发布日期**：2024-01-18
|
||||
> **原文链接**：https://arxiv.org/abs/2401.09670
|
||||
|
||||
---
|
||||
|
||||
## 论文元数据
|
||||
|
||||
- **arXiv ID**：2401.09670
|
||||
- **学科分类**：Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI)
|
||||
- **作者机构**：北京大学（School of Computer Science, Peking University）、StepFun、UC San Diego
|
||||
- **提交历史**：v1: 2024-01-18；v2: 2024-03-19；v3: 2024-06-06
|
||||
- **DOI**：https://doi.org/10.48550/arXiv.2401.09670
|
||||
|
||||
---
|
||||
|
||||
Yinmin Zhong Shengyu Liu Junda Chen Jianbo Hu Yibo Zhu Xuanzhe Liu
|
||||
|
||||
Xin Jin Hao Zhang
|
||||
|
||||
School of Computer Science, Peking University StepFun UC San Diego
|
||||
|
||||
## 摘要（Abstract）
|
||||
|
||||
DistServe improves the performance of large language models (LLMs) serving by disaggregating the prefill and decoding computation. Existing LLM serving systems colocate the two phases and batch the computation of prefill and decoding across all users and requests.
|
||||
We find that this strategy not only leads to strong prefill-decoding interferences but also couples the resource allocation and parallelism plans for both phases. LLM applications often emphasize individual latency for each phase: time to first token (TTFT) for the prefill phase and time per output token (TPOT) of each request for the decoding phase.
|
||||
In the presence of stringent latency requirements, existing systems have to prioritize one latency over the other, or over-provision compute resources to meet both.
|
||||
|
||||
DistServe assigns prefill and decoding computation to different GPUs, hence eliminating prefill-decoding interferences. Given the application’s TTFT and TPOT requirements, DistServe co-optimizes the resource allocation and parallelism strategy _tailored_ for each phase. DistServe also places the two phases according to the serving cluster’s bandwidth to minimize the communication caused by disaggregation. As a result, DistServe significantly improves LLM serving performance in terms of the maximum rate that can be served within both TTFT and TPOT constraints on each GPU.
|
||||
Our evaluations show that on various popular LLMs, applications, and latency requirements, DistServe can serve 7.4× more requests or 12.6× tighter SLO, compared to state-of-the-art systems, while staying within latency constraints for >90%90>90\%> 90 % of requests.
|
||||
|
||||
## 1 引言（Introduction）
|
||||
|
||||
Large language models (LLMs), such as GPT-4 [37], Bard [2], and LLaMA [51], represent a groundbreaking shift in generative AI. They start to reshape existing Internet services, ranging from search engines to personal assistants [4], and enable fundamentally new applications, like universal chatbots [1, 16] and programming assistants [15, 42]. Yet, these advances come with a significant challenge: processing an end-to-end LLM query can be substantially slower than a standard search query [41]. In order to meet the stringent latency requirements of various applications, service providers need to over-provision compute resources, particularly many GPUs, leading to a shortfall in cost efficiency. Therefore, optimizing the cost per LLM query while adhering to high SLO attainment (the proportion of requests that meet the SLOs) is becoming increasingly essential for all LLM services.
|
||||
|
||||
|
||||
|
||||
Figure 1:
|
||||
Performance when serving an LLM with 13B parameters under a synthetic workload with input length = 512 and output length = 64 on one NVIDIA 80GB A100. Upper: The P90 time-to-first-token (TTFT) latency comparing existing systems vs. a system serving only the prefill phase. Down: The P90 time-per-output-token (TPOT) latency comparing existing systems vs. a system serving only the decoding phase.
|
||||
|
||||
An LLM service responds to a user query in two phases. The _prefill phase_ processes a user’s prompt, composed of a sequence of tokens, to generate the first token of the response _in one step_. Following it, the _decoding phase_ sequentially generates subsequent tokens _in multiple steps_; each decoding step generates a new token based on tokens generated in previous steps, until reaching a termination token.
|
||||
This dual-phase process distinguishes LLM services from traditional services
|
||||
– an LLM service’s latency is uniquely measured by two key metrics: the _time to first token_ (TTFT), which is the duration of the prefill phase, and the _time per output token_ (TPOT), which represents the average time taken to generate a token for each request (except for the first token)111The overall request latency equals TTFT plus TPOT times the number of generated tokens in the decoding phase..
|
||||
Different applications place varying demands on each metric. For example, real-time chatbots [1] prioritize low TTFT for response promptness, while TPOT only remains important until it is faster than human reading speed (i.e., 250 words/min).
|
||||
Conversely, document summarization emphasizes low TPOT for faster generation of the summary.
|
||||
|
||||
Hence, given the application’s TTFT and TPOT requirements, an effective LLM serving system should balance these needs and maximize _per-GPU goodput_, defined as the maximum request rate that can be served adhering to the SLO attainment goal (say, 90%) for each GPU provisioned – higher per-GPU goodput directly translates into lower cost per query.
|
||||
|
||||
As the prefill and decoding phases share the LLM weights and working memory,
|
||||
existing LLM serving systems typically colocate both phases on GPUs and maximize the overall system throughput – tokens generated per second across all users and requests – by batching the prefill and decoding steps across requests [54, 31]. However, to meet latency requirements, we find these systems must over-provision compute resources. To see this, Figure 1 illustrates how the P90 TTFT and TPOT shift with increasing request rates when serving a 13B LLM using existing systems [32], with workload pattern and two latency constraints set to emulate using LLM to generate a short summary for an article. Under the SLO attainment of 90%, the maximum achievable goodput on a single A100 GPU, which is constrained by the more stringent one of TTFT and TPOT requirements, is about 1.6 requests per second (rps).
|
||||
The performance contrasts sharply when each phase is served independently on a separate GPU, shown by the orange and green curves, which achieve per-GPU goodput of 5.6 rps for the prefill phase and 10 rps for decoding. Ideally, by allocating 2 GPUs for prefill and 1 GPU for decoding, we can effectively serve the model with an overall goodput of 10 rps, or equally 3.3 rps per GPU, which is 2.1x higher than existing systems.
|
||||
The gap in goodput primarily stems from the colocation of the prefill and decoding – two phases with very distinct computational characteristics and latency requirements (§2.1).
|
||||
|
||||
First, colocation leads to strong _prefill-decoding interference_.
|
||||
A prefill step often takes much longer than a decoding step. When batched together, decoding steps in the batch are delayed by the prefill steps, significantly elongating their TPOT; similarly, the inclusion of decoding steps contributes to a non-trivial increase in TTFT, as evidenced in Figure 2.
|
||||
Even if we schedule them separately, issues persist as they begin to compete for resources. Decoding tasks awaiting GPU execution are subject to increased queuing delays due to ongoing prefill tasks, and vice versa. Prioritized scheduling of one phase risks failing the latency requirements of the other.
|
||||
|
||||
Second, the prefill and decoding computation differ in latency requirements and preference for different forms of parallelism (§3). Colocating prefill and decoding, however, couples their resource allocation, and prevents implementing different parallelism strategies more suited to meeting the specific latency requirements of each phase.
|
||||
|
||||
To overcome these challenges, we propose to disaggregate the prefill and decoding phases of LLM inference, assigning them to separate GPUs. Our approach has two benefits.
|
||||
First, operating each phase independently on different GPUs eliminates prefill-decoding interference. Second, it allows to scale each phase independently with tailored resource allocation and model parallelism strategies to meet their specific latency requirements.
|
||||
Although disaggregation causes communication of intermediate states between GPUs, we show that the communication overhead is insubstantial (§3.3) in modern GPU clusters, and when managed appropriately, disaggregation significantly improves per-GPU goodput.
|
||||
|
||||
Based on the above insights, in this work, we build DistServe 222https://github.com/LLMServe/DistServe, a goodput-optimized LLM serving system by disaggregating the prefill and decoding phases. Given TTFT and TPOT requirements, DistServe first scales each phase independently by co-optimizing the GPU allocation and parallelism strategies of the prefill and decoding phase assuming serving a single model replica. The optimization ensures maximizing the per-GPU goodput and may assign different numbers of GPUs and parallelism strategies to each phase depending on their respective latency requirements. DistServe then scales this allocation to multiple instances via replication until meeting the user-required traffic rate (§4).
|
||||
DistServe also features an algorithm to place the prefill and decoding computation according to their allocation schemes and the cluster’s bandwidth to minimize the overhead of communicating intermediate states between phases.
|
||||
|
||||
We implement DistServe as an orchestration layer on top of the LLM inference engine. We
|
||||
evaluate DistServe on various LLMs, varying the workloads based on three important real-world LLM applications: chatbots, programming assistant, and document summary. Compared to state-of-the-art solutions, DistServe can serve up to 7.4×7.4 × more requests or 12.6×12.6 × tighter SLO under various latency constraints. Our contributions are:
|
||||
|
||||
- •
|
||||
|
||||
Identify the problems of prefill-decoding interference and resource coupling in existing LLM serving systems and propose to disaggregate the two phases.
|
||||
- •
|
||||
|
||||
Design a novel placement algorithm to choose the goodput-optimal schema for prefill and decoding instances automatically.
|
||||
- •
|
||||
|
||||
Conduct a comprehensive evaluation of DistServe with realistic workloads.
|
||||
|
||||
## 4 方法（Method）
|
||||
|
||||
We built DistServe to solve the above challenges. Given the model, workload characteristic, latency requirements, and SLO attainment target, DistServe will determine (a) the parallelism strategies for prefill and decoding instances, (b) the number of each instance type to deploy, as well as (c) how to place them onto the physical cluster. We call the solution a placement. Our goal is to find a placement that maximizes the per-gpu goodput.
|
||||
|
||||
As explained in §3.3, a key design consideration is to manage communications between disaggregated prefill and decoding phases, given varying cluster setups.
|
||||
In this section, we first present two placement algorithms: one for clusters with high-speed cross-node networks (§4.1) and the other for environments lacking such infrastructure (§4.2); the latter introduces additional constraints. We then develop online scheduling optimizations that adapt to the nuances of real-world workloads (§4.3).
|
||||
|
||||
### 4.1 Placement for High Node-Affinity Cluster
|
||||
|
||||
Algorithm 1 High Node-Affinity Placement Algorithm
|
||||
|
||||
LLM G𝐺Gitalic_G, #node limit per-instance N𝑁Nitalic_N, #GPU per-node M𝑀Mitalic_M, GPU memory capacity C𝐶Citalic_C, workload W𝑊Witalic_W, traffic rate R𝑅Ritalic_R.
|
||||
|
||||
the placement 𝑏𝑒𝑠𝑡_𝑝𝑙𝑚.𝑏𝑒𝑠𝑡_𝑝𝑙𝑚{best\_plm}.italic_best _ italic_plm .
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔p,𝑐𝑜𝑛𝑓𝑖𝑔d←∅,∅formulae-sequence←
|
||||
|
||||
subscript𝑐𝑜𝑛𝑓𝑖𝑔𝑝subscript𝑐𝑜𝑛𝑓𝑖𝑔𝑑
|
||||
{config_{p}},{config_{d}},_config start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT , italic_config start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT ← ∅ , ∅
|
||||
|
||||
for 𝑖𝑛𝑡𝑟𝑎_𝑜𝑝∈{1,2,…,M}𝑖𝑛𝑡𝑟𝑎_𝑜𝑝12…𝑀{intra\_op}\{1,2,...,M\}italic_intra _ italic_op ∈ { 1 , 2 , … , italic_M } do
|
||||
|
||||
for 𝑖𝑛𝑡𝑒𝑟_𝑜𝑝∈{1,2,…,N×M𝑖𝑛𝑡𝑟𝑎_𝑜𝑝}𝑖𝑛𝑡𝑒𝑟_𝑜𝑝12…𝑁𝑀𝑖𝑛𝑡𝑟𝑎_𝑜𝑝{inter\_op}\{1,2,...,{N M}{{intra\_op}}\}italic_inter _ italic_op ∈ { 1 , 2 , … , divide start_ARG italic_N × italic_M end_ARG start_ARG italic_intra _ italic_op end_ARG } do
|
||||
|
||||
if G.size𝑖𝑛𝑡𝑒𝑟_𝑜𝑝×𝑖𝑛𝑡𝑟𝑎_𝑜𝑝<Cformulae-sequence𝐺𝑠𝑖𝑧𝑒𝑖𝑛𝑡𝑒𝑟_𝑜𝑝𝑖𝑛𝑡𝑟𝑎_𝑜𝑝𝐶{G.size}{{inter\_op}{intra\_op}}<Cdivide start_ARG italic_G . italic_s italic_i italic_z italic_e end_ARG start_ARG italic_inter _ italic_op × italic_intra _ italic_op end_ARG < italic_C then
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔←(𝑖𝑛𝑡𝑒𝑟_𝑜𝑝,𝑖𝑛𝑡𝑟𝑎_𝑜𝑝)←𝑐𝑜𝑛𝑓𝑖𝑔𝑖𝑛𝑡𝑒𝑟_𝑜𝑝𝑖𝑛𝑡𝑟𝑎_𝑜𝑝{config}({inter\_op},{intra\_op})italic_config ← ( italic_inter _ italic_op , italic_intra _ italic_op )
|
||||
|
||||
G^←parallel(G,𝑐𝑜𝑛𝑓𝑖𝑔)←^𝐺parallel𝐺𝑐𝑜𝑛𝑓𝑖𝑔{G}{parallel}(G,{config})over^ start_ARG italic_G end_ARG ← parallel ( italic_G , italic_config )
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔.𝑔𝑜𝑜𝑑𝑝𝑢𝑡←simu_prefill(G^,W)formulae-sequence𝑐𝑜𝑛𝑓𝑖𝑔←𝑔𝑜𝑜𝑑𝑝𝑢𝑡simu_prefill^𝐺𝑊{config.goodput}{simu\_prefill}({G},W)italic_config . italic_goodput ← simu_prefill ( over^ start_ARG italic_G end_ARG , italic_W )
|
||||
|
||||
if 𝑐𝑜𝑛𝑓𝑖𝑔p.𝑔𝑜𝑜𝑑𝑝𝑢𝑡configp.num_gpus<𝑐𝑜𝑛𝑓𝑖𝑔.𝑔𝑜𝑜𝑑𝑝𝑢𝑡config.num_gpusformulae-sequencesubscript𝑐𝑜𝑛𝑓𝑖𝑔𝑝𝑔𝑜𝑜𝑑𝑝𝑢𝑡formulae-sequence𝑐𝑜𝑛𝑓𝑖subscript𝑔𝑝𝑛𝑢𝑚_𝑔𝑝𝑢𝑠formulae-sequence𝑐𝑜𝑛𝑓𝑖𝑔𝑔𝑜𝑜𝑑𝑝𝑢𝑡formulae-sequence𝑐𝑜𝑛𝑓𝑖𝑔𝑛𝑢𝑚_𝑔𝑝𝑢𝑠{{config_{p}.goodput}}{config_{p}.num\_gpus}<{{config.%
|
||||
goodput}}{config.num\_gpus}divide start_ARG italic_config start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT . italic_goodput end_ARG start_ARG italic_c italic_o italic_n italic_f italic_i italic_g start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT . italic_n italic_u italic_m _ italic_g italic_p italic_u italic_s end_ARG < divide start_ARG italic_config . italic_goodput end_ARG start_ARG italic_c italic_o italic_n italic_f italic_i italic_g . italic_n italic_u italic_m _ italic_g italic_p italic_u italic_s end_ARG then
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔p←𝑐𝑜𝑛𝑓𝑖𝑔←subscript𝑐𝑜𝑛𝑓𝑖𝑔𝑝𝑐𝑜𝑛𝑓𝑖𝑔{config_{p}}{config}italic_config start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ← italic_config
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔.𝑔𝑜𝑜𝑑𝑝𝑢𝑡←simu_decode(G^,W)formulae-sequence𝑐𝑜𝑛𝑓𝑖𝑔←𝑔𝑜𝑜𝑑𝑝𝑢𝑡simu_decode^𝐺𝑊{config.goodput}{simu\_decode}({G},W)italic_config . italic_goodput ← simu_decode ( over^ start_ARG italic_G end_ARG , italic_W )
|
||||
|
||||
if 𝑐𝑜𝑛𝑓𝑖𝑔d.𝑔𝑜𝑜𝑑𝑝𝑢𝑡configd.num_gpus<𝑐𝑜𝑛𝑓𝑖𝑔.𝑔𝑜𝑜𝑑𝑝𝑢𝑡config.num_gpusformulae-sequencesubscript𝑐𝑜𝑛𝑓𝑖𝑔𝑑𝑔𝑜𝑜𝑑𝑝𝑢𝑡formulae-sequence𝑐𝑜𝑛𝑓𝑖subscript𝑔𝑑𝑛𝑢𝑚_𝑔𝑝𝑢𝑠formulae-sequence𝑐𝑜𝑛𝑓𝑖𝑔𝑔𝑜𝑜𝑑𝑝𝑢𝑡formulae-sequence𝑐𝑜𝑛𝑓𝑖𝑔𝑛𝑢𝑚_𝑔𝑝𝑢𝑠{{config_{d}.goodput}}{config_{d}.num\_gpus}<{{config.%
|
||||
goodput}}{config.num\_gpus}divide start_ARG italic_config start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT . italic_goodput end_ARG start_ARG italic_c italic_o italic_n italic_f italic_i italic_g start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT . italic_n italic_u italic_m _ italic_g italic_p italic_u italic_s end_ARG < divide start_ARG italic_config . italic_goodput end_ARG start_ARG italic_c italic_o italic_n italic_f italic_i italic_g . italic_n italic_u italic_m _ italic_g italic_p italic_u italic_s end_ARG then
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔d←𝑐𝑜𝑛𝑓𝑖𝑔←subscript𝑐𝑜𝑛𝑓𝑖𝑔𝑑𝑐𝑜𝑛𝑓𝑖𝑔{config_{d}}{config}italic_config start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT ← italic_config
|
||||
|
||||
n,m←⌈R𝑐𝑜𝑛𝑓𝑖𝑔p.𝑔𝑜𝑜𝑑𝑝𝑢𝑡⌉,⌈R𝑐𝑜𝑛𝑓𝑖𝑔d.𝑔𝑜𝑜𝑑𝑝𝑢𝑡⌉formulae-sequence←
|
||||
|
||||
𝑛𝑚
|
||||
𝑅formulae-sequencesubscript𝑐𝑜𝑛𝑓𝑖𝑔𝑝𝑔𝑜𝑜𝑑𝑝𝑢𝑡𝑅formulae-sequencesubscript𝑐𝑜𝑛𝑓𝑖𝑔𝑑𝑔𝑜𝑜𝑑𝑝𝑢𝑡n,m{R}{{config_{p}.goodput}},{R}{%
|
||||
{config_{d}.goodput}}_n , italic_m ← ⌈ divide start_ARG italic_R end_ARG start_ARG italic_config start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT . italic_goodput end_ARG ⌉ , ⌈ divide start_ARG italic_R end_ARG start_ARG italic_config start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT . italic_goodput end_ARG ⌉
|
||||
|
||||
𝑏𝑒𝑠𝑡_𝑝𝑙𝑚←(n,𝑐𝑜𝑛𝑓𝑖𝑔p,m,𝑐𝑜𝑛𝑓𝑖𝑔d)←𝑏𝑒𝑠𝑡_𝑝𝑙𝑚𝑛subscript𝑐𝑜𝑛𝑓𝑖𝑔𝑝𝑚subscript𝑐𝑜𝑛𝑓𝑖𝑔𝑑{best\_plm}(n,{config_{p}},m,{config_{d}})italic_best _ italic_plm ← ( italic_n , italic_config start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT , italic_m , italic_config start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT )
|
||||
|
||||
return 𝑏𝑒𝑠𝑡_𝑝𝑙𝑚𝑏𝑒𝑠𝑡_𝑝𝑙𝑚{best\_plm}italic_best _ italic_plm
|
||||
|
||||
On high node-affinity clusters equipped with Infiniband, KV caches transmission overhead across nodes is negligible, DistServe can deploy prefill and decoding instances across any two nodes without constraints.
|
||||
We propose a two-level placement algorithm for such scenarios: we first optimize the parallelism configurations for prefill and decoding instances separately to attain phase-level optimal per-gpu goodput; then, we use replication to match the overall traffic rate.
|
||||
|
||||
However, finding the optimal parallel configuration for a single instance type, such as for the prefill instance, is still challenging, due to the lack of a simple analytical formula to calculate the SLO attainment (a.k.a., percentage of requests that meet TTFT requirement), given that the workload has diverse input, output lengths, and irregular arrival patterns. Gauging the SLO via real-testbed profiling is time-prohibitive. We thus resort to building a simulator to estimate the SLO attainment, assuming prior knowledge of the workload’s arrival process and input and output length distributions.
|
||||
Although short-term interval is impossible to predict, the workload pattern over longer timescales (e.g.,
|
||||
hours or days) is often predictable [33, 55]. DistServe fits a distribution from the history request traces and resamples new traces from the distribution as the input workload to the simulator to compute the SLO attainment. Next, DistServe simply enumerates the placements and finds the maximum rate that meets the SLO attainment target with binary search and simulation trials.
|
||||
|
||||
Algorithm 1 outlines the process. We enumerate all feasible parallel configurations, subject to cluster capacity limit, for both prefill and decoding instances. Then, for a specific prefill phase configuration, we use `simu_prefill` to simulate and find its maximum goodput via binary search (similarly for using `simu_decode` for decoding).
|
||||
After determining the optimal parallel configurations for both prefill and decoding instances, we replicate them to achieve the user-required overall traffic rate according to their goodput.
|
||||
|
||||
The complexity of Algorithm 1 is O(NM2)𝑂𝑁superscript𝑀2O(NM^{)italic_O ( italic_N italic_M start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ), with N𝑁Nitalic_N as the node limit per instance and M𝑀Mitalic_M representing the typical number of GPUs per node in modern clusters (e.g., 8). The search space is manageable and the solving time is under 1.3 minutes in our largest setting, as demonstrated in §6.5.
|
||||
|
||||
Simulator building. Algorithm 1 relies on a simulator to estimate the goodput under various SLOs and SLO attainment goals given the workload and the parallelism plan.
|
||||
To build an accurate simulator, we analyze the FLOPs and the number of memory accesses for prefill and decoding phases respectively, and use a latency model to approximate the inference execution time. See details in Appendix A. The simulator aligns well with real profiling results, thanks to the high predictability of DNN workloads [23, 33], verified in §6.4.
|
||||
|
||||
By far, we have developed Algorithm 1 assuming we can place the prefill and decoding instance between any two nodes (or on the same node) of the cluster, and the KV cache transmission utilizes high bandwidth network. In many real clusters, GPUs inside a node access to high-bandwidth NVLINK while GPUs distributed across nodes have limited bandwidth. We next develop an algorithm to address this constraint.
|
||||
|
||||
Algorithm 2 Low Node-Affinity Placement Algorithm
|
||||
|
||||
LLM G𝐺Gitalic_G, #node limit per-instance N𝑁Nitalic_N, #GPU per-node M𝑀Mitalic_M, GPU memory capacity C𝐶Citalic_C, workload W𝑊Witalic_W, traffic rate R𝑅Ritalic_R.
|
||||
|
||||
the placement 𝑏𝑒𝑠𝑡_𝑝𝑙𝑚.𝑏𝑒𝑠𝑡_𝑝𝑙𝑚{best\_plm}.italic_best _ italic_plm .
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔∗←∅←superscript𝑐𝑜𝑛𝑓𝑖𝑔_config start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT ← ∅
|
||||
|
||||
for 𝑖𝑛𝑡𝑒𝑟_𝑜𝑝∈{1,2,…,N}𝑖𝑛𝑡𝑒𝑟_𝑜𝑝12…𝑁{inter\_op}\{1,2,...,N\}italic_inter _ italic_op ∈ { 1 , 2 , … , italic_N } do
|
||||
|
||||
𝒫←get_intra_node_configs(G,M,C,𝑖𝑛𝑡𝑒𝑟_𝑜𝑝)←𝒫get_intra_node_configs𝐺𝑀𝐶𝑖𝑛𝑡𝑒𝑟_𝑜𝑝{P}{get\_intra\_node\_configs}(G,M,C,{inter\_op})caligraphic_P ← get_intra_node_configs ( italic_G , italic_M , italic_C , italic_inter _ italic_op )
|
||||
|
||||
for Pp∈𝒫subscript𝑃𝑝𝒫P_{p}{P}italic_P start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ∈ caligraphic_P do
|
||||
|
||||
for Pd∈𝒫subscript𝑃𝑑𝒫P_{d}{P}italic_P start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT ∈ caligraphic_P do
|
||||
|
||||
if Pp.𝑛𝑢𝑚_𝑔𝑝𝑢𝑠+Pd.𝑛𝑢𝑚_𝑔𝑝𝑢𝑠≤Mformulae-sequencesubscript𝑃𝑝𝑛𝑢𝑚_𝑔𝑝𝑢𝑠subscript𝑃𝑑𝑛𝑢𝑚_𝑔𝑝𝑢𝑠𝑀P_{p}.{num\_gpus}+P_{d}.{num\_gpus} Mitalic_P start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT . italic_num _ italic_gpus + italic_P start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT . italic_num _ italic_gpus ≤ italic_M then
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔←(𝑖𝑛𝑡𝑒𝑟_𝑜𝑝,Pp,Pd)←𝑐𝑜𝑛𝑓𝑖𝑔𝑖𝑛𝑡𝑒𝑟_𝑜𝑝subscript𝑃𝑝subscript𝑃𝑑{config}({inter\_op},P_{p},P_{d})italic_config ← ( italic_inter _ italic_op , italic_P start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT , italic_P start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT )
|
||||
|
||||
G^p,G^d←parallel(G,𝑐𝑜𝑛𝑓𝑖𝑔)←
|
||||
|
||||
subscript^𝐺𝑝subscript^𝐺𝑑
|
||||
parallel𝐺𝑐𝑜𝑛𝑓𝑖𝑔{G}_{p},{G}_{d}{parallel}(G,{config})over^ start_ARG italic_G end_ARG start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT , over^ start_ARG italic_G end_ARG start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT ← parallel ( italic_G , italic_config )
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔.𝑔𝑜𝑜𝑑𝑝𝑢𝑡←simulate(G^p,G^d,W)formulae-sequence𝑐𝑜𝑛𝑓𝑖𝑔←𝑔𝑜𝑜𝑑𝑝𝑢𝑡simulatesubscript^𝐺𝑝subscript^𝐺𝑑𝑊{config.goodput}{simulate}({G}_{p},{G}_{d},W)italic_config . italic_goodput ← simulate ( over^ start_ARG italic_G end_ARG start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT , over^ start_ARG italic_G end_ARG start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT , italic_W )
|
||||
|
||||
if 𝑐𝑜𝑛𝑓𝑖𝑔.∗𝑔𝑜𝑜𝑑𝑝𝑢𝑡𝑐𝑜𝑛𝑓𝑖𝑔.∗𝑛𝑢𝑚_𝑔𝑝𝑢𝑠<𝑐𝑜𝑛𝑓𝑖𝑔.𝑔𝑜𝑜𝑑𝑝𝑢𝑡𝑐𝑜𝑛𝑓𝑖𝑔.𝑛𝑢𝑚_𝑔𝑝𝑢𝑠{{config.^{*}goodput}}{{config.^{*}num\_gpus}}<{%
|
||||
{config.goodput}}{{config.num\_gpus}}divide start_ARG italic_config . start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT italic_goodput end_ARG start_ARG italic_config . start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT italic_num _ italic_gpus end_ARG < divide start_ARG italic_config . italic_goodput end_ARG start_ARG italic_config . italic_num _ italic_gpus end_ARG then
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔∗←𝑐𝑜𝑛𝑓𝑖𝑔←superscript𝑐𝑜𝑛𝑓𝑖𝑔𝑐𝑜𝑛𝑓𝑖𝑔{config^{*}}{config}italic_config start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT ← italic_config
|
||||
|
||||
n←⌈R𝑐𝑜𝑛𝑓𝑖𝑔.∗𝑔𝑜𝑜𝑑𝑝𝑢𝑡⌉n{R}{{config.^{*}goodput}}_n ← ⌈ divide start_ARG italic_R end_ARG start_ARG italic_config . start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT italic_goodput end_ARG ⌉
|
||||
|
||||
𝑏𝑒𝑠𝑡_𝑝𝑙𝑚←(n,𝑐𝑜𝑛𝑓𝑖𝑔∗)←𝑏𝑒𝑠𝑡_𝑝𝑙𝑚𝑛superscript𝑐𝑜𝑛𝑓𝑖𝑔{best\_plm}(n,{config^{*}})italic_best _ italic_plm ← ( italic_n , italic_config start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT )
|
||||
|
||||
return 𝑏𝑒𝑠𝑡_𝑝𝑙𝑚𝑏𝑒𝑠𝑡_𝑝𝑙𝑚{best\_plm}italic_best _ italic_plm
|
||||
|
||||
### 4.2 Placement for Low Node-Affinity Cluster
|
||||
|
||||
A straightforward solution is to always colocate prefill and decoding instances on the same node, utilizing the NVLINK, which is commonly available inside a GPU node.
|
||||
For large models, e.g. with 175B parameters (350GB), we may be unable to even host a single pair of prefill and decoding instances in an 8-GPU node (80G×8=640G<350×2GB80𝐺8640𝐺3502𝐺𝐵80G 8=640G<350 2GB80 italic_G × 8 = 640 italic_G < 350 × 2 italic_G italic_B). We incorporate this as additional placement constraints and co-optimize it with model parallelism, presented in Algorithm 2.
|
||||
|
||||
The key insight is that KV cache transfer occurs exclusively between corresponding layers of prefill and decoding instances.
|
||||
Leveraging inter-op parallelism, we group layers into stages and divide each instance into segments, termed as instance segments, with each segment maintaining one specific inter-op stage.
|
||||
By colocating prefill and decoding segments of the same stage within a single node, we force the transfer of intermediate states to occur only via NVLINK. Inside a node, we set the same parallelism and resource allocation for segments of the same instance. Given the typical limitation of GPUs per node (usually 8), we can enumerate possible configurations inside one node and use the simulator to identify the configurations that yield the best goodput.
|
||||
|
||||
As outlined in Algorithm 2, we begin by enumerating inter-op parallelism degrees to get all the possible instance segments. For each segment, we get all possible intra-node parallelism configurations by calling `get_intra_node_configs`. Then we use simulation to find the optimal one and replicate it to satisfy the target traffic rate.
|
||||
|
||||
### 4.3 Online scheduling
|
||||
|
||||
The runtime architecture of DistServe is shown in Figure 6. DistServe operates with a simple FCFS scheduling policy. All incoming requests arrive at a centralized controller, then dispatched to the prefill instance with the shortest queue for prefill processing, followed by dispatch to the least loaded decoding instance for decoding steps. This setup, while simple, is optimized with several key enhancements tailored to the nuances of real-world workloads.
|
||||
|
||||
Reducing pipeline bubbles.
|
||||
To mitigate the pipeline bubbles caused by non-uniform prompt lengths (§3.3), we schedule the requests in a way that balances the execution time across all batches in the pipeline. This is achieved by noting that, for both prefill and decoding instances, the number of new tokens in the batch is a reliable indicator of the batch’s real execution time.
|
||||
For prefill instances, we profile the target model and GPU to figure out the shortest prompt length Lmsubscript𝐿𝑚L_{m}italic_L start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT needed to saturate the GPU. We schedule prefill batches with a total sequence length close to Lmsubscript𝐿𝑚L_{m}italic_L start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT, by either batching multiple requests shorter than Lmsubscript𝐿𝑚L_{m}italic_L start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT or individually scheduling requests longer than Lmsubscript𝐿𝑚L_{m}italic_L start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT. For decoding instances, we set Lmsubscript𝐿𝑚L_{m}italic_L start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT as the largest batch size.
|
||||
|
||||
Combat busrtiness.
|
||||
Burstiness in workloads can cause a deluge of KV caches to transfer from prefill to decoding instances, risking memory overload on decoding instances.
|
||||
To circumvent this, DistServe employs a “pull” method for KV cache transmission rather than a “push” approach – decoding instances fetch KV cache from prefill instances _as needed_, using the GPU memory of prefill instances as a queuing buffer. This way, the prefill instance can continue handling other prefill jobs by simply retaining the KV Cache in the GPU memory after processing the prompt. Hence, each type of instance operates at its own pace without complex coordination.
|
||||
|
||||
|
||||
|
||||
Figure 6: DistServe Runtime System Architecture
|
||||
|
||||
Replaning. The resource and parallelism plan in DistServe is optimized for a specific workload pattern, which may become suboptimal if the workload pattern changes over time. DistServe implement periodic replanning. A workload profiler monitors key parameters such as the average input and output length of the requests, the average arrival rate, etc. If a significant pattern shift is detected, DistServe will trigger a rerun of the placement algorithm based on recent historical data. This process is expedient – the proposed algorithm runs in seconds (§6.5) and reloading LLM weights can be completed within minutes – far shorter than the hourly scale at which real-world workload variations tend to occur.
|
||||
|
||||
Preemption and fault tolerance. DistServe does not implement advanced runtime policies like preemption [26] and fault tolerance [58], which are complementary to disaggregation. Nevertheless, we discuss how they fit into DistServe.
|
||||
In DistServe, the FCFS policy can lead to a “convoy effect”, where longer requests block shorter ones in the prefill stage. Incorporating preemptive strategies, as suggested in existing literature [53], could enhance efficiency and is feasible within our system’s architecture.
|
||||
While not a primary focus in the current DistServe, fault tolerance is a critical aspect for consideration. In traditional colocation- and replication-based systems, a fault in one instance typically does not disrupt other replica instances. However, in DistServe, the dependency between prefill and decoding instances introduces the risk of fault propagation. For example, a fault in a single decoding instance mapped to multiple prefill instances could potentially cripple the entire service and cluster. We leave both as future work.
|
||||
|
||||
## 9 结论（Conclusion）
|
||||
|
||||
We present DistServe, a new LLM serving architecture that disaggregates the prefill and decoding computation. DistServe maximizes the per-gpu goodput – the maximum request rate that can be served adhering to the SLO attainment goal for each GPU provisioned, hence resulting in up to 7.4×7.4 × lower cost per LLM query with guaranteed satisfaction of SLOs.
|
||||
Our findings affirm that as latency becomes an increasingly important metric for LLM services, prefill and decoding disaggregation is a vital strategy in promising improved performance and service quality guarantees.
|
||||
|
||||
Acknowledgments. We sincerely thank our shepherd and
|
||||
the anonymous reviewers for their valuable feedback. This work was
|
||||
supported by the National Natural Science Foundation of China under the grant numbers
|
||||
62172008, 62325201, and the National Natural Science Fund for the Excellent Young Scientists Fund
|
||||
Program (Overseas). Junda Chen is supported by UCSD fellowship and Hao Zhang is supported by UCSD faculty startup fund. Xin Jin is
|
||||
the corresponding author. Yinmin Zhong, Xuanzhe Liu, and Xin Jin are
|
||||
also with the Key Laboratory of High Confidence Software Technologies (Peking
|
||||
University), Ministry of Education.
|
||||
@@ -0,0 +1,91 @@
|
||||
# 📊 文章摘要：DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
|
||||
|
||||
> **原文**：[2024-01-18_DistServe.md](./2024-01-18_DistServe.md)
|
||||
> **原文链接**：https://arxiv.org/abs/2401.09670
|
||||
> **来源**：arXiv
|
||||
> **作者**：Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang（北京大学、StepFun、UC San Diego）
|
||||
> **发布日期**：2024-01-18
|
||||
> **摘要日期**：2026-08-06
|
||||
> **价值评级**：⭐⭐⭐ 高
|
||||
|
||||
---
|
||||
|
||||
## 核心命题
|
||||
|
||||
> **阶段解耦** — 将 prefill 与 decoding 两个计算特性迥异的阶段拆分到独立 GPU 池并各自定制资源与并行策略，以"每 GPU goodput"而非系统总吞吐为优化目标，是 LLM serving 成本优化的范式转折点。
|
||||
|
||||
---
|
||||
|
||||
## 文章概要
|
||||
|
||||
现有 LLM serving 系统（vLLM 等）将 prefill 与 decoding 两阶段放在同一批 GPU 上混合批处理，本文指出这一策略产生两类问题：一是强烈的 prefill-decoding 相互干扰（混合批处理时解码步被 prefill 步拖慢、TTFT 与 TPOT 相互挤压）；二是两阶段被耦合的资源配置与并行策略所绑定，无法各取所需。DistServe 将两阶段分配到不同 GPU，在 TTFT/TPOT 约束下联合优化各阶段的 GPU 数量与并行配置（含高低节点亲和集群两套放置算法），并用"拉取式"KV cache 传输、流水线气泡削减与周期性重规划支撑运行。在多种 LLM 与应用负载下，DistServe 相比当时 SOTA 系统可服务 7.4× 更多请求或满足 12.6× 更严格的 SLO（>90% 请求达标）。局限在于依赖工作负载可预测性（配置搜索基于模拟器）且未实现抢占与容错。
|
||||
|
||||
---
|
||||
|
||||
## 关键要点
|
||||
|
||||
1. **两阶段特性迥异是问题根源** — prefill 是计算密集型单步操作、吃 FLOPs；decoding 是内存带宽受限的多步自回归，延迟敏感。混合批处理必然顾此失彼。`[分类: 共识]`
|
||||
2. **"干扰 + 耦合"双重病因** — 同批共存导致解码步延迟（TPOT 变长）、prefill 排队恶化（TTFT 变长）；且共享权重与显存迫使两阶段用同一并行策略，无法分别优化。`[分类: 范式突破]`
|
||||
3. **解耦收益量化** — 13B 模型单 A100 上，混合部署仅 1.6 rps goodput；分离后 prefill 单 GPU 达 5.6 rps、decoding 达 10 rps，按 2:1 GPU 配比整体 goodput 为 10 rps（每 GPU 3.3 rps），是原方案的 2.1×。`[分类: 共识]`
|
||||
4. **goodput 目标重塑优化函数** — 以"每 GPU 满足 SLO 达标率（90%）的最大请求率"为优化目标，直接对应单次查询成本；相比传统的最大 token 吞吐目标更贴合商业诉求。`[分类: 范式突破]`
|
||||
5. **两套放置算法适配不同网络** — 高节点亲和集群（Infiniband，跨节点 KV cache 传输开销可忽略）用枚举并行配置 + 二进制搜索模拟；低节点亲和集群利用"KV cache 只在对应层之间传输"的特性，将 prefill/decoding 的同层段共置单节点内走 NVLINK，避免跨节点慢速传输。`[分类: 未探索]`
|
||||
6. **"拉取"式 KV cache 传输抗突发** — decoding 实例按需从 prefill 实例拉取 KV cache，prefill 显存充当排队缓冲，避免突发流量压垮 decoding 内存，两类实例无需复杂协调。`[分类: 未探索]`
|
||||
7. **故障传播是新风险** — 解耦引入 prefill↔decoding 实例依赖：单个 decoding 实例故障可能连带多个 prefill 实例、瘫痪整个服务；抢占（如缓解 convoy 效应）与容错均留作未来工作。`[分类: 未探索]`
|
||||
|
||||
---
|
||||
|
||||
## 批判性分析
|
||||
|
||||
### 假设前提
|
||||
|
||||
- 工作负载的到达过程与输入/输出长度分布在较长时间尺度上可预测（论文以小时/天级可预测为前提，用历史 trace 拟合分布驱动配置搜索）。
|
||||
- 现代集群具备足够的跨节点带宽，使解耦通信开销"不实质"（KV cache 传输相对推理时间可忽略）。
|
||||
- 应用对 TTFT 与 TPOT 的要求可明确量化并作为输入给定。
|
||||
- 单模型副本的优化可先于多实例扩展完成（分两步：先求最优单副本配置，再复制满足流量）。
|
||||
|
||||
### 论据与逻辑
|
||||
|
||||
- 论据链条完整：先以 Figure 1 的 13B 单卡实验证明"混合部署 goodput 远低于分阶段独立运行"（1.6 vs 5.6/10 rps），再推导出解耦的必要性；随后用模拟器 + 真实测试床验证配置搜索与 SLO 达标率对齐（论文声称模拟器与真实 profiling 高度吻合）。
|
||||
- 端到端评估覆盖三种真实应用（聊天、编程助手、文档摘要）与多模型，7.4×/12.6× 的收益数字有实验支撑。
|
||||
- 潜在弱点：7.4×/12.6× 是"up to"峰值表述，具体负载下提升幅度不同；解耦收益高度依赖集群网络质量，论文对此的敏感度分析着墨较少。
|
||||
|
||||
### 边界与局限
|
||||
|
||||
- 结论适用于两阶段特性差异明显、SLO 严格的场景；若 TTFT/TPOT 要求宽松，解耦收益会收窄。
|
||||
- 未处理抢占调度与故障容错，且解耦使故障传播范围扩大——生产部署需另行补充。
|
||||
- 配置搜索依赖 workload 预测，负载模式剧烈变化（短时间尺度不可预测）时可能退化为次优。
|
||||
- 评估硬件为 A100 世代，未覆盖 H100/Groq 等后续异构平台（此边界由后续论文补足）。
|
||||
|
||||
---
|
||||
|
||||
## 可引用金句
|
||||
|
||||
> "We find that this strategy not only leads to strong prefill-decoding interferences but also couples the resource allocation and parallelism plans for both phases."
|
||||
> （我们发现，这种策略不仅导致强烈的 prefill-decoding 干扰，还耦合了两个阶段的资源分配与并行计划。）
|
||||
|
||||
> "Our findings affirm that as latency becomes an increasingly important metric for LLM services, prefill and decoding disaggregation is a vital strategy in promising improved performance and service quality guarantees."
|
||||
> （我们的发现证实：随着延迟成为 LLM 服务日益重要的指标，prefill 与 decoding 解耦是带来性能与服务质量的提升的关键策略。）
|
||||
|
||||
---
|
||||
|
||||
## 总体评价
|
||||
|
||||
**亮点**：
|
||||
- 首个系统化论证并实现 prefill/decoding 解耦的 serving 系统之一，直击当时 colocation 共识的软肋，开启后续解耦架构浪潮
|
||||
- "per-GPU goodput"优化目标将系统性能与商业成本直接挂钩，视角独特且可操作
|
||||
- 同时覆盖高/低节点亲和两类集群的放置算法，工程完备度高；KV cache"拉取"机制与流水线气泡消减是实用的系统细节
|
||||
|
||||
**不足**：
|
||||
- 依赖工作负载可预测性做离线配置搜索，在线自适应能力有限
|
||||
- 抢占与容错缺失，且未充分讨论解耦故障传播的工程应对
|
||||
- "up to"峰值收益表述下，对网络带宽敏感度的量化分析不够深入
|
||||
|
||||
**适用场景**：面向 LLM serving 系统设计者、云厂商推理基础设施团队；为理解后续（Mooncake、Splitwise、vLLM 解耦版等）全部解耦系工作提供理论基础。
|
||||
|
||||
**关联建议**：可与同期论文对照阅读——Splitwise（异构硬件视角的阶段拆分）、TetriInfer（混合负载干扰 + 长度预测调度）、Mooncake（KVCache 为中心的规模化生产实践）；后续可追踪 vLLM 官方对 prefill/decode 解耦的采纳与 DistServe 作者后续工作（如 DeepSeek 的 DeepEP/EP 解耦方向）。
|
||||
|
||||
---
|
||||
|
||||
## 配图
|
||||
|
||||
|
||||
Reference in New Issue
Block a user