文档(金鹏): 2026-08-06 章节 48 篇文章摘要归档
- 46 篇原文+摘要双文件归档（按 来源/作者 分层，复用本地归档 20 篇+新抓取 26 篇） - 即梦生成 9 组主题配图（大图+列表缩略图）存入 知识/金鹏/20260806/ - 章节重组为 9 个主题分组并挂接摘要引用
This commit is contained in:
@@ -0,0 +1,237 @@
|
||||
# DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
|
||||
|
||||
> **来源**：arXiv
|
||||
> **作者**：Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang
|
||||
> **发布日期**：2024-01-18
|
||||
> **原文链接**：https://arxiv.org/abs/2401.09670
|
||||
|
||||
---
|
||||
|
||||
## 论文元数据
|
||||
|
||||
- **arXiv ID**：2401.09670
|
||||
- **学科分类**：Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI)
|
||||
- **作者机构**：北京大学（School of Computer Science, Peking University）、StepFun、UC San Diego
|
||||
- **提交历史**：v1: 2024-01-18；v2: 2024-03-19；v3: 2024-06-06
|
||||
- **DOI**：https://doi.org/10.48550/arXiv.2401.09670
|
||||
|
||||
---
|
||||
|
||||
Yinmin Zhong Shengyu Liu Junda Chen Jianbo Hu Yibo Zhu Xuanzhe Liu
|
||||
|
||||
Xin Jin Hao Zhang
|
||||
|
||||
School of Computer Science, Peking University StepFun UC San Diego
|
||||
|
||||
## 摘要（Abstract）
|
||||
|
||||
DistServe improves the performance of large language models (LLMs) serving by disaggregating the prefill and decoding computation. Existing LLM serving systems colocate the two phases and batch the computation of prefill and decoding across all users and requests.
|
||||
We find that this strategy not only leads to strong prefill-decoding interferences but also couples the resource allocation and parallelism plans for both phases. LLM applications often emphasize individual latency for each phase: time to first token (TTFT) for the prefill phase and time per output token (TPOT) of each request for the decoding phase.
|
||||
In the presence of stringent latency requirements, existing systems have to prioritize one latency over the other, or over-provision compute resources to meet both.
|
||||
|
||||
DistServe assigns prefill and decoding computation to different GPUs, hence eliminating prefill-decoding interferences. Given the application’s TTFT and TPOT requirements, DistServe co-optimizes the resource allocation and parallelism strategy _tailored_ for each phase. DistServe also places the two phases according to the serving cluster’s bandwidth to minimize the communication caused by disaggregation. As a result, DistServe significantly improves LLM serving performance in terms of the maximum rate that can be served within both TTFT and TPOT constraints on each GPU.
|
||||
Our evaluations show that on various popular LLMs, applications, and latency requirements, DistServe can serve 7.4× more requests or 12.6× tighter SLO, compared to state-of-the-art systems, while staying within latency constraints for >90%90>90\%> 90 % of requests.
|
||||
|
||||
## 1 引言（Introduction）
|
||||
|
||||
Large language models (LLMs), such as GPT-4 [37], Bard [2], and LLaMA [51], represent a groundbreaking shift in generative AI. They start to reshape existing Internet services, ranging from search engines to personal assistants [4], and enable fundamentally new applications, like universal chatbots [1, 16] and programming assistants [15, 42]. Yet, these advances come with a significant challenge: processing an end-to-end LLM query can be substantially slower than a standard search query [41]. In order to meet the stringent latency requirements of various applications, service providers need to over-provision compute resources, particularly many GPUs, leading to a shortfall in cost efficiency. Therefore, optimizing the cost per LLM query while adhering to high SLO attainment (the proportion of requests that meet the SLOs) is becoming increasingly essential for all LLM services.
|
||||
|
||||
|
||||
|
||||
Figure 1:
|
||||
Performance when serving an LLM with 13B parameters under a synthetic workload with input length = 512 and output length = 64 on one NVIDIA 80GB A100. Upper: The P90 time-to-first-token (TTFT) latency comparing existing systems vs. a system serving only the prefill phase. Down: The P90 time-per-output-token (TPOT) latency comparing existing systems vs. a system serving only the decoding phase.
|
||||
|
||||
An LLM service responds to a user query in two phases. The _prefill phase_ processes a user’s prompt, composed of a sequence of tokens, to generate the first token of the response _in one step_. Following it, the _decoding phase_ sequentially generates subsequent tokens _in multiple steps_; each decoding step generates a new token based on tokens generated in previous steps, until reaching a termination token.
|
||||
This dual-phase process distinguishes LLM services from traditional services
|
||||
– an LLM service’s latency is uniquely measured by two key metrics: the _time to first token_ (TTFT), which is the duration of the prefill phase, and the _time per output token_ (TPOT), which represents the average time taken to generate a token for each request (except for the first token)111The overall request latency equals TTFT plus TPOT times the number of generated tokens in the decoding phase..
|
||||
Different applications place varying demands on each metric. For example, real-time chatbots [1] prioritize low TTFT for response promptness, while TPOT only remains important until it is faster than human reading speed (i.e., 250 words/min).
|
||||
Conversely, document summarization emphasizes low TPOT for faster generation of the summary.
|
||||
|
||||
Hence, given the application’s TTFT and TPOT requirements, an effective LLM serving system should balance these needs and maximize _per-GPU goodput_, defined as the maximum request rate that can be served adhering to the SLO attainment goal (say, 90%) for each GPU provisioned – higher per-GPU goodput directly translates into lower cost per query.
|
||||
|
||||
As the prefill and decoding phases share the LLM weights and working memory,
|
||||
existing LLM serving systems typically colocate both phases on GPUs and maximize the overall system throughput – tokens generated per second across all users and requests – by batching the prefill and decoding steps across requests [54, 31]. However, to meet latency requirements, we find these systems must over-provision compute resources. To see this, Figure 1 illustrates how the P90 TTFT and TPOT shift with increasing request rates when serving a 13B LLM using existing systems [32], with workload pattern and two latency constraints set to emulate using LLM to generate a short summary for an article. Under the SLO attainment of 90%, the maximum achievable goodput on a single A100 GPU, which is constrained by the more stringent one of TTFT and TPOT requirements, is about 1.6 requests per second (rps).
|
||||
The performance contrasts sharply when each phase is served independently on a separate GPU, shown by the orange and green curves, which achieve per-GPU goodput of 5.6 rps for the prefill phase and 10 rps for decoding. Ideally, by allocating 2 GPUs for prefill and 1 GPU for decoding, we can effectively serve the model with an overall goodput of 10 rps, or equally 3.3 rps per GPU, which is 2.1x higher than existing systems.
|
||||
The gap in goodput primarily stems from the colocation of the prefill and decoding – two phases with very distinct computational characteristics and latency requirements (§2.1).
|
||||
|
||||
First, colocation leads to strong _prefill-decoding interference_.
|
||||
A prefill step often takes much longer than a decoding step. When batched together, decoding steps in the batch are delayed by the prefill steps, significantly elongating their TPOT; similarly, the inclusion of decoding steps contributes to a non-trivial increase in TTFT, as evidenced in Figure 2.
|
||||
Even if we schedule them separately, issues persist as they begin to compete for resources. Decoding tasks awaiting GPU execution are subject to increased queuing delays due to ongoing prefill tasks, and vice versa. Prioritized scheduling of one phase risks failing the latency requirements of the other.
|
||||
|
||||
Second, the prefill and decoding computation differ in latency requirements and preference for different forms of parallelism (§3). Colocating prefill and decoding, however, couples their resource allocation, and prevents implementing different parallelism strategies more suited to meeting the specific latency requirements of each phase.
|
||||
|
||||
To overcome these challenges, we propose to disaggregate the prefill and decoding phases of LLM inference, assigning them to separate GPUs. Our approach has two benefits.
|
||||
First, operating each phase independently on different GPUs eliminates prefill-decoding interference. Second, it allows to scale each phase independently with tailored resource allocation and model parallelism strategies to meet their specific latency requirements.
|
||||
Although disaggregation causes communication of intermediate states between GPUs, we show that the communication overhead is insubstantial (§3.3) in modern GPU clusters, and when managed appropriately, disaggregation significantly improves per-GPU goodput.
|
||||
|
||||
Based on the above insights, in this work, we build DistServe 222https://github.com/LLMServe/DistServe, a goodput-optimized LLM serving system by disaggregating the prefill and decoding phases. Given TTFT and TPOT requirements, DistServe first scales each phase independently by co-optimizing the GPU allocation and parallelism strategies of the prefill and decoding phase assuming serving a single model replica. The optimization ensures maximizing the per-GPU goodput and may assign different numbers of GPUs and parallelism strategies to each phase depending on their respective latency requirements. DistServe then scales this allocation to multiple instances via replication until meeting the user-required traffic rate (§4).
|
||||
DistServe also features an algorithm to place the prefill and decoding computation according to their allocation schemes and the cluster’s bandwidth to minimize the overhead of communicating intermediate states between phases.
|
||||
|
||||
We implement DistServe as an orchestration layer on top of the LLM inference engine. We
|
||||
evaluate DistServe on various LLMs, varying the workloads based on three important real-world LLM applications: chatbots, programming assistant, and document summary. Compared to state-of-the-art solutions, DistServe can serve up to 7.4×7.4 × more requests or 12.6×12.6 × tighter SLO under various latency constraints. Our contributions are:
|
||||
|
||||
- •
|
||||
|
||||
Identify the problems of prefill-decoding interference and resource coupling in existing LLM serving systems and propose to disaggregate the two phases.
|
||||
- •
|
||||
|
||||
Design a novel placement algorithm to choose the goodput-optimal schema for prefill and decoding instances automatically.
|
||||
- •
|
||||
|
||||
Conduct a comprehensive evaluation of DistServe with realistic workloads.
|
||||
|
||||
## 4 方法（Method）
|
||||
|
||||
We built DistServe to solve the above challenges. Given the model, workload characteristic, latency requirements, and SLO attainment target, DistServe will determine (a) the parallelism strategies for prefill and decoding instances, (b) the number of each instance type to deploy, as well as (c) how to place them onto the physical cluster. We call the solution a placement. Our goal is to find a placement that maximizes the per-gpu goodput.
|
||||
|
||||
As explained in §3.3, a key design consideration is to manage communications between disaggregated prefill and decoding phases, given varying cluster setups.
|
||||
In this section, we first present two placement algorithms: one for clusters with high-speed cross-node networks (§4.1) and the other for environments lacking such infrastructure (§4.2); the latter introduces additional constraints. We then develop online scheduling optimizations that adapt to the nuances of real-world workloads (§4.3).
|
||||
|
||||
### 4.1 Placement for High Node-Affinity Cluster
|
||||
|
||||
Algorithm 1 High Node-Affinity Placement Algorithm
|
||||
|
||||
LLM G𝐺Gitalic_G, #node limit per-instance N𝑁Nitalic_N, #GPU per-node M𝑀Mitalic_M, GPU memory capacity C𝐶Citalic_C, workload W𝑊Witalic_W, traffic rate R𝑅Ritalic_R.
|
||||
|
||||
the placement 𝑏𝑒𝑠𝑡_𝑝𝑙𝑚.𝑏𝑒𝑠𝑡_𝑝𝑙𝑚{best\_plm}.italic_best _ italic_plm .
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔p,𝑐𝑜𝑛𝑓𝑖𝑔d←∅,∅formulae-sequence←
|
||||
|
||||
subscript𝑐𝑜𝑛𝑓𝑖𝑔𝑝subscript𝑐𝑜𝑛𝑓𝑖𝑔𝑑
|
||||
{config_{p}},{config_{d}},_config start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT , italic_config start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT ← ∅ , ∅
|
||||
|
||||
for 𝑖𝑛𝑡𝑟𝑎_𝑜𝑝∈{1,2,…,M}𝑖𝑛𝑡𝑟𝑎_𝑜𝑝12…𝑀{intra\_op}\{1,2,...,M\}italic_intra _ italic_op ∈ { 1 , 2 , … , italic_M } do
|
||||
|
||||
for 𝑖𝑛𝑡𝑒𝑟_𝑜𝑝∈{1,2,…,N×M𝑖𝑛𝑡𝑟𝑎_𝑜𝑝}𝑖𝑛𝑡𝑒𝑟_𝑜𝑝12…𝑁𝑀𝑖𝑛𝑡𝑟𝑎_𝑜𝑝{inter\_op}\{1,2,...,{N M}{{intra\_op}}\}italic_inter _ italic_op ∈ { 1 , 2 , … , divide start_ARG italic_N × italic_M end_ARG start_ARG italic_intra _ italic_op end_ARG } do
|
||||
|
||||
if G.size𝑖𝑛𝑡𝑒𝑟_𝑜𝑝×𝑖𝑛𝑡𝑟𝑎_𝑜𝑝<Cformulae-sequence𝐺𝑠𝑖𝑧𝑒𝑖𝑛𝑡𝑒𝑟_𝑜𝑝𝑖𝑛𝑡𝑟𝑎_𝑜𝑝𝐶{G.size}{{inter\_op}{intra\_op}}<Cdivide start_ARG italic_G . italic_s italic_i italic_z italic_e end_ARG start_ARG italic_inter _ italic_op × italic_intra _ italic_op end_ARG < italic_C then
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔←(𝑖𝑛𝑡𝑒𝑟_𝑜𝑝,𝑖𝑛𝑡𝑟𝑎_𝑜𝑝)←𝑐𝑜𝑛𝑓𝑖𝑔𝑖𝑛𝑡𝑒𝑟_𝑜𝑝𝑖𝑛𝑡𝑟𝑎_𝑜𝑝{config}({inter\_op},{intra\_op})italic_config ← ( italic_inter _ italic_op , italic_intra _ italic_op )
|
||||
|
||||
G^←parallel(G,𝑐𝑜𝑛𝑓𝑖𝑔)←^𝐺parallel𝐺𝑐𝑜𝑛𝑓𝑖𝑔{G}{parallel}(G,{config})over^ start_ARG italic_G end_ARG ← parallel ( italic_G , italic_config )
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔.𝑔𝑜𝑜𝑑𝑝𝑢𝑡←simu_prefill(G^,W)formulae-sequence𝑐𝑜𝑛𝑓𝑖𝑔←𝑔𝑜𝑜𝑑𝑝𝑢𝑡simu_prefill^𝐺𝑊{config.goodput}{simu\_prefill}({G},W)italic_config . italic_goodput ← simu_prefill ( over^ start_ARG italic_G end_ARG , italic_W )
|
||||
|
||||
if 𝑐𝑜𝑛𝑓𝑖𝑔p.𝑔𝑜𝑜𝑑𝑝𝑢𝑡configp.num_gpus<𝑐𝑜𝑛𝑓𝑖𝑔.𝑔𝑜𝑜𝑑𝑝𝑢𝑡config.num_gpusformulae-sequencesubscript𝑐𝑜𝑛𝑓𝑖𝑔𝑝𝑔𝑜𝑜𝑑𝑝𝑢𝑡formulae-sequence𝑐𝑜𝑛𝑓𝑖subscript𝑔𝑝𝑛𝑢𝑚_𝑔𝑝𝑢𝑠formulae-sequence𝑐𝑜𝑛𝑓𝑖𝑔𝑔𝑜𝑜𝑑𝑝𝑢𝑡formulae-sequence𝑐𝑜𝑛𝑓𝑖𝑔𝑛𝑢𝑚_𝑔𝑝𝑢𝑠{{config_{p}.goodput}}{config_{p}.num\_gpus}<{{config.%
|
||||
goodput}}{config.num\_gpus}divide start_ARG italic_config start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT . italic_goodput end_ARG start_ARG italic_c italic_o italic_n italic_f italic_i italic_g start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT . italic_n italic_u italic_m _ italic_g italic_p italic_u italic_s end_ARG < divide start_ARG italic_config . italic_goodput end_ARG start_ARG italic_c italic_o italic_n italic_f italic_i italic_g . italic_n italic_u italic_m _ italic_g italic_p italic_u italic_s end_ARG then
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔p←𝑐𝑜𝑛𝑓𝑖𝑔←subscript𝑐𝑜𝑛𝑓𝑖𝑔𝑝𝑐𝑜𝑛𝑓𝑖𝑔{config_{p}}{config}italic_config start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ← italic_config
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔.𝑔𝑜𝑜𝑑𝑝𝑢𝑡←simu_decode(G^,W)formulae-sequence𝑐𝑜𝑛𝑓𝑖𝑔←𝑔𝑜𝑜𝑑𝑝𝑢𝑡simu_decode^𝐺𝑊{config.goodput}{simu\_decode}({G},W)italic_config . italic_goodput ← simu_decode ( over^ start_ARG italic_G end_ARG , italic_W )
|
||||
|
||||
if 𝑐𝑜𝑛𝑓𝑖𝑔d.𝑔𝑜𝑜𝑑𝑝𝑢𝑡configd.num_gpus<𝑐𝑜𝑛𝑓𝑖𝑔.𝑔𝑜𝑜𝑑𝑝𝑢𝑡config.num_gpusformulae-sequencesubscript𝑐𝑜𝑛𝑓𝑖𝑔𝑑𝑔𝑜𝑜𝑑𝑝𝑢𝑡formulae-sequence𝑐𝑜𝑛𝑓𝑖subscript𝑔𝑑𝑛𝑢𝑚_𝑔𝑝𝑢𝑠formulae-sequence𝑐𝑜𝑛𝑓𝑖𝑔𝑔𝑜𝑜𝑑𝑝𝑢𝑡formulae-sequence𝑐𝑜𝑛𝑓𝑖𝑔𝑛𝑢𝑚_𝑔𝑝𝑢𝑠{{config_{d}.goodput}}{config_{d}.num\_gpus}<{{config.%
|
||||
goodput}}{config.num\_gpus}divide start_ARG italic_config start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT . italic_goodput end_ARG start_ARG italic_c italic_o italic_n italic_f italic_i italic_g start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT . italic_n italic_u italic_m _ italic_g italic_p italic_u italic_s end_ARG < divide start_ARG italic_config . italic_goodput end_ARG start_ARG italic_c italic_o italic_n italic_f italic_i italic_g . italic_n italic_u italic_m _ italic_g italic_p italic_u italic_s end_ARG then
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔d←𝑐𝑜𝑛𝑓𝑖𝑔←subscript𝑐𝑜𝑛𝑓𝑖𝑔𝑑𝑐𝑜𝑛𝑓𝑖𝑔{config_{d}}{config}italic_config start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT ← italic_config
|
||||
|
||||
n,m←⌈R𝑐𝑜𝑛𝑓𝑖𝑔p.𝑔𝑜𝑜𝑑𝑝𝑢𝑡⌉,⌈R𝑐𝑜𝑛𝑓𝑖𝑔d.𝑔𝑜𝑜𝑑𝑝𝑢𝑡⌉formulae-sequence←
|
||||
|
||||
𝑛𝑚
|
||||
𝑅formulae-sequencesubscript𝑐𝑜𝑛𝑓𝑖𝑔𝑝𝑔𝑜𝑜𝑑𝑝𝑢𝑡𝑅formulae-sequencesubscript𝑐𝑜𝑛𝑓𝑖𝑔𝑑𝑔𝑜𝑜𝑑𝑝𝑢𝑡n,m{R}{{config_{p}.goodput}},{R}{%
|
||||
{config_{d}.goodput}}_n , italic_m ← ⌈ divide start_ARG italic_R end_ARG start_ARG italic_config start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT . italic_goodput end_ARG ⌉ , ⌈ divide start_ARG italic_R end_ARG start_ARG italic_config start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT . italic_goodput end_ARG ⌉
|
||||
|
||||
𝑏𝑒𝑠𝑡_𝑝𝑙𝑚←(n,𝑐𝑜𝑛𝑓𝑖𝑔p,m,𝑐𝑜𝑛𝑓𝑖𝑔d)←𝑏𝑒𝑠𝑡_𝑝𝑙𝑚𝑛subscript𝑐𝑜𝑛𝑓𝑖𝑔𝑝𝑚subscript𝑐𝑜𝑛𝑓𝑖𝑔𝑑{best\_plm}(n,{config_{p}},m,{config_{d}})italic_best _ italic_plm ← ( italic_n , italic_config start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT , italic_m , italic_config start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT )
|
||||
|
||||
return 𝑏𝑒𝑠𝑡_𝑝𝑙𝑚𝑏𝑒𝑠𝑡_𝑝𝑙𝑚{best\_plm}italic_best _ italic_plm
|
||||
|
||||
On high node-affinity clusters equipped with Infiniband, KV caches transmission overhead across nodes is negligible, DistServe can deploy prefill and decoding instances across any two nodes without constraints.
|
||||
We propose a two-level placement algorithm for such scenarios: we first optimize the parallelism configurations for prefill and decoding instances separately to attain phase-level optimal per-gpu goodput; then, we use replication to match the overall traffic rate.
|
||||
|
||||
However, finding the optimal parallel configuration for a single instance type, such as for the prefill instance, is still challenging, due to the lack of a simple analytical formula to calculate the SLO attainment (a.k.a., percentage of requests that meet TTFT requirement), given that the workload has diverse input, output lengths, and irregular arrival patterns. Gauging the SLO via real-testbed profiling is time-prohibitive. We thus resort to building a simulator to estimate the SLO attainment, assuming prior knowledge of the workload’s arrival process and input and output length distributions.
|
||||
Although short-term interval is impossible to predict, the workload pattern over longer timescales (e.g.,
|
||||
hours or days) is often predictable [33, 55]. DistServe fits a distribution from the history request traces and resamples new traces from the distribution as the input workload to the simulator to compute the SLO attainment. Next, DistServe simply enumerates the placements and finds the maximum rate that meets the SLO attainment target with binary search and simulation trials.
|
||||
|
||||
Algorithm 1 outlines the process. We enumerate all feasible parallel configurations, subject to cluster capacity limit, for both prefill and decoding instances. Then, for a specific prefill phase configuration, we use `simu_prefill` to simulate and find its maximum goodput via binary search (similarly for using `simu_decode` for decoding).
|
||||
After determining the optimal parallel configurations for both prefill and decoding instances, we replicate them to achieve the user-required overall traffic rate according to their goodput.
|
||||
|
||||
The complexity of Algorithm 1 is O(NM2)𝑂𝑁superscript𝑀2O(NM^{)italic_O ( italic_N italic_M start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ), with N𝑁Nitalic_N as the node limit per instance and M𝑀Mitalic_M representing the typical number of GPUs per node in modern clusters (e.g., 8). The search space is manageable and the solving time is under 1.3 minutes in our largest setting, as demonstrated in §6.5.
|
||||
|
||||
Simulator building. Algorithm 1 relies on a simulator to estimate the goodput under various SLOs and SLO attainment goals given the workload and the parallelism plan.
|
||||
To build an accurate simulator, we analyze the FLOPs and the number of memory accesses for prefill and decoding phases respectively, and use a latency model to approximate the inference execution time. See details in Appendix A. The simulator aligns well with real profiling results, thanks to the high predictability of DNN workloads [23, 33], verified in §6.4.
|
||||
|
||||
By far, we have developed Algorithm 1 assuming we can place the prefill and decoding instance between any two nodes (or on the same node) of the cluster, and the KV cache transmission utilizes high bandwidth network. In many real clusters, GPUs inside a node access to high-bandwidth NVLINK while GPUs distributed across nodes have limited bandwidth. We next develop an algorithm to address this constraint.
|
||||
|
||||
Algorithm 2 Low Node-Affinity Placement Algorithm
|
||||
|
||||
LLM G𝐺Gitalic_G, #node limit per-instance N𝑁Nitalic_N, #GPU per-node M𝑀Mitalic_M, GPU memory capacity C𝐶Citalic_C, workload W𝑊Witalic_W, traffic rate R𝑅Ritalic_R.
|
||||
|
||||
the placement 𝑏𝑒𝑠𝑡_𝑝𝑙𝑚.𝑏𝑒𝑠𝑡_𝑝𝑙𝑚{best\_plm}.italic_best _ italic_plm .
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔∗←∅←superscript𝑐𝑜𝑛𝑓𝑖𝑔_config start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT ← ∅
|
||||
|
||||
for 𝑖𝑛𝑡𝑒𝑟_𝑜𝑝∈{1,2,…,N}𝑖𝑛𝑡𝑒𝑟_𝑜𝑝12…𝑁{inter\_op}\{1,2,...,N\}italic_inter _ italic_op ∈ { 1 , 2 , … , italic_N } do
|
||||
|
||||
𝒫←get_intra_node_configs(G,M,C,𝑖𝑛𝑡𝑒𝑟_𝑜𝑝)←𝒫get_intra_node_configs𝐺𝑀𝐶𝑖𝑛𝑡𝑒𝑟_𝑜𝑝{P}{get\_intra\_node\_configs}(G,M,C,{inter\_op})caligraphic_P ← get_intra_node_configs ( italic_G , italic_M , italic_C , italic_inter _ italic_op )
|
||||
|
||||
for Pp∈𝒫subscript𝑃𝑝𝒫P_{p}{P}italic_P start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ∈ caligraphic_P do
|
||||
|
||||
for Pd∈𝒫subscript𝑃𝑑𝒫P_{d}{P}italic_P start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT ∈ caligraphic_P do
|
||||
|
||||
if Pp.𝑛𝑢𝑚_𝑔𝑝𝑢𝑠+Pd.𝑛𝑢𝑚_𝑔𝑝𝑢𝑠≤Mformulae-sequencesubscript𝑃𝑝𝑛𝑢𝑚_𝑔𝑝𝑢𝑠subscript𝑃𝑑𝑛𝑢𝑚_𝑔𝑝𝑢𝑠𝑀P_{p}.{num\_gpus}+P_{d}.{num\_gpus} Mitalic_P start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT . italic_num _ italic_gpus + italic_P start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT . italic_num _ italic_gpus ≤ italic_M then
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔←(𝑖𝑛𝑡𝑒𝑟_𝑜𝑝,Pp,Pd)←𝑐𝑜𝑛𝑓𝑖𝑔𝑖𝑛𝑡𝑒𝑟_𝑜𝑝subscript𝑃𝑝subscript𝑃𝑑{config}({inter\_op},P_{p},P_{d})italic_config ← ( italic_inter _ italic_op , italic_P start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT , italic_P start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT )
|
||||
|
||||
G^p,G^d←parallel(G,𝑐𝑜𝑛𝑓𝑖𝑔)←
|
||||
|
||||
subscript^𝐺𝑝subscript^𝐺𝑑
|
||||
parallel𝐺𝑐𝑜𝑛𝑓𝑖𝑔{G}_{p},{G}_{d}{parallel}(G,{config})over^ start_ARG italic_G end_ARG start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT , over^ start_ARG italic_G end_ARG start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT ← parallel ( italic_G , italic_config )
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔.𝑔𝑜𝑜𝑑𝑝𝑢𝑡←simulate(G^p,G^d,W)formulae-sequence𝑐𝑜𝑛𝑓𝑖𝑔←𝑔𝑜𝑜𝑑𝑝𝑢𝑡simulatesubscript^𝐺𝑝subscript^𝐺𝑑𝑊{config.goodput}{simulate}({G}_{p},{G}_{d},W)italic_config . italic_goodput ← simulate ( over^ start_ARG italic_G end_ARG start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT , over^ start_ARG italic_G end_ARG start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT , italic_W )
|
||||
|
||||
if 𝑐𝑜𝑛𝑓𝑖𝑔.∗𝑔𝑜𝑜𝑑𝑝𝑢𝑡𝑐𝑜𝑛𝑓𝑖𝑔.∗𝑛𝑢𝑚_𝑔𝑝𝑢𝑠<𝑐𝑜𝑛𝑓𝑖𝑔.𝑔𝑜𝑜𝑑𝑝𝑢𝑡𝑐𝑜𝑛𝑓𝑖𝑔.𝑛𝑢𝑚_𝑔𝑝𝑢𝑠{{config.^{*}goodput}}{{config.^{*}num\_gpus}}<{%
|
||||
{config.goodput}}{{config.num\_gpus}}divide start_ARG italic_config . start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT italic_goodput end_ARG start_ARG italic_config . start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT italic_num _ italic_gpus end_ARG < divide start_ARG italic_config . italic_goodput end_ARG start_ARG italic_config . italic_num _ italic_gpus end_ARG then
|
||||
|
||||
𝑐𝑜𝑛𝑓𝑖𝑔∗←𝑐𝑜𝑛𝑓𝑖𝑔←superscript𝑐𝑜𝑛𝑓𝑖𝑔𝑐𝑜𝑛𝑓𝑖𝑔{config^{*}}{config}italic_config start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT ← italic_config
|
||||
|
||||
n←⌈R𝑐𝑜𝑛𝑓𝑖𝑔.∗𝑔𝑜𝑜𝑑𝑝𝑢𝑡⌉n{R}{{config.^{*}goodput}}_n ← ⌈ divide start_ARG italic_R end_ARG start_ARG italic_config . start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT italic_goodput end_ARG ⌉
|
||||
|
||||
𝑏𝑒𝑠𝑡_𝑝𝑙𝑚←(n,𝑐𝑜𝑛𝑓𝑖𝑔∗)←𝑏𝑒𝑠𝑡_𝑝𝑙𝑚𝑛superscript𝑐𝑜𝑛𝑓𝑖𝑔{best\_plm}(n,{config^{*}})italic_best _ italic_plm ← ( italic_n , italic_config start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT )
|
||||
|
||||
return 𝑏𝑒𝑠𝑡_𝑝𝑙𝑚𝑏𝑒𝑠𝑡_𝑝𝑙𝑚{best\_plm}italic_best _ italic_plm
|
||||
|
||||
### 4.2 Placement for Low Node-Affinity Cluster
|
||||
|
||||
A straightforward solution is to always colocate prefill and decoding instances on the same node, utilizing the NVLINK, which is commonly available inside a GPU node.
|
||||
For large models, e.g. with 175B parameters (350GB), we may be unable to even host a single pair of prefill and decoding instances in an 8-GPU node (80G×8=640G<350×2GB80𝐺8640𝐺3502𝐺𝐵80G 8=640G<350 2GB80 italic_G × 8 = 640 italic_G < 350 × 2 italic_G italic_B). We incorporate this as additional placement constraints and co-optimize it with model parallelism, presented in Algorithm 2.
|
||||
|
||||
The key insight is that KV cache transfer occurs exclusively between corresponding layers of prefill and decoding instances.
|
||||
Leveraging inter-op parallelism, we group layers into stages and divide each instance into segments, termed as instance segments, with each segment maintaining one specific inter-op stage.
|
||||
By colocating prefill and decoding segments of the same stage within a single node, we force the transfer of intermediate states to occur only via NVLINK. Inside a node, we set the same parallelism and resource allocation for segments of the same instance. Given the typical limitation of GPUs per node (usually 8), we can enumerate possible configurations inside one node and use the simulator to identify the configurations that yield the best goodput.
|
||||
|
||||
As outlined in Algorithm 2, we begin by enumerating inter-op parallelism degrees to get all the possible instance segments. For each segment, we get all possible intra-node parallelism configurations by calling `get_intra_node_configs`. Then we use simulation to find the optimal one and replicate it to satisfy the target traffic rate.
|
||||
|
||||
### 4.3 Online scheduling
|
||||
|
||||
The runtime architecture of DistServe is shown in Figure 6. DistServe operates with a simple FCFS scheduling policy. All incoming requests arrive at a centralized controller, then dispatched to the prefill instance with the shortest queue for prefill processing, followed by dispatch to the least loaded decoding instance for decoding steps. This setup, while simple, is optimized with several key enhancements tailored to the nuances of real-world workloads.
|
||||
|
||||
Reducing pipeline bubbles.
|
||||
To mitigate the pipeline bubbles caused by non-uniform prompt lengths (§3.3), we schedule the requests in a way that balances the execution time across all batches in the pipeline. This is achieved by noting that, for both prefill and decoding instances, the number of new tokens in the batch is a reliable indicator of the batch’s real execution time.
|
||||
For prefill instances, we profile the target model and GPU to figure out the shortest prompt length Lmsubscript𝐿𝑚L_{m}italic_L start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT needed to saturate the GPU. We schedule prefill batches with a total sequence length close to Lmsubscript𝐿𝑚L_{m}italic_L start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT, by either batching multiple requests shorter than Lmsubscript𝐿𝑚L_{m}italic_L start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT or individually scheduling requests longer than Lmsubscript𝐿𝑚L_{m}italic_L start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT. For decoding instances, we set Lmsubscript𝐿𝑚L_{m}italic_L start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT as the largest batch size.
|
||||
|
||||
Combat busrtiness.
|
||||
Burstiness in workloads can cause a deluge of KV caches to transfer from prefill to decoding instances, risking memory overload on decoding instances.
|
||||
To circumvent this, DistServe employs a “pull” method for KV cache transmission rather than a “push” approach – decoding instances fetch KV cache from prefill instances _as needed_, using the GPU memory of prefill instances as a queuing buffer. This way, the prefill instance can continue handling other prefill jobs by simply retaining the KV Cache in the GPU memory after processing the prompt. Hence, each type of instance operates at its own pace without complex coordination.
|
||||
|
||||
|
||||
|
||||
Figure 6: DistServe Runtime System Architecture
|
||||
|
||||
Replaning. The resource and parallelism plan in DistServe is optimized for a specific workload pattern, which may become suboptimal if the workload pattern changes over time. DistServe implement periodic replanning. A workload profiler monitors key parameters such as the average input and output length of the requests, the average arrival rate, etc. If a significant pattern shift is detected, DistServe will trigger a rerun of the placement algorithm based on recent historical data. This process is expedient – the proposed algorithm runs in seconds (§6.5) and reloading LLM weights can be completed within minutes – far shorter than the hourly scale at which real-world workload variations tend to occur.
|
||||
|
||||
Preemption and fault tolerance. DistServe does not implement advanced runtime policies like preemption [26] and fault tolerance [58], which are complementary to disaggregation. Nevertheless, we discuss how they fit into DistServe.
|
||||
In DistServe, the FCFS policy can lead to a “convoy effect”, where longer requests block shorter ones in the prefill stage. Incorporating preemptive strategies, as suggested in existing literature [53], could enhance efficiency and is feasible within our system’s architecture.
|
||||
While not a primary focus in the current DistServe, fault tolerance is a critical aspect for consideration. In traditional colocation- and replication-based systems, a fault in one instance typically does not disrupt other replica instances. However, in DistServe, the dependency between prefill and decoding instances introduces the risk of fault propagation. For example, a fault in a single decoding instance mapped to multiple prefill instances could potentially cripple the entire service and cluster. We leave both as future work.
|
||||
|
||||
## 9 结论（Conclusion）
|
||||
|
||||
We present DistServe, a new LLM serving architecture that disaggregates the prefill and decoding computation. DistServe maximizes the per-gpu goodput – the maximum request rate that can be served adhering to the SLO attainment goal for each GPU provisioned, hence resulting in up to 7.4×7.4 × lower cost per LLM query with guaranteed satisfaction of SLOs.
|
||||
Our findings affirm that as latency becomes an increasingly important metric for LLM services, prefill and decoding disaggregation is a vital strategy in promising improved performance and service quality guarantees.
|
||||
|
||||
Acknowledgments. We sincerely thank our shepherd and
|
||||
the anonymous reviewers for their valuable feedback. This work was
|
||||
supported by the National Natural Science Foundation of China under the grant numbers
|
||||
62172008, 62325201, and the National Natural Science Fund for the Excellent Young Scientists Fund
|
||||
Program (Overseas). Junda Chen is supported by UCSD fellowship and Hao Zhang is supported by UCSD faculty startup fund. Xin Jin is
|
||||
the corresponding author. Yinmin Zhong, Xuanzhe Liu, and Xin Jin are
|
||||
also with the Key Laboratory of High Confidence Software Technologies (Peking
|
||||
University), Ministry of Education.
|
||||
@@ -0,0 +1,91 @@
|
||||
# 📊 文章摘要：DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
|
||||
|
||||
> **原文**：[2024-01-18_DistServe.md](./2024-01-18_DistServe.md)
|
||||
> **原文链接**：https://arxiv.org/abs/2401.09670
|
||||
> **来源**：arXiv
|
||||
> **作者**：Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang（北京大学、StepFun、UC San Diego）
|
||||
> **发布日期**：2024-01-18
|
||||
> **摘要日期**：2026-08-06
|
||||
> **价值评级**：⭐⭐⭐ 高
|
||||
|
||||
---
|
||||
|
||||
## 核心命题
|
||||
|
||||
> **阶段解耦** — 将 prefill 与 decoding 两个计算特性迥异的阶段拆分到独立 GPU 池并各自定制资源与并行策略，以"每 GPU goodput"而非系统总吞吐为优化目标，是 LLM serving 成本优化的范式转折点。
|
||||
|
||||
---
|
||||
|
||||
## 文章概要
|
||||
|
||||
现有 LLM serving 系统（vLLM 等）将 prefill 与 decoding 两阶段放在同一批 GPU 上混合批处理，本文指出这一策略产生两类问题：一是强烈的 prefill-decoding 相互干扰（混合批处理时解码步被 prefill 步拖慢、TTFT 与 TPOT 相互挤压）；二是两阶段被耦合的资源配置与并行策略所绑定，无法各取所需。DistServe 将两阶段分配到不同 GPU，在 TTFT/TPOT 约束下联合优化各阶段的 GPU 数量与并行配置（含高低节点亲和集群两套放置算法），并用"拉取式"KV cache 传输、流水线气泡削减与周期性重规划支撑运行。在多种 LLM 与应用负载下，DistServe 相比当时 SOTA 系统可服务 7.4× 更多请求或满足 12.6× 更严格的 SLO（>90% 请求达标）。局限在于依赖工作负载可预测性（配置搜索基于模拟器）且未实现抢占与容错。
|
||||
|
||||
---
|
||||
|
||||
## 关键要点
|
||||
|
||||
1. **两阶段特性迥异是问题根源** — prefill 是计算密集型单步操作、吃 FLOPs；decoding 是内存带宽受限的多步自回归，延迟敏感。混合批处理必然顾此失彼。`[分类: 共识]`
|
||||
2. **"干扰 + 耦合"双重病因** — 同批共存导致解码步延迟（TPOT 变长）、prefill 排队恶化（TTFT 变长）；且共享权重与显存迫使两阶段用同一并行策略，无法分别优化。`[分类: 范式突破]`
|
||||
3. **解耦收益量化** — 13B 模型单 A100 上，混合部署仅 1.6 rps goodput；分离后 prefill 单 GPU 达 5.6 rps、decoding 达 10 rps，按 2:1 GPU 配比整体 goodput 为 10 rps（每 GPU 3.3 rps），是原方案的 2.1×。`[分类: 共识]`
|
||||
4. **goodput 目标重塑优化函数** — 以"每 GPU 满足 SLO 达标率（90%）的最大请求率"为优化目标，直接对应单次查询成本；相比传统的最大 token 吞吐目标更贴合商业诉求。`[分类: 范式突破]`
|
||||
5. **两套放置算法适配不同网络** — 高节点亲和集群（Infiniband，跨节点 KV cache 传输开销可忽略）用枚举并行配置 + 二进制搜索模拟；低节点亲和集群利用"KV cache 只在对应层之间传输"的特性，将 prefill/decoding 的同层段共置单节点内走 NVLINK，避免跨节点慢速传输。`[分类: 未探索]`
|
||||
6. **"拉取"式 KV cache 传输抗突发** — decoding 实例按需从 prefill 实例拉取 KV cache，prefill 显存充当排队缓冲，避免突发流量压垮 decoding 内存，两类实例无需复杂协调。`[分类: 未探索]`
|
||||
7. **故障传播是新风险** — 解耦引入 prefill↔decoding 实例依赖：单个 decoding 实例故障可能连带多个 prefill 实例、瘫痪整个服务；抢占（如缓解 convoy 效应）与容错均留作未来工作。`[分类: 未探索]`
|
||||
|
||||
---
|
||||
|
||||
## 批判性分析
|
||||
|
||||
### 假设前提
|
||||
|
||||
- 工作负载的到达过程与输入/输出长度分布在较长时间尺度上可预测（论文以小时/天级可预测为前提，用历史 trace 拟合分布驱动配置搜索）。
|
||||
- 现代集群具备足够的跨节点带宽，使解耦通信开销"不实质"（KV cache 传输相对推理时间可忽略）。
|
||||
- 应用对 TTFT 与 TPOT 的要求可明确量化并作为输入给定。
|
||||
- 单模型副本的优化可先于多实例扩展完成（分两步：先求最优单副本配置，再复制满足流量）。
|
||||
|
||||
### 论据与逻辑
|
||||
|
||||
- 论据链条完整：先以 Figure 1 的 13B 单卡实验证明"混合部署 goodput 远低于分阶段独立运行"（1.6 vs 5.6/10 rps），再推导出解耦的必要性；随后用模拟器 + 真实测试床验证配置搜索与 SLO 达标率对齐（论文声称模拟器与真实 profiling 高度吻合）。
|
||||
- 端到端评估覆盖三种真实应用（聊天、编程助手、文档摘要）与多模型，7.4×/12.6× 的收益数字有实验支撑。
|
||||
- 潜在弱点：7.4×/12.6× 是"up to"峰值表述，具体负载下提升幅度不同；解耦收益高度依赖集群网络质量，论文对此的敏感度分析着墨较少。
|
||||
|
||||
### 边界与局限
|
||||
|
||||
- 结论适用于两阶段特性差异明显、SLO 严格的场景；若 TTFT/TPOT 要求宽松，解耦收益会收窄。
|
||||
- 未处理抢占调度与故障容错，且解耦使故障传播范围扩大——生产部署需另行补充。
|
||||
- 配置搜索依赖 workload 预测，负载模式剧烈变化（短时间尺度不可预测）时可能退化为次优。
|
||||
- 评估硬件为 A100 世代，未覆盖 H100/Groq 等后续异构平台（此边界由后续论文补足）。
|
||||
|
||||
---
|
||||
|
||||
## 可引用金句
|
||||
|
||||
> "We find that this strategy not only leads to strong prefill-decoding interferences but also couples the resource allocation and parallelism plans for both phases."
|
||||
> （我们发现，这种策略不仅导致强烈的 prefill-decoding 干扰，还耦合了两个阶段的资源分配与并行计划。）
|
||||
|
||||
> "Our findings affirm that as latency becomes an increasingly important metric for LLM services, prefill and decoding disaggregation is a vital strategy in promising improved performance and service quality guarantees."
|
||||
> （我们的发现证实：随着延迟成为 LLM 服务日益重要的指标，prefill 与 decoding 解耦是带来性能与服务质量的提升的关键策略。）
|
||||
|
||||
---
|
||||
|
||||
## 总体评价
|
||||
|
||||
**亮点**：
|
||||
- 首个系统化论证并实现 prefill/decoding 解耦的 serving 系统之一，直击当时 colocation 共识的软肋，开启后续解耦架构浪潮
|
||||
- "per-GPU goodput"优化目标将系统性能与商业成本直接挂钩，视角独特且可操作
|
||||
- 同时覆盖高/低节点亲和两类集群的放置算法，工程完备度高；KV cache"拉取"机制与流水线气泡消减是实用的系统细节
|
||||
|
||||
**不足**：
|
||||
- 依赖工作负载可预测性做离线配置搜索，在线自适应能力有限
|
||||
- 抢占与容错缺失，且未充分讨论解耦故障传播的工程应对
|
||||
- "up to"峰值收益表述下，对网络带宽敏感度的量化分析不够深入
|
||||
|
||||
**适用场景**：面向 LLM serving 系统设计者、云厂商推理基础设施团队；为理解后续（Mooncake、Splitwise、vLLM 解耦版等）全部解耦系工作提供理论基础。
|
||||
|
||||
**关联建议**：可与同期论文对照阅读——Splitwise（异构硬件视角的阶段拆分）、TetriInfer（混合负载干扰 + 长度预测调度）、Mooncake（KVCache 为中心的规模化生产实践）；后续可追踪 vLLM 官方对 prefill/decode 解耦的采纳与 DistServe 作者后续工作（如 DeepSeek 的 DeepEP/EP 解耦方向）。
|
||||
|
||||
---
|
||||
|
||||
## 配图
|
||||
|
||||
|
||||
Reference in New Issue
Block a user