文档(金鹏): 2026-08-06 章节 48 篇文章摘要归档
- 46 篇原文+摘要双文件归档（按 来源/作者 分层，复用本地归档 20 篇+新抓取 26 篇） - 即梦生成 9 组主题配图（大图+列表缩略图）存入 知识/金鹏/20260806/ - 章节重组为 9 个主题分组并挂接摘要引用
This commit is contained in:
@@ -0,0 +1,263 @@
|
||||
# TetriInfer: Inference without Interference - Disaggregate LLM Inference for Mixed Downstream Workloads
|
||||
|
||||
> **来源**：arXiv
|
||||
> **作者**：Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, Yizhou Shan（等，共 12 位）
|
||||
> **发布日期**：2024-01-20
|
||||
> **原文链接**：https://arxiv.org/abs/2401.11181
|
||||
|
||||
---
|
||||
|
||||
## 论文元数据
|
||||
|
||||
- **arXiv ID**：2401.11181
|
||||
- **学科分类**：Distributed, Parallel, and Cluster Computing (cs.DC)
|
||||
- **作者机构**：中国科学院大学、中科院计算所（ICT, CAS）、华为云（Huawei Cloud）
|
||||
- **提交历史**：v1: 2024-01-20
|
||||
- **DOI**：https://doi.org/10.48550/arXiv.2401.11181
|
||||
|
||||
---
|
||||
|
||||
Cunchen Hu1,2111Work done while intern at Huawei Cloud.,
|
||||
Heyang Huang1,2,
|
||||
Liangliang Xu3,
|
||||
Xusheng Chen3,
|
||||
Jiang Xu3,
|
||||
Shuang Chen3,
|
||||
|
||||
Hao Feng3,
|
||||
Chenxi Wang1,2,
|
||||
Sa Wang1,2,
|
||||
Yungang Bao1,2,
|
||||
Ninghui Sun1,2,
|
||||
Yizhou Shan3
|
||||
|
||||
1University of Chinese Academy of Sciences, 2ICT, CAS
|
||||
3Huawei Cloud
|
||||
|
||||
## 摘要（Abstract）
|
||||
|
||||
Transformer-based large language model (LLM) inference serving is now the backbone of many cloud services.
|
||||
LLM inference consists of a prefill phase and a decode phase.
|
||||
However, existing LLM deployment practices often overlook the distinct characteristics of these phases, leading to significant interference.
|
||||
To mitigate interference, our insight is to carefully schedule and group inference requests based on their characteristics. We realize this idea in TetriInfer through three pillars. First, it partitions prompts into fixed-size chunks so that the accelerator always runs close to its computation-saturated limit. Second, it disaggregates prefill and decode instances so each can run independently. Finally, it uses a smart two-level scheduling algorithm augmented with predicted resource usage to avoid decode scheduling hotspots.
|
||||
Results show that TetriInfer improves time-to-first-token (TTFT), job completion time (JCT), and inference efficiency in turns of performance per dollar by a large margin, e.g., it uses 38% less resources all the while lowering average TTFT and average JCT by 97% and 47%, respectively.
|
||||
|
||||
## 1 引言（Introduction）
|
||||
|
||||
Since the boom of ChatGPT, large language model (LLM) based services have now played a vital role in our daily lives[4, 38, 20, 9, 34, 31].
|
||||
Behind the scenes, all use cases boil down to LLM inference serving. To run an inference request, the LLM model will first take the user inputs to generate the first token (known as the prefill phase), and then generate outputs token-by-token in an auto-regressive manner (known as the decode phase).
|
||||
Numerous works were proposed to improve the cost efficiency of LLM inference [21, 41].
|
||||
|
||||
There are various ways to interact with LLM, from simple chats to more complex downstream tasks such as document summarization, content creation, etc.
|
||||
As a result, LLM-empowered services serve inference requests with dramatically different properties that can be categorized across two dimensions: the input prompt length during the prefill phase and the generated token length during the decode phase.
|
||||
As shown in Figure 1, summarization tasks have long input prompts and short generated tokens, while context creation tasks are the opposite.
|
||||
Token lengths of different downstream tasks can differ by more than two orders of magnitude.
|
||||
Given the significant variation in LLM inference requests from various downstream tasks, the first research question we ask in this paper is how do these inference requests perform when running together?.
|
||||
|
||||
To answer this question, we run extensive tests that mix LLM prefill and decode requests of different lengths.
|
||||
Unfortunately, we have observed serious interference across all combinations. For example, mixing prefill requests could result in a 10x slowdown, combining prefill and decode requests could lead to a 5x slowdown, and mixing decode requests with different lengths could take a 16% throughput hit (see §2.2).
|
||||
A naive solution to avoid interference is to provision resources for each downstream task statically. Given the high cost of LLM serving infrastructure, this solution is impractical.
|
||||
To this end, the second research question we ask in this paper is how to build a distributed LLM inference serving system that minimizes interferences?
|
||||
|
||||
We take a step back to examine why interference exists. We find the fundamental issue lies in the fact that current LLM deployment practices do not account for the distinct characteristics exhibited by LLM prefill and decode phases.
|
||||
Specifically, the prefill phase resembles a computation-heavy batch job, with its computation scaling quadratically with the input prompt length.
|
||||
The decode phase resembles a memory-intensive, latency-critical task, with its resource usage scaling sublinearly with the generated token length [33].
|
||||
Interferences observed in our tests are classic system problems.
|
||||
Running prefill requests leads to a serious slowdown because we continue adding computation-heavy jobs to an already saturated hardware (§2.2.1).
|
||||
Combining prefill and decode requests hurts both because we co-run batch and latency-critical jobs simultaneously (§2.2.2).
|
||||
Mixing decode requests leads to a throughput drop because we are unaware of the memory bandwidth and capacity usage, thus leading to contention and head-of-line blocking (§2.2.3).
|
||||
|
||||
To solve these issues, our insight is to carefully schedule and group requests based on their characteristics.
|
||||
We realize this idea in TetriInfer222The name of our system, TetriInfer, implies that it can efficiently organize LLM inference requests, similar to how tetris blocks are stacked., a cloud-scale LLM inference
|
||||
serving system designed to battle interferences.
|
||||
|
||||
Our designs are three-fold.
|
||||
First, to avoid interference running prefill, we propose limiting the number of tokens processed in a single prefill iteration so that hardware is fully utilized without incurring extra penalties. TetriInfer partitions and pads input prompts into fixed-size chunks so that the accelerator always runs close to its computation-saturated limit (§3.3).
|
||||
Second, to avoid interference in co-running prefill and decode, we propose disaggregating prefill from decode phases.
|
||||
TetriInfer has dedicated prefill and decode instances.
|
||||
During runtime, prefill instances transfer prefilled KV cache to decode instances.
|
||||
The prefill and decode instances are virtual concepts in that
|
||||
each can scale independently and flip roles if load changes (§3.5).
|
||||
Third, to avoid interference running decode requests, we propose using a smart two-level scheduling algorithm augmented with predicted resource usage to avoid scheduling hotspots (§3.4). TetriInfer incorporates an LLM-based length prediction model to speculate the number of generated tokens of decode requests, and then schedule them accordingly.
|
||||
|
||||
We implement TetriInfer’s disaggregated prefill and decode instances based on vLLM [21]. Most of our modules are implemented in Python, except for the network stack module, which utilizes C++ to interface with low-level APIs for KV cache transfer. The fine-tuning part uses Trainer APIs offered by HuggingFace Transformer [16]. Since we cannot access high-end hardware, we implement a mock mechanism to emulate varying network bandwidth connecting prefill and decode instances, as illustrated in Figure 9.
|
||||
|
||||
We compare TetriInfer with vanilla vLLM using public dataset [35] in terms of time-to-first-token (TTFT), job completion time (JCT), and efficiency as in performance per dollar (perf/$).
|
||||
We run them atop a real testbed with emulated network bandwidth ranging from 200Gbps to 300GBps.
|
||||
For light prefill and heavy decode workload, TetriInfer improves perf/$ by 2.4x (Figure 16). For common mixed workload, TetriInfer improves average TTFT and average JCT by 85% and 50%, respectively (Figure 16).
|
||||
Nevertheless, we also find that TetriInfer’s design is not ideal for heavy prefill and heavy decode workloads since the room for improvement is marginal, and the overhead we introduce cannot be offset (Figure 16).
|
||||
Overall, our ideas mentioned above are effective.
|
||||
TetriInfer achieves effective LLM inference serving, outperforming vLLM by a large margin in TTFT, JCT, and perf/$ running most common workloads (§5.1).
|
||||
|
||||
|
||||
|
||||
Figure 1: Length Distribution. Prompt Tokens for Prefill and Generated Tokens during Decode. Data sources: conversation [35], summarization [17], writing [18].
|
||||
|
||||
## 2 Background and Motivation
|
||||
|
||||
We present a brief primer on LLM inference and study interferences while running various LLM inference requests to motivate our work. For model and testbed details, see §5.
|
||||
|
||||
### 2.1 Generative LLM Inference
|
||||
|
||||
|
||||
|
||||
Figure 2: Prefill and Decode’s Characteristics. Decode’s GPU utilization fluctuates because the task is faster than our monitoring granularity.
|
||||
|
||||
LLM inference is a process that involves generating a sequence of output tokens in response to an input prompt. This process consists of two phases: prefill and decode.
|
||||
The prefill phase outputs the first token and generates the key and value cache (KV cache) for future decoding [21]. The decode phase uses the previous KV cache to generate new tokens step-by-step in an auto-regressive manner.
|
||||
Generally, the prefill phase is computation-bound, and the decode phase is memory-bound [33].
|
||||
We report this in Figure 2.
|
||||
Results indicate that the prefill phase’s throughput stays flat once the accelerator is saturated at a certain number of tokens (which we name the accelerator-saturate threshold).
|
||||
The decode phase’s throughput continues increasing with a larger batch size but plateaus once the memory bandwidth is saturated.
|
||||
|
||||
### 2.2 Motivation: Interference Study
|
||||
|
||||
This section studies the impact of running different inference requests concurrently.
|
||||
Inspired by Figure 1,
|
||||
we classify inference requests across two dimensions (prefill and decode length) and one property (light or heavy), resulting in four distinct request types:
|
||||
heavy prefill,
|
||||
light prefill,
|
||||
heavy decode, and
|
||||
light decode.
|
||||
Here, heavy refers to a long token length, while light refers to a short token length.
|
||||
Below, we study mixing prefill
|
||||
|
||||
ong decoding tasks across different on-demand decode instances. Each timeline comprises four rounds (R1 to R4), with the length of prefill and decode boxes representing their sequence length and the width of the decode box indicating its resource usage. A wider decode box indicates the presence of lengthy generated tokens, resulting in larger resource usage and decoding latency.
|
||||
(b) shows TetriInfer’s architecture with four core modules highlighted.
|
||||
|
||||
## 3 设计（Design）
|
||||
|
||||
### 3.1 Overview
|
||||
|
||||
We realize the above insights in TetriInfer, an LLM inference serving system designed to battle interferences.
|
||||
First, we run prefill in a fixed-size computation unit by partition and pad input prompts into fixed-size chunks such that the accelerator always runs close to its computation-saturated limit (§3.3).
|
||||
Second, we design instances dedicated to running the prefill or decode phases. We schedule prefill requests to prefill instances only, and the same goes for decode requests. Prefill instances will transfer prefilled KV cache to decode instances.
|
||||
Our prefill and decode instances are virtual concepts in that each can scale independently and flip roles if load changes (§3.5).
|
||||
Finally, we design a two-level scheduling algorithm for both prefill and decode request scheduling. We incorporate a length-prediction model to speculate decode requests’ resource usage and then schedule them accordingly (§3.4).
|
||||
|
||||
We show TetriInfer’s architecture in Figure 6 (b) with four modules highlighted: centralized control plane, prefill instance, decode instance, and length prediction model.
|
||||
|
||||
Centralized control plane.
|
||||
It consists of a global scheduler and a cluster monitor.
|
||||
The global scheduler sends requests to prefill instances based on load and receives streaming outputs from decode instances.
|
||||
The cluster monitor collects statistics from prefill and decode instances and regularly broadcasts load information to prefill instances. It adds, removes, and flips prefill or decodes instances.
|
||||
|
||||
Prefill Instances.
|
||||
They only run the prefill phase of an LLM inference request.
|
||||
Each prefill instance has a local scheduler, a length predictor, the main LLM engine, and a dispatcher.
|
||||
All requests undergo four steps.
|
||||
First, the local prefill scheduler sorts requests based on pre-defined policies.
|
||||
Second, the length predictor runs a prediction model to speculate the requests’ decode lengths, which are then used to estimate resource usage during the decoding phase.
|
||||
Third, the main LLM engine partitions all requests into fixed chunks.
|
||||
Finally, for each request, the dispatcher runs an inter-decode load-balancing algorithm to select a decode instance and then forwards the generated KV cache to it.
|
||||
|
||||
Decode instances.
|
||||
They are virtually disaggregated from prefill instances and only run the decode phase of an LLM inference request.
|
||||
Each decode instance can receive requests from any prefill instance.
|
||||
It runs a local scheduler with three pre-defined policies for selecting decode requests to run in the main LLM engine.
|
||||
|
||||
Length Prediction Model.
|
||||
The prediction model is a small LLM model fine-tuned offline for predicting the generation length of LLM inference requests. TetriInfer’s prefill dispatcher and decode instance’s local scheduler utilize the speculated information to schedule decode instances and avoid hotspots measured in §2.2.3. The prediction model is small and deployed at all prefill instances.
|
||||
|
||||
### 3.2 Control Plane
|
||||
|
||||
TetriInfer has a centralized control plane to
|
||||
manage inference clusters at the cloud scale.
|
||||
It consists of a cluster monitor that manages the lifecycle of prefill and decode instances and a global scheduler that managesthe lifecycle of inference requests.
|
||||
The centralized control plane is a distributed system without a single point of failure or processing bottlenecks.
|
||||
|
||||
The cluster monitor is responsible for collecting and broadcasting statistics and scaling instances. Both prefill and decode instances regularly send their load information to the cluster monitor (e.g., every 100 ms). Since we run decentralized decode request scheduling at prefill instances, the cluster monitor will aggregate decode instances’ load information and broadcast it to all prefill instances.
|
||||
|
||||
The global scheduler is responsible for forwarding inference requests from external services to prefill instances and sending inference outputs from decode instances back to external services in a streaming fashion.
|
||||
The global scheduler maintains a request status table, which stores requests’ arrival time, current phase (e.g., prefill or decode), SLA requirement, etc.
|
||||
When a request arrives, the global scheduler will choose a prefill instance with the least load and then insert the request into the table.
|
||||
Following our insight to disaggregate prefill and decode instances, the global scheduler only decides which prefill instance will handle the request. It is up to the prefill instance’s dispatcher to decide which decode instances to use with a speculated resource usage.
|
||||
|
||||
### 3.3 Prefill Instance
|
||||
|
||||
The prefill instance runs the prefill phase of an inference request.
|
||||
To avoid interference among prefill requests, we use a prefill scheduler and chunked prefill to sort and partition all prompts into fixed-size chunks.
|
||||
To help avoid interference during the decode phase, we run a length predictor and a decentralized dispatcher to choose decode instances based on speculated resource usage.
|
||||
|
||||
#### 3.3.1 Prefill Scheduler
|
||||
|
||||
The prefill instance’s scheduler is crucial for improving the prefill phase’s latency and throughput.
|
||||
The scheduler maintains a raw request queue that stores requests from the global scheduler and a scheduled queue that stores sorted requests.
|
||||
In this work, we have designed and implemented three scheduler policies: first-come-first-serve (FCFS), shortest-job-first (SJF), and longest-job-first (LJF).
|
||||
We can use the latter two policies because we can accurately estimate a request’s prefill time based on the number of tokens in its prompt.
|
||||
We only explore non-preemptive policies, though chunked prefill (described soon) has opened the door to preemptive and out-of-order prefill scheduling, such as shortest-remaining-time-first, which we leave for future work.
|
||||
|
||||
The scheduled requests are sent to the length predictor which executes scheduled requests as-is using fixed-size batch (§3.3.2), and the main LLM which uses chunked prefill (§3.3.3).
|
||||
In Figure 7, we illustrate the above three scheduler policies and how scheduled requests are partitioned and merged into fixed-size chunks.
|
||||
Specifically, FCFS keeps the original request arrival order.
|
||||
Prompt tokens are partitioned and merged into chunks sequentially.
|
||||
This policy is the easiest to implement and works best for inference requests with similar prompt lengths.
|
||||
However, FCFS can lead to head-of-line blocking and high average job completion time (JCT) when requests have long prompts. This is problematic since the length differences among LLM inference requests are more than three orders of magnitude (see Figure 1).
|
||||
|
||||
In response, we add the shortest-job-first
|
||||
(SJF), and longest-job-first (LJF) to overcome these issues.
|
||||
These two policies schedule prefill requests based on prompt token lengths in ascending or descending order. By design, they can achieve lower JCT compared to FCFS. Nevertheless, they are no panacea. They introduce starvation for either long or short requests. To avoid starvation, we propose using a prefill scheduling batch (i.e., PrefillSchedBatch) variable to control how many inference requests can be scheduled at a time. For example, assume the raw request queue has twenty requests awaiting scheduling. If we set the batch size to ten, we will schedule twice, each with ten requests sorted and put into the scheduled queue. This simple mechanism prevents starvation during the prefill phase.
|
||||
|
||||
Our scheduler is effective. Results in Figure 16 show that SJF lowers average prefill waiting time by 7.8% compared to FCFS when the batch size is set to 16. Additionaly, the improvement is even more pronounced with larger batch sizes.
|
||||
|
||||
|
||||
|
||||
Figure 7: Prefill Scheduler Policies. The left shows four raw inference requests (R1 to R4). The right shows scheduled requests using FCFS, SJF, and LJF. We show the chunked version to illustrate slicing and merging (C1 to C4).
|
||||
|
||||
#### 3.3.2 Length Predictor
|
||||
|
||||
To address the interference cases measured in §2.2.3, it is essential to determine the number of tokens that a decode request is likely to generate. This information will enable us to schedule decode requests in a length-aware manner.
|
||||
As such, the prefill instance runs a length predictor to predict the length range of an inference request’s generated tokens.
|
||||
The prefill instance’s dispatcher utilizes this information for inter-decode instance scheduling (§3.3.4), while the decoding instance’s local scheduler employs this information for intra-decode instance scheduling (§3.4).
|
||||
|
||||
Our length predictor uses a small LLM-based classification model called a "predict model" to classify the length of generated tokens into fixed-size buckets if the request were executed by a specific target LLM model.
|
||||
The predict model is intentionally small, containing millions of parameters while the target model is much larger, with billions of parameters. As we run the length predictor at the prefill instance, we aim to minimize its cost and avoid impacting the main LLM model. Therefore, approaches like using a giant LLM to predict length are not feasible for us [48].
|
||||
Fortunately, a small LLM model is much faster than a giant LLM and uses much less resources.
|
||||
For example, we use OPT-125M as the predict model and OPT-13B as the target model, the small one is roughly ten times faster than the larger one.
|
||||
|
||||
We opt to predict the length range instead of an exact number of tokens because the latter is extremely difficult to predict.
|
||||
Various inference parameters, such as temperature and top-p [3], result in significant response variations from the same LLM model to the same question in practice. Since our primary goal is to use the estimated length to guide our request scheduling decisions, an exact length estimation is unnecessary; a length range suffices.
|
||||
For instance, if we estimate the length to be between ten to twenty tokens, we can deduce its resource usage’s lower and upper bounds.
|
||||
|
||||
In this work, we have tested two execution modes: a sequential mode, where we first execute the predict model followed by the target model, and a parallel mode, where both models are run simultaneously.
|
||||
The sequential mode adds extra latency for the target LLM model, while the parallel mode may reduce the target LLM model’s throughput.
|
||||
Based on our findings in Figure 17, we opted to use the parallel mode because the main LLM is not affected for most requests (more than 80%), though throughput take a 10% hit under extreme stress test.
|
||||
|
||||
Figure 8 outlines the offline fine-tuning and online prediction workflow. In this process, the predict model (depicted in red) is trained to speculate the decoding behavior of a specific target model (depicted in blue).
|
||||
The fine-tuning of the predict model involves three key steps.
|
||||
Firstly, we assemble a prompt-only training dataset inherited from public datasets, a large target LLM model (e.g., OPT-13B), and a classification model for our predict model (e.g., 125M OPTForSequenceClassification [16]).
|
||||
Secondly, we send training prompts to the target LLM model, which generates responses.
|
||||
Subsequently, we categorize the generated responses into fixed-size buckets with a chosen granularity.
|
||||
For instance, using a granularity of 100, responses with token lengths between 0 to 200 are labeled with 0, 200-400 are labeled with 1, and so on. These labels are paired with the training prompts to create a new dataset. Lastly, we partition the new dataset into a training section and an evaluation section and then proceed to train and evaluate the predict model using this dataset.
|
||||
|
||||
The length range granularity plays a crucial role. If set to one, we fall back to predicting an exact number of tokens, which is not practical. If set to target model’s context window size (e.g., 2K), we fall back to no prediction at all and could run into interferences reported in §2.2.1. Intuitively, a smaller granularity means more accurate resource and performance estimation but lower accuracy in practice. A larger granularity means higher accuracy but essentially makes scheduling harder. Regardless of granularity, it’s easy to calculate resource usage’s upper and lower bound but not performance.
|
||||
In this work, we can predict a granularity of 200 tokens with 74.9% accuracy.
|
||||
Since improving prediction accuracy is not the focus of this work, we leave it for future work.
|
||||
|
||||
|
||||
|
||||
Figure 8: Predict Model’s Fine-tuning and Prediction Flow. The target model is the one that we want to predict its decoding behavior. The predict model is the one we train. This work does not explore online fine-tuning.
|
||||
|
||||
Discussions.
|
||||
We run the length predictor at each prefill instance, hence prefill instances can make well-informed decisions on which decode instances should have enough resources to run certain decoding requests.
|
||||
Nevertheless, we identify two alternative designs. The first design is to run the length predictor at each decode instance. As a result, the prefill instance can only schedule requests based on the load of decoding instances. However, this design cannot avoid interference cases we measured in §2.2.3. Indeed, one could migrate interference requests among decoding instances at runtime based on predicted length. This would be an overly complex solution. The second design is to run the length predictor at the global scheduler before dispatching requests to refill instances. This design could make the global scheduler a bottleneck. We believe our current design is easier and simpler to reason about and deploy compared to alternatives.
|
||||
|
||||
#### 3.3.3 Chunked Prefill
|
||||
|
||||
After the prefill scheduler, we concurrently execute the prefill phase of the main LLM alongside the length predictor.
|
||||
We employ fixed-size chunks for the LLM prefill rather than using fixed batch sizes [21].
|
||||
|
||||
As demonstrated in §2.2.1, we observe that as the number of tokens in a prefill iteration increases, the accelerator’s throughput remains constant, while the latency continues to rise after reaching a certain threshold. We refer to this threshold as ChunkSize. Compared to the traditional fixed batch size approach, running prefill in ChunkSize allows for the optimal utilization of accelerators without incurring additional latency penalties. The accelerator and the LLM model architecture determine the ChunkSize. Models with larger hidden dimensions and accelerators with lower capabilities typically result in a smaller ChunkSize. For example, in our test environment, the value is 512 tokens for OPT 13B.
|
||||
|
||||
Figure 7 illustrates how chunked prefill wor
|
||||
|
||||
ded and two-sided, similar to RDMA’s classification.
|
||||
Accelerators like GPU or NPU can do one-sided memory access as they have low-level primitives such as direct memory copies between devices [26, 14].
|
||||
|
||||
To navigate the complicated physical data links and ensure that TetriInfer can always use the most performant link once deployed, we design a unified network transfer abstraction to utilize the different network stack options listed in Figure 9. The stack exposes APIs such as send, receive, read, write, etc. Our dispatcher calls these APIs to transmit the KV cache to remote decode instances.
|
||||
|
||||
Discussion.
|
||||
We identify two unexplored research questions.
|
||||
The first question pertains to whether it is beneficial to simultaneously utilize multiple data links for transmitting the KV cache. While this approach could enhance performance, it may also introduce complex control logic.
|
||||
The second question involves the sender accelerator accessing the memory of the receiver accelerator without involving the receiver’s CPU. This scenario raises typical challenges associated with building large-scale RDMA-based memory systems [10, 12].
|
||||
Unfortunately, we cannot explore either of these ideas in this wo
|
||||
@@ -0,0 +1,93 @@
|
||||
# 📊 文章摘要：TetriInfer: Inference without Interference - Disaggregate LLM Inference for Mixed Downstream Workloads
|
||||
|
||||
> **原文**：[2024-01-20_TetriInfer.md](./2024-01-20_TetriInfer.md)
|
||||
> **原文链接**：https://arxiv.org/abs/2401.11181
|
||||
> **来源**：arXiv
|
||||
> **作者**：Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, Yizhou Shan 等 12 位（中国科学院大学、中科院计算所、华为云）
|
||||
> **发布日期**：2024-01-20
|
||||
> **摘要日期**：2026-08-06
|
||||
> **价值评级**：⭐⭐⭐ 高
|
||||
|
||||
---
|
||||
|
||||
## 核心命题
|
||||
|
||||
> **消除干扰** — 混合下游任务请求的 prefill/decode 长度差异可达两个数量级，共跑必然相互干扰；按请求特征调度分组（定长 chunk 的 prefill + 阶段解耦 + 长度预测调度）是消除干扰、提升性能/美元比的系统化方案。
|
||||
|
||||
---
|
||||
|
||||
## 文章概要
|
||||
|
||||
TetriInfer 研究"不同下游任务（摘要、创作、对话等）的请求混跑"时的干扰问题：请求的输入 prompt 长度与生成 token 长度差异超两个数量级，混跑会引发严重性能恶化（实测：混跑 prefill 请求 10× 减速、prefill 与 decode 混跑 5× 减速、不同长度 decode 混跑吞吐损失 16%）。作者把干扰根源归结为现有部署忽略了两阶段的本质差异（prefill 是计算密集的批作业，decode 是内存密集的延迟敏感任务），提出三支柱设计：将 prompt 切分为固定大小 chunk 使加速器始终贴近计算饱和点运行；prefill/decode 实例解耦（虚拟实例，可独立扩缩容与角色翻转）；两级调度配合小 LLM 长度预测模型（200 token 粒度预测准确率 74.9%）避免 decode 调度热点。相比 vanilla vLLM：资源使用减少 38%，平均 TTFT 与 JCT 分别降低 97% 与 47%；轻 prefill 重 decode 负载下性能/美元比提升 2.4×。局限：重 prefill + 重 decode 负载下收益边际化且开销无法抵消，且评估受限于模拟网络带宽与 OPT-13B 等旧模型。
|
||||
|
||||
---
|
||||
|
||||
## 关键要点
|
||||
|
||||
1. **混合负载干扰是真实且严重的问题** — 实测数据：混跑 prefill 请求 10× 减速（向已饱和硬件持续添加计算密集作业）、prefill+decode 混跑 5× 减速（批作业与延迟敏感任务共跑）、不同长度 decode 混跑 16% 吞吐损失（内存带宽/容量竞争与队头阻塞）。`[分类: 范式突破]`
|
||||
2. **按特征分组而非统一处理** — 核心洞见：不应把所有请求当同类处理，而应按 prefill/decode 长度（重/轻两维）分类调度；系统名 TetriInfer 即喻意像俄罗斯方块一样组织请求。`[分类: 范式突破]`
|
||||
3. **定长 chunk 的 prefill** — 加速器在 token 数达到饱和阈值后吞吐不再增长、延迟却继续上升；把 prompt 切成固定大小 chunk（OPT-13B 上为 512 token）使硬件始终贴近计算饱和点，避免额外延迟惩罚。`[分类: 范式突破]`
|
||||
4. **虚拟解耦实例** — prefill/decode 实例是"虚拟概念"：各自独立扩缩容、负载变化时可角色翻转，兼顾解耦隔离与资源弹性。`[分类: 共识]`
|
||||
5. **小 LLM 预测生成长度** — 用百万参数级的预测模型（OPT-125M）对十亿级目标模型（OPT-13B）的生成长度做分桶分类预测（而非精确预测），200 token 粒度准确率 74.9%；并行执行模式下 80% 以上的请求主模型不受影响，极端压测下吞吐损失 10%。`[分类: 未探索]`
|
||||
6. **两级调度避免热点** — prefill 实例的调度器选择 decode 实例时用预测资源占用做负载均衡，decode 实例内部再用长度感知策略调度，避免 §2.2.3 测得的 decode 热点问题。`[分类: 未探索]`
|
||||
7. **量化收益** — 相比 vLLM：资源减少 38%、平均 TTFT 降低 97%、平均 JCT 降低 47%；轻 prefill 重 decode 负载 perf/$ 提升 2.4×，常见混合负载 TTFT/JCT 改善 85%/50%。`[分类: 共识]`
|
||||
8. **诚实标注不适场景** — 重 prefill + 重 decode 负载下设计不理想：改进空间边际化、引入的开销无法抵消——这是少见的对自身方案适用边界的明确声明。`[分类: 争议]`
|
||||
|
||||
---
|
||||
|
||||
## 批判性分析
|
||||
|
||||
### 假设前提
|
||||
|
||||
- 下游任务请求的输入/输出长度分布差异显著且可测量、可分类（重/轻两维），且这种分类足以指导调度。
|
||||
- 用一个小模型（百万参数）预测大模型（十亿参数）的生成长度范围是可行的（以固定粒度分桶），且并行运行小模型不显著影响主模型吞吐。
|
||||
- 干扰现象（10×/5×/16%）在目标生产环境复现，且模拟网络带宽（200Gbps-300GBps）能代表真实数据中心网络。
|
||||
- 集中式控制面（全局调度器 + 集群监控器）不会成为云规模瓶颈（论文称其为无单点的分布式系统）。
|
||||
|
||||
### 论据与逻辑
|
||||
|
||||
- 干扰测量的数据（10×/5×/16%）是本文论据的地基，直接驱动设计决策，链条清晰；三支柱设计各自针对一种干扰类型，映射关系明确。
|
||||
- 端到端收益（38% 资源、97% TTFT、47% JCT）与摘要结论一致，且明确区分了"轻 prefill 重 decode"（2.4× perf/$）与"常见混合负载"（85%/50%）等不同语境。
|
||||
- 弱点：论文基于 OPT-13B/OPT-125M 等较旧模型，未在 GPT-4 级别或 MoE 模型上验证；评估在模拟带宽与模拟环境下进行（作者明示无法访问高端硬件）；所下载原文缺失完整实验章节，部分收益数字无法在原文内交叉核对（以摘要与引言为准）。
|
||||
|
||||
### 边界与局限
|
||||
|
||||
- 明确不适用场景：重 prefill + 重 decode 负载（收益无法抵消开销）。
|
||||
- 长度预测粒度（200 token、74.9% 准确率）意味着对长度分布接近粒度边界的请求，调度决策可能偏差。
|
||||
- 依赖小模型与目标模型行为的一致性；换目标模型需重新微调预测模型。
|
||||
- 未探索在线微调预测模型、可抢占/乱序 prefill 调度（chunked prefill 已打开该可能，留作未来工作）。
|
||||
|
||||
---
|
||||
|
||||
## 可引用金句
|
||||
|
||||
> "We find the fundamental issue lies in the fact that current LLM deployment practices do not account for the distinct characteristics exhibited by LLM prefill and decode phases."
|
||||
> （我们发现根本问题在于：当前的 LLM 部署实践没有考虑 prefill 与 decode 阶段各自迥异的特性。）
|
||||
|
||||
> "We take a step back to examine why interference exists. We find the fundamental issue lies in the fact that current LLM deployment practices do not account for the distinct characteristics exhibited by LLM prefill and decode phases."
|
||||
> （我们退一步审视干扰为何存在，发现根本问题在于现有部署实践忽视了 prefill 与 decode 阶段的本质差异——prefill 像计算密集的批作业，decode 像内存密集的延迟敏感任务。）
|
||||
|
||||
---
|
||||
|
||||
## 总体评价
|
||||
|
||||
**亮点**：
|
||||
- 首个系统量化"混合下游任务干扰"的工作，10×/5×/16% 的干扰数据直观有力
|
||||
- 长度预测驱动的调度是独特贡献，用分桶分类规避了精确预测的不可能，工程务实
|
||||
- 定长 chunk prefill 与虚拟解耦实例的设计简洁可落地，且明确声明适用边界（重+重负载不适用），学术诚实度高
|
||||
- 与 DistServe、Splitwise 同期（2023.11-2024.1）独立验证了解耦思路，并补充了干扰测量与预测调度两个新维度
|
||||
|
||||
**不足**：
|
||||
- 评估环境受限（模拟带宽、旧模型 OPT-13B），无真实生产部署验证
|
||||
- 预测模型需要随目标模型重新微调，落地成本未量化
|
||||
- 下载版原文缺失实验与结论章节，部分细节只能依赖摘要与引言
|
||||
|
||||
**适用场景**：多任务混合流量（对话 + 摘要 + 创作等）的 serving 系统设计者；研究 LLM serving 干扰表征、长度预测调度的研究人员；云厂商推理平台团队。
|
||||
|
||||
**关联建议**：与 DistServe（阶段解耦的 goodput 理论）、Splitwise（异构硬件）、Mooncake（生产系统 KVCache 中心化）构成解耦 serving 四篇同期代表作，可联合精读；"输出长度预测"方向后续可关注 speculative decoding、MoE 路由预测等相关工作。
|
||||
|
||||
---
|
||||
|
||||
## 配图
|
||||
|
||||
|
||||
Reference in New Issue
Block a user