tech/知识/arXiv/Cunchen_Hu/2024-01-20_TetriInfer.md

# TetriInfer: Inference without Interference - Disaggregate LLM Inference for Mixed Downstream Workloads

> **来源**：arXiv  
> **作者**：Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, Yizhou Shan（等，共 12 位）  
> **发布日期**：2024-01-20  
> **原文链接**：https://arxiv.org/abs/2401.11181

---

## 论文元数据

- **arXiv ID**：2401.11181
- **学科分类**：Distributed, Parallel, and Cluster Computing (cs.DC)
- **作者机构**：中国科学院大学、中科院计算所（ICT, CAS）、华为云（Huawei Cloud）
- **提交历史**：v1: 2024-01-20
- **DOI**：https://doi.org/10.48550/arXiv.2401.11181

---

Cunchen Hu1,2111Work done while intern at Huawei Cloud.,
Heyang Huang1,2,
Liangliang Xu3,
Xusheng Chen3,
Jiang Xu3,
Shuang Chen3,
 
Hao Feng3,
Chenxi Wang1,2,
Sa Wang1,2,
Yungang Bao1,2,
Ninghui Sun1,2,
Yizhou Shan3

1University of Chinese Academy of Sciences, 2ICT, CAS
3Huawei Cloud

## 摘要（Abstract）

Transformer-based large language model (LLM) inference serving is now the backbone of many cloud services.
LLM inference consists of a prefill phase and a decode phase.
However, existing LLM deployment practices often overlook the distinct characteristics of these phases, leading to significant interference.
To mitigate interference, our insight is to carefully schedule and group inference requests based on their characteristics. We realize this idea in TetriInfer through three pillars. First, it partitions prompts into fixed-size chunks so that the accelerator always runs close to its computation-saturated limit. Second, it disaggregates prefill and decode instances so each can run independently. Finally, it uses a smart two-level scheduling algorithm augmented with predicted resource usage to avoid decode scheduling hotspots.
Results show that TetriInfer improves time-to-first-token (TTFT), job completion time (JCT), and inference efficiency in turns of performance per dollar by a large margin, e.g., it uses 38% less resources all the while lowering average TTFT and average JCT by 97% and 47%, respectively.

## 1 引言（Introduction）

Since the boom of ChatGPT, large language model (LLM) based services have now played a vital role in our daily lives[4, 38, 20, 9, 34, 31].
Behind the scenes, all use cases boil down to LLM inference serving. To run an inference request, the LLM model will first take the user inputs to generate the first token (known as the prefill phase), and then generate outputs token-by-token in an auto-regressive manner (known as the decode phase).
Numerous works were proposed to improve the cost efficiency of LLM inference [21, 41].

There are various ways to interact with LLM, from simple chats to more complex downstream tasks such as document summarization, content creation, etc.
As a result, LLM-empowered services serve inference requests with dramatically different properties that can be categorized across two dimensions: the input prompt length during the prefill phase and the generated token length during the decode phase.
As shown in Figure 1, summarization tasks have long input prompts and short generated tokens, while context creation tasks are the opposite.
Token lengths of different downstream tasks can differ by more than two orders of magnitude.
Given the significant variation in LLM inference requests from various downstream tasks, the first research question we ask in this paper is how do these inference requests perform when running together?.

To answer this question, we run extensive tests that mix LLM prefill and decode requests of different lengths.
Unfortunately, we have observed serious interference across all combinations. For example, mixing prefill requests could result in a 10x slowdown, combining prefill and decode requests could lead to a 5x slowdown, and mixing decode requests with different lengths could take a 16% throughput hit (see §2.2).
A naive solution to avoid interference is to provision resources for each downstream task statically. Given the high cost of LLM serving infrastructure, this solution is impractical.
To this end, the second research question we ask in this paper is how to build a distributed LLM inference serving system that minimizes interferences?

We take a step back to examine why interference exists. We find the fundamental issue lies in the fact that current LLM deployment practices do not account for the distinct characteristics exhibited by LLM prefill and decode phases.
Specifically, the prefill phase resembles a computation-heavy batch job, with its computation scaling quadratically with the input prompt length.
The decode phase resembles a memory-intensive, latency-critical task, with its resource usage scaling sublinearly with the generated token length [33].
Interferences observed in our tests are classic system problems.
Running prefill requests leads to a serious slowdown because we continue adding computation-heavy jobs to an already saturated hardware (§2.2.1).
Combining prefill and decode requests hurts both because we co-run batch and latency-critical jobs simultaneously (§2.2.2).
Mixing decode requests leads to a throughput drop because we are unaware of the memory bandwidth and capacity usage, thus leading to contention and head-of-line blocking (§2.2.3).

To solve these issues, our insight is to carefully schedule and group requests based on their characteristics.
We realize this idea in TetriInfer222The name of our system, TetriInfer, implies that it can efficiently organize LLM inference requests, similar to how tetris blocks are stacked., a cloud-scale LLM inference
serving system designed to battle interferences.

Our designs are three-fold.
First, to avoid interference running prefill, we propose limiting the number of tokens processed in a single prefill iteration so that hardware is fully utilized without incurring extra penalties. TetriInfer partitions and pads input prompts into fixed-size chunks so that the accelerator always runs close to its computation-saturated limit (§3.3).
Second, to avoid interference in co-running prefill and decode, we propose disaggregating prefill from decode phases.
TetriInfer has dedicated prefill and decode instances.
During runtime, prefill instances transfer prefilled KV cache to decode instances.
The prefill and decode instances are virtual concepts in that
each can scale independently and flip roles if load changes (§3.5).
Third, to avoid interference running decode requests, we propose using a smart two-level scheduling algorithm augmented with predicted resource usage to avoid scheduling hotspots (§3.4). TetriInfer incorporates an LLM-based length prediction model to speculate the number of generated tokens of decode requests, and then schedule them accordingly.

We implement TetriInfer’s disaggregated prefill and decode instances based on vLLM [21]. Most of our modules are implemented in Python, except for the network stack module, which utilizes C++ to interface with low-level APIs for KV cache transfer. The fine-tuning part uses Trainer APIs offered by HuggingFace Transformer [16]. Since we cannot access high-end hardware, we implement a mock mechanism to emulate varying network bandwidth connecting prefill and decode instances, as illustrated in Figure 9.

We compare TetriInfer with vanilla vLLM using public dataset [35] in terms of time-to-first-token (TTFT), job completion time (JCT), and efficiency as in performance per dollar (perf/$).
We run them atop a real testbed with emulated network bandwidth ranging from 200Gbps to 300GBps.
For light prefill and heavy decode workload, TetriInfer improves perf/$ by 2.4x (Figure 16). For common mixed workload, TetriInfer improves average TTFT and average JCT by 85% and 50%, respectively (Figure 16).
Nevertheless, we also find that TetriInfer’s design is not ideal for heavy prefill and heavy decode workloads since the room for improvement is marginal, and the overhead we introduce cannot be offset (Figure 16).
Overall, our ideas mentioned above are effective.
TetriInfer achieves effective LLM inference serving, outperforming vLLM by a large margin in TTFT, JCT, and perf/$ running most common workloads (§5.1).

![Image 1: Refer to caption](https://arxiv.org/x1.png)

Figure 1: Length Distribution. Prompt Tokens for Prefill and Generated Tokens during Decode. Data sources: conversation [35], summarization [17], writing [18].

## 2 Background and Motivation

We present a brief primer on LLM inference and study interferences while running various LLM inference requests to motivate our work. For model and testbed details, see §5.

### 2.1 Generative LLM Inference

![Image 2: Refer to caption](https://arxiv.org/x2.png)

Figure 2: Prefill and Decode’s Characteristics. Decode’s GPU utilization fluctuates because the task is faster than our monitoring granularity.

LLM inference is a process that involves generating a sequence of output tokens in response to an input prompt. This process consists of two phases: prefill and decode.
The prefill phase outputs the first token and generates the key and value cache (KV cache) for future decoding [21]. The decode phase uses the previous KV cache to generate new tokens step-by-step in an auto-regressive manner.
Generally, the prefill phase is computation-bound, and the decode phase is memory-bound [33].
We report this in Figure 2.
Results indicate that the prefill phase’s throughput stays flat once the accelerator is saturated at a certain number of tokens (which we name the accelerator-saturate threshold).
The decode phase’s throughput continues increasing with a larger batch size but plateaus once the memory bandwidth is saturated.

### 2.2 Motivation: Interference Study

This section studies the impact of running different inference requests concurrently.
Inspired by Figure 1,
we classify inference requests across two dimensions (prefill and decode length) and one property (light or heavy), resulting in four distinct request types:
heavy prefill,
light prefill,
heavy decode, and
light decode.
Here, heavy refers to a long token length, while light refers to a short token length.
Below, we study mixing prefill 

ong decoding tasks across different on-demand decode instances. Each timeline comprises four rounds (R1 to R4), with the length of prefill and decode boxes representing their sequence length and the width of the decode box indicating its resource usage. A wider decode box indicates the presence of lengthy generated tokens, resulting in larger resource usage and decoding latency.
(b) shows TetriInfer’s architecture with four core modules highlighted.

## 3 设计（Design）

### 3.1 Overview

We realize the above insights in TetriInfer, an LLM inference serving system designed to battle interferences.
First, we run prefill in a fixed-size computation unit by partition and pad input prompts into fixed-size chunks such that the accelerator always runs close to its computation-saturated limit (§3.3).
Second, we design instances dedicated to running the prefill or decode phases. We schedule prefill requests to prefill instances only, and the same goes for decode requests. Prefill instances will transfer prefilled KV cache to decode instances.
Our prefill and decode instances are virtual concepts in that each can scale independently and flip roles if load changes (§3.5).
Finally, we design a two-level scheduling algorithm for both prefill and decode request scheduling. We incorporate a length-prediction model to speculate decode requests’ resource usage and then schedule them accordingly (§3.4).

We show TetriInfer’s architecture in Figure 6 (b) with four modules highlighted: centralized control plane, prefill instance, decode instance, and length prediction model.

Centralized control plane.
It consists of a global scheduler and a cluster monitor.
The global scheduler sends requests to prefill instances based on load and receives streaming outputs from decode instances.
The cluster monitor collects statistics from prefill and decode instances and regularly broadcasts load information to prefill instances. It adds, removes, and flips prefill or decodes instances.

Prefill Instances.
They only run the prefill phase of an LLM inference request.
Each prefill instance has a local scheduler, a length predictor, the main LLM engine, and a dispatcher.
All requests undergo four steps.
First, the local prefill scheduler sorts requests based on pre-defined policies.
Second, the length predictor runs a prediction model to speculate the requests’ decode lengths, which are then used to estimate resource usage during the decoding phase.
Third, the main LLM engine partitions all requests into fixed chunks.
Finally, for each request, the dispatcher runs an inter-decode load-balancing algorithm to select a decode instance and then forwards the generated KV cache to it.

Decode instances.
They are virtually disaggregated from prefill instances and only run the decode phase of an LLM inference request.
Each decode instance can receive requests from any prefill instance.
It runs a local scheduler with three pre-defined policies for selecting decode requests to run in the main LLM engine.

Length Prediction Model.
The prediction model is a small LLM model fine-tuned offline for predicting the generation length of LLM inference requests. TetriInfer’s prefill dispatcher and decode instance’s local scheduler utilize the speculated information to schedule decode instances and avoid hotspots measured in §2.2.3. The prediction model is small and deployed at all prefill instances.

### 3.2 Control Plane

TetriInfer has a centralized control plane to
manage inference clusters at the cloud scale.
It consists of a cluster monitor that manages the lifecycle of prefill and decode instances and a global scheduler that managesthe lifecycle of inference requests.
The centralized control plane is a distributed system without a single point of failure or processing bottlenecks.

The cluster monitor is responsible for collecting and broadcasting statistics and scaling instances. Both prefill and decode instances regularly send their load information to the cluster monitor (e.g., every 100 ms). Since we run decentralized decode request scheduling at prefill instances, the cluster monitor will aggregate decode instances’ load information and broadcast it to all prefill instances.

The global scheduler is responsible for forwarding inference requests from external services to prefill instances and sending inference outputs from decode instances back to external services in a streaming fashion.
The global scheduler maintains a request status table, which stores requests’ arrival time, current phase (e.g., prefill or decode), SLA requirement, etc.
When a request arrives, the global scheduler will choose a prefill instance with the least load and then insert the request into the table.
Following our insight to disaggregate prefill and decode instances, the global scheduler only decides which prefill instance will handle the request. It is up to the prefill instance’s dispatcher to decide which decode instances to use with a speculated resource usage.

### 3.3 Prefill Instance

The prefill instance runs the prefill phase of an inference request.
To avoid interference among prefill requests, we use a prefill scheduler and chunked prefill to sort and partition all prompts into fixed-size chunks.
To help avoid interference during the decode phase, we run a length predictor and a decentralized dispatcher to choose decode instances based on speculated resource usage.

#### 3.3.1 Prefill Scheduler

The prefill instance’s scheduler is crucial for improving the prefill phase’s latency and throughput.
The scheduler maintains a raw request queue that stores requests from the global scheduler and a scheduled queue that stores sorted requests.
In this work, we have designed and implemented three scheduler policies: first-come-first-serve (FCFS), shortest-job-first (SJF), and longest-job-first (LJF).
We can use the latter two policies because we can accurately estimate a request’s prefill time based on the number of tokens in its prompt.
We only explore non-preemptive policies, though chunked prefill (described soon) has opened the door to preemptive and out-of-order prefill scheduling, such as shortest-remaining-time-first, which we leave for future work.

The scheduled requests are sent to the length predictor which executes scheduled requests as-is using fixed-size batch (§3.3.2), and the main LLM which uses chunked prefill (§3.3.3).
In Figure 7, we illustrate the above three scheduler policies and how scheduled requests are partitioned and merged into fixed-size chunks.
Specifically, FCFS keeps the original request arrival order.
Prompt tokens are partitioned and merged into chunks sequentially.
This policy is the easiest to implement and works best for inference requests with similar prompt lengths.
However, FCFS can lead to head-of-line blocking and high average job completion time (JCT) when requests have long prompts. This is problematic since the length differences among LLM inference requests are more than three orders of magnitude (see Figure 1).

In response, we add the shortest-job-first
(SJF), and longest-job-first (LJF) to overcome these issues.
These two policies schedule prefill requests based on prompt token lengths in ascending or descending order. By design, they can achieve lower JCT compared to FCFS. Nevertheless, they are no panacea. They introduce starvation for either long or short requests. To avoid starvation, we propose using a prefill scheduling batch (i.e., PrefillSchedBatch) variable to control how many inference requests can be scheduled at a time. For example, assume the raw request queue has twenty requests awaiting scheduling. If we set the batch size to ten, we will schedule twice, each with ten requests sorted and put into the scheduled queue. This simple mechanism prevents starvation during the prefill phase.

Our scheduler is effective. Results in Figure 16 show that SJF lowers average prefill waiting time by 7.8% compared to FCFS when the batch size is set to 16. Additionaly, the improvement is even more pronounced with larger batch sizes.

![Image 8: Refer to caption](https://arxiv.org/x8.png)

Figure 7: Prefill Scheduler Policies. The left shows four raw inference requests (R1 to R4). The right shows scheduled requests using FCFS, SJF, and LJF. We show the chunked version to illustrate slicing and merging (C1 to C4).

#### 3.3.2 Length Predictor

To address the interference cases measured in §2.2.3, it is essential to determine the number of tokens that a decode request is likely to generate. This information will enable us to schedule decode requests in a length-aware manner.
As such, the prefill instance runs a length predictor to predict the length range of an inference request’s generated tokens.
The prefill instance’s dispatcher utilizes this information for inter-decode instance scheduling (§3.3.4), while the decoding instance’s local scheduler employs this information for intra-decode instance scheduling (§3.4).

Our length predictor uses a small LLM-based classification model called a "predict model" to classify the length of generated tokens into fixed-size buckets if the request were executed by a specific target LLM model.
The predict model is intentionally small, containing millions of parameters while the target model is much larger, with billions of parameters. As we run the length predictor at the prefill instance, we aim to minimize its cost and avoid impacting the main LLM model. Therefore, approaches like using a giant LLM to predict length are not feasible for us [48].
Fortunately, a small LLM model is much faster than a giant LLM and uses much less resources.
For example, we use OPT-125M as the predict model and OPT-13B as the target model, the small one is roughly ten times faster than the larger one.

We opt to predict the length range instead of an exact number of tokens because the latter is extremely difficult to predict.
Various inference parameters, such as temperature and top-p [3], result in significant response variations from the same LLM model to the same question in practice. Since our primary goal is to use the estimated length to guide our request scheduling decisions, an exact length estimation is unnecessary; a length range suffices.
For instance, if we estimate the length to be between ten to twenty tokens, we can deduce its resource usage’s lower and upper bounds.

In this work, we have tested two execution modes: a sequential mode, where we first execute the predict model followed by the target model, and a parallel mode, where both models are run simultaneously.
The sequential mode adds extra latency for the target LLM model, while the parallel mode may reduce the target LLM model’s throughput.
Based on our findings in Figure 17, we opted to use the parallel mode because the main LLM is not affected for most requests (more than 80%), though throughput take a 10% hit under extreme stress test.

Figure 8 outlines the offline fine-tuning and online prediction workflow. In this process, the predict model (depicted in red) is trained to speculate the decoding behavior of a specific target model (depicted in blue).
The fine-tuning of the predict model involves three key steps.
Firstly, we assemble a prompt-only training dataset inherited from public datasets, a large target LLM model (e.g., OPT-13B), and a classification model for our predict model (e.g., 125M OPTForSequenceClassification [16]).
Secondly, we send training prompts to the target LLM model, which generates responses.
Subsequently, we categorize the generated responses into fixed-size buckets with a chosen granularity.
For instance, using a granularity of 100, responses with token lengths between 0 to 200 are labeled with 0, 200-400 are labeled with 1, and so on. These labels are paired with the training prompts to create a new dataset. Lastly, we partition the new dataset into a training section and an evaluation section and then proceed to train and evaluate the predict model using this dataset.

The length range granularity plays a crucial role. If set to one, we fall back to predicting an exact number of tokens, which is not practical. If set to target model’s context window size (e.g., 2K), we fall back to no prediction at all and could run into interferences reported in §2.2.1. Intuitively, a smaller granularity means more accurate resource and performance estimation but lower accuracy in practice. A larger granularity means higher accuracy but essentially makes scheduling harder. Regardless of granularity, it’s easy to calculate resource usage’s upper and lower bound but not performance.
In this work, we can predict a granularity of 200 tokens with 74.9% accuracy.
Since improving prediction accuracy is not the focus of this work, we leave it for future work.

![Image 9: Refer to caption](https://arxiv.org/x9.png)

Figure 8: Predict Model’s Fine-tuning and Prediction Flow. The target model is the one that we want to predict its decoding behavior. The predict model is the one we train. This work does not explore online fine-tuning.

Discussions.
We run the length predictor at each prefill instance, hence prefill instances can make well-informed decisions on which decode instances should have enough resources to run certain decoding requests.
Nevertheless, we identify two alternative designs. The first design is to run the length predictor at each decode instance. As a result, the prefill instance can only schedule requests based on the load of decoding instances. However, this design cannot avoid interference cases we measured in §2.2.3. Indeed, one could migrate interference requests among decoding instances at runtime based on predicted length. This would be an overly complex solution. The second design is to run the length predictor at the global scheduler before dispatching requests to refill instances. This design could make the global scheduler a bottleneck. We believe our current design is easier and simpler to reason about and deploy compared to alternatives.

#### 3.3.3 Chunked Prefill

After the prefill scheduler, we concurrently execute the prefill phase of the main LLM alongside the length predictor.
We employ fixed-size chunks for the LLM prefill rather than using fixed batch sizes [21].

As demonstrated in §2.2.1, we observe that as the number of tokens in a prefill iteration increases, the accelerator’s throughput remains constant, while the latency continues to rise after reaching a certain threshold. We refer to this threshold as ChunkSize. Compared to the traditional fixed batch size approach, running prefill in ChunkSize allows for the optimal utilization of accelerators without incurring additional latency penalties. The accelerator and the LLM model architecture determine the ChunkSize. Models with larger hidden dimensions and accelerators with lower capabilities typically result in a smaller ChunkSize. For example, in our test environment, the value is 512 tokens for OPT 13B.

Figure 7 illustrates how chunked prefill wor

ded and two-sided, similar to RDMA’s classification.
Accelerators like GPU or NPU can do one-sided memory access as they have low-level primitives such as direct memory copies between devices [26, 14].

To navigate the complicated physical data links and ensure that TetriInfer can always use the most performant link once deployed, we design a unified network transfer abstraction to utilize the different network stack options listed in Figure 9. The stack exposes APIs such as send, receive, read, write, etc. Our dispatcher calls these APIs to transmit the KV cache to remote decode instances.

Discussion.
We identify two unexplored research questions.
The first question pertains to whether it is beneficial to simultaneously utilize multiple data links for transmitting the KV cache. While this approach could enhance performance, it may also introduce complex control logic.
The second question involves the sender accelerator accessing the memory of the receiver accelerator without involving the receiver’s CPU. This scenario raises typical challenges associated with building large-scale RDMA-based memory systems [10, 12].
Unfortunately, we cannot explore either of these ideas in this wo