文档(金鹏): 2026-08-06 章节 48 篇文章摘要归档
- 46 篇原文+摘要双文件归档（按 来源/作者 分层，复用本地归档 20 篇+新抓取 26 篇） - 即梦生成 9 组主题配图（大图+列表缩略图）存入 知识/金鹏/20260806/ - 章节重组为 9 个主题分组并挂接摘要引用
This commit is contained in:
@@ -0,0 +1,276 @@
|
||||
# Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
|
||||
|
||||
> **来源**：arXiv
|
||||
> **作者**：Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu
|
||||
> **发布日期**：2024-06-24
|
||||
> **原文链接**：https://arxiv.org/abs/2407.00079
|
||||
|
||||
---
|
||||
|
||||
## 论文元数据
|
||||
|
||||
- **arXiv ID**：2407.00079
|
||||
- **学科分类**：Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Hardware Architecture (cs.AR)
|
||||
- **作者机构**：月之暗面（Moonshot AI）、清华大学
|
||||
- **提交历史**：v1: 2024-06-24；v2: 2024-07-02；v3: 2024-07-09；v4: 2025-09-03
|
||||
- **DOI**：https://doi.org/10.48550/arXiv.2407.00079
|
||||
|
||||
---
|
||||
|
||||
11脚注: Ruoyu Qin’s part of work done as an intern at Moonshot AI, contributed equally with Zheming Li.22脚注: Corresponding to zhang_mingxing@mail.tsinghua.edu.cn, xuxinran@moonshot.ai.
|
||||
|
||||
Ruoyu Qin♠♡1 Zheming Li♠1 Weiran He♠
|
||||
|
||||
&Mingxing Zhang♡2 Yongwei Wu♡ Weimin Zheng♡ Xinran Xu♠2
|
||||
|
||||
♠Moonshot AI ♡Tsinghua University
|
||||
|
||||
## 摘要（Abstract）
|
||||
|
||||
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI. It features a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters. It also leverages the underutilized CPU, DRAM, and SSD resources of the GPU cluster to implement a disaggregated cache of KVCache. The core of Mooncake is its KVCache-centric scheduler, which balances maximizing overall effective throughput while meeting latency-related Service Level Objectives (SLOs). Unlike traditional studies that assume all requests will be processed, Mooncake faces challenges due to highly overloaded scenarios. To mitigate these, we developed a prediction-based early rejection policy. Experiments show that Mooncake excels in long-context scenarios. Compared to the baseline method, Mooncake can achieve up to a 525% increase in throughput in certain simulated scenarios while adhering to SLOs. Under real workloads, Mooncake’s innovative architecture enables Kimi to handle 75% more requests.
|
||||
|
||||
## 1 引言（Introduction）
|
||||
|
||||
### 1.1 Motivation of Developing Mooncacke
|
||||
|
||||
With the rapid adoption of large language models (LLMs) in various scenarios [1, 2, 3, 4], the workloads for LLM serving have become significantly diversified. These workloads differ in input/output length, frequency and distribution of arrival, and, most importantly, demand different kinds of Service Level Objectives (SLOs). As a Model as a Service (MaaS) provider, one of the primary goals of Kimi [5] is to solve an optimization problem with multiple complex constraints. The optimization goal is to maximize overall effective throughput, which directly impacts revenue, while the constraints reflect varying levels of SLOs. These SLOs typically involve meeting latency-related requirements, mainly the time to first token (TTFT) and the time between tokens (TBT).
|
||||
|
||||
|
||||
|
||||
Figure 1: Mooncake Architecture.
|
||||
|
||||
To achieve this goal, a prerequisite is to make the best use of the various kinds of resources available in the GPU cluster.
|
||||
Specifically, although GPU servers are currently provided as highly integrated nodes (e.g., DGX/HGX supercomputers [6]), it is necessary to decouple and restructure them into several disaggregated resource pools, each optimized for different but collaborative goals. For example, many researchers [7, 8, 9] have suggested separating prefill servers from decoding servers because these two stages of LLM serving have very different computational characteristics, in which the KVCache shifts with requests moving from prefill to decoding servers.
|
||||
|
||||
Building on this idea, we found that the scheduling of KVCache is central to LLM serving scheduling. To improve overall throughput, there are typically two general approaches: 1) reuse KVCache as much as possible to reduce the required computation resources; and 2) maximize the number of tokens in each batch to improve the Model FLOPs Utilization (MFU). However, reusing KVCache from a remote location will prolong the TTFT, and a large batch size will lead to a larger TBT. Thus, the utilization of both these throughput-oriented optimizations may lead to violations of latency-related SLOs.
|
||||
|
||||
According to the above guidelines, we propose a disaggregated design that is centered around KVCache for scheduling and optimization. Figure 1 presents our current KVCache-centric disaggregated architecture for LLM serving, named Mooncake. For each request, the global scheduler (Conductor) needs to select a pair of prefill and decoding instances and schedule the request in the following steps: 1) transfer as much reusable KVCache as possible to the selected prefill instance; 2) complete the prefill stage in chunks/layers and continuously stream the output KVCache to the corresponding decoding instance; 3) load the KVCache and add the request to the continuous batching process at the decoding instance for generating request outputs.
|
||||
|
||||
Although this process seems straightforward, the selection policy is complex due to many restrictions. In the prefill stage, the main objective is to reuse the KVCache as much as possible to avoid redundant computation. However, waiting for KVCache stored on lower-tier storage may violate the TTFT SLO. Additionally, high demand on the KVCache server can lead to network congestion, prolonging the waiting time.
|
||||
Thus Conductor is also responsible for predicting the future usage of KVCache blocks and executing scheduling operations such as swapping and replication accordingly.
|
||||
The hottest blocks should be replicated to multiple nodes to avoid fetching congestion, while the coldest ones should be swapped out to reduce reserving costs.
|
||||
Prefill scheduling is also constrained by the availability of DRAM space in the prefill node, especially when much of the memory is reserved for the global KVCache pool.
|
||||
|
||||
In contrast, the decoding stage has different optimization goals and constraints. The aim is to aggregate as many tokens as possible in a decoding batch to improve MFU.
|
||||
However, this objective is restricted not only by the TBT SLO but also by the total size of the aggregated KVCache that can be contained in the VRAM.
|
||||
|
||||
More importantly, existing research on LLM serving assumes sufficient resources and focuses on improving resource utilization. In contrast, the current GPU/accelerator supply is limited, and many MaaS providers face severe overload problems, especially during peak times. Scheduling in such scenarios presents unique challenges that existing works have not explored. For example, we need to predict future loads and reject certain requests early if there will be no available decoding slots after the prefill stage, to save wasted computation resources.
|
||||
However, a straightforward implementation of such an early reject policy surprisingly leads to fluctuations in the overloads. This has led us to aim at predicting the generation length of specific queries and making overall load predictions in the short-term future to implement a better rejection policy. It is also necessary to classify different request priorities to implement priority-based scheduling.
|
||||
In this paper, we summarize these problems as overload-oriented scheduling and present our preliminary study results.
|
||||
|
||||
### 1.2 Design and Results of Mooncacke
|
||||
|
||||
In the following sections of this paper, we first present an overview of Mooncake’s architecture, including its main components and the typical workflow for processing a request (§3). Then, we describe the main design choices made during its implementation, especially those not covered in current research.
|
||||
|
||||
First, in §5, we discuss how to implement a separate prefill node pool that seamlessly handles the dynamic distribution of context length. We employ a chunked pipeline parallelism (CPP) mechanism to scale the processing of a single request across multiple nodes, which is necessary for reducing the TTFT of long-context inputs. Compared to traditional sequence parallelism (SP) based solutions, CPP reduces network consumption and simplifies the reliance on frequent elastic scaling. This mechanism is further supplemented with layer-wise prefill that enables stream transferring of KVCache to overlap latency.
|
||||
|
||||
Next, in §6, we detail our KVCache-centric request scheduling algorithm, which balances instance loads and user experience as measured by TTFT and TBT SLOs. This includes a heuristic-based automated hot-spot migration scheme that replicates hot KVCache blocks without requiring precise predictions of future KVCache usage. Experimental results show that our cache-aware scheduling can significantly lower TTFT in real-world scenarios. In end-to-end experiments using public datasets, simulated data, and real workloads, Mooncake excels in long-context scenarios. Compared to the baseline method, Mooncake can achieve up to a 525% increase in throughput while meeting SLOs. Under real workloads, Mooncake enables Kimi to handle 75% more requests.
|
||||
|
||||
Finally, unlike existing work on LLM serving that assumes all requests will be processed, Mooncake consistently faces overload due to Kimi’s rapid growth in user requests. Thus, Mooncake’s scheduling involves determining whether to accept or reject incoming requests based on the system load. In §7, we discuss our implementation of a unique early rejection policy that reduces wasted computational resources in overloaded scenarios. We further explore the load fluctuation problem caused by straightforward early rejection and how predicting future load can mitigate this issue.
|
||||
|
||||
Mooncake is currently the primary platform for serving Kimi and has successfully handled exponential workload growth, proving its effectiveness in scaling out to large and highly overloaded workloads. However, many more problems need to be explored, and these future directions are also included in the paper.
|
||||
|
||||
To protect proprietary information and facilitate reproducibility, all the experimental results reported in this paper are based on replayed traces of real workloads, but using a dummy model that follows the same architecture as LLaMA2-70B. The trace includes only the timing of request arrivals, the number of input tokens, and the number of output tokens, the remaped block hash, without any real user content. The trace is open-sourced at https://github.com/kvcache-ai/Mooncake.
|
||||
|
||||
## 2 Preliminary and Problem Definition
|
||||
|
||||
Modern large language models (LLMs) are based on the Transformer architecture, which utilizes attention mechanisms and multilayer perceptrons (MLPs) to process input. Popular Transformer-based models, such as GPT [10] and LLaMA [11], employ a decoder-only structure. Each inference request is logically divided into two stages: the prefill stage and the decoding stage.
|
||||
|
||||
|
||||
|
||||
Figure 2: Normalized throughput and latency of prefill and decoding stages with different sequence lengths or batch sizes for the dummy
|
||||
LLaMA2-70B model.
|
||||
|
||||
In the prefill stage, all input tokens are processed in parallel. This stage generates the first output token while storing intermediate results of computed keys and values, referred to as the KVCache. The decoding stage then uses this KVCache to autoregressively generate new tokens, adding new keys and values from the computation to the KVCache. The ability to process input tokens simultaneously in the prefill stage typically makes it computationally intensive, except for short requests. Since the computational complexity of attention networks scales quadratically with input length while the complexity of MLP scales linearly, computation time in the prefill stage generally increases superlinearly with input length, as shown in the left part of Figure 2.
|
||||
|
||||
In contrast, the decoding stage processes only one token at a time per batch due to the limitation of autoregressive generation. This makes it memory-constrained and causes computation time to increase sublinearly with batch size, as shown in the right part of Figure 2. A widely used optimization in the decoding stage is continuous batching [12, 13]. Before each iteration, the scheduler checks the status of all requests, adding newly arrived requests to the batch’s prefill stage while
|
||||
|
||||
ng contexts, bringing no significant overhead for short context prefill and avoiding frequent dynamic adjustment of node partitioning.
|
||||
This pipeline-based acceleration method has been explored in training systems [24], but to our knowledge, this is the first application in the inference stage, as long context inference has only recently emerged.
|
||||
|
||||
### 5.2 Layer-wise Prefill
|
||||
|
||||
Beyond computational power, the limited size of VRAM is also a precious resource, and we aim to minimize the VRAM occupation by states, primarily the KVCache.
|
||||
Theoretically, if the KVCache size of a request is SS and the processing time is TT, its occupation cost is S∗TS*T.
|
||||
If a request is chunked and the processing of each chunk is inlined with other decoding requests in chunked prefill, TT will increase, leading to a larger occupation cost.
|
||||
|
||||
|
||||
|
||||
Figure 7: Latency of storing KVCache of different request lengths (Layer-wise latency refers to the difference in latency between Layer-wise Prefill and Prefill without storing KVCache).
|
||||
|
||||
Moreover, since prefill is processed layer-by-layer and is computation-bound, it is possible to overlap the transferring and dumping of KVCache with computation, further reducing its occupation cost.
|
||||
In Mooncake, KVCache loading and storing are executed asynchronously via launch and wait operations. Before each layer’s attention computation begins, the model waits for the asynchronous loading of that layer’s KVCache to complete and triggers the next layer’s asynchronous KVCache loading. After the attention calculation is complete, asynchronous storage of that layer’s KVCache is launched. Once all layers’ computations are finished, the process waits for the completion of all asynchronous storage operations. Transfer overlapping allows the prefill instance’s execution time to be roughly equivalent to either the KVCache loading time or the standard prefilling time, depending on the prefix cache proportion relative to the input length. The experimental result of KVCache storing latency, as shown in Figure 7, demonstrates that the layer-wise prefill can effectively reduce the latency for long-context requests.
|
||||
|
||||
The main advantage of this overlap effectiveness is that it enables us to disregard the available VRAM size in prefill scheduling, as long as it can contain a single request.
|
||||
As shown in Figure 1, the scheduling of prefill nodes only considers the KVCache distribution and the available DRAM size.
|
||||
|
||||
In the future, we intend to explore more uses for this free VRAM. For example, OpenAI recently proposed the use of batch APIs [25], which enable users to send asynchronous groups of requests at 50% lower costs, but with only a clear 24-hour turnaround time. This service is ideal for processing jobs that do not require immediate responses. Since there is no stringent TBT for these batch requests, we can inline even the decoding stage of these requests into prefill processing for better MFU, if there is enough VRAM space to hold the corresponding KVCache.
|
||||
|
||||
## 6 以 KV Cache 为中心的调度
|
||||
|
||||
In this section, we mainly discuss how Conductor schedules the requests and KVCache blocks under normal conditions, leaving the discussion on overload scenarios for the next section.
|
||||
|
||||
Algorithm 1 KVCache-centric Scheduling Algorithm
|
||||
|
||||
1:prefill instance pool PP, decoding instance pool DD, request RR, cache block size BB.
|
||||
|
||||
2:the prefill and decoding instances (p,d)(p,d) to process RR.
|
||||
|
||||
3:𝑏𝑙𝑜𝑐𝑘_𝑘𝑒𝑦𝑠←PrefixHash(R.𝑝𝑟𝑜𝑚𝑝𝑡_𝑡𝑜𝑘𝑒𝑛𝑠,B){block\_keys}{PrefixHash}(R.{prompt\_tokens},B)
|
||||
|
||||
4:𝑇𝑇𝐹𝑇←inf{TTFT}
|
||||
|
||||
5:p←∅p
|
||||
|
||||
6:𝑏𝑒𝑠𝑡_𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛,𝑏𝑒𝑠𝑡_𝑚𝑎𝑡𝑐ℎ𝑒𝑑_𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒←FindBestPrefixMatch(P,𝑏𝑙𝑜𝑐𝑘_𝑘𝑒𝑦𝑠){best\_prefix\_len},{best\_matched\_instance}{FindBestPrefixMatch}(P,{block\_keys})
|
||||
|
||||
7:for 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒∈P{instance} P do
|
||||
|
||||
8: 𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛←𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒.𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛{prefix\_len}{instance.prefix\_len}
|
||||
|
||||
9: T𝑞𝑢𝑒𝑢𝑒←EstimatePrefillQueueTime(𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒){T_{queue}}{EstimatePrefillQueueTime}({instance})
|
||||
|
||||
10: if 𝑏𝑒𝑠𝑡_𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛<𝐤𝐯𝐜𝐚𝐜𝐡𝐞_𝐛𝐚𝐥𝐚𝐧𝐜𝐢𝐧𝐠_𝐭𝐡𝐫𝐞𝐬𝐡𝐨𝐥𝐝{{best\_prefix\_len}}{{prefix\_len}}<{ kvcache\_balancing\_threshold} then
|
||||
⊳ Cache-aware prefill scheduling
|
||||
|
||||
11: T𝑝𝑟𝑒𝑓𝑖𝑙𝑙←EstimatePrefillExecutionTime(len(R.𝑝𝑟𝑜𝑚𝑝𝑡_𝑡𝑜𝑘𝑒𝑛𝑠),𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛){T_{prefill}}{EstimatePrefillExecutionTime}({len}(R.{prompt\_tokens}),{prefix\_len})
|
||||
|
||||
12: if 𝑇𝑇𝐹𝑇>T𝑞𝑢𝑒𝑢𝑒+T𝑝𝑟𝑒𝑓𝑖𝑙𝑙{TTFT}>{T_{queue}}+{T_{prefill}} then
|
||||
|
||||
13: 𝑇𝑇𝐹𝑇←T𝑞𝑢𝑒𝑢𝑒+T𝑝𝑟𝑒𝑓𝑖𝑙𝑙{TTFT}{T_{queue}}+{T_{prefill}}
|
||||
|
||||
14: p←𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒p{instance}
|
||||
|
||||
15: end if
|
||||
|
||||
16: else⊳ Cache-aware and -balancing prefill scheduling
|
||||
|
||||
17: 𝑡𝑟𝑎𝑛𝑠𝑓𝑒𝑟_𝑙𝑒𝑛←𝑏𝑒𝑠𝑡_𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛−𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛{transfer\_len}{best\_prefix\_len}-{prefix\_len}
|
||||
|
||||
18: T𝑡𝑟𝑎𝑛𝑠𝑓𝑒𝑟←EstimateKVCacheTransferTime(𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒,𝑏𝑒𝑠𝑡_𝑚𝑎𝑡𝑐ℎ𝑒𝑑_𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒,𝑡𝑟𝑎𝑛𝑠𝑓𝑒𝑟_𝑙𝑒𝑛){T_{transfer}}{EstimateKVCacheTransferTime}({instance},{best\_matched\_instance},{transfer\_len})
|
||||
|
||||
19: T𝑝𝑟𝑒𝑓𝑖𝑙𝑙←EstimatePrefillExecutionTime(len(R.𝑝𝑟𝑜𝑚𝑝𝑡_𝑡𝑜𝑘𝑒𝑛𝑠),𝑏𝑒𝑠𝑡_𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛){T_{prefill}}{EstimatePrefillExecutionTime}({len}(R.{prompt\_tokens}),{best\_prefix\_len})
|
||||
|
||||
20: if 𝑇𝑇𝐹𝑇>T𝑡𝑟𝑎𝑛𝑠𝑓𝑒𝑟+T𝑞𝑢𝑒𝑢𝑒+T𝑝𝑟𝑒𝑓𝑖𝑙𝑙{TTFT}>{T_{transfer}}+{T_{queue}}+{T_{prefill}} then
|
||||
|
||||
21: 𝑇𝑇𝐹𝑇←T𝑡𝑟𝑎𝑛𝑠𝑓𝑒𝑟+T𝑞𝑢𝑒𝑢𝑒+T𝑝𝑟𝑒𝑓𝑖𝑙𝑙{TTFT}{T_{transfer}}+{T_{queue}}+{T_{prefill}}
|
||||
|
||||
22: p←𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒p{instance}
|
||||
|
||||
23: end if
|
||||
|
||||
24: end if
|
||||
|
||||
25:end for
|
||||
|
||||
26:d,𝑇𝐵𝑇←SelectDecodingInstance(D)d,{TBT}{SelectDecodingInstance}(D)
|
||||
⊳ Load-balancing decoding scheduling
|
||||
|
||||
27:if 𝑇𝑇𝐹𝑇>𝑇𝑇𝐹𝑇_𝑆𝐿𝑂{TTFT}>{TTFT\_SLO} or 𝑇𝐵𝑇>𝑇𝐵𝑇_𝑆𝐿𝑂{TBT}>{TBT\_SLO} then
|
||||
|
||||
28: reject RR; return
|
||||
|
||||
29:end if
|
||||
|
||||
30:if 𝑏𝑒𝑠𝑡_𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛p.𝑝𝑟𝑒𝑓𝑖𝑥_𝑙𝑒𝑛>𝐤𝐯𝐜𝐚𝐜𝐡𝐞_𝐛𝐚𝐥𝐚𝐧𝐜𝐢𝐧𝐠_𝐭𝐡𝐫𝐞𝐬𝐡𝐨𝐥𝐝{{best\_prefix\_len}}{p.{prefix\_len}}>{ kvcache\_balancing\_threshold} then
|
||||
|
||||
31: TransferKVCache(𝑏𝑒𝑠𝑡_𝑚𝑎𝑡𝑐ℎ𝑒𝑑_𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒,p){TransferKVCache}({best\_matched\_instance},p)
|
||||
⊳ KVCache hot-spot migration
|
||||
|
||||
32:end if
|
||||
|
||||
33:return (p,d)(p,d)
|
||||
|
||||
### 6.1 Prefill Global Scheduling
|
||||
|
||||
Previous research on LLM servi
|
||||
|
||||
ng typically uses a load-balancing strategy that evaluates the load on each instance based on the number of assigned requests. In Mooncake, however, the selection of prefill instances considers additional factors—not just load but also the prefix cache hit length and the distribution of reusable KVCache blocks. While there is a preference to route requests to prefill instances with longer prefix cache lengths to reduce computation costs, it may be beneficial to schedule them to other nodes to ensure overall system balance and meet TTFT SLOs. To address these complexities, we propose a cache-aware global scheduling algorithm that accounts for both the prefill time due to the prefix cache and the queuing time associated with the load on the instance.
|
||||
|
||||
Algorithm 1 details the mechanism for our cache-aware prefill scheduling. For every new request, its input tokens are divided into several blocks, and a hash key is computed for each block. This involves generating a hash key of tokens in a block concatenated with the hash key of the previous block (if available). The request’s block keys are then compared one by one against each prefill instance’s cache keys to identify the prefix match length (prefix_lenprefix\_len). Similar reuse logic is already implemented in vLLM, but the open-source version of vLLM only supports local KVCache caching.
|
||||
|
||||
With this matching information, Conductor estimates the corresponding execution time based on the request length and prefix_lenprefix\_len (which varies by instance). It then adds the estimated waiting time for that request to get the TTFT on that instance. Finally, Conductor assigns the request to the instance with the shortest TTFT and updates the cache and queue times for that instance accordingly. If the SLO is not achievable, Conductor directly returns the HTTP 429 Too Many Requests response status code to the upper layers.
|
||||
|
||||
The backbone of this scheduling framework is straightforward, but complexities are hidden in the engineering implementation of various components. For example, to predict the computation time of the prefill stage for a request, we employ a predictive model derived from offline test data. This model estimates the prefill duration based on the request’s length and prefix cache hit length. Thanks to the regular computation pattern of Transformers, the error bound of this prediction is small as long as enough offline data is available. The queuing time for a request is calculated by aggregating the prefill times of all queued requests. In practical implementations, TTFTs are computed in parallel, rendering the processing time negligible compared to the inference time.
|
||||
|
||||
More difficulty lies in predicting the transfer time because it is determined not only by the size of the transferred data but also by the current network status, especially whether the sending node is under congestion. This also necessitates the replication of hot KVCache blocks, which will be discussed in the next section.
|
||||
|
||||
### 6.2 Cache Load Balancing
|
||||
|
||||
In our Mooncake cluster, each prefill machine manages its own set of local prefix caches. The usage frequency of these caches varies significantly. For example, system prompts are accessed by almost every request, whereas caches storing content from a local long document may be used by only one user. As discussed in §6.1, Conductor’s role is crucial in achieving an optimal balance between cache matching and instance load. Thus, from the perspective of the distributed cache system, load balancing also plays an important role. Specifically, it involves strategizing on how to back up caches to ensure that global prefill scheduling can achieve both high cache hits and low load.
|
||||
|
||||
A straw-man solution to this KVCache scheduling problem could be collecting the global usages of each block, using a prediction model to forecast their future usages, and making scheduling decisions accordingly. However, unlike the estimation of prefill time, workloads are highly dynamic and change significantly over time. Especially for a MaaS provider experiencing rapid growth in its user base, it is impossible to accurately predict future usage. Thus, we propose a heuristic-based automated hot-spot migration scheme to enhance cache load balancing.
|
||||
|
||||
|
||||
|
||||
Figure 8: The prefill scheduling experiment in the Mooncake cluster.
|
||||
|
||||
As previously noted, requests may not always be directed to the prefill instance with the longest prefix cache length due to high instance load. In such cases, the conductor forwards the cache’s location and the request to an alternative instance if the estimated additional prefill time is shorter than the transfer time. This instance proactively retrieves the KVCache from the holder and stores it locally. More importantly, we prefer to compute the input tokens if the best remote prefix match length is no larger than the current local reusable prefix multiplied by a threshold111This threshold is currently adjusted manually, but can be adaptively adjusted by an algorithm in the future. Both strategies not only reduce the prefill time for requests but also facilitate the automatic replication of hot-spot caches, allowing for their broader distribution across multiple machines.
|
||||
|
||||
To validate the effectiveness of our strategy, we conducted a scheduling experiment that compares random scheduling and load-balancing scheduling with our strategy. We further compare the cache-aware scheduling described in §6.1 and the KVCache-centric scheduling described in this section that considers cache load balancing. In random scheduling, a prefill instance is selected arbitrarily for each request. In load-balancing scheduling, the instance with the lightest load is chosen. To evaluate, we built a Mooncake cluster consisting of 8 prefill instances and 8 decoding instances, using idle machines overnight, and replayed 23,000 real-world requests for the experiment. We assessed the performance of each scheduling algorithm using the average TTFT and the TTFT SLO attainment rate. The experimental results, depicted in Figure 8, demonstrate that both the cache-aware strategy and the cache load balancing strategy significantly reduce the TTFT of requests. Our KVCache-centric scheduling algorithm outperforms both random and load-balancing scheduling across both metrics. More experiment results can be found in §8.
|
||||
|
||||
## 7 面向过载的调度
|
||||
|
||||
Most existing work on LLM serving assumes that all requests will be processed, optimizing the throughput or the TTFT and TBT of requests accordingly. However, in real scenarios, processing every incoming request is neither economical nor realistic. For commercial inference services facing rapidly increasing volumes of user requests, the growth rate of the cluster’s inference resources is far slower than the increase in incoming requests. As a result, overload is a common issue in current LLM serving, especially during peak times.
|
||||
|
||||
To balance costs and user experience, the system should process as many requests as possible until the system load reaches a predefined threshold. After this point, remaining requests will be either directly rejected or deferred for later retry. Mooncake, implemented as a disaggregated inference system, allows for more flexible scheduling strategies but also confronts unique scheduling challenges not present in non-disaggregated systems and not mentioned in previous works[7, 8, 9].
|
||||
|
||||
In this section, we describe an early rejection policy designed specifically for a disaggregated architecture and address the load fluctuation caused by this approach. We then explore how predicting the generation length is necessary to mitigate these problems.
|
||||
|
||||
### 7.1 Scheduling in Overload Scenarios
|
||||
|
||||
In scenarios where system overload occurs, scheduling involves determining whether to accept or reject incoming requests based on the system load. A critical aspect of this process is defining what constitutes the “system load”, as this definition influences the threshold at which requests are rejected. In conventional coupled systems, the prediction of TTFT and TBT can be complicated by interference between the prefill and decoding stages. Therefore, the load is often measured simply by the ratio of the number of requests being processed to the system’s maximum capacity.
|
||||
|
||||
In contrast, Mooncake, with its disaggregated architecture, processes the prefill and decoding stages independently. Thus we use SLO satisfaction as a direct load measurement. Specifically, we define lttftl_{ttft} and ltbtl_{tbt} as the TTFT and TBT SLO constraints for requests, respectively. The load for prefill and decoding instances is then determined by comparing the predicted maximum TTFT and TBT on an instance against lttftl_{ttft} and ltbtl_{tbt}. With these two criteria, Mooncake’s scheduling requires two key decisions: first, whether to accept the prefill stage based on the prefill instance’s load, and second, whether to proceed with the decoding stage depending on the decoding instance’s load.
|
||||
|
||||
### 7.2 Early Rejection
|
||||
|
||||
In practice, the individual load on prefill or decoding instances does not accurately reflect the actual number of requests processed by the system. This discrepancy arises due to a time lag between scheduling prefill and decoding instances for a single request. If a request is rejected by the decoding instance due to high load after the prefill stage has been completed, the computational resources expended during the prefill stage are wasted. Consequently, the actual number of successfully processed requests during prefill is less than that indicated by the load metric.
|
||||
|
||||
To address this issue, it is natural to advance the load assessment of the decoding instance to precede the beginning of the prefill stage. We refer to this strategy as Early Rejection. Upon the arrival of a request, Conductor evaluates whether to accept the request based on the greater load between the prefill and decoding pools. Early Rejection significantly reduces ineffective computations from rejected requests and enhances load balancing.
|
||||
|
||||
|
||||
|
||||
Figure 9: The load of prefill and decoding instances over 20 minutes, before using the prediction-based early rejection.
|
||||
|
||||
### 7.3 Load Fluctuation Caused by Early Rejection
|
||||
|
||||
However, Early Rejection introduces new challenges. Figure 9 shows the observed real-world instance load over a 20-minute period in a cluster of 20 machines after using the Early Rejection strategy. It highlights significant anti-phase fluctuations between prefill and decoding machines. This phenomenon becomes more pronounced in clusters with fewer prefill machines and in scenarios where the prefill stage takes longer.
|
||||
|
||||
Upon further exploration, we found that this load fluctuation problem is rooted in the time lag between predicting the decoding load and its actual execution. Scheduling based on the current decoding load is inherently delayed. This delay causes fluctuations and phase staggering between the loads on prefill and decoding instances, as illustrated in the theoretical example described in Figure 10(a). The green curve represents the load of prefill instances (scaled from 0 to 1), and the yellow curve represents the load of decoding instances.
|
||||
|
||||
|
||||
|
||||
(a) Early Rejection.
|
||||
|
||||
|
||||
|
||||
(b) Early Rejection Based on Prediction.
|
||||
|
||||
Figure 10: Instance load when applying Early Rejection and Early Rejection Based on Prediction.
|
||||
|
||||
In Stage 1, the load on both prefill and decoding instances is low, so Conductor accepts a large number of requests until the load on prefill instances reaches its limit. In Stage 2, requests processed by prefill instances are scheduled to decoding instances, causing the load on decoding instances to be high. Consequently, Conductor rejects incoming requests, leading to a lower load on prefill instances. In Stage 3, no new requests enter the decoding stage, resulting in a decreased load. At this point, Conductor again accepts a large number of requests until the prefill instances are fully loaded. In Stage 4, as the load on decoding instances increases, Conductor rejects requests, causing a low load on prefill instances. This severe fluctuation in load between prefill and decoding instances results in poor resource utilization of the inference cluster.
|
||||
|
||||
### 7.4 Early Rejection Based on Prediction
|
||||
|
||||
To solve the load fluctuation problem, we propose a framework of Early Rejection Based on Prediction to address scheduling challenges in overload scenarios for disaggregated LLM serving systems like Mooncake. As illustrated in Figure 10(b), this framework predicts the decoding load after the prefill stage of incoming requests and uses this prediction to decide whether to accept the requests, which helps mitigate the fluctuation problem. The core component of this strategy is the accurate prediction of the decoding load for the subsequent period. We introduce two approaches for this:
|
||||
|
||||
Request level: Previous work highlights a significant challenge in predicting loads for LLM serving: the unknown output length of each request. If we could determine the output length in advance, it would be possible to estimate the TTFT and TBT much more accurately. This, in turn, would help predict the number of requests a decoding instance can complete and the number of new requests that will be added after a specified time, thereby obtaining the load at that time. However, predicting each request’s output length is challenging due to high costs [9] or low accuracy, especially under overload conditions where resources are scarce and accurate predictions are necessary, making request-level predictions particularly difficult.
|
||||
|
||||
System level: In contrast to request-level predictions, system-level predictions do not attempt to predict the completion time
|
||||
|
||||
e TTFT SLO. However, while approximately 100% of the requests for Mooncake-[10P+10D] satisfy the TBT SLO, only 57% of the requests for vLLM-[20M] meet this criterion, with some requests exhibiting extremely high TBTs. In this experiment, Mooncake can process approximately 75% more requests while adhering to the SLOs.
|
||||
|
||||
### 8.2 Performance in Overload Scenarios
|
||||
|
||||
In this section, we evaluate performance under overload scenarios, focusing on the maximum number of requests the system can handle, as discussed in §7. The baseline strategy, which rejects requests based on load before both stages start, leads to resource wastage by rejecting requests already processed in the prefill stage. In contrast, we propose the Early Rejection and Early Rejection based on Prediction strategies, detailed in §7.2 and §7.4, respectively. These strategies take the system’s load into comprehensive consideration, and hence reduce unnecessary request rejections.
|
||||
|
||||
Specifically, we built a Mooncake cluster with 8 prefill instances and 8 decoding instances and tested it using real traces from 23,000 requests. To simulate overload scenarios, we increased the replay speed to 2x.
|
||||
|
||||
Table 3: Number of requests rejected by the system under the overloaded-scenario experiment.
|
||||
|
||||
Table 3 shows Mooncake’s performance under different strategies. With the baseline strategy, the system rejects 4,183 requests. In contrast, under the Early Rejection and Early Rejection based on Prediction strategies, Mooncake rejects 3,771 and 3,589 requests, respectively. This demonstrates that by rejecting requests early, Mooncake can avoid unnecessary prefill computations, thereby improving the effective utilization of system resources. Furthermore, by predicting the load of decoding instances, Mooncake can mitigate load fluctuations, increasing the request handling capacity.
|
||||
|
||||
## 9 Related Work
|
||||
|
||||
Significant efforts have been dedicated to enhancing the efficiency of LLM serving systems through scheduling, memory management, a
|
||||
Reference in New Issue
Block a user