文档(金鹏): 2026-08-06 章节 48 篇文章摘要归档
- 46 篇原文+摘要双文件归档（按 来源/作者 分层，复用本地归档 20 篇+新抓取 26 篇） - 即梦生成 9 组主题配图（大图+列表缩略图）存入 知识/金鹏/20260806/ - 章节重组为 9 个主题分组并挂接摘要引用
This commit is contained in:
@@ -0,0 +1,388 @@
|
||||
# Splitwise: Efficient Generative LLM Inference Using Phase Splitting
|
||||
|
||||
> **来源**：arXiv
|
||||
> **作者**：Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini
|
||||
> **发布日期**：2023-11-30
|
||||
> **原文链接**：https://arxiv.org/abs/2311.18677
|
||||
|
||||
---
|
||||
|
||||
## 论文元数据
|
||||
|
||||
- **arXiv ID**：2311.18677
|
||||
- **学科分类**：Distributed, Parallel, and Cluster Computing (cs.DC)
|
||||
- **作者机构**：华盛顿大学（University of Washington）、微软（Microsoft）
|
||||
- **提交历史**：v1: 2023-11-30；v2: 2024-05-20
|
||||
- **DOI**：https://doi.org/10.48550/arXiv.2311.18677
|
||||
|
||||
---
|
||||
|
||||
Pratyush Patel1,
|
||||
Esha Choukse2,
|
||||
Chaojie Zhang2,
|
||||
|
||||
Aashaka Shah2,
|
||||
Íñigo Goiri2,
|
||||
Saeed Maleki2,
|
||||
Ricardo Bianchini2
|
||||
|
||||
1University of Washington 2Microsoft
|
||||
|
||||
## 摘要（Abstract）
|
||||
|
||||
Generative large language model (LLM) applications are growing rapidly, leading to large-scale deployments of expensive and power-hungry GPUs.
|
||||
Our characterization of LLM inference shows that each inference request undergoes two phases: a compute-intensive prompt computation phase and a memory-intensive token generation phase, each with distinct latency, throughput, memory, and power characteristics.
|
||||
Despite state-of-the-art batching and scheduling, the token generation phase underutilizes compute resources.
|
||||
Unlike prompt computation, token generation does not need the compute capability of the latest GPUs and can be run with lower power and cost.
|
||||
|
||||
Based on these insights, we propose Splitwise, a model deployment and scheduling technique that splits the two phases of LLM inference requests on to separate machines.
|
||||
Splitwise enables phase-specific resource management using hardware that is well suited for each phase.
|
||||
Request state is transferred efficiently between machines using optimized network libraries on the fast back-plane interconnects available in today’s GPU clusters.
|
||||
Using Splitwise, we design homogeneous and heterogeneous LLM inference clusters optimized for throughput, cost, and power.
|
||||
Compared to current designs, Splitwise clusters achieve up to 1.4×1.4 × higher throughput at 20% lower cost. Alternatively, they can deliver 2.35×2.35 × more throughput under the same power and cost budgets.
|
||||
|
||||
## I 引言（Introduction）
|
||||
|
||||
Recent advancements in generative large language models (LLMs) have significantly improved their response quality and accuracy [18, 71].
|
||||
These trends have led to the widespread adoption of LLMs across various domains [6, 21].
|
||||
Most modern LLMs are built using the transformer architecture [78, 77] and exhibit similar characteristics [63].
|
||||
Transformer model sizes have grown steadily, from the early BERT models [36] having 340 million parameters, to GPT-3 [28] with a staggering 175 billion parameters, and GPT-4 rumored to have even more.
|
||||
|
||||
LLMs typically run on expensive and power-hungry GPUs [16].
|
||||
The sudden and large-scale deployment of LLMs has led to a worldwide GPU capacity crunch [14].
|
||||
The computational demand for LLM inference far exceeds that of training due to the vast number of applications leveraging LLMs.
|
||||
Furthermore, since training LLMs requires expensive and dedicated supercomputers [60, 56], a large number of inferences are necessary to amortize the high training costs.
|
||||
LLM inference jobs, although orders of magnitude smaller than training, are still expensive given the compute involved.
|
||||
11脚注: Work partly done as an intern at Microsoft.
|
||||
|
||||
TABLE I: NVIDIA A100 vs. H100 specifications.
|
||||
|
||||
Generative LLM inference for a single request consists of several forward passes through the model, since the output tokens are generated one by one.
|
||||
This inherently has two contrasting phases of computation.
|
||||
First, the _prompt computation phase_, in which all the input prompt tokens run through the forward pass of the model in parallel to generate the first output token.
|
||||
This phase tends to be computationally intensive and requires the high FLOPs (floating point operations per second) of the latest GPUs today.
|
||||
Second, the _token generation phase_, in which subsequent output tokens are generated sequentially based on the forward pass of the last token and all the cached context from previous tokens in the sequence.
|
||||
Given the lack of compute parallelism, this phase tends to be more memory bandwidth and capacity bound, despite state-of-the-art batching.
|
||||
Running both phases on the same machine often leads to inconsistent end-to-end latencies due to the arbitrary batching of prompt and token phases.
|
||||
Due to these challenges, services need to over-provision expensive GPUs to meet tight inference service level objectives (SLOs) for interactive applications.
|
||||
At the same time, cloud service providers (CSPs) are having to build a lot of new datacenters to meet the GPU demand, and are running into a power wall [19].
|
||||
|
||||
The industry continues to release new computationally powerful GPUs, each much more power hungry and expensive than the last.
|
||||
However, as shown in Table I, the high-bandwidth memory (HBM) capacity and bandwidth on these GPUs has not scaled at the same rate recently.
|
||||
The latest NVIDIA H100 GPUs have 3.43×3.43 × more compute and 1.75×1.75 × more power compared to their predecessor A100 GPUs.
|
||||
However, their memory bandwidth only grew by 1.6×1.6 ×, with no increase in memory capacity.
|
||||
|
||||
Our work.
|
||||
Given the distinct properties of prompt computation and token generation phases, we propose splitting the inference request and running them on separate machines.
|
||||
Doing so allows us to separately manage hardware resources for each phase, thereby increasing the GPU utilization and the overall efficiency of the system.
|
||||
It also enables using different, better-suited hardware for each phase.
|
||||
To realize such a setup, the cached context from the prompt computation needs to be communicated over from the prompt processing machine to the token generation machine at low latency.
|
||||
We implement these transfers in an optimized manner over the back-end Infiniband interconnects avaialble in datacenters today, allowing us to increase efficiency without any perceived performance loss.
|
||||
|
||||
With Splitwise, we design clusters optimized for cost, throughput, and power, using production traces of LLM inference requests [4].
|
||||
Given the diverging memory and compute scaling rates across GPU generations, we also evaluate different GPUs and power caps for the different inference phases.
|
||||
This allows us to target better performance per dollar (Perf/$) for users, and better performance per watt (Perf/W) for CSPs.
|
||||
Additionally, users can target older GPUs, which are likely more readily available to them.
|
||||
|
||||
We show that Splitwise-based LLM inference clusters can achieve 1.4× higher throughput at 20% lower cost than existing clusters. Alternatively, they can deliver 2.35× more throughput with the same cost and power budgets.
|
||||
|
||||
Summary.
|
||||
We make the following contributions:
|
||||
|
||||
1. 1.
|
||||
|
||||
An extensive characterization of the differences in the execution and utilization patterns of the prompt and token generation phases in LLM inference on the NVIDIA A100 and H100 GPUs using production traces.
|
||||
2. 2.
|
||||
|
||||
Splitwise, our technique for optimized utilization of available hardware, which splits the prompt computation and token generation phases onto separate machines.
|
||||
3. 3.
|
||||
|
||||
A design exploration of homogeneous and heterogeneous cluster deployments with Splitwise to optimize the overall cost, request throughput, and provisioned power.
|
||||
4. 4.
|
||||
|
||||
An evaluation of the systems designed with Splitwise using production traces.
|
||||
|
||||
## II Background
|
||||
|
||||
### II-A Large Language Models
|
||||
|
||||
Modern LLMs are based on transformers.
|
||||
Transformer models use attention [77] and multi-layer-perceptron layers to understand the inputs and generate an output, respectively.
|
||||
Transformer-based LLMs include encoder-only [36, 54], decoder-only [67, 69, 71], and encoder-decoder [70] models.
|
||||
Generative LLMs, the focus of this paper, are usually either decoder-only, or encoder-decoder models.
|
||||
|
||||
### II-B Generative LLM inference phases
|
||||
|
||||
Figure 1 shows an example of generative LLM inference.
|
||||
Once the prompt query is received, all the input tokens are computed in parallel, within a single iteration, to generate the first token.
|
||||
We call this the prompt processing phase.
|
||||
The context generated from the attention layers during the prompt computation is saved in the key-value (KV) cache, since it is needed for all the future token generation iterations.
|
||||
After the first token is generated, the following tokens only use the last generated token and the KV-cache as inputs to the forward pass of the model.
|
||||
This makes the subsequent token generation more memory bandwidth and capacity intensive than the computationally heavy prompt phase.
|
||||
|
||||
Figure 1: An LLM inference example.
|
||||
|
||||
### II-C Performance metrics for LLMs
|
||||
|
||||
Prior work has proposed three main metrics for LLM inference: end-to-end (E2E) latency, time to first token (TTFT), and throughput.
|
||||
We add another latency metric: time between tokens (TBT), to track the online streaming throughput of the tokens as they are generated serially.
|
||||
Table II summarizes the key performance metrics that we consider in this work.
|
||||
|
||||
TABLE II: Performance metrics for LLMs.
|
||||
|
||||
Generative LLMs may be used for a variety of tasks with different kinds of SLOs.
|
||||
For batch tasks (_e.g._, summarization), TTFT or TBT latency metrics are less important than throughput.
|
||||
On the other hand, for latency-sensitive tasks (_e.g._, conversational APIs), TTFT and TBT are the more important metrics with tighter SLOs.
|
||||
|
||||
Figure 2: Batching mechanisms and their latency impact on the pro
|
||||
|
||||
ut keeps scaling up with the batch size until the machine runs out of memory.
|
||||
For this reason, the MLS tracks the memory and starts queueing tokens once the machine is close to running out of memory.
|
||||
|
||||
Mixed machines.
|
||||
To meet the TTFT SLO, the MLS must prioritize running prompts and schedule any new prompts in the pending queue immediately.
|
||||
If the machine is running token phases and has no additional capacity to run the prompt phase, the MLS will _preempt_ tokens.
|
||||
To avoid _starvation_ of the token phase due to preemption, we increase the priority of the token with age and limit the number of preemptions that each request can have.
|
||||
|
||||
### IV-C KV-cache transfer
|
||||
|
||||
As discussed in Section II, the KV-cache is generated during the prompt phase of the request, and it continuously grows during the token generation phase.
|
||||
In Splitwise, we need to transfer the KV-cache from the prompt machine to the token machine (shown in Figure 10) to complete the inference.
|
||||
This transfer delay is the main overhead associated with Splitwise.
|
||||
In this section, we discuss the impact of KV-cache transfer and how we optimize it.
|
||||
|
||||
(a)
|
||||
|
||||
(b)
|
||||
|
||||
Figure 11: Optimizing KV-cache transfer in Splitwise.
|
||||
|
||||
Figure 11(a) shows the Gantt chart for the prompt phase, the KV-cache transfer, and the token generation phase for a single batch of requests when naively transferring the KV cache in a serialized way.
|
||||
The KV-cache transfer starts only after the prompt phase has finished and the first token is generated.
|
||||
Further, it needs to complete before the next output token can be generated in the token generation phase.
|
||||
This directly impacts the maximum TBT and end-to-end latency of inference.
|
||||
|
||||
The time required for the transfer depends on the size of the KV cache (which is directly proportional to the number of prompt tokens) and on the bandwidth of the interconnect between the prompt and the token machines.
|
||||
Even when using fast InfiniBand links, the transfer overhead for large prompt sizes could become a significant fraction of the TBT.
|
||||
|
||||
In Splitwise, we optimize the KV-cache transfer by overlapping it with the computation in the prompt phase.
|
||||
As each layer in the LLM gets calculated in the prompt machine, the KV cache corresponding to that layer is also generated.
|
||||
At the end of each layer, we trigger an asynchronous transfer of the KV-cache for that layer while the prompt computation continues to the next layer.
|
||||
Figure 11(b) shows this asynchronous transfer which reduces the transfer overheads.
|
||||
Layer-wise transfer also enables other optimizations, such as earlier start of the token phase in the token machines, as well as earlier release of KV-cache memory on the prompt machines.
|
||||
|
||||
Layer-wise KV-cache transfer happens in parallel with the prompt computation for the next layer.
|
||||
This requires fine-grained synchronization per layer for correctness.
|
||||
Thus, it is possible to incur performance interference and increase the TTFT, especially for smaller prompts.
|
||||
However, for small prompts the total KV-cache size is small and does not need the layer-wise transfer to hide the latency.
|
||||
Since the number of tokens in a batch is already known at the start of computation, Splitwise picks the best technique for KV-cache transfer.
|
||||
It uses serialized KV-cache transfer for smaller prompts and layer-wise transfer and for larger prompts.
|
||||
We show that the overall transfer and interference overheads are relatively small in Section VI-A.
|
||||
|
||||
TABLE V: Evaluated Splitwise designs all normalized to DGX-A100
|
||||
|
||||
### IV-D Provisioning with Splitwise
|
||||
|
||||
We leverage Splitwise to optimize LLM inference cluster deployments for power, cost, and throughput.
|
||||
|
||||
Type of machines.
|
||||
We propose four main variants of Splitwise-based systems:
|
||||
_Splitwise-AA_,
|
||||
_Splitwise-HH_,
|
||||
_Splitwise-HA_,
|
||||
and _Splitwise-HHcap_.
|
||||
The nomenclature is simply drawn from the first letter representing the Prompt machine type, and the second letter representing the Token machine type.
|
||||
“A” represents a DGX-A100 machine, “H” represents a DGX-H100 machine,
|
||||
and “Hcap” represents a power-capped DGX-H100 machine.
|
||||
Table V shows a summary of the cost, power, and hardware in each of our evaluated systems.
|
||||
|
||||
Splitwise-AA uses DGX-A100 for both prompt and token pools, while Splitwise-HH uses DGX-H100 for both.
|
||||
These two variants represent the commonly available setups in providers where machines are homogeneous and interchangeable.
|
||||
|
||||
Splitwise-HA uses DGX-H100 for the prompt pool and DGX-A100 for the token pool.
|
||||
We choose this configuration based on Table IV, and the Insight VII (_i.e._, A100s can be more cost- and power-efficient for the token phase).
|
||||
|
||||
Splitwise-HHcap uses DGX-H100 machines for both prompt and token pools.
|
||||
However, we power cap the token machines down to 70% of their rated power, with each GPU capped by 50% of the power.
|
||||
We propose this design based on Figure 9 and Insight VII (_i.e._, the prompts phase is impacted by power caps while token has no performance impact with 50% lower power cap per GPU).
|
||||
|
||||
Number of machines.
|
||||
The LLM inference cluster deployment must be sized with the appropriate number of prompt and token machines.
|
||||
Our methodology involves searching the design space using our event-driven cluster simulator, which is described in detail in Section V.
|
||||
We need to provide as input:
|
||||
(1) the target cluster design (_e.g._, Splitwise-HA or Splitwise-HHcap),
|
||||
(2) an LLM-specific performance model that can estimate the TTFT and TBT at various input, output, and batch sizes,
|
||||
(3) a short trace derived from the target prompt and token size distributions for the service (_e.g._, Figure 3),
|
||||
(4) the SLOs (_e.g._, Table VI),
|
||||
(5) the constraints (_e.g._, throughput),
|
||||
and (6) the optimization goal (_e.g._, minimize cost).
|
||||
Using this information, our provisioning framework searches the space for the desired optimal point.
|
||||
For example, searching with a throughput constraint and a cost minimization goal gives us iso-throughput cost-optimized clusters across different designs.
|
||||
|
||||
|
||||
|
||||
Figure 12: Design space for provisioning a Splitwise-HH cluster.
|
||||
Cluster configurations targets a peak throughput of 70 RPS.
|
||||
The cost-optimal Splitwise-HH configuration is marked with ⋆⋆⋆ (27 prompt and 3 token machines).
|
||||
|
||||
Search space.
|
||||
Figure 12 shows an example of the two-dimensional search space for the number of prompt and token machines under Splitwise-HH for the coding workload (using a 2-minute trace).
|
||||
The simulator outputs the various percentiles for TTFT, TBT, and E2E latencies.
|
||||
Then, we select the clusters that meet the SLOs for each of these metrics and optimize our target function.
|
||||
For example, Figure 12 shows a ⋆⋆⋆ for the setup with 27 prompt and 3 token machines with the lowest cost that achieves 70 RPS.
|
||||
We call this setup _iso-throughput cost-optimized_.
|
||||
|
||||
Optimization.
|
||||
We can use three optimization goals:
|
||||
_throughput_, _cost_, and _power_.
|
||||
Throughput optimization is important for both, the cloud service provider (CSP) and the user.
|
||||
Cost optimization has different importance levels to the CSP and the user.
|
||||
For the CSP, a higher cost for the same throughput might be acceptable if there are gains in power and space requirements for the cluster.
|
||||
However, for the end-user, a higher cost at the same throughput is generally unacceptable.
|
||||
Finally, power optimization is attractive for a CSP, since it enables more GPUs to be deployed in the same datacenter [62, 63], but it may not be as important to the user.
|
||||
We only consider the provisioned power, and not the dynamic power utilization, in our study.
|
||||
|
||||
### IV-E Practical Considerations
|
||||
|
||||
Accuracy impact.
|
||||
Splitwise does not impact accuracy since it uses lossless KV-cache transfer and does not add any randomization.
|
||||
It executes inference with the same parameters and state as on a single machine.
|
||||
|
||||
Scalability.
|
||||
Since LLM requests are much longer than typical ML requests [37, 38], they incur lower scheduling overhead for similar cluster sizes.
|
||||
However, the CLS may become a scalability bottleneck for large clusters.
|
||||
Insights from prior work on partitioned or replicated scheduling could help improve scalability [61, 27, 72] and are orthogonal to Splitwise.
|
||||
|
||||
Reliability and fault tolerance.
|
||||
If the prompt or the token machine fail, Splitwise simply restarts requests from scratch, similar to today’s LLM serving systems [51, 44].
|
||||
Alternatively, Splitwise could checkpoint the KV-cache generated after prompt computation into an in-memory database.
|
||||
To recover, Splitwise can use this cache to skip prompt recomputation, and start right away with the token phase.
|
||||
The KV-cache could also be checkpointed periodically during the token phase.
|
||||
Designing safe and efficient failure recovery is out of scope for our paper.
|
||||
|
||||
## V Methodology
|
||||
|
||||
### V-A Experimental setup
|
||||
|
||||
To evaluate our proposal on real hardware, we implement Splitwise’s KV-cache transfer mechanism on top of vLLM [51]. Our implementation is open source [1].
|
||||
We run this modified vLLM on two DGX-A100 and two DGX-H10 virtual machines (VMs) on Microsoft Azure with specifications from Table I.
|
||||
These are the VMs used to collect the characterization data in Section III.
|
||||
These machines are connected with InfiniBand and the DGX-H100s have double the bandwidth (_i.e._, 400 Gbps).
|
||||
|
||||
Since vanilla vLLM only supports continuous batching with token preemption which can lead to much higher TBT, we implement state-of-the-art mixed continuous batching [81] as discussed earlier in Figure 2(c).
|
||||
|
||||
Our implementation of the Splitwise technique assigns machines either a prompt role, or a token role.
|
||||
As the prompt machine generates the first token, it transfers the KV-cache to the token machine using the technique described in Section IV-C.
|
||||
We use MSCCL++ [11], an optimized GPU-driven communication library, to implement the naive and layer-wise KV cache transfers.
|
||||
|
||||
In our implementation, the prompt machine uses the zero-copy one-sided put primitive of MSCCL++ to send KV-cache data over InfiniBand as soon as it is ready, without requiring the token machine to issue any receive instructions.
|
||||
Once we have issued a put for all layers, the prompt machine signals a semaphore that the token machine waits on.
|
||||
The synchronization done with the help of semaphores uses the same InfiniBand connection used to send KV-cache data.
|
||||
When processing a batch of prompts, each request is assigned a different semaphore since it may be routed to different token machines.
|
||||
We ship the KV-caches block-by-block in vLLM.
|
||||
To minimize the number of transfers, we also consider the contiguity of KV blocks as long as they use the same semaphore.
|
||||
|
||||
### V-B Simulator setup
|
||||
|
||||
We build a simulator to explore cluster designs and evaluate Splitwise at scale.
|
||||
The simulator code is open source [20].
|
||||
|
||||
|
||||
|
||||
Figure 13: Overview of the design of the Splitwise simulator.
|
||||
|
||||
Figure 13 shows the design of our simulator.
|
||||
The simulator is event-driven and faithfully models the Splitwise machine pools, schedulers, machine-level memory and queues, and KV-cache transfer.
|
||||
We first profile the LLM on the target hardware with various input/output sizes .
|
||||
Based on the characterization profiles, we build a performance model.
|
||||
The simulator takes as input the request traces, SLOs, the performance model, and the configurations for cluster and scheduler .
|
||||
For our evaluation, we use the prompt and token size distributions from the production traces in Section III.
|
||||
We tune the Poisson arrival rate to increase and decrease the load (requests per second) for cluster sizing.
|
||||
The simulator provides the achieved metrics per request (TTFT, TBT, E2E), and the machine utilization levels .
|
||||
We cross-validated the performance model with hardware experiments to ensure accuracy; we also validated the simulator end-to-end using production load with over 50K iterations to ensure fidelity .
|
||||
|
||||
Performance model.
|
||||
We build a piece-wise linear performance model using performance profiles at various batch sizes, input sizes, output sizes, in the required parallelism configuration on A100 and H100 machines from Section III.
|
||||
We validate that our performance model has high accuracy; it incurs a mean absolute percentage error (MAPE) of less than 3% when evaluated with a 80:20 train:test dataset split.
|
||||
|
||||
Communication model.
|
||||
In our evaluation, KV-cache transfers cause inter-machine communication, whereas tensor parallelism only causes intra-machine communication.
|
||||
We model inter-machine communication overheads by benchmarking our KV-cache transfer implementation over Infiniband in Section VI-A.
|
||||
|
||||
SLOs.
|
||||
To determine the maximum throughput that can be supported by a given cluster design, we use P50, P90, and P99 SLOs for TTFT, TBT, and E2E latency metrics.
|
||||
Table VI shows our SLO definition using DGX-A100 as a reference.
|
||||
We require all nine SLOs to be met.
|
||||
SLOs on TTFT are slightly looser, since it has a much smaller impact on the E2E latency.
|
||||
|
||||
TABLE VI: SLO expressed as slowdown compared to a request running on DGX-A100 under no contention.
|
||||
|
||||
Baselines.
|
||||
We compare our Splitwise designs against Baseline-A100 and Baseline-H100.
|
||||
The clusters in these baselines consist of just DGX-A100s and DGX-H100s, respectively.
|
||||
Both baselines use the same mixed continuous batching that Splitwise uses for mixed pool machines (described in Section IV-A).
|
||||
|
||||
## VI Evaluation
|
||||
|
||||
### VI-A Experimental results
|
||||
|
||||
KV-cache transfer latency.
|
||||
We first measure the latency to transfer the KV-cache as the prompt size grows.
|
||||
Figure 14 shows the visible transfer latency on both A100 and H100 setups with the naive and optimized transfer design as discussed in Figure 11.
|
||||
Compared to the prompt computation time, the overhead is minimal (<7%7<7\%< 7 %).
|
||||
The time for serialized transfers linearly increases with the prompt size since the size of the KV-cache also increases.
|
||||
The optimized per-layer transfer, on the other hand, hides much of the latency.
|
||||
For these transfers, we see a constant non-overlapped transfer time of around 8ms for the A100 and around 5ms for the H100 setup.
|
||||
The H100 setup has double the bandwidth of the A100 setup (_i.e._, 200 vs 400 Gbps), and the impact of this can be clearly seen with transfers in the H100 setup happening about twice as fast as those in the A100 setup.
|
||||
|
||||
As discussed in Section IV-C, for small prompt sizes (<512absent512<512< 512 in H100), Splitwise uses the serialized KV-cache transfer and for larger prompts, it uses per-layer transfers.
|
||||
|
||||
|
||||
|
||||
Figure 14: Overhead of the KV-cache transfer as the prompt size increases on A100s and H100s.
|
||||
|
||||
End-to-end impact.
|
||||
Next, we run the coding trace on the 2-machine Splitwise setups without batching, and compare the observed latency metrics to a 1-machine baseline setup with no batching.
|
||||
Figure 15 shows our results.
|
||||
The latency impact of serially transferring the KV-cache grows up to 3% of the E2E with large prompts.
|
||||
However, Splitwise only incurs 0.8% of E2E.
|
||||
In a user-facing inference, the only visible impact of KV-cache transfer overhead is the latency for the second token.
|
||||
Splitwise adds a 16.5% latency to the second token, as compared to the 64% overhead from a serialized transfer.
|
||||
Overall, the transfer impact in Splitwise is hardly perceivable even in a user-facing inference.
|
||||
|
||||
|
||||
|
||||
Figure 15: Overhead of KV cache transfer on TTFT, E2E latency for coding trace for A100 and H100.
|
||||
|
||||
### VI-B Iso-power throughput-optimized clusters
|
||||
|
||||
Cluster provisioning.
|
||||
We provision clusters using the methodology described in Section IV-D.
|
||||
We target a specific workload (_e.g._, conversation) at a peak load with the same power (_i.e._, iso-power) for each cluster design.
|
||||
For the baseline, we use the power for 40 DGX-H100 machines as our target peak power.
|
||||
For the A100 baseline, we can fit 70 DGX-A100 machines under the same power budget.
|
||||
We denote these two designs as 40P/T and 70P/T respectively, since they both use mixed batching in all machines.
|
||||
|
||||
For Splitwise cluster designs under the coding trace, Splitwise-AA provisions 55 prompt machines and 15 for the token pool, denoted as (55P, 15P).
|
||||
Note that like Baseline-A100, Splitwise-AA also provisions 75% more machines than Baseline-H100.
|
||||
The legends in Figure 16 show the different provisioning choices under coding and conversation workloads.
|
||||
Request size distributions reflect in the machine pool sizing.
|
||||
For example, we provision
|
||||
|
||||
[18].
|
||||
However, in the future, services may have enough GPU capacity to cache the context and avoid recomputation.
|
||||
This could sway the memory utilization pattern of the prompt phase from our characterization.
|
||||
Furthermore, it may require transferring the KV-cache back to a prompt machine to be ready for the next conversation request.
|
||||
|
||||
## VIII Related Work
|
||||
|
||||
Heterogeneous scheduling and dataflow systems.
|
||||
Prior work has studied heterogeneous scheduling for a variety of interactive services [83, 65, 68].
|
||||
These works exploit hardware heterogeneity to strike a balance between different objectives such as cost, energy, and performance.
|
||||
However, they run the entire workload on the same machine.
|
||||
Research on heterogeneous multiprocessor CPU scheduling attempts to match workload heterogeneity to hardware heterogeneity [40, 76, 41, 50, 80, 29].
|
||||
These works use profiling or online monitoring with metrics like request length or hardware performance counters to identify workload phases and allocate them appropriately on heterogeneous processors.
|
||||
However, they do not consider the complexities with batching.
|
||||
Distributed dataflow systems orchestrate large-scale computational graphs and aim to provide general-purpose programmability [34, 46, 75, 82].
|
||||
LLM inference under Splitwise can be viewed as a static computational graph with two stages, so it could be implemented using distributed frameworks that provide efficient GPU abstractions [59].
|
||||
Splitwise differs from these works since it uses a spe
|
||||
@@ -0,0 +1,93 @@
|
||||
# 📊 文章摘要：Splitwise: Efficient Generative LLM Inference Using Phase Splitting
|
||||
|
||||
> **原文**：[2023-11-30_Splitwise.md](./2023-11-30_Splitwise.md)
|
||||
> **原文链接**：https://arxiv.org/abs/2311.18677
|
||||
> **来源**：arXiv
|
||||
> **作者**：Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini（华盛顿大学、微软）
|
||||
> **发布日期**：2023-11-30
|
||||
> **摘要日期**：2026-08-06
|
||||
> **价值评级**：⭐⭐⭐ 高
|
||||
|
||||
---
|
||||
|
||||
## 核心命题
|
||||
|
||||
> **相位拆分** — LLM 推理的两阶段（计算密集的 prompt 处理、内存密集的 token 生成）对硬件的需求截然不同，把它们拆分到各自合适的机器（含异构、降配硬件），是突破"GPU 算力与内存带宽增长失衡"约束的成本与功耗优化路径。
|
||||
|
||||
---
|
||||
|
||||
## 文章概要
|
||||
|
||||
本文用生产 trace 对 A100/H100 上的 LLM 推理做表征分析，发现同一请求的两个阶段特性对立：prompt 计算阶段吃算力（FLOPs），token 生成阶段受内存带宽与容量约束、即使最优批处理也浪费算力。由此提出 Splitwise：把两阶段拆分到不同机器，用层级异步传输把 KV cache 的迁移开销与 prompt 计算重叠（小 prompt 走串行传输），并利用生产 trace + 事件驱动模拟器探索同构/异构集群设计（AA、HH、HA、HHcap 四类）。相比现有集群，Splitwise 可在成本降低 20% 的同时提升 1.4× 吞吐，或在同等成本与功耗预算下提供 2.35× 吞吐。局限：KV cache 传输是固有开销、依赖数据中心高速互连（InfiniBand/NVLink），集中式调度器在超大集群下可能成为扩展性瓶颈。
|
||||
|
||||
---
|
||||
|
||||
## 关键要点
|
||||
|
||||
1. **硬件失衡是动因** — H100 相对 A100 算力提升 3.43×、功耗提升 1.75×，但内存带宽仅增长 1.6×、容量零增长；最新 GPU 的算力优势在内存密集的 token 生成阶段被浪费。`[分类: 共识]`
|
||||
2. **两阶段资源画像对立** — prompt 阶段计算密集、受 FLOPs 约束；token 生成阶段内存带宽/容量受限，即使 state-of-the-art 批处理也显著低效利用算力，可降配硬件运行。`[分类: 范式突破]`
|
||||
3. **拆分而非解耦竞争** — 与 DistServe 同期提出阶段拆分思路，但出发点不同：DistServe 追求 goodput/SLO，Splitwise 强调异构硬件选型与成本/功耗优化（Perf/$ 与 Perf/W）。`[分类: 共识]`
|
||||
4. **KV cache 传输的工程化优化** — 逐层异步传输与 prompt 计算重叠，使 E2E 延迟影响仅 0.8%（串行传输为 3%）；第二 token 延迟增加 16.5%（串行方案 64%），用户几乎无感知；传输总开销 <7% 的 prompt 计算时间。`[分类: 范式突破]`
|
||||
5. **异构集群设计矩阵** — 四种设计：AA（A100 全同构）、HH（H100 全同构）、HA（H100 跑 prompt + A100 跑 token，token 阶段 A100 更划算）、HHcap（H100 双池但 token 机功耗上限 70%、单 GPU 限 50% 功耗——token 阶段对功耗限制不敏感）。`[分类: 范式突破]`
|
||||
6. **量化收益** — 同等功耗预算下：40 台 H100 baseline 被 55 prompt + 15 token 的 Splitwise-AA 方案超越；相比现有集群吞吐提升 1.4× 且成本降 20%，同成本同功耗下吞吐可达 2.35×。`[分类: 共识]`
|
||||
7. **模拟驱动的集群供给** — 事件驱动模拟器 + 分段线性性能模型（MAPE < 3%，与真实硬件实验交叉验证，端到端 5 万+ 迭代验证），在 TTFT/TBT/E2E 九个 SLO 约束下搜索异构机群配比（如 70 RPS 目标的成本最优配置为 27 prompt + 3 token 机器）。`[分类: 未探索]`
|
||||
8. **可靠性以重启为代价** — 节点故障时从头重启请求；论文提出可将 KV cache 检查点化到内存数据库以跳过 prompt 重算，但安全高效的故障恢复留作未来工作。`[分类: 未探索]`
|
||||
|
||||
---
|
||||
|
||||
## 批判性分析
|
||||
|
||||
### 假设前提
|
||||
|
||||
- GPU 集群存在（或未来将出现）供异构部署的多种机型，且异构机型间的价格/功耗/可用性差异足以驱动拆分决策。
|
||||
- 数据中心普遍具备高速后端互连（InfiniBand 等），KV cache 跨机传输可被有效隐藏。
|
||||
- token 生成阶段使用上一代或降配硬件不影响 SLO 达标（对交互式负载尤其依赖此假设）。
|
||||
- 生产请求的 prompt/token 长度分布可表征，且集群按峰值负载供给（论文只考虑供给功耗而非动态功耗）。
|
||||
|
||||
### 论据与逻辑
|
||||
|
||||
- 论据扎实：表征数据来自真实硬件（Azure 上的 DGX-A100/H100）+ 生产 trace；KV cache 传输开销有端到端实测（0.8% E2E、16.5% 第二 token）；模拟器性能模型 MAPE <3% 且经 5 万+ 迭代端到端验证，结论的量化支撑较强。
|
||||
- 逻辑链条完整：硬件失衡观测 → 阶段资源画像差异 → 拆分设计 → 传输优化 → 异构供给搜索 → 收益量化。
|
||||
- 弱点：1.4×/2.35× 等收益来自模拟器而非全系统真实部署；"第二 token 延迟"这类交互体验指标的受众感知评估主观；对大规模集群下集中式调度器（CLS）瓶颈仅以"正交于 Splitwise"带过，未量化。
|
||||
|
||||
### 边界与局限
|
||||
|
||||
- 结论适用于两阶段分离收益大于传输开销的场景：batch 处理类任务（摘要等）收益最大，交互式任务受第二 token 延迟影响。
|
||||
- 前提是高速互连数据中心；无 InfiniBand/NVLink 级带宽的环境下拆分收益会显著缩水。
|
||||
- 未来若 GPU 缓存技术（如长上下文缓存避免重算）改变 prompt 阶段的内存画像，表征结论可能失效（论文自身也承认此点）。
|
||||
- 容错、调度器扩展性、动态功耗等生产关键问题未解决。
|
||||
|
||||
---
|
||||
|
||||
## 可引用金句
|
||||
|
||||
> "Unlike prompt computation, token generation does not need the compute capability of the latest GPUs and can be run with lower power and cost."
|
||||
> （与 prompt 计算不同，token 生成并不需要最新 GPU 的计算能力，可以用更低的功耗与成本运行。）
|
||||
|
||||
> "Running both phases on the same machine often leads to inconsistent end-to-end latencies due to the arbitrary batching of prompt and token phases."
|
||||
> （在同一台机器上运行两个阶段，常因 prompt 与 token 阶段的随意混合批处理而导致端到端延迟不稳定。）
|
||||
|
||||
---
|
||||
|
||||
## 总体评价
|
||||
|
||||
**亮点**：
|
||||
- 系统化表征了 LLM 推理两阶段的硬件资源画像差异，数据一手且翔实（A100/H100 实测）
|
||||
- "异构降配 + 功耗上限"的集群设计思路（HA、HHcap）直接把成本/功耗优化落到集群拓扑层面，工程可操作性强
|
||||
- KV cache 逐层异步传输的工程方案精炼，与 Mooncake 的逐层 prefill 形成呼应
|
||||
- 模拟器 + 性能模型方法论严谨（MAPE <3%），可复用于集群供给规划
|
||||
|
||||
**不足**：
|
||||
- 收益数字主要来自模拟器评估，缺大规模真实部署验证
|
||||
- 对调度器扩展性、容错等生产约束讨论偏简略
|
||||
- 未考虑动态功耗、缓存普及对未来硬件画像的影响
|
||||
|
||||
**适用场景**：云厂商推理集群规划者、追求成本/功耗优化的 serving 基础设施团队；研究异构调度与数据中心级 LLM 部署的研究人员。
|
||||
|
||||
**关联建议**：与 DistServe（goodput 视角的阶段解耦）、TetriInfer（混合负载干扰）、Mooncake（生产系统 KVCache 中心化）对照，可形成"解耦 serving"的完整谱系；后续可关注微软相关后续工作与 vLLM 解耦功能的演进。
|
||||
|
||||
---
|
||||
|
||||
## 配图
|
||||
|
||||
|
||||
Reference in New Issue
Block a user