CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference

Generative LLM inference computation. LLM inference is a process that involves generating a sequence of output tokens in response to an input prompt, which is a list of tokens. This process consists of two phases: prefill and decoding. During the prefill phase, the input sequence is used to generate the key-value cache (KV cache) for each transformer layer; the decoding phase then uses the previous KV cache to generate new tokens step-by-step and update the KV cache accordingly. The computation of one transformer layer can be summarized as follows. Denote the batch size by $B$, the prompt sequence length by $L$, the hidden dimension of the transformer by $H$, and the intermediate size by $H^{\prime}$. We have weight matrices of the $i$-th transformer layer: $\mathbf{W}_{K}^{i},\mathbf{W}_{Q}^{i},\mathbf{W}_{V}^{i},\mathbf{W}_{O}^{i}\in\mathbb{R}^{H\times H}$, $\mathbf{W}_{1}^{i}\in\mathbb{R}^{H\times H^{\prime}}$, and $\mathbf{W}_{2}^{i}\in\mathbb{R}^{H^{\prime}\times H}$. During the prefill phase, let $\mathbf{x}^{i}$ be the input of the $i$-th transformer layer, and the key, value, query, and output of the attention layer respectively specified as $\mathbf{x}_{K}^{i},\mathbf{x}_{V}^{i},\mathbf{x}_{Q}^{i},\mathbf{x}_{\text{Out}}^{i}\in\mathbb{R}^{B\times L\times H}$. The computation of the cached key, value is given by $\mathbf{x}_{K}^{i}=\mathbf{x}^{i}\cdot\mathbf{W}_{K}^{i}$ and $\mathbf{x}_{V}^{i}=\mathbf{x}^{i}\cdot\mathbf{W}_{V}^{i}$. The remaining computation in this transformer layer is given by

LLM adaption. Adapting LLMs in a parameter-efficient manner is a popular approach to enhancing their performance for domain-specific tasks or customizing the model inference results to align with human intents [17, 18]. One notable approach is Low-Rank Adaptation or LoRA [9], which introduces an adapter to modify the intermediate LLM inference results while keeping the original LLM parameters unchanged. Specifically, given a pre-trained weight matrix $\mathbf{W}\in\mathbb{R}^{H_{1}\times H_{2}}$, an adapter consists of two low-rank matrices $\mathbf{A}\in\mathbb{R}^{H_{1}\times r}$ and $\mathbf{B}\in\mathbb{R}^{r\times H_{2}}$, where $r$ is the LoRA rank. LoRA adapts this weight matrix to $\mathbf{W}^{\prime}=\mathbf{W}+\mathbf{AB}$. Let $\mathbf{y}$ be the original output of this layer given by $\mathbf{y}=\mathbf{x}\mathbf{W}$. With LoRA adaption, the updated computation becomes

The need of LLM-multiplexing. A naive way to serve a LoRA adapter [9] is to merge its weights into the weights of the base LLM, which introduces no additional computational overhead when deploying the adapted model for inference. However, this approach does not scale to multi-tenant LoRA serving: because one base model can only merge with one LoRA adapter at a time, serving $n$ different LoRA models requires duplicating $n$ copies of the base LLM, wasting GPU memory and missing opportunities for batch inference [11].

Continuous batching. Existing LLM serving systems employ a continuous batching strategy optimized for LLM’s iterative auto-regressive generation process [31, 11, 16, 10]. Continuous batching operates at the iteration level, where completed requests are immediately removed from the running batch after each iteration to make room for new requests to join. This allows a new request to be incorporated in just one iteration without waiting for the entire batch inference to complete. Continuous batching significantly improves the token generation throughput while minimizing the request queuing delays. Fig. 2 illustrates this batching process used in existing systems [10, 11, 16], where the decoding and prefill phases interleave as new requests arrive. Upon a request’s arrival, the decoding phase (Dec) is preempted to perform prompt processing, which involves loading the requested LoRA adapter (Load) and prefilling (Pre). Once completed, the new requests join the running batch, and the system combines them together to continue the decoding process.

C1: High cold-start overhead. To save GPU memory, existing systems only cache the base LLM on GPU while keeping all its LoRA adapters in host memory [23, 1]. When a new request arrives, the system fetches the corresponding adapter from the host to the GPU, leading to an adapter loading phase that must complete before the prefill phase begins (Fig. 2). This results in a severe cold-start problem, where loading an adapter from the host to a GPU can take between a few to tens of milliseconds, depending on the adapter size (Fig. 3-Right). Cold-start degrades the service responsiveness, measured by time-to-first-token [4, 23]. Moreover, under continuous batching, each time a new request arrives, the decoding phase of in-flight requests is blocked until the new arrival’s prefill phase completes (Fig. 2). As new requests keep arriving, their cold-start overhead cumulatively delays the token generation of an in-flight request (as shown in Fig. 2, where R1 experiences two cold-starts due to the arrivals of R2 and R3). We empirically validate this issue by multiplexing a Llama2-7B model with a group of 512 LoRA adapters (rank=64). These adapters have skewed popularity (Fig. 12) following the Microsoft Azure Function (MAF) trace [21]. We configured Poisson request arrivals with various aggregate loads. Fig. 3-Left shows the proportion distribution of the cold-start overhead, which, on average, accounts for 10%, 16%, and 20% of the entire request serving time when the aggregate load is 3, 6, and 9 requests per second, respectively.

C2: Request scheduling for heterogeneous LoRA serving. In multi-tenant LoRA serving, users often request to use heterogeneous LoRA adapters with varying ranks [23]. These heterogeneous adapters can be batched together to multiplex one base LLM using specialized kernel implementations, such as the Batched Gather Matrix-Vector Multiplication (BGMV) kernel in Punica [1] or the Multi-size Batched Gather Matrix-Vector Multiplication (MBGMV) kernel in S-LoRA [23]. Specifically, when batching a set of heterogeneous LoRA adapters, BGMV pads adapters of smaller ranks to the highest rank to perform batch operations, while MBGMV does not use padding [23]. As a result, BGMV’s performance is determined by the maximum rank in the batch, whereas MBGMV’s performance depends on the average rank. We measure the decoding latency of batch serving heterogeneous LoRA adapters using these two kernels with various batch configurations, and the results are depicted in Fig. 4. We observe significant performance variations when batching different sets of heterogeneous adapters. This highlights the need for intelligent request scheduling that takes into account the rank heterogeneity and the batching performance of a specific kernel implementation.

GPU LoRA. As the base LLM is “pinned” on GPU, running LoRA adapters on the same device saves the communication overhead and is usually more efficient than running them on CPU. To maximize the token throughput, LoRA computations (i.e., $\mathbf{xAB}$ in Eq. (1)) are batched in each attention layer during base LLM inference. This can be achieved with a specialized CUDA operator [1, 23]. In CaraServe, we adapt the Batched Gather Matrix-Vector Multiplication (BGMV) operator [1], which parallelizes the LoRA weight gathering and computation for efficient execution. The LoRA output is then added to the base output in the self-attention computation, following in Eq. (1). For an efficient implementation, we incorporate the operators of GPU LoRA computation into the base LLM inference process, as shown in Fig. 7.

CPU LoRA. LoRA computation can also be executed using the CPU, which requires layer-wise synchronization with the base LLM inference running on the GPU. Specifically, at each attention layer, the base inference process transfers the input tensor $\mathbf{x}$ in Eq. (1) from the GPU device memory to the host memory (Fig. 7). The CPU LoRA process then performs computation and transfers the result $\mathbf{xAB}$ back to the GPU device. In the meantime, the base inference process proceeds to compute $\mathbf{xW}$, which is finally adapted with the received LoRA output following Eq. (1). Although CPU LoRA requires synchronization, it can start immediately because the LoRA weights are already in memory. We hence utilize it to address the cold-start problem that arises in GPU LoRA (C1 in §2.3).

Mitigating GPU cold-start with CPU assistance. As illustrated in Fig. 1, when a new request arrives and the corresponding adapter is not available on the GPU, the server fetches it from host memory and, in the meantime, starts its prefill computation using the CPU. Once the adapter is fully loaded, the GPU LoRA takes over, finishing the remaining prefill computation not done by the CPU, if any, and the subsequent decoding process. Fig. 7 illustrates how CPU and GPU LoRA computations are coordinated in our design, where we run CPU LoRA adapters as isolated, concurrent processes for resource/failure isolation and improved performance.

Challenges. Though hosting LoRA computation in isolated CPU processes effectively addresses the cold-start problem, it poses three challenges to system implementation. First, running LoRA in CPU processes requires layer-wise synchronization between the GPU-based LLM inference to ensure data validity. Second, frequent triggering of LoRA computation in each attention layer leads to high invocation overhead, such as inter-process data transfer. Third, using CPU to compute adaptation can be slow given its limited parallelization capability, especially when the input prompt is long.

Sync-free CPU LoRA invocation. Most LLM serving systems achieve low latency through asynchronous GPU computation in PyTorch-like frameworks [11, 16, 28, 23, 10]. However, adapter serving requires careful coordination between base LLM inference running on GPU and LoRA invocation running on CPU to ensure correctness and good performance.

Shared memory data transfer. Transferring data and signals between the base LLM process and the isolated CPU LoRA processes requires inter-process communication (IPC). This is a one-to-N communication involving one base LLM inference process and multiple CPU LoRA processes. (We explain why multiple CPU LoRA processes later.) We utilize shared memory for fast inter-process data transfer, eliminating the need for data copying and serialization (Fig. 7). After the base LLM process executes our customized operator (see Fig. 8), the CPU LoRA processes will soon be signaled to start reading the input matrix $\mathbf{x}$ from the shared memory and perform the computation $\mathbf{xAB}$. They then write $\mathbf{xAB}$ back to the shared memory and notify the LLM inference process to incorporate the adaptation results (Eq. (1)). Micro-benchmark evaluations (§7.4) demonstrate that the use of shared memory reduces data transfer overhead to less than 1 ms (Fig. 17), substantially outperforming the message passing IPC employed by existing LLM frameworks [16].

Profiling-guided LoRA parallelization. Given that the CPU has lower computing power and limited parallelization capability compared to the GPU, performing LoRA adaptation using a single CPU is not scalable. Therefore, we propose a profiling-guided parallelization scheme to accelerate LoRA adaptation using multiple CPU cores. As discussed in §2.1, the adaptation computation is $\mathbf{xAB}$, where $\mathbf{x}\in\mathbb{R}^{B\times L\times H}$ is the input matrix for $B$ requests with $L$ tokens, totaling $B\times L$ tokens. We first profile the performance achieved by a single core under varying workloads (Fig. 18-Left) and set the maximum workload for a single CPU, which is the maximum number of tokens a CPU core can handle for computation. For example, if one core can handle $c$ tokens, we allocate $\lceil\frac{L}{c}\rceil$ cores for computing the adaptation results of each request with weight matrix $\mathbf{W}$. Each core is dedicated to an isolated CPU process to avoid interference. Specifically, the CPU process reads a slice of $\mathbf{x}$ from the shared memory region, performs the computation, writes the results back to the shared memory, and notifies the base LLM process accordingly. Compared to PyTorch’s native multi-threading module [7], this approach achieves $1.7\times$ speedup when using 8 CPUs for the same workload (Fig. 18-Right).

Model and server configurations. We adopt Llama2 [28] models with 7B, 13B and 70B parameters for evaluation (details in Tab. 2), where LoRA adapters are applied to $\mathbf{W}_{Q}$, $\mathbf{W}_{K}$, and $\mathbf{W}_{V}$ (§2) in LLM’s attention layers following the standard settings [5, 9, 23]^*** Following the setting in [1, 23], we use dummy weights for LoRA models, which do not affect system performance. .

Metrics. We use the following metrics in evaluation, which are considered essential in user-facing LLM serving [4, 23].

Baselines. We consider the following baselines.

• •

  Cached represents an Oracle method where all required LoRA adapters are pre-cached in unlimited GPU memory. It has no adapter loading overhead, thus achieving performance upper bound.

• •

  OnDmd loads LoRA adapters on demand. It will suffer from the cold-start overhead if the required LoRA adapters are not on GPUs.

• •

  S-LoRA [23] represents a state-of-the-art multi-tenant LoRA serving framework, which is also built on top of LightLLM [16]. It loads LoRA adapters on demand and uses an adapted CUDA kernel for GPU LoRA computation.

Note that we equip baselines other than S-LoRA with the BGMV kernel [1] to perform GPU LoRA computation for a fair comparison in the single GPU case.

Workloads. We use both synthetic and scaled production workloads in our evaluation.

For both workloads, we set each request’s input prompt and output length according to the Alpaca dataset [26, 11], which contains input and output texts of real LLM services. Like S-LoRA [23], we run each workload for 5 minutes.

Synthetic workloads. We generate traces using a Poisson process with an aggregate $RPS=9$ and set the LoRA adapter rank to 64. We measure the performance of each baseline using the metrics discussed in §7.1. Fig. 10 plots the CDFs of time metrics, demonstrating that CaraServe can rival Cached and outperform OnDmd/S-LoRA.

Scaled production workloads. We next evaluate CaraServe on a production workload based on the MAF trace [21]. Fig. 12 illustrates the skewed distribution of function popularity. We evaluate each baseline with an increasing number of LoRAs and their workloads in a single LLM inference server. More LoRA adapters mean heavier request loads, and each new request is more likely to invoke a new LoRA adapter that needs to be loaded onto GPU on demand (Fig. 12). The average aggregate $RPS$ for $128/256/512$ adapters is $1.5/3.6/7.7$, respectively, scaled from the original trace.

Sync-free CPU LoRA invocation. To analyze the performance of our optimized CPU LoRA invocation kernel, we use the Llama2-7B model on one A10 GPU to measure the prefill latency of the PyTorch’s native implementation and our optimized kernels. As Fig. 16 shows, our customized kernel performs better than the default PyTorch kernel. As the total number of tokens in prefill phase increases, CaraServe’s kernel gains up to a $16\%$ performance increase.

Shared memory data transfer. We compare the latency of computing CPU LoRA with different IPC methods: shared memory and UNIX domain socket. We measure the time it takes to perform LoRA computation and the data round trip cost. We increase the number of receiver processes to represent the increase in the number of CPU LoRA processes (§4.2). Fig. 17 shows that as the number of receiver processes increases, the domain socket-based approach suffers from linear time increase in initialization and serialization overhead, whereas the shared memory-based approach obtains near-constant performance.

Multi-CPU computation. We first profile the LoRA computation performance during a prefill phase with a single CPU. We profile it with different workloads (number of tokens to process). We run the profiling on a Llama2-7B model with a single A10 card. As shown in Fig. 18-Left, the CPU has limited parallelism and does not scale to fit high workloads. Fig. 18-Right illustrates the performance of prefilling a prompt of 128 tokens with CaraServe’s multi-CPU design (§4.2) or the native multi-core utilization of PyTorch multi-threading module [7]. We can see that CaraServe’s design achieves up to $1.7\times$ speedup.

Baselines. Upon the arrival of new requests, we consider the following scheduling baselines for comparison:

Setup. Following [33], we run experiments in two settings: a large-scale simulation and an 8-instance real-world testbed.

Large-scale simulation. We first evaluate the scheduler’s performance through simulation, where we obtain the prefill and decoding latency of the simulator by profiling. We include all 40,000 functions from the MAF trace [21], with aggregated $RPS\approx 340$, and use 60 simulated servers. We set the SLO regarding time per token, as it corresponds to the perceived "speed" of the inference service. The SLO is set to $1.5\times$ higher than that achieved by the HF-PEFT solution (§1). Fig. 19-Top shows that with S-LoRA’s MBGMV, CaraServe’s scheduler achieves an SLO attainment of $99\%$ and speeds up the average time per token by $16.1/18.8/36.4\%$ compared to the MostIdle/Random/FirstFit. Fig. 19-Bottom shows the performance with Punica’s BGMV kernel, which is also adopted in CaraServe (§4.1). Our scheduler has $99\%$ SLO attainment and accelerates time per token up to $36.0\%$.

Testbed. Next, we evaluate the scheduler in a small-scale testbed, which has 8$\times$A10 GPUs to support 8 Llama2-7B models. Due to the limited number of available CPUs, we use Cached (§7.1) as the LoRA serving backend, as our CPU-assisted design can rival its performance in various settings (§7.2, §7.3). We randomly sample 1,200 requests with an aggregated $RPS\approx 60$ from the MAF trace [21]. The SLO is also set regarding time per token, which is $1.5\times$ higher than that achieved by the HF-PEFT solution. As illustrated in Fig. 20, CaraServe outperforms other baselines by achieving the highest SLO attainment of 80%.