Distributed Inference Performance Optimization for LLMs on CPUs

With the unprecedented success of Large Language Models (LLMs) across diverse domains (Cui et al. (2023), Thirunavukarasu et al. (2023)), the performance of LLM inference is paramount for extensive LLM applications (Zhao et al. (2023)). In the deployment of LLMs, we encounter numerous challenges, including substantial memory consumption, stringent latency targets, and long sequence lengths. Moreover, these challenges hindered the practical applications in low-resource environments.

As we known, LLMs primarily utilize the Transformers architecture (Vaswani et al. (2017)), which exhibits high parallelism. However, efficiently deploying these models in practical applications presents challenges. This is because inference generation occurs token by token, with each token’s computation relying on previously generated tokens. Multiple deployment optimization solution for LLM has been proposed, such as Miao et al. (2023), Agrawal et al. (2023) and Zheng et al. (2023). While these solutions are primarily designed for GPUs, when GPU hardware resources are limited, we can explore alternative options on CPUs. Therefore, an efficient distributed solution for LLM on CPUs is proposed. A better distributed solution for LLM inference performance optimization on CPU is crucial for cost-savings, efficient hardware usage, and optimal inference strategies. It could help achieve the required scalability and efficient low-latency inference.

In this paper, we propose three approaches which helps optimize the distributed inference performance for LLMs on CPUs. We conduct experiments with the proposed solution on 5th Gen Intel^® Xeon^® Scalable Processors, and the results indicate that the LLM with 72B parameters achieves a time per output token of 140 ms, significantly surpassing the average human reading speed of approximately 200 ms per token.

To optimize the distributed inference performance, minimizing communication cost wherever possible is important (Bajović et al. (2016)). We utilize the oneAPI Collective CommunicationsLibrary (oneCCL) designed with the aim of creating a unified standard API compatible with various types of hardware accelerators. Furthermore, we proposed multiple optimization approaches to enhance the LLM inference performance on CPUs as follows.

We conducted experiments using Qwen which is a large-scale pre-trained model developed by Alibaba Group (Bai et al. (2023)) with model parameter sizes of 72B. Qwen-72B is a Transformer-based large language model. To illustrate performance results, we measured the per token time of the next token generation on the 4 * Intel^® Xeon^® Scalable Processors 8575C, where each device has 1 socket, and each socket has 48 cores. With input size = 512 tokens and batch size = 1, the results shows time per output token is 140 ms/token, much faster than human reading speed.

We propose an efficient distributed inference performance optimization solution for LLMs on CPUs by leveraging oneCCL. The experiment results shows the promising per-token generation latency is 140 ms. In our future endeavors, we aim to enhance distributed LLM inference as a contribution to the open-source community. Furthermore, we intend to expand our approach to encompass a wider variety of CPUs, thereby empowering generative AI on CPUs in resource-limited environments.