Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models

The outstanding code generation capabilities exhibited by LLMs have attracted considerable attention from researchers (Wang et al., 2021; Li et al., 2022; Fried et al., 2022; Xu et al., 2022; Roziere et al., 2023; Zheng et al., 2023a). Some representative Code LLMs, such as CodeX (Chen et al., 2021), CodeGen (Nijkamp et al., 2022), and AlphaCode (Li et al., 2022), have achieved notable performance in code generation, program repair and code translation. Currently, research on LLMs for code primarily focuses on data and pretraining methods. For training data collection, WizardCoder (Luo et al., 2024) introduce code instruction-following training constructed by Evol-Instruct to enhance the capabilities of Code LLMs. For pretraining methods, StarCoder (Li et al., 2023a) and DeepSeek-Coder (Guo et al., 2024) incorporate fill-in-the-middle training task to enhance the model’s capability to handle various structural arrangements in code. With the rapid advancement of Code LLM capabilities, there is an increasing demand for reliable and comprehensive code evaluation benchmarks.

The existing benchmarks for LLM-based code (Ni et al., 2023), such as HumanEval (Chen et al., 2021), APPS (Hendrycks et al., 2021), MBPP (Austin et al., 2021), CodeContests (Li et al., 2022), and DS-1000 (Lai et al., 2023), focusing on the correctness of generated code in scenarios such as code exercises, data science, and competitions (Yan et al., 2023; Li et al., 2023b; Shinn et al., 2024). However, these efforts only focus on the correctness of the generated code, using the pass rate of test cases as the sole evaluation metric. Meanwhile, there has been a recent trend in considering other dimensions (Li et al., 2024; Jain et al., 2024b; Tian et al., 2024); for example, Huang et al. (2024) evaluate the efficiency of the generated code, while Dillmann et al. (2024) bridge the connection between cross-entropy and logical lines of code (LLOC). Nevertheless, these studies neither account for the demand-dependent nature of these dimensions nor systematically evaluate the LLM’s code capabilities across multiple dimensions.

To investigate the impact of incorporating customized instructions on code correctness, we evaluate the accuracy of the LLM-generated code on the original benchmark tasks and also calculate the accuracy when provided with instructions containing customization requirements.

To thoroughly investigate the impact of customized instructions across various tasks, we select the following datasets: HumanEval+ and MBPP+ (Liu et al., 2024) for code exercise problems, ClassEval (Du et al., 2023) for class-level code generation, and LeetCode (Guo et al., 2024) for coding competition problems. To mitigate the bias introduced by additional information in the original dataset on the customized requirements, we remove such information from the datasets.

To measure code correctness, we calculate the macro accuracy at the dataset level, which is the proportion of generated code that passes all test cases for the corresponding problems.

In real-world development scenarios, code is required to adhere to a consistent style to ensure comprehensibility and reduce the time cost of maintaining the code, which refers to code readability (Börstler et al., 2023). Specifically, the code length is the most straightforward aspect of style; excessively long lines of code can lead to incomplete screen display, severely affecting readability. Meanwhile, good and consistent naming styles help developers quickly understand the functionality of interfaces, and comments assist in rapidly comprehending the implementation logic of the code. Therefore, we summarize the code readability into three representative factors: Length, Naming Convention, and Comment. Based on the real-world development requirements, we collect corresponding customizable options for different factors.

For the Length factor, the readability requirements for code length vary due to the differences in display scales across different user scenarios. Therefore, we refer to the PEP8 style for Python, and define the following user requirements concerning code length: (60, 20), (70, 30), and (79, 40), with the parentheses corresponding to the maximum line length and the maximum lines of functions, respectively. For the Naming Convention factor, camel-case and snake-case are commonly used naming methods in computer programming, with varying preferences across different projects for naming functions and variables. Consequently, we offer the choice between camel-case and snake-case based on the naming convention used for functions or variables as customization options. For the Comment factor, different levels of granularity serve varying purposes and needs. Line-level comments aid in understanding the implementation details of the code, while function-level comments assist in comprehending the functionality and usage of functions. Additionally, line-level comments are particularly beneficial for novice programmers. Consequently, we have defined two customization options: function-level comments and line-level comments.

Code exercise tasks are derived from snippets of real-world development tasks, encapsulating scenarios encountered in actual development environments. Subsequently, leveraging these code exercise scenarios, we evaluate the generated code against customized readability requirements on HumanEval+ datasets to assess its alignment with specific criteria.

Furthermore, to measure code readability, we employed abstract syntax tree analysis and heuristic methods to assess code length, examine naming conventions, and distinguish between different levels of comment granularity.

The maintainability of code significantly impacts the long-term health of software and the efficiency of development teams. Many quality models propose empirical quantitative equations for maintainability. Simultaneously, the Single Responsibility Principle (SRP) is a crucial part of code design principles to avoid excessive functional coupling. Therefore, based on these principles, we summarize two factors for code maintainability: Maintainability Metric and Modularity.

For the Maintainability Metric factor, we use the Maintainability Index (MI) (Coleman et al., 1994) to measure how maintainable the code is, which is widely used in the Microsoft Visual Studio 2010 development environment. This index is a four-metric polynomial equation, resulting in a value between 0 and 100, with higher values indicating greater maintainability. The formulation is as follows:

where $V$ is Halstead Volume to identify measurable properties of the code, $G$ is Cyclomatic Complexity corresponding to the number of decisions a block of code contains plus 1, $L$ is the number of source lines of code, and $C$ is the percent of comment lines. To comprehensively assess the maintainability requirements satisfaction of LLM-generated code, we employ ClassEval (Du et al., 2023) dataset, to ensure the complexity of the code problems.

For the Modularity factor, different application requirements dictate varying levels of modularization. Achieving compactness often necessitates implementing functionality using a single function, whereas maximizing code reusability demands the use of multiple functions. Accordingly, we define the following customization options: implementing functionality using 1, 2, or 3 functions. We choose to measure the modularity of generated code on LeetCode (Guo et al., 2024) dataset, which is more challenging to ensure better discriminative capability. In addition, we design corresponding rule-based methods to check the degree of modularity in the generated code.

In most applications, code efficiency is directly linked to user experience or business process efficiency. Generally, efficiency is assessed using time complexity and space complexity. Based on this principle, we define them as factors. Due to varying user-side hardware conditions, achieving a balance between execution time and memory usage, or optimizing one of these aspects to the extreme, is a common practice to ensure code efficiency. Recognizing these scenarios, we gather 101 cases from LeetCode programming problems designed to simulate such conditions. These cases are customized with specific time complexity requirements, space complexity requirements, or both, to evaluate the extent to which the LLM-generated code meets the efficiency requirements.

To measure code efficiency, we propose the Normalized Index (NI), i.e., to measure the degree to which the generated code satisfies the complexity requirement. Given two solution codes with time and space complexity $C_{1}^{T},C_{1}^{S}$ and $C_{2}^{T},C_{2}^{S}$, respectively, where $C_{1}^{T}$ and $C_{2}^{S}$ are better, and given their total running time $T_{1},T_{2}$ ($T_{1}<T_{2}$) and memory usage $S_{1},S_{2}$ ($S_{1}>S_{2}$) on all test cases. Now there is a code $\hat{C}$ to be evaluated, which has a running time $\hat{T}$ and memory usage $\hat{S}$, with requirements $C_{1}^{T},C_{1}^{S}$, then the normalized index is:

$\mathrm{NI}_{T}$ indicates the degree of time complexity toward $C_{1}^{T}$, and $\mathrm{NI}_{S}$ indicates the degree of space complexity toward $C_{2}^{S}$.

To conduct a more in-depth analysis of how various factors across different dimensions influence overall code quality, we analyze the correlations between different factors in each model. Specifically, we first compute the proportion of the generated code that is both correct and follows customized instructions across 8 factors for 18 Code LLMs. Subsequently, we calculate Pearson correlation coefficients between these factors.

The results of the correlation analysis are presented in Figure 4, which demonstrate that readability serves as a critical indicator of overall code quality. Specifically, significant correlations are observed between accuracy and almost all the factors, with correlation coefficients mostly exceeding 0.8, notably exceeding 0.9 in relation to correctness. For instance, if a segment of LLM-generated code exhibits consistent naming conventions, appropriate length constraints, or suitable comments, it is more likely to be of overall high quality. This finding aligns with conclusions from  Börstler et al. (2023), which identifies code readability as a decisive factor in code quality, suggesting that enhancing the readability of code generated by LLMs is a crucial avenue for improvement. Furthermore, we analyze the comment factor within the readability dimension and compare the changes in the accuracy of LLM-generated code before and after incorporating comments. As illustrated in Figure 5, requiring models to include comments in appropriate sections enhances the performance of some LLMs. We posit that this phenomenon could be attributed to an emerging ability in large-scale LLMs (Wei et al., 2022; Schaeffer et al., 2024), wherein comments serve an implicit chain-of-thought role, thereby improving the accuracy of generated code.

To investigate whether the model’s inherent preferences affect its ability to follow user instructions, we conduct a more fine-grained comparison of various incorporated user requirements. Specifically, we focus on naming conventions, requiring LLMs to consistently use either camel-case or snake-case for both function names and variable names. Additionally, regarding code length, LLMs are tasked with generating single-line lengths not exceeding 60, 70, and 79 characters, as well as ensuring that individual method lengths do not exceed 20, 30, and 40 lines. For loop structures, the requirement is for LLMs to use only either for or while statements to implement necessary loop constructs. Finally, we calculate the proportion of LLM-generated code that follows these customized requirements, i.e. the rate of instruction-following (IF).

Figure 6 demonstrates the IF rates of 18 LLMs across all the customized requirements above. We find that the majority of LLMs exhibit an inherent preference bias towards generating code in specific styles. This bias often results in these LLMs being unable to follow user instructions effectively if the requested style differs from that prevalent in their training data. Specifically, for naming conventions, Python conventionally employs snake-case for function and variable names. When LLMs are requested to use camel-case, most LLMs, such as CodeLlama and CodeQwen, almost fail to comprehend and fulfill this requirement, with instruction-following rates below 20%. For code length, a 79-character single-line length limit is a common style in Python. When dealing with more stringent requirements, the IF rates of most instruct-type LLMs drop by nearly 20%, while DeepSeek-Coder-V2-236B maintains the best instruction-following rates. For loop structures, all LLMs except GPT series and DeepSeek-Coder-V2-236B exhibit a pronounced tendency to use "for" statements. When tasked with using "while" statements, LLMs struggle to transform between different statements, reflected in the IF rates generally below 70%. These findings imply that most LLMs may simply learn the inherent patterns of the next token from examples, without a clear understanding of the logic of code comprehension. However, GPT-3.5, GPT-4o, and DeepSeek-Coder-V2-236B perform well in this aspect, with an IF rate above 90%. Such preference bias can lead to the ossification of code style in Code LLMs, thereby hindering their ability to meet specific real-world project requirements and consequently affecting the adaptability and scalability of generated code. This issue may be more pronounced in programming languages like Perl, JavaScript, and PHP, where there is no strict, widely accepted standard code style.