PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs

The concepts encompass language-specific constructs like variables, data types, control flow, and conditions to generic constructs like Algorithms, OOPs, etc. We, therefore, propose a hierarchy of programming concepts in which a complex concept might require knowledge of several basic concepts. For example, sorting algorithms like Quicksort or Mergesort require a thorough understanding of data structures such as arrays and linked lists, as well as proficiency in algorithmic analysis and time complexity²²2https://shorturl.at/nrBTX. Each programming concept is an intrinsic feature of a problem. We next describe the proposed hierarchy:

• •

  Basic Concepts: At the basic level, concepts involve the application of elementary syntax principles, encompassing the utilization of variables, manipulation of diverse data types, basic input/output operations, comprehension of control flow and conditional statements, basic handling of data structures, functions, and knowledge of essential built-in libraries. Problems leveraging basic concepts primarily aim to evaluate the core competencies within a designated programming language.

• •

  Intermediate Concepts: Intermediate-level concepts involve a comprehensive understanding of multiple foundational concepts and their adept integration. For example, extending basic data structures to implement Stack, Hash, Queue, etc. Problems comprising intermediate concepts evaluate a higher level of proficiency in programming.

• •

  Advance Concepts: Concepts include implementation knowledge of advanced data structures such as Tree, Heap, etc., algorithmic paradigms such as Greedy, Divide and Conquer, and Dynamic Programming, and Concurrent and Parallel Programming. Problems comprising advanced concepts focus on evaluating sophisticated problem-solving and design capabilities.

We curate a list of 38 programming concepts from three popular coding platforms (GeeksForGeeks, 2023; LeetCode, 2023; hackerearth, 2023). We further assign each concept to one of the three hierarchy levels. Table 1 presents the curated concepts and the proposed hierarchy.

An annotator, with their expertise and experience in programming, can perceive a programming problem as belonging to one of three difficulty levels: Easy, Medium, or Hard (Hendrycks et al., 2021). Thus, difficulty level is an extrinsic feature of a problem. This subjective assessment is based on a complex combination of factors, such as knowledge of programming concepts, problem-solving skills, experience with similar problems, and coding proficiency. It is important to note that perceived difficulty is subjective and can vary significantly between annotators. A problem considered easy by one annotator due to their prior experience and knowledge might be deemed challenging by another who lacks those same advantages. Furthermore, the perceived difficulty of a problem can also evolve over time as an annotator develops their skills and knowledge. A problem that initially seemed challenging may become easier with practice and exposure to similar problems. Conversely, an annotator may encounter a problem that initially appears straightforward but then finds themselves struggling due to hidden complexities or unforeseen challenges.

In this paper, we focus on Python Programming language and conduct human experiments with two popular Python-based code generation benchmarks to showcase extensive selection bias and poor diversity in the curated problems. The following section describes the human experiments in detail.

This study is grounded on the two most widely recognized Python code generation benchmarks: (i) HumanEval (Chen et al., 2021) and (ii) MBPP (Austin et al., 2021). Recent Code-LLMs including STARCODER (Li et al., 2023), LLaMA (Touvron et al., 2023), METAGPT (Hong et al., 2023), Code Llama (Roziere et al., 2023), SANTACODER (Allal et al., 2023), CodeGeeX (Zheng et al., 2023), Gemma (Team et al., 2024), and Claude 3 (Anthropic, 2024) have employed these two benchmarks to assess their performance. We next briefly describe these benchmarks.

• •

  HumanEval Dataset: HumanEval dataset was introduced alongside Codex (Chen et al., 2021)³³3Codex is a GPT-based language model fine-tuned on publicly available codes from GitHub.. It comprises 164 hand-crafted Python programming problems⁴⁴4Dataset is available here: https://github.com/openai/human-eval. Each problem description contains a function signature, docstring, body, and multiple unit tests. Figure 4 illustrates a representative problem. On average, each problem is associated with 7.7 unit tests.

• •

  Mostly Basic Programming Problems (MBPP) Dataset: The MBPP dataset (Austin et al., 2021) evaluates models that can synthesize short Python programs from natural language descriptions. The benchmark⁵⁵5Dataset is available here: https://github.com/google-research/google-research/tree/master/mbpp consists of about 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. Each problem consists of a task description, code solution, and three automated test cases. Figure 5 presents a representative problem.

Both benchmarks evaluate model performances against one of the most popular metrics pass@k. We formally define pass@k in appendix A.1.

Next, we conducted two human annotation studies to gain insights into the diversity in programming concepts and difficulty levels of the two proposed benchmarks. Each study involved the recruitment of the same set of five annotators. Each annotator is a postgraduate student in Computer Science with at least three years of experience in Python programming and competitive programming. It is noteworthy that each participant willingly volunteered throughout the entire duration of the experiment, and no remuneration was provided. Internet access was prohibited during the entire annotation period. Annotators were encouraged to utilize any brute-force technique they considered appropriate without prioritizing optimized solutions. No time constraints were imposed to prevent hasty or fatigue-induced decisions. Each annotator was presented with 164 problems from HumanEval and a randomly selected set of 164 problems from MBPP. We next describe the two annotation studies:

• •

  Programming Concepts Diversity: In this study, we adopted a single-concept annotation approach, where annotators assigned one programming concept (detailed in Section 3.1) to each problem. This selection represented the concept they considered most crucial for successful problem-solving. Our annotation guidelines explicitly prohibited assigning multiple concepts to any single problem, ensuring focused and unambiguous mapping between problems and relevant concepts.

• •

  Difficulty Level Diversity: In this study, each annotator categorized the problems into three distinct difficulty levels: Easy, Medium, and Hard, based on their individual expertise and experience.

We notice a considerable selection bias favoring certain programming concepts and simpler problems in both benchmarks. Consequently, we assert that current benchmarks overstate the generalization abilities of existing code-LLMs.

Aligned with the problem curation strategies employed in established benchmarks Hendrycks et al. (2021); Lai et al. (2023); Zhu et al. (2022), we curate problems from two prominent coding platforms: GeekForGeeks (GeeksForGeeks, 2023) and LeetCode (LeetCode, 2023). To comprehensively represent each proposed programming concept (detailed in Section 3.1), we curated five problems per concept. This diverse set comprises one Easy problem, two Medium problems, and two Hard problems, ensuring a balanced distribution across difficulty levels (20%, 40%, and 40%, respectively) within the PythonSaga Dataset.The difficulty levels for each question were specified by their respective source platforms, GeekForGeeks, and LeetCode.

To enhance human-friendliness and ground the problems in realistic contexts, each shortlisted problem statement undergoes a manual rephrasing process without any aid from AI tools. Furthermore, a comprehensive description of input and output formats, accompanied by relevant examples, is supplied with each problem statement to ensure a thorough understanding of the task by Code-LLM. This multi-step approach aims to retain the core knowledge and essential solution steps while integrating them into relatable real-world scenarios. This reconstruction involves reformulating the entire problem statement while preserving its fundamental functionality. This deliberate transformation enhances the challenge for Code-LLMs, requiring them to move beyond simple pattern matching and grasp the nuanced context embedded within the prompt to devise a solution effectively. For example, the problem statement “Given a string str, find the length of the longest substring without repeating characters.” is paraphrased as “Let’s say you attend a car show where cars of different brands are showcased in a row. Find the length of the longest stretch where no two cars are of the same brand. Take the input from the user for the brands of the cars in the order they are placed in the row. Print the length of the longest stretch where no two cars are of the same brand”.

Overall, PythonSaga comprises five problem instances from each programming concept, resulting in a total size of 185 problems. Each problem is associated with a maximum of four test cases, with an average of 3.7 test cases per problem. PythonSaga’s structure resembles HumanEval and MBPP, wherein each problem comprises a function signature, docstring, body, and multiple unit tests. A representative example is present in Appendix A.2.

Next, we benchmark several open and closed-source LLMs on PythonSaga. Open-source models include StarCoderBase (Li et al., 2023), StarCoder2 (Lozhkov et al., 2024), Code Llama (Roziere et al., 2023), CodeQwen1.5-Chat (Bai et al., 2023), Nxcode-CQ-orpo (NTQAI, 2024), Mistral-Instruct-v0.1 (Jiang et al., 2023), Code Llama Instruct (Roziere et al., 2023), Deepseek Coder Instruct (Guo et al., 2024), Code Llama Python (Roziere et al., 2023) 7B & 13B, Llama 3 (Meta, 2024), Phi-2 (Javaheripi et al., 2023), OpenCodeInterpreter-DS (Zheng et al., 2024), and Deepseek Coder (Guo et al., 2024). Except for Mistral-Instruct-v0.1, the rest are Code-LLMs. In addition, we benchmark two closed-source models, including GPT variants GPT-3.5 (OpenAI, 2022) and GPT-4 (OpenAI and Achiam, 2024). While larger open-source options exist, our selection was restricted to models with less than 13B parameters due to computational resource limitations, which were limited to a single Tesla V100 in our case.

We evaluate model performances using pass@k metric. Adhering to previous studies like HumanEval (Chen et al., 2021), StarCoder (Li et al., 2023), Deepseek Coder (Guo et al., 2024) etc, we primarily utilized $k=1$, signifying that a model is considered successful if at least one of its generated solutions passes the defined evaluation criteria. However, we additionally explored $k=10$ to analyze model consistency across larger sets of responses. Notably, unlike prior works that varied the number of sampled responses ($n$), we consistently generated $n=20$ samples from both open and closed source models for a consistent evaluation.

Table 2 compares models against pass@1 and pass@10 metrics. Closed-source models performed considerably better than open-source models. Among open-source models, Code Llama Python (Roziere et al., 2023) 13B performed best, whereas, among closed-source models, GPT-4 (OpenAI and Achiam, 2024) performed best. Notably, the performance of closed-source models on PythonSaga is significantly lower than the respective performances in HumanEval and MBPP benchmarks (see Figure 6 present in appendix for more details).

Figure 3 illustrates the performance of each LLM on problems within specific programming concepts in the PythonSaga. We consider a model has successfully solved a problem if any one of the n(=20) generated samples passes all the test cases. As anticipated, all models exhibited better performance in solving problems associated with basic concepts compared to intermediate or advanced concepts. For example, Code Llama Python 13B solved 28.8%, 21.6%, and 10.9% of problems in these categories, respectively. Whereas GPT-4 solved 42.3%, 46.6%, and 32.8% of problems, respectively. In contrast to open-source models, closed-source models have successfully solved at least one problem from a majority of the concepts. Interestingly, none of the models could successfully solve any problem within six specific concepts, Basic Data Structures, Recursion, Hashing, Context Managers, Concurrancy and Parallelism and Max Flow. Notably, closed-source models exhibited a more consistent performance across categorization compared to open-source models, suggesting a potential advantage in handling diverse problem complexities.

Through our assessment of Code-LLMs using PythonSaga benchmark, we identified various error types that explain subpar performance of these LLMs.

• •

  Invalid syntax: Code-LLMs frequently produce syntactically incorrect Python code, including misused keywords, unmatched parentheses, and other grammatical errors that prevent successful execution. This is one of the most common errors exhibited by all Code-LLMs during code generation.

• •

  Incomplete code: Often, the LLMs generate incomplete code, halting mid-way through a line or statement, resulting in unusable fragments. This trait is evident in all open-source Code-LLMs but is particularly pronounced in models like StarCoder, Code Llama, Mistral, and Nxcode.

• •

  Invalid return statements: LLMs sometimes produce return statements that are incorrect or inappropriate for the given function, leading to runtime errors or incorrect outputs. All open-source Code-LLMs occasionally produce invalid return statements, particularly when addressing advanced-level questions.

• •

  Invalid declaration of functions and variables: LLMs sometimes incorrectly declare functions and variables, including wrong parameter lists, improper variable types, or undeclared identifiers, leading to code malfunctions. The first five Code-LLMs listed in Table 2 frequently call undeclared functions and variables or reference variables declared in prompt examples.

• •

  Subjective answers instead of code: LLMs sometimes provide explanations or descriptions instead of generating the required code, rendering the output unusable. Models like Llama 3, Phi-2, and GPT-3.5, trained for tasks beyond code generation, are particularly prone to this error.

• •

  Copying example answers: LLMs occasionally replicate answers provided in examples rather than generating original solutions. For complex or advanced problems, Code-LLMs like Code Llama Instruct and Deepseek Coder Instruct often use examples provided in the prompt rather than generalizing to other inputs.

• •

  Failure to pass test cases: A significant portion of the generated code fails to pass the given test cases. This demonstrates the model’s inability to produce functionally correct and robust solutions. Figure 3 clearly illustrates the inability of each model to pass test cases for various tasks.

• •

  Hallucination and irritable content: Code-LLMs often generate irrelevant code and off-topic content, including random elements, facts, or links, which detract from the intended solution. All open-source Code-LLMs frequently generate URLs, often linking to GitHub, LeetCode, or other coding sites. In contrast, GPT-3 and GPT-4 exhibit this behavior almost negligibly.

• •

  Non-compliance with problem statements: LLMs often fail to follow the format or methods outlined in problem statements, leading to invalid solutions that do not meet the specified requirements. Nearly all Code-LLMs struggle with generating compliant code when prompts become slightly complex, particularly in cases like Max Flow or Recursion.

By identifying and categorizing these errors, we highlight the need for improved benchmarks and evaluation methods that accurately reflect the capabilities and limitations of Code-LLMs.