Revisiting the Impact of Pursuing Modularity for Code Generation

With recent advances in the capabilities of large language models (LLMs; OpenAI, 2024; Gemini Team, 2024; inter alia), their application areas have expanded beyond simple text-based tasks. Among these, coding assistants are becoming practically essential for programmers, enhancing their efficiency through tasks such as natural language to code (NL2Code) generation.

Similar to other use cases of LLMs, coding assistants are typically utilized in zero- or few-shot manners, without task-specific fine-tuning. The problem is that as the length of code is usually much longer than that of a sentence, the number of code examples available for each run is strictly limited. Furthermore, the same functionality can be represented with different forms of code, making it challenging for users to select a proper example for a target task. It is thus important to understand what characteristics of the code provided to the agents contribute to the final performance of such models. Among the many possible properties that influence the characteristics of code snippets, this work investigates the impact of code modularity on the performance of LLMs for NL2Code generation.

Modular programming, the practice of building software with independent components, has long been considered a cornerstone of good software development. While this paradigm facilitates desirable properties of code for human programmers, such as reusability, readability, and maintainability, it remains an open question whether it offers the same level of effectiveness for LLMs.

Notably, Jain et al. (2024) argued that leveraging a set of modular functions can improve code generation accuracy for both in-context learning (ICL) and fine-tuning. As it is not trivial to guarantee the modularity of each code snippet, the authors asked GPT-3.5-Turbo¹¹1https://platform.openai.com/docs/models/gpt-3-5-turbo to convert an existing code snippet into a more modular one, while ensuring its functional correctness.

However, we claim that their report warrants revisiting for two reasons. First, since LLMs are notorious for their verbosity, it is unclear whether the conversion process aimed solely for modularity or accidentally introduced unexpected side effects. Second, the lack of a formally defined quantitative method for estimating modularity hinders more extensive analyses related to the problem.

In this paper, we (re-)investigate the effectiveness of pursuing modularity in NL2Code generation. We aim to push the boundaries of previous work by (1) introducing a novel metric that quantifies the modularity of a code snippet using numeric values. Based on the metric, we (2) classify code snippets as modular or non-modular without relying on LLMs, and evaluate how each category contributes to performance.²²2Note that this was infeasible in the previous study Jain et al. (2024) as there was no clear standard for determining whether each code snippet is modular or not. Moreover, beyond previous work, we (3) conduct experiments on models with parameters exceeding 7B (i.e., 33B and 34B) to investigate the impact of model scale. Figure 1 illustrates the core research question of this work.

In experiments, we discover that contrary to conventional wisdom in the literature, the modularity of a code example may not be the crucial factor for performance. We also explore potential explanations for why LLMs do not exhibit a preference for modular code compared to non-modular code.

To assess the impact of code modularity, the first essential step is to develop a method that provides a measurable score for code modularity. While the previous study Jain et al. (2024) bypassed this vital step,³³3The authors instead utilized LLMs to transform all code snippets into supposedly modular ones. we present a reasonable metric for estimating code modularity, which is challenging due to the inherent subjectivity of the concept itself.

Inspired by the software engineering literature, we employ the concept of Cyclomatic Complexity (CC) McCabe (1976) to determine the ideal number of modules, $m^{*}$, for a given code snippet. CC counts the number of independent execution paths in the control-flow graph (CFG) representation of the target code. CC can also be calculated as $E$ - $N$ + 2, where $E$ and $N$ correspond to the number of edges and nodes in the CFG. The CC values are computed at either the whole code level (total CC; $\text{CC}_{\text{total}}$) or the function level (meaning the average CC across all functions in the code; $\text{CC}_{\text{avg}}$).

A high CC value generally indicates a complex code structure. It functions as a guideline for code decomposition, suggesting that a function whose CC is exceeding a certain threshold value $\tau$, e.g., 5 McCabe (1976) or 10 McConnell (2004), might benefit from being broken down into smaller sub-functions. Based on the concept, we assume that the average CC of an ideal modular code example, denoted by $\text{CC}_{\text{avg}^{*}}$, should be equal to the threshold $\tau$.⁴⁴4Given two choices for $\tau$, i.e., 5 or 10, we set $\tau$ to 5 to encourage a sparser distribution of modularity scores (MoS). In other words, ideally, every function within a modular code snippet is expected to have a CC value of $\tau$. Following the intuition, we define $m^{*}$, the number of ideal modules, as follows:

Finally, we define the modularity score, dubbed MoS, as follows:

where $n$ is equal to the actual number of modules in the target code. That is, the closer $n$ (actual number of modules) is to $m^{*}$ (ideal number of modules), the higher the modularity is considered to be.⁵⁵5In extreme cases where $m=0$ (no modularization required), the modularity score is set to 0 if no actual modules are used ($n=0$) and 1 otherwise ($n>0$). The process of deriving MoS is illustrated in Figure 2.

With a way to quantify code modularity, we can now classify a code dataset into two categories—modular and non-modular (= singular). We further leverage prior research by including LLM-based code transformations and their corresponding manually recovered counterparts for controlled experiments. This allows us to create four distinct clusters of code separated by their modularity levels.⁶⁶6Figure 3 in Appendix displays examples of code from each category for a code generation problem.

Modular Code (MC) is a collection of code snippets with the highest MoS among solutions for each problem in a dataset.

Singular Code (SC) represents another set of solution code examples for the same problems corresponding to MC, with MoS being 0.

Transformed Modular Code (TMC) can be obtained by utilizing GPT-3.5-Turbo ($f$) to transform SC into code with high MoS. The conversion process can be represented by the following:

where $I$ represents a transformation instruction and $Q$ is the problem description of SC.⁷⁷7See Figure 4 for prompt details on the conversion process.

Transformed Singular Code (TSC) is a variation from TMC, whose modularity is manually removed by human programmers. Specifically, TSC is created by replacing module invocation parts in TMC with the body code of corresponding modules and removing the modules from the program. By minimizing the influence of factors other than modularity through the comparison of TSC and TMC, it contributes to a rigorous evaluation of the impact of modularity.

We explore the impact of modularity by comparing how the four code collections, categorized by their modularity levels, affect performance. To mimic real-world usage, we focus on the case of utilizing code LLMs with few-shot in-context learning. We leverage two-shot demonstrations (providing two code examples) unless otherwise specified.⁸⁸8Refer to Figure 5 and Figure 6 for prompt details.

Table 1 and Table 2 present results on APPS and CodeContests, categorized by the modularity of the code demonstrations. All results are the average of five independent runs with different random seeds.

In Table 1, we observe, as previously reported, that the performance of TMC is slightly better than TSC.¹¹¹¹11For APPS, we conducted experiments only with TMC and TSC due to computational constraints. However, their marginal performance gaps raise questions about the impact of modularity.

In Table 2, the relationship between modularity and performance becomes less clear. When comparing MC to SC, we observe that MC consistently underperforms SC, which contradicts previous findings. Furthermore, the comparison between TMC and TSC—a more controlled setting for evaluating modularity—shows no clear correlation between code modularity and performance. This is despite the fact that the transformation process by GPT-3.5-Turbo (SC $\rightarrow$ TMC) seems to contribute to non-trivial increases in performance. Therefore, we argue that the previously reported effectiveness of modularity on performance was likely due to unforeseen consequences of the transformation process, rather than the modularity itself.

In this work, we propose a metric, called MoS, for quantifying the modularity of code snippets and evaluate its impact on performance. Our evaluation reveals no significant correlation, or even a possible weak negative correlation, between modularity and performance. This suggests that factors influencing the usefulness of code examples may differ between human and LLM perspectives. Exploring the influence of other code properties beyond modularity is a promising direction for future work.

Due to limited computational resources, we focused on designing experimental settings that are both targeted and generalizable. This limitation restricted the scope of our investigation, but considering more extensive configurations in future work—such as fine-tuning, employing much larger models, and evaluating other programming languages—will help validate and potentially broaden the applicability of our findings. Despite these limitations, we believe our findings offer valuable insights, thanks to our comprehensive exploration of the feasible configurations within the available resources. Additionally, identifying a core factor besides modularity that directly affects performance holds significant promise for improving code generation.