NoviCode: Generating Programs
from Natural Language Utterances by Novices

The current Text-to-Code paradigm focuses on generating code-lines from technical descriptions produced by trained programmers. In this work, we move from this Text-to-Code paradigm towards generating programs with intricate structures from intuitive, day-to-day language descriptions produced by non-technical individuals. This novel reconstruction of the task is challenging on two levels: (a) the generated programs have non-trivial control-flow structures — with API calls to a novel API that the model was not necessarily trained on — rather than generic one-liners; and (b) we are interested in instructions provided by laypeople in everyday natural language, without using technical concepts or jargon. We term this task Natural Language Programming (NLProg).

NLProg differs from standard code generation models, tasked to convert natural language instructions or descriptions into executable code, in several ways. In the Text-to-Code paradigm, models require the user to use technical terms, like flow structures (e.g., conditions or loops), data types, variables, and programming concepts (e.g., sorting algorithms, object-oriented concepts, recursions, etc.). In contrast, in NLProg we aim to convert plain, everyday language descriptions, devoid of any technical jargon, into functional code using independent and unseen API specifications.

NLProg is particularly challenging when the descriptions implicitly allude to code constructs as control flow elements such as sequences, loops, or conditional statements, without explicitly mentioning them. For example, given a standard API of email and calendar applications and a user request “Check that I received confirmation emails from all advisors in the committee or cancel my meeting with them”, we expect the model to interpret the conjunction “or” as indicating a condition, and to recognize “all advisors in the committee” as an iteration over a particular set. The model should then generate a proper executable program based on the API (Figure 1).

Recent work on natural language synthesis to code (Chen et al., 2021, Wang et al., 2023a, Nijkamp et al., 2023) have shown remarkable success in code generation tasks using LLMs. Trained on a large fraction of GitHub repositories (Iyer et al., 2018, Husain et al., 2019, Lu et al., 2021, Wang et al., 2021, Wang et al., 2023a) or StackOverflow question-answer pairs (Yin et al., 2018), these code-oriented LLMs learn rich contextual representations that can be transferred to various code-related downstream tasks.

However, these works share several limitations. They predominantly focus on technically precise descriptions given by programmers, using jargon-laden, code-centric discourse geared towards seasoned programmers. Other datasets in intent recognition and slot fillings (Gupta et al., 2018, Chen et al., 2020) focus on plain language description from non-programmers, or novices, but these are translated into simple code that exhibits no complexity in its flow execution and is inherently derived from the NL descriptions in its datasets.

To bridge this gap, we present NoviCode, a novel task that aims to translate NL descriptions from novices into complex executable programs. We curated a dataset where novice-centric NL instructions are paired with complex code programs. These code programs adhere to API specifications to facilitate the functional execution of the produced code. Having defined this task and created the benchmark, two additional challenges present themselves, namely: (i) How can we evaluate the efficacy of models on such complex code synthesis tasks? and (ii) What would be good models for addressing the non-trivial challenge of translating high-level language to low-level intricate code?

To address the evaluation challenge, we deliver, along with the task data, test suites that simulate diverse scenarios of execution flows to gauge the functional correctness of the generated code. Our evaluation methodology is designed to assess the functional correctness of the control flow elements within the generated complex programs, rather than merely looking at the form of the code or assessing an end state only.

To address the modeling challenge, we hypothesize that learning the alignment of the natural language queries with codes’ hierarchical structures, rather than learning a simple end-to-end text-to-code model, can improve performance, particularly with programs as complex as we target here.

Our experiments with standard LLMs assessed by our functional evaluation suites demonstrate that the task is indeed challenging. Existing models struggle to produce the expected executable code with complex constructs. We further empirically confirm our hypothesis that learning the alignment of natural language spans to compositional hierarchical code structures is better than the de-facto standard end-to-end modeling.

The contribution of this paper is thus multifold. First, we define a novel code-synthesis task based on plain non-technical natural language. Second, we deliver a dataset of NL instructions obtained from non-programmers, paired with an expert-crafted evaluation suite that focuses on the function, rather than form of the generated code. Finally, we provide a novel modeling approach that aligns NL spans with the explicit compositional structures of the code. We further show that this approach outperforms the standard end-to-end baseline models on this task, even with the most capable contemporary generative LLMs.

Our benchmark creation begins with collecting natural language utterances from novices, in domains that are understandable to them (Section 4.1). Next, we assemble an evaluation dataset that combines these utterances with corresponding test suites to assess the quality of the generated code (Section 4.2). In addition, we synthetically generate NL utterance-code pairs to facilitate the training of models for the task (Section 4.3).

To provide code frameworks that bridge the translation of NL descriptions to their respective code programs, we created an API that generically aligns spans in the user description to code data types and actions. The generated code must correctly utilize the API endpoints and be executable, allowing us to assess its functionality by executing corresponding tests. The API specifications followed the nine domain apps defined for sourcing the prompts (see Section 4.1). Nonetheless, the formulation of the API was designed independently from the process of collecting natural language user utterances in these domains.

The API code comprises multiple classes and is designed to provide a comprehensive and modular framework for interacting with the system’s functionalities. 49 interfaces define the different data type entities in our APIs. Location, Contact, DateTime, are examples of such data types. To perform actions with these entities, we expose an additional 11 classes with 34 action methods that are available to be executed. For instance, the Messages class defines methods to send a message and find messages received from specific senders, with desired content, or at a specific time.

Users provide natural language descriptions detailing the actions they want to execute and the specifics of each action. The model must interpret descriptions and associate them with the appropriate API classes and methods.

To map from user-specified text spans to domain objects, we assumed the API provides the methods resolve_from_text and resolve_many_from_text, which discern relevant spans of word sequences in an utterance and correlate them with specific entities. For instance, when a user queries for "the weather on independence day" the method DateTime.resolve_from_text is invoked with the temporal description spans from the request ("on independence day") and returns a DateTime object which is stored in the code. These variables, representing entities, are then used in other API methods to execute the actions requested in the natural language description (e.g., get_weather_forecast). It is the API implementor’s responsibility to provide suitable implementations for these text-resolution functions. In our evaluation suite, we provide suitable mock implementation (see Section 6.3).

In certain cases, an entity needs to be inferred from another entity. For such instances, the API offers extra methods such as resolve_from_entity or resolve_many_from_entity. For instance, the phrase "After every Astros game" refers to a set of events on a public calendar. To infer the DateTime entity from each event we iterate on the events and call resolve_from_entity on each event (see Figure 3).

Instead of translating natural language utterances directly to code, we propose to map the natural language to an intermediary structure that better encapsulates the control flow elements of the language. By explicitly representing the structure of the target complex program through a hierarchical structure, one that expresses the compositional control flows expressed in the code, we hope to improve the compositional generalization thus improving code generation accuracy relative to standard end-to-end text-to-code methods (Herzig and Berant, 2020).

Formally, given a natural language utterance $x$, our goal is to convert $x$ into a surface code $y$. Instead of mapping directly to $y$, we learn to translate $x$ to an intermediary logical form $z$, which can be deterministically converted to $y$. We use the text-to-code models to perform the transformation $x\rightarrow z$, and then use a deterministic transformation $z\rightarrow y$ to obtain $y$. Specifically, $z$ is based on a linearized compact AST (cAST) which is derived from the abstract syntax tree (AST) of the code, and is linearized into text using a bracketted notation (Figure 4). The AST is a tree structure representing the structure of the program. Using the AST allows the model to focus more on the logic and flow of the complex program rather than mere syntax (e.g., keywords, symbols, indentation), reducing the complexity of the output space. On top of that, we define a compact AST (cAST) that only retains the control flow logic structures, such as sequences, conditions, and loops, in a tree form. To create compact ASTs, we revert basic operations such as variable assignments or function calls to their fundamental code syntax. These elements are transformed from their AST hierarchical representation to appear as leaves in the tree’s code form.

To translate the linearized cAST $z$ back to code $y$, we parse the string into its cAST tree form and expand it back to the AST form using a tool we provide. The ASTs are translated into actual code using the official Python package for this (ast).

We note that previous works (Yin and Neubig, 2017, Yin and Neubig, 2018) also utilized ASTs to synthesize code better. Yet, these works model code generation as a series of classification problems of grammar rules required to reconstruct the AST. Our method uses the hierarchical tree form as a vehicle to directly map texts to hierarchical code structures.

We set out to evaluate how existing models cope with the NoviCode task. Moreover, we aim to assess whether our representation-based learning approach (Section 8) outperforms a simple end-to-end Text-to-Code baseline.

To do so, we evaluated decoder-only generative large language models (LLMs) using in-context learning scenarios. We also tested pre-trained encoder-decoder LLMs that are fine-tuned with the synthesized trainset. We tested both a simple end-to-end scenario (denoted base) and one where we map to the hierarchical compact AST (denoted cAST), as defined above.

Previous text-to-code datasets were delivered to facilitate the training and testing of the code generation capacity of contemporary models. Notable datasets include the CoNaLa Dataset (Yin et al., 2018), which contains programming questions from Stack Overflow along with code solutions, the CONCODE dataset (Iyer et al., 2018), the CodeSearchNet Corpus (Husain et al., 2019), and the CodeXGLUE dataset (Lu et al., 2021), constructed using code-comment pairs from GitHub across numerous domains in various programming languages. The majority of these include very technical jargon and relatively short statements, in contrast with our novice users language and lengthier intent expressions (Section 2.1).

Identifying instructions in Task-Oriented datasets can also be seen as equivalent to interpreting description as executions. Notable Task-oriented datasets are TOP (Gupta et al., 2018) and TOPv2 (Chen et al., 2020) datasets. However, these datasets fail to provide high-level abstractions that elicit complex code with sequences and control flow structures (e.g., loops and conditions).

On the front of evaluating text-to-code generation, automatic metrics for code generation evaluation initially adopted techniques similar to those used in machine translation. BLEU (Papineni et al., 2002) has been extensively used for evaluating code generation. However, its limitations have been increasingly recognized, including the disregard for semantic correctness and functionality. CodeBLEU (Ren et al., 2020) enhances BLEU by incorporating crucial code-related features such as syntactic and semantic similarity, data flow, and variable misuse. Despite its improvements, CodeBLEU still relies on a reference implementation, which may hinder its efficacy in cases where multiple correct solutions exist.

The HumanEval dataset (Chen et al., 2021) presented a recent and significant effort in this direction, where functional correctness tests evaluate the generated code. The evaluation tests in HumanEval present basic programming problems. Austin et al., 2021 presents a similar approach for evaluating correctness. Additional code generation benchmarks extended this approach using more challenging programming problems, which require an understanding of algorithms (Li et al., 2022, Hendrycks et al., 2021), or usage of external and advanced Python packages (Lai et al., 2022, Wang et al., 2023b).

Our benchmark presents code-generation tasks that concentrate on everyday tasks for laypeople, such as scheduling meetings in a calendar or checking weather forecasts. Furthermore, to support and extend the complexity, our benchmark introduces a private and unseen API, in contrast to using standard Python packages.

Other benchmarks challenged models by providing class-level Python code-generation tasks (Du et al., 2023). The challenge introduced in this paper is orthogonal to this challenge and does not generate class-level code.

Functional tests on the generated code are prone to false positives or true negatives upon low coverage of test edge cases. Some works (Liu et al., 2023) mitigate this using an automatic test input generation engine. In contrast with this approach, our evaluation method validates the data model affected by the generated code and not the generated code itself.

Few code generation benchmarks have also concentrated on the population composing the NL prompts in the benchmarks for code LLMs, specifically targeting non-expert beginner programmers (Babe et al., 2023). In contrast, the prompts in our benchmark were crafted by novice, non-programmers lacking any programming proficiency.

Last, other corpora capture how non-programmers express if-then clauses (Quirk et al., 2015). Yet, the tasks in this benchmark are expressed in a highly structured and noisy language and exhibit only a single control flow paradigm – conditionals.

In this paper we introduce NoviCode, a novel NL programming task of generating executable complex programs with control flow structures from novice NL user requests and API specifications that the program should comply with. As an integral part of our task, we propose an evaluation framework to assess model efficacy on this task, based on functional execution and denotation rather than on the code form. Building upon this task, we propose a novel representation that explicitly reflects the hierarchical structure of code, which outperforms all baseline models, open source (e.g., CodeT5+) or closed source (e.g., GPT-4-Turbo) models, in the evaluation tests. Finally, our analysis suggests that while generating working code based on language is a feasible task, ours is still a challenging one. Yet NoviCode constitutes a promising approach towards true natural language programming — where humans program in their native tongues —encouraging future research and development of this domain.

We gratefully acknowledge the contribution of Tamar Gur for her invaluable assistance in this work. Additionally, we extend our gratitude to Royi Rassin, Royi Lachmy, Avshalom Manevich, and Shira Kritchman for their helpful comments and discussions. We also thank the anonymous reviewers and the action editor for their valuable suggestions.

This project received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program, grant agreement No. 677352 (NLPRO) and grant agreement No. 802774 (iEXTRACT).