ITERTL: An Iterative Framework for Fine-tuning LLMs for RTL Code Generation

Previous works(blocklove2023chip; chang2023chipgpt) have attempted to directly prompt LLMs to generate RTL code, achieving notable results. Chip-chat(blocklove2023chip) design a 8-bit accumulator-based microprocessor with the assistance of commercial LLMs. In practical applications, it is challenging to directly generate usable code through human instructions. As a result, recent works(thakur2023autochip; delorenzo2024make) explore optimizing the RTL codes generated by LLMs with feedback from Verilog tools. AutoChip(thakur2023autochip) utilize error reports from compilers and simulators to help LLMs to rectify faulty code. (delorenzo2024make) develops a Monte Carlo tree-search (MCTS) algorithm to enhance LLMs to generate correct and PPA-efficient code with the feedback from compilers and synthesis tools.

Instead of using off-the-shelf LLMs directly, some researchers(liu2023verilogeval) opt to train LLMs to make it specialized in hardware design. VeriGen(thakur2023benchmarking) leverage corpora from GitHub and Verilog textbooks to fine-tune open source LLMs, defeating the state-of-the-art commercial Codex(chen2021evaluating) LLM on 17 Verilog problems. ChipNeMo(liu2023chipnemo) customize LLaMa2(touvron2023llama) for applications in chip design such as chatbot, generating EDA tool script, and bug summarization. RTLCoder(liu2023rtlcoder) develops a automated flow to generate instruction-code pairs for supervised fine-tuning (SFT) and propose a new SFT method leveraging code quality assessment.

In order to better measure the effect of LLMs on Verilog code generation, researchers have proposed corresponding evaluation benchmarks. VerilogEval(liu2023verilogeval) has released an open-source evaluation dataset , including 156 questions along with their corresponding golden solutions. RTLLM(lu2024rtllm) present an open-source benchmark evaluating the quality of the generated code from three progressive perspectives: syntax, functionality, and design quality.

To tailor LLMs for RTL code generation, researchers often need to fine-tune LLMs with domain-specific data. VeriGen(thakur2023benchmarking) fine-tune LLMs by predicting next token on corpora from open-source code and textbooks, which can be regarded as an continual pre-training. To facilitate the model’s ability to follow instructions, VerilogEval(liu2023verilogeval) applies SFT on LLMs using synthetic instruction-code pairs. Furthermore, RTLCoder(liu2023rtlcoder) introduces a new fine-tuning algorithm which harnesses code quality evaluation upon candidates sampled from pretrained LLMs. However, during the training process, the aforementioned methods are constrained by static and unchanging training samples. According to previous research(yuan2023rrhf; liu2022brio) in natural language, updating train samples using in-training LLMs can significantly bootstrap the model performance. However, related research in the task of Verilog code generation remains scarce. Our work can be seen as a pioneering attempt.

As an iterative training scheme, our approach boost the generation capability by alternately sampling and training. During a single loop, we basically follows RTLCoder(liu2023rtlcoder) for ease of implementation. The key distinction lies in our core idea of alternating iterations, which can significantly enhances the performance.

We briefly introduce the procedure in a single loop at first, and then explain the iteration process. As shown in Figure  1, in the $t$-th round of training, for each input instruction $s$, there are $K$ corresponding output responses $a_{k}^{t}$, $1\leq k\leq K$. Among these, the first $K-1$ responses are sampled from the model $\pi_{t}$ acquired from previous training iteration. While the last response $a_{K}^{t}$, which serve as the reference data, can be obtained from another teacher LLM or human, denoted as $\pi_{teacher}$. Assume the distribution of $\pi_{teacher}$ remains unchanged, there is little impact from repeated sampling. Therefore, we omit the superscript $t$ from $a_{K}^{t}$. Each response $a_{k}^{t}$ is assigned a quality score $z_{k}^{t}$ utilizing Pyverilog and Rough-L following (liu2023rtlcoder).

The loss function comprises two components: the ranking loss(liu2023rtlcoder; yuan2023rrhf; liu2022brio) and the Maximum Likelihood Estimation (MLE) loss.

Conditional log probability (length-normalized) is calculated leveraging the model $\pi$ that is being trained:

Combining the quality score $z$ and log probability $p$, the ranking loss(liu2023rtlcoder; yuan2023rrhf; liu2022brio) can be computed:

Unlike RTLCoder(liu2023rtlcoder), we discard the use of softmax normalization to $p$ based on subsequent experimental results.

Another loss component is the common cross entropy loss relative to the reference sample $a_{K}$:

The overall loss function is a weighted sum of both:

To control the scale relationship between the two loss functions, we set $\lambda$ equal to $sg(L_{ce})$, where $sg(\cdot)$ represents a stop-gradient operation.

After the initial round of training, we obtained the converged model $\pi_{t+1}$. Even though the capability of model $\pi_{t+1}$ has improved comparing to $\pi_{t}$, there remains room for further enhancement. So we propose to resample new responses $a_{k}^{t+1}$ using the new model $\pi_{t+1}$ and assess new quality scores $z_{k}^{t+1}$. Resampling can bring new feedback as the model’s distribution has shifted after previous round of training. We compute the loss function $L^{t+1}$ and update parameters with new data. This process is repeated until the training concludes.

Inspired by the previous work(li2023policy), we regard fine-tuning LLMs as a reward maximization problem, which can be formulated as follows:

where $\mathbb{E}$ denotes the expected value, $s$ represents the instruction (prompt), following the distribution $\rho(\cdot)$. $a$ represents the response (code) generated by the LLM $\pi$ which is being optimized. $r$ represents the reward for the response, in other words, the quality of the generated code.

Ideally, the true reward for the generated Verilog code should incorporate the evaluation of its functional correctness, as well as PPA metrics. However, it’s intractable to implement because verifying the functional correctness requires a comprehensive corresponding testbench, which is nearly impossible for researchers to develop given the large amount of samples for fine-tuning. Additionally, assessing PPA relies on logic synthesis, which also consumes a significant amount of time. Therefore, it is necessary to approximate the reward function to make it more manageable.

Let’s first consider a naive approach that directly fine-tunes LLMs using reference data drawn from $\pi_{teacher}$, which can be considered a form of knowledge distillation. The optimization objective can be derived as follows:

The first equality holds because $P_{teacher}$ is independent of $\pi$. While the approximate equality in the third line is based on the law of large numbers. Comparing equations  5 and  6, it is clear that this approach leads to systematic errors in two aspects¹¹1If $s$ is synthesized by another model rather than obtained from real samples, the estimation is also biased. we omit this aspect in this work since it’s not the focus of our research.: (1) replacing the distribution of the model being trained $\pi(\cdot\mid s)$ with that of $\pi_{teacher}(\cdot\mid s)$, and (2) the use of the negative Kullback-Leibler divergence (KL divergence) as the surrogate reward function between the probability distribution of the reference data and $\pi$.

Next, Let’s consider a non-iterative degenerate version of our approach, where a constant distribution $\pi_{c}$ is used to generate training samples. This version is quite similar with RTLCoder(liu2023rtlcoder). We can also transform the optimization objective into a reward maximization problem. To simplify the derivation, here we only present the ranking loss component from Equation  2. And the term associated with the reference data in the ranking loss are also omitted, which does not interfere with our conclusions.

Here, we denote the surrogate reward function as $r_{\pi}$ to show its dependency on $\pi$. Comparing equations  6 and  7, we find that this approach offers two improvements over the previous naive approach. (1) $\pi_{c}$ can be adjusted to an appropriate distribution to reduce the mismatch with $\pi$, for instance, selecting the initial distribution $\pi_{0}$. (2) The surrogate reward function incorporates the evaluation of code quality, which is more reasonable than a simple KL divergence.

Based on the analysis of the above two cases, we can formally reveal the superiority of our method. Using similar notations, our optimization scheme can be formulated as:

After $t$-th iteration, $\pi$ is updated, progressively deviating from the previous distribution $\pi_{old}$ ($\pi_{0}$,…$\pi_{t-1}$). By replacing $\pi_{old}$ with $\pi_{t}$ for sampling, our method effectively mitigates the distribution mismatch. Furthermore, due to the law of large numbers, increasing the number of sampling promotes better estimation expected values.

we can also analyze the potential bottleneck of our approach from Equation  8. Since the surrogate reward function $r_{\pi}$ evaluates code quality merely from a syntax perspective, ignoring functional correctness and PPA, there is a mismatch between $r_{\pi}$ and the true reward function $r$. As a result, although $r_{\pi}$ may increase with each iteration, the potential improvement for model is still constrained, which is verified by our experiments in Section  4. We anticipate that a better-designed surrogate reward function could further enhance the effectiveness of our approach in the future.

From another perspective, our approach can also be regarded as a degenerate variant of the generalized Expectation-Maximization (EM) algorithm. Consider the following optimization problem:

Since $\pi$ appears in both the subscript of the expected symbol and the reward function, an EM-style optimization procedure would be like:

Equation  10 corresponds to our optimization approach, while optimizing Equation  11 typically requires reinforcement learning methods like policy gradients, which may increase the complexity and instability of the whole scheme. So we simply set $\pi_{t+2}$ equal to $\pi_{t+1}$ to avoid these issues. We leave the exploration of more precise optimization methods to future researches.

Benchmark: We choose a comprehensive evaluation dataset, named VerilogEval(liu2023verilogeval), as the benchmark to measure the performance. This dataset consists of diverse Verilog tasks ranging from simple to complex, such as combinational circuits, finite state machines, code debugging, constructing testbenches, and so on. Two sets of design instructions are provided: the first one is generated by LLM, named VerilogEval-machine, containing 143 samples; the other one is manually written, named VerilogEval-human, comprising 156 samples. Functional correctness evaluation is conducted via ICARUS Verilog simulator by comparing the outputs of the generated design with that of the golden solution.

Metric: As with many code generation researches, pass@$k$ metric(kulal2019spoc) is employed to measure the performance, by regard a problem as solved if at least one of the $k$ generated code samples passes the unit tests. Specifically, for each problem, the model generated $n\geq k$ candidates, where $c\leq n$ samples pass the unit tests. The pass@$k$ metric is estimated unbiasedly use the following expression(chen2021evaluating):

To avoid the impact of randomness, we mainly focus on the pass@1 value under greedy decoding. For a comprehensive evaluation, we also measure the pass@$k=\{1,5,10\}$ metrics setting $n=10$ under Top-p decoding.

Decoding Strategy: During the training stage, in order to enhance the diversity and promote the exploration, we use Top-p decoding strategy with $top_{p}=0.95$ and $temperature=0.5$. In the testing stage, to mitigate random errors, the model is prompted to generate responses with $temperature=\{0(greedy\,decoding),0.2,0.5,0.8\}$ and $top_{p}=0.95$. For each test metric (pass@1, pass@5, and pass@10), the best result is chosen.

Training Details: DeepSeek-Coder-Instruct-6.7B(guo2024deepseek) is chosen as the pre-trained model. We use the open-source instruction-code pairs from (liu2023rtlcoder) as reference data. This dataset contains nearly 27k samples. We randomly sample only 10,000 entries to train the final version of our model. The learning rate is set to $10^{-5}$. AdamW optimizer is employed with $\beta_{1}=0.9$ and $\beta_{2}=0.999$. The bp16 Mixed-precision Training is adopted to avoid overflow. During each iteration, the model is trained for 3 epochs. The total number of iterations is set to 7. We conduct all experiments on 4 NVIDIA A800 GPUs. As for parameters mentioned in Section  3.1, we set the number of candidates $K=4$. The hyper-parameters $\alpha$ and $\beta$ in Equation  2 are set to 0.3 and 0.2, respectively.

As Table  1 shows, our model reach the state-of-the-art level among open-source models. On Pass@1 metric, our model trained with only 10k reference samples surpasses RTLCoder with 27k reference samples by 1% and 1.3% on VerilogEval-machine and VerilogEval-human respectively. Leveraging an equal amount of reference samples, our model significantly outperforms RTLCoder-DeepSeek-10k on all metrics, with a relative improvement of 12.5% and 16.9% in pass@1 especially. Considering the size of the parameters, with only 6.7 billion parameters, our model significantly outperforms VerilogEval(liu2023verilogeval) with 16 billion parameters and a similar volume of reference samples (8502), further proving the efficiency and superiority of our method. Additionally, we incorporated a general-purpose code model, Codegen(nijkamp2022codegen), into our comparison. Its subpar performance on the VerilogEval benchmark underscores the importance of tailoring LLMs for RTL code.

Relative to closed-source models, our model comprehensively surpasses GPT-3.5 on the benchmark. Against GPT-4, our model performs better on VerilogEval-machine but lags on VerilogEval-human. Considering the vast differences in training costs and the lightweight advantage of our model, these performance deficits are acceptable.

To delve deeper into the impact of iteration, we plot how pass@1 on VerilogEval-human varies with the number of iterations in Figure  2. It can be clearly seen that, with 5 iterations and only 10k reference data, our model beats the baseline model with with 27k reference data. From the first to the fifth iteration, the pass@1 rate exhibits a clear upward trend, indicating the efficacy of iterative training. From the fifth to the seventh iteration, the pass@1 rate gradually decreases, which can be explained by the mismatch between the surrogate reward function and the true reward function mentioned in Section  3.2. In fact, similar discoveries were found on LLMs trained using reward models on natural languages. Our work reveals that the surrogate reward function based on code quality evaluation and reward model on natural languages have similar properties, providing inspiration for subsequent improvement work.

Figure  3 depicts different loss functions curves across each iteration. Even though the loss function converges within each round, the loss value can still decrease by leveraging newly sampled data points from updated model, validating the effectiveness of the proposed iterative training approach. Another observed phenomenon is that as the number of iterations increases, the marginal decrease in the loss function gradually diminishes, which indicates that the surrogate reward function is nearing its optimization ceiling.

The baseline method(liu2023rtlcoder) employ a softmax normalization for conditional log probability after Equation  1:

We conduct an ablation study to investigate its impact in iteration training. All other settings remain unchanged except for softmax normalization. From Figure  4, it can be observed that the pass@1 of the approach with softmax shows smoother variations across iterations, which may be caused by vanishing gradients brought by softmax saturation. We choose to discard softmax to maximize performance. While the softmax normalization can also be considered when a more stable and smooth training process is required.