HDLCopilot: Hardware Design Library Querying with Natural Language

LLMs have been extensively applied to various tasks in the hardware design process from verilog code generation, EDA scripting to RTL bug fixing. Takhur et al. (Thakur et al., 2023) introduced the first finetuned LLM for verilog code generation. Wu et al. (Wu et al., 2024) introduced ChatEDA, demonstrating how LLMs can be used for EDA tool scripting and automation. Tsai et al. (Tsai et al., 2023) explored the use of LLMs for fixing RTL code bugs. Several studies have also explored LLMs for question-answering tasks in hardware design. For example, Liu et al. (Liu et al., 2023a) proposed ChipNeMo, an LLM fine-tuned for hardware design tasks including question answering.

Retrieval augmented generation emerged as a powerful approach to enhance the performance and reliability of LLMs by connecting them to external knowledge sources. Lewis et al.  (Lewis et al., 2020) introduced the RAG model, which combines a pre-trained neural retriever with a sequence-to-sequence model for open-domain question answering. Guu et al. (Guu et al., 2020) proposed REALM (Retrieval-Augmented Language Model Pre-Training), which integrates retrieval during model pre-training, showing improvements in open domain question-answering tasks. Shuster et al. (Shuster et al., 2021) have shown that using retrieval can reduce LLM hallucination in knowledge-grounded dialogue tasks.

The transformation of natural language queries into SQL using LLMs has been a focus of recent research. Pourezza et al. (Pourreza and Rafiei, 2024) proposed decomposing the user question into sub-questions and feeding the solution of those sub-questions to the LLM to generate the final SQL query. Cyren et al. (Wang et al., 2023) proposed a multi-agent collaborative framework for converting the user question to SQL by first semantically choosing relevant tables, then performing query decomposition into multiple sub-queries.

While these works demonstrate the broad applicability of LLMs in hardware design and the potential of text-to-SQL in other domains, the application of text-to-SQL techniques for PDK management remains an underexplored area. HDLCopilot addresses this gap by combining the strengths of LLMs, text-to-SQL conversion, and structured PDK data to provide an efficient and accurate natural language interface for PDK queries.

To facilitate integration with LLM-based retrieval, we first convert the PDK files into a relational database. LLM agents can then retrieve information from the database by dynamically generating SQL queries based on the user input. We mainly focus on three views in the PDK files: liberty for storing timing information at different operating conditions, LEF for abstract layout information for each cell, and Technology LEF for storing metal stack properties. We propose a schema for each view. These schema are designed to support storing information across different standard cell libraries different operating conditions within the PDK.

The liberty schema is shown in Fig. 3. This schema comprises tables capturing operating conditions, cell attributes, pin properties, and timing data. The LEF schema is shown in Fig. 4. This schema comprises tables that capture abstract physical information about various macros (cells), including cell dimensions, obstruction layers in cell layouts, and pin physical attributes such as antenna gate area and pin shapes. Fig. 5 depicts the schema for technology LEF files, which stores technology-specific information. This includes details about routing layers (such as preferred routing direction, width, and spacing rules, resistance), via layers, and their associated antenna ratios.

HDLCopilot comprises four LLM agents: : the Dispatcher, Selector, SQL-Generator, and Interpreter, which collaborate together to ensure a reliable and accurate SQL generation pipeline.

Dispatcher: The main objective of the dispatcher agent is to route the user question to the appropriate standard cell library, library view, and operating conditions. Given the user question $\mathcal{Q}$, the available libraries $\mathcal{L}$ and library views $\mathcal{V}$ in the PDK, the dispatcher selects the relevant library $\mathcal{L^{\prime}}$, library view $\mathcal{V^{\prime}}$, and operating conditions $\mathcal{C^{\prime}}$ (if applicable). The function of the dispatcher agent is described in Eq. 1, where $f_{dispatcher}(.|\mathcal{M})$ represents LLM agent $\mathcal{M}$.

The selected library view $\mathcal{V^{\prime}}$ is then used to filter the PDK tables, retaining only those relevant to the routed view as described in Eq. 2. The selected tables $\mathcal{T^{\prime}}$ are then passed to to the selector agent to perform further fine-grained table selection by choosing only the tables relevant to the user question.

Selector: The selector agent performs a more refined reduction of the tables. Given the user question $\mathcal{Q}$ and the schema description of the routed tables $\mathcal{T^{\prime}}$, the selector narrows down the the set of the routed tables to only those most relevant to the user question. The function of the selector agent is described in Eq. 3. The main purpose of this table reduction process is to make the text-to-SQL task easier by having the SQL-Generator only examine the relevant tables. This focused approach enhances the efficiency and accuracy of the SQL generation process.

SQL-Generator: The SQL-generator serves as the core agent of the framework. Its primary function is to construct a SQL query that accurately retrieves the required information from the PDK database to address the user’s question. The generator employs a query decomposition approach proposed in (Pourreza and Rafiei, 2024; Wang et al., 2023) that breaks down the user question into smaller, manageable sub-questions. For each sub-question, the generator produces a corresponding sub-query. These sub-queries are then combined to form the final SQL query. This step-wise approach enhances accuracy and allows for handling complex user questions. The generator function is describe in Eq. 4. It takes as input the user question $\mathcal{Q}$, the schema description of the selected tables $\mathcal{T^{\prime\prime}}$, the relevant standard cell library $\mathcal{L^{\prime}}$, and operating conditions $\mathcal{C^{\prime}}$.

Interpreter: The interpreter main role is to translate the raw database results into a coherent, natural language response that directly addresses the user’s question. This agent processes the user question $\mathcal{Q}$ and the result $\mathcal{R}$ obtained from executing the generated SQL query (Eq. 5) and then then formulates an output answer $\mathcal{O}$ in natural language format, as formalized in Eq. 6.

To evaluate HDLCopilot, we created an evaluation set of $52$ user questions with corresponding SQL-queries. This set encompasses a diverse range of complexities, from simple single-table selections to complex multi-table joins with sub-queries and multiple conditions. As shown in Table 1, the set incorporates various SQL clauses, aggregation functions, and sub-queries, providing a comprehensive test of the framework’s SQL handling capabilities.

Following text-to-SQL work  (Wang et al., 2023; Li et al., 2024; Zhong et al., 2020), we use the Execution Accuracy (EX) and Valid Efficiency Score (VES) to evaluate the performance of our proposed framework.

The Execution Accuracy (EX) quantifies the framework’s ability to generate SQL queries that produce correct results. It measures the proportion of questions in the evaluation set where the execution results match those of the ground truth queries. It is formally defined in Eq. 7, where $N$ defines the number of questions in the evaluation set, $V_{i}$ defines the set returned by the ground truth SQL query and $\hat{V_{i}}$ defines the set returned by the predicted SQL query. $1\!\!1(.)$ is an indicator function that is equal to $1$ if both the ground truth set and the predicted set are equal and $0$ otherwise.

The Valid Efficiency Score (VES) evaluates the correctly generated SQLs by comparing their execution time against those of the ground truth SQLs. VES is formally defined in Eq. 8, where $R(.)$ is the relative efficiency of the predicted SQL and the ground truth SQL, and $E(.)$ is the execution time of each SQL in the database. The VES metric provides insights into both the correctness and the computational efficiency of the generated SQL queries.

First, we present three qualitative examples that showcase HDLCopilot’s capability in generating complex SQL queries, retrieving relevant information, and providing precise answers to user questions. Fig. 6 shows a user question that asks for comparing the width of the 4-input MUX across all libraries. The framework is able to answer the question with high precision. This analysis is potentially useful in providing designers with immediate insights into a specific cell’s footprint variation among different library options, helping them determine which library is most suitable for their design requirement. Fig. 7 also presents a cross-library comparison of leakage power in flip-flop cells. This analysis is useful for designers focusing on low-power applications, allowing them to quickly identify the most power-efficient cell for their specific needs. Fig. 8 showcases the framework’s ability to handle more sophisticated queries. In this example, the framework generates and executes a complex SQL query to compare the propagation delay of a 2-input MUX cell between two specific libraries. These three example highlight the system’s ability to perform diverse set of analyses that would be time-consuming if done manually.

We also present quantitative assessment of the system’s performance on the $52$ examples in our evaluation set. We first evaluate the accuracy of the dispatcher and the selector independently to determine the most reliable setup for these agents. We conduct evaluations both with and without few-shot examples. Table. 2 shows that few-shot demonstrations generally improves the accuracy for both GPT3.5 and GPT-4 models. GPT-4-o achieves the highest overall accuracy even without few-shot examples, showcasing its ability to comprehend and execute the task without additional context. The best setup is achieved by using GPT-4 and GPT-4o models, with an overall dispatch accuracy of 99.35% across all three routing tasks and table selection accuracy of 98.07%.

Table. 3 shows the Execution Accuracy (EX) and valid Efficiency Score (VES) of the entire framework. GPT-4o demonstrates superior performance, achieving the highest overall EX of 94.23% and VES of 98.58%. The results demonstrate that liberty queries are particularly harder than the LEF and TechLef queries. This is mainly because liberty files contains more attributes and diverse data types. Nonetheless, GPT4-o demonstrates a high execution accuracy of 91.30% for the liberty set, and even generates more efficient SQL queries than the hand-crafted ground truth SQLs, as shown by its VES of 101.57%.