HDLCopilot: Hardware Design Library Querying with Natural Language
Abstract.
Hardware design engineers routinely work with multiple Process Design Kits (PDKs) from various fabrication labs, each containing several standard cell libraries, optimized for specific metric such as speed, power, or density. These libraries include multiple views such as liberty files for timing information, LEF files for abstract layout details, and technology LEF for process design rules. Navigating this complex landscape to retrieve specific information about gates or design rules is often time-consuming and error-prone. To address this, we present HDLCopilot, an LLM-powered PDK query system that allows engineers to streamline interactions with PDKs in natural language format, making information retrieval accurate and more efficient. HDLCopilot achieves an accuracy of 94.23% on an evaluation set comprised of diverse and complex natural language queries. HDLCopilot positions itself as a powerful assistant in the hardware design process, enhancing productivity and reducing potential human errors.
1. Introduction
![Refer to caption](x1.png)
At the core of semiconductor design workflows lies a critical component: the Process Design Kit (PDK). PDKs serve as a comprehensive library of building blocks that are used for synthesizing abstract circuit definitions into manufacturable chips. PDKs usually contain different standard cell libraries, each optimized for a specific metric, such as speed, density, or power. These libraries include detailed files on cell timing information at different process corners, physical layout data, and metal stack properties.
Traditionally, hardware engineers navigate this complex landscape manually, parsing through extensive library files containing thousands of cells, each with numerous attributes, in order to locate specific information relevant to their current design task. This manual process is not only time consuming but also prone to human errors. Therefore, there is a need for more automated and efficient tools to assist engineers in managing and utilizing PDKs effectively, potentially accelerating the design process, and enhancing accuracy.
Large Language Models (LLMs) have enhanced productivity in various engineering domains, including hardware design. They’ve shown promise in tasks like Verilog code generation, RTL code bug fixing, and EDA tool scripting (Thakur et al., 2022; Pearce et al., 2020; Blocklove et al., 2023; Lu et al., 2024; Liu et al., 2023b, c, a; Tsai et al., 2023; Qiu et al., 2024; Wu et al., 2024). However, their application to PDK management remains underexplored. LLMs could potentially enhance engineer-PDK interactions through natural language interfaces, automated data retrieval, and intelligent suggestions, accelerating the design process and reducing errors.
Currently, Large Language Models (LLMs) are not inherently aware of Process Design Kits (PDK) specifics. Domain adaptive pre-training could be one way of enhancing LLMs knowledge with PDK-specific data (Gururangan et al., 2020). However, it would require expensive and time-consuming pre-training of the LLM on a large scale dataset of different PDKs from different manufacturers and process nodes. Moreover updating the PDK information would require retraining the LLM, making it impractical to maintain the LLM’s relevance to the most recent PDK version.
In contrast, Retrieval Augmented Generation (RAG) offers a more flexible and maintainable solution. RAG enhances LLM capabilities by grounding their responses in external knowledge sources, which can be easily updated without retraining the base model (Lewis et al., 2020). RAG has been applied to various data types: unstructured (e.g., text documents), semi-structured (e.g., JSON), and fully structured (e.g., relational databases) (Wang et al., 2023; Li et al., 2024; Zhong et al., 2020). For unstructured data, RAG uses semantic similarity matching, while for semi-structured data, it combines semantic and structural information. With fully structured data, RAG transforms the retrieval into a text-to-SQL task, generating SQL queries based on user questions and database schemas, then using the query results as context for the LLM’s response.
Since PDKs generally follow a structured format, they are well-suited for conversion into relational databases. By transforming PDK data into a SQL database, we can leverage the benefits of SQL-based retrieval while maintaining the flexibility to update and expand the knowledge base without retraining the underlying LLM. The SQL-based retrieval also allows for human verification of the generated SQL queries, ensuring reliability and transparency in the information retrieval process.
In light of this, we introduce HDLCopilot, an LLM-powered PDK query, designed for interacting with Process Design Kits (PDKs) using natural language. HDLCopilot harnesses the power of RAG and text-to-SQL conversion to provide an intuitive, efficient, and accurate interface for hardware designers to access and utilize PDK information. Fig. 1 presents an overview of the HDLCopilot framework, illustrating the flow of converting natural language queries to SQL queries to retrieve relevant data from the PDK database.
Our contributions are summarized as follows:
-
•
We introduce HDLCopilot, a multi-agent collaborative framework, designed to streamline interactions with Process Design Kits (PDKs) in natural language format, offering a novel approach for enhancing hardware design engineers efficiency.
-
•
We propose a database schema for storing PDK information in relational tables, which facilitates easy and seamless integration with LLM-enabled applications. This structured approach allows LLMs to perform precise data retrieval by dynamically generating SQL queries based on natural language inputs.
-
•
Experimental results show that HDLCopilot can answer diverse and complex user questions with high precision across different libraries in the PDK, achieving an accuracy of 94.23%. HDLCopilot also demonstrates high capability in generating efficient SQL queries with an efficiency score of 98.07%.
2. Related Work
In this section, we review relevant work in three key areas: LLMs for hardware design process in Section. 2.1, Retrieval augmented generation in Section. 2.2, LLMs for text-to-SQL applications in Section 2.3.
2.1. LLMs for Hardware Design
LLMs have been extensively applied to various tasks in the hardware design process from verilog code generation, EDA scripting to RTL bug fixing. Takhur et al. (Thakur et al., 2023) introduced the first finetuned LLM for verilog code generation. Wu et al. (Wu et al., 2024) introduced ChatEDA, demonstrating how LLMs can be used for EDA tool scripting and automation. Tsai et al. (Tsai et al., 2023) explored the use of LLMs for fixing RTL code bugs. Several studies have also explored LLMs for question-answering tasks in hardware design. For example, Liu et al. (Liu et al., 2023a) proposed ChipNeMo, an LLM fine-tuned for hardware design tasks including question answering.
2.2. Retrieval Augmented Generation (RAG)
Retrieval augmented generation emerged as a powerful approach to enhance the performance and reliability of LLMs by connecting them to external knowledge sources. Lewis et al. (Lewis et al., 2020) introduced the RAG model, which combines a pre-trained neural retriever with a sequence-to-sequence model for open-domain question answering. Guu et al. (Guu et al., 2020) proposed REALM (Retrieval-Augmented Language Model Pre-Training), which integrates retrieval during model pre-training, showing improvements in open domain question-answering tasks. Shuster et al. (Shuster et al., 2021) have shown that using retrieval can reduce LLM hallucination in knowledge-grounded dialogue tasks.
![Refer to caption](x2.png)
2.3. LLMs for Text-to-SQL
The transformation of natural language queries into SQL using LLMs has been a focus of recent research. Pourezza et al. (Pourreza and Rafiei, 2024) proposed decomposing the user question into sub-questions and feeding the solution of those sub-questions to the LLM to generate the final SQL query. Cyren et al. (Wang et al., 2023) proposed a multi-agent collaborative framework for converting the user question to SQL by first semantically choosing relevant tables, then performing query decomposition into multiple sub-queries.
While these works demonstrate the broad applicability of LLMs in hardware design and the potential of text-to-SQL in other domains, the application of text-to-SQL techniques for PDK management remains an underexplored area. HDLCopilot addresses this gap by combining the strengths of LLMs, text-to-SQL conversion, and structured PDK data to provide an efficient and accurate natural language interface for PDK queries.
3. Proposed HDLCopilot Framework
In Fig. 2, we present HDLCopilot, a multi-agent collaborative framework for streamlining interactions with Process Design Kits (PDKs) using natural language. HDLCopilot employs four specialized LLM agents, each designed to perform a specific function. This section first provides an overview of the PDK files conversion to a relational database, followed by a detailed explanation of each agent’s functionality and purpose.
3.1. LLM-Compatible PDK Database Schema
To facilitate integration with LLM-based retrieval, we first convert the PDK files into a relational database. LLM agents can then retrieve information from the database by dynamically generating SQL queries based on the user input. We mainly focus on three views in the PDK files: liberty for storing timing information at different operating conditions, LEF for abstract layout information for each cell, and Technology LEF for storing metal stack properties. We propose a schema for each view. These schema are designed to support storing information across different standard cell libraries different operating conditions within the PDK.
The liberty schema is shown in Fig. 3. This schema comprises tables capturing operating conditions, cell attributes, pin properties, and timing data. The LEF schema is shown in Fig. 4. This schema comprises tables that capture abstract physical information about various macros (cells), including cell dimensions, obstruction layers in cell layouts, and pin physical attributes such as antenna gate area and pin shapes. Fig. 5 depicts the schema for technology LEF files, which stores technology-specific information. This includes details about routing layers (such as preferred routing direction, width, and spacing rules, resistance), via layers, and their associated antenna ratios.
3.2. LLM Agents
HDLCopilot comprises four LLM agents: : the Dispatcher, Selector, SQL-Generator, and Interpreter, which collaborate together to ensure a reliable and accurate SQL generation pipeline.
Dispatcher: The main objective of the dispatcher agent is to route the user question to the appropriate standard cell library, library view, and operating conditions. Given the user question , the available libraries and library views in the PDK, the dispatcher selects the relevant library , library view , and operating conditions (if applicable). The function of the dispatcher agent is described in Eq. 1, where represents LLM agent .
(1) |
The selected library view is then used to filter the PDK tables, retaining only those relevant to the routed view as described in Eq. 2. The selected tables are then passed to to the selector agent to perform further fine-grained table selection by choosing only the tables relevant to the user question.
(2) |
Selector: The selector agent performs a more refined reduction of the tables. Given the user question and the schema description of the routed tables , the selector narrows down the the set of the routed tables to only those most relevant to the user question. The function of the selector agent is described in Eq. 3. The main purpose of this table reduction process is to make the text-to-SQL task easier by having the SQL-Generator only examine the relevant tables. This focused approach enhances the efficiency and accuracy of the SQL generation process.
(3) |
SQL-Generator: The SQL-generator serves as the core agent of the framework. Its primary function is to construct a SQL query that accurately retrieves the required information from the PDK database to address the user’s question. The generator employs a query decomposition approach proposed in (Pourreza and Rafiei, 2024; Wang et al., 2023) that breaks down the user question into smaller, manageable sub-questions. For each sub-question, the generator produces a corresponding sub-query. These sub-queries are then combined to form the final SQL query. This step-wise approach enhances accuracy and allows for handling complex user questions. The generator function is describe in Eq. 4. It takes as input the user question , the schema description of the selected tables , the relevant standard cell library , and operating conditions .
(4) |
Interpreter: The interpreter main role is to translate the raw database results into a coherent, natural language response that directly addresses the user’s question. This agent processes the user question and the result obtained from executing the generated SQL query (Eq. 5) and then then formulates an output answer in natural language format, as formalized in Eq. 6.
(5) |
(6) |
4. Experimental Results
We conduct all experiments using OpenAI’s GPT models. All models were accessed through their API, specifically gpt-3.5-turbo-0125 for GPT3.5, gpt-4-turbo-2024-04-09 for GPP4, and gpt-4o-2024-05-13 for GPT4-o. For the Process Design Kit (PDK), we utilize the open-source Skywater 130nm (sky, [n. d.]). This PDK encompasses Standard Cell Libraries (SCLs). First, we converted the PDK files to a database using our proposed schema. The resulting Skywater database comprises tables, different cell entries, entries for cell timing information at different process corners, with a total size of . This comprehensive large-scale database provides a robust platform for evaluating our framework.
4.1. Evaluation Set
To evaluate HDLCopilot, we created an evaluation set of user questions with corresponding SQL-queries. This set encompasses a diverse range of complexities, from simple single-table selections to complex multi-table joins with sub-queries and multiple conditions. As shown in Table 1, the set incorporates various SQL clauses, aggregation functions, and sub-queries, providing a comprehensive test of the framework’s SQL handling capabilities.
4.2. Evaluation Metrics
Statistic | Total | TechLef | LEF | Liberty |
Total Queries | 52 | 13 | 16 | 23 |
Clause Occurrences: | ||||
JOIN | 7 | - | - | 7 |
ORDER BY | 13 | 2 | 3 | 8 |
WHERE | 113 | 17 | 28 | 68 |
GROUP BY | 2 | - | - | 2 |
Aggregation Functions | 48 | 6 | 1 | 41 |
Sub-queries | 62 | 3 | 12 | 47 |
Average Length (chars) | 222 | 130 | 137 | 333 |
Maximum Length (chars) | 1075 | 202 | 258 | 1075 |
Following text-to-SQL work (Wang et al., 2023; Li et al., 2024; Zhong et al., 2020), we use the Execution Accuracy (EX) and Valid Efficiency Score (VES) to evaluate the performance of our proposed framework.
The Execution Accuracy (EX) quantifies the framework’s ability to generate SQL queries that produce correct results. It measures the proportion of questions in the evaluation set where the execution results match those of the ground truth queries. It is formally defined in Eq. 7, where defines the number of questions in the evaluation set, defines the set returned by the ground truth SQL query and defines the set returned by the predicted SQL query. is an indicator function that is equal to if both the ground truth set and the predicted set are equal and otherwise.
(7) |
The Valid Efficiency Score (VES) evaluates the correctly generated SQLs by comparing their execution time against those of the ground truth SQLs. VES is formally defined in Eq. 8, where is the relative efficiency of the predicted SQL and the ground truth SQL, and is the execution time of each SQL in the database. The VES metric provides insights into both the correctness and the computational efficiency of the generated SQL queries.
(8) |
4.3. Main Results
First, we present three qualitative examples that showcase HDLCopilot’s capability in generating complex SQL queries, retrieving relevant information, and providing precise answers to user questions. Fig. 6 shows a user question that asks for comparing the width of the 4-input MUX across all libraries. The framework is able to answer the question with high precision. This analysis is potentially useful in providing designers with immediate insights into a specific cell’s footprint variation among different library options, helping them determine which library is most suitable for their design requirement. Fig. 7 also presents a cross-library comparison of leakage power in flip-flop cells. This analysis is useful for designers focusing on low-power applications, allowing them to quickly identify the most power-efficient cell for their specific needs. Fig. 8 showcases the framework’s ability to handle more sophisticated queries. In this example, the framework generates and executes a complex SQL query to compare the propagation delay of a 2-input MUX cell between two specific libraries. These three example highlight the system’s ability to perform diverse set of analyses that would be time-consuming if done manually.
We also present quantitative assessment of the system’s performance on the examples in our evaluation set. We first evaluate the accuracy of the dispatcher and the selector independently to determine the most reliable setup for these agents. We conduct evaluations both with and without few-shot examples. Table. 2 shows that few-shot demonstrations generally improves the accuracy for both GPT3.5 and GPT-4 models. GPT-4-o achieves the highest overall accuracy even without few-shot examples, showcasing its ability to comprehend and execute the task without additional context. The best setup is achieved by using GPT-4 and GPT-4o models, with an overall dispatch accuracy of 99.35% across all three routing tasks and table selection accuracy of 98.07%.
Dispatcher* | Selector† | ||||
SCL | View | Corner | |||
GPT-3.5 | w/ few-shot | 98.07% | 96.15% | 95.65% | 69.23% |
w/o few-shot | 78.85% | 96.15% | 61.04% | 36.53% | |
GPT-4 | w/ few-shot | 98.07% | 100.0% | 100.0% | 98.07% |
w/o few-shot | 94.23% | 100.0% | 97.83% | 69.23% | |
GPT-4-o | w/ few-shot | 98.07% | 100.0% | 100.0% | 98.07% |
w/o few-shot | 98.07% | 100.0% | 100.0% | 84.61% |
-
*
Dispatcher w/few-shot configuration uses 6-shot examples for SCL dispatching and 3-shot examples for view and corner dispatching.
-
†
Selector w/few-shot configuration uses 3-shot examples.
Tech. | LEF | Lib. | Overall | |||||
EX | VES | EX | VES | EX | VES | EX | VES | |
GPT3.5 | 92.30 | |||||||
GPT-4 | 92.30 | 100.00 | ||||||
GPT-4o | 92.30 | 93.62 | 100.00 | 98.31 | 91.30 | 101.57 | 94.23 | 98.58 |
Table. 3 shows the Execution Accuracy (EX) and valid Efficiency Score (VES) of the entire framework. GPT-4o demonstrates superior performance, achieving the highest overall EX of 94.23% and VES of 98.58%. The results demonstrate that liberty queries are particularly harder than the LEF and TechLef queries. This is mainly because liberty files contains more attributes and diverse data types. Nonetheless, GPT4-o demonstrates a high execution accuracy of 91.30% for the liberty set, and even generates more efficient SQL queries than the hand-crafted ground truth SQLs, as shown by its VES of 101.57%.
5. Conclusion
In this paper, we introduced HDLCopilot, an LLM-powered multi-agent collaborative framework, designed to streamline interactions with Process Design Kits (PDK) in natural language format. To facilitate integration with LLM-Agents, the PDK information are first converted to a relational database, which HDLCopilot agent interacts with by generating SQL queries to retrieve relevant information. HDLCopilot also has the potential of integrating well with other hardware design copilots in order to give LLMs PDK awareness.
References
- (1)
- sky ([n. d.]) [n. d.]. Open source process design kit for usage with SkyWater Technology Foundry’s 130nm node. https://github.com/google/skywater-pdk
- Blocklove et al. (2023) Jason Blocklove, Siddharth Garg, Ramesh Karri, and Hammond Pearce. 2023. Chip-chat: Challenges and opportunities in conversational hardware design. In 2023 ACM/IEEE 5th Workshop on Machine Learning for CAD (MLCAD). IEEE, 1–6.
- Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964 (2020).
- Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning. PMLR, 3929–3938.
- Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33 (2020), 9459–9474.
- Li et al. (2024) Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. 2024. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems 36 (2024).
- Liu et al. (2023a) Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, et al. 2023a. Chipnemo: Domain-adapted llms for chip design. arXiv preprint arXiv:2311.00176 (2023).
- Liu et al. (2023c) Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren. 2023c. Invited Paper: VerilogEval: Evaluating Large Language Models for Verilog Code Generation. In 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD). 1–8. https://doi.org/10.1109/ICCAD57390.2023.10323812
- Liu et al. (2023b) Shang Liu, Wenji Fang, Yao Lu, Qijun Zhang, Hongce Zhang, and Zhiyao Xie. 2023b. Rtlcoder: Outperforming gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution. arXiv preprint arXiv:2312.08617 (2023).
- Lu et al. (2024) Yao Lu, Shang Liu, Qijun Zhang, and Zhiyao Xie. 2024. RTLLM: An open-source benchmark for design rtl generation with large language model. In 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC). IEEE, 722–727.
- Pearce et al. (2020) Hammond Pearce, Benjamin Tan, and Ramesh Karri. 2020. Dave: Deriving automatically verilog from english. In Proceedings of the 2020 ACM/IEEE Workshop on Machine Learning for CAD. 27–32.
- Pourreza and Rafiei (2024) Mohammadreza Pourreza and Davood Rafiei. 2024. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. Advances in Neural Information Processing Systems 36 (2024).
- Qiu et al. (2024) Siyu Qiu, Benjamin Tan, and Hammond Pearce. 2024. Explaining EDA synthesis errors with LLMs. arXiv preprint arXiv:2404.07235 (2024).
- Shuster et al. (2021) Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567 (2021).
- Thakur et al. (2022) Shailja Thakur, Baleegh Ahmad, Zhenxing Fan, Hammond Pearce, Benjamin Tan, Ramesh Karri, Brendan Dolan-Gavitt, and Siddharth Garg. 2022. Benchmarking Large Language Models for Automated Verilog RTL Code Generation. https://doi.org/10.48550/ARXIV.2212.11140
- Thakur et al. (2023) Shailja Thakur, Baleegh Ahmad, Hammond Pearce, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri, and Siddharth Garg. 2023. Verigen: A large language model for verilog code generation. ACM Transactions on Design Automation of Electronic Systems (2023).
- Tsai et al. (2023) YunDa Tsai, Mingjie Liu, and Haoxing Ren. 2023. Rtlfixer: Automatically fixing rtl syntax errors with large language models. arXiv preprint arXiv:2311.16543 (2023).
- Wang et al. (2023) Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, Qian-Wen Zhang, Zhao Yan, and Zhoujun Li. 2023. Mac-sql: Multi-agent collaboration for text-to-sql. arXiv preprint arXiv:2312.11242 (2023).
- Wu et al. (2024) Haoyuan Wu, Zhuolun He, Xinyun Zhang, Xufeng Yao, Su Zheng, Haisheng Zheng, and Bei Yu. 2024. Chateda: A large language model powered autonomous agent for eda. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (2024).
- Zhong et al. (2020) Ruiqi Zhong, Tao Yu, and Dan Klein. 2020. Semantic evaluation for text-to-SQL with distilled test suites. arXiv preprint arXiv:2010.02840 (2020).