Redefining Developer Assistance: Through Large Language Models in Software Ecosystem
Abstract
In this paper, we delve into the advancement of domain-specific Large Language Models (LLMs) with a focus on their application in software development. We introduce DevAssistLlama, a model developed through instruction tuning, to assist developers in processing software-related natural language queries. This model, a variant of instruction tuned LLM, is particularly adept at handling intricate technical documentation, enhancing developer capability in software specific tasks. The creation of DevAssistLlama involved constructing an extensive instruction dataset from various software systems, enabling effective handling of Named Entity Recognition (NER), Relation Extraction (RE), and Link Prediction (LP). Our results demonstrate DevAssistLlama’s superior capabilities in these tasks, in comparison with other models including ChatGPT. This research not only highlights the potential of specialized LLMs in software development also the pioneer LLM for this domain.
![[Uncaptioned image]](extracted/5474139/softllama.png)
1 Introduction
The advent of Large Language Models (LLMs) has brought a revolutionary shift in various industries, with a significant impact in the field of software development. These models, ranging from General-Purpose Transformers like GPT by OpenAI Brown et al. (2020) to specialized ones such as LLaMA Introducing LLaMA: A foundational and Alpaca Taori et al. (2023), have redefined human-computer interaction and automation in software engineering tasks. In the software industry, LLMs have been instrumental in automating tasks such as code generation Ross et al. (2023); Zhang et al. (2023), test case creation Schäfer et al. (2023), and program debugging Chen et al. (2023). Tools like OpenAI’s Codex che (2021) and GitHub Copilot Dakhel et al. (2023) are exemplary in facilitating code-related tasks and enhancing developer productivity.
However, despite these advancements in the field, the software industry 111https://insights.sei.cmu.edu/blog/application-of-large-language-models-llms-in-software-engineering-overblown-hype-or-disruptive-change/ confronts substantial challenges beyond the realm of coding. While coding is an integral part of a developer’s role, it occupies about 59.5% of their work time 222https://www.software.com/reports/future-of-work 333https://thenewstack.io/how-much-time-do-developers-spend-actually-writing-code/, reflecting a notable focus on this aspect of software development. Surprisingly, a significant portion of a developer’s time, nearly 40-45%, involves tasks beyond coding. These include code maintenance 444https://thenewstack.io/how-much-time-do-developers-spend-actually-writing-code/, testing, security issues 555https://www.techtarget.com/searchitoperations/feature/The-promises-and-risks-of-AI-in-software-development, and other non-coding responsibilities that are essential for the holistic development of software. This division of time highlights the necessity for tools that cater not only to the coding aspect but also to the broader scope of software development. Tasks involving key concepts from Named Entity Recognition (NER), Relation Extraction (RE), Link Prediction (LP), Question Answering (QA), and Forum Answer Ranking (FAR) are pivotal in this broader scope Banerjee et al. (2024b, a); Hazra et al. (2023a, b). They play a critical role in interpreting complex technical documentation 666https://pub.aimind.so/interrogate-your-technical-documentation-using-free-and-paid-llms-f2a7664ff2bd, managing software project requirements 777https://www.linkedin.com/pulse/requirement-engineering-using-llm-madhavan-vivekanandan/, and facilitating effective communication within developer communities 888https://www.smehorizon.com/large-language-models-5-takeaways-for-smes/.
Existing models fall short in addressing the unique requirements and complexities within the software domain considering facts. Their application in understanding technical specifications, extracting relevant information from software documentation, and assisting in the conceptualization of new software projects is not fully optimized. Moreover, in the realm of community engagement, particularly in online forums, the need for sophisticated models that can effectively manage and prioritize a wide range of tasks is increasingly evident. Current models have primarily focused on individual aspects of software development. However, the complexity and diversity of tasks in these forums demand a more integrated approach, combining all applicable tasks strengths.
Recognizing this gap, our research proposes the development of a comprehensive model DevAssistLlama 999Our model card? designed to address these specific tasks in the software domain. This model aims to provide advanced unified solutions for NER, Relation Extraction, Link Prediction, Forum Answer Ranking, and Question Answering. The goal is to create a tool that not only enhances the efficiency of software developers but also provides nuanced and context-aware support in online software development communities. We explore the foundational technologies that empower these models and demonstrate how our proposed model is poised to reshape the paradigms of software development and community interaction, filling a critical gap in the current landscape.
![Refer to caption](x1.png)
Models | NER | RE | LP | |||
---|---|---|---|---|---|---|
DistALANER | StackOverflow | GitHub | WikiSER | DistALARE | BLANCA-L | |
Alpaca | 0.001 | 0.000 | 0.002 | 0.001 | 0.001 | 0.102 |
Vicuna | 0.028 | 0.000 | 0.003 | 0.000 | 0.008 | 0.899 |
ChatGPT | 0.350 | 0.420 | 0.300 | 0.450 | 0.400 | 0.850 |
LlaMa 2 13B | 0.010 | 0.001 | 0.001 | 0.002 | 0.150 | 0.812 |
DevAssistLlama | 0.427 | 0.597 | 0.607 | 0.272 | 0.423 | 0.869 |
Models | FAR |
---|---|
BLANCA-R | |
Alpaca | 0.001 |
Vicuna | 0.165 |
chatGPT | 0.290 |
LlaMa 2 13B | 0.297 |
DevAssistLlama | 0.312 |
Models | QA | ||
---|---|---|---|
ServerFault | askUbuntu | Android | |
Alpaca | 0.42 | 0.48 | 0.44 |
Vicuna | 0.49 | 0.43 | 0.53 |
chatGPT | 0.76 | 0.72 | 0.64 |
LlaMa 2 13B | 0.76 | 0.76 | 0.71 |
DevAssistLlama | 0.80 | 0.81 | 0.77 |
2 Dataset
Dataset | Number of Examples |
---|---|
DistAlaner |
450,000 |
WikiSER |
24,602 |
StackOverflow |
10,807 |
Link Prediction |
23,516 |
Forum Answer Ranking |
330,151 |
Android |
16,523 |
ServerFault |
152,082 |
AskUbuntu |
134,655 |
We mainly utilize our model for five state-of-art elementary tasks which are basic component of different high level tasks – named entity recognition, relation extraction, forum answer ranking, forum questions relateness, question-answering (See Figure 1).
Named entity recognition (NER) Anonymous (2023); Tabassum et al. (2020); Nguyen et al. (2023): NER is an elementary method for automated requirement extraction Haris and Kurniawan (2021), code document analysis Chew et al. (2023) and bug tracking and resolution. The task is to extract the entities of specific entity types given a software related textual information. We consider four software system related NER datasets. The DistalaNER Anonymous (2023) data consists of nine different software entity types. The textual information is the bug description. The entities are packages, operating system, organization, commands, error, file extension, peripheral, software component and computer architecture. Total number of instances present in the dataset is 170K. We used the Stackoverflow and Github data proposed in paper Tabassum et al. (2020). In these datasets, the texts are the sentences of the questions. Both of them has around 20 entity types where 8 code related entities and 12 natural language entities. The stackoverflow dataset consists of 15,372 sentences with 19,122 entities. The github data consists of 6,510 sentences with 10,963 entities. The Wikiser Nguyen et al. (2023) dataset consists of 12 entity types. It contains the wikipedia textual information. This dataset consists of 1.7M sentences with 79K unique software entities.
Relation extraction (RE) Anonymous (2023): In this dataset, textual information, along with two entities, serves as input, and the relation type acts as the output. It encompasses five relation types: dependency, conflict, affected version, cause and effect, and interaction/control. The textual information comprises bugs from the Ubuntu ecosystem. This dataset contains 500 instances.
Forum answer ranking (FAR) Abdelaziz et al. (2022): We use the forum answer ranking dataset from the Blanca benchmark proposed in paper Abdelaziz et al. (2022). The task is to predict the best answers relative to other answers provided for a given question in a forum. The dataset consists of 500K questions, each having at least 3 answers, with an average of 4.9 answers per question.
Forum link prediction (LP) Abdelaziz et al. (2022): The task involves predicting whether two questions are related or not. The dataset consists of 23,516 pairs of posts, with 11,758 related pairs and 11,758 unrelated pairs.
Question answering 101010 https://serverfault.com/ 111111https://askubuntu.com/ 121212https://android.stackexchange.com/:
The task requires the model to generate answers for software-related questions. We compile datasets from three community question answering systems: Android CQA with 21,560 pairs, AskUbuntu with 175,000 pairs, and ServerFault with 210,500 pairs, all from early 2023.
3 Developer assistance through instruction tuning
In this work, our main objective is to build a model which would assist developers to address natural language software queries in various stages. We utilize the instruction tuning method on pretrained LLMs to achieve this objective. Instruction tuning is one of the fine tuning methods for aligning the llm to specific tasks or specific domain. For each pair of instruction and desired response in the instruction dataset, we update the model parameters to reduce the difference between the model’s output and desired output . Further, this process iteratively update to improve the model’s performance in following instructions and producing the desired outputs. In our work, this methodology involves the refinement of pre-existing llama2-13b. There are various existing works on instruction tuning where the main focus is to solve specific tasks Zhou et al. (2023); Lu et al. (2023). Whereas we investigate how these pretrained LLMs as well as instruction tuned LLM can be employed as a component in the pipeline of various software system tasks. In the subsequent sections, we describe the construction of instruction dataset, instruction tuning process and evaluation tasks.
System Message |
[INST]You work with software developers from software industries. Your task is to {Task}.[\INST] |
Task: {Named Entity Recognition} |
Instructions: |
Assume that you are a named entity recognizer. You will be given text, entity type as inputs and you have to extract possible entities of given entity type from the text. |
Task: {Link prediction} |
Instructions: |
Assume that you are a software question relevance checker. You will be given two questions as inputs and you have to identify whether two questions are relevant or not. Use 0 for non-relevant and 1 for relevant. |
Task: {Community Question Answering} |
Instructions: |
Assume that you are a helpful software community question answering expert. You will be given a question as inputs and you have to provide the answer. Your answer should be very crisp and clear. If you don’t know the answer, do not try to make up the answer. |
Task: {Forum Answer Ranking} |
Instructions: |
Assume that you are an answer ranker in a community question answering portal. Given the question, rank the following answers based on their relevance from most relevant (1) to least relevant (0). If an answer is completely irrelevant, it should be ranked last. Strictly follow the Output Format: "OPTION [answer number]: Rank [ranking number]". DO NOT include any other additional text. |
3.1 Instruction dataset construction
For instruction tuning, we build an instruction dataset from various software system datasets. This dataset is meticulously curated to encompass a broad spectrum of instructional formats, contexts and ensuring comprehensive exposure to various types of related tasks. We build instruction dataset from different datasets of major tasks in software systems such as named entity recognition, forum answer generation, forum answer ranking and forum related questions prediction. Firstly, we merge training datasets present for all these tasks. We provide training dataset statistics in Table 4. Secondly, we create instruction for each of the tasks and prepare the instruction set. We provide prompts of each task in Table 5. Further, we shuffle each datasets and considered 5000 instances from each dataset. Our final training set contains 45000 instances with five different type of instructions.
3.2 Instruction tuning
After preparing the instruction dataset, we implement instruction tuning on the Large Language Model (LLM) to specifically adapt it for various tasks commonly handled by software developers. This fine-tuning involves inputting both the definitions of tasks and relevant textual information into the LLM. Such an approach ensures that the model becomes finely attuned to the particular nuances and specific requirements of domain-centric tasks.
3.3 Negative sampling
Our methodology crucially includes negative sampling in the instruction dataset to enhance the model’s ability to differentiate between relevant and irrelevant or non-existent data. In Named Entity Recognition (NER) tasks, for example, we introduce scenarios where text falsely associates with a non-existent entity type, expecting a null response from the model. This trains the model to recognize both the presence and absence of entities, reducing false positives. For instance, if a text mentions software tools but no programming languages, and the entity type is ’programming language’, the model should produce an empty output. Similarly, for question-answering tasks, we include questions outside the software system domain, like historical or culinary queries, to teach the model to filter out irrelevant inquiries. Negative sampling is vital for enhancing the model’s precision and reliability, equipping it to handle scenarios where the absence of relevant information is crucial. This approach ensures accurate responses and avoids misleading information when the model lacks necessary context.
4 Experimental setup
4.1 Model setup
In our study, we employ a model setup inspired by Chiang et al. (2023). The cornerstone of our experimentation is Llama 2, fine-tuned to enhance its efficacy in our designated tasks. This approach is in line with cutting-edge methodologies in natural language processing. For evaluation, we select up to 1,000 instances from three distinct random splits, ensuring a diverse and representative test dataset. This strategy aims to emulate real-world scenarios and offers a comprehensive evaluation of the model’s capabilities.
4.2 Compared models
We evaluate the performance of several prominent Large language models. ChatGPT (gpt-3.5-turbo-0301) is renowned for its conversational abilities and versatility in various language tasks, initially based on the GPT-2 model from Radford et al. (2019). Vicuna, credited to Chiang et al. (2023), stands out for its efficiency, being fine-tuned on a LLaMA base model using conversational data. Similarly, Alpaca, developed by Taori et al. (2023), uses the LLaMA architecture but is specifically tuned for domain-specific dialogues. Our focus, Llama 2, is fine-tuned following tou (2023) guidelines, showcasing its capabilities in complex language processing scenarios and offering insights into the evolving landscape of vaious NLP tasks.
4.3 LoRA approach for fine-tuning
Low-Rank Adapters (LoRA) represent a pivotal advancement in the field of natural language processing, particularly in the fine-tuning of large language models. First introduced by Hu et al. (2021), LoRA allows for the efficient and effective modification of pre-trained models, significantly reducing the computational overhead typically associated with training large-scale neural networks. In our implementation, we integrate LoRA adapters directly into the attention mechanisms and train both the language model (LM) heads and the adapters concurrently, diverging from traditional methods that prioritize the initial training of embeddings alone, as seen in models like the Chinese LLaMA Cui et al. (2023). Our infrastructure utilizes the robust capabilities of six NVIDIA A100 GPUs, each with 46GB of memory, enabling us to train our model, DevAssistLlama, over three epochs spanning a total of 63 hours. We showcased important hyperparameters in Table 6.
5 Results
Our evaluation of various models in the NER, RE, and LP tasks, measured using the macro F1 score, offers significant insights, as detailed in Table 1. In the NER tasks across DistALANER, StackOverflow, GitHub, and WikiSER datasets, DevAssistLlama leads with scores of 0.427, 0.597, 0.607, and 0.272, respectively. ChatGPT also shows impressive performance with scores of 0.350, 0.420, 0.300, and 0.450 in the same tasks, indicating its strong capabilities in named entity recognition.
For the RE task on DistALARE, DevAssistLlama tops with a score of 0.423, closely followed by ChatGPT with a score of 0.400. In the LP task on BLANCA-L, DevAssistLlama again leads with a score of 0.869, with ChatGPT showing a commendable score of 0.850.
FAR Performance: Table 2 illustrates the FAR performance, using the Mean Reciprocal Rank (MRR) as the evaluation metric. Here, DevAssistLlama achieves the highest MRR score of 0.312, with LlaMa 2 13B at 0.297. ChatGPT’s performance is also noteworthy, with an MRR score of 0.290, showcasing its efficiency in fault analysis and recovery.
QA Performance: In QA tasks covering ServerFault, askUbuntu, and Android domains (Table 3), DevAssistLlama outshines others with scores of 0.80, 0.81, and 0.77, respectively. ChatGPT, however, delivers the highest performance in this category, with impressive scores of 0.76, 0.72, and 0.64, indicating its strong suitability for QA tasks.
Overall Assessment: This comprehensive evaluation across diverse NLP tasks using macro F1 score, MRR for FAR, and FactSUMM for QA tasks, highlights the exceptional capabilities of DevAssistLlama and ChatGPT. DevAssistLlama consistently tops in NER, RE, and LP tasks, while ChatGPT excels notably in QA tasks, affirming its proficiency in complex language processing challenges.
5.1 Evaluation metrics
To evaluate our model, we employ a range of metrics to assess performance across different tasks. For Named Entity Recognition (NER), Relation Extraction (RE), and forum link prediction tasks, we utilize the F1 score. This metric is preferred in machine learning for its ability to assess predictive skill by focusing on class-wise performance. The F1 score combines precision and recall, using their harmonic mean, and is particularly useful in situations where datasets are class-imbalanced 131313https://tinyurl.com/adkbk2yk.
For tasks like forum answer ranking, we apply the Mean Reciprocal Rank (MRR). MRR is a measure that determines the mean of the reciprocal ranks of the actual relevant item in the ranking list of items provided in response to queries 141414https://en.wikipedia.org/wiki/Mean_reciprocal_rank. For question answering datasets, we use factsumm metric which measure the facts overlap between actual answer and generated answer.
Important hyperparameters of DevAssistLlama |
---|
Learning Rate: 2e-4| Max sequence length: 4096| |
LoRA Alpha: 16| Training Precision: FP16| |
Epochs: 3| Optim: paged_adamw_32bit| |
fp16=True| bf16=True| Warmup ratio=0.03| |
LR scheduler=Cosine| Trainer= SFTT| |
Per device training batch=8 |
6 Limitation and future work
The DevAssistLlama, a fine-tuned version of the Llama 2 model specifically for the software domain, exhibits certain limitations and areas for future development.
Contextual Understanding: While DevAssistLlama may excel in general language processing, its understanding of complex software terminology and concepts could be limited. This could affect its ability to accurately interpret and respond to highly technical queries.
Code Generation and Debugging: The model might struggle with generating error-free code or providing accurate debugging solutions, as these tasks require a deep understanding of programming logic and syntax.
Data Bias and Quality: The model’s performance heavily depends on the quality and diversity of the training data. Any biases or limitations in the dataset can lead to suboptimal or skewed outputs.
Real-time Adaptation: DevAssistLlama might face challenges in adapting to the rapidly evolving software technologies and terminologies, leading to outdated or irrelevant responses.
To address these limitations we can explore certain steps. Enhanced Technical Training: Integrating more comprehensive and diverse software-specific datasets could improve the model’s understanding of complex technical concepts and terminologies. Collaboration with Developers: Working closely with software developers to fine-tune the model could lead to more accurate and contextually relevant outputs. Real-Time Learning Mechanisms: Implementing mechanisms for real-time learning and adaptation could help the model stay updated with the latest software trends and practices. Robust Code Generation and Debugging: Focusing on enhancing the model’s capabilities in code generation and debugging could make it a more valuable tool for software development.
References
- che [2021] Evaluating large language models trained on code, 2021.
- tou [2023] Llama 2: Open foundation and fine-tuned chat models, 2023.
- Abdelaziz et al. [2022] Ibrahim Abdelaziz, Julian Dolby, Jamie McCusker, and Kavitha Srinivas. Can machines read coding manuals yet? – a benchmark for building better language models for code understanding. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2022), 2022.
- Anonymous [2023] Anonymous. Anonymous, June 2023. URL https://doi.org/10.5281/zenodo.8075578.
- Banerjee et al. [2024a] Somnath Banerjee, Avik Dutta, Aaditya Agrawal, Rima Hazra, and Animesh Mukherjee. Distalaner: Distantly supervised active learning augmented named entity recognition in the open source software ecosystem. CoRR, abs/2402.16159, 2024a. doi: 10.48550/ARXIV.2402.16159. URL https://doi.org/10.48550/arXiv.2402.16159.
- Banerjee et al. [2024b] Somnath Banerjee, Amruit Sahoo, Sayan Layek, Avik Dutta, Rima Hazra, and Animesh Mukherjee. Context matters: Pushing the boundaries of open-ended answer generation with graph-structured knowledge context. CoRR, abs/2401.12671, 2024b. doi: 10.48550/ARXIV.2401.12671. URL https://doi.org/10.48550/arXiv.2401.12671.
- Brown et al. [2020] Tom B Brown et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
- Chen et al. [2023] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug, 2023.
- Chew et al. [2023] Robert Chew, John Bollenbacher, Michael Wenger, Jessica Speer, and Annice Kim. Llm-assisted content analysis: Using large language models to support deductive coding, 2023.
- Chiang et al. [2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/.
- Cui et al. [2023] Yiming Cui, Ziqing Yang, and Xin Yao. Efficient and effective text encoding for chinese llama and alpaca, 2023.
- Dakhel et al. [2023] Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, Zhen Ming, and Jiang. Github copilot ai pair programmer: Asset or liability?, 2023.
- Haris and Kurniawan [2021] M Syauqi Haris and Tri Astoto Kurniawan. Automated requirement sentences extraction from software requirement specification document. In Proceedings of the 5th International Conference on Sustainable Information Engineering and Technology, page 142–147, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450376051. doi: 10.1145/3427423.3427450.
- Hazra et al. [2023a] Rima Hazra, Agnik Saha, Somnath Banerjee, and Animesh Mukherjee. Evaluating the ebb and flow: An in-depth analysis of question-answering trends across diverse platforms. CoRR, abs/2309.05961, 2023a. doi: 10.48550/ARXIV.2309.05961. URL https://doi.org/10.48550/arXiv.2309.05961.
- Hazra et al. [2023b] Rima Hazra, Debanjan Saha, Amruit Sahoo, Somnath Banerjee, and Animesh Mukherjee. Duplicate question retrieval and confirmation time prediction in software communities. CoRR, abs/2309.05035, 2023b. doi: 10.48550/ARXIV.2309.05035. URL https://doi.org/10.48550/arXiv.2309.05035.
- Hu et al. [2021] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
- [17] 65-billion-parameter language model Introducing LLaMA: A foundational. URL https://ai.meta.com/blog/large-language-model-llama-meta-ai/.
- Lu et al. [2023] Jinghui Lu, Rui Zhao, Brian Mac Namee, and Fei Tan. Punifiedner: A prompting-based unified ner system for diverse datasets. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11):13327–13335, Jun. 2023. doi: 10.1609/aaai.v37i11.26564. URL https://ojs.aaai.org/index.php/AAAI/article/view/26564.
- Nguyen et al. [2023] Tai Nguyen, Yifeng Di, Joohan Lee, Muhao Chen, and Tianyi Zhang. Software entity recognition with noise-robust learning. In Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering (ASE’23). IEEE/ACM, 2023.
- Radford et al. [2019] Alec Radford, Jeff Wu, Rewon Child, D. Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. URL https://www.semanticscholar.org/paper/Language-Models-are-Unsupervised-Multitask-Learners-Radford-Wu/9405cc0d6169988371b2755e573cc28650d14dfe.
- Ross et al. [2023] Steven I. Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D. Weisz. The programmer’s assistant: Conversational interaction with a large language model for software development. In Proceedings of the 28th International Conference on Intelligent User Interfaces, IUI ’23, page 491–514, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701061. doi: 10.1145/3581641.3584037.
- Schäfer et al. [2023] Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. An empirical evaluation of using large language models for automated unit test generation, 2023.
- Tabassum et al. [2020] Jeniya Tabassum, Mounica Maddela, Wei Xu, and Alan Ritter. Code and named entity recognition in stackoverflow. In The Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
- Taori et al. [2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023.
- Zhang et al. [2023] Jieyu Zhang, Ranjay Krishna, Ahmed H Awadallah, and Chi Wang. Ecoassistant: Using llm assistant more affordably and accurately. arXiv preprint arXiv:2310.03046, 2023.
- Zhou et al. [2023] Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. Universalner: Targeted distillation from large language models for open named entity recognition. 2023.