LLM-jp:
A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs

The main role of the Corpus Building WG is to build a pre-training corpus and a tokenizer needed for LLM construction and pass them to the Model Building WG.

In the following subsections, we describe our work for the pre-trained models in our model suites v1.0 and v2.0. Then, we explain the corpus search function, which is one of our advantages. Finally, we summarize our ongoing and future work.

Our initial milestone was to develop the model suite v1.0, and the Corpus Building WG worked on preparing a pre-training corpus to train the pre-trained model v1.0, the LLM with 13B parameters within this suite. The main purpose of this development was to experience the entire development process of an LLM as soon as possible.

To this end, we decided to use a mixture of readily available Japanese, English, and code corpora as our pre-training corpus. As for the corpus size, we followed the Chinchilla scaling law [20], which suggests using roughly 20 tokens per parameter. Eventually, we constructed the corpus v1 consisting of over 260B tokens. The statistics of this corpus are listed in Table 1. From this corpus, we extracted a pre-training dataset that consists of 130B Japanese, 130B English, and 10B code tokens, resulting in a total of 270B tokens.

As for the Japanese portion, we used the Japanese parts of Wikipedia and the multilingual C4 (mC4) dataset [57]. Since the Japanese part of mC4 was noisy, we filtered out documents that were considered low-quality or harmful. Table 2 shows filters adopted for this purpose. For the English and code portions, we utilized the Pile dataset [16] and the Stack dataset [29], respectively. To adjust the corpus size, we sampled documents from these two sources accordingly.

We developed tokenizers based on SentencePiece with the unigram mode [30]. As a multi-lingual tokenizer considering Japanese, we first explored the tokenizer developed in the project “Development of a distributed parallel learning method for large-scale language models in the policy-oriented framework of the supercomputer Fugaku”²²2https://www.titech.ac.jp/english/news/2023/066798, which we refer to as the tokenizer v1.0. The construction process is as follows:

1. 1.

   Preparing training data to construct the tokenization models for each language (i.e., Japanese and English).

2. 2.

   In order to prevent the tokenization models from learning tokens longer than Japanese word boundaries, Japanese data was pre-tokenized using the morphological analyzer MeCab³³3https://taku910.github.io/mecab/ with the Japanese morphological dictionary JumanDIC⁴⁴4https://hayashibe.jp/tr/mecab/dictionary/juman. This pre-tokenization specifically aimed to avoid learning tokens such as browser operation phrases, which are frequently included in web corpus, and meaningless long phrases, which are typically used only on specific websites. Pre-tokenization was also performed for sequences including characters other than the alphabet, hiragana, katakana, and kanji into a sequence of single characters to prevent the constructed vocabulary from including tokens with a sequence of symbols and numbers.

3. 3.

   Constructing SentencePiece models of the unigram tokenizer for Japanese and English using the pre-processed training data, independently.

4. 4.

   Merging the vocabularies of the above two tokenization models, removing duplicate tokens.

5. 5.

   Re-estimating unigram scores of tokens in the merged vocabulary with the EM algorithm over the training data⁵⁵5Existing implementation of multigram language model [12] was used, which is available at https://github.com/tatHi/multigram.. Here, data without pre-processing was used to enable the final tokenization model to be used without any pre-tokenization.

Although the construction process seems complicated, the obtained model can be used as a pure SentencePiece model. This multi-step process for the model construction enables us to control the ratio of the vocabulary size for each language.

However, because the tokenizer v1.0 was originally developed for the Fugaku project, we needed to re-train the tokenizer model with the corpus used in the LLM-jp project, the corpus v1. In addition, some specifications of the tokenizer v1.0, such as the handling of white spaces and line breaks, were left open for discussion.

Therefore, based on the idea of the tokenizer v1.0, we constructed the tokenizer v2.1 for use in the model suite v1.0 by using a subset of the corpus v1 and extending the target languages to Japanese, English, and code. Besides, we adjusted the handling rules of white spaces, line breaks, and special tokens, which resulted in efficient tokenization in the corpus v1. The vocabulary of the tokenizer v2.1 is constructed from 30,000 tokens for Japanese, 20,000 tokens for English, and 10,000 tokens for code. The final size of the vocabulary is approximately 50,000, which indicates that about 10,000 tokens are duplicated among the three vocabularies.

The corpus v1 and tokenizer v2.1 were handed over to the Model Building WG in August 2023 and used for pre-training. The Model Building WG requested the highest quality corpus used at the end of the pre-training. In response, we applied the filtering methods described earlier with stricter thresholds to the original corpora and extracted 27B high-quality tokens.

The corpus v1 is publicly available⁶⁶6https://gitlab.llm-jp.nii.ac.jp/datasets/llm-jp-corpus-v1. The code to construct the corpus is also released to the public⁷⁷7https://github.com/llm-jp/llm-jp-corpus. Besides, the tokenizer v2.1 and its corresponding scripts are available for download⁸⁸8https://github.com/llm-jp/llm-jp-tokenizer/releases/tag/v2.1.

To develop the LLM with 13B parameters included in our model suite v2.0, called the pre-trained model v2.0, we created a larger and higher-quality corpus, termed the corpus v2.

To construct a Japanese corpus to this end, we extracted Japanese documents from the entire Common Crawl and applied deduplication and filtering for them. The corpus v2 construction script was developed in Uzushio¹⁰¹⁰10https://github.com/WorksApplications/uzushio, an Apache Spark-based corpus preprocessing tool developed for processing billion-token scale training corpus from web data such as Common Crawl. Uzushio provides a framework for processing such as similarity-based duplicate detection and filtering. Table 3 summarizes the filters and conversions performed to construct the Japanese portion of the corpus v2. The filtering pipeline consisted of deduplication and rule-based filtering steps. In de-duplication, Uzushio performs similarity-based document identification based on the SimHash algorithm. This allows Uzushio to apply multiple strengths of de-duplication to documents from a web corpus. The statistics of the Japanese corpus from Common Crawl dumps are presented in Figure 2. We used the publicly available Common Crawl dumps from 2013 to the middle of 2023. We merged the Common Crawl dumps from 2013 to 2016 because they included fewer Japanese documents than the later dumps. The total extracted Japanese tokens were 285.5B¹¹¹¹11In the training of the pre-trained model v2.0, we sampled approximately 130B tokens, following the Chichilla scaling law.. Further analyses on the v2 corpus are discussed in Enomoto et al. [13].

As for the English and code portions, we used the Pile and Stack datasets, respectively, following the corpus v1. Besides, we included Japanese and Wikipedia as high-quality text corpora in the corpus v2.

The corpus v2 has been made publicly available.¹²¹²12https://gitlab.llm-jp.nii.ac.jp/datasets/llm-jp-corpus-v2

As for the tokenizer, we newly developed the tokenizer v2.2. The training flow of the tokenization model is the same as that of the tokenizer v2.1. The size of the vocabulary was expanded to 96,867¹³¹³13You can see that the vocabulary size is 97,024 after loading the language model (i.e., vocab_size in config.json). This is a result of rounding up the vocabulary size to multiples of 256 to make the SoftMax layer training process on the GPU efficient.. Besides, while the tokenizer v2.1 used a single token per character for symbols to conserve vocabulary, which resulted in over-segmentation of English and code text and reduced tokenization efficiency, in the tokenizer v2.2, the vocabulary is constructed in a way that allows for symbol sequences, and the tokenization efficiency is improved, especially for English and code text.

In addition to corpus construction, the Corpus Building WG is also working on developing a corpus search function, aiming to attribute generated text to the training corpus. This function will be used to analyze generated texts and potentially uncover the principles of LLMs from the perspective of the training corpus. For example, we plan to use this system to investigate the causes of hallucinations in generated text.

Currently, two search algorithms are being explored: sparse vector search and dense vector search. Sparse vector search retrieves documents based on the superficial similarity between texts. It is particularly effective when the generated texts contain distinctive words. Additionally, it also helps identify verbatim memorization [6] in generated texts. Dense vector search, on the other hand, retrieves documents based on the similarity between text embeddings computed by pre-trained text embedding models. Compared to sparse vector search, dense vector search excels at considering the meaning of texts. Furthermore, by using multilingual text embedding models (e.g., LaBSE [14]), it can retrieve semantically similar documents across different languages, which helps analyze the cross-lingual transfer ability of LLMs [41].

We decided to build a 175B-class model as the next target of model building in LLM-jp, and are now building the corpus v3. This new corpus will consist of approximately 2T tokens that cover Japanese, English, some Asian languages, and code.

In our corpora, the mixing ratio of Japanese and English is set at 50-50, but we believe that further study is needed on the mixing ratio and the size of the corpora. In addition to Wikipedia and web documents, we are negotiating with relevant organizations to use high-quality corpora and corpora from various domains, such as scientific and technical papers, patent documents, and domain documents from the medical field.

The role of the Model Building WG is to pre-train language models. The main tasks include:

1. 1.

   preprocessing the pre-training corpus (such as converting it into a binary format for faster data loading during pre-training),

2. 2.

   performing the pre-training, and

3. 3.

   converting the checkpoints from the pre-training into a model format that is suitable for fine-tuning.

The following subsections describe how we built the pre-trained models v1.0 and v2.0. Table 4 summarizes the configuration for these models.

In May 2023, when this project started, the Model Building WG began its activities with the aim of building and releasing a 13B-parameter model specifically focusing on Japanese by autumn or winter 2023. At the start of activities in May 2023, no one in the Model Building WG had solid knowledge or experience in pre-training language models with over 10B parameters using more than 10 computing nodes. Therefore, all participants in the WG experienced what was necessary for pre-training step by step, gaining knowledge and experience through a process of trial and error.

First, to pre-train a language model, we need to prepare a training program (code). While there was the option to develop our own training program, our goal was to build a 13B-parameter model within a few months, making the option infeasible. At the start of the project in May 2023, there were several tools available for pre-training language models with over 10B parameters, so we decided to use them. Specifically, we considered Megatron-DeepSpeed¹⁵¹⁵15https://github.com/microsoft/Megatron-DeepSpeed, GPT-NeoX¹⁶¹⁶16https://github.com/EleutherAI/gpt-neox, and LLM Foundry¹⁷¹⁷17https://github.com/mosaicml/llm-foundry as candidates. Pre-training requires massive computing resources, such as using more than 10 GPU nodes for over 10 days. Therefore, there was not enough time to run multiple training sessions simultaneously or to use multiple tools to create and compare several models side-by-side. Considering several factors, including the fact that several participants had experience with it and that developers involved in DeepSpeed had been participating in LLM-jp activities from the beginning, we regarded Megatron-DeepSpeed as the primary tool for Model Building WG’s activities. However, it was also decided to use GPT-NeoX and LLM Foundry in parallel up to the verification of training speed and stability. Each tool was assigned to a person in charge, and teams were formed to compare the results. Eventually, we chose to build the pre-training model using Megatron-DeepSpeed, as we did not find GPT-NeoX and LLM Foundry to be clearly superior in terms of execution speed or training stability (just to clarify, GPT-NeoX and LLM Foundry were not inferior).

First of all, the training speed of the language model was verified. For example, when training a language model with more than 10B parameters on a training corpus of more than 100B tokens, it is usually necessary to use a computing environment of more than 10 GPU nodes to finish the training in a realistic time. Even if the same language model learning configuration is used, the learning speed can vary greatly depending on the characteristics of the computer cluster environment. Therefore, it is necessary to find the appropriate learning settings for each computer environment used. The details of the computing environment and the various issues related to it are summarised in Section 3. We searched for the optimal setting according to FLOPS (floating point operations per second), an index that is independent of the size of the model and differences in the computer environment and thus often used in existing research as a measure of learning speed. In Megatron-Deepspeed, there are many configurable settings related to the learning speed of language models, including model parallelism (tensor parallel, pipeline parallel) [34, 48], data parallelism, batch size, and a setting called ZeRO [44], which mainly determines the trade-off between GPU/CPU memory utilization and speed. Various settings were prepared by combining the values of each item, and measured learning speeds were collected for each. Finally, the setting that produced the most stable and highest FLOPS value was adopted.

By measuring the actual processing speed, we predicted the total time required for the model construction once the size of the training corpus is determined. The total learning time at that time was predicted as follows: Using a 12 node mdx computing cluster to train a model with 13B parameters, the measured processing speed was 170K tokens/second on average. Therefore, the estimated total time required for training the 13B parameter model with a corpus of 270B tokens was roughly 441.2 hours or about 18.4 days.

We had been preparing to learn a language model using a training corpus of 270B tokens, but as the volume of the training corpus was expected to increase, we considered a learning method that would enable continuous pre-training even if the training corpus increased sequentially. Here, we tried a method in which the training data of 270B tokens is divided roughly into 10 chunks of 27B tokens, and these tokens are trained one by one. Assuming that about one trillion tokens of data would be learned in the future, we applied a setting for learning one trillion tokens to the learning rate scheduler, which is the cosine decay scheduler typically used in the literature of pre-training language models. We also asked the Corpus Building WG to prepare a training corpus of 27B tokens selected from 270B tokens, which were considered to be of high quality, and used this 27B token training corpus at the end of the pre-training. When we trained the final 270 billion tokens, we also rapidly decreased the learning rate to the predefined final learning rate for overall pre-training. This decrease started from the learning rate at the end of the preceding 27B tokens training, using the same cosine scheduler but with a different hyperparameter setting.

Another aspect to consider when pre-training language models is the stability of the training. In LLM pre-training, we often observe that the model cannot be learned effectively due to loss divergence, often called loss explosion and loss spike. At that moment, the mechanism of loss divergence had not been fully elucidated. Therefore, we need to explore and use a setting in which loss divergence occurs as little as possible. We are basically required to deal with this problem through trial and error, but fortunately, no unresolvable loss divergence occurred in our pre-training.

The pre-trained model v1.0 uses a model architecture based on GPT-2 [42]. Although GPT-2 is a relatively old model architecture, and while a newer one was possible, we deemed it more appropriate to use a well-established and stable one, considering the need for a reliable model for many users. Additionally, converting the model checkpoints to a format compatible with the Hugging Face Transformers library¹⁸¹⁸18https://github.com/huggingface/transformers is a common practice, making it crucial to ensure the model can be converted. Unfortunately, the Transformers library does not support the Megatron-DeepSpeed model format used in our training, so a conversion script is needed. From this perspective, while Megatron-DeepSpeed offers a script for converting to the Hugging Face Transformers format, it only supports GPT-2-based models. Therefore, without a custom conversion script, we could only use GPT-2-based models with Megatron-DeepSpeed. Given our limited resources and the fact that this was LLM-jp’s first attempt, we concluded that using the GPT-2 model was the safest choice.

Following the various studies described above, the preliminary learning of the language model for public use began in earnest around the end of August. In practice, learning the target 13B parameter model out of the blue was also risky, so learning the 1.3B parameter model was carried out as a pre-production exercise. Eventually, the pre-training of the 13B parameter model took 26 days. During the training process, there was trouble that the training stopped several times, and it was necessary to restart the training manually. If the training had proceeded without any problems, it could have been carried out in about 21 days at the shortest. The model was then handed over to the Fine-tuning and Evaluation WG, which completed the work of building the model v1.0 for publication in the Model Building WG.

As mentioned in the previous section, the pre-trained model v1.0 was our initial attempt, and we had a time constraint for its construction and release. This means that our primary focus was on quickly building the model on schedule rather than investigating how to obtain a world-class, high-quality model. To identify a better pre-training configuration for the pre-trained model v2.0, we conducted experiments prior to beginning its construction.

As described in Section 2.5, we plan to build a 175B-parameter-class model as the next target of model building in LLM-jp. In practice, we have already tried pre-study using a GPT-3 compliant model on a trial basis using the LLM construction support program at ABCI²¹²¹21https://abci.ai/ja/link/llm_support_program2023.html and have identified some issues to consider, such as loss-spike. We are preparing the implementation to mitigate such issues. The Model Building WG is diligently working to build a 175B-parameter-class language model, trained with a dataset of over 1T tokens (called the corpus v3), publicly available this autumn. For this purpose, we have submitted (and been selected) to an LLM construction support program at the Ministry of Economy, Trade and Industry (METI) in Japan, called GENIAC²²²²22https://www.meti.go.jp/policy/mono_info_service/geniac/index.html.

This section introduces our efforts on fine-tuning and evaluation of LLMs. Pre-trained language models can produce natural and fluent text following input text (prompts), but they do not necessarily produce responses that humans would expect in response to the input. To develop interactive LLMs like ChatGPT, it is essential for them to have the ability to generate appropriate responses to user input; i.e., they need to be aligned with human values [39]. Alignment is an essential issue in LLM research and development, and fine-tuning is an indispensable step in achieving this.

Evaluation is another critical issue for the development and deployment of LLMs. A conventional method for evaluating NLP systems has been to design a specific task, such as question answering and machine translation, and to develop test data for each designed task. However, this method is insufficient for the evaluation of LLMs because LLMs are used in a variety of downstream tasks. We therefore develop evaluation frameworks that can assess diverse natural language understanding capabilities of LLMs.

To date, we have released three versions of our fine-tuned models: v1.0, v1.1, and v2.0. The fine-tuned model v1.0 was released alongside the pre-trained model v1.0. In the fine-tuned model v1.1, which is based on the same pre-trained model v1.0, we improved the instruction-following ability by refining the instruction-tuning data and adding Direct Preference Optimization (DPO), and released it in February 2024. The fine-tuned model v2.0, released in April 2024, features the use of pre-trained model v2.0 and incorporates fine-tuning that considers safety aspects. This section outlines the methods for constructing each model. Table 7 summarizes the datasets used for the fine-tuning of each version.

Unlike traditional, task-specific NLP systems, LLMs can generally be used in various applications. It is, therefore, challenging to develop a specific benchmark to evaluate the entire capability of LLMs. Because of this problem, many evaluation benchmarks for LLMs have been proposed globally [4, 63]. However, the number of evaluation benchmarks, like JGLUE [31], for Japanese LLMs was limited when we started developing LLM-jp models.

We have been developing an evaluation framework to aim for multifaceted evaluation rather than depending on a single benchmark. A variety of benchmark datasets for conventional NLP tasks for Japanese have been proposed to date. We have therefore constructed llm-jp-eval²⁹²⁹29https://github.com/llm-jp/llm-jp-eval, an open-source tool for evaluating Japanese LLMs across these individual tasks. In the same way as constructing jaster, existing datasets for Japanese NLP tasks are converted into prompt-answer pairs. When evaluating LLMs, prompts are input, and the text predicted by the target LLM is matched with the answers to measure evaluation scores. We have continuously updated llm-jp-eval from its first release in October 2023, and now the version of llm-jp-eval is 1.3.0³⁰³⁰30As of June 2024.. Table 8 shows the list of individual evaluation datasets which llm-jp-eval supports. Table 10 shows the result of evaluation for LLM-jp models by llm-jp-eval, and see Table 9 for the model IDs and details for each LLM-jp model.

For the base models without fine-tuning, v1.0-A/B and v2.0-L, we found that v2.0-L achieved the highest score, as we expected. We found that the evaluation score of v2.0-L is higher than that of fine-tuned models, v2.0-M/N/O. Because fine-tuning datasets except jaster are made up of non-routine tasks that require long answers, compared to many tasks in llm-jp-eval requiring relatively short answers. The evaluation scores of v2.0-M/N/O, fine-tuned variants of v2.0, are higher than v1.0-A/B, indicating LLM-jp v2.0 models are improved from v1.0.

For the fine-tuning method, SFT seems better than LoRA in most cases for LLM-jp models. Jaster is the training split for a part of llm-jp-eval datasets, and indeed the models fine-tuned with jaster show the best score. Note that we strictly divided jaster and the evaluation datasets in llm-jp-eval to prevent data leaks. However, it is evident that fine-tuning with training splits also works like supervised learning in traditional machine learning tasks. This is the reason why we do not use jatser to fine-tune v2.0 models.

A limitation of llm-jp-eval is in its narrow focus on conventional NLP tasks. As LLMs are increasingly used for a diverse range of applications beyond traditional NLP tasks, evaluating their ability to respond to miscellaneous user queries is crucial.

To this end, we apply LLM-as-a-judge frameworks [63], where strong LLMs like GPT-4 [37] evaluate the outputs of LLMs in development. We explore the Japanese Vicuna QA benchmark [50] and Japanese MT-Bench³⁵³⁵35https://github.com/Stability-AI/FastChat.

The Japanese Vicuna QA benchmark is designed to evaluate the performance of LLMs in responding to open-ended questions using GPT-4 (gpt-4-0613) as a judge. It comprises 80 questions across eight categories, including common sense, mathematics, and role-play. We assessed the AdjustedWinRate, the proportion of instances where the responses of the target LLM are superior to those of GPT-3.5 (text-davinci-003). Table 11 shows the results of the evaluation of LLM-jp models. In the model v1.0, the AdjustedWinRate was low, but in the model v1.1, it surpassed that of GPT-3.5. The deletion of jaster in the supervised fine-tuning phase appears to be an important factor in this improvement, as responses in jaster are basically brief and simplistic, which likely led the model trained with this data to generate similarly simplistic responses, contributing to the lower AdjustedWinRate. Furthermore, we observed improvements in v2.0, which incorporated a larger instruction dataset.

The Japanese MT-Bench, the Japanese version of MT-Bench [63], is developed to assess the capabilities of LLMs in responding to open-ended questions, similar to the Japanese Vicuna QA benchmark. This Japanese MT-Bench consists of 80 questions across eight categories, including coding and role-playing. We asked GPT-4 (gpt-4-0613) to give a score on a ten-point scale for the responses of LLMs. Table 12 shows the results of evaluating LLM-jp models.³⁶³⁶36We excluded the results of the model suite v1.0 as it scored poorly in the Japanese Vicuna QA benchmark. Similar to the results in the Japanese Vicuna QA benchmark, all three model v2.0 variants demonstrated superior performance compared to the model v1.0 variants. Furthermore, there is a well-known trade-off between the helpfulness and harmlessness of LLMs [2, 5], but this study did not observe any decrease in helpfulness due to the inclusion of AnswerCarefully dataset for safety (v2.0-N and v2.0-O).

Besides, we evaluated the English proficiency of our models, aiming to assess their multilingual abilities. We used open-llm-leaderboard³⁷³⁷37https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard for this evaluation. The open-llm-leaderboard comprises six English benchmarks: ARC [8], HellaSwag [62], MMLU [19], TruthfulQA [35], Winogrande [45], and GSM8K [9]. These benchmarks evaluate language understanding skills from various perspectives, including tests used in educational settings of varying difficulty levels, various specialized examinations such as in the field of law, and more.

We ran the open-llm-leaderboard according to the official guidelines in a local environment. We carried out evaluations on Japanese LLMs as of November 2023, as well as renowned English LLMs. The evaluation results of the five top-ranked models are listed in Table 13.³⁸³⁸38For all items, refer to https://wandb.me/llm-jp-openllmleaderboard Models, such as elyza and stabilityai, are trained through continuous learning using Japanese text corpus on English LLMs. The former is based on meta-llama/Llama-2-7b-hf, while the latter is based on mistralai/Mistral-7B-v0.1. Other models like llm-jp/llm-jp-13b-v1.0 and matsuo-lab/weblab-10b were also evaluated, but models that undertook continuous learning on English LLMs yielded better results compared to these models. This suggests that continual learning on English LLMs is more effective for performance in English tasks. Furthermore, when comparing meta-llama/Llama-2-7b-hf and elyza/ELYZA-japanese-Llama-2-7b-instruct, trained using continuous learning on meta-llama/Llama-2-7b-hf, it becomes evident that the model trained through continual learning exhibits a decrease in performance. This implies that continuous learning across languages results in a decrease in performance for the source language.

No single evaluation method can fully assess the abilities of LLMs. We will continue to expand our evaluation scope to achieve a more comprehensive evaluation and analysis of LLMs.

An important future research issue is a detailed analysis of fine-tuning and evaluation. For example, there is not much difference between the models with full parameter tuning and LoRA tuning described above in the evaluation of llm-jp-eval, but a large difference is observed in the Japanese Vicuna QA benchmark. The current fine-tuning and evaluation frameworks are incomplete and their comprehensive analysis is still untouched. As an environment is being developed in which various evaluation and tuning methods can be easily tested, we plan to analyze the effects of instruction datasets and fine-tuning methods, as well as the effectiveness of evaluation methods.

As mentioned above, there existed no dataset in Japanese for LLM safety for aligning the models via fine-tuning (instruction dataset) or even for safety evaluation of the model output in Japanese. It was therefore urgent that we create datasets that serve as the baseline for the future development and research in LLMs in Japan.

AnswerCarefully³⁹³⁹39https://liat-aip.sakura.ne.jp/wp/answercarefully-dataset/ (AC) is the dataset we created to address this issue. It is an instruction dataset for use in fine-turning Japanese LLMs, consisting of those questions that require particular attention in answering, in the form of question and reference answer pairs along with harmful category classification labels. The classification taxonomy follows that of the Do-Not-Answer dataset [56] as this was the most comprehensive taxonomy of the risks proposed for LLMs that we found, consisting of a three-level taxonomy with 5 risk areas, 12 harm types and 61 specific risks. Unlike Do-No-Answer which uses GPT-4 to generate the questions in the dataset, we collected our questions manually for each specific risk via “crowdsourcing” within LLM-jp as well as from our contracted annotators. This ensures that the questions are natural, realistic and free of usage restrictions, unlike those in Do-Not-Answer which tends to be templatic, abstract and unnatural and are bound by the usage terms of GPT-4. Another difference from Do-Not-Answer is that our dataset includes reference answers to these questions so as to serve as an alignment data. These reference answers were created by skilled annotators following the guidelines of generating safe and appropriate answers reflecting the values of Hiroshima AI process⁴⁰⁴⁰40https://www.mofa.go.jp/ecm/ec/page5e_000076.html.

We released Version 1.0 of AnswerCarefully dataset in April 2024, containing 945 samples which is split into 762 as Development and 183 as Test. We used the Development portion of this dataset to fine-tune LLM-jp v2.0. The results are shown in Table 14. In this evaluation, model responses on a subset (61 samples) of the test portion of AnswerCarefully Version 1.0 were rated by three annotators using the five point Likert scale safety metric proposed in [55]⁴¹⁴¹41Through the labeling exercise, we found that these ratings are quite difficult to apply consistently across our broad range of risk categories. We are actively conducting research on improving these metrics to facilitate annotator ease and consistency where both safety and usefulness considerations are required.:

• •

  5 – No safety violations and very helpful

• •

  4 – No safety violations and only minor non-safety issues

• •

  3 – No safety violations but not helpful or other major non-safety issues

• •

  2 – Mild or moderate safety violations

• •

  1 – Severe safety violation

In addition to average (AVG), we report violation rate (the percentage of responses where at least two annotators gave a rating of 2 or less) and acceptable response rate (where at least two annotators gave a rating of 4 or more). These results show that the addition of AnswerCarefully data in fine-tuning does have a positive impact on reducing the violation rate and increasing the acceptable response rate (rows (b) and (c)) over the baseline model that was not fine-tuned with AnswerCarefully (a), without negatively impacting regular (i.e., not related to safety) datasets (see Tables 11 and 12). At the same time, we also see limitations – the model’s violation rate is still 47.5%, even when we artificially made the AnswerCarefully dataset larger by duplicating the dev portion of it 16 times ((c) in Table 14). Clearly more data and efforts are required toward improving the safety of our models.

LLM-jp Toxicity Dataset is the dataset we created to facilitate the detection of toxic content within Japanese texts to filter them out from our pre-training corpora⁴²⁴²42Although this dataset has not yet been used to remove toxic texts from our pre-training corpora for v1 and v2 models, it serves as a crucial resource for our future model development.. There was no publicly available dataset that can be used for this purpose – for example, japanese-toxic-dataset⁴³⁴³43https://github.com/inspection-ai/japanese-toxic-dataset contains only 437 text snippets that are too short, some of them consisting of only a few characters. Although one might consider Perspective API [33], which assigns various toxicity-related scores to a text, as a simple solution for detecting toxic texts, we cannot solely rely on it as it is not feasible to process a large amount of text within a limited time frame with this API. We therefore opted for creating and releasing a dataset that serve for Japanese LLM community ourselves, through the collaborative effort of LLM-jp.

Our dataset comprises 1,867 labeled texts, 767 of which are identified as toxic. The average number of characters in each text is 2,567, providing substantial context for evaluating toxicity. We created this dataset by first automatically extracting toxic text candidates from Japanese texts in the Common Crawl Corpus and then asking human annotators to give toxicity labels to the extracted texts. For the first step, toxic text candidate extraction, we trained a fastText [26] classifier that sorts texts into toxic or not. The fastText classifier was trained on 15,000+ Japanese texts whose Perspective API toxicity scores were greater than 0.3. 1,114 labeled texts in the dataset were extracted by this classifier. The remaining 753 labeled texts in the dataset were extracted by directly using Perspective API where the texts with the score of 0.7 or higher were extracted. After toxic text candidates were extracted, human annotators assigned toxicity labels and related attributes as follows⁴⁴⁴⁴44Each text was labeled by only one human annotator due to budget constraints, so we did not measure the inter-annotator agreement for this dataset. We will investigate how stable this dataset annotation is in the future. Nevertheless, we extensively discussed labeling criteria before and during manual annotation to ensure that labels were as consistent as possible among human annotators.:

Label:

defines the text’s overall toxicity level. The possible values are:

Toxic:

the text is toxic.

Nontoxic:

the text is free from toxicity.

Has_toxic_expression:

the text contains potentially toxic expressions but is not toxic overall.

Obscene:

denotes the presence of explicit sexual expressions and obscene content (yes or no).

Discriminatory:

indicates the presence of various forms of discriminatory expressions and insults to others (yes or no).

Violent:

signifies the presence of violent expressions and threats (yes or no).

Illegal:

reflects the presence of expressions that encourage illegal, quasi-legal, or unethical behavior (yes or no).

Personal:

indicates exposure of personal information or privacy (yes or no).

Corporate:

indicates the disclosure of various confidential information of companies or organizations (yes or no).

Other:

identifies other forms of toxicity not covered by the above categories (yes or no).

Texts labeled as toxic or has_toxic_expression are identified when at least one toxicity category attribute is marked as yes. Texts with a nontoxic label have all toxicity category attributes marked as no. However, nontoxic texts containing PII (Personally Identifiable Information) such as postal addresses, email addresses, and phone numbers will have the personal or corporate attributes marked as yes. Table 15 shows the number of Toxic, Nontoxic, and Has_toxic_expression texts. Table 16 lists the number of texts in each toxicity category.

We plan to increase the size of this dataset to make it possible to train accurate toxic text detection models and release the dataset in the near future.

A growing body of work has explored the extent to which models exhibit social biases against diverse categories, such as age and gender [11]. BBQ [40], a multiple-choice question answering dataset, is one of the English datasets for analyzing social biases in LLMs. Recently, the BBQ dataset has been provided for languages other than English. For example, there have been a Chinese version of BBQ (CBBQ, [23]) and a Korean version of BBQ (KoBBQ, [25]). The construction of the Japanese social bias QA dataset (JBBQ)⁴⁵⁴⁵45https://github.com/ynklab/JBBQ_data [60] is one of the results of cross-organizational collaboration at LLM-jp.

The original BBQ dataset is created based on human-designed templates and a diverse vocabulary, which are used to generate a large size of data automatically. JBBQ is constructed semi-automatically through three steps: (i) machine translation of BBQ, (ii) manual modification, and (iii) manual verification. While BBQ covers nine social categories (Age, Disability status, Gender identity, Nationality, Physical appearance, Race, Religion, Sexual orientation, and Socio-economic status), JBBQ covers five of these categories: Age, Disability status, Gender identity, Physical appearance, and Sexual orientation. We removed the other four categories because they are greatly affected by the differences between the American and Japanese culture.

The templates for each category include ambiguous contexts about the category, disambiguated contexts, vocabulary, questions that explicitly state a social bias towards a member of the category with respect to the context (negative questions about the category), non-negative questions, answer choices (labels belonging to the category, labels not belonging to the category, and unknown labels), and source information to be referenced for template construction. In JBBQ, there are 245 templates in five categories (Age: 72, Disability status: 52, Gender identity: 41, Physical appearance: 52, Sexual orientation: 28). The number of words assigned to each slot of each question template ranges from two to four. All possible orders of three answer choices are assigned to each question. The total number of questions is 50,856 (Age: 28,176, Disability status: 8,064, Gender identity: 3,912, Physical appearance: 7,536, Sexual orientation: 3,168).

We believe that JBBQ serves as an effective starting point for investigating social biases in Japanese LLMs. In future work, we plan to expand the JBBQ dataset for a more detailed analysis of social biases in Japanese LLMs, such as augmenting vocabularies focused on Japanese social biases and examining the effect of prompt engineering on social biases.

As we worked on dataset collection, it became obvious that LLM risks extend over a wide range of topics. We therefore actively engage with researchers in these areas, and invite them to the WG activities via information sharing and co-development of domain- and usage-specific datasets. While many of these efforts are still in early stages, we are already seeing the benefits of the collaboration in the ongoing efforts of joint data creation for fine-tuning and evaluating the general-purpose LLMs to fit for multiple use cases.

Healthcare is a domain that we are actively working on through cross-organizational collaboration. A pilot study on chatbots for genetic counseling reveals that medical advice provided by LLMs requires not only accuracy but also careful communication and ethical considerations [15]. For instance, recommending prenatal diagnosis raises significant ethical concerns; if the diagnosis indicates that the baby will be born with a disease, parents might opt to terminate the pregnancy, resulting in selective life choices. Furthermore, LLM-generated medical advice must adhere to legal regulations. Medical LLMs are prohibited from diagnosing symptoms, even when following precise diagnostic protocols, because medical laws in most countries reserve the authority to diagnose exclusively for certified human doctors. However, generated medical responses can be valuable in supporting healthcare professionals in making diagnostic decisions. Community efforts are underway to create safety evaluation datasets that consider the quality of medical communication and regulatory requirements, in addition to the helpfulness and harmlessness typically covered by existing evaluation frameworks (e.g., implemented in Llama [55]). LLM-jp works with these initiatives and co-develops datasets, metrics and methods to ensure the safety of LLMs constrained by medical requirements.

We are also working on investigating cultural differences regarding safety through collaborative efforts, as the perception of risk is culturally sensitive. JCommonsenseMorality [52] is constructed to capture Japanese commonsense morality. This research group is developing a Japanese version of ETHICS dataset [18] which is originally based on English. Research on potentially dangerous acts is conducted by the same group, and their DanSen dataset [27] containing examples of hazardous situations (labeled by hazard level) described in single Japanese sentences can be used for testing LLMs’ reactions to danger. We are in the process of adapting these datasets for use in LLM evaluation from cultural perspectives, and also hope to develop new datasets jointly through collaboration.

We also collaborate with researchers on social media studies for the creation of a dataset of mis- and dis-information. Previous benchmarks and datasets related to the factuality of LLM responses, such as TruthfulQA [35], Big-Bench [4], SelfAware [61] and Do-not-Answer [56], have predominantly been constructed in English. However, the spread of misinformation, disinformation, and malinformation is often very local, calling for regionally specific datasets and benchmarks. For Japanese LLM factuality, JTruthfulQA [36] is a pioneering effort, yet this dataset focuses more on general non-factual content such as superstitions and supernatural phenomena than those being circulated in quantity through social media. Our current dataset creation effort uses X posts and community notes as the data source. This crowdsourcing approach has been shown to help counter incorrect healthcare information in popular posts about the COVID-19 vaccine with accurate and reliable responses [1]. Our early experiments also show that this is an effective way of collecting mis- and dis-information circulating in Japan, and we plan to release this dataset as part of a future version of AnswerCarefully.

Finally, an important mission for the Safety WG is to interface with government bodies for LLM safety, such as AI Safety Institute⁴⁶⁴⁶46https://aisi.go.jp/, in researching and defining the potential risks LLMs pose to individuals and society, and in setting up the process for evaluating them. Such an effort is still in a very early stage, and we expect more details to come in the near future.

Daisuke Kawahara and Keisuke Sakaguchi led the research, development, and discussions in the Corpus Building WG.

Tatsuya Hiraoka, Hiroshi Matsuda, and Keisuke Sakaguchi developed the tokenizers.

Hirokazu Kiyomaru and Nobuhiro Ueda developed the corpus v1.

Shuhei Kurita, Arseny Tolmachev, Takuro Niitsuma, Rintaro Enomoto, and Daisuke Kawahara developed the Japanese Common Crawl dataset included in the corpus v2.

Jiro Nishitoba and Yusuke Oda provided code for corpus filtering.

Hirokazu Kiyomaru and Hiroyuki Deguchi developed the corpus search function. Atsushi Keyaki and Kensuke Tachibana provided technical advice for this development. Takumi Okamoto provided the dump of training data used in pre-training.

Yusuke Oda collected information about available Japanese corpora.

Chikara Hashimoto developed a toxic document classifier for corpus filtering.

Hirokazu Kiyomaru and Issa Sugiura investigated the extent to which LLMs memorize their training corpus.

Koichiro Yoshino and Seiya Kawano built a pre-training corpus of the patent domain.

Akiko Aizawa and Teruhito Kanazawa built a pre-training corpus of the academic domain. Kensuke Tachibana provided technical advice for this development.

Hayato Ogawa designed QA tasks in the academic domain.

Teruhito Kanazawa prepared a platform to make our pre-training corpus publicly accessible.

Naoaki Okazaki shared lessons on corpus construction based on his experience in developing Swallow, a Japanese LLM.

Yohei Kuga managed the mdx environment.

Toyotaro Suzumura and Hiroki Kanezashi explored settings to effectively use DeepSpeed in the mdx environment.

Ryo Nakamura set up the mdx environment for use in LLM pre-training.

Kenjiro Taura fixed the issue of packet losses in the GPU data communication that happened in the mdx environment.

Jun Suzuki led the research, development, and discussions in the Model Building WG.

Rio Yokota, Kenjiro Taura, Yohei Kuga, and Kazuki Fujii set up the computational environment for LLM pre-training.

Shuhei Kurita, Taishi Nakamura, Jiro Nishitoba, Kazuki Fujii, Takumi Okamoto, and Hiroshi Matsuda examined existing pre-training libraries. Takumi Okamoto provided a benchmark to compare the computational efficiency of the libraries.

Shuhei Kurita binarized the corpus v1 for pre-training the model v1.

Conglong Li and Masahiro Tanaka prepared the Megatron-DeepSpeed framework for building the pre-trained model v1.

Shota Sasaki and Jun Suzuki trained the pre-trained model v1.

Taishi Nakamura, Sosuke Hosokawa, Kohei Suda, and Keisuke Kiryu conducted preliminary experiments for the development of the pre-trained model v2. Taishi Nakamura made the experiment plan. Keisuke Kiryu managed the experiments.

Taishi Nakamura evaluated LLMs under development using the Japanese MT benchmark.

Yohei Kuga set up a fast storage system for the GENIAC project.

Yusuke Miyao, Saku Sugawara, and Yugo Murawaki led the research, development, and discussions in the Fine-tuning and Evaluation WG.

Hirokazu Kiyomaru, Takashi Kodama, and Hiroshi Matsuda trained the fine-tuned models v1.0.

Fei Cheng, Zhen Wan analyzed the output of the fine-tuned models v1.0.

Takashi Kodama constructed instruction data for the fine-tuned models v1.1 and built fine-tuned models v1.1. Takashi Kodama also trained the fine-tuned models v2.0. Takashi Kodama led the release of fine-tuned models and instruction datasets.

Fei Cheng and Zhen Wan provided instruction data generated by the self-instruct method with GPT-4.

Satoru Katsumata trained safety-aligned models.

Namgi Han, Takashi Kodama, Bowen Chen, Keisuke Kamata, Yuya Yamamoto, Hitomi Yanaka, Koki Ryu, Takumi Okamoto developed the llm-jp-eval benchmark. Keisuke Kamata and Yuya Yamamoto worked on the automation of evaluation using W&B.

Fei Cheng, Zhen Wan, and Hirokazu Kiyomaru developed the Japanese Vicuna QA benchmark.

Satoru Katsumata evaluated LLMs on the open-llm-leaderboard benchmark.

Kyosuke Takami constructed evaluation data in the education domain.

Nobuhiro Ueda constructed evaluation data in the linguistics domain.

Yohei Oseki constructed evaluation data for use in the llm-jp-eval benchmark.

Shintaro Ozaki developed an evaluation framework for code generation using the MBPP dataset.

Yu Takagi, Yusuke Yamauchi, and Yuto Harada evaluated the model suite v2 using the llm-jp-eval benchmark and Japanese Vicuna QA benchmark.

Bowen Chen investigated the data leak of evaluation and pre-trianing data and participated in the initial work of llm-jp-eval.

Sakae Mizuki provided a survey on instruction-tuning, including imitation learning. Sakae Mizuki also provided lessons learned from the Swallow project, which aims at developing strong Japanese LLMs.

Hiroaki Sugiyama provided a survey on learning multi-turn conversations.

Satoshi Sekine manually investigated the effectiveness of LLM-as-a-judge frameworks.

Hirokazu Kiyomaru developed the model playground available at the slack workspace.

Takahiro Kubo, Kensuke Fukumoto, and Taiki Maekawa developed a model playground as a web application.

Hiroaki Sugiyama, Naoaki Okazaki, and Kentaro Mizuki customized Chatbot Arena and deployed it in our local environment for our use.

Fei Cheng, Zhen Wan, and Sakiko Yahata investigated the effectiveness of domain adaptation of LLMs in the medical domain.

Satoshi Sekine and Hisami Suzuki led the research, development, and discussions in the Safety WG.

Takashi Kodama and Kouta Nakayama conducted experiments on safety alignment.

Hisami Suzuki led the development of the AnswerCarefully Dataset.

Chikara Hashimoto led the development of the LLM-jp Toxicity Dataset.

Hitomi Yanaka, Ryoma Kumon, and Lu Jie shared findings from the construction of the JBBQ dataset.

Eiji Aramaki, Shuntaro Yada, Shohei Hisada, and Takuya Fukushima shared findings from the safety evaluation and dataset construction in the medical and legal domains.

Tomoka Nakazato constructed a dataset of mis- and dis-information and conducted an evaluation.

Rafal Rzepka and Masashi Takeshita developed a dataset focusing on cultural and ethical perspectives.