Does Incomplete Syntax Influence Korean Language Model?
Focusing on Word Order and Case Markers

Syntactic information is crucial for humans to understand sentence meanings accurately (Chomsky, 2002). One prominent syntactic element, word order, has been a central topic in recent Language Model (LM) research. Studies have explored various aspects of word order: its significance to LMs (Pham et al., 2020; Sinha et al., 2020), methods to improve model performance via word order adjustments (Sinha et al., 2021), and insights into how LMs progressively learn human language (Abdou et al., 2022).

Korean follows a canonical Subject-Object-Verb (S-O-V) word order. However, the presence of case markers, a type of postposition, allows Korean to exhibit relatively flexible word order (Lee & Im, 1997; Yeon & Brown, 2013; Sohn, 2005). Case markers denote the grammatical roles of constituents within a sentence, like their function as a subject or object. Conversely, due to its canonical word order, Korean often experiences omissions of case markers. Our aim is to verify whether this flexibility is reflected in Korean LMs.

Figuree 1 exemplifies the flexibility of a Korean sentence, “나비가 꿀을 마신다,” in terms of word order and case markers. Even with word order changes (Arrow a), the case markers ‘가’ and ‘을’ identify ‘나비’ and ‘꿀’ as subject and object, respectively. Conversely, in S-O-V order (Arrow b), subject and object are identifiable by position, even without case markers. Of course, the freedom from case markers and word order comes with limitations. Sentences lacking case markers or with rearranged word order might appear unnatural or have the potential to be interpreted differently from the original sentence. More detailed information about Korean word order and structure is as follows:

• •

  Korean typically follows an SOV (Subject-Object-Verb) order.

• •

  Major reorderings occur among full phrases (e.g., noun phrase arguments) and usually take place before the verb.

• •

  Moving the verb to an earlier position is much rarer.

• •

  Splitting a noun phrase argument (e.g., placing an adjectival modifier before the noun) is also much rarer.

• •

  Naturally dropping case markers is more common with subjects than with objects (or vice versa, depending on the context).

We aimed to determine if the syntactic flexibility inherent to the Korean language is mirrored in Korean LMs. Leveraging these unique linguistic characteristics, our objective was to improve the performance of Korean LMs. To this end, we established a methodology for generating Syntactically Incomplete Korean (SIKO) data and subsequently constructed a comprehensive dataset. Further, we evaluated the effectiveness of the SIKO dataset as a resource for fine-tuning LMs, focusing on its practical applicability in enhancing language processing capabilities.

We designed various experiments using the SIKO data. We employed four fundamental tasks used for assessing LM performance: Text Classification (TC), Natural Language Inference (NLI), dialogue topic classification, and dialogue summarization. For the experiments, we used the well-established KLUE benchmark (Park et al., 2021) and the AI-hub data²²2It is a data platform operated by the Korea National Information Society Agency (NIA). It collects and processes artificial intelligence data in six fields and makes it publicly available. https://www.aihub.or.kr/ released by national authorities. This compilation includes a wide range of Korean language expressions, from news headlines to conversational texts, thereby expanding our experimental scope to cover the extensive usage of Korean. Additionally, we examined the characteristics and performance across three different Large Language Models (LLMs) with varying sizes and structures, to uncover both commonalities and unique aspects in processing syntactic phenomena.

In our study, we observed that expressions featuring syntactic incompleteness are prevalent in real-world Korean language applications. We were able to confirm through inference experiments using SIKO data that LLMs are responding to such syntactic incompleteness. Our analysis reveals that the SIKO dataset surpasses representative data augmentation methods in efficacy, especially for fine-tuning purposes. Significantly, it demonstrates superior capability in improving the handling of inputs that are syntactically incomplete, a common phenomenon in typical Korean usage.

The contributions of our study are as follows:

• •

  This study is the first to explore syntactically incomplete Korean sentences.

• •

  We constructed the Syntactically Incomplete Korean (SIKO) dataset.

• •

  Through downstream testing with SIKO, we verified that the Korean LLM exhibits flexibility when dealing with syntactically incomplete inputs.

• •

  Models tuned using SIKO demonstrated improved performance in handling incomplete data commonly found in Korean.

We explored the frequency of case marker omissions and word order changes in practical Korean usage. For this investigation, we took 1,000 samples each from two tasks within the KLUE benchmark and conversational data from AI-hub. We assessed the presence of omitted case markers and deviations in word order, and determined whether such phenomena led to ambiguity or challenges in interpreting the meaning of the sentences.

Additionally, we restored omitted case markers and corrected word order in 300 samples for each task. This process allowed us to identify the number of case marker positions, assess omission rates across datasets, and evaluate the scale of word order modifications. The findings are showcased in Table 1. All tasks described in this section were manually performed, and details about the individuals involved are provided in the Experiment Settings Section ( 5.4).

The rate of case marker omission was calculated by comparing the number of omitted case markers to the total number of case markers after their restoration. The word order correction rate was determined by taking the word-level edit distance between the original data and the corrected word order data, then dividing by the total word count in the data.

From this analysis, we ascertained that the omission of case markers and alterations in word order are common in ordinary Korean usage. Specifically, case marker omission occurred in 97.4% of KLUE-TC instances and non-canonical word orders were found in 35% of AI-hub dialogues. The scale of these phenomena is underscored by their extensive occurrence: an average omission rate of 80.1% for case markers in KLUE-TC and an average word order correction rate of 45.35% in KLUE-TC.

Notably, even with such syntactical incompleteness, the sentences largely retained their clarity of meaning. In dialogue data, there were instances where interpretation was challenged due to case marker omissions. However, it’s important to note that these were extracted from conversations, potentially lacking some contextual or background information. This investigation reaffirms that while Korean showcases syntactic flexibility, it remains bounded to ensure semantic clarity.

In Section 3, we demonstrated Korean’s extensive syntactic flexibility and its frequent occurrence in actual use. We aimed to determine if Korean LMs reflect this variability. To this end, we generated datasets with deliberate syntactic incompleteness. This approach occasionally led to reductions in syntactic detail and interpretability.

We randomly sampled 22,000 instances from three data types (KLUE-TC, KLUE-NLI, AIhub-dialogue) and developed five variants of SIKO data. Of these, 2,000 instances (and their SIKO derivatives) served as the test set in Section 6.1, while 20,000 (with their SIKO derivatives) were used for fine-tuning in Section 6.2. Additional details on data statistics, prompts, and reviewer information are documented in Appendix B.

In this study, we established that Korean’s syntactic flexibility are indeed mirrored in LMs. We also demonstrated that data generated via the SIKO method enhances comprehension of ordinary Korean. The scope of our research was intentionally limited to Korean, constrained by our linguistic expertise and the experimental scope. Yet, it’s noteworthy that numerous languages, including Japanese, certain Turkic languages, Mongolian, and Hungarian, possess case markers or exhibit flexibility in sentence structure, like Japanese, Russian, and Latin. We anticipate that our findings will significantly benefit LMs developed for these languages.

Our methodology leveraged training data infused with syntactic noise to boost model performance. By experimenting with various rates of data augmentation, we ascertained the necessity for moderation in augmentation to avoid detrimental effects. Future studies will aim to pinpoint the optimal augmentation level, evaluate potential drawbacks, and assess the method’s utility across different tasks.

We sought to determine whether Korean LMs reflect the inherent syntactic flexibility of the Korean language, especially regarding case marker usage and word order. Our initial findings confirmed the frequent use of incomplete case markers and word orders in Korean through publicly available benchmark datasets. To delve deeper, we introduced the Syntactically Incomplete Korean (SIKO) dataset. Through SIKO, it became evident that Korean LMs can flexibly handle syntactically incomplete inputs. Fine-tuning with SIKO enhances this capability to process common incomplete syntactic forms in Korean. Moreover, the dataset’s low construction cost, coupled with significant performance enhancements, solidifies its standing as an effective data augmentation technique.

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [No.2022-0-00641, XVoice: Multi-Modal Voice Meta Learning]