MTFinEval:A Multi-domain Chinese Financial Benchmark with Eurypalynous questions

Our research is dedicated to strengthening the capacity of these models to tackle economic challenges fundamentally and systematically.

The MTFinEval dataset comprises 360 university economics questions, spanning six major sub-topics: macroeconomics, microeconomics, accounting, management, e-commerce, and strategic—management. Each subtopic includes single choice, multiple choice, and true or false questions. All questions and answers are manually extracted from college textbooks\parenciteLi2023TextbooksAA and exam papers to ensure they are foundational and introductory. The data collection process has been meticulously scrutinized through a series of systematic checks. We began by manually entering the paper questions into CSV files, ensuring the completeness of each entry.

Subsequently, to verify the accuracy of the questions and answers, we enlisted the expertise of six economists, each with high proficiency in various subfields. They were tasked with reviewing the clarity of the descriptions and the correctness of the answers, for which they were compensated at a rate of $3 per question. In cases where a question was deemed questionable during the expert review stage, a collective discussion was held among all experts to determine the correct answer and to decide if the question should be omitted. The types and numbers of questions for all subtopics are shown in Figure 1.

For problem type $T$, given the large model $\mathcal{M}$, the corpus set $d$ used for training and fine-tuning the large model, input the question q, and the large model returns the answer $a$\parenciteNarayan2018DontGM.

In the zero-shot scenario for true or false statements, the objective function is defined as follows:

$\displaystyle r=\underset{a\in\{0,1\}}{\arg\max}P(a|q^{T};\mathcal{M},d^{T})$

(1)

Similarly, for single-choice questions in the zero-shot scenario, the objective function is specified as:

$\displaystyle r=\underset{a\in\{A,B,C,D\}}{\arg\max}P(a|q^{T};\mathcal{M},d^{T})$

(2)

Lastly, for multiple-choice questions in the zero-shot scenario, the objective function is:

$\displaystyle r=\underset{a\in\{A,B,C,AB,AC,BC,ABC...\}}{\arg\max}P(a|q^{T};\mathcal{M},d^{T})$

(3)

On the surface, choices and judgments are both classify problems. Fundamentally, for the decoder only model, the underlying task is a generation task. In the training of LLMs, While pursuing to reduce cross entropy, maximum likelihood estimation is also used to reproduce similar sentences in the corpus. Therefore, although only the selection of the judgment problem, the bottom layer still recalls the relevant corpus content during the training of the LLMs.

For problem type T, given the large model M,the corpus set d used for training and finetuning the large model, input the question q, and the large model returns the answer a. In the zero-shot scenario for judge questions the objective function is define To simulate the most direct use cases effectively, we introduce a brief prompt to the question. This prompt includes a description of the question type, all under zero-shot conditions\parenciteBrown2020LanguageMA. The prompts for the three question types are similiar with follows\parenciteWei2022ChainOT\parencitexcot\parencitelow_resource_template\parencitehlt_mt\parenciteTai2023ExploringCS: multiple choice: You are a financial knowledge expert, please combine your financial knowledge to answer the following multiple choice, note that there is not only one answer to multiple choice, to think and reason whether each option is correct but do not output thinking process, directly return the option letter in markdown format, each option in the answer is separated by a space. When it comes to judgment questions, the LLM is instructed to provide answers limited to ”yes,” ”wrong,” or ”do not know.”. Subsequently, key words from these answers are extracted and mapped onto these three specific options. In the realm of traditional economics, LLMs are typically based on the Llama series\parenciteTouvron2023Llama2O\parenciteDubey2024TheL3 for processing. To promote decentralization, the models participating in this evaluation are exclusively open-source models that were released in the year 2024.

Regardless of little changes in the LLMs’ architecture, the diverse answers provided by various LLMs can reveal the extent and quality of the training data used. This insight can then be used to reasonably gauge the model’s capability level. Specialization in Subjects: Certain models demonstrate expertise in specific domains. For instance, the 01-ai/Yi-1.5-9B-Chat-16K\parenciteYoung2024YiOF model excels in accounting, e-commerce, and strategic management, suggesting a rich dataset in these areas. This proficiency likely stems from its capacity to interpret complex financial data, manage online platforms efficiently, and formulate strategic business plans. Macroeconomic Understanding: The THUDM/glm-4-9b-chat\parenciteZeng2024ChatGLMAF\parenciteBommasani2021OnTO model’s elevated score in macroeconomics indicates a robust grasp of economic trends and policy impacts. This proficiency may be due to its exposure to a diverse range of economic data, enabling precise assessments. Strategic Management Insight: The google/Gemma-2-9B-IT model’s high strategic management score reflects its potential to support business planning. Its ability to integrate information from various sources to offer valuable strategic insights is likely the reason for this performance. Overall Performance: The Qwen/Qwen2-7B-Instruct\parenciteBai2023QwenTR\parenciteYang2024Qwen2TR model’s lead in overall performance with a 63.33 score signifies its emergence as a leading contender in the field of financial language model technology. Its comprehensive score not only reflects its proficiency in individual subjects but also suggests a robust and well-rounded training approach that has equipped it to handle a variety of economic analyses. This model’s performance is indicative of its potential to become the new baseline, or pedestal\parenciteNori2023CanGF, for financial LLMs\parenciteChen2023TigerBotAO, possibly outperforming or replacing previous models such as Llama\parenciteCui2023EfficientAE in certain applications. Reflection on Weaker Performances: Models like the meta-llama/Meta-Llama-3-8B-Instruct, which scored lower overall, may have been trained on less diverse or pertinent data for certain subjects, or they may have architectural limitations hindering effective processing and analysis of economic information. Implications for Model Improvement: To enhance performance in specific subjects, models should be trained with more domain-specific data. Moreover, ongoing updates and refinements to the model architecture, informed by performance data, can further augment capabilities. In conclusion, LLM performance in economic subjects mirrors the quality of their training data and architecture. While specialized models are better suited for particular applications, a well-rounded model like Qwen/Qwen2-7B-Instruct serves as a multifaceted tool for a range of economic analyses. Ongoing refinement of training data and model architecture is essential for advancing their capabilities in economics.