CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Models
Abstract
Due to the rapid advancements in multimodal large language models, evaluating their multimodal mathematical capabilities continues to receive wide attention. Despite the datasets like MathVista proposed benchmarks for assessing mathematical capabilities in multimodal scenarios, there is still a lack of corresponding evaluation tools and datasets for fine-grained assessment in the context of K12 education in Chinese language. To systematically evaluate the capability of multimodal large models in solving Chinese multimodal mathematical problems, we propose a Chinese Multi-modal Math Skill Evaluation Benchmark, named CMMaTH, contraining 23k multimodal K12 math related questions, forming the largest Chinese multimodal mathematical problem benchmark to date. CMMaTH questions from elementary to high school levels, provide increased diversity in problem types, solution objectives, visual elements, detailed knowledge points, and standard solution annotations. We have constructed an open-source tool GradeGPT integrated with the CMMaTH dataset, facilitating stable, rapid, and cost-free model evaluation. Our data and code are available.
UTF8gbsn
CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Models
Zhong-Zhi Li1,2††thanks: Equal Contribution , Ming-Liang Zhang1,211footnotemark: 1, Fei Yin1,2, Zhi-Long Ji3, Jin-Feng Bai3, Zhen-Ru Pan3, Fan-Hu Zeng1,2, Jian Xu1,2, Jia-Xin Zhang1,2, Cheng-Lin Liu1,2††thanks: Corresponding Author School of Artifcial Intelligence, University of Chinese Academy of Sciences1 MAIS, Institute of Automation of Chinese Academy of Sciences2, Tomorrow Advancing Life3 {lizhongzhi2022, zhangmingliang2018}@ia.ac.cn, {jizhilong, baijinfeng, panzhenru,}@tal.com, {fyin, liucl}@nlpr.ia.ac.cn
1 Introduction
Large language models(LLMs) excel in various language tasks, while multimodal models effectively handle visual-language problems. They advance natural language processing and computer vision fields, providing powerful solutions for complex tasks. Multimodal large models demonstrate potential as versatile solvers for multimodal problems.
The systematic evaluation of large models’ performance across various mathematical reasoning scenarios has been a subject of extensive research. GSM8K and MATHCobbe et al. (2021); Hendrycks et al. (2021b) assessed the ability in multi-step mathematical reasoning by constructing a high-quality set of elementary school math word problems or various competition mathematics problems. By collecting a diverse set of mathematical problems containing both textual and visual components, Lu et al. (2023); Wang et al. (2024); Zhang et al. (2024b) systematically evaluated the ability of large multimodal models to perceive visual elements and solve corresponding multimodal problems. Shi et al. (2023) constructed a multilingual mathematical reasoning dataset, MGSM, for evaluating the LLM reasoning ability in multilingual environments.
However, in non-English multimodal contexts, especially in Chinese scenarios, there is still a lack of sufficiently detailed and diverse benchmarks for assessing mathematical abilities. To assess the capability of large language models in non-English contexts, Huang et al. (2023) and Zhang et al. (2024a) constructed multidisciplinary Chinese question answering datasets C-Eval and CMMMU to evaluate the knowledge and reasoning abilities of multimodal large models. However, C-Eval lacks evaluation in multimodal contexts, while CMMMU’s dataset has relatively low diversity, consisting of only 540 questions.
![Refer to caption](x1.png)
Existing Math benchmarks for answer evaluation can be categorized into two types:Rule-based Cobbe et al. (2021); Hendrycks et al. (2021b); He et al. (2024) and API-based methods Lu et al. (2023); Zhang et al. (2024b); Hendrycks et al. (2021a). API-based methods are very costly and time-consuming, and they often result in unstable and inconsistent evaluation results. Rule-based methods, on the other hand, struggle to handle highly diverse contents of benchmarks. Also, it is difficult to maintain handcrafted rules for dynamically updated benchmarks. Current multimodal math benchmark evaluations often resort to multiple-choice or true/false question formats, using rules or API-based LLM to extract options for assessing answers.
Based on above considerations, we propose a new multimodal mathematical benchmark CMMaTH. Compared to previous benchmarks, our benchmark demonstrates greater diversity, increased depth of reasoning, and finer-grained knowledge annotation for multimodal models to grasp different levels and types of knowledge. We provided and open-sourced a lightweight answer comparator called GradeGPT, designed to compare the consistency between outputs from different LLM/LMMs and standard answers, thus avoiding expensive evaluation costs. Leveraging the CMMaTH dataset and GradeGPT tool, we evaluated mainstream open-source and commercial multimodal large models in Table 4, reporting comprehensive evaluation results along with extensive case analyses. In summary, our paper makes the following contributions:
-
•
We introduce the largest high-quality Chinese multimodal mathematics benchmark with the most detailed annotation granularity to date. We also provide an English version of this dataset. The CMMaTH dataset is a dynamically maintained and will be periodically updated.
-
•
Compared to previous multimodal mathematical benchmarks, our dataset exhibits great depth of reasoning and diversity. Our benchmark simulates more realistic educational Q&A scenarios, encompassing a wider variety of question types and answer formats. Additionally, we annotate each question with detailed knowledge points and corresponding skills to evaluate the mastery level of current large models.
-
•
We build an evaluation assistant named GradeGPT on the CMMaTH dataset, which allows for comparing the proximity of model responses to standard answers and assessing the correctness of results and processes. GradeGPT features lightweight open-source characteristics, avoiding the instability and high costs associated with commercial models.
-
•
We conduct a systematic evaluation of existing mainstream multimodal large models, quantitatively and qualitatively comparing with existing models.
2 Related Work
2.1 Assessment of mathematical abilities
To evaluate the performance of large models in mathematical reasoning and examine hallucinations during the reasoning process, numerous benchmarks have been proposed for evaluating the mathematical reasoning capabilities of large models. GSM8KCobbe et al. (2021) is the first and most widely used mathematical dataset used for large model math evaluation, consisting of 1k math word problem test samples and corresponding answers. The MATHHendrycks et al. (2021b) dataset, in comparison to GSM8K, presents a greater challenge in terms of reasoning difficulty. This dataset demands a more profound understanding and intuition in various mathematical domains such as Algebra, Number Theory, and Geometry. MathVistaLu et al. (2023) is the first dataset used to evaluate the multimodal mathematical capabilities of large models, but it has relatively simple reasoning depth. MATH-VISIONWang et al. (2024) has richer visual elements and deeper reasoning difficulty. MathVerseZhang et al. (2024c) constructed several subsets of datasets to assess whether existing multimodal large models can truly understand mathematical abstract forms.
The CMMaTH Benchmark, in comparison to existing works on the evaluation of mathematical proficiency, places a greater emphasis on the analysis of mathematical abilities within the context of the Chinese language. The data distribution of the CMMaTH dataset more closely aligns with the actual distribution found in K12 educational settings, and it provides detailed annotations of mathematical knowledge points to facilitate the assessment of models’ mastery of knowledge and skills.
2.2 Large Model Evaluation Tool
Due to their strong generalization capabilities and extensive world knowledge, large language models have achieved outstanding results in tasks such as machine translationZhu et al. (2023), question answeringKamalloo et al. (2023), dialogueDuan et al. (2023) and so on by generating text. Evaluating the comprehensive abilities of large models, such as clarity, adherence to instructions, comprehensiveness, formality, and mathematical reasoning ability, has received widespread attentionKe et al. (2023). Currently, many works opt to use powerful commercial model APIs, such as GPT-4, to assist in evaluating the comprehensive abilities of large models. For instance, MathVistaLu et al. (2023) and GeoEvalZhang et al. (2024b) use GPT-4’s API to extract correct answers for evaluation. These methods face several challenges: they are costly and time-consuming, and they struggle to keep up with rapid model iterations. Besides, these methods face challenges in terms of consistency and reproducibilityWang et al. (2023a); Ke et al. (2023).
Recent methods have proposed using metrics such as BERT scoreZhang et al. (2020) or MAUVEPillutla et al. (2021) for evaluation. However, the numerical indicators produced by these methods are difficult to interpret when it comes to the erroneous responses generated by LLM. PandaLM and CritiqueLLM Wang et al. (2023b); Ke et al. (2023) are similar to our work. They proposed a fine-tuning method based on open-source LLMs, distilling the evaluation capabilities of GPT-3.5 into a series of smaller open-source models. However, they are focused on the automated evaluation of more general text generation tasks, while we are targeting the automated evaluation of responses from large models for multimodal mathematics problems.
Unlike PandaLMWang et al. (2023b) trying to evalution relative conciseness, clarity and so on, our evaluation model, GradeGPT, is a dataset-oriented answer comparator that can provide specific reasons based on the standard answer and a model’s response. We distilled the answer comparison capability of GPT-4 using the Cross-Lingual Judge-of-Chain method and enhanced GradeGPT’s answer discrimination ability.
3 CMMaTH Dataset
Statistic | Number |
Total questions | 23856 |
- multiple-choice questions | 18191 |
- Free-form questions | 5665 |
- Questions in the testmini set | 1000 |
Single-choice questions | 13706(75.3%) |
- Proportion of answers A | 2694(14.8%) |
- Proportion of answers B | 3903(21.4%) |
- Proportion of answers C | 3961(21.7%) |
- Proportion of answers D | 3148(17.5%) |
Multiple-choice & Multi-turn questions | 4485(24.7%) |
knowledge point number | 2299 |
Levels | 5 |
Visual Subjects | 13 |
Maximum question length | 593 |
Minimum question length | 6 |
Average question length | 75.1 |
Grade Distribution Elementary(1-6) | 800 |
Junior(7-9) | 5082 |
Senior(10-12) | 17972 |
3.1 Overview of CMMaTH
![Refer to caption](x2.png)
We selected diverse multimodal mathematical problems from a vast pool of K12 educational questions, comprising 23856 items across 13 visual themes, 5 difficulty levels, and encompassing 150 types of knowledge points. More detailed statistical data can be found in Table 1.
For the convenience of evaluation, we provide a miniaturized test set of CMMaTH, called CMMaTH-testmin, containing 1500 samples. Testmin retains the diversity of the CMMaTH dataset and shows similar overall performance to the entire CMMaTH dataset. Evaluators can conduct quick tests and generate preliminary analyses based on CMMaTH-testmin.
UTF8gbsn
Image Type | #Num | Image Type | #Num | Image Type | #Num | Image Type | #Num |
视觉表格 Visual-Table | 1513 | 折叠展开图 Folded Image Graph | 235 | 立体几何图 Solid Geometry | 2054 | 解析几何图 Analatic Geometry | 3060 |
流程图 Flow Chart | 3120 | 条形图 Bar Chart | 4924 | 散点图 Scatter Chart | 517 | 平面几何图 Plane Chart | 3834 |
折线图 Line Chart | 846 | 饼状图 Fan Chart | 175 | 雷达图 LiDAR Chart | 73 | 抽象类比图 Abatract Analog Graph | 440 |
三视图 Three View Graph | 22 | 枝页图 Stem-and-Leaf display | 23 | 其他 Other Image type | 240 |
3.2 Collectioin Guidelines
We collected a large number of multimodal mathematics questions from a vast K12 educational question bank, including elements such as statistical charts, plane geometry, three-view diagrams, flowcharts, set notation diagrams, etc. The quality and distribution of the data were guided by the following criteria during collection.
-
•
Diverse Mathematical Visual Elements. We have collected solutions to multimodal mathematical problems that rely on understanding image content, especially those containing a large amount of Chinese visual content such as text and symbols. Table 2 shows some visual elements subject of CMMaTH.
-
•
High relevance to the K12 math knowledge and skill. The annotator, who is well-versed in knowledge, needs to ensure that the multimodal question assesses a specific K-12 mathematics knowledge point during the question collection process. It primarily includes mathematics questions related to K12 education, facilitating the assessment of the application potential of large-scale multimodal capabilities in the field of mathematics education.
-
•
High-quality images and answers. During the collection phase, we instruct collectors to disregard multimodal math questions with erroneous symbols or low-quality images (blurry images). Collectors are required to ensure that the collected questions are generally solvable.
3.3 Data Collections
Collection from Diverse Multimodal Math Sources CMMaTH’s data is based on a million-level private database. The private database we used comes from questions collected from the Internet and undergoes rigorous data checking. The project’s data has undergone multiple rounds of collection. We first sampled 45,000 multimodal math questions: 14,000 each from elementary, high, and junior high schools. Then, we added 34,000 more questions featuring algorithm block diagrams, statistics, and geometry diagrams to enhance visual diversity.
Data Filtering We filtered out all questions without images in the question stems, including questions with multi-graph reasoning, questions in non-Chinese languages, and questions not relying on visual content to solve. To ensure the quality of the images and text questions, we removed all images whose width and height were less than 100, then used the GPT4 API to score the data quality and filter out questions suspected of being unsolvable and questions with garbled text in the question text.
Data Labeling For K-12 mathematics knowledge points, we have scraped the mathematics section from Jiaoyan Cloud111https://www.jiaoyanyun.com/ and organized all the knowledge points into a knowledge tree including a total of 5,531 knowledge points. We retained 2,299 knowledge points more relevant to multimodal mathematics in K-12. Subsequently, all questions were classified according to knowledge points by GPT-4 and a fine-tuned LLM, followed by manual multi-level verification. Questions that did not match any K-12 multimodal mathematics knowledge points were filtered out.
3.4 Comparison with Existing Benchmarks
Dataset | Size | Image&Supplementary Input | Format | Source | Answer | Knowledge Annotation | Lanugage Domain | Knowledge Domain |
VQAv2Goyal et al. (2017) | M | V | I+T | Annotated | Open/MC/TF | ✗ | En | Allgemein |
SEEDLi et al. (2023a) | 19K | V | I+T | Annotated | MC | ✗ | En | Allgemein |
MMBenchLiu et al. (2023) | 3K | V | I+T | Repurposed | MC | ✗ | En | Allgemein |
MM-VetYu et al. (2023) | 0.2K | V | I+T | Annotated | Öffnen Sie | ✗ | En | Allgemein |
ScienceQALu et al. (2022) | 6K | V | I+T | Textbooks | MC | ✗ | En | Science |
MathVistaLu et al. (2023) | 1K/6K | V(5 Types)+OC | I+T | Synthesized | Open/MC/TF | ✗ | En/ZH | Math |
MMMUYue et al. (2023) | 11.5K | V(30 Types)+OC | Interleaved | Textbooks | Open/MC | ✗ | - | Allgemein |
CMMMUZhang et al. (2024a) | K(Math Part) | V(5 Types)+OC | Interleaved | Internet | Open/MC | ✗ | ZH | Allgemein |
OlympiadBenchHe et al. (2024) | 6.5K(Math Part) | V(5 Types) | Interleaved | Internet | Öffnen Sie | ✗ | ZH/EH | Math/Physics |
MathVerseZhang et al. (2024c) | 2.6K/15K | V(3 Types) | I+T | Synthesized | MC | ✗ | ZH/EH | Math |
MATH-VisionWang et al. (2024) | 3K | V(16 Types)+IC | I+T | Synthesized | Open/MC | ✗ | EH | Math |
CMMaTH | 23K | V(13 Types)+OC+IC | I+T | Internet/Annotated | Open/MC/TF | ✓ | ZH | K12 Math |
The CMMaTH dataset is primarily used to evaluate multimodal reasoning capabilities in K-12 educational scenarios. We compared the current mainstream multimodal mathematical datasets and large model benchmarks in Table 3. Compared to existing multimodal benchmarks and multimodal reasoning benchmarks, the CMMaTH dataset has the following characteristics:
Extreme Diversity Currently, there is a severe lack of high-quality Chinese multimodal mathematics datasets. MATH-VISION lacks a Chinese component, the MATH-VISTA dataset contains only a small number of Chinese samples, and CMMMU contains only 540 math problems, which are not fine-grained and comprehensive enough. We have included about 23k fine-grained multimodal mathematics assessment samples, covering 13 K12 mathematics visual categories, making it the largest known multimodal Chinese dataset to date.
Real and High Quality & Multilingual MathVista features a substantial number of problems that are associated with natural and synthetic images. However, these images do not accurately represent the genuine data distribution encountered in K12 mathematics educational settings. OlympiadBench is an Olympiad-level bilingual multimodal benchmark. However, this benchmark is overly challenging and deviates from the application of LMM in real K12 multimodal math scenarios. Additionally, the variety of multimodal visual elements is relatively limited. Instead, we collect multimodal data specifically tailored to the K12 education context. Additionally, MathVista incorporates a significant amount of data from GeoQA and synthetic images, which have relatively poor image quality. Our multimodal visual image elements have all undergone stringent image quality assessments. Unlike CMMMU, CEval, and CMath, our dataset is a bilingual dataset that considers a large number of Chinese scenes. In addition to the text of the questions being in Chinese, the visual elements related to the questions also contain Chinese text/symbols.
High-quality Fine-grained Annotation and Evaluation Tool Every question in our dataset is meticulously annotated with standardized answers, solutions expressed in natural language, associated multimodal knowledge points, visual element categories, and K-12 grade levels. This fine-grained annotation enables a more nuanced evaluation of multimodal mathematical proficiency within the K-12 educational context. While MathVista and GeoEval rely on GPT-4 for answer extraction and validation, we introduce an open-source model named GradeGPT. GradeGPT stands out by providing a stable, cost-free, and swift accuracy evaluation specifically tailored for the CMMaTH dataset.
![Refer to caption](x3.png)
4 GradeGPT
The CMMaTH dataset encompasses a large variety of problem-solving objectives, such as mathematical expressions, multiple-choice options, numerical outcomes, coordinate points, conclusion figures, and correctness assessments. Traditionally, in reasoning or evaluation contexts, problems have been formulated as multiple-choice or true/false questions to facilitate comparison and to simplify the extraction of results. Also, it is difficult to maintain dynamically updated benchmark. Employing API models for evaluation is prohibitively expensive, and the resulting evaluations are not consistently stable, which also hampers the iterative development of models on benchmarks, such as hyperparameter selection.
To provide a stable, free, fast, and easy-to-update model response evaluation tool, we introduce GradeGPT, an answer comparison model tailored for the CMMaTH dataset. GradeGPT is designed to receive a question, its standard answers, and a model-generated response. It extracts key steps including results from Chinese output. Determine whether the result is consistent with the standard answer. Our GradeGPT is a streamlined, open-source model. When integrated with frameworks such as vLLM using the 14B model, it can swiftly compare a myriad of model-generated answers, accomplishing a remarkable judgment accuracy of 96.1% for assessing responses comparable with GPT4 API.
Prompt Format
In the prompt input of GradeGPT, there are "questions," "reference answers," and "model output answers." The model is required to provide an answer in the form of "<Yes>" or "<No>" indicating whether the model output answer is equivalent to the standard reference answer. We have designed an instruction format named Cross-Lingual-Judge-of-Chain for the purpose of determining answer consistency. Cross-Lingual-Judge-of-Chain first analyzes the model response and finds the key sentences that give the answer in the model response, understand key chinese sentences in English. Then analyze the standard answer, determine the type of the standard answer, and then determine whether the standard answer is included in the model response. More details can be found in Appendix E
Instruction Construction
We first generate inference results on CMMaTH using multiple Multimodal LLMs and provide GPT-4 with a detailed few-shot prompt to synthesize answer judgments in the form of a Cross-Lingual Judge-of-Chain response. By employing GPT4’s In-Context Learning, as showned in Figure 3, we have established a procedure for synthesizing instruction data and have produced approximately 56k cross-lingual result judge instruction pairs. Through fine-tuning the model with these instructions, we are able to obtain an expert model, GradeGPT, which possesses the capability to compare answers.
5 Experiments
We conducted a series of experiment to evaluate various models on the CMMaTH dataset. We evaluated various LLM/LMM models, including open-source and closed-source models. More model details can be found in Table 13. We employed a method similar to GeoEval and MathVista, generating captions through an GPT4V, and assessed them using MetaMath, and DeepSeekMath equipped with caption information. Our empirical research reveals that even the most advanced models struggle to achieve satisfactory accuracy levels. Furthermore, we conducted an exhaustive error analysis on a sufficiently strong commercial multimodal model, GPT-4V, examining its error distribution and presenting illustrative qualitative examples. Our investigation also revealed that the inclusion of multilingual thought chains does not mitigate the substantial difficulties presented by Chinese multimodal mathematical reasoning scenarios. We postulate that the richness of non-English contextual information contained within the images necessitates models equipped with enhanced multilingual OCR and sophisticated multimodal diagram reasoning capabilities.
5.1 Main Experiments on LLM/LMMs
![Refer to caption](x4.png)
![Refer to caption](x5.png)
Model | Overall | Flow | Bar | Scatter | Line Plot | Fan | LiDAR | Visual-Table | Three View | Folded Image | Analytic | Solid | Plane | Venn | Abt-Analogy |
LLMs (Text Only) | |||||||||||||||
LLama2-70B | 4.5 | 4.7 | 2.5 | 4.4 | 7.9 | 7.4 | 8.1 | 3.4 | 5.4 | 5.1 | 5.3 | 4.1 | 5.3 | 5.9 | 4.5 |
MetaMath-70B | 5.7 | 4.6 | 3.3 | 6.6 | 8.7 | 5.7 | 0.2 | 4.2 | 4.1 | 8.5 | 7.2 | 4.8 | 8.5 | 9.8 | 5.4 |
DeepSeek-Math | 14.0 | 13.4 | 6.7 | 14.7 | 13.1 | 12.5 | 12.2 | 8.1 | 13.5 | 12.3 | 17.2 | 16.5 | 21.6 | 19.5 | 13.8 |
Baichuan-13B | 8.4 | 6.7 | 4.8 | 12.2 | 12.4 | 13.1 | 16.2 | 5.4 | 4.1 | 8.5 | 11.1 | 6.7 | 13.7 | 12.8 | 9.3 |
Qwen-14B | 13.7 | 15.5 | 7.3 | 14.3 | 16.9 | 13.6 | 10.8 | 11.4 | 12.8 | 14.8 | 15.9 | 12.7 | 17.8 | 20.4 | 19.3 |
Math LLMs (Text + OCR Caption) | |||||||||||||||
LLama2-70B | 5.6 | 4.9 | 2.3 | 4.8 | 7.9 | 7.1 | 8.0 | 4.4 | 6.4 | 9.1 | 3.3 | 4.8 | 6.3 | 6.9 | 5.5 |
MetaMath-70B | 5.1 | 4.3 | 3.2 | 6.9 | 8.1 | 5.3 | 0.0 | 4.4 | 4.2 | 8.8 | 7.1 | 4.4 | 8.3 | 9.1 | 5.2 |
DeepSeek-Math | 15.3 | 13.2 | 6.9 | 14.1 | 12.6 | 12.3 | 12.1 | 8.9 | 14.4 | 14.1 | 17.9 | 19.3 | 22.7 | 21.5 | 13.9 |
Baichuan-13B | 8.1 | 6.9 | 4.3 | 12.4 | 11.5 | 12.3 | 14.9 | 3.4 | 4.4 | 9.3 | 11.6 | 6.8 | 13.2 | 12.9 | 9.9 |
Qwen-14B | 13.3 | 14.1 | 7.4 | 13.3 | 16.2 | 13.2 | 11.8 | 10.6 | 11.8 | 19.8 | 5.9 | 11.7 | 13.8 | 21.4 | 16.3 |
Open-source LMMs (Text + Image) | |||||||||||||||
LLaVA-v1.5-7B | 5.5 | 1.5 | 4.2 | 5.4 | 6.2 | 5.4 | 3.6 | 4.0 | 4.2 | 5.3 | 4.8 | 3.9 | 8.4 | 6.1 | 4.2 |
InternLM-XComposer2-VL | 3.4 | 3.3 | 5.3 | 3.2 | 6.2 | 11.3 | 6.2 | 5.4 | 4.0 | 0.5 | 0.4 | 3.6 | 1.5 | 1.8 | 3.6 |
Yi-VL-34B | 8.3 | 7.1 | 4.6 | 10.2 | 14.6 | 8.5 | 6.8 | 7.7 | 5.9 | 6.4 | 10.1 | 7.8 | 12.2 | 11.3 | 7.9 |
CogAgent-Chat | 10.6 | 12.2 | 5.2 | 10.8 | 13.7 | 8.0 | 9.5 | 8.8 | 11.2 | 10.2 | 13.2 | 10.5 | 11.8 | 19.9 | 12.2 |
Closed-source LMMs (Text + Image) | |||||||||||||||
GPT4V | 27.0 | 39.3 | 12.5 | 30.2 | 21.0 | 22.9 | 38.6 | 16.9 | 18.3 | 20.0 | 37.5 | 15.8 | 21.5 | 58.0 | 29.9 |
GPT4o | 35.2 | 59.4 | 18.8 | 54.5 | 31.7 | 58.4 | 32.4 | 31.7 | 28.7 | 23.8 | 40.6 | 31.6 | 33.6 | 57.4 | 29.7 |
Human Performance | |||||||||||||||
Human (testmini) | 80.1 | 73.7 | 78.9 | 96.2 | 95.1 | 57.4 | 91.7 | 83.5 | 69.2 | 63.2 | 67.5 | 51.6 | 72.1 | 89.1 | 83.1 |
We evaluated the results of mainstream multimodal large models and mathematical expert models in Table B. We analyzed the trend of existing large models in descending with problems and conditions, as well as the effectiveness of techniques such as Cross-Lingual Prompting in solving Chinese multimodal mathematical problems. The experimental in Table 4 results indicates that our data exhibits extremely strong diversity and relatively challenging reasoning depth. Figure 1 and Table 4 shows models such as GPT4V struggle to comprehend our multimodal content and reasoning questions effectively, resulting in significant performance gaps between open-source and proprietary models. In certain rare visual domains, multimodal large models achieve very low reasoning outcomes.
Accuracy on various question types. We evaluated the accuracy of GPT4V on various target-solving tasks in Figure 4. The results indicate that when solving free-form problems, especially those with more diverse targets such as expressions, coordinates, and conclusion judgments, the multimodal large language model shows poorer performance.
Is OCR information sufficient for CMMaTH? We also referred to works like MathVista, attempting to use LLMs combined with OCR information from diagrams to assist in mathematical reasoning in Table 4. We found that, in our benchmark, a small amount of OCR information (such as mathematical symbols in diagrams, axis values, and image titles) made it very difficult to complete our multimodal mathematical reasoning tasks. The results indicate that solving problems in CMMaTH requires stronger multimodal mathematical chart capabilities, beyond just OCR.
K12 Multimodal Knowledge Richness of current LMMs. We systematically evaluated the proficiency of existing multimodal large models in the K12 domain regarding multimodal reasoning skills in Figure 5. The results revealed a significant knowledge gap in existing multimodal K12 educational resources. Compared to other existing LMMs, GPT4V possesses a richer knowledge base, thereby substantially reducing the illusion of reasoning in multimodal mathematical inference.
5.2 Experiments of Cross-language Reason Technology
We also attempted several multilingual Chain-of-Thought approaches such as En-CoT, CLP(Cross-Lingual Prompting) used by Qin et al. (2023) to observe whether multimodal mathematical problems could be enhanced through context learning techniques without training. The results indicate that multilingual CoT methods face challenges in solving, possibly due to the abundance of Chinese contextual text in the image content, which may necessitate the model to demonstrate excellent cross-lingual OCR capabilities. We have included more details on the implementation of Cross-Lingual Prompting and En-CoT on the CMMaTH dataset in the Table 5.
LMM | Overall-Acc |
LLaVA-v15 | 4.2 |
InternLM-XComposer2-VL | 3.4 |
LLaVA-v15 + En-CoT | 9.4 |
InternLM-XComposer2-VL + En-CoT | 16.9 |
LLaVA-v15 + CLP | 12.7 |
InternLM-XComposer2-VL + CLP | 17.1 |
5.3 Error Analysis
![Refer to caption](x6.png)
We conducted a detailed analysis and evaluation of GPT4V on CMMaTH-testmin, categorizing errors into four types: perceptual errors, reasoning errors, calculation errors, and Reject Errors. The error type distribution of GPT4V on CMMaTH is shown in the Figure 6.
Perception Errors
Perception Error refers to the model’s erroneous interpretation and utilization of diagram content during reasoning. For example, incorrect OCR, misidentification of numerical relationships, geometric relationships, logical relationships, etc.
Reasoning Errors
Reasoning Error are quite common during the solving process. For instance, the model may misinterpret symbols or use incorrect logic or knowledge for inference. The frequency of Reasoning Errors reflects the model’s logical and mathematical reasoning capabilities.
Calculation Errors
Calculation Error refers to the model performing incorrect mathematical operations, such as writing equations or solving equations incorrectly.
Reject Errors
Reject Error refers to the model’s inability to solve a problem that is actually solvable. The frequency of such errors reflects the model’s ability to follow instructions.
6 Conclusions
We introduce CMMaTH, a detailed Chinese math reasoning benchmark with diverse question types, vivid visuals, and complex reasoning. The benchmark includes detailed knowledge points, standard thought processes, and grade levels to measure the mastery of knowledge points in the K-12 multimodal math skill. To evaluate large multimodal models quickly and affordably, we built GradeGPT, an open-source tool for assessing results. Extensive experimental results on CMMaTH manifest the limitations of current models in multilingual, multimodal math reasoning.
Limitation & Potential Impact
Our dataset CMMaTH, as a multimodal mathematics dataset aimed at the K-12 education sector, can facilitate model evaluation and iteration of multimodal large models in this field, and may promote the research and development of educational artificial intelligence. CMMaTH primarily consists of single-image problems, without considering multi-image contextual reasoning or scenarios requiring auxiliary line drawing and similar tasks. GradeGPT is a result-oriented, relatively coarse reasoning response evaluator. How to construct a process evaluation model for fine-grained assessment of the reasoning ability of large models can continue to be explored in the future.
References
- Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR, abs/2110.14168.
- Duan et al. (2023) Haodong Duan, Jueqi Wei, Chonghua Wang, Hongwei Liu, Yixiao Fang, Songyang Zhang, Dahua Lin, and Kai Chen. 2023. Botchat: Evaluating llms’ capabilities of having multi-turn dialogues. CoRR, abs/2310.13650.
- Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 6325–6334. IEEE Computer Society.
- He et al. (2024) Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. 2024. Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. CoRR, abs/2402.14008.
- Hendrycks et al. (2021a) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
- Hendrycks et al. (2021b) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.
- Huang et al. (2023) Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.
- Kamalloo et al. (2023) Ehsan Kamalloo, Nouha Dziri, Charles L. A. Clarke, and Davood Rafiei. 2023. Evaluating open-domain question answering in the era of large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 5591–5606. Association for Computational Linguistics.
- Ke et al. (2023) Pei Ke, Bosi Wen, Zhuoer Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, Jie Tang, and Minlie Huang. 2023. Critiquellm: Scaling llm-as-critic for effective and explainable evaluation of large language model generation. CoRR, abs/2311.18702.
- Li et al. (2023a) Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023a. Seed-bench: Benchmarking multimodal llms with generative comprehension. CoRR, abs/2307.16125.
- Li et al. (2023b) Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023b. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 292–305. Association for Computational Linguistics.
- Liu et al. (2023) Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2023. Mmbench: Is your multi-modal model an all-around player? CoRR, abs/2307.06281.
- Lu et al. (2023) Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating math reasoning in visual contexts with gpt-4v, bard, and other large multimodal models. CoRR, abs/2310.02255.
- Lu et al. (2022) Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.
- Pillutla et al. (2021) Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaïd Harchaoui. 2021. MAUVE: measuring the gap between neural text and human text using divergence frontiers. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 4816–4828.
- Qin et al. (2023) Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. 2023. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 2695–2709. Association for Computational Linguistics.
- Shi et al. (2023) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
- Wang et al. (2024) Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. 2024. Measuring multimodal mathematical reasoning with math-vision dataset. Preprint, arXiv:2402.14804.
- Wang et al. (2023a) Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023a. Large language models are not fair evaluators. CoRR, abs/2305.17926.
- Wang et al. (2023b) Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. 2023b. Pandalm: An automatic evaluation benchmark for LLM instruction tuning optimization. CoRR, abs/2306.05087.
- Yu et al. (2023) Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. CoRR, abs/2308.02490.
- Yue et al. (2023) Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. CoRR, abs/2311.16502.
- Zhang et al. (2024a) Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng, Chunpu Xu, Shuyue Guo, Haoran Zhang, Xingwei Qu, Junjie Wang, Ruibin Yuan, Yizhi Li, Zekun Wang, Yudong Liu, Yu-Hsuan Tsai, Fengji Zhang, Chenghua Lin, Wenhao Huang, Wenhu Chen, and Jie Fu. 2024a. CMMMU: A chinese massive multi-discipline multimodal understanding benchmark. CoRR, abs/2401.11944.
- Zhang et al. (2024b) Jiaxin Zhang, Zhongzhi Li, Mingliang Zhang, Fei Yin, Chenglin Liu, and Yashar Moshfeghi. 2024b. Geoeval: Benchmark for evaluating llms and multi-modal models on geometry problem-solving. CoRR, abs/2402.10104.
- Zhang et al. (2024c) Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, and Hongsheng Li. 2024c. Mathverse: Does your multi-modal LLM truly see the diagrams in visual math problems? CoRR, abs/2403.14624.
- Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
- Zhu et al. (2023) Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Jiajun Chen, Lei Li, and Shujian Huang. 2023. Multilingual machine translation with large language models: Empirical results and analysis. CoRR, abs/2304.04675.
Appendix A More Related Work About Multimodal Large Model Evaluation
The multimodal large models face serious hallucination issues in perceiving objects and executing inference. To systematically evaluate the various capabilities of multimodal large models, diverse multimodal benchmarks are utilized for assessing the abilities of large models and aiding iterative development. POPELi et al. (2023b) is used to evaluate the accuracy of large models in identifying perceptual objects. MMMU and CMMMUYue et al. (2023); Zhang et al. (2024a) are comprehensive subject datasets designed to assess the proficiency of large models in mastering massive multimodal multi-disciplinary knowledge. SEED-Bench designed 19,000 diverse multimodal questions spanning video and image modalities to evaluate the spatiotemporal capabilities of multimodal large models Li et al. (2023a). MMVetYu et al. (2023) attempts to design datasets to evaluate the integrated capabilities of different multimodal large model systems in combining various Vision-Language skills.
Appendix B Model Generation Details
B.1 Model Weight Version
We evaluated models on CMMaTH, including open-source models such as LLaVA-v1.5, Deepseek-Math, InternLM-XComposer2-VL, Yi-VL-34B, CogAgent-Chat, MetaMath-70B, LLama-70B, Baichuan-13B and Qwen-14B as well as state-of-the-art commercial models GPT4V. We have listed the parameter versions and the Hugging Face repository names of the open-source models used in Table 12.
B.2 Model Sampling Parameter
We have listed the corresponding hyperparameters used by the models in Table 13. For API models, we have indicated the corresponding release versions. Models using vLLM for inference are annotated.
B.3 Data quality control
To ensure the high quality of the final data, we conducted sampling and manual verification. We performed three random samples, each consisting of 500 multimodal samples, to check the data quality and ensure the consistency of the knowledge points and data.
Appendix C Prompt Details
![Refer to caption](x7.png)
C.1 Prompt For Step Response Generation
Element | Prompt |
system-prompt | 你是一个数学题解题助手,你的输入是一道数学题,以及这道题目的图像,你的任务是输出这道题目的求解思路以及答案. 输出格式是一步一步的方式. |
Example1 | Question: 小明有2000元, 一个苹果2元, 小明买了10个苹果。小明还有多少钱。 Model response: 首先计算小明买苹果的开销为2乘10等于20元. <STEP1> 用2000元减去所有的开销, 2000-20=1980. <STEP2> 所以最终的答案为1980元. <STEP3> |
Example2 | Question: 如图,在△ABC中,∠C=90°,∠B=30°,以A为圆心,任意长为半径画弧分别交AB、AC于点M和N,再分别以M、N为圆心,大于MN/2的长为半径画弧,两弧交于点P,连结AP并延长交BC于点D,则下列说法中正确的个数是( )①AD是∠BAC的平分线;②∠ADC=60°;③点D在AB的中垂线上;④BD=2CD.A.4 B.3 C.2 D.1 Model response: 首先确定△ABC的几何性质。 <STEP1>∠C=90°,∠B=30°,∠A=120°。 <STEP2>△ABC是一个等腰三角形。 <STEP3>根据等腰三角形的性质,AD是∠BAC的平分线。 <STEP4>①AD是∠BAC的平分线,正确。 <STEP5>②∠ADC=60°,错误。 <STEP6>③点D在AB的中垂线上,错误。 <STEP7>④BD=2CD,正确。 <STEP8>因此,正确的个数为4。 <STEP9>答案为A.4。 <STEP10>请你根据这个例子,解决下面的数学题。问题:在△ABC中,∠C=90°,∠B=30°,以A为圆心,任意长为半径画弧分别交AB、AC于点M和N,再分别以M、N为圆心,大于MN/2的长为半径画弧,两弧交于点P,连结AP并延长交BC于点D,则下列说法中正确的个数是( )①AD是∠BAC的平分线;②∠ADC=60°;③点D在AB的中垂线上;④BD=2CD.A.4 B.3 C.2 D.1 求解步骤:首先确定△ABC的几何性质。∠C=90°,∠B=30°,∠A=120°。△ABC是一个等腰三角形。根据等腰三角形的性质,AD是∠BAC的平分线。①AD是∠BAC的平分线,正确。②∠ADC=60°,错误。③点D在AB的中垂线上,错误。④BD=2CD,正确。因此,正确的个数为4。答案为A.4。 |
When evaluating hallucinations during the assessment process, we use a few-shot prompt format to elicit step-by-step outputs from the model as showed in Table 6.
C.2 Prompt For GradeGPT
Element | Prompt |
system-prompt | You are an answer evaluation assistant. Based on the following questions and reference answers, determine if the model’s output is the same answer. When making the determination, please provide the analysis process in the following format. Note, for multiple-choice questions, it is only correct if the output includes all the corresponding options. (For example: If the reference answer is: A, C, and the model response is: C, the answer consistency is: No) Note, if the model response does not specify a particular option, and its correctness cannot be determined, the answer consistency should be: No. |
Example | Question: 如图所示,AD是的中线,点E在AC上,BE交AD于点F.当时, Model Response: 首先,我们需要找到中线AD,然后找到点E在AC上的位置。找到中线AD,可以通过勾股定理求解,。设, ,则有 。<STEP1> 解这个方程,我们得到 。现在我们知道点E在AC上,所以E的横坐标为y,纵坐标为x。 接下来,我们需要找到点F在BD上的位置。由于F点在BD上,我们可以设F点在BD的一半,即。<STEP2> 现在我们可以计算AF和FD的长度。, 将已知的x和y代入公式,得到: 简化得到: 这意味着。<STEP3> 现在我们可以求解题目中的条件: 将已知的AF和FD代入公式,得到:化简得到:所以,答案为 <STEP4> Reference Response: Judge Result: The relevant paragraph in the Model Response that provides the answer is 所以,答案为AE / AC = 1 / 2. Understand the question in Chinese. The Chinese model response means: Therefore, the answer is AE / AC = 1 / 2. Next, let’s analyze whether the correct answer has been given. The reference answer is a numerical answer, and the output given in the model response is 1/2. The consistency of the answers is: <No>. |
We also listed the prompts used by GradeGPT in Tables 7.
C.3 Prompt For Cross-Lingual Prompting and En-CoT
We have listed the specific prompts used for En-CoT and Cross-Lingual Prompt during actual execution in Table 11. Unlike the original Cross-Lingual Prompt paper, for experimental simplicity, we only adopted a single-turn format. However, this suffices to illustrate the varying inferential capabilities across different languages in current LMMs.
Appendix D CMMaTH Dataset Details
D.1 Data Collection Details
To more clearly elucidate our data collection process, we have depicted the overall pipeline of data collection in Figure 7.
D.2 Knowledge Point Details
We provided detailed annotations of knowledge points for our dataset and conducted preliminary clustering of these knowledge points. The distribution of knowledge points in different clusters is as follows:
![Refer to caption](extracted/5692186/figures/wordbrain.png)
We have formulated a Knowledge Successful Solve Rate(SSR) as a structural metric to gauge the proficiency level of multi-modal extensive models in mastering knowledge points. is the total number of knowledge point of CMMaTH. is the of questions about ’th knowledge point. denotes an indicator function.
(1) |
It is our contention that a knowledge point can be deemed comprehensively understood only when the accuracy rate of solving problems related to that knowledge point surpasses a predefined threshold, denoted as . For the purpose of our investigation, we have established at the values of 0.1, 0.2, 0.3, and 0.6 to demarcate the levels of mastery.
D.3 Characteristics Of Annotators
We utilized a standard team of four people, who spent two weeks annotating the data. All annotators have a university undergraduate education and are well-versed in basic knowledge of the K12 education field. To ensure quality, each question was verified by at least two people.
Appendix E GradeGPT details
E.1 GradeGPT Prompt Detail
We have listed detailed Fewshot Examples using the GPT4-generated GradeGPT model responses in Table 11. Through this table, you can observe the specific form of the Cross-Lingual-Judge-of-Chain that we have used.
E.2 GradeGPT Performance Metric
GradeGPT performance evaluation metric is precision in comparison. We constructed a model that responds to a test set containing outputs from various large models (including both correct and incorrect model outputs). Each output is labeled as correct or incorrect based on its result. GradeGPT is tasked with assessing whether the model responses are correct or incorrect, and this performance evaluation metric is a binary classification metric.
(2) |
E.3 GradeGPT Training Details
We generated cross-lingual evaluation instruction pairs using the outputs from InternLM-XComposer, LLaVA-v1.5, CogAgent-18B and Yi-VL-34B. These outputs were produced using GPT-4 Fewshot. The generated evaluation instructions were filtered based on specific rules, retaining only those responses from GPT-4 that contained the fields: <Yes>/<No>. Ultimately, we constructed a cross-lingual format instruction set comprising 56k instruction pairs.
GradeGPT was trained on 8 H800, with the Qwen-14B-Chat version used as the base model. The model’s batch size was set to 16. The learning rate was set to 1e-4, and the gradient accumulation step was set to 16. It was trained for 10 epochs on a 40k bilingual Judge-of-Chain dataset. A detail example of instruction can refer to Figure 9.
![Refer to caption](x8.png)
LLM | |
Qwen-7B-Chat(4-Shot) | 35.1 |
Naive Outcome Finetune | 51.5 |
Judge-of-Chain | 65.3 |
Cross-Lingual-Judge-of-Chain | 85.1 |
Qwen-14B-Chat(4-Shot) | 43.7 |
GradeGPT(14B) | 96.1 |
GPT4(4-Shot) | 97.2 |
E.4 Futher More Ablation Study
We conducted experiments on a development set comprising outputs from a 0.5k model. The development set was sampled from a subset of 0.5k questions on CMMaTH. Each question was accompanied by answers provided by GPT-4V, GPT-4o, and middle school students. Each answer was manually annotated to indicate whether it was correct. We use 2 to measure the answer judgment capability of different LMMs, including Zershot LMMs and LLMs after Finetune.
Ablation On Instruction Format We conducted experiments on various instruction enhancement techniques used by GradeGPT and compared the results with GPT4 in Table 8. The results suggest that after various instruction enhancements, the accuracy of GradeGPT in model response judgment on CMMaTH can be improved to 96.1%, significantly surpassing the accuracy of GPT4. The proposed strategy can significantly enhance GradeGPT’s ability to judge results. It is only slightly weaker than the performance of GPT4(Fewshot) executed with a large number of examples. Our GradeGPT, as an open-source parameter model of approximately 14B, can serve as a stable, low-cost, and efficient alternative to GPT4.
The Baseline we compared, Qwen-7B/14B(4-Shot), GPT4(4-Shot), Naive Outcome Finetune, Judge-of-Chain. In the Naive Outcome Finetune format of instructions, the model is required to output its results indicating whether they are correct in the form of "<Yes>"/"<No>".. Judge-of-Chain also includes the understanding of results and natural language descriptions of model outputs, but does not include the part of extracting key Chinese outputs and translating them into English. Compared to having the language model directly predict the <Yes>/<No> judgment labels, directly using Chinese Judge-of-Chain to construct Judge-of-Chain improves the model’s performance in answer evaluation. However, they didn’t yield good results. The performance disparity may stem from an imbalance in the quality of Chinese and English components within some bilingual base models due to the training corpus.
Our findings show that fine-tuning with Cross-Lingual-Judge-of-Chain for detailed thought chain refinement significantly improves the performance of open-source models in outcome analysis tasks.
Additionally, we discovered that using bilingual thought chains instead of Chinese-only thought chains for base model fine-tuning effectively enhances performance in outcome determination tasks. By using and synthesizing the instructions in the form of Cross-Lingual-Judge-of-Chain that we designed, we are able to efficiently distill the answer reviewing capabilities of GPT4.
Ablation On Instruction Data Source
The instruction data for Cross-Lingual Judge-of-Chain Prompts comes from outputs of various LLMs on CMMaTH. We conducted ablation experiments on the sources of instruction data, which showed the impact of using different LLM models on constructing diverse and effective instruction data.
Model Response Source | |
LLaVA-v1.5 response | 77.2 |
InterLM-XComposer2-VL response | 83.1 |
CogAgent-Chat response | 84.2 |
Yi-VL-34B response | 85.1 |
Appendix F Case Study
We conducted a fine-grained manual evaluation of GPT4V’s output on CMMaTH, with the results shown in Figure 10 to Figure 22.
Element | Prompt |
system-prompt | You are a math problem-solving assistant. Your input is a math problem and an image of the problem. Your task is to provide the solution steps and the final answer. The output format should be step-by-step. |
Example | Question: Xiaoming has 2000 yuan, an apple costs 2 yuan, Xiaoming bought 10 apples. How much money does Xiaoming have left? Model response: First, calculate the cost of the apples Xiaoming bought, which is 2 times 10 equals 20 yuan. <STEP1> Subtract the total cost from 2000 yuan, 2000-20=1980. <STEP2> So the final answer is 1980 yuan. <STEP3> |
Element | Prompt |
system-prompt | Please act as an expert in multi-lingual understanding in Chinese. After understanding, you should act as an expert in arithmetic reasoning in English. Let’s resolve the task you understand above step-by-step! Finally, you should format your answer as ’Answer: [num]’. |
Example | Request: 玛丽比琼小两岁,琼比杰莎大五岁。如果杰莎20岁,这三个女孩的年龄总和是多少? Reason Result: Let’s understand the task in English step-by-step! 1. "玛丽比琼小两岁" means "Mary is two years younger than Joan." 2. "琼比杰莎大五岁" means "Joan is five years older than Jessie. " 3. "如果杰莎 20 岁" means "If Jessie is 20 years old. " Then, Let’s solve the task step-by-step: 1. According to the first sentence, Mary is two years younger than Joan. Let’s suppose Joan’s age is J, then Mary’s age is J - 2. 2. According to the second sentence, Joan is five years older than Jessie. If Jessie’s age is x, then Joan’s age is x + 5. Answer: 68. |
Model | Name |
Llama-2-70B | meta-llama/Llama-2-70b-hf |
MetaMath-70B | meta-math/MetaMath-70B-V1.0 |
DeepSeek-Math-7B | deepseek-ai/deepseek-math-7b-instruct |
Baichuan-13B | baichuan-inc/Baichuan2-13B-Chat |
Qwen-14B | Qwen/Qwen-14B-Chat |
LLaVA-v1.5 | liuhaotian/llava-v1.5-13b |
InterLM-XComposer2-VL | internlm/internlm-7b |
Yi-VL-34B | 01-ai/Yi-VL-34B |
CogAgent-Chat | THUDM/cogagent-chat-hf |
Model Name | Generation Parameters | Kommentare |
Llama-2-70B | do_sample=True, top_k=0.5, top_p=0.5, max_tokens=512 | model=""Salesforce/codegen2-16B" |
GPT-4 | temperature=0.2, max_tokens=2048 | version="gpt-4-1106-preview" |
llava-7B-V1.5 | temperature=0.2, max_new_tokens=2048 | llava package |
DeepSeek-Math-7B | temperature=0.2, max_new_tokens=2048 | vllm package |
Baichuan-13B | temperature=0.2, max_new_tokens=2048 | vllm package |
Qwen-14B | temperature=0.2, max_new_tokens=2048 | vllm package |
InterLM-XComposer2-VL | temperature=0.2, max_new_tokens=2048 | Huggingface |
Yi-VL-34B | temperature=0.2, max_new_tokens=2048 | Huggingface |
CogAgent-Chat | temperature=0.2, max_new_tokens=2048 | Huggingface |
GPT4V | temperature=0.2, max_tokens=2048 | version="gpt-4-vision-2023-05-15" |
GPT4o | temperature=0.2, max_tokens=2048 | version="gpt-4o-2024-02-01" |
![Refer to caption](x9.png)
![Refer to caption](x10.png)
![Refer to caption](x11.png)
![Refer to caption](x12.png)
![Refer to caption](x13.png)
![Refer to caption](x14.png)
![Refer to caption](x15.png)
![Refer to caption](x16.png)
![Refer to caption](x17.png)
![Refer to caption](x18.png)
![Refer to caption](x19.png)
![Refer to caption](x20.png)
![Refer to caption](x21.png)