CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Models

To evaluate the performance of large models in mathematical reasoning and examine hallucinations during the reasoning process, numerous benchmarks have been proposed for evaluating the mathematical reasoning capabilities of large models. GSM8KCobbe et al. (2021) is the first and most widely used mathematical dataset used for large model math evaluation, consisting of 1k math word problem test samples and corresponding answers. The MATHHendrycks et al. (2021b) dataset, in comparison to GSM8K, presents a greater challenge in terms of reasoning difficulty. This dataset demands a more profound understanding and intuition in various mathematical domains such as Algebra, Number Theory, and Geometry. MathVistaLu et al. (2023) is the first dataset used to evaluate the multimodal mathematical capabilities of large models, but it has relatively simple reasoning depth. MATH-VISIONWang et al. (2024) has richer visual elements and deeper reasoning difficulty. MathVerseZhang et al. (2024c) constructed several subsets of datasets to assess whether existing multimodal large models can truly understand mathematical abstract forms.

The CMMaTH Benchmark, in comparison to existing works on the evaluation of mathematical proficiency, places a greater emphasis on the analysis of mathematical abilities within the context of the Chinese language. The data distribution of the CMMaTH dataset more closely aligns with the actual distribution found in K12 educational settings, and it provides detailed annotations of mathematical knowledge points to facilitate the assessment of models’ mastery of knowledge and skills.

Due to their strong generalization capabilities and extensive world knowledge, large language models have achieved outstanding results in tasks such as machine translationZhu et al. (2023), question answeringKamalloo et al. (2023), dialogueDuan et al. (2023) and so on by generating text. Evaluating the comprehensive abilities of large models, such as clarity, adherence to instructions, comprehensiveness, formality, and mathematical reasoning ability, has received widespread attentionKe et al. (2023). Currently, many works opt to use powerful commercial model APIs, such as GPT-4, to assist in evaluating the comprehensive abilities of large models. For instance, MathVistaLu et al. (2023) and GeoEvalZhang et al. (2024b) use GPT-4’s API to extract correct answers for evaluation. These methods face several challenges: they are costly and time-consuming, and they struggle to keep up with rapid model iterations. Besides, these methods face challenges in terms of consistency and reproducibilityWang et al. (2023a); Ke et al. (2023).

Recent methods have proposed using metrics such as BERT scoreZhang et al. (2020) or MAUVEPillutla et al. (2021) for evaluation. However, the numerical indicators produced by these methods are difficult to interpret when it comes to the erroneous responses generated by LLM. PandaLM and CritiqueLLM Wang et al. (2023b); Ke et al. (2023) are similar to our work. They proposed a fine-tuning method based on open-source LLMs, distilling the evaluation capabilities of GPT-3.5 into a series of smaller open-source models. However, they are focused on the automated evaluation of more general text generation tasks, while we are targeting the automated evaluation of responses from large models for multimodal mathematics problems.

Unlike PandaLMWang et al. (2023b) trying to evalution relative conciseness, clarity and so on, our evaluation model, GradeGPT, is a dataset-oriented answer comparator that can provide specific reasons based on the standard answer and a model’s response. We distilled the answer comparison capability of GPT-4 using the Cross-Lingual Judge-of-Chain method and enhanced GradeGPT’s answer discrimination ability.

We selected diverse multimodal mathematical problems from a vast pool of K12 educational questions, comprising 23856 items across 13 visual themes, 5 difficulty levels, and encompassing 150 types of knowledge points. More detailed statistical data can be found in Table 1.

For the convenience of evaluation, we provide a miniaturized test set of CMMaTH, called CMMaTH-testmin, containing 1500 samples. Testmin retains the diversity of the CMMaTH dataset and shows similar overall performance to the entire CMMaTH dataset. Evaluators can conduct quick tests and generate preliminary analyses based on CMMaTH-testmin.

UTF8gbsn

We collected a large number of multimodal mathematics questions from a vast K12 educational question bank, including elements such as statistical charts, plane geometry, three-view diagrams, flowcharts, set notation diagrams, etc. The quality and distribution of the data were guided by the following criteria during collection.

• •

  Diverse Mathematical Visual Elements. We have collected solutions to multimodal mathematical problems that rely on understanding image content, especially those containing a large amount of Chinese visual content such as text and symbols. Table 2 shows some visual elements subject of CMMaTH.

• •

  High relevance to the K12 math knowledge and skill. The annotator, who is well-versed in knowledge, needs to ensure that the multimodal question assesses a specific K-12 mathematics knowledge point during the question collection process. It primarily includes mathematics questions related to K12 education, facilitating the assessment of the application potential of large-scale multimodal capabilities in the field of mathematics education.

• •

  High-quality images and answers. During the collection phase, we instruct collectors to disregard multimodal math questions with erroneous symbols or low-quality images (blurry images). Collectors are required to ensure that the collected questions are generally solvable.

Collection from Diverse Multimodal Math Sources CMMaTH’s data is based on a million-level private database. The private database we used comes from questions collected from the Internet and undergoes rigorous data checking. The project’s data has undergone multiple rounds of collection. We first sampled 45,000 multimodal math questions: 14,000 each from elementary, high, and junior high schools. Then, we added 34,000 more questions featuring algorithm block diagrams, statistics, and geometry diagrams to enhance visual diversity.
Data Filtering We filtered out all questions without images in the question stems, including questions with multi-graph reasoning, questions in non-Chinese languages, and questions not relying on visual content to solve. To ensure the quality of the images and text questions, we removed all images whose width and height were less than 100, then used the GPT4 API to score the data quality and filter out questions suspected of being unsolvable and questions with garbled text in the question text.
Data Labeling For K-12 mathematics knowledge points, we have scraped the mathematics section from Jiaoyan Cloud¹¹1https://www.jiaoyanyun.com/ and organized all the knowledge points into a knowledge tree including a total of 5,531 knowledge points. We retained 2,299 knowledge points more relevant to multimodal mathematics in K-12. Subsequently, all questions were classified according to knowledge points by GPT-4 and a fine-tuned LLM, followed by manual multi-level verification. Questions that did not match any K-12 multimodal mathematics knowledge points were filtered out.

The CMMaTH dataset is primarily used to evaluate multimodal reasoning capabilities in K-12 educational scenarios. We compared the current mainstream multimodal mathematical datasets and large model benchmarks in Table 3. Compared to existing multimodal benchmarks and multimodal reasoning benchmarks, the CMMaTH dataset has the following characteristics:
Extreme Diversity Currently, there is a severe lack of high-quality Chinese multimodal mathematics datasets. MATH-VISION lacks a Chinese component, the MATH-VISTA dataset contains only a small number of Chinese samples, and CMMMU contains only 540 math problems, which are not fine-grained and comprehensive enough. We have included about 23k fine-grained multimodal mathematics assessment samples, covering 13 K12 mathematics visual categories, making it the largest known multimodal Chinese dataset to date.
Real and High Quality & Multilingual MathVista features a substantial number of problems that are associated with natural and synthetic images. However, these images do not accurately represent the genuine data distribution encountered in K12 mathematics educational settings. OlympiadBench is an Olympiad-level bilingual multimodal benchmark. However, this benchmark is overly challenging and deviates from the application of LMM in real K12 multimodal math scenarios. Additionally, the variety of multimodal visual elements is relatively limited. Instead, we collect multimodal data specifically tailored to the K12 education context. Additionally, MathVista incorporates a significant amount of data from GeoQA and synthetic images, which have relatively poor image quality. Our multimodal visual image elements have all undergone stringent image quality assessments. Unlike CMMMU, CEval, and CMath, our dataset is a bilingual dataset that considers a large number of Chinese scenes. In addition to the text of the questions being in Chinese, the visual elements related to the questions also contain Chinese text/symbols.
High-quality Fine-grained Annotation and Evaluation Tool Every question in our dataset is meticulously annotated with standardized answers, solutions expressed in natural language, associated multimodal knowledge points, visual element categories, and K-12 grade levels. This fine-grained annotation enables a more nuanced evaluation of multimodal mathematical proficiency within the K-12 educational context. While MathVista and GeoEval rely on GPT-4 for answer extraction and validation, we introduce an open-source model named GradeGPT. GradeGPT stands out by providing a stable, cost-free, and swift accuracy evaluation specifically tailored for the CMMaTH dataset.

We evaluated the results of mainstream multimodal large models and mathematical expert models in Table B. We analyzed the trend of existing large models in descending with problems and conditions, as well as the effectiveness of techniques such as Cross-Lingual Prompting in solving Chinese multimodal mathematical problems. The experimental in Table 4 results indicates that our data exhibits extremely strong diversity and relatively challenging reasoning depth. Figure 1 and Table 4 shows models such as GPT4V struggle to comprehend our multimodal content and reasoning questions effectively, resulting in significant performance gaps between open-source and proprietary models. In certain rare visual domains, multimodal large models achieve very low reasoning outcomes.
Accuracy on various question types. We evaluated the accuracy of GPT4V on various target-solving tasks in Figure 4. The results indicate that when solving free-form problems, especially those with more diverse targets such as expressions, coordinates, and conclusion judgments, the multimodal large language model shows poorer performance.
Is OCR information sufficient for CMMaTH? We also referred to works like MathVista, attempting to use LLMs combined with OCR information from diagrams to assist in mathematical reasoning in Table 4. We found that, in our benchmark, a small amount of OCR information (such as mathematical symbols in diagrams, axis values, and image titles) made it very difficult to complete our multimodal mathematical reasoning tasks. The results indicate that solving problems in CMMaTH requires stronger multimodal mathematical chart capabilities, beyond just OCR.
K12 Multimodal Knowledge Richness of current LMMs. We systematically evaluated the proficiency of existing multimodal large models in the K12 domain regarding multimodal reasoning skills in Figure 5. The results revealed a significant knowledge gap in existing multimodal K12 educational resources. Compared to other existing LMMs, GPT4V possesses a richer knowledge base, thereby substantially reducing the illusion of reasoning in multimodal mathematical inference.

We also attempted several multilingual Chain-of-Thought approaches such as En-CoT, CLP(Cross-Lingual Prompting) used by Qin et al. (2023) to observe whether multimodal mathematical problems could be enhanced through context learning techniques without training. The results indicate that multilingual CoT methods face challenges in solving, possibly due to the abundance of Chinese contextual text in the image content, which may necessitate the model to demonstrate excellent cross-lingual OCR capabilities. We have included more details on the implementation of Cross-Lingual Prompting and En-CoT on the CMMaTH dataset in the Table 5.

We conducted a detailed analysis and evaluation of GPT4V on CMMaTH-testmin, categorizing errors into four types: perceptual errors, reasoning errors, calculation errors, and Reject Errors. The error type distribution of GPT4V on CMMaTH is shown in the Figure 6.
Perception Errors
Perception Error refers to the model’s erroneous interpretation and utilization of diagram content during reasoning. For example, incorrect OCR, misidentification of numerical relationships, geometric relationships, logical relationships, etc.
Reasoning Errors
Reasoning Error are quite common during the solving process. For instance, the model may misinterpret symbols or use incorrect logic or knowledge for inference. The frequency of Reasoning Errors reflects the model’s logical and mathematical reasoning capabilities.
Calculation Errors
Calculation Error refers to the model performing incorrect mathematical operations, such as writing equations or solving equations incorrectly.
Reject Errors
Reject Error refers to the model’s inability to solve a problem that is actually solvable. The frequency of such errors reflects the model’s ability to follow instructions.