LANS: A Layout-Aware Neural Solver for Plane Geometry Problem

GPS is a special type of multimodal reasoning that examines geometric spatial structure cognition and mathematical logical reasoning, and also requires the application of geometric theorem knowledge, which make it highly challenging. Existing works of GPS can be classified into two categories: symbolic solvers and neural solvers. The symbolic solvers Seo et al. (2015); Sachan and Xing (2017); Lu et al. (2021); Peng et al. (2023) parse the diagram and textual problem into a unified formal language first, and then perform symbolic reasoning by path search and condition matching based on the geometric theorem knowledge. However, symbolic solvers are carefully designed with complex rules and are hard to extend. The neural solvers treat GPS as a visual question answering task and design a special interpretable program to represent the problem-solving process. NGS Chen et al. (2021) and Geoformer Chen et al. (2022) use auxiliary self-supervised tasks such as location prediction, elements prediction, and knowledge classification to boost cross-modal semantic representation. PGPSNet Zhang et al. (2023) expresses the geometry diagram with textual clauses and fuses multi-modal information through structural and semantic pre-training, data augmentation, and self-limited decoding. SCA-GPS Ning et al. (2023) tries to align characters in text and diagram and enhance the diagram understanding through multi-label classification and masked image modeling pre-training. Although existing neural solvers have achieved impressive performance, they are still coarse-grained at the modal understanding and fusion, especially for geometry diagrams with complex layouts. In this paper, we propose a layout-aware neural solver to improve the understanding and fusion of geometry diagrams and therefore promote GPS.

Multimodal pre-training realizes alignment and understanding between different modalities by a series of designed auxiliary tasks and then applies to the specific downstream tasks. Common strategies involve image-text contrastive learning Radford et al. (2021), image-text matching Kim et al. (2021), image-grounded text generation Cho et al. (2021), and masked object classification Li et al. (2020). With a large amount of pre-training data, these strategies exhibit good performance in multimodal tasks for natural images. However, their alignment methods are coarse-grained and straightforward and do not fit for complex multi-level and fine-grained tasks. Most relevant to our work is the research on document analysis Liu et al. (2023a). Existing advanced document pre-training methods Xu et al. (2020, 2021) incorporate textual and visual blocks with fine-grained position embeddings, and adopt masked visual-language modeling and text-image alignment to pretrain document layout, whereas they still do not apply to GPS due to the specificity of geometry objects and small-scale of GPS datasets. DocFormer Appalaraju et al. (2021) and LayoutReader Wang et al. (2021) employ meticulously designed attention mechanisms targeting information within text boxes to enhance their perception abilities regarding document content. Our LANS proposes targeted and data-efficient pre-training methods and a geometry layout-aware attention to implement geometry layout awareness.

To fully understand and represent the geometry diagram, we propose a layout-aware neural solver called LANS as displayed in Figure 2. First, the diagram is parsed into the textual clauses using the geometry diagram parser PGDPNet Zhang et al. (2022), where the structural clauses $T_{stru}$ describe the connection relations among geometric primitives and the semantic clauses $T_{sem}$ depict the semantic relations between non-geometric primitives and geometric primitives Zhang et al. (2023). Besides, the visual information of diagram image is represented as patches. Therefore, the input of LANS could be further expressed as $\left\{D\!=\!\left\{d_{i}\right\}_{i=1}^{N_{D}},T\!\!=\!\!\left\{T_{stru},T_{sem},T_{prob}\right\}\!\!=\!\!\left\{t_{j}\right\}_{j=1}^{N_{T}}\right\}$ after token concatenation, where $N_{D}$ is the diagram patch number and $N_{T}$ is the text token number. Then, these modal tokens are fed into the multimodal layout-aware pre-trained language model (MLA-PLM) and input into the bidirectional GRU encoder to perform fusion encoding. Next, the mixed encoding context $H=\left\{h_{i}\right\}_{i=1}^{N_{D}\!+\!N_{T}}$ leverages the layout-aware fusion attention (LA-FA) to further boost diagram layout awareness. Finally, the enhanced context is decoded by the self-limited GRU decoder and generates the sequential solution program $S$ in the manner of autoregressive.

Geometry problems are often solved by humans by depicting the geometric structure in visual form no matter whether it has the geometry diagram or not. Previous neural geometric solvers, such as the NGS Chen et al. (2021), PGPSNet Zhang et al. (2023) and SCA-GPS Ning et al. (2023), do not utilize the diagram layout adequately, thus resulting in unsatisfactory performance of GPS. In this paper, we propose the multimodal layout-aware pre-trained language model (MLA-PLM), with two pre-training strategies: structural-semantic pre-training (SSP) and point matching pre-training (PMP) illustrated in Figure 3, to boost the diagram layout-aware ability during the pre-training stage.

Although LANS has already acquired a certain level of layout understanding through the pre-training strategies above, this ability can fade to some extent during downstream training because of the different training targets of GPS. To compensate for the loss of layout awareness in the GPS training phase, we propose layout-aware fusion attention (LA-FA) to enhance the intra-modal and cross-modal token fusion. LA-FA is located between the bidirectional GRU encoder and the self-limited GRU decoder.

As shown in Figure 4, the LA-FA module is similar to the transformer encoder block Vaswani et al. (2017) which also contains layer normalization, feed-forward layer, and residual connection except the layout-aware self-attention. Our layout-aware self-attention uses the carefully designed layout-aware attention mask which allows visibility to all intra-modality tokens but restricts cross-modality visibility if the textual point is not inside the image patch in the visual space. Specifically, we construct the mask matrix $M_{i,j}\;(1\!\leq\!i,j\!\leq\!N_{D}\!+\!N_{T})$, which consists of value 0 as invisible and value 1 as visible:

$$\textstyle M_{i,j}\!=\!\left\{\begin{array}[]{ll}1,&\text{if }(i,j)\in V\!V\\ 1,&\text{if }(i,j)\in T\!T\\ 1,&\text{if }(i,j)\in V\!T\&\text{Pos}(t_{j})\in\text{Reg}(d_{i})\\ 0,&\text{otherwise}\end{array}\right.$$

(3)

where $V\!V=\{(i,j)\,|\,1\!\leq\!i,j\!\leq\!N_{D}\}$ is the mask region of visual intra-modality, $T\!T=\{(i,j)\,|\,N_{D}\!\!+\!\!1\!\leq\!i,j\!\leq\!N_{D}\!\!+\!\!N_{T}\}$ is the mask region of textual intra-modality, $V\!T=\{(i,j)\,|\,1\!\leq\!i,j\!\leq\!N_{D}\!+\!N_{T}\}\!-\!V\!V\!-\!T\!T$ is the mask region of cross-modality, $\text{Pos}(t_{j})$ denotes the visual position of point token $t_{j}$ and $\text{Reg}(d_{i})$ refers to the visual region of image patch $d_{i}$. Moreover, layout-aware fusion-attention (LA-FA) could be computed by:

$$\text{LA-FA}(Q,K,V,M)\!=\!\text{softmax}\!\left(\frac{QK^{T}}{\sqrt{m_{k}}}\cdot M\!\right)V$$

(4)

where $Q,K,V$ are query matrix, key matrix, and value matrix all transformed from encoding context $H$, and $m_{k}$ is the dimension of the key vector.

In summary, in the process of cross-modal fusion, LA-FA leverages the point position to guide the attention between diagram and text, strengthening the understanding of diagram layout. For mitigating the optimization burden, we only use one LA-FA block, as adding more blocks does not bring extra improvement according to our experiments.

We compare LANS with state-of-the-art models, including neural solvers, symbolic solvers, and multimodal large models in Table 1, in terms of both performance and parameter quantity. The results indicate that our LANS achieves excellent model performance by incorporating efficient parameters.

As to symbolic solvers, InterGPS Lu et al. (2021) solved geometry problems by searching and matching with unified formal language. According to the input source of formal language, InterGPS presents three types of results, e.g., “Predict" means that all formal language is predicted by its parsers, “Digram GT" denotes that formal clauses of diagram use ground truth, and “All GT" indicates that formal clauses of diagram and textual problem are all ground truth. GeoDRL Peng et al. (2023) improved the search strategy of Inter-GPS with logical graph deduction and deep reinforcement learning. Experimental results show that our LANS outperforms symbolic solvers on all datasets and in all evaluation metrics. Even compared with InterGPS (All GT) which uses annotated formal clauses designed carefully, LANS gains a 3.1% improvement in Completion and a 6.4% improvement in Choice mode on Geometry3K Dataset.

As to neural solvers, NGS Chen et al. (2021) and Geoformer Chen et al. (2022) relied primarily on textual problems to solve problems. Even though re-implementing them with the textual clauses parsed from the diagram and the same augmentation strategies, performance gaps between these two solvers and our LANS are still significant, 32.6% and 31.1% lower in Completion on PGPS9K, respectively. SCA-GPS Ning et al. (2023) shows similar performance as InterGPS (All GT) because diagram understanding methods, character alignments, and masked image modeling, are coarse-grained and ineffective. PGPSNet Zhang et al. (2023) employed textual clauses to model diagram layout but lost lots of visual information. Our LANS is enhanced at modal alignment and fusion for better layout awareness and surpasses PGPSNet by 7.1% and 4.0% in Completion on Geometry3K and PGPS9K. The improvements in Top-3 are less than in Completion because most of the correct solutions are concentrated among highly confident candidates.

Our approach far surpasses the performance of current multimodal large models. This may be attributed to the presence of complex symblic OCR information, layout details, and abstract elements in geometric images, where the perception capabilities of the Gemini Team et al. (2023) and GPT-4V Achiam et al. (2023b) models are insufficient. Similar phenomena have also been observed in the MathVista Lu et al. (2024) benchmark. For evaluation results and detailed information on the Completion and Choice multimodal large-scale models, please refer to section C.

We also conducted a case analysis to discuss the strengths and weaknesses of solvers. Figure 5 displays four plane geometry problems (a)-(d) involving various geometric layouts, and they rely on good layout awareness to solve them. In case (a), the position of C relative to F determines if $\angle$JIK and $\angle$DEC are corresponding or alternate angles. Results show LANS identifies corresponding angles accurately, unlike other solvers. In case (b), the perception of polygon edge number is the key to solving this problem. Contrary to LANS, other solvers cannot count edge numbers correctly through the diagram or textual clauses, resulting in a wrong solution. Case (c) is the same problem as shown in Figure 1 in which textual clauses cannot identify diagram uniquely. In contrast with PGPSNet, LANS can judge the orientation and type of $\angle$ABC and get the right solution.