AMuRD: Annotated Arabic-English Receipt Dataset for Key Information Extraction and Classification

Our first step was to gather a large and diverse collection of receipts. To do this, we utilized the DISCO application²²2https://discoapp.ai/, a platform that allows registered users to upload real receipts from a variety of commercial establishments, including restaurants, supermarkets, and retail stores. We collected a total of one hundred thousand receipts, carefully chosen to cover a wide range of industries, products, and transaction types. This approach ensured that our dataset was not only rich and relevant but also accurately reflected the complexity and variability of real-world transactional documents. Figure LABEL:fig:examples_receipts provides a visual representation of the diversity of receipts included in our dataset.

To ensure consistency and precision in the annotation process, we developed comprehensive guidelines. These guidelines provide clear instructions and definitions for each element that needs to be annotated, including item names, classes, weight, number of units, size of units, price, total price, packaging, units, and brand names.

Our annotation team consisted of five expert annotators. Each annotator was provided with these guidelines to ensure a uniform process of annotation across the entire dataset. The use of a small, dedicated team of annotators helped to maintain high standards of accuracy and consistency in the annotations. This rigorous approach to annotation has contributed to the high quality of our dataset.

Once we had our collection of receipts, the annotators carefully examined each one against several criteria. These criteria included the clarity of the text, the completeness of the receipt image, and the character of the document (it had to be a receipt and not an unrelated bill, such as an electricity bill). This selection ensured that the subsequent annotation process was based on high-quality images and relevant documents.

In this step, each receipt was processed through the Connectionist Text Proposal Network (CTPN) (Tian et al., 2016), a tool known for its efficacy in detecting text on a line-by-line basis across various scales and languages, without necessitating further post-processing. Its innovative approach, diverging from traditional bottom-up methods that require extensive post-filtering, reaches an F-measures of 0.88 and 0.61 on the ICDAR 2013 and 2015 benchmarks (Karatzas et al., 2013, 2015), respectively. This step was crucial for preparing each receipt for detailed annotation by isolating every line of text for further analysis.

After a receipt passed the initial screening, it was then passed on to annotators. The annotators went through each line item on the receipt, carefully transcribing every entry. This process prepared the data for the next phase of processing.

This phase involved expert annotators in the process of information extraction. They took the itemized data and processed it, labeling and classifying each element. This stage is crucial as it transforms the raw, annotated data into structured information that is suitable for further analysis.

The final step in our annotation process was overseen by the head of the annotator team. This individual checked the accuracy of the information extraction and classification. If any discrepancies or errors were identified, the previous steps were reviewed. This rigorous validation process ensured the integrity and reliability of our dataset.

AMuRD consists of $47,720$ items, encompassing both Arabic and English content, and serves as a comprehensive resource for key information extraction. Each receipt image contains approximately critical text fields, including item names, unit prices, and total prices. For a visual representation, refer to Figure LABEL:fig:examples_receipts, which showcases sample receipt images.

Item classification from receipts presents unique challenges due to the wide variety of item names, and categories, and the multilingual nature of the data. Our methodology to tackle these challenges comprises preprocessing, feature extraction, model training, and evaluation.

Information extraction from receipts is critical for transforming unstructured data into actionable insights. We deployed a suite of state-of-the-art models to address the intricacies of this task.

We evaluate our models using standard metrics such as precision, recall, F1-score, and accuracy. By using these metrics, we ensure a comprehensive and robust evaluation of our models’ performance. Our dataset was split into three subsets: a test set comprising 4,773 samples, a training set with 38,177 samples, and a validation set containing 4,772 samples.

This section outlines the experimental setup used for training the LLaMA V1 and V2 model, leveraging the DeepSpeed framework (Rajbhandari et al., 2020; Rasley et al., 2020). We conduct fine-tuning using the official LLaMA model with the Huggingface Transformers toolkit (Wolf et al., 2020). Our primary goal during training is to fine-tune LLaMa to produce the reference response while minimizing the cross-entropy loss. Considering the time and computational resource limitations, we choose a parameter-efficient fine-tuning approach over full-model fine-tuning for most of our experiments by using DeepSpeed. DeepSpeed offers techniques and automated parameter tuning to optimize training efficiency and memory utilization. We have tailored the training process by utilizing DeepSpeed’s configuration options to achieve the best results. We used mixed precision training with bfloat16 (bf16) precision, a technique that accelerates training while preserving model accuracy. We chose the AdamW optimizer (Loshchilov and Hutter, 2017) as our optimizer and let DeepSpeed automatically determine its hyperparameters. To regulate the learning rate, we employed the WarmupDecayLR scheduler. All experiments are carried out on 4 Nvidia A100 48GB GPUs.

We have employed a variety of machine learning algorithms and Large Language Models (LLMs), both in their original and fine-tuned forms, to gauge their efficacy in accurately classifying receipt items.

Traditional machine learning models form our baseline. The Gradient Boost algorithm shows moderate precision and recall, leading to an F1 score of 73.19% and an accuracy of 70.32%. The K-Nearest Neighbors (KNN) algorithm significantly improves on this, with both precision and recall just above 90%, resulting in a similar F1 score and accuracy. Naive Bayes further pushes the boundary with slightly improved scores across all metrics, notably a 92.20% accuracy. Random Forest classifiers show a robust performance with an F1 score of 94.18% and an accuracy of 94.32%, while Support Vector Machine (SVM) tops the traditional models with impressive precision and recall, both slightly over 96%, and the highest F1 score and accuracy, 96.30% and 96.43% respectively. These models establish a strong benchmark for item classification as shown in Table. 2.

The non-fine-tuned LLMs, display varied results. DeciLM has a relatively low precision but higher recall, which is reflected in a modest F1 score of 16.17% and accuracy of 8.79%. LLaMA V1 and V2 without fine-tuning show limited precision and recall, resulting in F1 scores of 19.13% and 8.19% respectively. Similarly, Mistral, LLaMA V2 (13B), and Mixtral display challenges in achieving high scores, with F1 results not exceeding 31.07%. GPT-3.5 achieves an F1 score of 21.83% and an accuracy of 12.25%, indicating that without fine-tuning, even the most powerful LLMs struggle with the specificity required for item classification on our dataset.

Table 3 presents the nuanced capabilities of our fine-tuned models, LLaMA V1 and LLaMA V2, both wielding 7 billion parameters. The comprehensive results across diverse information extraction and classification categories speak to their effectiveness. In discerning brand details, the models exhibit high prowess, with F1 scores soaring above 97%, reflecting their acute brand recognition capabilities. Weight and quantity details, often challenging due to their variability, are accurately captured by the models, as evidenced by F1 scores near 96% for weight and nearly 99% for both number of units and size of units. Such high scores are indicative of the models’ precise identification and classification processes.

The adeptness of these models extends to the crucial financial metrics of total price and price, where they perform well, achieving F1 scores and accuracies above 96%. This demonstrates their capacity to handle the intricacies of numerical data extraction from receipts, which can vary greatly in format. In other intricate categories like packaging and units, the models again prove their merit with F1 scores in the mid-90s and accuracies that reflect a high level of consistency and reliability. Overall, the models achieve remarkable F1 scores of above 97% and accuracies nearing 95%, showcasing their comprehensive effectiveness in both information extraction and classification tasks.

The exploration of zero-shot capabilities within our study presents a novel approach to information extraction without direct examples for training. Table 4 showcases the performance metrics of various language models, including LLaMA V1 and V2, under zero-shot conditions across a spectrum of categories. Zero-shot, by design, challenges models to apply learned knowledge from unrelated tasks to make accurate predictions on unseen data, a task that demands a high degree of generalization.

In the context of zero-shot information extraction, the models exhibited varied levels of proficiency across different categories. Notably, in the Brand and Weight categories, the performance metrics indicate a struggle to maintain high precision and recall, which translates to modest F1 scores and accuracy. This trend highlights the inherent challenge of zero-shot performance, where models must rely solely on their pre-existing knowledge base without the benefit of specific training examples.

Among the models, Zephyr and Falcon showed relatively higher F1 scores in several categories, suggesting a better generalization capability in certain contexts. However, LLaMA V1 and V2’s performance, while not leading in the zero-shot scenario, still demonstrates the potential of large language models to adapt their extensive pre-trained knowledge to specific tasks.

The Number of Units and Size of Units categories further illustrate the models’ challenges in accurately interpreting and extracting numerical data without explicit examples. However, some models like LLaMA V2 (13B) show promising results, indicating a capacity for significant improvement with fine-tuning and targeted training.

The Total Price and Price extraction tasks reveal an interesting aspect of zero-shot learning, where even without specific training, models like LLaMA V1 and V2 can leverage their linguistic and numerical understanding to achieve modest success in extracting financial information.

Packaging (Pack) and Unit categories underscore the complexity of zero-shot information extraction, with generally lower performance across the models because such information is typically not explicitly mentioned in items. Hence, these categories likely suffer from the lack of direct examples that would help models understand the specific context and terminology associated with packaging information and unit measurements.

Table 5 delineates the performance of various language models in a one-shot learning scenario across multiple categories. In one-shot learning, the models are expected to generalize from a single example to new instances, which is a challenging task, especially in the context of information extraction from diverse datasets like receipts.

When comparing the models’ one-shot capabilities, it becomes evident that certain models excel in specific categories. For instance, in the Brand category, Falcon demonstrates the highest precision, while Mixtral leads with the highest recall. However, it is LLaMA V1 that balances both metrics most effectively, achieving the highest F1 score and accuracy overall.

In the Weight category, once again, LLaMA V1 stands out with the highest F1 score and accuracy, suggesting that it has a better understanding of contextual cues related to product weight, despite the varied ways in which this information can be presented in receipts.

The Number of Units and Size of Units categories are particularly demanding since they require the model to interpret potentially ambiguous numerical data accurately. In both categories, LLaMA V2 (13B) shows an impressive recall, but it is LLaMA V1 that maintains a superior balance as reflected in its F1 scores and accuracy.

The Total Price and Price categories are dominated by LLaMA V1 in terms of F1 and accuracy, reinforcing its superior performance in the one-shot extraction of numerical data.

Packaging (Pack) and Units represent categories where the models generally perform less effectively, with Zephyr showing an unexpectedly higher F1 score in Pack, and LLaMA V2 (13B) leading in the Units category.

Finally, in few-shot learning, as Table 6 shows, the models are provided with a few examples (2 or 3) from which to learn, allowing them to better understand the nuances of the data.

When examining the performance improvements between one-shot and few-shot scenarios, it is clear that additional examples aid the models significantly. For example, LLaMA V2 (7B) jumps from an F1 score of 46.73% to 78.52% in the Overall category when moving from one to two shots, and LLaMA V1 shows similar improvements.

Across specific categories, the advantages of few-shot learning become even more pronounced. In Brand extraction, LLaMA V2 (7B) improves markedly in precision, recall, F1, and accuracy with each additional example provided. In the Weight and Number of Units categories, we see substantial enhancements in F1 scores and accuracies for almost all models, with LLaMA V2 (13B) showing significant jumps in performance.

The Total Price and Price categories also benefit from the few-shot approach, with most models showing noticeable improvements in precision and recall, leading to higher F1 scores. This suggests that with even a small increase in data, the models can better capture the patterns necessary for extracting financial information.

Interestingly, the Pack and Units categories, which were challenging in the one-shot setting, see varied improvements with few-shot learning. While some models like LLaMA V2 (7B) and Mistral exhibit substantial gains, others like Falcon and Zephyr show little to no improvement, indicating that these categories might be inherently more challenging or require a different approach for learning.