-
Improving Zero-Shot Text Matching for Financial Auditing with Large Language Models
Authors:
Lars Hillebrand,
Armin Berger,
Tobias Deußer,
Tim Dilmaghani,
Mohamed Khaled,
Bernd Kliem,
Rüdiger Loitz,
Maren Pielka,
David Leonhard,
Christian Bauckhage,
Rafet Sifa
Abstract:
Auditing financial documents is a very tedious and time-consuming process. As of today, it can already be simplified by employing AI-based solutions to recommend relevant text passages from a report for each legal requirement of rigorous accounting standards. However, these methods need to be fine-tuned regularly, and they require abundant annotated data, which is often lacking in industrial envir…
▽ More
Auditing financial documents is a very tedious and time-consuming process. As of today, it can already be simplified by employing AI-based solutions to recommend relevant text passages from a report for each legal requirement of rigorous accounting standards. However, these methods need to be fine-tuned regularly, and they require abundant annotated data, which is often lacking in industrial environments. Hence, we present ZeroShotALI, a novel recommender system that leverages a state-of-the-art large language model (LLM) in conjunction with a domain-specifically optimized transformer-based text-matching solution. We find that a two-step approach of first retrieving a number of best matching document sections per legal requirement with a custom BERT-based model and second filtering these selections using an LLM yields significant performance improvements over existing approaches.
△ Less
Submitted 14 August, 2023; v1 submitted 11 August, 2023;
originally announced August 2023.
-
sustain.AI: a Recommender System to analyze Sustainability Reports
Authors:
Lars Hillebrand,
Maren Pielka,
David Leonhard,
Tobias Deußer,
Tim Dilmaghani,
Bernd Kliem,
Rüdiger Loitz,
Milad Morad,
Christian Temath,
Thiago Bell,
Robin Stenzel,
Rafet Sifa
Abstract:
We present sustainAI, an intelligent, context-aware recommender system that assists auditors and financial investors as well as the general public to efficiently analyze companies' sustainability reports. The tool leverages an end-to-end trainable architecture that couples a BERT-based encoding module with a multi-label classification head to match relevant text passages from sustainability report…
▽ More
We present sustainAI, an intelligent, context-aware recommender system that assists auditors and financial investors as well as the general public to efficiently analyze companies' sustainability reports. The tool leverages an end-to-end trainable architecture that couples a BERT-based encoding module with a multi-label classification head to match relevant text passages from sustainability reports to their respective law regulations from the Global Reporting Initiative (GRI) standards. We evaluate our model on two novel German sustainability reporting data sets and consistently achieve a significantly higher recommendation performance compared to multiple strong baselines. Furthermore, sustainAI is publicly available for everyone at https://sustain.ki.nrw/.
△ Less
Submitted 26 May, 2023; v1 submitted 15 May, 2023;
originally announced May 2023.
-
Zero-Shot Text Matching for Automated Auditing using Sentence Transformers
Authors:
David Biesner,
Maren Pielka,
Rajkumar Ramamurthy,
Tim Dilmaghani,
Bernd Kliem,
Rüdiger Loitz,
Rafet Sifa
Abstract:
Natural language processing methods have several applications in automated auditing, including document or passage classification, information retrieval, and question answering. However, training such models requires a large amount of annotated data which is scarce in industrial settings. At the same time, techniques like zero-shot and unsupervised learning allow for application of models pre-trai…
▽ More
Natural language processing methods have several applications in automated auditing, including document or passage classification, information retrieval, and question answering. However, training such models requires a large amount of annotated data which is scarce in industrial settings. At the same time, techniques like zero-shot and unsupervised learning allow for application of models pre-trained using general domain data to unseen domains.
In this work, we study the efficiency of unsupervised text matching using Sentence-Bert, a transformer-based model, by applying it to the semantic similarity of financial passages. Experimental results show that this model is robust to documents from in- and out-of-domain data.
△ Less
Submitted 28 October, 2022;
originally announced November 2022.
-
Towards automating Numerical Consistency Checks in Financial Reports
Authors:
Lars Hillebrand,
Tobias Deußer,
Tim Dilmaghani,
Bernd Kliem,
Rüdiger Loitz,
Christian Bauckhage,
Rafet Sifa
Abstract:
We introduce KPI-Check, a novel system that automatically identifies and cross-checks semantically equivalent key performance indicators (KPIs), e.g. "revenue" or "total costs", in real-world German financial reports. It combines a financial named entity and relation extraction module with a BERT-based filtering and text pair classification component to extract KPIs from unstructured sentences bef…
▽ More
We introduce KPI-Check, a novel system that automatically identifies and cross-checks semantically equivalent key performance indicators (KPIs), e.g. "revenue" or "total costs", in real-world German financial reports. It combines a financial named entity and relation extraction module with a BERT-based filtering and text pair classification component to extract KPIs from unstructured sentences before linking them to synonymous occurrences in the balance sheet and profit & loss statement. The tool achieves a high matching performance of $73.00$% micro F$_1$ on a hold out test set and is currently being deployed for a globally operating major auditing firm to assist the auditing procedure of financial statements.
△ Less
Submitted 11 November, 2022;
originally announced November 2022.
-
KPI-BERT: A Joint Named Entity Recognition and Relation Extraction Model for Financial Reports
Authors:
Lars Hillebrand,
Tobias Deußer,
Tim Dilmaghani,
Bernd Kliem,
Rüdiger Loitz,
Christian Bauckhage,
Rafet Sifa
Abstract:
We present KPI-BERT, a system which employs novel methods of named entity recognition (NER) and relation extraction (RE) to extract and link key performance indicators (KPIs), e.g. "revenue" or "interest expenses", of companies from real-world German financial documents. Specifically, we introduce an end-to-end trainable architecture that is based on Bidirectional Encoder Representations from Tran…
▽ More
We present KPI-BERT, a system which employs novel methods of named entity recognition (NER) and relation extraction (RE) to extract and link key performance indicators (KPIs), e.g. "revenue" or "interest expenses", of companies from real-world German financial documents. Specifically, we introduce an end-to-end trainable architecture that is based on Bidirectional Encoder Representations from Transformers (BERT) combining a recurrent neural network (RNN) with conditional label masking to sequentially tag entities before it classifies their relations. Our model also introduces a learnable RNN-based pooling mechanism and incorporates domain expert knowledge by explicitly filtering impossible relations. We achieve a substantially higher prediction performance on a new practical dataset of German financial reports, outperforming several strong baselines including a competing state-of-the-art span-based entity tagging approach.
△ Less
Submitted 3 August, 2022;
originally announced August 2022.