-
Unraveling the Truth: Do LLMs really Understand Charts? A Deep Dive into Consistency and Robustness
Authors:
Srija Mukhopadhyay,
Adnan Qidwai,
Aparna Garimella,
Pritika Ramu,
Vivek Gupta,
Dan Roth
Abstract:
Chart question answering (CQA) is a crucial area of Visual Language Understanding. However, the robustness and consistency of current Visual Language Models (VLMs) in this field remain under-explored. This paper evaluates state-of-the-art VLMs on comprehensive datasets, developed specifically for this study, encompassing diverse question categories and chart formats. We investigate two key aspects…
▽ More
Chart question answering (CQA) is a crucial area of Visual Language Understanding. However, the robustness and consistency of current Visual Language Models (VLMs) in this field remain under-explored. This paper evaluates state-of-the-art VLMs on comprehensive datasets, developed specifically for this study, encompassing diverse question categories and chart formats. We investigate two key aspects: 1) the models' ability to handle varying levels of chart and question complexity, and 2) their robustness across different visual representations of the same underlying data. Our analysis reveals significant performance variations based on question and chart types, highlighting both strengths and weaknesses of current models. Additionally, we identify areas for improvement and propose future research directions to build more robust and reliable CQA systems. This study sheds light on the limitations of current models and paves the way for future advancements in the field.
△ Less
Submitted 15 July, 2024;
originally announced July 2024.
-
Is this a bad table? A Closer Look at the Evaluation of Table Generation from Text
Authors:
Pritika Ramu,
Aparna Garimella,
Sambaran Bandyopadhyay
Abstract:
Understanding whether a generated table is of good quality is important to be able to use it in creating or editing documents using automatic methods. In this work, we underline that existing measures for table quality evaluation fail to capture the overall semantics of the tables, and sometimes unfairly penalize good tables and reward bad ones. We propose TabEval, a novel table evaluation strateg…
▽ More
Understanding whether a generated table is of good quality is important to be able to use it in creating or editing documents using automatic methods. In this work, we underline that existing measures for table quality evaluation fail to capture the overall semantics of the tables, and sometimes unfairly penalize good tables and reward bad ones. We propose TabEval, a novel table evaluation strategy that captures table semantics by first breaking down a table into a list of natural language atomic statements and then compares them with ground truth statements using entailment-based measures. To validate our approach, we curate a dataset comprising of text descriptions for 1,250 diverse Wikipedia tables, covering a range of topics and structures, in contrast to the limited scope of existing datasets. We compare TabEval with existing metrics using unsupervised and supervised text-to-table generation methods, demonstrating its stronger correlation with human judgments of table quality across four datasets.
△ Less
Submitted 20 June, 2024;
originally announced June 2024.
-
RE$^2$: Region-Aware Relation Extraction from Visually Rich Documents
Authors:
Pritika Ramu,
Sijia Wang,
Lalla Mouatadid,
Joy Rimchala,
Lifu Huang
Abstract:
Current research in form understanding predominantly relies on large pre-trained language models, necessitating extensive data for pre-training. However, the importance of layout structure (i.e., the spatial relationship between the entity blocks in the visually rich document) to relation extraction has been overlooked. In this paper, we propose REgion-Aware Relation Extraction (RE$^2$) that lever…
▽ More
Current research in form understanding predominantly relies on large pre-trained language models, necessitating extensive data for pre-training. However, the importance of layout structure (i.e., the spatial relationship between the entity blocks in the visually rich document) to relation extraction has been overlooked. In this paper, we propose REgion-Aware Relation Extraction (RE$^2$) that leverages region-level spatial structure among the entity blocks to improve their relation prediction. We design an edge-aware graph attention network to learn the interaction between entities while considering their spatial relationship defined by their region-level representations. We also introduce a constraint objective to regularize the model towards consistency with the inherent constraints of the relation extraction task. Extensive experiments across various datasets, languages and domains demonstrate the superiority of our proposed approach.
△ Less
Submitted 3 June, 2024; v1 submitted 23 May, 2023;
originally announced May 2023.
-
Autonomous Vehicles in 5G and Beyond: A Survey
Authors:
Saqib Hakak,
Thippa Reddy Gadekallu,
Swarna Priya Ramu,
Parimala M,
Praveen Kumar Reddy Maddikunta,
Chamitha de Alwis,
Madhusanka Liyanage
Abstract:
Fifth Generation (5G) technology is an emerging and fast adopting technology which is being utilized in most of the novel applications that require highly reliable low-latency communications. It has the capability to provide greater coverage, better access, and best suited for high density networks. Having all these benefits, it clearly implies that 5G could be used to satisfy the requirements of…
▽ More
Fifth Generation (5G) technology is an emerging and fast adopting technology which is being utilized in most of the novel applications that require highly reliable low-latency communications. It has the capability to provide greater coverage, better access, and best suited for high density networks. Having all these benefits, it clearly implies that 5G could be used to satisfy the requirements of Autonomous vehicles. Automated driving Vehicles and systems are developed with a promise to provide comfort, safe and efficient drive reducing the risk of life. But, recently there are fatalities due to these autonomous vehicles and systems. This is due to the lack of robust state-of-art which has to be improved further. With the advent of 5G technology and rise of autonomous vehicles (AVs), road safety is going to get more secure with less human errors. However, integration of 5G and AV is still at its infant stage with several research challenges that needs to be addressed. This survey first starts with a discussion on the current advancements in AVs, automation levels, enabling technologies and 5G requirements. Then, we focus on the emerging techniques required for integrating 5G technology with AVs, impact of 5G and B5G technologies on AVs along with security concerns in AVs. The paper also provides a comprehensive survey of recent developments in terms of standardisation activities on 5G autonomous vehicle technology and current projects. The article is finally concluded with lessons learnt, future research directions and challenges.
△ Less
Submitted 21 July, 2022;
originally announced July 2022.
