-
Deciphering the Role of Representation Disentanglement: Investigating Compositional Generalization in CLIP Models
Authors:
Reza Abbasi,
Mohammad Hossein Rohban,
Mahdieh Soleymani Baghshah
Abstract:
CLIP models have recently shown to exhibit Out of Distribution (OoD) generalization capabilities. However, Compositional Out of Distribution (C-OoD) generalization, which is a crucial aspect of a model's ability to understand unseen compositions of known concepts, is relatively unexplored for the CLIP models. Our goal is to address this problem and identify the factors that contribute to the C-OoD…
▽ More
CLIP models have recently shown to exhibit Out of Distribution (OoD) generalization capabilities. However, Compositional Out of Distribution (C-OoD) generalization, which is a crucial aspect of a model's ability to understand unseen compositions of known concepts, is relatively unexplored for the CLIP models. Our goal is to address this problem and identify the factors that contribute to the C-OoD in CLIPs. We noted that previous studies regarding compositional understanding of CLIPs frequently fail to ensure that test samples are genuinely novel relative to the CLIP training data. To this end, we carefully synthesized a large and diverse dataset in the single object setting, comprising attributes for objects that are highly unlikely to be encountered in the combined training datasets of various CLIP models. This dataset enables an authentic evaluation of C-OoD generalization. Our observations reveal varying levels of C-OoD generalization across different CLIP models. We propose that the disentanglement of CLIP representations serves as a critical indicator in this context. By utilizing our synthesized datasets and other existing datasets, we assess various disentanglement metrics of text and image representations. Our study reveals that the disentanglement of image and text representations, particularly with respect to their compositional elements, plays a crucial role in improving the generalization of CLIP models in out-of-distribution settings. This finding suggests promising opportunities for advancing out-of-distribution generalization in CLIPs.
△ Less
Submitted 16 July, 2024; v1 submitted 8 July, 2024;
originally announced July 2024.
-
Language Plays a Pivotal Role in the Object-Attribute Compositional Generalization of CLIP
Authors:
Reza Abbasi,
Mohammad Samiei,
Mohammad Hossein Rohban,
Mahdieh Soleymani Baghshah
Abstract:
Vision-language models, such as CLIP, have shown promising Out-of-Distribution (OoD) generalization under various types of distribution shifts. Recent studies attempted to investigate the leading cause of this capability. In this work, we follow the same path, but focus on a specific type of OoD data - images with novel compositions of attribute-object pairs - and study whether such models can suc…
▽ More
Vision-language models, such as CLIP, have shown promising Out-of-Distribution (OoD) generalization under various types of distribution shifts. Recent studies attempted to investigate the leading cause of this capability. In this work, we follow the same path, but focus on a specific type of OoD data - images with novel compositions of attribute-object pairs - and study whether such models can successfully classify those images into composition classes. We carefully designed an authentic image test dataset called ImageNet-AO, consisting of attributes for objects that are unlikely encountered in the CLIP training sets. We found that CLIPs trained with large datasets such as OpenAI CLIP, LAION-400M, and LAION-2B show orders-of-magnitude improvement in effective compositional OoD generalization compared to both supervised models and CLIPs trained with smaller datasets, such as CC-12M and YFCC-15M. Our results provide evidence that the scale and diversity of training data and language supervision play a key role in unlocking the compositional generalization abilities of vision-language models.
△ Less
Submitted 27 March, 2024;
originally announced March 2024.
-
Human Emotions Analysis and Recognition Using EEG Signals in Response to 360$^\circ$ Videos
Authors:
Haseeb ur Rahman Abbasi,
Zeeshan Rashid,
Muhammad Majid,
Syed Muhammad Anwar
Abstract:
Emotion recognition (ER) technology is an integral part for developing innovative applications such as drowsiness detection and health monitoring that plays a pivotal role in contemporary society. This study delves into ER using electroencephalography (EEG), within immersive virtual reality (VR) environments. There are four main stages in our proposed methodology including data acquisition, pre-pr…
▽ More
Emotion recognition (ER) technology is an integral part for developing innovative applications such as drowsiness detection and health monitoring that plays a pivotal role in contemporary society. This study delves into ER using electroencephalography (EEG), within immersive virtual reality (VR) environments. There are four main stages in our proposed methodology including data acquisition, pre-processing, feature extraction, and emotion classification. Acknowledging the limitations of existing 2D datasets, we introduce a groundbreaking 3D VR dataset to elevate the precision of emotion elicitation. Leveraging the Interaxon Muse headband for EEG recording and Oculus Quest 2 for VR stimuli, we meticulously recorded data from 40 participants, prioritizing subjects without reported mental illnesses. Pre-processing entails rigorous cleaning, uniform truncation, and the application of a Savitzky-Golay filter to the EEG data. Feature extraction encompasses a comprehensive analysis of metrics such as power spectral density, correlation, rational and divisional asymmetry, and power spectrum. To ensure the robustness of our model, we employed a 10-fold cross-validation, revealing an average validation accuracy of 85.54\%, with a noteworthy maximum accuracy of 90.20\% in the best fold. Subsequently, the trained model demonstrated a commendable test accuracy of 82.03\%, promising favorable outcomes.
△ Less
Submitted 6 February, 2024;
originally announced February 2024.
-
Spuriosity Rankings for Free: A Simple Framework for Last Layer Retraining Based on Object Detection
Authors:
Mohammad Azizmalayeri,
Reza Abbasi,
Amir Hosein Haji Mohammad rezaie,
Reihaneh Zohrabi,
Mahdi Amiri,
Mohammad Taghi Manzuri,
Mohammad Hossein Rohban
Abstract:
Deep neural networks have exhibited remarkable performance in various domains. However, the reliance of these models on spurious features has raised concerns about their reliability. A promising solution to this problem is last-layer retraining, which involves retraining the linear classifier head on a small subset of data without spurious cues. Nevertheless, selecting this subset requires human s…
▽ More
Deep neural networks have exhibited remarkable performance in various domains. However, the reliance of these models on spurious features has raised concerns about their reliability. A promising solution to this problem is last-layer retraining, which involves retraining the linear classifier head on a small subset of data without spurious cues. Nevertheless, selecting this subset requires human supervision, which reduces its scalability. Moreover, spurious cues may still exist in the selected subset. As a solution to this problem, we propose a novel ranking framework that leverages an open vocabulary object detection technique to identify images without spurious cues. More specifically, we use the object detector as a measure to score the presence of the target object in the images. Next, the images are sorted based on this score, and the last-layer of the model is retrained on a subset of the data with the highest scores. Our experiments on the ImageNet-1k dataset demonstrate the effectiveness of this ranking framework in sorting images based on spuriousness and using them for last-layer retraining.
△ Less
Submitted 31 October, 2023;
originally announced November 2023.
-
Observation of high-energy neutrinos from the Galactic plane
Authors:
R. Abbasi,
M. Ackermann,
J. Adams,
J. A. Aguilar,
M. Ahlers,
M. Ahrens,
J. M. Alameddine,
A. A. Alves Jr.,
N. M. Amin,
K. Andeen,
T. Anderson,
G. Anton,
C. Argüelles,
Y. Ashida,
S. Athanasiadou,
S. Axani,
X. Bai,
A. Balagopal V.,
S. W. Barwick,
V. Basu,
S. Baur,
R. Bay,
J. J. Beatty,
K. -H. Becker,
J. Becker Tjus
, et al. (364 additional authors not shown)
Abstract:
The origin of high-energy cosmic rays, atomic nuclei that continuously impact Earth's atmosphere, has been a mystery for over a century. Due to deflection in interstellar magnetic fields, cosmic rays from the Milky Way arrive at Earth from random directions. However, near their sources and during propagation, cosmic rays interact with matter and produce high-energy neutrinos. We search for neutrin…
▽ More
The origin of high-energy cosmic rays, atomic nuclei that continuously impact Earth's atmosphere, has been a mystery for over a century. Due to deflection in interstellar magnetic fields, cosmic rays from the Milky Way arrive at Earth from random directions. However, near their sources and during propagation, cosmic rays interact with matter and produce high-energy neutrinos. We search for neutrino emission using machine learning techniques applied to ten years of data from the IceCube Neutrino Observatory. We identify neutrino emission from the Galactic plane at the 4.5$σ$ level of significance, by comparing diffuse emission models to a background-only hypothesis. The signal is consistent with modeled diffuse emission from the Galactic plane, but could also arise from a population of unresolved point sources.
△ Less
Submitted 10 July, 2023;
originally announced July 2023.
-
Classifying Text-Based Conspiracy Tweets related to COVID-19 using Contextualized Word Embeddings
Authors:
Abdul Rehman,
Rabeeh Ayaz Abbasi,
Irfan ul Haq Qureshi,
Akmal Saeed Khattak
Abstract:
The FakeNews task in MediaEval 2022 investigates the challenge of finding accurate and high-performance models for the classification of conspiracy tweets related to COVID-19. In this paper, we used BERT, ELMO, and their combination for feature extraction and RandomForest as classifier. The results show that ELMO performs slightly better than BERT, however their combination at feature level reduce…
▽ More
The FakeNews task in MediaEval 2022 investigates the challenge of finding accurate and high-performance models for the classification of conspiracy tweets related to COVID-19. In this paper, we used BERT, ELMO, and their combination for feature extraction and RandomForest as classifier. The results show that ELMO performs slightly better than BERT, however their combination at feature level reduces the performance.
△ Less
Submitted 7 March, 2023;
originally announced March 2023.
-
Identifying Misinformation Spreaders: A Graph-Based Semi-Supervised Learning Approach
Authors:
Atta Ullah,
Rabeeh Ayaz Abbasi,
Akmal Saeed Khattak,
Anwar Said
Abstract:
In this paper we proposed a Graph-Based conspiracy source detection method for the MediaEval task 2022 FakeNews: Corona Virus and Conspiracies Multimedia Analysis Task. The goal of this study was to apply SOTA graph neural network methods to the problem of misinformation spreading in online social networks. We explore three different Graph Neural Network models: GCN, GraphSAGE and DGCNN. Experimen…
▽ More
In this paper we proposed a Graph-Based conspiracy source detection method for the MediaEval task 2022 FakeNews: Corona Virus and Conspiracies Multimedia Analysis Task. The goal of this study was to apply SOTA graph neural network methods to the problem of misinformation spreading in online social networks. We explore three different Graph Neural Network models: GCN, GraphSAGE and DGCNN. Experimental results demonstrate that DGCNN outperforms in terms of accuracy.
△ Less
Submitted 7 March, 2023;
originally announced March 2023.
-
Graph Neural Networks for Low-Energy Event Classification & Reconstruction in IceCube
Authors:
R. Abbasi,
M. Ackermann,
J. Adams,
N. Aggarwal,
J. A. Aguilar,
M. Ahlers,
M. Ahrens,
J. M. Alameddine,
A. A. Alves Jr.,
N. M. Amin,
K. Andeen,
T. Anderson,
G. Anton,
C. Argüelles,
Y. Ashida,
S. Athanasiadou,
S. Axani,
X. Bai,
A. Balagopal V.,
M. Baricevic,
S. W. Barwick,
V. Basu,
R. Bay,
J. J. Beatty,
K. -H. Becker
, et al. (359 additional authors not shown)
Abstract:
IceCube, a cubic-kilometer array of optical sensors built to detect atmospheric and astrophysical neutrinos between 1 GeV and 1 PeV, is deployed 1.45 km to 2.45 km below the surface of the ice sheet at the South Pole. The classification and reconstruction of events from the in-ice detectors play a central role in the analysis of data from IceCube. Reconstructing and classifying events is a challen…
▽ More
IceCube, a cubic-kilometer array of optical sensors built to detect atmospheric and astrophysical neutrinos between 1 GeV and 1 PeV, is deployed 1.45 km to 2.45 km below the surface of the ice sheet at the South Pole. The classification and reconstruction of events from the in-ice detectors play a central role in the analysis of data from IceCube. Reconstructing and classifying events is a challenge due to the irregular detector geometry, inhomogeneous scattering and absorption of light in the ice and, below 100 GeV, the relatively low number of signal photons produced per event. To address this challenge, it is possible to represent IceCube events as point cloud graphs and use a Graph Neural Network (GNN) as the classification and reconstruction method. The GNN is capable of distinguishing neutrino events from cosmic-ray backgrounds, classifying different neutrino event types, and reconstructing the deposited energy, direction and interaction vertex. Based on simulation, we provide a comparison in the 1-100 GeV energy range to the current state-of-the-art maximum likelihood techniques used in current IceCube analyses, including the effects of known systematic uncertainties. For neutrino event classification, the GNN increases the signal efficiency by 18% at a fixed false positive rate (FPR), compared to current IceCube methods. Alternatively, the GNN offers a reduction of the FPR by over a factor 8 (to below half a percent) at a fixed signal efficiency. For the reconstruction of energy, direction, and interaction vertex, the resolution improves by an average of 13%-20% compared to current maximum likelihood techniques in the energy range of 1-30 GeV. The GNN, when run on a GPU, is capable of processing IceCube events at a rate nearly double of the median IceCube trigger rate of 2.7 kHz, which opens the possibility of using low energy neutrinos in online searches for transient events.
△ Less
Submitted 11 October, 2022; v1 submitted 7 September, 2022;
originally announced September 2022.
-
A Convolutional Neural Network based Cascade Reconstruction for the IceCube Neutrino Observatory
Authors:
R. Abbasi,
M. Ackermann,
J. Adams,
J. A. Aguilar,
M. Ahlers,
M. Ahrens,
C. Alispach,
A. A. Alves Jr.,
N. M. Amin,
R. An,
K. Andeen,
T. Anderson,
I. Ansseau,
G. Anton,
C. Argüelles,
S. Axani,
X. Bai,
A. Balagopal V.,
A. Barbano,
S. W. Barwick,
B. Bastian,
V. Basu,
V. Baum,
S. Baur,
R. Bay
, et al. (343 additional authors not shown)
Abstract:
Continued improvements on existing reconstruction methods are vital to the success of high-energy physics experiments, such as the IceCube Neutrino Observatory. In IceCube, further challenges arise as the detector is situated at the geographic South Pole where computational resources are limited. However, to perform real-time analyses and to issue alerts to telescopes around the world, powerful an…
▽ More
Continued improvements on existing reconstruction methods are vital to the success of high-energy physics experiments, such as the IceCube Neutrino Observatory. In IceCube, further challenges arise as the detector is situated at the geographic South Pole where computational resources are limited. However, to perform real-time analyses and to issue alerts to telescopes around the world, powerful and fast reconstruction methods are desired. Deep neural networks can be extremely powerful, and their usage is computationally inexpensive once the networks are trained. These characteristics make a deep learning-based approach an excellent candidate for the application in IceCube. A reconstruction method based on convolutional architectures and hexagonally shaped kernels is presented. The presented method is robust towards systematic uncertainties in the simulation and has been tested on experimental data. In comparison to standard reconstruction methods in IceCube, it can improve upon the reconstruction accuracy, while reducing the time necessary to run the reconstruction by two to three orders of magnitude.
△ Less
Submitted 26 July, 2021; v1 submitted 27 January, 2021;
originally announced January 2021.
-
Event Detection in Twitter Stream using Weighted Dynamic Heartbeat Graph Approach
Authors:
Zafar Saeed,
Rabeeh Ayaz Abbasi,
Muhammad Imran Razzak,
Guandong Xu
Abstract:
Tweets about everyday events are published on Twitter. Detecting such events is a challenging task due to the diverse and noisy contents of Twitter. In this paper, we propose a novel approach named Weighted Dynamic Heartbeat Graph (WDHG) to detect events from the Twitter stream. Once an event is detected in a Twitter stream, WDHG suppresses it in later stages, in order to detect new emerging event…
▽ More
Tweets about everyday events are published on Twitter. Detecting such events is a challenging task due to the diverse and noisy contents of Twitter. In this paper, we propose a novel approach named Weighted Dynamic Heartbeat Graph (WDHG) to detect events from the Twitter stream. Once an event is detected in a Twitter stream, WDHG suppresses it in later stages, in order to detect new emerging events. This unique characteristic makes the proposed approach sensitive to capture emerging events efficiently. Experiments are performed on three real-life benchmark datasets: FA Cup Final 2012, Super Tuesday 2012, and the US Elections 2012. Results show considerable improvement over existing event detection methods in most cases.
△ Less
Submitted 22 February, 2019;
originally announced February 2019.
-
Verification of Asynchronous Systems with an Unspecified Component
Authors:
Rosa Abbasi,
Fatemeh Ghassemi,
Ramtin Khosravi
Abstract:
Component-based systems evolve as a new component is added or an existing one is replaced by a newer version. Hence, it is appealing to assure the new system still preserves its safety properties. However, instead of inspecting the new system as a whole, which may result in a large state space, it is beneficial to reuse the verification results by inspecting the newly added component in isolation.…
▽ More
Component-based systems evolve as a new component is added or an existing one is replaced by a newer version. Hence, it is appealing to assure the new system still preserves its safety properties. However, instead of inspecting the new system as a whole, which may result in a large state space, it is beneficial to reuse the verification results by inspecting the newly added component in isolation. To this aim, we study the problem of model checking component-based asynchronously communicating systems in the presence of an unspecified component against safety properties. Our solution is based on assume-guarantee reasoning, adopted for asynchronous environments, which generates the weakest assumption. If the newly added component conforms to the assumption, then the whole system still satisfies the property. To make the approach efficient and convergent, we produce an overapproximated interface of the missing component and by its composition with the rest of the system components, we achieve an overapproximated specification of the system, from which we remove those traces of the system that violate the property and generate an assumption for the missing component.
We have implemented our approach on two case studies. Furthermore, we compared our results with the state of the art direct approach. Our resulting assumptions are smaller in size and achieved faster.
△ Less
Submitted 11 September, 2017;
originally announced September 2017.
-
The IceProd Framework: Distributed Data Processing for the IceCube Neutrino Observatory
Authors:
M. G. Aartsen,
R. Abbasi,
M. Ackermann,
J. Adams,
J. A. Aguilar,
M. Ahlers,
D. Altmann,
C. Arguelles,
J. Auffenberg,
X. Bai,
M. Baker,
S. W. Barwick,
V. Baum,
R. Bay,
J. J. Beatty,
J. Becker Tjus,
K. -H. Becker,
S. BenZvi,
P. Berghaus,
D. Berley,
E. Bernardini,
A. Bernhard,
D. Z. Besson,
G. Binder,
D. Bindig
, et al. (262 additional authors not shown)
Abstract:
IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, iden- tify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. IceProd is a distributed management system based on Python, XML-RPC and GridFTP. It…
▽ More
IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, iden- tify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. IceProd is a distributed management system based on Python, XML-RPC and GridFTP. It is driven by a central database in order to coordinate and admin- ister production of simulations and processing of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, Condor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework.
△ Less
Submitted 22 August, 2014; v1 submitted 22 November, 2013;
originally announced November 2013.
