-
Attend-Fusion: Efficient Audio-Visual Fusion for Video Classification
Authors:
Mahrukh Awan,
Asmar Nadeem,
Muhammad Junaid Awan,
Armin Mustafa,
Syed Sameed Husain
Abstract:
Exploiting both audio and visual modalities for video classification is a challenging task, as the existing methods require large model architectures, leading to high computational complexity and resource requirements. Smaller architectures, on the other hand, struggle to achieve optimal performance. In this paper, we propose Attend-Fusion, an audio-visual (AV) fusion approach that introduces a co…
▽ More
Exploiting both audio and visual modalities for video classification is a challenging task, as the existing methods require large model architectures, leading to high computational complexity and resource requirements. Smaller architectures, on the other hand, struggle to achieve optimal performance. In this paper, we propose Attend-Fusion, an audio-visual (AV) fusion approach that introduces a compact model architecture specifically designed to capture intricate audio-visual relationships in video data. Through extensive experiments on the challenging YouTube-8M dataset, we demonstrate that Attend-Fusion achieves an F1 score of 75.64\% with only 72M parameters, which is comparable to the performance of larger baseline models such as Fully-Connected Late Fusion (75.96\% F1 score, 341M parameters). Attend-Fusion achieves similar performance to the larger baseline model while reducing the model size by nearly 80\%, highlighting its efficiency in terms of model complexity. Our work demonstrates that the Attend-Fusion model effectively combines audio and visual information for video classification, achieving competitive performance with significantly reduced model size. This approach opens new possibilities for deploying high-performance video understanding systems in resource-constrained environments across various applications.
△ Less
Submitted 26 August, 2024;
originally announced August 2024.
-
NarrativeBridge: Enhancing Video Captioning with Causal-Temporal Narrative
Authors:
Asmar Nadeem,
Faegheh Sardari,
Robert Dawes,
Syed Sameed Husain,
Adrian Hilton,
Armin Mustafa
Abstract:
Existing video captioning benchmarks and models lack coherent representations of causal-temporal narrative, which is sequences of events linked through cause and effect, unfolding over time and driven by characters or agents. This lack of narrative restricts models' ability to generate text descriptions that capture the causal and temporal dynamics inherent in video content. To address this gap, w…
▽ More
Existing video captioning benchmarks and models lack coherent representations of causal-temporal narrative, which is sequences of events linked through cause and effect, unfolding over time and driven by characters or agents. This lack of narrative restricts models' ability to generate text descriptions that capture the causal and temporal dynamics inherent in video content. To address this gap, we propose NarrativeBridge, an approach comprising of: (1) a novel Causal-Temporal Narrative (CTN) captions benchmark generated using a large language model and few-shot prompting, explicitly encoding cause-effect temporal relationships in video descriptions, evaluated automatically to ensure caption quality and relevance; and (2) a dedicated Cause-Effect Network (CEN) architecture with separate encoders for capturing cause and effect dynamics independently, enabling effective learning and generation of captions with causal-temporal narrative. Extensive experiments demonstrate that CEN is more accurate in articulating the causal and temporal aspects of video content than the second best model (GIT): 17.88 and 17.44 CIDEr on the MSVD and MSR-VTT datasets, respectively. The proposed framework understands and generates nuanced text descriptions with intricate causal-temporal narrative structures present in videos, addressing a critical limitation in video captioning. For project details, visit https://narrativebridge.github.io/.
△ Less
Submitted 10 June, 2024;
originally announced June 2024.
-
Single-cell Subcellular Protein Localisation Using Novel Ensembles of Diverse Deep Architectures
Authors:
Syed Sameed Husain,
Eng-Jon Ong,
Dmitry Minskiy,
Mikel Bober-Irizar,
Amaia Irizar,
Miroslaw Bober
Abstract:
Unravelling protein distributions within individual cells is key to understanding their function and state and indispensable to developing new treatments. Here we present the Hybrid subCellular Protein Localiser (HCPL), which learns from weakly labelled data to robustly localise single-cell subcellular protein patterns. It comprises innovative DNN architectures exploiting wavelet filters and learn…
▽ More
Unravelling protein distributions within individual cells is key to understanding their function and state and indispensable to developing new treatments. Here we present the Hybrid subCellular Protein Localiser (HCPL), which learns from weakly labelled data to robustly localise single-cell subcellular protein patterns. It comprises innovative DNN architectures exploiting wavelet filters and learnt parametric activations that successfully tackle drastic cell variability. HCPL features correlation-based ensembling of novel architectures that boosts performance and aids generalisation. Large-scale data annotation is made feasible by our "AI-trains-AI" approach, which determines the visual integrity of cells and emphasises reliable labels for efficient training. In the Human Protein Atlas context, we demonstrate that HCPL defines state-of-the-art in the single-cell classification of protein localisation patterns. To better understand the inner workings of HCPL and assess its biological relevance, we analyse the contributions of each system component and dissect the emergent features from which the localisation predictions are derived.
△ Less
Submitted 16 September, 2022; v1 submitted 19 May, 2022;
originally announced May 2022.
-
ACTNET: end-to-end learning of feature activations and multi-stream aggregation for effective instance image retrieval
Authors:
Syed Sameed Husain,
Eng-Jon Ong,
Miroslaw Bober
Abstract:
We propose a novel CNN architecture called ACTNET for robust instance image retrieval from large-scale datasets. Our key innovation is a learnable activation layer designed to improve the signal-to-noise ratio (SNR) of deep convolutional feature maps. Further, we introduce a controlled multi-stream aggregation, where complementary deep features from different convolutional layers are optimally tra…
▽ More
We propose a novel CNN architecture called ACTNET for robust instance image retrieval from large-scale datasets. Our key innovation is a learnable activation layer designed to improve the signal-to-noise ratio (SNR) of deep convolutional feature maps. Further, we introduce a controlled multi-stream aggregation, where complementary deep features from different convolutional layers are optimally transformed and balanced using our novel activation layers, before aggregation into a global descriptor. Importantly, the learnable parameters of our activation blocks are explicitly trained, together with the CNN parameters, in an end-to-end manner minimising triplet loss. This means that our network jointly learns the CNN filters and their optimal activation and aggregation for retrieval tasks. To our knowledge, this is the first time parametric functions have been used to control and learn optimal aggregation. We conduct an in-depth experimental study on three non-linear activation functions: Sine-Hyperbolic, Exponential and modified Weibull, showing that while all bring significant gains the Weibull function performs best thanks to its ability to equalise strong activations. The results clearly demonstrate that our ACTNET architecture significantly enhances the discriminative power of deep features, improving significantly over the state-of-the-art retrieval results on all datasets.
△ Less
Submitted 23 October, 2020; v1 submitted 12 July, 2019;
originally announced July 2019.
-
REMAP: Multi-layer entropy-guided pooling of dense CNN features for image retrieval
Authors:
Syed Sameed Husain,
Miroslaw Bober
Abstract:
This paper addresses the problem of very large-scale image retrieval, focusing on improving its accuracy and robustness. We target enhanced robustness of search to factors such as variations in illumination, object appearance and scale, partial occlusions, and cluttered backgrounds - particularly important when search is performed across very large datasets with significant variability. We propose…
▽ More
This paper addresses the problem of very large-scale image retrieval, focusing on improving its accuracy and robustness. We target enhanced robustness of search to factors such as variations in illumination, object appearance and scale, partial occlusions, and cluttered backgrounds - particularly important when search is performed across very large datasets with significant variability. We propose a novel CNN-based global descriptor, called REMAP, which learns and aggregates a hierarchy of deep features from multiple CNN layers, and is trained end-to-end with a triplet loss. REMAP explicitly learns discriminative features which are mutually-supportive and complementary at various semantic levels of visual abstraction. These dense local features are max-pooled spatially at each layer, within multi-scale overlapping regions, before aggregation into a single image-level descriptor. To identify the semantically useful regions and layers for retrieval, we propose to measure the information gain of each region and layer using KL-divergence. Our system effectively learns during training how useful various regions and layers are and weights them accordingly. We show that such relative entropy-guided aggregation outperforms classical CNN-based aggregation controlled by SGD. The entire framework is trained in an end-to-end fashion, outperforming the latest state-of-the-art results. On image retrieval datasets Holidays, Oxford and MPEG, the REMAP descriptor achieves mAP of 95.5%, 91.5%, and 80.1% respectively, outperforming any results published to date. REMAP also formed the core of the winning submission to the Google Landmark Retrieval Challenge on Kaggle.
△ Less
Submitted 15 June, 2019;
originally announced June 2019.
-
A Model for Innovation Diffusion with Intergroup Suppression
Authors:
Anirban Chakraborti,
Syed S. Husain,
Joseph Whitmeyer
Abstract:
We present a new model for the diffusion of innovation. Here, the population is segmented into distinct groups. Adoption by a particular group of some cultural product may be inhibited both by large numbers of its own members already having adopted but also, in particular, by members of another group having adopted. Intergroup migration is also permitted. We determine the equilibrium points and ca…
▽ More
We present a new model for the diffusion of innovation. Here, the population is segmented into distinct groups. Adoption by a particular group of some cultural product may be inhibited both by large numbers of its own members already having adopted but also, in particular, by members of another group having adopted. Intergroup migration is also permitted. We determine the equilibrium points and carry out stability analysis for the model for a two-group population. We also simulate a discrete time version of the model. Lastly, we present data on tablet use in eight countries from 2012-2016 and show that the relationship between use in the "under 25" age group and "55+" age group conforms to the model.
△ Less
Submitted 24 February, 2018;
originally announced February 2018.
-
Identifying the global terror hubs and vulnerable motifs using complex network dynamics
Authors:
Syed Shariq Husain,
Kiran Sharma,
Vishwas Kukreti,
Anirban Chakraborti
Abstract:
Terrorism instills fear in the minds of people and takes away the freedom of individuals to act as they will. Terrorism has turned out to be an international menace today. Here, we study the terrorist attack incidents which occurred in the last half-century across the globe from the open source, Global Terrorism Database, and develop a view on their spatio-temporal dynamics. We construct a complex…
▽ More
Terrorism instills fear in the minds of people and takes away the freedom of individuals to act as they will. Terrorism has turned out to be an international menace today. Here, we study the terrorist attack incidents which occurred in the last half-century across the globe from the open source, Global Terrorism Database, and develop a view on their spatio-temporal dynamics. We construct a complex network of global terrorism and study its growth dynamics, along with the statistical properties of the anti-social network, which are quite intriguing. Normally, each nation pursues its own vision of international security based upon its mandate and particular notions of politics and its policies to counter the threat of terrorism that could naturally include the use of tactical measures and strategic negotiations, or even physical power. We study the network resilience against targeted attacks and random failures, which could guide the counterterrorist outfits in designing strategies to fight terrorism. We then use a disparity filter method to isolate backbone of the network, and identify the terror hubs and vulnerable motifs of global terrorism. We also examine evolution of the hubs and motifs in a few exemplary cases like Afghanistan, Colombia, India, Israel, Pakistan and the United Kingdom. The dynamics of the terror hubs and the vulnerable motifs that we discover in the network backbone turn out to be very significant, and may provide deep insight on their formations and spreading, and thereby help in contending terrorism or framing public policies that can check their spread.
△ Less
Submitted 21 October, 2019; v1 submitted 4 February, 2018;
originally announced February 2018.