Showing 1–7 of 7 results for author: Husain, S S

Attend-Fusion: Efficient Audio-Visual Fusion for Video Classification

Authors: Mahrukh Awan, Asmar Nadeem, Muhammad Junaid Awan, Armin Mustafa, Syed Sameed Husain

Abstract: Exploiting both audio and visual modalities for video classification is a challenging task, as the existing methods require large model architectures, leading to high computational complexity and resource requirements. Smaller architectures, on the other hand, struggle to achieve optimal performance. In this paper, we propose Attend-Fusion, an audio-visual (AV) fusion approach that introduces a co… ▽ More Exploiting both audio and visual modalities for video classification is a challenging task, as the existing methods require large model architectures, leading to high computational complexity and resource requirements. Smaller architectures, on the other hand, struggle to achieve optimal performance. In this paper, we propose Attend-Fusion, an audio-visual (AV) fusion approach that introduces a compact model architecture specifically designed to capture intricate audio-visual relationships in video data. Through extensive experiments on the challenging YouTube-8M dataset, we demonstrate that Attend-Fusion achieves an F1 score of 75.64\% with only 72M parameters, which is comparable to the performance of larger baseline models such as Fully-Connected Late Fusion (75.96\% F1 score, 341M parameters). Attend-Fusion achieves similar performance to the larger baseline model while reducing the model size by nearly 80\%, highlighting its efficiency in terms of model complexity. Our work demonstrates that the Attend-Fusion model effectively combines audio and visual information for video classification, achieving competitive performance with significantly reduced model size. This approach opens new possibilities for deploying high-performance video understanding systems in resource-constrained environments across various applications. △ Less

Submitted 26 August, 2024; originally announced August 2024.

NarrativeBridge: Enhancing Video Captioning with Causal-Temporal Narrative

Authors: Asmar Nadeem, Faegheh Sardari, Robert Dawes, Syed Sameed Husain, Adrian Hilton, Armin Mustafa

Abstract: Existing video captioning benchmarks and models lack coherent representations of causal-temporal narrative, which is sequences of events linked through cause and effect, unfolding over time and driven by characters or agents. This lack of narrative restricts models' ability to generate text descriptions that capture the causal and temporal dynamics inherent in video content. To address this gap, w… ▽ More Existing video captioning benchmarks and models lack coherent representations of causal-temporal narrative, which is sequences of events linked through cause and effect, unfolding over time and driven by characters or agents. This lack of narrative restricts models' ability to generate text descriptions that capture the causal and temporal dynamics inherent in video content. To address this gap, we propose NarrativeBridge, an approach comprising of: (1) a novel Causal-Temporal Narrative (CTN) captions benchmark generated using a large language model and few-shot prompting, explicitly encoding cause-effect temporal relationships in video descriptions, evaluated automatically to ensure caption quality and relevance; and (2) a dedicated Cause-Effect Network (CEN) architecture with separate encoders for capturing cause and effect dynamics independently, enabling effective learning and generation of captions with causal-temporal narrative. Extensive experiments demonstrate that CEN is more accurate in articulating the causal and temporal aspects of video content than the second best model (GIT): 17.88 and 17.44 CIDEr on the MSVD and MSR-VTT datasets, respectively. The proposed framework understands and generates nuanced text descriptions with intricate causal-temporal narrative structures present in videos, addressing a critical limitation in video captioning. For project details, visit https://narrativebridge.github.io/. △ Less

Submitted 10 June, 2024; originally announced June 2024.

Single-cell Subcellular Protein Localisation Using Novel Ensembles of Diverse Deep Architectures

Authors: Syed Sameed Husain, Eng-Jon Ong, Dmitry Minskiy, Mikel Bober-Irizar, Amaia Irizar, Miroslaw Bober

Abstract: Unravelling protein distributions within individual cells is key to understanding their function and state and indispensable to developing new treatments. Here we present the Hybrid subCellular Protein Localiser (HCPL), which learns from weakly labelled data to robustly localise single-cell subcellular protein patterns. It comprises innovative DNN architectures exploiting wavelet filters and learn… ▽ More Unravelling protein distributions within individual cells is key to understanding their function and state and indispensable to developing new treatments. Here we present the Hybrid subCellular Protein Localiser (HCPL), which learns from weakly labelled data to robustly localise single-cell subcellular protein patterns. It comprises innovative DNN architectures exploiting wavelet filters and learnt parametric activations that successfully tackle drastic cell variability. HCPL features correlation-based ensembling of novel architectures that boosts performance and aids generalisation. Large-scale data annotation is made feasible by our "AI-trains-AI" approach, which determines the visual integrity of cells and emphasises reliable labels for efficient training. In the Human Protein Atlas context, we demonstrate that HCPL defines state-of-the-art in the single-cell classification of protein localisation patterns. To better understand the inner workings of HCPL and assess its biological relevance, we analyse the contributions of each system component and dissect the emergent features from which the localisation predictions are derived. △ Less

Submitted 16 September, 2022; v1 submitted 19 May, 2022; originally announced May 2022.

ACTNET: end-to-end learning of feature activations and multi-stream aggregation for effective instance image retrieval

Authors: Syed Sameed Husain, Eng-Jon Ong, Miroslaw Bober

Abstract: We propose a novel CNN architecture called ACTNET for robust instance image retrieval from large-scale datasets. Our key innovation is a learnable activation layer designed to improve the signal-to-noise ratio (SNR) of deep convolutional feature maps. Further, we introduce a controlled multi-stream aggregation, where complementary deep features from different convolutional layers are optimally tra… ▽ More We propose a novel CNN architecture called ACTNET for robust instance image retrieval from large-scale datasets. Our key innovation is a learnable activation layer designed to improve the signal-to-noise ratio (SNR) of deep convolutional feature maps. Further, we introduce a controlled multi-stream aggregation, where complementary deep features from different convolutional layers are optimally transformed and balanced using our novel activation layers, before aggregation into a global descriptor. Importantly, the learnable parameters of our activation blocks are explicitly trained, together with the CNN parameters, in an end-to-end manner minimising triplet loss. This means that our network jointly learns the CNN filters and their optimal activation and aggregation for retrieval tasks. To our knowledge, this is the first time parametric functions have been used to control and learn optimal aggregation. We conduct an in-depth experimental study on three non-linear activation functions: Sine-Hyperbolic, Exponential and modified Weibull, showing that while all bring significant gains the Weibull function performs best thanks to its ability to equalise strong activations. The results clearly demonstrate that our ACTNET architecture significantly enhances the discriminative power of deep features, improving significantly over the state-of-the-art retrieval results on all datasets. △ Less

Submitted 23 October, 2020; v1 submitted 12 July, 2019; originally announced July 2019.

REMAP: Multi-layer entropy-guided pooling of dense CNN features for image retrieval

Authors: Syed Sameed Husain, Miroslaw Bober

Abstract: This paper addresses the problem of very large-scale image retrieval, focusing on improving its accuracy and robustness. We target enhanced robustness of search to factors such as variations in illumination, object appearance and scale, partial occlusions, and cluttered backgrounds - particularly important when search is performed across very large datasets with significant variability. We propose… ▽ More This paper addresses the problem of very large-scale image retrieval, focusing on improving its accuracy and robustness. We target enhanced robustness of search to factors such as variations in illumination, object appearance and scale, partial occlusions, and cluttered backgrounds - particularly important when search is performed across very large datasets with significant variability. We propose a novel CNN-based global descriptor, called REMAP, which learns and aggregates a hierarchy of deep features from multiple CNN layers, and is trained end-to-end with a triplet loss. REMAP explicitly learns discriminative features which are mutually-supportive and complementary at various semantic levels of visual abstraction. These dense local features are max-pooled spatially at each layer, within multi-scale overlapping regions, before aggregation into a single image-level descriptor. To identify the semantically useful regions and layers for retrieval, we propose to measure the information gain of each region and layer using KL-divergence. Our system effectively learns during training how useful various regions and layers are and weights them accordingly. We show that such relative entropy-guided aggregation outperforms classical CNN-based aggregation controlled by SGD. The entire framework is trained in an end-to-end fashion, outperforming the latest state-of-the-art results. On image retrieval datasets Holidays, Oxford and MPEG, the REMAP descriptor achieves mAP of 95.5%, 91.5%, and 80.1% respectively, outperforming any results published to date. REMAP also formed the core of the winning submission to the Google Landmark Retrieval Challenge on Kaggle. △ Less

Submitted 15 June, 2019; originally announced June 2019.

Comments: Submitted to IEEE Trans. Image Processing on 24 May 2018, published 22 May 2019

Journal ref: IEEE Transactions on Image Processing, Early Access 22 May 2019

A Model for Innovation Diffusion with Intergroup Suppression

Authors: Anirban Chakraborti, Syed S. Husain, Joseph Whitmeyer

Abstract: We present a new model for the diffusion of innovation. Here, the population is segmented into distinct groups. Adoption by a particular group of some cultural product may be inhibited both by large numbers of its own members already having adopted but also, in particular, by members of another group having adopted. Intergroup migration is also permitted. We determine the equilibrium points and ca… ▽ More We present a new model for the diffusion of innovation. Here, the population is segmented into distinct groups. Adoption by a particular group of some cultural product may be inhibited both by large numbers of its own members already having adopted but also, in particular, by members of another group having adopted. Intergroup migration is also permitted. We determine the equilibrium points and carry out stability analysis for the model for a two-group population. We also simulate a discrete time version of the model. Lastly, we present data on tablet use in eight countries from 2012-2016 and show that the relationship between use in the "under 25" age group and "55+" age group conforms to the model. △ Less

Submitted 24 February, 2018; originally announced February 2018.

Identifying the global terror hubs and vulnerable motifs using complex network dynamics

Authors: Syed Shariq Husain, Kiran Sharma, Vishwas Kukreti, Anirban Chakraborti

Abstract: Terrorism instills fear in the minds of people and takes away the freedom of individuals to act as they will. Terrorism has turned out to be an international menace today. Here, we study the terrorist attack incidents which occurred in the last half-century across the globe from the open source, Global Terrorism Database, and develop a view on their spatio-temporal dynamics. We construct a complex… ▽ More Terrorism instills fear in the minds of people and takes away the freedom of individuals to act as they will. Terrorism has turned out to be an international menace today. Here, we study the terrorist attack incidents which occurred in the last half-century across the globe from the open source, Global Terrorism Database, and develop a view on their spatio-temporal dynamics. We construct a complex network of global terrorism and study its growth dynamics, along with the statistical properties of the anti-social network, which are quite intriguing. Normally, each nation pursues its own vision of international security based upon its mandate and particular notions of politics and its policies to counter the threat of terrorism that could naturally include the use of tactical measures and strategic negotiations, or even physical power. We study the network resilience against targeted attacks and random failures, which could guide the counterterrorist outfits in designing strategies to fight terrorism. We then use a disparity filter method to isolate backbone of the network, and identify the terror hubs and vulnerable motifs of global terrorism. We also examine evolution of the hubs and motifs in a few exemplary cases like Afghanistan, Colombia, India, Israel, Pakistan and the United Kingdom. The dynamics of the terror hubs and the vulnerable motifs that we discover in the network backbone turn out to be very significant, and may provide deep insight on their formations and spreading, and thereby help in contending terrorism or framing public policies that can check their spread. △ Less

Submitted 21 October, 2019; v1 submitted 4 February, 2018; originally announced February 2018.

Comments: 33 pages, 11 figures, 5 tables