-
Open Vocabulary Multi-Label Video Classification
Authors:
Rohit Gupta,
Mamshad Nayeem Rizve,
Jayakrishnan Unnikrishnan,
Ashish Tawari,
Son Tran,
Mubarak Shah,
Benjamin Yao,
Trishul Chilimbi
Abstract:
Pre-trained vision-language models (VLMs) have enabled significant progress in open vocabulary computer vision tasks such as image classification, object detection and image segmentation. Some recent works have focused on extending VLMs to open vocabulary single label action classification in videos. However, previous methods fall short in holistic video understanding which requires the ability to…
▽ More
Pre-trained vision-language models (VLMs) have enabled significant progress in open vocabulary computer vision tasks such as image classification, object detection and image segmentation. Some recent works have focused on extending VLMs to open vocabulary single label action classification in videos. However, previous methods fall short in holistic video understanding which requires the ability to simultaneously recognize multiple actions and entities e.g., objects in the video in an open vocabulary setting. We formulate this problem as open vocabulary multilabel video classification and propose a method to adapt a pre-trained VLM such as CLIP to solve this task. We leverage large language models (LLMs) to provide semantic guidance to the VLM about class labels to improve its open vocabulary performance with two key contributions. First, we propose an end-to-end trainable architecture that learns to prompt an LLM to generate soft attributes for the CLIP text-encoder to enable it to recognize novel classes. Second, we integrate a temporal modeling module into CLIP's vision encoder to effectively model the spatio-temporal dynamics of video concepts as well as propose a novel regularized finetuning technique to ensure strong open vocabulary classification performance in the video domain. Our extensive experimentation showcases the efficacy of our approach on multiple benchmark datasets.
△ Less
Submitted 12 July, 2024;
originally announced July 2024.
-
VidLA: Video-Language Alignment at Scale
Authors:
Mamshad Nayeem Rizve,
Fan Fei,
Jayakrishnan Unnikrishnan,
Son Tran,
Benjamin Z. Yao,
Belinda Zeng,
Mubarak Shah,
Trishul Chilimbi
Abstract:
In this paper, we propose VidLA, an approach for video-language alignment at scale. There are two major limitations of previous video-language alignment approaches. First, they do not capture both short-range and long-range temporal dependencies and typically employ complex hierarchical deep network architectures that are hard to integrate with existing pretrained image-text foundation models. To…
▽ More
In this paper, we propose VidLA, an approach for video-language alignment at scale. There are two major limitations of previous video-language alignment approaches. First, they do not capture both short-range and long-range temporal dependencies and typically employ complex hierarchical deep network architectures that are hard to integrate with existing pretrained image-text foundation models. To effectively address this limitation, we instead keep the network architecture simple and use a set of data tokens that operate at different temporal resolutions in a hierarchical manner, accounting for the temporally hierarchical nature of videos. By employing a simple two-tower architecture, we are able to initialize our video-language model with pretrained image-text foundation models, thereby boosting the final performance. Second, existing video-language alignment works struggle due to the lack of semantically aligned large-scale training data. To overcome it, we leverage recent LLMs to curate the largest video-language dataset to date with better visual grounding. Furthermore, unlike existing video-text datasets which only contain short clips, our dataset is enriched with video clips of varying durations to aid our temporally hierarchical data tokens in extracting better representations at varying temporal scales. Overall, empirical results show that our proposed approach surpasses state-of-the-art methods on multiple retrieval benchmarks, especially on longer videos, and performs competitively on classification benchmarks.
△ Less
Submitted 21 March, 2024;
originally announced March 2024.
-
Joint Beamforming and Association Design for MIMO Radar
Authors:
Urs Niesen,
Jayakrishnan Unnikrishnan
Abstract:
A critical task of a radar receiver is data association, which assigns radar target detections to target filter tracks. Motivated by its importance, this paper introduces the problem of jointly designing multiple-input multiple-output (MIMO) radar transmit beam patterns and the corresponding data association schemes. We show that the coupling of the beamforming and the association subproblems can…
▽ More
A critical task of a radar receiver is data association, which assigns radar target detections to target filter tracks. Motivated by its importance, this paper introduces the problem of jointly designing multiple-input multiple-output (MIMO) radar transmit beam patterns and the corresponding data association schemes. We show that the coupling of the beamforming and the association subproblems can be conveniently parameterized by what we term an ambiguity graph, which prescribes if two targets are to be disambiguated by the beamforming design or by the data association scheme. The choice of ambiguity graph determines which of the two subproblems is more difficult and therefore allows to trade performance of one versus the other, resulting in a detection-association trade-off. This paper shows how to design both the beam pattern and the association scheme for a given ambiguity graph. It then discusses how to choose an ambiguity graph achieving close to the optimal detection-association trade-off.
△ Less
Submitted 10 September, 2018;
originally announced September 2018.
-
Where You Are Is Who You Are: User Identification by Matching Statistics
Authors:
Farid M. Naini,
Jayakrishnan Unnikrishnan,
Patrick Thiran,
Martin Vetterli
Abstract:
Most users of online services have unique behavioral or usage patterns. These behavioral patterns can be exploited to identify and track users by using only the observed patterns in the behavior. We study the task of identifying users from statistics of their behavioral patterns. Specifically, we focus on the setting in which we are given histograms of users' data collected during two different ex…
▽ More
Most users of online services have unique behavioral or usage patterns. These behavioral patterns can be exploited to identify and track users by using only the observed patterns in the behavior. We study the task of identifying users from statistics of their behavioral patterns. Specifically, we focus on the setting in which we are given histograms of users' data collected during two different experiments. We assume that, in the first dataset, the users' identities are anonymized or hidden and that, in the second dataset, their identities are known. We study the task of identifying the users by matching the histograms of their data in the first dataset with the histograms from the second dataset. In recent works, the optimal algorithm for this user identification task is introduced. In this paper, we evaluate the effectiveness of this method on three different types of datasets and in multiple scenarios. Using datasets such as call data records, web browsing histories, and GPS trajectories, we show that a large fraction of users can be easily identified given only histograms of their data; hence these histograms can act as users' fingerprints. We also verify that simultaneous identification of users achieves better performance compared to one-by-one user identification. We show that using the optimal method for identification gives higher identification accuracy than heuristics-based approaches in practical scenarios. The accuracy obtained under this optimal method can thus be used to quantify the maximum level of user identification that is possible in such settings. We show that the key factors affecting the accuracy of the optimal identification algorithm are the duration of the data collection, the number of users in the anonymized dataset, and the resolution of the dataset. We analyze the effectiveness of k-anonymization in resisting user identification attacks on these datasets.
△ Less
Submitted 9 December, 2015;
originally announced December 2015.
-
Unlabeled Sensing with Random Linear Measurements
Authors:
Jayakrishnan Unnikrishnan,
Saeid Haghighatshoar,
Martin Vetterli
Abstract:
We study the problem of solving a linear sensing system when the observations are unlabeled. Specifically we seek a solution to a linear system of equations y = Ax when the order of the observations in the vector y is unknown. Focusing on the setting in which A is a random matrix with i.i.d. entries, we show that if the sensing matrix A admits an oversampling ratio of 2 or higher, then with probab…
▽ More
We study the problem of solving a linear sensing system when the observations are unlabeled. Specifically we seek a solution to a linear system of equations y = Ax when the order of the observations in the vector y is unknown. Focusing on the setting in which A is a random matrix with i.i.d. entries, we show that if the sensing matrix A admits an oversampling ratio of 2 or higher, then with probability 1 it is possible to recover x exactly without the knowledge of the order of the observations in y. Furthermore, if x is of dimension K, then any 2K entries of y are sufficient to recover x. This result implies the existence of deterministic unlabeled sensing matrices with an oversampling factor of 2 that admit perfect reconstruction. The result is universal in that recovery is guaranteed for all possible choices of x. While the proof is constructive, it uses a combinatorial algorithm which is not practical, leaving the question of complexity open. We also analyze a noisy version of the problem and show that local stability is guaranteed by the solution. In particular, for every x, the recovery error tends to zero as the signal-to-noise-ratio tends to infinity. The question of universal stability is unclear. We also obtain a converse of the result in the noiseless case: If the number of observations in y is less than 2K, then with probability 1, universal recovery fails, i.e., with probability 1, there exists distinct choices of x which lead to the same unordered list of observations in y. In terms of applications, the unlabeled sensing problem is related to data association problems encountered in different domains including robotics where it is appears in a method called "simultaneous localization and mapping" (SLAM), multi-target tracking applications, and in sampling signals in the presence of jitter.
△ Less
Submitted 30 November, 2015;
originally announced December 2015.
-
Asymptotically Optimal Matching of Multiple Sequences to Source Distributions and Training Sequences
Authors:
Jayakrishnan Unnikrishnan
Abstract:
Consider a finite set of sources, each producing i.i.d. observations that follow a unique probability distribution on a finite alphabet. We study the problem of matching a finite set of observed sequences to the set of sources under the constraint that the observed sequences are produced by distinct sources. In general, the number of sequences $N$ may be different from the number of sources $M$, a…
▽ More
Consider a finite set of sources, each producing i.i.d. observations that follow a unique probability distribution on a finite alphabet. We study the problem of matching a finite set of observed sequences to the set of sources under the constraint that the observed sequences are produced by distinct sources. In general, the number of sequences $N$ may be different from the number of sources $M$, and only some $K \leq \min\{M,N\}$ of the observed sequences may be produced by a source from the set of sources of interest. We consider two versions of the problem -- one in which the probability laws of the sources are known, and another in which the probability laws of the sources are unspecified but one training sequence from each of the sources is available. We show that both these problems can be solved using a sequence of tests that are allowed to produce "no-match" decisions. The tests ensure exponential decay of the probabilities of incorrect matching as the sequence lengths increase, and minimize the "no-match" decisions. Both tests can be implemented using variants of the minimum weight matching algorithm applied to a weighted bipartite graph. We also compare the performances obtained by using these tests with those obtained by using tests that do not take into account the constraint that the sequences are produced by distinct sources. For the version of the problem in which the probability laws of the sources are known, we compute the rejection exponents and error exponents of the tests and show that tests that make use of the constraint have better exponents than tests that do not make use of this information.
△ Less
Submitted 8 December, 2014;
originally announced December 2014.
-
On Minimal Trajectories for Mobile Sampling of Bandlimited Fields
Authors:
Karlheinz Gröchenig,
José Luis Romero,
Jayakrishnan Unnikrishnan,
Martin Vetterli
Abstract:
We study the design of sampling trajectories for stable sampling and the reconstruction of bandlimited spatial fields using mobile sensors. The spectrum is assumed to be a symmetric convex set. As a performance metric we use the path density of the set of sampling trajectories that is defined as the total distance traveled by the moving sensors per unit spatial volume of the spatial region being m…
▽ More
We study the design of sampling trajectories for stable sampling and the reconstruction of bandlimited spatial fields using mobile sensors. The spectrum is assumed to be a symmetric convex set. As a performance metric we use the path density of the set of sampling trajectories that is defined as the total distance traveled by the moving sensors per unit spatial volume of the spatial region being monitored. Focussing first on parallel lines, we identify the set of parallel lines with minimal path density that contains a set of stable sampling for fields bandlimited to a known set. We then show that the problem becomes ill-posed when the optimization is performed over all trajectories by demonstrating a feasible trajectory set with arbitrarily low path density. However, the problem becomes well-posed if we explicitly specify the stability margins. We demonstrate this by obtaining a non-trivial lower bound on the path density of an arbitrary set of trajectories that contain a sampling set with explicitly specified stability bounds.
△ Less
Submitted 24 November, 2014; v1 submitted 30 December, 2013;
originally announced December 2013.
-
Approximate dynamic programming using fluid and diffusion approximations with applications to power management
Authors:
Wei Chen,
Dayu Huang,
Ankur A. Kulkarni,
Jayakrishnan Unnikrishnan,
Quanyan Zhu,
Prashant Mehta,
Sean Meyn,
Adam Wierman
Abstract:
Neuro-dynamic programming is a class of powerful techniques for approximating the solution to dynamic programming equations. In their most computationally attractive formulations, these techniques provide the approximate solution only within a prescribed finite-dimensional function class. Thus, the question that always arises is how should the function class be chosen? The goal of this paper is to…
▽ More
Neuro-dynamic programming is a class of powerful techniques for approximating the solution to dynamic programming equations. In their most computationally attractive formulations, these techniques provide the approximate solution only within a prescribed finite-dimensional function class. Thus, the question that always arises is how should the function class be chosen? The goal of this paper is to propose an approach using the solutions to associated fluid and diffusion approximations. In order to illustrate this approach, the paper focuses on an application to dynamic speed scaling for power management in computer processors.
△ Less
Submitted 9 July, 2013; v1 submitted 6 July, 2013;
originally announced July 2013.
-
Sampling and Reconstruction of Spatial Fields using Mobile Sensors
Authors:
Jayakrishnan Unnikrishnan,
Martin Vetterli
Abstract:
Spatial sampling is traditionally studied in a static setting where static sensors scattered around space take measurements of the spatial field at their locations. In this paper we study the emerging paradigm of sampling and reconstructing spatial fields using sensors that move through space. We show that mobile sensing offers some unique advantages over static sensing in sensing time-invariant b…
▽ More
Spatial sampling is traditionally studied in a static setting where static sensors scattered around space take measurements of the spatial field at their locations. In this paper we study the emerging paradigm of sampling and reconstructing spatial fields using sensors that move through space. We show that mobile sensing offers some unique advantages over static sensing in sensing time-invariant bandlimited spatial fields. Since a moving sensor encounters such a spatial field along its path as a time-domain signal, a time-domain anti-aliasing filter can be employed prior to sampling the signal received at the sensor. Such a filtering procedure, when used by a configuration of sensors moving at constant speeds along equispaced parallel lines, leads to a complete suppression of spatial aliasing in the direction of motion of the sensors. We analytically quantify the advantage of using such a sampling scheme over a static sampling scheme by computing the reduction in sampling noise due to the filter. We also analyze the effects of non-uniform sensor speeds on the reconstruction accuracy. Using simulation examples we demonstrate the advantages of mobile sampling over static sampling in practical problems.
We extend our analysis to sampling and reconstruction schemes for monitoring time-varying bandlimited fields using mobile sensors. We demonstrate that in some situations we require a lower density of sensors when using a mobile sensing scheme instead of the conventional static sensing scheme. The exact advantage is quantified for a problem of sampling and reconstructing an audio field.
△ Less
Submitted 1 November, 2012;
originally announced November 2012.
-
Sampling High-Dimensional Bandlimited Fields on Low-Dimensional Manifolds
Authors:
Jayakrishnan Unnikrishnan,
Martin Vetterli
Abstract:
Consider the task of sampling and reconstructing a bandlimited spatial field in $\Re^2$ using moving sensors that take measurements along their path. It is inexpensive to increase the sampling rate along the paths of the sensors but more expensive to increase the total distance traveled by the sensors per unit area, which we call the \emph{path density}. In this paper we introduce the problem of d…
▽ More
Consider the task of sampling and reconstructing a bandlimited spatial field in $\Re^2$ using moving sensors that take measurements along their path. It is inexpensive to increase the sampling rate along the paths of the sensors but more expensive to increase the total distance traveled by the sensors per unit area, which we call the \emph{path density}. In this paper we introduce the problem of designing sensor trajectories that are minimal in path density subject to the condition that the measurements of the field on these trajectories admit perfect reconstruction of bandlimited fields. We study various possible designs of sampling trajectories. Generalizing some ideas from the classical theory of sampling on lattices, we obtain necessary and sufficient conditions on the trajectories for perfect reconstruction. We show that a single set of equispaced parallel lines has the lowest path density from certain restricted classes of trajectories that admit perfect reconstruction.
We then generalize some of our results to higher dimensions. We first obtain results on designing sampling trajectories in higher dimensional fields. Further, interpreting trajectories as 1-dimensional manifolds, we extend some of our ideas to higher dimensional sampling manifolds. We formulate the problem of designing $κ$-dimensional sampling manifolds for $d$-dimensional spatial fields that are minimal in \emph{manifold density}, a natural generalization of the path density. We show that our results on sampling trajectories for fields in $\Re^2$ can be generalized to analogous results on $d-1$-dimensional sampling manifolds for $d$-dimensional spatial fields.
△ Less
Submitted 1 November, 2012; v1 submitted 1 December, 2011;
originally announced December 2011.
-
Minimax Robust Quickest Change Detection
Authors:
Jayakrishnan Unnikrishnan,
Venugopal V. Veeravalli,
Sean Meyn
Abstract:
The popular criteria of optimality for quickest change detection procedures are the Lorden criterion, the Shiryaev-Roberts-Pollak criterion, and the Bayesian criterion. In this paper a robust version of these quickest change detection problems is considered when the pre-change and post-change distributions are not known exactly but belong to known uncertainty classes of distributions. For uncertai…
▽ More
The popular criteria of optimality for quickest change detection procedures are the Lorden criterion, the Shiryaev-Roberts-Pollak criterion, and the Bayesian criterion. In this paper a robust version of these quickest change detection problems is considered when the pre-change and post-change distributions are not known exactly but belong to known uncertainty classes of distributions. For uncertainty classes that satisfy a specific condition, it is shown that one can identify least favorable distributions (LFDs) from the uncertainty classes, such that the detection rule designed for the LFDs is optimal for the robust problem in a minimax sense. The condition is similar to that required for the identification of LFDs for the robust hypothesis testing problem originally studied by Huber. An upper bound on the delay incurred by the robust test is also obtained in the asymptotic setting under the Lorden criterion of optimality. This bound quantifies the delay penalty incurred to guarantee robustness. When the LFDs can be identified, the proposed test is easier to implement than the CUSUM test based on the Generalized Likelihood Ratio (GLR) statistic which is a popular approach for such robust change detection problems. The proposed test is also shown to give better performance than the GLR test in simulations for some parameter values.
△ Less
Submitted 31 May, 2010; v1 submitted 13 November, 2009;
originally announced November 2009.
-
Universal and Composite Hypothesis Testing via Mismatched Divergence
Authors:
Jayakrishnan Unnikrishnan,
Dayu Huang,
Sean Meyn,
Amit Surana,
Venugopal Veeravalli
Abstract:
For the universal hypothesis testing problem, where the goal is to decide between the known null hypothesis distribution and some other unknown distribution, Hoeffding proposed a universal test in the nineteen sixties. Hoeffding's universal test statistic can be written in terms of Kullback-Leibler (K-L) divergence between the empirical distribution of the observations and the null hypothesis dist…
▽ More
For the universal hypothesis testing problem, where the goal is to decide between the known null hypothesis distribution and some other unknown distribution, Hoeffding proposed a universal test in the nineteen sixties. Hoeffding's universal test statistic can be written in terms of Kullback-Leibler (K-L) divergence between the empirical distribution of the observations and the null hypothesis distribution. In this paper a modification of Hoeffding's test is considered based on a relaxation of the K-L divergence test statistic, referred to as the mismatched divergence. The resulting mismatched test is shown to be a generalized likelihood-ratio test (GLRT) for the case where the alternate distribution lies in a parametric family of the distributions characterized by a finite dimensional parameter, i.e., it is a solution to the corresponding composite hypothesis testing problem. For certain choices of the alternate distribution, it is shown that both the Hoeffding test and the mismatched test have the same asymptotic performance in terms of error exponents. A consequence of this result is that the GLRT is optimal in differentiating a particular distribution from others in an exponential family. It is also shown that the mismatched test has a significant advantage over the Hoeffding test in terms of finite sample size performance. This advantage is due to the difference in the asymptotic variances of the two test statistics under the null hypothesis. In particular, the variance of the K-L divergence grows linearly with the alphabet size, making the test impractical for applications involving large alphabet distributions. The variance of the mismatched divergence on the other hand grows linearly with the dimension of the parameter space, and can hence be controlled through a prudent choice of the function class defining the mismatched divergence.
△ Less
Submitted 9 September, 2010; v1 submitted 11 September, 2009;
originally announced September 2009.
-
Algorithms for Dynamic Spectrum Access with Learning for Cognitive Radio
Authors:
Jayakrishnan Unnikrishnan,
Venugopal Veeravalli
Abstract:
We study the problem of dynamic spectrum sensing and access in cognitive radio systems as a partially observed Markov decision process (POMDP). A group of cognitive users cooperatively tries to exploit vacancies in primary (licensed) channels whose occupancies follow a Markovian evolution. We first consider the scenario where the cognitive users have perfect knowledge of the distribution of the…
▽ More
We study the problem of dynamic spectrum sensing and access in cognitive radio systems as a partially observed Markov decision process (POMDP). A group of cognitive users cooperatively tries to exploit vacancies in primary (licensed) channels whose occupancies follow a Markovian evolution. We first consider the scenario where the cognitive users have perfect knowledge of the distribution of the signals they receive from the primary users. For this problem, we obtain a greedy channel selection and access policy that maximizes the instantaneous reward, while satisfying a constraint on the probability of interfering with licensed transmissions. We also derive an analytical universal upper bound on the performance of the optimal policy. Through simulation, we show that our scheme achieves good performance relative to the upper bound and improved performance relative to an existing scheme.
We then consider the more practical scenario where the exact distribution of the signal from the primary is unknown. We assume a parametric model for the distribution and develop an algorithm that can learn the true distribution, still guaranteeing the constraint on the interference probability. We show that this algorithm outperforms the naive design that assumes a worst case value for the parameter. We also provide a proof for the convergence of the learning algorithm.
△ Less
Submitted 6 February, 2010; v1 submitted 16 July, 2008;
originally announced July 2008.
