-
Tradeoffs Between Contrastive and Supervised Learning: An Empirical Study
Authors:
Ananya Karthik,
Mike Wu,
Noah Goodman,
Alex Tamkin
Abstract:
Contrastive learning has made considerable progress in computer vision, outperforming supervised pretraining on a range of downstream datasets. However, is contrastive learning the better choice in all situations? We demonstrate two cases where it is not. First, under sufficiently small pretraining budgets, supervised pretraining on ImageNet consistently outperforms a comparable contrastive model…
▽ More
Contrastive learning has made considerable progress in computer vision, outperforming supervised pretraining on a range of downstream datasets. However, is contrastive learning the better choice in all situations? We demonstrate two cases where it is not. First, under sufficiently small pretraining budgets, supervised pretraining on ImageNet consistently outperforms a comparable contrastive model on eight diverse image classification datasets. This suggests that the common practice of comparing pretraining approaches at hundreds or thousands of epochs may not produce actionable insights for those with more limited compute budgets. Second, even with larger pretraining budgets we identify tasks where supervised learning prevails, perhaps because the object-centric bias of supervised pretraining makes the model more resilient to common corruptions and spurious foreground-background correlations. These results underscore the need to characterize tradeoffs of different pretraining objectives across a wider range of contexts and training regimes.
△ Less
Submitted 10 December, 2021;
originally announced December 2021.
-
Storage Class Memory: Principles, Problems, and Possibilities
Authors:
Aditya K Kamath,
Leslie Monis,
A Tarun Karthik,
Basavaraj Talawar
Abstract:
Storage Class Memory (SCM) is a class of memory technology which has recently become viable for use. Their namearises from the fact that they exhibit non-volatility of data, similar to secondary storage while also having latencies comparable toprimary memory and byte-addressibility. In this area, Phase Change Memory (PCM), Spin-Transfer-Torque Random Access Memory(STT-RAM), and Resistive RAM (ReRA…
▽ More
Storage Class Memory (SCM) is a class of memory technology which has recently become viable for use. Their namearises from the fact that they exhibit non-volatility of data, similar to secondary storage while also having latencies comparable toprimary memory and byte-addressibility. In this area, Phase Change Memory (PCM), Spin-Transfer-Torque Random Access Memory(STT-RAM), and Resistive RAM (ReRAM) have emerged as the major contenders for commercial and industrial use. In this paper, wedescribe how these memory types function, while highlighting the problems of endurance and performance that these memory typesface. We also discuss the future possibilities of Multi-Level Cells (MLCs), as well as how SCM can be used to construct accelerators.
△ Less
Submitted 26 September, 2019;
originally announced September 2019.
-
Estimation Theory Based Robust Phase Offset Estimation in the Presence of Delay Attacks
Authors:
Anantha K. Karthik,
Rick S. Blum
Abstract:
This paper addresses the problem of robust clock phase offset estimation for the IEEE 1588 precision time protocol (PTP) in the presence of delay attacks. Delay attacks are one of the most effective cyber attacks in PTP, as they cannot be mitigated using typical security measures. In this paper, we consider the case where the slave node can exchange synchronization messages with multiple master no…
▽ More
This paper addresses the problem of robust clock phase offset estimation for the IEEE 1588 precision time protocol (PTP) in the presence of delay attacks. Delay attacks are one of the most effective cyber attacks in PTP, as they cannot be mitigated using typical security measures. In this paper, we consider the case where the slave node can exchange synchronization messages with multiple master nodes synchronized to the same clock. We first provide lower bounds on the best achievable performance for any phase offset estimation scheme in the presence of delay attacks. We then present a novel phase offset estimation scheme that employs the Expectation-Maximization algorithm for detecting which of the master-slave communication links have been subject to delay attacks. After discarding information from the links identified as attacked, which we show to be optimal, the optimal vector location parameter estimator is employed to estimate the phase offset of the slave node. Simulation results are presented to show that the proposed phase offset estimation scheme exhibits performance close to the lower bounds in a wide variety of scenarios.
△ Less
Submitted 15 November, 2016;
originally announced November 2016.
-
Randomized Assignment of Jobs to Servers in Heterogeneous Clusters of Shared Servers for Low Delay
Authors:
Arpan Mukhopadhyay,
A. Karthik,
Ravi R. Mazumdar
Abstract:
We consider the job assignment problem in a multi-server system consisting of $N$ parallel processor sharing servers, categorized into $M$ ($\ll N$) different types according to their processing capacity or speed. Jobs of random sizes arrive at the system according to a Poisson process with rate $N λ$. Upon each arrival, a small number of servers from each type is sampled uniformly at random. The…
▽ More
We consider the job assignment problem in a multi-server system consisting of $N$ parallel processor sharing servers, categorized into $M$ ($\ll N$) different types according to their processing capacity or speed. Jobs of random sizes arrive at the system according to a Poisson process with rate $N λ$. Upon each arrival, a small number of servers from each type is sampled uniformly at random. The job is then assigned to one of the sampled servers based on a selection rule. We propose two schemes, each corresponding to a specific selection rule that aims at reducing the mean sojourn time of jobs in the system.
We first show that both methods achieve the maximal stability region. We then analyze the system operating under the proposed schemes as $N \to \infty$ which corresponds to the mean field. Our results show that asymptotic independence among servers holds even when $M$ is finite and exchangeability holds only within servers of the same type. We further establish the existence and uniqueness of stationary solution of the mean field and show that the tail distribution of server occupancy decays doubly exponentially for each server type. When the estimates of arrival rates are not available, the proposed schemes offer simpler alternatives to achieving lower mean sojourn time of jobs, as shown by our numerical studies.
△ Less
Submitted 20 February, 2015;
originally announced February 2015.
