-
Multi-Resolution Weak Supervision for Sequential Data
Authors:
Frederic Sala,
Paroma Varma,
Jason Fries,
Daniel Y. Fu,
Shiori Sagawa,
Saelig Khattar,
Ashwini Ramamoorthy,
Ke Xiao,
Kayvon Fatahalian,
James Priest,
Christopher Ré
Abstract:
Since manually labeling training data is slow and expensive, recent industrial and scientific research efforts have turned to weaker or noisier forms of supervision sources. However, existing weak supervision approaches fail to model multi-resolution sources for sequential data, like video, that can assign labels to individual elements or collections of elements in a sequence. A key challenge in w…
▽ More
Since manually labeling training data is slow and expensive, recent industrial and scientific research efforts have turned to weaker or noisier forms of supervision sources. However, existing weak supervision approaches fail to model multi-resolution sources for sequential data, like video, that can assign labels to individual elements or collections of elements in a sequence. A key challenge in weak supervision is estimating the unknown accuracies and correlations of these sources without using labeled data. Multi-resolution sources exacerbate this challenge due to complex correlations and sample complexity that scales in the length of the sequence. We propose Dugong, the first framework to model multi-resolution weak supervision sources with complex correlations to assign probabilistic labels to training data. Theoretically, we prove that Dugong, under mild conditions, can uniquely recover the unobserved accuracy and correlation parameters and use parameter sharing to improve sample complexity. Our method assigns clinician-validated labels to population-scale biomedical video repositories, helping outperform traditional supervision by 36.8 F1 points and addressing a key use case where machine learning has been severely limited by the lack of expert labeled data. On average, Dugong improves over traditional supervision by 16.0 F1 points and existing weak supervision approaches by 24.2 F1 points across several video and sensor classification tasks.
△ Less
Submitted 21 October, 2019;
originally announced October 2019.
-
Scene Graph Prediction with Limited Labels
Authors:
Vincent S. Chen,
Paroma Varma,
Ranjay Krishna,
Michael Bernstein,
Christopher Re,
Li Fei-Fei
Abstract:
Visual knowledge bases such as Visual Genome power numerous applications in computer vision, including visual question answering and captioning, but suffer from sparse, incomplete relationships. All scene graph models to date are limited to training on a small set of visual relationships that have thousands of training labels each. Hiring human annotators is expensive, and using textual knowledge…
▽ More
Visual knowledge bases such as Visual Genome power numerous applications in computer vision, including visual question answering and captioning, but suffer from sparse, incomplete relationships. All scene graph models to date are limited to training on a small set of visual relationships that have thousands of training labels each. Hiring human annotators is expensive, and using textual knowledge base completion methods are incompatible with visual data. In this paper, we introduce a semi-supervised method that assigns probabilistic relationship labels to a large number of unlabeled images using few labeled examples. We analyze visual relationships to suggest two types of image-agnostic features that are used to generate noisy heuristics, whose outputs are aggregated using a factor graph-based generative model. With as few as 10 labeled examples per relationship, the generative model creates enough training data to train any existing state-of-the-art scene graph model. We demonstrate that our method outperforms all baseline approaches on scene graph prediction by 5.16 recall@100 for PREDCLS. In our limited label setting, we define a complexity metric for relationships that serves as an indicator (R^2 = 0.778) for conditions under which our method succeeds over transfer learning, the de-facto approach for training with limited labels.
△ Less
Submitted 30 November, 2019; v1 submitted 25 April, 2019;
originally announced April 2019.
-
Learning Dependency Structures for Weak Supervision Models
Authors:
Paroma Varma,
Frederic Sala,
Ann He,
Alexander Ratner,
Christopher Ré
Abstract:
Labeling training data is a key bottleneck in the modern machine learning pipeline. Recent weak supervision approaches combine labels from multiple noisy sources by estimating their accuracies without access to ground truth labels; however, estimating the dependencies among these sources is a critical challenge. We focus on a robust PCA-based algorithm for learning these dependency structures, est…
▽ More
Labeling training data is a key bottleneck in the modern machine learning pipeline. Recent weak supervision approaches combine labels from multiple noisy sources by estimating their accuracies without access to ground truth labels; however, estimating the dependencies among these sources is a critical challenge. We focus on a robust PCA-based algorithm for learning these dependency structures, establish improved theoretical recovery rates, and outperform existing methods on various real-world tasks. Under certain conditions, we show that the amount of unlabeled data needed can scale sublinearly or even logarithmically with the number of sources $m$, improving over previous efforts that ignore the sparsity pattern in the dependency structure and scale linearly in $m$. We provide an information-theoretic lower bound on the minimum sample complexity of the weak supervision setting. Our method outperforms weak supervision approaches that assume conditionally-independent sources by up to 4.64 F1 points and previous structure learning approaches by up to 4.41 F1 points on real-world relation extraction and image classification tasks.
△ Less
Submitted 14 March, 2019;
originally announced March 2019.
-
Training Classifiers with Natural Language Explanations
Authors:
Braden Hancock,
Paroma Varma,
Stephanie Wang,
Martin Bringmann,
Percy Liang,
Christopher Ré
Abstract:
Training accurate classifiers requires many labels, but each label provides only limited information (one bit for binary classification). In this work, we propose BabbleLabble, a framework for training classifiers in which an annotator provides a natural language explanation for each labeling decision. A semantic parser converts these explanations into programmatic labeling functions that generate…
▽ More
Training accurate classifiers requires many labels, but each label provides only limited information (one bit for binary classification). In this work, we propose BabbleLabble, a framework for training classifiers in which an annotator provides a natural language explanation for each labeling decision. A semantic parser converts these explanations into programmatic labeling functions that generate noisy labels for an arbitrary amount of unlabeled data, which is used to train a classifier. On three relation extraction tasks, we find that users are able to train classifiers with comparable F1 scores from 5-100$\times$ faster by providing explanations instead of just labels. Furthermore, given the inherent imperfection of labeling functions, we find that a simple rule-based semantic parser suffices.
△ Less
Submitted 25 August, 2018; v1 submitted 10 May, 2018;
originally announced May 2018.
-
Inferring Generative Model Structure with Static Analysis
Authors:
Paroma Varma,
Bryan He,
Payal Bajaj,
Imon Banerjee,
Nishith Khandwala,
Daniel L. Rubin,
Christopher Ré
Abstract:
Obtaining enough labeled data to robustly train complex discriminative models is a major bottleneck in the machine learning pipeline. A popular solution is combining multiple sources of weak supervision using generative models. The structure of these models affects training label quality, but is difficult to learn without any ground truth labels. We instead rely on these weak supervision sources h…
▽ More
Obtaining enough labeled data to robustly train complex discriminative models is a major bottleneck in the machine learning pipeline. A popular solution is combining multiple sources of weak supervision using generative models. The structure of these models affects training label quality, but is difficult to learn without any ground truth labels. We instead rely on these weak supervision sources having some structure by virtue of being encoded programmatically. We present Coral, a paradigm that infers generative model structure by statically analyzing the code for these heuristics, thus reducing the data required to learn structure significantly. We prove that Coral's sample complexity scales quasilinearly with the number of heuristics and number of relations found, improving over the standard sample complexity, which is exponential in $n$ for identifying $n^{\textrm{th}}$ degree relations. Experimentally, Coral matches or outperforms traditional structure learning approaches by up to 3.81 F1 points. Using Coral to model dependencies instead of assuming independence results in better performance than a fully supervised model by 3.07 accuracy points when heuristics are used to label radiology data without ground truth labels.
△ Less
Submitted 7 September, 2017;
originally announced September 2017.
-
Socratic Learning: Augmenting Generative Models to Incorporate Latent Subsets in Training Data
Authors:
Paroma Varma,
Bryan He,
Dan Iter,
Peng Xu,
Rose Yu,
Christopher De Sa,
Christopher Ré
Abstract:
A challenge in training discriminative models like neural networks is obtaining enough labeled training data. Recent approaches use generative models to combine weak supervision sources, like user-defined heuristics or knowledge bases, to label training data. Prior work has explored learning accuracies for these sources even without ground truth labels, but they assume that a single accuracy param…
▽ More
A challenge in training discriminative models like neural networks is obtaining enough labeled training data. Recent approaches use generative models to combine weak supervision sources, like user-defined heuristics or knowledge bases, to label training data. Prior work has explored learning accuracies for these sources even without ground truth labels, but they assume that a single accuracy parameter is sufficient to model the behavior of these sources over the entire training set. In particular, they fail to model latent subsets in the training data in which the supervision sources perform differently than on average. We present Socratic learning, a paradigm that uses feedback from a corresponding discriminative model to automatically identify these subsets and augments the structure of the generative model accordingly. Experimentally, we show that without any ground truth labels, the augmented generative model reduces error by up to 56.06% for a relation extraction task compared to a state-of-the-art weak supervision technique that utilizes generative models.
△ Less
Submitted 28 September, 2017; v1 submitted 25 October, 2016;
originally announced October 2016.
-
Closed-form Approximations for Coverage and Rate in a Multi-tier Heterogeneous Network in Nakagami-m Fading
Authors:
G. V. S. S. Praneeth Varma,
G. V. V. Sharma,
A. Kumar
Abstract:
In this paper, we consider the downlink in a K-tier heterogeneous network in the presence of Nakagami-m fading and noise. For such a system, we derive closed-form approximations of coverage probability and average rate achievable. A piece-wise linear approximation is employed in obtaining the simplified expressions. The proposed results are verified numerically through simulations. A comparison wi…
▽ More
In this paper, we consider the downlink in a K-tier heterogeneous network in the presence of Nakagami-m fading and noise. For such a system, we derive closed-form approximations of coverage probability and average rate achievable. A piece-wise linear approximation is employed in obtaining the simplified expressions. The proposed results are verified numerically through simulations. A comparison with existing work shows that the proposed work is a good approximation.
△ Less
Submitted 25 April, 2016;
originally announced April 2016.
