-
Optimizing Black-box Metrics with Iterative Example Weighting
Authors:
Gaurush Hiranandani,
Jatin Mathur,
Harikrishna Narasimhan,
Mahdi Milani Fard,
Oluwasanmi Koyejo
Abstract:
We consider learning to optimize a classification metric defined by a black-box function of the confusion matrix. Such black-box learning settings are ubiquitous, for example, when the learner only has query access to the metric of interest, or in noisy-label and domain adaptation applications where the learner must evaluate the metric via performance evaluation using a small validation sample. Ou…
▽ More
We consider learning to optimize a classification metric defined by a black-box function of the confusion matrix. Such black-box learning settings are ubiquitous, for example, when the learner only has query access to the metric of interest, or in noisy-label and domain adaptation applications where the learner must evaluate the metric via performance evaluation using a small validation sample. Our approach is to adaptively learn example weights on the training dataset such that the resulting weighted objective best approximates the metric on the validation sample. We show how to model and estimate the example weights and use them to iteratively post-shift a pre-trained class probability estimator to construct a classifier. We also analyze the resulting procedure's statistical properties. Experiments on various label noise, domain shift, and fair classification setups confirm that our proposal compares favorably to the state-of-the-art baselines for each application.
△ Less
Submitted 23 June, 2021; v1 submitted 18 February, 2021;
originally announced February 2021.
-
Quadratic Metric Elicitation for Fairness and Beyond
Authors:
Gaurush Hiranandani,
Jatin Mathur,
Harikrishna Narasimhan,
Oluwasanmi Koyejo
Abstract:
Metric elicitation is a recent framework for eliciting classification performance metrics that best reflect implicit user preferences based on the task and context. However, available elicitation strategies have been limited to linear (or quasi-linear) functions of predictive rates, which can be practically restrictive for many applications including fairness. This paper develops a strategy for el…
▽ More
Metric elicitation is a recent framework for eliciting classification performance metrics that best reflect implicit user preferences based on the task and context. However, available elicitation strategies have been limited to linear (or quasi-linear) functions of predictive rates, which can be practically restrictive for many applications including fairness. This paper develops a strategy for eliciting more flexible multiclass metrics defined by quadratic functions of rates, designed to reflect human preferences better. We show its application in eliciting quadratic violation-based group-fair metrics. Our strategy requires only relative preference feedback, is robust to noise, and achieves near-optimal query complexity. We further extend this strategy to eliciting polynomial metrics -- thus broadening the use cases for metric elicitation.
△ Less
Submitted 21 August, 2022; v1 submitted 3 November, 2020;
originally announced November 2020.
-
Predicting cell phone adoption metrics using satellite imagery
Authors:
Edward J. Oughton,
Jatin Mathur
Abstract:
Approximately half of the global population does not have access to the internet, even though digital connectivity can reduce poverty by revolutionizing economic development opportunities. Due to a lack of data, Mobile Network Operators and governments struggle to effectively determine if infrastructure investments are viable, especially in greenfield areas where demand is unknown. This leads to a…
▽ More
Approximately half of the global population does not have access to the internet, even though digital connectivity can reduce poverty by revolutionizing economic development opportunities. Due to a lack of data, Mobile Network Operators and governments struggle to effectively determine if infrastructure investments are viable, especially in greenfield areas where demand is unknown. This leads to a lack of investment in network infrastructure, resulting in a phenomenon commonly referred to as the `digital divide`. In this paper we present a machine learning method that uses publicly available satellite imagery to predict telecoms demand metrics, including cell phone adoption and spending on mobile services, and apply the method to Malawi and Ethiopia. Our predictive machine learning approach consistently outperforms baseline models which use population density or nightlight luminosity, with an improvement in data variance prediction of at least 40%. The method is a starting point for developing more sophisticated predictive models of infrastructure demand using machine learning and publicly available satellite imagery. The evidence produced can help to better inform infrastructure investment and policy decisions.
△ Less
Submitted 8 June, 2021; v1 submitted 12 June, 2020;
originally announced June 2020.
