-
Poisoning Web-Scale Training Datasets is Practical
Authors:
Nicholas Carlini,
Matthew Jagielski,
Christopher A. Choquette-Choo,
Daniel Paleka,
Will Pearce,
Hyrum Anderson,
Andreas Terzis,
Kurt Thomas,
Florian Tramèr
Abstract:
Deep learning models are often trained on distributed, web-scale datasets crawled from the internet. In this paper, we introduce two new dataset poisoning attacks that intentionally introduce malicious examples to a model's performance. Our attacks are immediately practical and could, today, poison 10 popular datasets. Our first attack, split-view poisoning, exploits the mutable nature of internet…
▽ More
Deep learning models are often trained on distributed, web-scale datasets crawled from the internet. In this paper, we introduce two new dataset poisoning attacks that intentionally introduce malicious examples to a model's performance. Our attacks are immediately practical and could, today, poison 10 popular datasets. Our first attack, split-view poisoning, exploits the mutable nature of internet content to ensure a dataset annotator's initial view of the dataset differs from the view downloaded by subsequent clients. By exploiting specific invalid trust assumptions, we show how we could have poisoned 0.01% of the LAION-400M or COYO-700M datasets for just $60 USD. Our second attack, frontrunning poisoning, targets web-scale datasets that periodically snapshot crowd-sourced content -- such as Wikipedia -- where an attacker only needs a time-limited window to inject malicious examples. In light of both attacks, we notify the maintainers of each affected dataset and recommended several low-overhead defenses.
△ Less
Submitted 6 May, 2024; v1 submitted 20 February, 2023;
originally announced February 2023.
-
Bad Citrus: Reducing Adversarial Costs with Model Distances
Authors:
Giorgio Severi,
Will Pearce,
Alina Oprea
Abstract:
Recent work by Jia et al., showed the possibility of effectively computing pairwise model distances in weight space, using a model explanation technique known as LIME. This method requires query-only access to the two models under examination. We argue this insight can be leveraged by an adversary to reduce the net cost (number of queries) of launching an evasion campaign against a deployed model.…
▽ More
Recent work by Jia et al., showed the possibility of effectively computing pairwise model distances in weight space, using a model explanation technique known as LIME. This method requires query-only access to the two models under examination. We argue this insight can be leveraged by an adversary to reduce the net cost (number of queries) of launching an evasion campaign against a deployed model. We show that there is a strong negative correlation between the success rate of adversarial transfer and the distance between the victim model and the surrogate used to generate the evasive samples. Thus, we propose and evaluate a method to reduce adversarial costs by finding the closest surrogate model for adversarial transfer.
△ Less
Submitted 6 October, 2022;
originally announced October 2022.
-
Growing polarisation around climate change on social media
Authors:
Max Falkenberg,
Alessandro Galeazzi,
Maddalena Torricelli,
Niccolo Di Marco,
Francesca Larosa,
Madalina Sas,
Amin Mekacher,
Warren Pearce,
Fabiana Zollo,
Walter Quattrociocchi,
Andrea Baronchelli
Abstract:
Climate change and political polarisation are two of the 21st century's critical socio-political issues. Here, we investigate their intersection by studying the discussion around the UN Conference of The Parties on Climate Change (COP) using Twitter data from 2014 to 2021. First, we reveal a large increase in ideological polarisation during COP26, following low polarisation between COP20 and COP25…
▽ More
Climate change and political polarisation are two of the 21st century's critical socio-political issues. Here, we investigate their intersection by studying the discussion around the UN Conference of The Parties on Climate Change (COP) using Twitter data from 2014 to 2021. First, we reveal a large increase in ideological polarisation during COP26, following low polarisation between COP20 and COP25. Second, we show that this increase is driven by growing right-wing activity, a 4-fold increase since COP21 relative to pro-climate groups. Finally, we identify a broad range of ''climate contrarian'' views during COP26, emphasising the theme of ''political hypocrisy'' as a topic of cross-ideological appeal; contrarian views and accusations of hypocrisy have become key themes in the Twitter climate discussion since 2019. With future climate action reliant on negotiations at COP27 and beyond, our results highlight the importance of monitoring polarisation, and its impacts, in the public climate discourse.
△ Less
Submitted 14 November, 2022; v1 submitted 22 December, 2021;
originally announced December 2021.
-
Machine Learning for Offensive Security: Sandbox Classification Using Decision Trees and Artificial Neural Networks
Authors:
Will Pearce,
Nick Landers,
Nancy Fulda
Abstract:
The merits of machine learning in information security have primarily focused on bolstering defenses. However, machine learning (ML) techniques are not reserved for organizations with deep pockets and massive data repositories; the democratization of ML has lead to a rise in the number of security teams using ML to support offensive operations. The research presented here will explore two models t…
▽ More
The merits of machine learning in information security have primarily focused on bolstering defenses. However, machine learning (ML) techniques are not reserved for organizations with deep pockets and massive data repositories; the democratization of ML has lead to a rise in the number of security teams using ML to support offensive operations. The research presented here will explore two models that our team has used to solve a single offensive task, detecting a sandbox. Using process list data gathered with phishing emails, we will demonstrate the use of Decision Trees and Artificial Neural Networks to successfully classify sandboxes, thereby avoiding unsafe execution. This paper aims to give unique insight into how a real offensive team is using machine learning to support offensive operations.
△ Less
Submitted 13 July, 2020;
originally announced July 2020.
