-
The Sky's the Limit: Re-lightable Outdoor Scenes via a Sky-pixel Constrained Illumination Prior and Outside-In Visibility
Authors:
James A. D. Gardner,
Evgenii Kashin,
Bernhard Egger,
William A. P. Smith
Abstract:
Inverse rendering of outdoor scenes from unconstrained image collections is a challenging task, particularly illumination/albedo ambiguities and occlusion of the illumination environment (shadowing) caused by geometry. However, there are many cues in an image that can aid in the disentanglement of geometry, albedo and shadows. Whilst sky is frequently masked out in state-of-the-art methods, we exp…
▽ More
Inverse rendering of outdoor scenes from unconstrained image collections is a challenging task, particularly illumination/albedo ambiguities and occlusion of the illumination environment (shadowing) caused by geometry. However, there are many cues in an image that can aid in the disentanglement of geometry, albedo and shadows. Whilst sky is frequently masked out in state-of-the-art methods, we exploit the fact that any sky pixel provides a direct observation of distant lighting in the corresponding direction and, via a neural illumination prior, a statistical cue to derive the remaining illumination environment. The incorporation of our illumination prior is enabled by a novel `outside-in' method for computing differentiable sky visibility based on a neural directional distance function. This is highly efficient and can be trained in parallel with the neural scene representation, allowing gradients from appearance loss to flow from shadows to influence the estimation of illumination and geometry. Our method estimates high-quality albedo, geometry, illumination and sky visibility, achieving state-of-the-art results on the NeRF-OSR relighting benchmark. Our code and models can be found at https://github.com/JADGardner/neusky
△ Less
Submitted 30 July, 2024; v1 submitted 28 November, 2023;
originally announced November 2023.
-
RENI++ A Rotation-Equivariant, Scale-Invariant, Natural Illumination Prior
Authors:
James A. D. Gardner,
Bernhard Egger,
William A. P. Smith
Abstract:
Inverse rendering is an ill-posed problem. Previous work has sought to resolve this by focussing on priors for object or scene shape or appearance. In this work, we instead focus on a prior for natural illuminations. Current methods rely on spherical harmonic lighting or other generic representations and, at best, a simplistic prior on the parameters. This results in limitations for the inverse se…
▽ More
Inverse rendering is an ill-posed problem. Previous work has sought to resolve this by focussing on priors for object or scene shape or appearance. In this work, we instead focus on a prior for natural illuminations. Current methods rely on spherical harmonic lighting or other generic representations and, at best, a simplistic prior on the parameters. This results in limitations for the inverse setting in terms of the expressivity of the illumination conditions, especially when taking specular reflections into account. We propose a conditional neural field representation based on a variational auto-decoder and a transformer decoder. We extend Vector Neurons to build equivariance directly into our architecture, and leveraging insights from depth estimation through a scale-invariant loss function, we enable the accurate representation of High Dynamic Range (HDR) images. The result is a compact, rotation-equivariant HDR neural illumination model capable of capturing complex, high-frequency features in natural environment maps. Training our model on a curated dataset of 1.6K HDR environment maps of natural scenes, we compare it against traditional representations, demonstrate its applicability for an inverse rendering task and show environment map completion from partial observations. We share our PyTorch implementation, dataset and trained models at https://github.com/JADGardner/ns_reni
△ Less
Submitted 15 November, 2023;
originally announced November 2023.
-
Integrated dataset for air travel and reported Zika virus cases in Colombia (Data and Resources Paper)
Authors:
Aiman Soliman,
Priyam Mazumdar,
Aaron Hoyle-Katz,
Brian Allan,
Allison Gardner
Abstract:
This open-access dataset provides consistent records of air travel volumes between 205 airport catchments in Colombia and the associated number of reported human cases of Zika virus within these catchments during the arbovirus outbreak between October 2015 and September 2016. We associated in this dataset the monthly air travel volumes provided by the Colombian Civil Aviation Authority (AEROCIVIL)…
▽ More
This open-access dataset provides consistent records of air travel volumes between 205 airport catchments in Colombia and the associated number of reported human cases of Zika virus within these catchments during the arbovirus outbreak between October 2015 and September 2016. We associated in this dataset the monthly air travel volumes provided by the Colombian Civil Aviation Authority (AEROCIVIL) with the reported human cases of Zika Virus published by The Pan American Health Organization (PAHO). Our methodology consists of geocoding all the reported airports and identifying the catchment of each airport using the municipalities' boundaries since reported human cases of Zika Virus are available at the municipal level. In addition, we calculated the total population at risk in each airport catchment by combining the total population count in a catchment with the environmental suitability of the Aedes aegypti mosquito, the vector for the Zika virus. We separated the monthly air travel volumes into domestic and international based on the location of the origin airport. The current dataset includes the total air travel volumes of 23,539,364 passengers on domestic flights and 11,592,197 on international ones. We validated our dataset by comparing the monthly aggregated air travel volumes between airport catchments to those predicted by the gravity model. We hope the novel dataset will provide a resource to researchers studying the role of human mobility in the spread of mosquito-borne diseases and modeling disease spread in realistic networks.
△ Less
Submitted 14 August, 2023;
originally announced August 2023.
-
Synthetic pre-training for neural-network interatomic potentials
Authors:
John L. A. Gardner,
Kathryn T. Baker,
Volker L. Deringer
Abstract:
Machine learning (ML) based interatomic potentials have transformed the field of atomistic materials modelling. However, ML potentials depend critically on the quality and quantity of quantum-mechanical reference data with which they are trained, and therefore developing datasets and training pipelines is becoming an increasingly central challenge. Leveraging the idea of "synthetic" (artificial) d…
▽ More
Machine learning (ML) based interatomic potentials have transformed the field of atomistic materials modelling. However, ML potentials depend critically on the quality and quantity of quantum-mechanical reference data with which they are trained, and therefore developing datasets and training pipelines is becoming an increasingly central challenge. Leveraging the idea of "synthetic" (artificial) data that is common in other areas of ML research, we here show that synthetic atomistic data, themselves obtained at scale with an existing ML potential, constitute a useful pre-training task for neural-network interatomic potential models. Once pre-trained with a large synthetic dataset, these models can be fine-tuned on a much smaller, quantum-mechanical one, improving numerical accuracy and stability in computational practice. We demonstrate feasibility for a series of equivariant graph-neural-network potentials for carbon, and we carry out initial experiments to test the limits of the approach.
△ Less
Submitted 24 July, 2023;
originally announced July 2023.
-
Beyond Intuition, a Framework for Applying GPs to Real-World Data
Authors:
Kenza Tazi,
Jihao Andreas Lin,
Ross Viljoen,
Alex Gardner,
ST John,
Hong Ge,
Richard E. Turner
Abstract:
Gaussian Processes (GPs) offer an attractive method for regression over small, structured and correlated datasets. However, their deployment is hindered by computational costs and limited guidelines on how to apply GPs beyond simple low-dimensional datasets. We propose a framework to identify the suitability of GPs to a given problem and how to set up a robust and well-specified GP model. The guid…
▽ More
Gaussian Processes (GPs) offer an attractive method for regression over small, structured and correlated datasets. However, their deployment is hindered by computational costs and limited guidelines on how to apply GPs beyond simple low-dimensional datasets. We propose a framework to identify the suitability of GPs to a given problem and how to set up a robust and well-specified GP model. The guidelines formalise the decisions of experienced GP practitioners, with an emphasis on kernel design and options for computational scalability. The framework is then applied to a case study of glacier elevation change yielding more accurate results at test time.
△ Less
Submitted 17 July, 2023; v1 submitted 6 July, 2023;
originally announced July 2023.
-
Synthetic data enable experiments in atomistic machine learning
Authors:
John L. A. Gardner,
Zoé Faure Beaulieu,
Volker L. Deringer
Abstract:
Machine-learning models are increasingly used to predict properties of atoms in chemical systems. There have been major advances in developing descriptors and regression frameworks for this task, typically starting from (relatively) small sets of quantum-mechanical reference data. Larger datasets of this kind are becoming available, but remain expensive to generate. Here we demonstrate the use of…
▽ More
Machine-learning models are increasingly used to predict properties of atoms in chemical systems. There have been major advances in developing descriptors and regression frameworks for this task, typically starting from (relatively) small sets of quantum-mechanical reference data. Larger datasets of this kind are becoming available, but remain expensive to generate. Here we demonstrate the use of a large dataset that we have "synthetically" labelled with per-atom energies from an existing ML potential model. The cheapness of this process, compared to the quantum-mechanical ground truth, allows us to generate millions of datapoints, in turn enabling rapid experimentation with atomistic ML models from the small- to the large-data regime. This approach allows us here to compare regression frameworks in depth, and to explore visualisation based on learned representations. We also show that learning synthetic data labels can be a useful pre-training task for subsequent fine-tuning on small datasets. In the future, we expect that our open-sourced dataset, and similar ones, will be useful in rapidly exploring deep-learning models in the limit of abundant chemical data.
△ Less
Submitted 29 November, 2022;
originally announced November 2022.
-
Rotation-Equivariant Conditional Spherical Neural Fields for Learning a Natural Illumination Prior
Authors:
James A. D. Gardner,
Bernhard Egger,
William A. P. Smith
Abstract:
Inverse rendering is an ill-posed problem. Previous work has sought to resolve this by focussing on priors for object or scene shape or appearance. In this work, we instead focus on a prior for natural illuminations. Current methods rely on spherical harmonic lighting or other generic representations and, at best, a simplistic prior on the parameters. We propose a conditional neural field represen…
▽ More
Inverse rendering is an ill-posed problem. Previous work has sought to resolve this by focussing on priors for object or scene shape or appearance. In this work, we instead focus on a prior for natural illuminations. Current methods rely on spherical harmonic lighting or other generic representations and, at best, a simplistic prior on the parameters. We propose a conditional neural field representation based on a variational auto-decoder with a SIREN network and, extending Vector Neurons, build equivariance directly into the network. Using this, we develop a rotation-equivariant, high dynamic range (HDR) neural illumination model that is compact and able to express complex, high-frequency features of natural environment maps. Training our model on a curated dataset of 1.6K HDR environment maps of natural scenes, we compare it against traditional representations, demonstrate its applicability for an inverse rendering task and show environment map completion from partial observations. A PyTorch implementation, our dataset and trained models can be found at jadgardner.github.io/RENI.
△ Less
Submitted 14 October, 2022; v1 submitted 7 June, 2022;
originally announced June 2022.
-
On partitioning of an SHM problem and parallels with transfer learning
Authors:
G. Tsialiamanis,
D. J. Wagg,
P. A. Gardner,
N. Dervilis,
K. Worden
Abstract:
In the current work, a problem-splitting approach and a scheme motivated by transfer learning is applied to a structural health monitoring problem. The specific problem in this case is that of localising damage on an aircraft wing. The original experiment is described, together with the initial approach, in which a neural network was trained to localise damage. The results were not ideal, partly b…
▽ More
In the current work, a problem-splitting approach and a scheme motivated by transfer learning is applied to a structural health monitoring problem. The specific problem in this case is that of localising damage on an aircraft wing. The original experiment is described, together with the initial approach, in which a neural network was trained to localise damage. The results were not ideal, partly because of a scarcity of training data, and partly because of the difficulty in resolving two of the damage cases. In the current paper, the problem is split into two sub-problems and an increase in classification accuracy is obtained. The sub-problems are obtained by separating out the most difficult-to-classify damage cases. A second approach to the problem is considered by adopting ideas from transfer learning (usually applied in much deeper) networks to see if a network trained on the simpler damage cases can help with feature extraction in the more difficult cases. The transfer of a fixed trained batch of layers between the networks is found to improve classification by making the classes more separable in the feature space and to speed up convergence.
△ Less
Submitted 3 March, 2022;
originally announced March 2022.
-
Bayesian Modelling of Multivalued Power Curves from an Operational Wind Farm
Authors:
L. A. Bull,
P. A. Gardner,
T. J. Rogers,
N. Dervilis,
E. J. Cross,
E. Papatheou,
A. E. Maguire,
C. Campos,
K. Worden
Abstract:
Power curves capture the relationship between wind speed and output power for a specific wind turbine. Accurate regression models of this function prove useful in monitoring, maintenance, design, and planning. In practice, however, the measurements do not always correspond to the ideal curve: power curtailments will appear as (additional) functional components. Such multivalued relationships canno…
▽ More
Power curves capture the relationship between wind speed and output power for a specific wind turbine. Accurate regression models of this function prove useful in monitoring, maintenance, design, and planning. In practice, however, the measurements do not always correspond to the ideal curve: power curtailments will appear as (additional) functional components. Such multivalued relationships cannot be modelled by conventional regression, and the associated data are usually removed during pre-processing. The current work suggests an alternative method to infer multivalued relationships in curtailed power data. Using a population-based approach, an overlapping mixture of probabilistic regression models is applied to signals recorded from turbines within an operational wind farm. The model is shown to provide an accurate representation of practical power data across the population.
△ Less
Submitted 30 November, 2021;
originally announced November 2021.
-
DVERGE: Diversifying Vulnerabilities for Enhanced Robust Generation of Ensembles
Authors:
Huanrui Yang,
Jingyang Zhang,
Hongliang Dong,
Nathan Inkawhich,
Andrew Gardner,
Andrew Touchet,
Wesley Wilkes,
Heath Berry,
Hai Li
Abstract:
Recent research finds CNN models for image classification demonstrate overlapped adversarial vulnerabilities: adversarial attacks can mislead CNN models with small perturbations, which can effectively transfer between different models trained on the same dataset. Adversarial training, as a general robustness improvement technique, eliminates the vulnerability in a single model by forcing it to lea…
▽ More
Recent research finds CNN models for image classification demonstrate overlapped adversarial vulnerabilities: adversarial attacks can mislead CNN models with small perturbations, which can effectively transfer between different models trained on the same dataset. Adversarial training, as a general robustness improvement technique, eliminates the vulnerability in a single model by forcing it to learn robust features. The process is hard, often requires models with large capacity, and suffers from significant loss on clean data accuracy. Alternatively, ensemble methods are proposed to induce sub-models with diverse outputs against a transfer adversarial example, making the ensemble robust against transfer attacks even if each sub-model is individually non-robust. Only small clean accuracy drop is observed in the process. However, previous ensemble training methods are not efficacious in inducing such diversity and thus ineffective on reaching robust ensemble. We propose DVERGE, which isolates the adversarial vulnerability in each sub-model by distilling non-robust features, and diversifies the adversarial vulnerability to induce diverse outputs against a transfer attack. The novel diversity metric and training procedure enables DVERGE to achieve higher robustness against transfer attacks comparing to previous ensemble methods, and enables the improved robustness when more sub-models are added to the ensemble. The code of this work is available at https://github.com/zjysteven/DVERGE
△ Less
Submitted 18 October, 2020; v1 submitted 30 September, 2020;
originally announced September 2020.
-
Deep Detector Health Management under Adversarial Campaigns
Authors:
Javier Echauz,
Keith Kenemer,
Sarfaraz Hussein,
Jay Dhaliwal,
Saurabh Shintre,
Slawomir Grzonkowski,
Andrew Gardner
Abstract:
Machine learning models are vulnerable to adversarial inputs that induce seemingly unjustifiable errors. As automated classifiers are increasingly used in industrial control systems and machinery, these adversarial errors could grow to be a serious problem. Despite numerous studies over the past few years, the field of adversarial ML is still considered alchemy, with no practical unbroken defenses…
▽ More
Machine learning models are vulnerable to adversarial inputs that induce seemingly unjustifiable errors. As automated classifiers are increasingly used in industrial control systems and machinery, these adversarial errors could grow to be a serious problem. Despite numerous studies over the past few years, the field of adversarial ML is still considered alchemy, with no practical unbroken defenses demonstrated to date, leaving PHM practitioners with few meaningful ways of addressing the problem. We introduce turbidity detection as a practical superset of the adversarial input detection problem, coping with adversarial campaigns rather than statistically invisible one-offs. This perspective is coupled with ROC-theoretic design guidance that prescribes an inexpensive domain adaptation layer at the output of a deep learning model during an attack campaign. The result aims to approximate the Bayes optimal mitigation that ameliorates the detection model's degraded health. A proactively reactive type of prognostics is achieved via Monte Carlo simulation of various adversarial campaign scenarios, by sampling from the model's own turbidity distribution to quickly deploy the correct mitigation during a real-world campaign.
△ Less
Submitted 18 November, 2019;
originally announced November 2019.
-
Detecting DGA domains with recurrent neural networks and side information
Authors:
Ryan R. Curtin,
Andrew B. Gardner,
Slawomir Grzonkowski,
Alexey Kleymenov,
Alejandro Mosquera
Abstract:
Modern malware typically makes use of a domain generation algorithm (DGA) to avoid command and control domains or IPs being seized or sinkholed. This means that an infected system may attempt to access many domains in an attempt to contact the command and control server. Therefore, the automatic detection of DGA domains is an important task, both for the sake of blocking malicious domains and iden…
▽ More
Modern malware typically makes use of a domain generation algorithm (DGA) to avoid command and control domains or IPs being seized or sinkholed. This means that an infected system may attempt to access many domains in an attempt to contact the command and control server. Therefore, the automatic detection of DGA domains is an important task, both for the sake of blocking malicious domains and identifying compromised hosts. However, many DGAs use English wordlists to generate plausibly clean-looking domain names; this makes automatic detection difficult. In this work, we devise a notion of difficulty for DGA families called the smashword score; this measures how much a DGA family looks like English words. We find that this measure accurately reflects how much a DGA family's domains look like they are made from natural English words. We then describe our new modeling approach, which is a combination of a novel recurrent neural network architecture with domain registration side information. Our experiments show the model is capable of effectively identifying domains generated by difficult DGA families. Our experiments also show that our model outperforms existing approaches, and is able to reliably detect difficult DGA families such as matsnu, suppobox, rovnix, and others. The model's performance compared to the state of the art is best for DGA families that resemble English words. We believe that this model could either be used in a standalone DGA domain detector---such as an endpoint security application---or alternately the model could be used as a part of a larger malware detection system.
△ Less
Submitted 21 June, 2019; v1 submitted 3 October, 2018;
originally announced October 2018.
-
Classifying Unordered Feature Sets with Convolutional Deep Averaging Networks
Authors:
Andrew Gardner,
Jinko Kanno,
Christian A. Duncan,
Rastko R. Selmic
Abstract:
Unordered feature sets are a nonstandard data structure that traditional neural networks are incapable of addressing in a principled manner. Providing a concatenation of features in an arbitrary order may lead to the learning of spurious patterns or biases that do not actually exist. Another complication is introduced if the number of features varies between each set. We propose convolutional deep…
▽ More
Unordered feature sets are a nonstandard data structure that traditional neural networks are incapable of addressing in a principled manner. Providing a concatenation of features in an arbitrary order may lead to the learning of spurious patterns or biases that do not actually exist. Another complication is introduced if the number of features varies between each set. We propose convolutional deep averaging networks (CDANs) for classifying and learning representations of datasets whose instances comprise variable-size, unordered feature sets. CDANs are efficient, permutation-invariant, and capable of accepting sets of arbitrary size. We emphasize the importance of nonlinear feature embeddings for obtaining effective CDAN classifiers and illustrate their advantages in experiments versus linear embeddings and alternative permutation-invariant and -equivariant architectures.
△ Less
Submitted 9 September, 2017;
originally announced September 2017.
-
Detecting Adversarial Samples from Artifacts
Authors:
Reuben Feinman,
Ryan R. Curtin,
Saurabh Shintre,
Andrew B. Gardner
Abstract:
Deep neural networks (DNNs) are powerful nonlinear architectures that are known to be robust to random perturbations of the input. However, these models are vulnerable to adversarial perturbations--small input changes crafted explicitly to fool the model. In this paper, we ask whether a DNN can distinguish adversarial samples from their normal and noisy counterparts. We investigate model confidenc…
▽ More
Deep neural networks (DNNs) are powerful nonlinear architectures that are known to be robust to random perturbations of the input. However, these models are vulnerable to adversarial perturbations--small input changes crafted explicitly to fool the model. In this paper, we ask whether a DNN can distinguish adversarial samples from their normal and noisy counterparts. We investigate model confidence on adversarial samples by looking at Bayesian uncertainty estimates, available in dropout neural networks, and by performing density estimation in the subspace of deep features learned by the model. The result is a method for implicit adversarial detection that is oblivious to the attack algorithm. We evaluate this method on a variety of standard datasets including MNIST and CIFAR-10 and show that it generalizes well across different architectures and attacks. Our findings report that 85-93% ROC-AUC can be achieved on a number of standard classification tasks with a negative class that consists of both normal and noisy samples.
△ Less
Submitted 15 November, 2017; v1 submitted 1 March, 2017;
originally announced March 2017.
-
Fast approximate furthest neighbors with data-dependent hashing
Authors:
Ryan R. Curtin,
Andrew B. Gardner
Abstract:
We present a novel hashing strategy for approximate furthest neighbor search that selects projection bases using the data distribution. This strategy leads to an algorithm, which we call DrusillaHash, that is able to outperform existing approximate furthest neighbor strategies. Our strategy is motivated by an empirical study of the behavior of the furthest neighbor search problem, which lends intu…
▽ More
We present a novel hashing strategy for approximate furthest neighbor search that selects projection bases using the data distribution. This strategy leads to an algorithm, which we call DrusillaHash, that is able to outperform existing approximate furthest neighbor strategies. Our strategy is motivated by an empirical study of the behavior of the furthest neighbor search problem, which lends intuition for where our algorithm is most useful. We also present a variant of the algorithm that gives an absolute approximation guarantee; to our knowledge, this is the first such approximate furthest neighbor hashing approach to give such a guarantee. Performance studies indicate that DrusillaHash can achieve comparable levels of approximation to other algorithms while giving up to an order of magnitude speedup. An implementation is available in the mlpack machine learning library (found at http://www.mlpack.org).
△ Less
Submitted 31 May, 2016;
originally announced May 2016.
-
On the Definiteness of Earth Mover's Distance and Its Relation to Set Intersection
Authors:
Andrew Gardner,
Christian A. Duncan,
Jinko Kanno,
Rastko R. Selmic
Abstract:
Positive definite kernels are an important tool in machine learning that enable efficient solutions to otherwise difficult or intractable problems by implicitly linearizing the problem geometry. In this paper we develop a set-theoretic interpretation of the Earth Mover's Distance (EMD) and propose Earth Mover's Intersection (EMI), a positive definite analog to EMD for sets of different sizes. We p…
▽ More
Positive definite kernels are an important tool in machine learning that enable efficient solutions to otherwise difficult or intractable problems by implicitly linearizing the problem geometry. In this paper we develop a set-theoretic interpretation of the Earth Mover's Distance (EMD) and propose Earth Mover's Intersection (EMI), a positive definite analog to EMD for sets of different sizes. We provide conditions under which EMD or certain approximations to EMD are negative definite. We also present a positive-definite-preserving transformation that can be applied to any kernel and can also be used to derive positive definite EMD-based kernels and show that the Jaccard index is simply the result of this transformation. Finally, we evaluate kernels based on EMI and the proposed transformation versus EMD in various computer vision tasks and show that EMD is generally inferior even with indefinite kernel techniques.
△ Less
Submitted 21 August, 2018; v1 submitted 9 October, 2015;
originally announced October 2015.
