Showing 1–31 of 31 results for author: Lewis, J

On Exact Bit-level Reversible Transformers Without Changing Architectures

Authors: Guoqiang Zhang, J. P. Lewis, W. B. Kleijn

Abstract: In the literature, various reversible deep neural networks (DNN) models have been proposed to reduce memory consumption or improve data-throughput in the training process. However, almost all existing reversible DNNs either are constrained to have special structures or are constructed by modifying the original DNN architectures considerably to enable reversibility. In this work, we propose exact b… ▽ More In the literature, various reversible deep neural networks (DNN) models have been proposed to reduce memory consumption or improve data-throughput in the training process. However, almost all existing reversible DNNs either are constrained to have special structures or are constructed by modifying the original DNN architectures considerably to enable reversibility. In this work, we propose exact bit-level reversible transformers without changing the architectures in the inference procedure. The basic idea is to first treat each transformer block as the Euler integration approximation for solving an ordinary differential equation (ODE) and then incorporate the technique of bidirectional integration approximation (BDIA) (see [26]) for BDIA-based diffusion inversion) into the neural architecture together with activation quantization to make it exactly bit-level reversible, referred to as BDIA-transformer. In the training process, we let a hyper-parameter $γ$ in BDIA-transformer randomly take one of the two values $\{0.5, -0.5\}$ per transformer block for averaging two consecutive integration approximations, which regularizes the models for improving the validation accuracy. Light-weight side information per transformer block is required to be stored in the forward process to account for binary quantization loss to enable exact bit-level reversibility. In the inference procedure, the expectation $\mathbb{E}(γ)=0$ is taken to make the resulting architectures of BDIA-transformer be identical to transformers up to activation quantization. Empirical study indicates that BDIA-transformers outperform their original counterparts notably due to the regularization effect of the $γ$ parameter. △ Less

Submitted 12 July, 2024; originally announced July 2024.

Can AI Understand Human Personality? -- Comparing Human Experts and AI Systems at Predicting Personality Correlations

Authors: Philipp Schoenegger, Spencer Greenberg, Alexander Grishin, Joshua Lewis, Lucius Caviola

Abstract: We test the abilities of specialised deep neural networks like PersonalityMap as well as general LLMs like GPT-4o and Claude 3 Opus in understanding human personality. Specifically, we compare their ability to predict correlations between personality items to the abilities of lay people and academic experts. We find that when compared with individual humans, all AI models make better predictions t… ▽ More We test the abilities of specialised deep neural networks like PersonalityMap as well as general LLMs like GPT-4o and Claude 3 Opus in understanding human personality. Specifically, we compare their ability to predict correlations between personality items to the abilities of lay people and academic experts. We find that when compared with individual humans, all AI models make better predictions than the vast majority of lay people and academic experts. However, when selecting the median prediction for each item, we find a different pattern: Experts and PersonalityMap outperform LLMs and lay people on most measures. Our results suggest that while frontier LLMs' are better than most individual humans at predicting correlations between personality items, specialised models like PersonalityMap continue to match or exceed expert human performance even on some outcome measures where LLMs underperform. This provides evidence both in favour of the general capabilities of large language models and in favour of the continued place for specialised models trained and deployed for specific domains. △ Less

Submitted 12 June, 2024; originally announced June 2024.

Comments: 45 pages, 6 figures

MSC Class: K.4.0; J.4

Autonomous Navigation in Complex Environments

Authors: Andrew Gerstenslager, Jomol Lewis, Liam McKenna, Poorva Patel

Abstract: This paper explores the application of CNN-DNN network fusion to construct a robot navigation controller within a simulated environment. The simulated environment is constructed to model a subterranean rescue situation, such that an autonomous agent is tasked with finding a goal within an unknown cavernous system. Imitation learning is used to train the control algorithm to use LiDAR and camera da… ▽ More This paper explores the application of CNN-DNN network fusion to construct a robot navigation controller within a simulated environment. The simulated environment is constructed to model a subterranean rescue situation, such that an autonomous agent is tasked with finding a goal within an unknown cavernous system. Imitation learning is used to train the control algorithm to use LiDAR and camera data to navigate the space and find the goal. The trained model is then tested for robustness using Monte-Carlo. △ Less

Submitted 6 January, 2024; originally announced January 2024.

Comments: 7 pages, 3 figures, independent paper

TrailBlazer: Trajectory Control for Diffusion-Based Video Generation

Authors: Wan-Duo Kurt Ma, J. P. Lewis, W. Bastiaan Kleijn

Abstract: Within recent approaches to text-to-video (T2V) generation, achieving controllability in the synthesized video is often a challenge. Typically, this issue is addressed by providing low-level per-frame guidance in the form of edge maps, depth maps, or an existing video to be altered. However, the process of obtaining such guidance can be labor-intensive. This paper focuses on enhancing controllabil… ▽ More Within recent approaches to text-to-video (T2V) generation, achieving controllability in the synthesized video is often a challenge. Typically, this issue is addressed by providing low-level per-frame guidance in the form of edge maps, depth maps, or an existing video to be altered. However, the process of obtaining such guidance can be labor-intensive. This paper focuses on enhancing controllability in video synthesis by employing straightforward bounding boxes to guide the subject in various ways, all without the need for neural network training, finetuning, optimization at inference time, or the use of pre-existing videos. Our algorithm, TrailBlazer, is constructed upon a pre-trained (T2V) model, and easy to implement. The subject is directed by a bounding box through the proposed spatial and temporal attention map editing. Moreover, we introduce the concept of keyframing, allowing the subject trajectory and overall appearance to be guided by both a moving bounding box and corresponding prompts, without the need to provide a detailed mask. The method is efficient, with negligible additional computation relative to the underlying pre-trained model. Despite the simplicity of the bounding box guidance, the resulting motion is surprisingly natural, with emergent effects including perspective and movement toward the virtual camera as the box size increases. △ Less

Submitted 8 April, 2024; v1 submitted 31 December, 2023; originally announced January 2024.

Comments: 14 pages, 18 figures, Project Page: https://hohonu-vicml.github.io/Trailblazer.Page/

FDLS: A Deep Learning Approach to Production Quality, Controllable, and Retargetable Facial Performances

Authors: Wan-Duo Kurt Ma, Muhammad Ghifary, J. P. Lewis, Byungkuk Choi, Haekwang Eom

Abstract: Visual effects commonly requires both the creation of realistic synthetic humans as well as retargeting actors' performances to humanoid characters such as aliens and monsters. Achieving the expressive performances demanded in entertainment requires manipulating complex models with hundreds of parameters. Full creative control requires the freedom to make edits at any stage of the production, whic… ▽ More Visual effects commonly requires both the creation of realistic synthetic humans as well as retargeting actors' performances to humanoid characters such as aliens and monsters. Achieving the expressive performances demanded in entertainment requires manipulating complex models with hundreds of parameters. Full creative control requires the freedom to make edits at any stage of the production, which prohibits the use of a fully automatic ``black box'' solution with uninterpretable parameters. On the other hand, producing realistic animation with these sophisticated models is difficult and laborious. This paper describes FDLS (Facial Deep Learning Solver), which is Weta Digital's solution to these challenges. FDLS adopts a coarse-to-fine and human-in-the-loop strategy, allowing a solved performance to be verified and edited at several stages in the solving process. To train FDLS, we first transform the raw motion-captured data into robust graph features. Secondly, based on the observation that the artists typically finalize the jaw pass animation before proceeding to finer detail, we solve for the jaw motion first and predict fine expressions with region-based networks conditioned on the jaw position. Finally, artists can optionally invoke a non-linear finetuning process on top of the FDLS solution to follow the motion-captured virtual markers as closely as possible. FDLS supports editing if needed to improve the results of the deep learning solution and it can handle small daily changes in the actor's face shape. FDLS permits reliable and production-quality performance solving with minimal training and little or no manual effort in many cases, while also allowing the solve to be guided and edited in unusual and difficult cases. The system has been under development for several years and has been used in major movies. △ Less

Submitted 26 September, 2023; originally announced September 2023.

Comments: DigiPro '22: The Digital Production Symposium

Automated segmentation of rheumatoid arthritis immunohistochemistry stained synovial tissue

Authors: Amaya Gallagher-Syed, Abbas Khan, Felice Rivellese, Costantino Pitzalis, Myles J. Lewis, Gregory Slabaugh, Michael R. Barnes

Abstract: Rheumatoid Arthritis (RA) is a chronic, autoimmune disease which primarily affects the joint's synovial tissue. It is a highly heterogeneous disease, with wide cellular and molecular variability observed in synovial tissues. Over the last two decades, the methods available for their study have advanced considerably. In particular, Immunohistochemistry stains are well suited to highlighting the fun… ▽ More Rheumatoid Arthritis (RA) is a chronic, autoimmune disease which primarily affects the joint's synovial tissue. It is a highly heterogeneous disease, with wide cellular and molecular variability observed in synovial tissues. Over the last two decades, the methods available for their study have advanced considerably. In particular, Immunohistochemistry stains are well suited to highlighting the functional organisation of samples. Yet, analysis of IHC-stained synovial tissue samples is still overwhelmingly done manually and semi-quantitatively by expert pathologists. This is because in addition to the fragmented nature of IHC stained synovial tissue, there exist wide variations in intensity and colour, strong clinical centre batch effect, as well as the presence of many undesirable artefacts present in gigapixel Whole Slide Images (WSIs), such as water droplets, pen annotation, folded tissue, blurriness, etc. There is therefore a strong need for a robust, repeatable automated tissue segmentation algorithm which can cope with this variability and provide support to imaging pipelines. We train a UNET on a hand-curated, heterogeneous real-world multi-centre clinical dataset R4RA, which contains multiple types of IHC staining. The model obtains a DICE score of 0.865 and successfully segments different types of IHC staining, as well as dealing with variance in colours, intensity and common WSIs artefacts from the different clinical centres. It can be used as the first step in an automated image analysis pipeline for synovial tissue samples stained with IHC, increasing speed, reproducibility and robustness. △ Less

Submitted 13 September, 2023; originally announced September 2023.

The Still Secret Ballot: The Limited Privacy Cost of Transparent Election Results

Authors: Shiro Kuriwaki, Jeffrey B. Lewis, Michael Morse

Abstract: After an election, should officials release an electronic record of each ballot? The release of ballots could bolster the legitimacy of the result. But it may also facilitate vote revelation, where an analyst unravels the secret ballot by uniquely linking votes on an anonymous ballot to the voter's name and address in the public voter file. We first provide a theoretical model of how vote revelati… ▽ More After an election, should officials release an electronic record of each ballot? The release of ballots could bolster the legitimacy of the result. But it may also facilitate vote revelation, where an analyst unravels the secret ballot by uniquely linking votes on an anonymous ballot to the voter's name and address in the public voter file. We first provide a theoretical model of how vote revelation could occur under various election-reporting regimes. Perhaps counterintuitively, releasing ballot records is no more revelatory than the typical practice of releasing aggregate vote tallies by precinct and method. We then present the first empirical evaluation of vote revelation, using the 2020 election in Maricopa County, Arizona, as a case study. For 99.8% of voters, the release of ballot records led to no revelation of any vote choice. We conclude the ballot can be both public and still as secret as it is under typical reporting practices. △ Less

Submitted 9 April, 2024; v1 submitted 8 August, 2023; originally announced August 2023.

Comments: Minor updates from v2. 50 pages, 14 figures, 5 tables

SCOL: Supervised Contrastive Ordinal Loss for Abdominal Aortic Calcification Scoring on Vertebral Fracture Assessment Scans

Authors: Afsah Saleem, Zaid Ilyas, David Suter, Ghulam Mubashar Hassan, Siobhan Reid, John T. Schousboe, Richard Prince, William D. Leslie, Joshua R. Lewis, Syed Zulqarnain Gilani

Abstract: Abdominal Aortic Calcification (AAC) is a known marker of asymptomatic Atherosclerotic Cardiovascular Diseases (ASCVDs). AAC can be observed on Vertebral Fracture Assessment (VFA) scans acquired using Dual-Energy X-ray Absorptiometry (DXA) machines. Thus, the automatic quantification of AAC on VFA DXA scans may be used to screen for CVD risks, allowing early interventions. In this research, we for… ▽ More Abdominal Aortic Calcification (AAC) is a known marker of asymptomatic Atherosclerotic Cardiovascular Diseases (ASCVDs). AAC can be observed on Vertebral Fracture Assessment (VFA) scans acquired using Dual-Energy X-ray Absorptiometry (DXA) machines. Thus, the automatic quantification of AAC on VFA DXA scans may be used to screen for CVD risks, allowing early interventions. In this research, we formulate the quantification of AAC as an ordinal regression problem. We propose a novel Supervised Contrastive Ordinal Loss (SCOL) by incorporating a label-dependent distance metric with existing supervised contrastive loss to leverage the ordinal information inherent in discrete AAC regression labels. We develop a Dual-encoder Contrastive Ordinal Learning (DCOL) framework that learns the contrastive ordinal representation at global and local levels to improve the feature separability and class diversity in latent space among the AAC-24 genera. We evaluate the performance of the proposed framework using two clinical VFA DXA scan datasets and compare our work with state-of-the-art methods. Furthermore, for predicted AAC scores, we provide a clinical analysis to predict the future risk of a Major Acute Cardiovascular Event (MACE). Our results demonstrate that this learning enhances inter-class separability and strengthens intra-class consistency, which results in predicting the high-risk AAC classes with high sensitivity and high accuracy. △ Less

Submitted 22 July, 2023; originally announced July 2023.

Comments: Accepted in conference MICCAI 2023

Exact Diffusion Inversion via Bi-directional Integration Approximation

Authors: Guoqiang Zhang, J. P. Lewis, W. Bastiaan Kleijn

Abstract: Recently, various methods have been proposed to address the inconsistency issue of DDIM inversion to enable image editing, such as EDICT [36] and Null-text inversion [22]. However, the above methods introduce considerable computational overhead. In this paper, we propose a new technique, named \emph{bi-directional integration approximation} (BDIA), to perform exact diffusion inversion with neglibl… ▽ More Recently, various methods have been proposed to address the inconsistency issue of DDIM inversion to enable image editing, such as EDICT [36] and Null-text inversion [22]. However, the above methods introduce considerable computational overhead. In this paper, we propose a new technique, named \emph{bi-directional integration approximation} (BDIA), to perform exact diffusion inversion with neglible computational overhead. Suppose we would like to estimate the next diffusion state $\boldsymbol{z}_{i-1}$ at timestep $t_i$ with the historical information $(i,\boldsymbol{z}_i)$ and $(i+1,\boldsymbol{z}_{i+1})$. We first obtain the estimated Gaussian noise $\hat{\boldsymbolε}(\boldsymbol{z}_i,i)$, and then apply the DDIM update procedure twice for approximating the ODE integration over the next time-slot $[t_i, t_{i-1}]$ in the forward manner and the previous time-slot $[t_i, t_{t+1}]$ in the backward manner. The DDIM step for the previous time-slot is used to refine the integration approximation made earlier when computing $\boldsymbol{z}_i$. A nice property of BDIA-DDIM is that the update expression for $\boldsymbol{z}_{i-1}$ is a linear combination of $(\boldsymbol{z}_{i+1}, \boldsymbol{z}_i, \hat{\boldsymbolε}(\boldsymbol{z}_i,i))$. This allows for exact backward computation of $\boldsymbol{z}_{i+1}$ given $(\boldsymbol{z}_i, \boldsymbol{z}_{i-1})$, thus leading to exact diffusion inversion. It is demonstrated with experiments that (round-trip) BDIA-DDIM is particularly effective for image editing. Our experiments further show that BDIA-DDIM produces markedly better image sampling qualities than DDIM for text-to-image generation. BDIA can also be applied to improve the performance of other ODE solvers in addition to DDIM. In our work, it is found that applying BDIA to the EDM sampling procedure produces consistently better performance over four pre-trained models. △ Less

Submitted 26 November, 2023; v1 submitted 10 July, 2023; originally announced July 2023.

Comments: arXiv admin note: text overlap with arXiv:2304.11328. Our code is available at https://github.com/guoqiang-zhang-x/BDIA

Training-Free Neural Matte Extraction for Visual Effects

Authors: Sharif Elcott, J. P. Lewis, Nori Kanazawa, Christoph Bregler

Abstract: Alpha matting is widely used in video conferencing as well as in movies, television, and social media sites. Deep learning approaches to the matte extraction problem are well suited to video conferencing due to the consistent subject matter (front-facing humans), however training-based approaches are somewhat pointless for entertainment videos where varied subjects (spaceships, monsters, etc.) may… ▽ More Alpha matting is widely used in video conferencing as well as in movies, television, and social media sites. Deep learning approaches to the matte extraction problem are well suited to video conferencing due to the consistent subject matter (front-facing humans), however training-based approaches are somewhat pointless for entertainment videos where varied subjects (spaceships, monsters, etc.) may appear only a few times in a single movie -- if a method of creating ground truth for training exists, just use that method to produce the desired mattes. We introduce a training-free high quality neural matte extraction approach that specifically targets the assumptions of visual effects production. Our approach is based on the deep image prior, which optimizes a deep neural network to fit a single image, thereby providing a deep encoding of the particular image. We make use of the representations in the penultimate layer to interpolate coarse and incomplete "trimap" constraints. Videos processed with this approach are temporally consistent. The algorithm is both very simple and surprisingly effective. △ Less

Submitted 29 June, 2023; originally announced June 2023.

ACM Class: I.4.6

Journal ref: SIGGRAPH Asia 2022 Technical Communications

Directed Diffusion: Direct Control of Object Placement through Attention Guidance

Authors: Wan-Duo Kurt Ma, J. P. Lewis, Avisek Lahiri, Thomas Leung, W. Bastiaan Kleijn

Abstract: Text-guided diffusion models such as DALLE-2, Imagen, eDiff-I, and Stable Diffusion are able to generate an effectively endless variety of images given only a short text prompt describing the desired image content. In many cases the images are of very high quality. However, these models often struggle to compose scenes containing several key objects such as characters in specified positional relat… ▽ More Text-guided diffusion models such as DALLE-2, Imagen, eDiff-I, and Stable Diffusion are able to generate an effectively endless variety of images given only a short text prompt describing the desired image content. In many cases the images are of very high quality. However, these models often struggle to compose scenes containing several key objects such as characters in specified positional relationships. The missing capability to ``direct'' the placement of characters and objects both within and across images is crucial in storytelling, as recognized in the literature on film and animation theory. In this work, we take a particularly straightforward approach to providing the needed direction. Drawing on the observation that the cross-attention maps for prompt words reflect the spatial layout of objects denoted by those words, we introduce an optimization objective that produces ``activation'' at desired positions in these cross-attention maps. The resulting approach is a step toward generalizing the applicability of text-guided diffusion models beyond single images to collections of related images, as in storybooks. Directed Diffusion provides easy high-level positional control over multiple objects, while making use of an existing pre-trained model and maintaining a coherent blend between the positioned objects and the background. Moreover, it requires only a few lines to implement. △ Less

Submitted 26 September, 2023; v1 submitted 25 February, 2023; originally announced February 2023.

Comments: Our project page: https://hohonu-vicml.github.io/DirectedDiffusion.Page

CONSISTENT: Open-Ended Question Generation From News Articles

Authors: Tuhin Chakrabarty, Justin Lewis, Smaranda Muresan

Abstract: Recent work on question generation has largely focused on factoid questions such as who, what, where, when about basic facts. Generating open-ended why, how, what, etc. questions that require long-form answers have proven more difficult. To facilitate the generation of open-ended questions, we propose CONSISTENT, a new end-to-end system for generating open-ended questions that are answerable from… ▽ More Recent work on question generation has largely focused on factoid questions such as who, what, where, when about basic facts. Generating open-ended why, how, what, etc. questions that require long-form answers have proven more difficult. To facilitate the generation of open-ended questions, we propose CONSISTENT, a new end-to-end system for generating open-ended questions that are answerable from and faithful to the input text. Using news articles as a trustworthy foundation for experimentation, we demonstrate our model's strength over several baselines using both automatic and human=based evaluations. We contribute an evaluation dataset of expert-generated open-ended questions.We discuss potential downstream applications for news media organizations. △ Less

Submitted 20 October, 2022; originally announced October 2022.

Comments: EMNLP 2022 Findings

Multi-level Adversarial Spatio-temporal Learning for Footstep Pressure based FoG Detection

Authors: Kun Hu, Shaohui Mei, Wei Wang, Kaylena A. Ehgoetz Martens, Liang Wang, Simon J. G. Lewis, David D. Feng, Zhiyong Wang

Abstract: Freezing of gait (FoG) is one of the most common symptoms of Parkinson's disease, which is a neurodegenerative disorder of the central nervous system impacting millions of people around the world. To address the pressing need to improve the quality of treatment for FoG, devising a computer-aided detection and quantification tool for FoG has been increasingly important. As a non-invasive technique… ▽ More Freezing of gait (FoG) is one of the most common symptoms of Parkinson's disease, which is a neurodegenerative disorder of the central nervous system impacting millions of people around the world. To address the pressing need to improve the quality of treatment for FoG, devising a computer-aided detection and quantification tool for FoG has been increasingly important. As a non-invasive technique for collecting motion patterns, the footstep pressure sequences obtained from pressure sensitive gait mats provide a great opportunity for evaluating FoG in the clinic and potentially in the home environment. In this study, FoG detection is formulated as a sequential modelling task and a novel deep learning architecture, namely Adversarial Spatio-temporal Network (ASTN), is proposed to learn FoG patterns across multiple levels. A novel adversarial training scheme is introduced with a multi-level subject discriminator to obtain subject-independent FoG representations, which helps to reduce the over-fitting risk due to the high inter-subject variance. As a result, robust FoG detection can be achieved for unseen subjects. The proposed scheme also sheds light on improving subject-level clinical studies from other scenarios as it can be integrated with many existing deep architectures. To the best of our knowledge, this is one of the first studies of footstep pressure-based FoG detection and the approach of utilizing ASTN is the first deep neural network architecture in pursuit of subject-independent representations. Experimental results on 393 trials collected from 21 subjects demonstrate encouraging performance of the proposed ASTN for FoG detection with an AUC 0.85. △ Less

Submitted 22 September, 2022; originally announced September 2022.

NewsStories: Illustrating articles with visual summaries

Authors: Reuben Tan, Bryan A. Plummer, Kate Saenko, JP Lewis, Avneesh Sud, Thomas Leung

Abstract: Recent self-supervised approaches have used large-scale image-text datasets to learn powerful representations that transfer to many tasks without finetuning. These methods often assume that there is one-to-one correspondence between its images and their (short) captions. However, many tasks require reasoning about multiple images and long text narratives, such as describing news articles with visu… ▽ More Recent self-supervised approaches have used large-scale image-text datasets to learn powerful representations that transfer to many tasks without finetuning. These methods often assume that there is one-to-one correspondence between its images and their (short) captions. However, many tasks require reasoning about multiple images and long text narratives, such as describing news articles with visual summaries. Thus, we explore a novel setting where the goal is to learn a self-supervised visual-language representation that is robust to varying text length and the number of images. In addition, unlike prior work which assumed captions have a literal relation to the image, we assume images only contain loose illustrative correspondence with the text. To explore this problem, we introduce a large-scale multimodal dataset containing over 31M articles, 22M images and 1M videos. We show that state-of-the-art image-text alignment methods are not robust to longer narratives with multiple images. Finally, we introduce an intuitive baseline that outperforms these methods on zero-shot image-set retrieval by 10% on the GoodNews dataset. △ Less

Submitted 14 August, 2022; v1 submitted 26 July, 2022; originally announced July 2022.

Comments: Accepted at ECCV 2022

LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalization

Authors: Avisek Lahiri, Vivek Kwatra, Christian Frueh, John Lewis, Chris Bregler

Abstract: In this paper, we present a video-based learning framework for animating personalized 3D talking faces from audio. We introduce two training-time data normalizations that significantly improve data sample efficiency. First, we isolate and represent faces in a normalized space that decouples 3D geometry, head pose, and texture. This decomposes the prediction problem into regressions over the 3D fac… ▽ More In this paper, we present a video-based learning framework for animating personalized 3D talking faces from audio. We introduce two training-time data normalizations that significantly improve data sample efficiency. First, we isolate and represent faces in a normalized space that decouples 3D geometry, head pose, and texture. This decomposes the prediction problem into regressions over the 3D face shape and the corresponding 2D texture atlas. Second, we leverage facial symmetry and approximate albedo constancy of skin to isolate and remove spatio-temporal lighting variations. Together, these normalizations allow simple networks to generate high fidelity lip-sync videos under novel ambient illumination while training with just a single speaker-specific video. Further, to stabilize temporal dynamics, we introduce an auto-regressive approach that conditions the model on its previous visual state. Human ratings and objective metrics demonstrate that our method outperforms contemporary state-of-the-art audio-driven video reenactment benchmarks in terms of realism, lip-sync and visual quality scores. We illustrate several applications enabled by our framework. △ Less

Submitted 8 June, 2021; originally announced June 2021.

Comments: Accepted to IEEE CVPR 2021. Brief demo video available at: https://www.youtube.com/watch?v=L1StbX9OznY

Virtual Surfaces and Attitude Aware Planning and Behaviours for Negative Obstacle Navigation

Authors: Thomas Hines, Kazys Stepanas, Fletcher Talbot, Inkyu Sa, Jake Lewis, Emili Hernandez, Navinda Kottege, Nicolas Hudson

Abstract: This paper presents an autonomous navigation system for ground robots traversing aggressive unstructured terrain through a cohesive arrangement of mapping, deliberative planning and reactive behaviour modules. All systems are aware of terrain slope, visibility and vehicle orientation, enabling robots to recognize, plan and react around unobserved areas and overcome negative obstacles, slopes, step… ▽ More This paper presents an autonomous navigation system for ground robots traversing aggressive unstructured terrain through a cohesive arrangement of mapping, deliberative planning and reactive behaviour modules. All systems are aware of terrain slope, visibility and vehicle orientation, enabling robots to recognize, plan and react around unobserved areas and overcome negative obstacles, slopes, steps, overhangs and narrow passageways. This is one of pioneer works to explicitly and simultaneously couple mapping, planning and reactive components in dealing with negative obstacles. The system was deployed on three heterogeneous ground robots for the DARPA Subterranean Challenge, and we present results in Urban and Cave environments, along with simulated scenarios, that demonstrate this approach. △ Less

Submitted 21 January, 2021; v1 submitted 29 October, 2020; originally announced October 2020.

Comments: 8 pages, 11 figures, submitted to RA-L

The tree search game for two players

Authors: Ravi B. Boppana, Joel Brewster Lewis

Abstract: We consider a two-player search game on a tree $T$. One vertex (unknown to the players) is randomly selected as the target. The players alternately guess vertices. If a guess $v$ is not the target, then both players are informed in which subtree of $T \smallsetminus v$ the target lies. The winner is the player who guesses the target. When both players play optimally, we show that each of them wi… ▽ More We consider a two-player search game on a tree $T$. One vertex (unknown to the players) is randomly selected as the target. The players alternately guess vertices. If a guess $v$ is not the target, then both players are informed in which subtree of $T \smallsetminus v$ the target lies. The winner is the player who guesses the target. When both players play optimally, we show that each of them wins with probability approximately $1/2$. When one player plays optimally and the other plays randomly, we show that the player with the optimal strategy wins with probability between $9/16$ and $2/3$ (asymptotically). When both players play randomly, we show that each wins with probability between $13/30$ and $17/30$ (asymptotically). △ Less

Submitted 4 February, 2022; v1 submitted 26 August, 2020; originally announced August 2020.

Comments: 24 pages. Essentially the same as published version (http://ajc.maths.uq.edu.au/pdf/82/ajc_v82_p119.pdf)

Journal ref: Australasian J. of Combinatorics. Volume 82(2) (2022), pages 119-145

NASA: Neural Articulated Shape Approximation

Authors: Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, Andrea Tagliasacchi

Abstract: Efficient representation of articulated objects such as human bodies is an important problem in computer vision and graphics. To efficiently simulate deformation, existing approaches represent 3D objects using polygonal meshes and deform them using skinning techniques. This paper introduces neural articulated shape approximation (NASA), an alternative framework that enables efficient representatio… ▽ More Efficient representation of articulated objects such as human bodies is an important problem in computer vision and graphics. To efficiently simulate deformation, existing approaches represent 3D objects using polygonal meshes and deform them using skinning techniques. This paper introduces neural articulated shape approximation (NASA), an alternative framework that enables efficient representation of articulated deformable objects using neural indicator functions that are conditioned on pose. Occupancy testing using NASA is straightforward, circumventing the complexity of meshes and the issue of water-tightness. We demonstrate the effectiveness of NASA for 3D tracking applications, and discuss other potential extensions. △ Less

Submitted 21 July, 2022; v1 submitted 6 December, 2019; originally announced December 2019.

Comments: ECCV 2020; Project Page: https://nasa-eccv20.github.io/

Predicting intelligence based on cortical WM/GM contrast, cortical thickness and volumetry

Authors: Juan Miguel Valverde, Vandad Imani, John D. Lewis, Jussi Tohka

Abstract: We propose a four-layer fully-connected neural network (FNN) for predicting fluid intelligence scores from T1-weighted MR images for the ABCD-challenge. In addition to the volumes of brain structures, the FNN uses cortical WM/GM contrast and cortical thickness at 78 cortical regions. These last two measurements were derived from the T1-weighted MR images using cortical surfaces produced by the CIV… ▽ More We propose a four-layer fully-connected neural network (FNN) for predicting fluid intelligence scores from T1-weighted MR images for the ABCD-challenge. In addition to the volumes of brain structures, the FNN uses cortical WM/GM contrast and cortical thickness at 78 cortical regions. These last two measurements were derived from the T1-weighted MR images using cortical surfaces produced by the CIVET pipeline. The age and gender of the subjects and the scanner manufacturer are also used as features for the learning algorithm. This yielded 283 features provided to the FNN with two hidden layers of 20 and 15 nodes. The method was applied to the data from the ABCD study. Trained with a training set of 3736 subjects, the proposed method achieved a MSE of 71.596 and a correlation of 0.151 in the validation set of 415 subjects. For the final submission, the model was trained with 3568 subjects and it achieved a MSE of 94.0270 in the test set comprised of 4383 subjects. △ Less

Submitted 9 September, 2019; originally announced September 2019.

Comments: Submission to the ABCD Neurocognitive Prediction Challenge at MICCAI 2019

Deep Learning-based Radiomic Features for Improving Neoadjuvant Chemoradiation Response Prediction in Locally Advanced Rectal Cancer

Authors: Jie Fu, Xinran Zhong, Ning Li, Ritchell Van Dams, John Lewis, Kyunghyun Sung, Ann C. Raldow, Jing Jin, X. Sharon Qi

Abstract: Radiomic features achieve promising results in cancer diagnosis, treatment response prediction, and survival prediction. Our goal is to compare the handcrafted (explicitly designed) and deep learning (DL)-based radiomic features extracted from pre-treatment diffusion-weighted magnetic resonance images (DWIs) for predicting neoadjuvant chemoradiation treatment (nCRT) response in patients with local… ▽ More Radiomic features achieve promising results in cancer diagnosis, treatment response prediction, and survival prediction. Our goal is to compare the handcrafted (explicitly designed) and deep learning (DL)-based radiomic features extracted from pre-treatment diffusion-weighted magnetic resonance images (DWIs) for predicting neoadjuvant chemoradiation treatment (nCRT) response in patients with locally advanced rectal cancer (LARC). 43 patients receiving nCRT were included. All patients underwent DWIs before nCRT and total mesorectal excision surgery 6-12 weeks after completion of nCRT. Gross tumor volume (GTV) contours were drawn by an experienced radiation oncologist on DWIs. The patient-cohort was split into the responder group (n=22) and the non-responder group (n=21) based on the post-nCRT response assessed by postoperative pathology, MRI or colonoscopy. Handcrafted and DL-based features were extracted from the apparent diffusion coefficient (ADC) map of the DWI using conventional computer-aided diagnosis methods and a pre-trained convolution neural network, respectively. Least absolute shrinkage and selection operator (LASSO)-logistic regression models were constructed using extracted features for predicting treatment response. The model performance was evaluated with repeated 20 times stratified 4-fold cross-validation using receiver operating characteristic (ROC) curves and compared using the corrected resampled t-test. The model built with handcrafted features achieved the mean area under the ROC curve (AUC) of 0.64, while the one built with DL-based features yielded the mean AUC of 0.73. The corrected resampled t-test on AUC showed P-value < 0.05. DL-based features extracted from pre-treatment DWIs achieved significantly better classification performance compared with handcrafted features for predicting nCRT response in patients with LARC. △ Less

Submitted 9 September, 2019; originally announced September 2019.

Comments: Review in progress

Journal ref: 2020 Phys. Med. Biol

The HSIC Bottleneck: Deep Learning without Back-Propagation

Authors: Wan-Duo Kurt Ma, J. P. Lewis, W. Bastiaan Kleijn

Abstract: We introduce the HSIC (Hilbert-Schmidt independence criterion) bottleneck for training deep neural networks. The HSIC bottleneck is an alternative to the conventional cross-entropy loss and backpropagation that has a number of distinct advantages. It mitigates exploding and vanishing gradients, resulting in the ability to learn very deep networks without skip connections. There is no requirement f… ▽ More We introduce the HSIC (Hilbert-Schmidt independence criterion) bottleneck for training deep neural networks. The HSIC bottleneck is an alternative to the conventional cross-entropy loss and backpropagation that has a number of distinct advantages. It mitigates exploding and vanishing gradients, resulting in the ability to learn very deep networks without skip connections. There is no requirement for symmetric feedback or update locking. We find that the HSIC bottleneck provides performance on MNIST/FashionMNIST/CIFAR10 classification comparable to backpropagation with a cross-entropy target, even when the system is not encouraged to make the output resemble the classification labels. Appending a single layer trained with SGD (without backpropagation) to reformat the information further improves performance. △ Less

Submitted 5 December, 2019; v1 submitted 5 August, 2019; originally announced August 2019.

VIPER: Volume Invariant Position-based Elastic Rods

Authors: Baptiste Angles, Daniel Rebain, Miles Macklin, Brian Wyvill, Loic Barthe, JP Lewis, Javier von der Pahlen, Shahram Izadi, Julien Valentin, Sofien Bouaziz, Andrea Tagliasacchi

Abstract: We extend the formulation of position-based rods to include elastic volumetric deformations. We achieve this by introducing an additional degree of freedom per vertex -- isotropic scale (and its velocity). Including scale enriches the space of possible deformations, allowing the simulation of volumetric effects, such as a reduction in cross-sectional area when a rod is stretched. We rigorously der… ▽ More We extend the formulation of position-based rods to include elastic volumetric deformations. We achieve this by introducing an additional degree of freedom per vertex -- isotropic scale (and its velocity). Including scale enriches the space of possible deformations, allowing the simulation of volumetric effects, such as a reduction in cross-sectional area when a rod is stretched. We rigorously derive the continuous formulation of its elastic energy potentials, and hence its associated position-based dynamics (PBD) updates to realize this model, enabling the simulation of up to 26000 DOFs at 140 Hz in our GPU implementation. We further show how rods can provide a compact alternative to tetrahedral meshes for the representation of complex muscle deformations, as well as providing a convenient representation for collision detection. This is achieved by modeling a muscle as a bundle of rods, for which we also introduce a technique to automatically convert a muscle surface mesh into a rods-bundle. Finally, we show how rods and/or bundles can be skinned to a surface mesh to drive its deformation, resulting in an alternative to cages for real-time volumetric deformation. △ Less

Submitted 12 June, 2019; originally announced June 2019.

Provable Certificates for Adversarial Examples: Fitting a Ball in the Union of Polytopes

Authors: Matt Jordan, Justin Lewis, Alexandros G. Dimakis

Abstract: We propose a novel method for computing exact pointwise robustness of deep neural networks for all convex $\ell_p$ norms. Our algorithm, GeoCert, finds the largest $\ell_p$ ball centered at an input point $x_0$, within which the output class of a given neural network with ReLU nonlinearities remains unchanged. We relate the problem of computing pointwise robustness of these networks to that of com… ▽ More We propose a novel method for computing exact pointwise robustness of deep neural networks for all convex $\ell_p$ norms. Our algorithm, GeoCert, finds the largest $\ell_p$ ball centered at an input point $x_0$, within which the output class of a given neural network with ReLU nonlinearities remains unchanged. We relate the problem of computing pointwise robustness of these networks to that of computing the maximum norm ball with a fixed center that can be contained in a non-convex polytope. This is a challenging problem in general, however we show that there exists an efficient algorithm to compute this for polyhedral complices. Further we show that piecewise linear neural networks partition the input space into a polyhedral complex. Our algorithm has the ability to almost immediately output a nontrivial lower bound to the pointwise robustness which is iteratively improved until it ultimately becomes tight. We empirically show that our approach generates distance lower bounds that are tighter compared to prior work, under moderate time constraints. △ Less

Submitted 3 June, 2019; v1 submitted 20 March, 2019; originally announced March 2019.

Comments: Code can be found here: https://github.com/revbucket/geometric-certificates

Immersive VR as a Tool to Enhance Relaxation for Undergraduate Students with the Aim of Reducing Anxiety - A Pilot Study

Authors: James Lewis, Benedikte Rorstad

Abstract: Despite extensive use in related domains, Virtual Reality (VR) for generalised anxiety disorder (GAD) has received little previous attention. We report upon a VR environment created for the Oculus Rift and Unreal Engine 4 (UE4) to investigate the potential of a VR simulation to be used as an anxiety management tool. We introduce the broad topic of GAD and related publications on the application of… ▽ More Despite extensive use in related domains, Virtual Reality (VR) for generalised anxiety disorder (GAD) has received little previous attention. We report upon a VR environment created for the Oculus Rift and Unreal Engine 4 (UE4) to investigate the potential of a VR simulation to be used as an anxiety management tool. We introduce the broad topic of GAD and related publications on the application of VR to this, and similar, mental health conditions. We then describe the development of a real time simulation tool, based upon the passive VR experience of a tranquil, rural alpine scene experienced from a seated position with head tracking. Evaluation focused upon qualitative feedback on the application. Testing was carried out over the period of two weeks on a sample group of eleven students studying at Nottingham Trent University. All participants were asked to complete the Depression, Anxiety and Stress Scale - 21 Items (DASS21) at the beginning and at the end of the study order to assess their profile, and hence suitability to comment upon the software. Qualitative feedback was very encouraging, with all participants reporting that they believed the experience helped and that they would consider utilising it if it was available. Additionally, a psychologist was asked to test the application to provide a specialist opinion on whether it would be appropriate for use as an anxiety management tool. The results highlight several areas for improvement but are positive overall in terms of its potential as a therapeutic tool. △ Less

Submitted 4 March, 2019; originally announced March 2019.

Comments: 8 pages, 8 figures

Multi-robot Dubins Coverage with Autonomous Surface Vehicles

Authors: Nare Karapetyan, Jason Moulton, Jeremy S. Lewis, Alberto Quattrini Li, Jason M. O'Kane, Ioannis Rekleitis

Abstract: In large scale coverage operations, such as marine exploration or aerial monitoring, single robot approaches are not ideal, as they may take too long to cover a large area. In such scenarios, multi-robot approaches are preferable. Furthermore, several real world vehicles are non-holonomic, but can be modeled using Dubins vehicle kinematics. This paper focuses on environmental monitoring of aquatic… ▽ More In large scale coverage operations, such as marine exploration or aerial monitoring, single robot approaches are not ideal, as they may take too long to cover a large area. In such scenarios, multi-robot approaches are preferable. Furthermore, several real world vehicles are non-holonomic, but can be modeled using Dubins vehicle kinematics. This paper focuses on environmental monitoring of aquatic environments using Autonomous Surface Vehicles (ASVs). In particular, we propose a novel approach for solving the problem of complete coverage of a known environment by a multi-robot team consisting of Dubins vehicles. It is worth noting that both multi-robot coverage and Dubins vehicle coverage are NP-complete problems. As such, we present two heuristics methods based on a variant of the traveling salesman problem -- k-TSP -- formulation and clustering algorithms that efficiently solve the problem. The proposed methods are tested both in simulations to assess their scalability and with a team of ASVs operating on a lake to ensure their applicability in real world. △ Less

Submitted 7 August, 2018; originally announced August 2018.

Comments: In proceedings of IEEE International Conference on Robotics and Automation (ICRA), 2018

Rook theory of the finite general linear group

Authors: Joel Brewster Lewis, Alejandro H. Morales

Abstract: Matrices over a finite field having fixed rank and restricted support are a natural $q$-analogue of rook placements on a board. We develop this $q$-rook theory by defining a corresponding analogue of the hit numbers. Using tools from coding theory, we show that these $q$-hit and $q$-rook numbers obey a variety of identities analogous to the classical case. We also explore connections to earlier… ▽ More Matrices over a finite field having fixed rank and restricted support are a natural $q$-analogue of rook placements on a board. We develop this $q$-rook theory by defining a corresponding analogue of the hit numbers. Using tools from coding theory, we show that these $q$-hit and $q$-rook numbers obey a variety of identities analogous to the classical case. We also explore connections to earlier $q$-analogues of rook theory, as well as settling a polynomiality conjecture and finding a counterexample of a positivity conjecture of the authors and Klein. △ Less

Submitted 5 September, 2017; v1 submitted 25 July, 2017; originally announced July 2017.

Comments: 25 pages, 10 figure files. Minor change in definition of q-hit numbers changes notation but doesn't substantively affect results

Journal ref: Experimental Mathematics 29:3 (2020), 328-346

Negentropic Planar Symmetry Detector

Authors: Agata Migalska, JP Lewis

Abstract: In this paper we observe that information theoretical concepts are valuable tools for extracting information from images and, in particular, information on image symmetries. It is shown that the problem of detecting reflectional and rotational symmetries in a two-dimensional image can be reduced to the problem of detecting point-symmetry and periodicity in one-dimensional negentropy functions. Bas… ▽ More In this paper we observe that information theoretical concepts are valuable tools for extracting information from images and, in particular, information on image symmetries. It is shown that the problem of detecting reflectional and rotational symmetries in a two-dimensional image can be reduced to the problem of detecting point-symmetry and periodicity in one-dimensional negentropy functions. Based on these findings a detector of reflectional and rotational global symmetries in greyscale images is constructed. We discuss the importance of high precision in symmetry detection in applications arising from quality control and illustrate how the proposed method satisfies this requirement. Finally, a superior performance of our method to other existing methods, demonstrated by the results of a rigorous experimental verification, is an indication that our approach rooted in information theory is a promising direction in a development of a robust and widely applicable symmetry detector. △ Less

Submitted 11 March, 2017; originally announced March 2017.

Comments: 25 pages, 18 figures

The Shattered Gradients Problem: If resnets are the answer, then what is the question?

Authors: David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, Brian McWilliams

Abstract: A long-standing obstacle to progress in deep learning is the problem of vanishing and exploding gradients. Although, the problem has largely been overcome via carefully constructed initializations and batch normalization, architectures incorporating skip-connections such as highway and resnets perform much better than standard feedforward architectures despite well-chosen initialization and batch… ▽ More A long-standing obstacle to progress in deep learning is the problem of vanishing and exploding gradients. Although, the problem has largely been overcome via carefully constructed initializations and batch normalization, architectures incorporating skip-connections such as highway and resnets perform much better than standard feedforward architectures despite well-chosen initialization and batch normalization. In this paper, we identify the shattered gradients problem. Specifically, we show that the correlation between gradients in standard feedforward networks decays exponentially with depth resulting in gradients that resemble white noise whereas, in contrast, the gradients in architectures with skip-connections are far more resistant to shattering, decaying sublinearly. Detailed empirical evidence is presented in support of the analysis, on both fully-connected networks and convnets. Finally, we present a new "looks linear" (LL) initialization that prevents shattering, with preliminary experiments showing the new initialization allows to train very deep networks without the addition of skip-connections. △ Less

Submitted 6 June, 2018; v1 submitted 27 February, 2017; originally announced February 2017.

Comments: ICML 2017, final version

Journal ref: PMLR volume 70 (2017)

A Base Camp for Scaling AI

Authors: C. J. C. Burges, T. Hart, Z. Yang, S. Cucerzan, R. W. White, A. Pastusiak, J. Lewis

Abstract: Modern statistical machine learning (SML) methods share a major limitation with the early approaches to AI: there is no scalable way to adapt them to new domains. Human learning solves this in part by leveraging a rich, shared, updateable world model. Such scalability requires modularity: updating part of the world model should not impact unrelated parts. We have argued that such modularity will r… ▽ More Modern statistical machine learning (SML) methods share a major limitation with the early approaches to AI: there is no scalable way to adapt them to new domains. Human learning solves this in part by leveraging a rich, shared, updateable world model. Such scalability requires modularity: updating part of the world model should not impact unrelated parts. We have argued that such modularity will require both "correctability" (so that errors can be corrected without introducing new errors) and "interpretability" (so that we can understand what components need correcting). To achieve this, one could attempt to adapt state of the art SML systems to be interpretable and correctable; or one could see how far the simplest possible interpretable, correctable learning methods can take us, and try to control the limitations of SML methods by applying them only where needed. Here we focus on the latter approach and we investigate two main ideas: "Teacher Assisted Learning", which leverages crowd sourcing to learn language; and "Factored Dialog Learning", which factors the process of application development into roles where the language competencies needed are isolated, enabling non-experts to quickly create new applications. We test these ideas in an "Automated Personal Assistant" (APA) setting, with two scenarios: that of detecting user intent from a user-APA dialog; and that of creating a class of event reminder applications, where a non-expert "teacher" can then create specific apps. For the intent detection task, we use a dataset of a thousand labeled utterances from user dialogs with Cortana, and we show that our approach matches state of the art SML methods, but in addition provides full transparency: the whole (editable) model can be summarized on one human-readable page. For the reminder app task, we ran small user studies to verify the efficacy of the approach. △ Less

Submitted 23 December, 2016; originally announced December 2016.

Hierarchical Evidence Accumulation in the Pseiki System and Experiments in Model-Driven Mobile Robot Navigation

Authors: A. C. Kak, K. M. Andress, C. Lopez-Abadia, M. S. Carroll, J. R. Lewis

Abstract: In this paper, we will review the process of evidence accumulation in the PSEIKI system for expectation-driven interpretation of images of 3-D scenes. Expectations are presented to PSEIKI as a geometrical hierarchy of abstractions. PSEIKI's job is then to construct abstraction hierarchies in the perceived image taking cues from the abstraction hierarchies in the expectations. The Dempster-Shafe… ▽ More In this paper, we will review the process of evidence accumulation in the PSEIKI system for expectation-driven interpretation of images of 3-D scenes. Expectations are presented to PSEIKI as a geometrical hierarchy of abstractions. PSEIKI's job is then to construct abstraction hierarchies in the perceived image taking cues from the abstraction hierarchies in the expectations. The Dempster-Shafer formalism is used for associating belief values with the different possible labels for the constructed abstractions in the perceived image. This system has been used successfully for autonomous navigation of a mobile robot in indoor environments. △ Less

Submitted 27 March, 2013; originally announced April 2013.

Comments: Appears in Proceedings of the Fifth Conference on Uncertainty in Artificial Intelligence (UAI1989)

Report number: UAI-P-1989-PG-194-207

Phishing, Personality Traits and Facebook

Authors: Tzipora Halevi, Jim Lewis, Nasir Memon

Abstract: Phishing attacks have become an increasing threat to online users. Recent research has begun to focus on the factors that cause people to respond to them. Our study examines the correlation between the Big Five personality traits and email phishing response. We also examine how these factors affect users behavior on Facebook, including posting personal information and choosing Facebook privacy set… ▽ More Phishing attacks have become an increasing threat to online users. Recent research has begun to focus on the factors that cause people to respond to them. Our study examines the correlation between the Big Five personality traits and email phishing response. We also examine how these factors affect users behavior on Facebook, including posting personal information and choosing Facebook privacy settings. Our research shows that when using a prize phishing email, we find a strong correlation between gender and the response to the phishing email. In addition, we find that the neuroticism is the factor most correlated to responding to this email. Our study also found that people who score high on the openness factor tend to both post more information on Facebook as well as have less strict privacy settings, which may cause them to be susceptible to privacy attacks. In addition, our work detected no correlation between the participants estimate of being vulnerable to phishing attacks and actually being phished, which suggests susceptibility to phishing is not due to lack of awareness of the phishing risks and that realtime response to phishing is hard to predict in advance by online users. We believe that better understanding of the traits which contribute to online vulnerability can help develop methods for increasing users privacy and security in the future. △ Less

Submitted 7 February, 2013; v1 submitted 31 January, 2013; originally announced January 2013.