Showing 1–31 of 31 results for author: Nowak, A

Sparser, Better, Deeper, Stronger: Improving Sparse Training with Exact Orthogonal Initialization

Authors: Aleksandra Irena Nowak, Łukasz Gniecki, Filip Szatkowski, Jacek Tabor

Abstract: Static sparse training aims to train sparse models from scratch, achieving remarkable results in recent years. A key design choice is given by the sparse initialization, which determines the trainable sub-network through a binary mask. Existing methods mainly select such mask based on a predefined dense initialization. Such an approach may not efficiently leverage the mask's potential impact on th… ▽ More Static sparse training aims to train sparse models from scratch, achieving remarkable results in recent years. A key design choice is given by the sparse initialization, which determines the trainable sub-network through a binary mask. Existing methods mainly select such mask based on a predefined dense initialization. Such an approach may not efficiently leverage the mask's potential impact on the optimization. An alternative direction, inspired by research into dynamical isometry, is to introduce orthogonality in the sparse subnetwork, which helps in stabilizing the gradient signal. In this work, we propose Exact Orthogonal Initialization (EOI), a novel sparse orthogonal initialization scheme based on composing random Givens rotations. Contrary to other existing approaches, our method provides exact (not approximated) orthogonality and enables the creation of layers with arbitrary densities. We demonstrate the superior effectiveness and efficiency of EOI through experiments, consistently outperforming common sparse initialization techniques. Our method enables training highly sparse 1000-layer MLP and CNN networks without residual connections or normalization techniques, emphasizing the crucial role of weight initialization in static sparse training alongside sparse mask selection. The code is available at https://github.com/woocash2/sparser-better-deeper-stronger △ Less

Submitted 3 June, 2024; originally announced June 2024.

Comments: ICML 2024

On the Role of Summary Content Units in Text Summarization Evaluation

Authors: Marcel Nawrath, Agnieszka Nowak, Tristan Ratz, Danilo C. Walenta, Juri Opitz, Leonardo F. R. Ribeiro, João Sedoc, Daniel Deutsch, Simon Mille, Yixin Liu, Lining Zhang, Sebastian Gehrmann, Saad Mahamood, Miruna Clinciu, Khyathi Chandu, Yufang Hou

Abstract: At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs). These SCUs are concise sentences that decompose a summary into small facts. Such SCUs can be used to judge the quality of a candidate summary, possibly partially automated via natural language inference (NLI) systems. Interestingly, with the aim to fully automate the Pyramid evaluat… ▽ More At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs). These SCUs are concise sentences that decompose a summary into small facts. Such SCUs can be used to judge the quality of a candidate summary, possibly partially automated via natural language inference (NLI) systems. Interestingly, with the aim to fully automate the Pyramid evaluation, Zhang and Bansal (2021) show that SCUs can be approximated by automatically generated semantic role triplets (STUs). However, several questions currently lack answers, in particular: i) Are there other ways of approximating SCUs that can offer advantages? ii) Under which conditions are SCUs (or their approximations) offering the most value? In this work, we examine two novel strategies to approximate SCUs: generating SCU approximations from AMR meaning representations (SMUs) and from large language models (SGUs), respectively. We find that while STUs and SMUs are competitive, the best approximation quality is achieved by SGUs. We also show through a simple sentence-decomposition baseline (SSUs) that SCUs (and their approximations) offer the most value when ranking short summaries, but may not help as much when ranking systems or longer summaries. △ Less

Submitted 2 April, 2024; originally announced April 2024.

Comments: 10 Pages, 3 Figures, 3 Tables, camera ready version accepted at NAACL 2024

Conditional cooperation with longer memory

Authors: Nikoleta E. Glynatsi, Martin A. Nowak, Christian Hilbe

Abstract: Direct reciprocity is a wide-spread mechanism for evolution of cooperation. In repeated interactions, players can condition their behavior on previous outcomes. A well known approach is given by reactive strategies, which respond to the co-player's previous move. Here we extend reactive strategies to longer memories. A reactive-$n$ strategy takes into account the sequence of the last $n$ moves of… ▽ More Direct reciprocity is a wide-spread mechanism for evolution of cooperation. In repeated interactions, players can condition their behavior on previous outcomes. A well known approach is given by reactive strategies, which respond to the co-player's previous move. Here we extend reactive strategies to longer memories. A reactive-$n$ strategy takes into account the sequence of the last $n$ moves of the co-player. A reactive-$n$ counting strategy records how often the co-player has cooperated during the last $n$ rounds. We derive an algorithm to identify all partner strategies among reactive-$n$ strategies. We give explicit conditions for all partner strategies among reactive-2, reactive-3 strategies, and reactive-$n$ counting strategies. Partner strategies are those that ensure mutual cooperation without exploitation. We perform evolutionary simulations and find that longer memory increases the average cooperation rate for reactive-$n$ strategies but not for reactive counting strategies. Paying attention to the sequence of moves is necessary for reaping the advantages of longer memory. △ Less

Submitted 4 February, 2024; originally announced February 2024.

Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training

Authors: Aleksandra I. Nowak, Bram Grooten, Decebal Constantin Mocanu, Jacek Tabor

Abstract: Dynamic Sparse Training (DST) is a rapidly evolving area of research that seeks to optimize the sparse initialization of a neural network by adapting its topology during training. It has been shown that under specific conditions, DST is able to outperform dense models. The key components of this framework are the pruning and growing criteria, which are repeatedly applied during the training proces… ▽ More Dynamic Sparse Training (DST) is a rapidly evolving area of research that seeks to optimize the sparse initialization of a neural network by adapting its topology during training. It has been shown that under specific conditions, DST is able to outperform dense models. The key components of this framework are the pruning and growing criteria, which are repeatedly applied during the training process to adjust the network's sparse connectivity. While the growing criterion's impact on DST performance is relatively well studied, the influence of the pruning criterion remains overlooked. To address this issue, we design and perform an extensive empirical analysis of various pruning criteria to better understand their impact on the dynamics of DST solutions. Surprisingly, we find that most of the studied methods yield similar results. The differences become more significant in the low-density regime, where the best performance is predominantly given by the simplest technique: magnitude-based pruning. The code is provided at https://github.com/alooow/fantastic_weights_paper △ Less

Submitted 29 November, 2023; v1 submitted 21 June, 2023; originally announced June 2023.

Comments: NeurIPS 2023

In Sync: Exploring Synchronization to Increase Trust Between Humans and Non-humanoid Robots

Authors: Wieslaw Bartkowski, Andrzej Nowak, Filip Ignacy Czajkowski, Albrecht Schmidt, Florian Müller

Abstract: When we go for a walk with friends, we can observe an interesting effect: From step lengths to arm movements - our movements unconsciously align; they synchronize. Prior research found that this synchronization is a crucial aspect of human relations that strengthens social cohesion and trust. Generalizing from these findings in synchronization theory, we propose a dynamical approach that can be ap… ▽ More When we go for a walk with friends, we can observe an interesting effect: From step lengths to arm movements - our movements unconsciously align; they synchronize. Prior research found that this synchronization is a crucial aspect of human relations that strengthens social cohesion and trust. Generalizing from these findings in synchronization theory, we propose a dynamical approach that can be applied in the design of non-humanoid robots to increase trust. We contribute the results of a controlled experiment with 51 participants exploring our concept in a between-subjects design. For this, we built a prototype of a simple non-humanoid robot that can bend to follow human movements and vary the movement synchronization patterns. We found that synchronized movements lead to significantly higher ratings in an established questionnaire on trust between people and automation but did not influence the willingness to spend money in a trust game. △ Less

Submitted 6 April, 2023; v1 submitted 28 March, 2023; originally announced March 2023.

Comments: To appear in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI 23), April 23-28, 2023, Hamburg, Germany. ACM, New York, NY, USA, 14 pages

Journal ref: In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 367, 1-14

See or Hear? Exploring the Effect of Visual and Audio Hints and Gaze-assisted Task Feedback for Visual Search Tasks in Augmented Reality

Authors: Yuchong Zhang, Adam Nowak, Yueming Xuan, Andrzej Romanowski, Morten Fjeld

Abstract: Augmented reality (AR) is emerging in visual search tasks for increasingly immersive interactions with virtual objects. We propose an AR approach providing visual and audio hints along with gaze-assisted instant post-task feedback for search tasks based on mobile head-mounted display (HMD). The target case was a book-searching task, in which we aimed to explore the effect of the hints together wit… ▽ More Augmented reality (AR) is emerging in visual search tasks for increasingly immersive interactions with virtual objects. We propose an AR approach providing visual and audio hints along with gaze-assisted instant post-task feedback for search tasks based on mobile head-mounted display (HMD). The target case was a book-searching task, in which we aimed to explore the effect of the hints together with the task feedback with two hypotheses. H1: Since visual and audio hints can positively affect AR search tasks, the combination outperforms the individuals. H2: The gaze-assisted instant post-task feedback can positively affect AR search tasks. The proof-of-concept was demonstrated by an AR app in HMD and a comprehensive user study (n=96) consisting of two sub-studies, Study I (n=48) without task feedback and Study II (n=48) with task feedback. Following quantitative and qualitative analysis, our results partially verified H1 and completely verified H2, enabling us to conclude that the synthesis of visual and audio hints conditionally improves the AR visual search task efficiency when coupled with task feedback. △ Less

Submitted 14 November, 2023; v1 submitted 3 February, 2023; originally announced February 2023.

Comments: In Proceedings of 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)

Journal ref: 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (pp. 1113-1122). IEEE

Trust Your $\nabla$: Gradient-based Intervention Targeting for Causal Discovery

Authors: Mateusz Olko, Michał Zając, Aleksandra Nowak, Nino Scherrer, Yashas Annadani, Stefan Bauer, Łukasz Kuciński, Piotr Miłoś

Abstract: Inferring causal structure from data is a challenging task of fundamental importance in science. Observational data are often insufficient to identify a system's causal structure uniquely. While conducting interventions (i.e., experiments) can improve the identifiability, such samples are usually challenging and expensive to obtain. Hence, experimental design approaches for causal discovery aim to… ▽ More Inferring causal structure from data is a challenging task of fundamental importance in science. Observational data are often insufficient to identify a system's causal structure uniquely. While conducting interventions (i.e., experiments) can improve the identifiability, such samples are usually challenging and expensive to obtain. Hence, experimental design approaches for causal discovery aim to minimize the number of interventions by estimating the most informative intervention target. In this work, we propose a novel Gradient-based Intervention Targeting method, abbreviated GIT, that 'trusts' the gradient estimator of a gradient-based causal discovery framework to provide signals for the intervention acquisition function. We provide extensive experiments in simulated and real-world datasets and demonstrate that GIT performs on par with competitive baselines, surpassing them in the low-data regime. △ Less

Submitted 3 April, 2024; v1 submitted 24 November, 2022; originally announced November 2022.

Comments: Accepted to 37th Conference on Neural Information Processing Systems (NeurIPS 2023)

Trust based attachment

Authors: Julian Kates-Harbeck, Martin A. Nowak

Abstract: In social systems subject to indirect reciprocity, a positive reputation is key for increasing one's likelihood of future positive interactions. The flow of gossip can amplify the impact of a person's actions on their reputation depending on how widely it spreads across the social network, which leads to a percolation problem. To quantify this notion, we calculate the expected number of individual… ▽ More In social systems subject to indirect reciprocity, a positive reputation is key for increasing one's likelihood of future positive interactions. The flow of gossip can amplify the impact of a person's actions on their reputation depending on how widely it spreads across the social network, which leads to a percolation problem. To quantify this notion, we calculate the expected number of individuals, the "audience", who find out about a particular interaction. For a potential donor, a larger audience constitutes higher reputational stakes, and thus a higher incentive, to perform "good" actions in line with current social norms. For a receiver, a larger audience therefore increases the trust that the partner will be cooperative. This idea can be used for an algorithm that generates social networks, which we call trust based attachment (TBA). TBA produces graphs that share crucial quantitative properties with real-world networks, such as high clustering, small-world behavior, and power law degree distributions. We also show that TBA can be approximated by simple friend-of-friend routines based on triadic closure, which are known to be highly effective at generating realistic social network structures. Therefore, our work provides a new justification for triadic closure in social contexts based on notions of trust, gossip, and social information spread. These factors are thus identified as potential significant influences on how humans form social ties. △ Less

Submitted 29 August, 2023; v1 submitted 4 August, 2022; originally announced August 2022.

Structural measures of similarity and complementarity in complex networks

Authors: Szymon Talaga, Andrzej Nowak

Abstract: The principle of similarity, or homophily, is often used to explain patterns observed in complex networks such as transitivity and the abundance of triangles (3-cycles). However, many phenomena from division of labor to protein-protein interactions (PPI) are driven by complementarity (differences and synergy). Here we show that the principle of complementarity is linked to the abundance of quadran… ▽ More The principle of similarity, or homophily, is often used to explain patterns observed in complex networks such as transitivity and the abundance of triangles (3-cycles). However, many phenomena from division of labor to protein-protein interactions (PPI) are driven by complementarity (differences and synergy). Here we show that the principle of complementarity is linked to the abundance of quadrangles (4-cycles) and dense bipartite-like subgraphs. We link both principles to their characteristic motifs and introduce two families of coefficients of: (1) structural similarity, which generalize local clustering and closure coefficients and capture the full spectrum of similarity-driven structures; (2) structural complementarity, defined analogously but based on quadrangles instead of triangles. Using multiple social and biological networks, we demonstrate that the coefficients capture structural properties related to meaningful domain-specific phenomena. We show that they allow distinguishing between different kinds of social relations as well as measuring an increasing structural diversity of PPI networks across the tree of life. Our results indicate that some types of relations are better explained by complementarity than homophily, and may be useful for improving existing link prediction methods. We also introduce a Python package implementing efficient algorithms for calculating the proposed coefficients. △ Less

Submitted 10 October, 2022; v1 submitted 10 January, 2022; originally announced January 2022.

Comments: Authors' copy of the published version

Journal ref: Sci Rep 12, 16580 (2022)

Non-Gaussian Gaussian Processes for Few-Shot Regression

Authors: Marcin Sendera, Jacek Tabor, Aleksandra Nowak, Andrzej Bedychaj, Massimiliano Patacchiola, Tomasz Trzciński, Przemysław Spurek, Maciej Zięba

Abstract: Gaussian Processes (GPs) have been widely used in machine learning to model distributions over functions, with applications including multi-modal regression, time-series prediction, and few-shot learning. GPs are particularly useful in the last application since they rely on Normal distributions and enable closed-form computation of the posterior probability function. Unfortunately, because the re… ▽ More Gaussian Processes (GPs) have been widely used in machine learning to model distributions over functions, with applications including multi-modal regression, time-series prediction, and few-shot learning. GPs are particularly useful in the last application since they rely on Normal distributions and enable closed-form computation of the posterior probability function. Unfortunately, because the resulting posterior is not flexible enough to capture complex distributions, GPs assume high similarity between subsequent tasks - a requirement rarely met in real-world conditions. In this work, we address this limitation by leveraging the flexibility of Normalizing Flows to modulate the posterior predictive distribution of the GP. This makes the GP posterior locally non-Gaussian, therefore we name our method Non-Gaussian Gaussian Processes (NGGPs). More precisely, we propose an invertible ODE-based mapping that operates on each component of the random variable vectors and shares the parameters across all of them. We empirically tested the flexibility of NGGPs on various few-shot learning regression datasets, showing that the mapping can incorporate context embedding information to model different noise levels for periodic functions. As a result, our method shares the structure of the problem between subsequent tasks, but the contextualization allows for adaptation to dissimilarities. NGGPs outperform the competing state-of-the-art approaches on a diversified set of benchmarks and applications. △ Less

Submitted 26 October, 2021; originally announced October 2021.

On the relationship between disentanglement and multi-task learning

Authors: Łukasz Maziarka, Aleksandra Nowak, Maciej Wołczyk, Andrzej Bedychaj

Abstract: One of the main arguments behind studying disentangled representations is the assumption that they can be easily reused in different tasks. At the same time finding a joint, adaptable representation of data is one of the key challenges in the multi-task learning setting. In this paper, we take a closer look at the relationship between disentanglement and multi-task learning based on hard parameter… ▽ More One of the main arguments behind studying disentangled representations is the assumption that they can be easily reused in different tasks. At the same time finding a joint, adaptable representation of data is one of the key challenges in the multi-task learning setting. In this paper, we take a closer look at the relationship between disentanglement and multi-task learning based on hard parameter sharing. We perform a thorough empirical study of the representations obtained by neural networks trained on automatically generated supervised tasks. Using a set of standard metrics we show that disentanglement appears naturally during the process of multi-task neural network training. △ Less

Submitted 7 October, 2021; originally announced October 2021.

Neural networks adapting to datasets: learning network size and topology

Authors: Romuald A. Janik, Aleksandra Nowak

Abstract: We introduce a flexible setup allowing for a neural network to learn both its size and topology during the course of a standard gradient-based training. The resulting network has the structure of a graph tailored to the particular learning task and dataset. The obtained networks can also be trained from scratch and achieve virtually identical performance. We explore the properties of the network a… ▽ More We introduce a flexible setup allowing for a neural network to learn both its size and topology during the course of a standard gradient-based training. The resulting network has the structure of a graph tailored to the particular learning task and dataset. The obtained networks can also be trained from scratch and achieve virtually identical performance. We explore the properties of the network architectures for a number of datasets of varying difficulty observing systematic regularities. The obtained graphs can be therefore understood as encoding nontrivial characteristics of the particular classification tasks. △ Less

Submitted 15 July, 2020; v1 submitted 22 June, 2020; originally announced June 2020.

Comments: Fixed blank page

Analyzing Neural Networks Based on Random Graphs

Authors: Romuald A. Janik, Aleksandra Nowak

Abstract: We perform a massive evaluation of neural networks with architectures corresponding to random graphs of various types. We investigate various structural and numerical properties of the graphs in relation to neural network test accuracy. We find that none of the classical numerical graph invariants by itself allows to single out the best networks. Consequently, we introduce a new numerical graph ch… ▽ More We perform a massive evaluation of neural networks with architectures corresponding to random graphs of various types. We investigate various structural and numerical properties of the graphs in relation to neural network test accuracy. We find that none of the classical numerical graph invariants by itself allows to single out the best networks. Consequently, we introduce a new numerical graph characteristic that selects a set of quasi-1-dimensional graphs, which are a majority among the best performing networks. We also find that networks with primarily short-range connections perform better than networks which allow for many long-range connections. Moreover, many resolution reducing pathways are beneficial. We provide a dataset of 1020 graphs and the test accuracies of their corresponding neural networks at https://github.com/rmldj/random-graph-nn-paper △ Less

Submitted 2 December, 2020; v1 submitted 19 February, 2020; originally announced February 2020.

Comments: Added new results and discussion

WICA: nonlinear weighted ICA

Authors: Andrzej Bedychaj, Przemysław Spurek, Aleksandra Nowak, Jacek Tabor

Abstract: Independent Component Analysis (ICA) aims to find a coordinate system in which the components of the data are independent. In this paper we construct a new nonlinear ICA model, called WICA, which obtains better and more stable results than other algorithms. A crucial tool is given by a new efficient method of verifying nonlinear dependence with the use of computation of correlation coefficients fo… ▽ More Independent Component Analysis (ICA) aims to find a coordinate system in which the components of the data are independent. In this paper we construct a new nonlinear ICA model, called WICA, which obtains better and more stable results than other algorithms. A crucial tool is given by a new efficient method of verifying nonlinear dependence with the use of computation of correlation coefficients for normally weighted data. In addition, authors propose a new baseline nonlinear mixing to perform comparable experiments, and a~reliable measure which allows fair comparison of nonlinear models. Our code for WICA is available on Github https://github.com/gmum/wica. △ Less

Submitted 9 December, 2020; v1 submitted 13 January, 2020; originally announced January 2020.

Homophily as a Process Generating Social Networks: Insights from Social Distance Attachment Model

Authors: Szymon Talaga, Andrzej Nowak

Abstract: Real-world social networks often exhibit high levels of clustering, positive degree assortativity, short average path lengths (small-world property) and right-skewed but rarely power law degree distributions. On the other hand homophily, defined as the propensity of similar agents to connect to each other, is one of the most fundamental social processes observed in many human and animal societies.… ▽ More Real-world social networks often exhibit high levels of clustering, positive degree assortativity, short average path lengths (small-world property) and right-skewed but rarely power law degree distributions. On the other hand homophily, defined as the propensity of similar agents to connect to each other, is one of the most fundamental social processes observed in many human and animal societies. In this paper we examine the extent to which homophily is sufficient to produce the typical structural properties of social networks. To do so, we conduct a simulation study based on the Social Distance Attachment (SDA) model, a particular kind of Random Geometric Graph (RGG), in which nodes are embedded in a social space and connection probabilities depend functionally on distances between nodes. We derive the form of the model from first principles based on existing analytical results and argue that the mathematical construction of RGGs corresponds directly to the homophily principle, so they provide a good model for it. We find that homophily, especially when combined with a random edge rewiring, is sufficient to reproduce many of the characteristic features of social networks. Additionally, we devise a hybrid model combining SDA with the configuration model that allows generating homophilic networks with arbitrary degree sequences and we use it to study interactions of homophily with processes imposing constraints on degree distributions. We show that the effects of homophily on clustering are robust with respect to distribution constraints, while degree assortativity can be highly dependent on the particular kind of enforced degree sequence. △ Less

Submitted 2 April, 2020; v1 submitted 16 July, 2019; originally announced July 2019.

Comments: 22 pages; 11 figures. Published version. Introduction, theory and discussion differ somewhat from the earlier preprint versions

Journal ref: Journal of Artificial Societies and Social Simulation, 23(2) 6, 2020

Non-linear ICA based on Cramer-Wold metric

Authors: Przemysław Spurek, Aleksandra Nowak, Jacek Tabor, Łukasz Maziarka, Stanisław Jastrzębski

Abstract: Non-linear source separation is a challenging open problem with many applications. We extend a recently proposed Adversarial Non-linear ICA (ANICA) model, and introduce Cramer-Wold ICA (CW-ICA). In contrast to ANICA we use a simple, closed--form optimization target instead of a discriminator--based independence measure. Our results show that CW-ICA achieves comparable results to ANICA, while foreg… ▽ More Non-linear source separation is a challenging open problem with many applications. We extend a recently proposed Adversarial Non-linear ICA (ANICA) model, and introduce Cramer-Wold ICA (CW-ICA). In contrast to ANICA we use a simple, closed--form optimization target instead of a discriminator--based independence measure. Our results show that CW-ICA achieves comparable results to ANICA, while foregoing the need for adversarial training. △ Less

Submitted 1 March, 2019; originally announced March 2019.

Journal ref: Neural Information Processing. ICONIP 2020

Team EP at TAC 2018: Automating data extraction in systematic reviews of environmental agents

Authors: Artur Nowak, Paweł Kunstman

Abstract: We describe our entry for the Systematic Review Information Extraction track of the 2018 Text Analysis Conference. Our solution is an end-to-end, deep learning, sequence tagging model based on the BI-LSTM-CRF architecture. However, we use interleaved, alternating LSTM layers with highway connections instead of the more traditional approach, where last hidden states of both directions are concatena… ▽ More We describe our entry for the Systematic Review Information Extraction track of the 2018 Text Analysis Conference. Our solution is an end-to-end, deep learning, sequence tagging model based on the BI-LSTM-CRF architecture. However, we use interleaved, alternating LSTM layers with highway connections instead of the more traditional approach, where last hidden states of both directions are concatenated to create an input to the next layer. We also make extensive use of pre-trained word embeddings, namely GloVe and ELMo. Thanks to a number of regularization techniques, we were able to achieve relatively large capacity of the model (31.3M+ of trainable parameters) for the size of training set (100 documents, less than 200K tokens). The system's official score was 60.9% (micro-F1) and it ranked first for the Task 1. Additionally, after rectifying an obvious mistake in the submission format, the system scored 67.35%. △ Less

Submitted 7 January, 2019; originally announced January 2019.

Comments: 7 pages, 3 figures, to appear in the proceedings of Text Analysis Conference (TAC) 2018

Fixation probability and fixation time in structured populations

Authors: Josef Tkadlec, Andreas Pavlogiannis, Krishnendu Chatterjee, Martin A. Nowak

Abstract: The rate of biological evolution depends on the fixation probability and on the fixation time of new mutants. Intensive research has focused on identifying population structures that augment the fixation probability of advantageous mutants. But these `amplifiers of natural selection' typically increase fixation time. Here we study population structures that achieve a trade-off between high fixatio… ▽ More The rate of biological evolution depends on the fixation probability and on the fixation time of new mutants. Intensive research has focused on identifying population structures that augment the fixation probability of advantageous mutants. But these `amplifiers of natural selection' typically increase fixation time. Here we study population structures that achieve a trade-off between high fixation probability and short fixation time. First, we show that no amplifiers can have asymptotically lower absorption time than the well-mixed population. Then we design population structures that substantially augment the fixation probability with just a minor increase in fixation time. Finally, we show that those structures enable higher effective rate of evolution than the well-mixed population provided that the rate of generating advantageous mutants is relatively low. Our work sheds light on how population structure affects the rate of evolution. Moreover, our structures could be useful for lab-based, medical or industrial applications of evolutionary optimization. △ Less

Submitted 8 March, 2019; v1 submitted 27 September, 2018; originally announced October 2018.

Set Aggregation Network as a Trainable Pooling Layer

Authors: Łukasz Maziarka, Marek Śmieja, Aleksandra Nowak, Jacek Tabor, Łukasz Struski, Przemysław Spurek

Abstract: Global pooling, such as max- or sum-pooling, is one of the key ingredients in deep neural networks used for processing images, texts, graphs and other types of structured data. Based on the recent DeepSets architecture proposed by Zaheer et al. (NIPS 2017), we introduce a Set Aggregation Network (SAN) as an alternative global pooling layer. In contrast to typical pooling operators, SAN allows to e… ▽ More Global pooling, such as max- or sum-pooling, is one of the key ingredients in deep neural networks used for processing images, texts, graphs and other types of structured data. Based on the recent DeepSets architecture proposed by Zaheer et al. (NIPS 2017), we introduce a Set Aggregation Network (SAN) as an alternative global pooling layer. In contrast to typical pooling operators, SAN allows to embed a given set of features to a vector representation of arbitrary size. We show that by adjusting the size of embedding, SAN is capable of preserving the whole information from the input. In experiments, we demonstrate that replacing global pooling layer by SAN leads to the improvement of classification accuracy. Moreover, it is less prone to overfitting and can be used as a regularizer. △ Less

Submitted 25 November, 2019; v1 submitted 3 October, 2018; originally announced October 2018.

Comments: ICONIP 2019

Journal ref: Neural Information Processing. ICONIP 2019

Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function

Authors: Wojciech Tarnowski, Piotr Warchoł, Stanisław Jastrzębski, Jacek Tabor, Maciej A. Nowak

Abstract: We demonstrate that in residual neural networks (ResNets) dynamical isometry is achievable irrespectively of the activation function used. We do that by deriving, with the help of Free Probability and Random Matrix Theories, a universal formula for the spectral density of the input-output Jacobian at initialization, in the large network width and depth limit. The resulting singular value spectrum… ▽ More We demonstrate that in residual neural networks (ResNets) dynamical isometry is achievable irrespectively of the activation function used. We do that by deriving, with the help of Free Probability and Random Matrix Theories, a universal formula for the spectral density of the input-output Jacobian at initialization, in the large network width and depth limit. The resulting singular value spectrum depends on a single parameter, which we calculate for a variety of popular activation functions, by analyzing the signal propagation in the artificial neural network. We corroborate our results with numerical simulations of both random matrices and ResNets applied to the CIFAR-10 classification problem. Moreover, we study the consequence of this universal behavior for the initial and late phases of the learning processes. We conclude by drawing attention to the simple fact, that initialization acts as a confounding factor between the choice of activation function and the rate of learning. We propose that in ResNets this can be resolved based on our results, by ensuring the same level of dynamical isometry at initialization. △ Less

Submitted 4 March, 2019; v1 submitted 24 September, 2018; originally announced September 2018.

Journal ref: AISTATS 2019

Evolution of Cooperation on Stochastic Block Models

Authors: Babak Fotouhi, Naghmeh Momeni, Benjamin Allen, Martin A. Nowak

Abstract: Cooperation is a major factor in the evolution of human societies. The structure of human social networks, which affects the dynamics of cooperation and other interpersonal phenomena, have common structural signatures. One of these signatures is the tendency to organize as groups. Among the generative models that network theorists use to emulate this feature is the Stochastic Block Model (SBM). In… ▽ More Cooperation is a major factor in the evolution of human societies. The structure of human social networks, which affects the dynamics of cooperation and other interpersonal phenomena, have common structural signatures. One of these signatures is the tendency to organize as groups. Among the generative models that network theorists use to emulate this feature is the Stochastic Block Model (SBM). In this paper, we study evolutionary game dynamics on SBM networks. Using a recently-discovered duality between evolutionary games and coalescing random walks, we obtain analytical conditions such that natural selection favors cooperation over defection. We calculate the transition point for each community to favor cooperation. We find that a critical inter-community link creation probability exists for given group density, such that the overall network supports cooperation even if individual communities inhibit it. As a byproduct, we present mean-field solutions for the critical benefit-to-cost ratio which perform with remarkable accuracy for diverse generative network models, including those with community structure and heavy-tailed degree distributions. We also demonstrate the generalizability of the results to arbitrary two-player games. △ Less

Submitted 9 July, 2018; originally announced July 2018.

Conjoining uncooperative societies facilitates evolution of cooperation

Authors: Babak Fotouhi, Naghmeh Momeni, Benjamin Allen, Martin A. Nowak

Abstract: Social structure affects the emergence and maintenance of cooperation. Here we study the evolutionary dynamics of cooperation in fragmented societies, and show that conjoining segregated cooperation-inhibiting groups, if done properly, rescues the fate of collective cooperation. We highlight the essential role of inter-group ties, that sew the patches of the social network together and facilitate… ▽ More Social structure affects the emergence and maintenance of cooperation. Here we study the evolutionary dynamics of cooperation in fragmented societies, and show that conjoining segregated cooperation-inhibiting groups, if done properly, rescues the fate of collective cooperation. We highlight the essential role of inter-group ties, that sew the patches of the social network together and facilitate cooperation. We point out several examples of this phenomenon in actual settings. We explore random and non-random graphs, as well as empirical networks. In many cases we find a marked reduction of the critical benefit-to-cost ratio needed for sustaining cooperation. Our finding gives hope that the increasing worldwide connectivity, if managed properly, can promote global cooperation. △ Less

Submitted 14 June, 2018; v1 submitted 30 May, 2018; originally announced May 2018.

Journal ref: Nature Human Behaviour, volume 2 (2018)

Successful Nash Equilibrium Agent for a 3-Player Imperfect-Information Game

Authors: Sam Ganzfried, Austin Nowak, Joannier Pinales

Abstract: Creating strong agents for games with more than two players is a major open problem in AI. Common approaches are based on approximating game-theoretic solution concepts such as Nash equilibrium, which have strong theoretical guarantees in two-player zero-sum games, but no guarantees in non-zero-sum games or in games with more than two players. We describe an agent that is able to defeat a variety… ▽ More Creating strong agents for games with more than two players is a major open problem in AI. Common approaches are based on approximating game-theoretic solution concepts such as Nash equilibrium, which have strong theoretical guarantees in two-player zero-sum games, but no guarantees in non-zero-sum games or in games with more than two players. We describe an agent that is able to defeat a variety of realistic opponents using an exact Nash equilibrium strategy in a 3-player imperfect-information game. This shows that, despite a lack of theoretical guarantees, agents based on Nash equilibrium strategies can be successful in multiplayer games after all. △ Less

Submitted 13 April, 2018; originally announced April 2018.

Strong Amplifiers of Natural Selection: Proofs

Authors: Andreas Pavlogiannis, Josef Tkadlec, Krishnendu Chatterjee, Martin A. Nowak

Abstract: We consider the modified Moran process on graphs to study the spread of genetic and cultural mutations on structured populations. An initial mutant arises either spontaneously (aka \emph{uniform initialization}), or during reproduction (aka \emph{temperature initialization}) in a population of $n$ individuals, and has a fixed fitness advantage $r>1$ over the residents of the population. The fixati… ▽ More We consider the modified Moran process on graphs to study the spread of genetic and cultural mutations on structured populations. An initial mutant arises either spontaneously (aka \emph{uniform initialization}), or during reproduction (aka \emph{temperature initialization}) in a population of $n$ individuals, and has a fixed fitness advantage $r>1$ over the residents of the population. The fixation probability is the probability that the mutant takes over the entire population. Graphs that ensure fixation probability of~1 in the limit of infinite populations are called \emph{strong amplifiers}. Previously, only a few examples of strong amplifiers were known for uniform initialization, whereas no strong amplifiers were known for temperature initialization. In this work, we study necessary and sufficient conditions for strong amplification, and prove negative and positive results. We show that for temperature initialization, graphs that are unweighted and/or self-loop-free have fixation probability upper-bounded by $1-1/f(r)$, where $f(r)$ is a function linear in $r$. Similarly, we show that for uniform initialization, bounded-degree graphs that are unweighted and/or self-loop-free have fixation probability upper-bounded by $1-1/g(r,c)$, where $c$ is the degree bound and $g(r,c)$ a function linear in $r$. Our main positive result complements these negative results, and is as follows: every family of undirected graphs with (i)~self loops and (ii)~diameter bounded by $n^{1-ε}$, for some fixed $ε>0$, can be assigned weights that makes it a strong amplifier, both for uniform and temperature initialization. △ Less

Submitted 14 May, 2018; v1 submitted 7 February, 2018; originally announced February 2018.

Variational auto-encoding of protein sequences

Authors: Sam Sinai, Eric Kelsic, George M. Church, Martin A. Nowak

Abstract: Proteins are responsible for the most diverse set of functions in biology. The ability to extract information from protein sequences and to predict the effects of mutations is extremely valuable in many domains of biology and medicine. However the mapping between protein sequence and function is complex and poorly understood. Here we present an embedding of natural protein sequences using a Variat… ▽ More Proteins are responsible for the most diverse set of functions in biology. The ability to extract information from protein sequences and to predict the effects of mutations is extremely valuable in many domains of biology and medicine. However the mapping between protein sequence and function is complex and poorly understood. Here we present an embedding of natural protein sequences using a Variational Auto-Encoder and use it to predict how mutations affect protein function. We use this unsupervised approach to cluster natural variants and learn interactions between sets of positions within a protein. This approach generally performs better than baseline methods that consider no interactions within sequences, and in some cases better than the state-of-the-art approaches that use the inverse-Potts model. This generative model can be used to computationally guide exploration of protein sequence space and to better inform rational and automatic protein design. △ Less

Submitted 3 January, 2018; v1 submitted 9 December, 2017; originally announced December 2017.

Comments: Abstract for oral presentation at NIPS 2017 Workshop on Machine Learning in Computational Biology

MPI_XSTAR: MPI-based Parallelization of the XSTAR Photoionization Program

Authors: Ashkbiz Danehkar, Michael A. Nowak, Julia C. Lee, Randall K. Smith

Abstract: We describe a program for the parallel implementation of multiple runs of XSTAR, a photoionization code that is used to predict the physical properties of an ionized gas from its emission and/or absorption lines. The parallelization program, called MPI_XSTAR, has been developed and implemented in the C++ language by using the Message Passing Interface (MPI) protocol, a conventional standard of par… ▽ More We describe a program for the parallel implementation of multiple runs of XSTAR, a photoionization code that is used to predict the physical properties of an ionized gas from its emission and/or absorption lines. The parallelization program, called MPI_XSTAR, has been developed and implemented in the C++ language by using the Message Passing Interface (MPI) protocol, a conventional standard of parallel computing. We have benchmarked parallel multiprocessing executions of XSTAR, using MPI_XSTAR, against a serial execution of XSTAR, in terms of the parallelization speedup and the computing resource efficiency. Our experience indicates that the parallel execution runs significantly faster than the serial execution, however, the efficiency in terms of the computing resource usage decreases with increasing the number of processors used in the parallel computing. △ Less

Submitted 28 November, 2017; originally announced December 2017.

Comments: 5 pages, 1 figure, accepted for publication in Publications of the Astronomical Society of the Pacific (PASP)

Journal ref: Publ.Astron.Soc.Pac.130:024501,2018

Revised Note on Learning Algorithms for Quadratic Assignment with Graph Neural Networks

Authors: Alex Nowak, Soledad Villar, Afonso S. Bandeira, Joan Bruna

Abstract: Inverse problems correspond to a certain type of optimization problems formulated over appropriate input distributions. Recently, there has been a growing interest in understanding the computational hardness of these optimization problems, not only in the worst case, but in an average-complexity sense under this same input distribution. In this revised note, we are interested in studying another… ▽ More Inverse problems correspond to a certain type of optimization problems formulated over appropriate input distributions. Recently, there has been a growing interest in understanding the computational hardness of these optimization problems, not only in the worst case, but in an average-complexity sense under this same input distribution. In this revised note, we are interested in studying another aspect of hardness, related to the ability to learn how to solve a problem by simply observing a collection of previously solved instances. These 'planted solutions' are used to supervise the training of an appropriate predictive model that parametrizes a broad class of algorithms, with the hope that the resulting model will provide good accuracy-complexity tradeoffs in the average sense. We illustrate this setup on the Quadratic Assignment Problem, a fundamental problem in Network Science. We observe that data-driven models based on Graph Neural Networks offer intriguingly good performance, even in regimes where standard relaxation based techniques appear to suffer. △ Less

Submitted 30 August, 2018; v1 submitted 22 June, 2017; originally announced June 2017.

Comments: Revised note to arXiv:1706.07450v1 that appeared in IEEE Data Science Workshop 2018

Highly curved image sensors: a practical approach for improved optical performance

Authors: Brian Guenter, Neel Joshi, Richard Stoakley, Andrew Keefe, Kevin Geary, Ryan Freeman, Jake Hundley, Pamela Patterson, David Hammon, Guillermo Herrera, Elena Sherman, Andrew Nowak, Randall Schubert, Peter Brewer, Louis Yang, Russell Mott, Geoff McKnight

Abstract: The significant optical and size benefits of using a curved focal surface for imaging systems have been well studied yet never brought to market for lack of a high-quality, mass-producible, curved image sensor. In this work we demonstrate that commercial silicon CMOS image sensors can be thinned and formed into accurate, highly curved optical surfaces with undiminished functionality. Our key devel… ▽ More The significant optical and size benefits of using a curved focal surface for imaging systems have been well studied yet never brought to market for lack of a high-quality, mass-producible, curved image sensor. In this work we demonstrate that commercial silicon CMOS image sensors can be thinned and formed into accurate, highly curved optical surfaces with undiminished functionality. Our key development is a pneumatic forming process that avoids rigid mechanical constraints and suppresses wrinkling instabilities. A combination of forming-mold design, pressure membrane elastic properties, and controlled friction forces enables us to gradually contact the die at the corners and smoothly press the sensor into a spherical shape. Allowing the die to slide into the concave target shape enables a threefold increase in the spherical curvature over prior approaches having mechanical constraints that resist deformation, and create a high-stress, stretch-dominated state. Our process creates a bridge between the high precision and low-cost but planar CMOS process, and ideal non-planar component shapes such as spherical imagers for improved optical systems. We demonstrate these curved sensors in prototype cameras with custom lenses, measuring exceptional resolution of 3220 line-widths per picture height at an aperture of f/1.2 and nearly 100% relative illumination across the field. Though we use a 1/2.3" format image sensor in this report, we also show this process is generally compatible with many state of the art imaging sensor formats. By example, we report photogrammetry test data for an APS-C sized silicon die formed to a 30$^\circ$ subtended spherical angle. These gains in sharpness and relative illumination enable a new generation of ultra-high performance, manufacturable, digital imaging systems for scientific, industrial, and artistic use. △ Less

Submitted 20 June, 2017; originally announced June 2017.

Journal ref: Opt. Express 25, 13010-13023 (2017)

Faster Monte-Carlo Algorithms for Fixation Probability of the Moran Process on Undirected Graphs

Authors: Krishnendu Chatterjee, Rasmus Ibsen-Jensen, Martin A. Nowak

Abstract: Evolutionary graph theory studies the evolutionary dynamics in a population structure given as a connected graph. Each node of the graph represents an individual of the population, and edges determine how offspring are placed. We consider the classical birth-death Moran process where there are two types of individuals, namely, the residents with fitness 1 and mutants with fitness r. The fitness in… ▽ More Evolutionary graph theory studies the evolutionary dynamics in a population structure given as a connected graph. Each node of the graph represents an individual of the population, and edges determine how offspring are placed. We consider the classical birth-death Moran process where there are two types of individuals, namely, the residents with fitness 1 and mutants with fitness r. The fitness indicates the reproductive strength. The evolutionary dynamics happens as follows: in the initial step, in a population of all resident individuals a mutant is introduced, and then at each step, an individual is chosen proportional to the fitness of its type to reproduce, and the offspring replaces a neighbor uniformly at random. The process stops when all individuals are either residents or mutants. The probability that all individuals in the end are mutants is called the fixation probability. We present faster polynomial-time Monte-Carlo algorithms for finidng the fixation probability on undirected graphs. Our algorithms are always at least a factor O(n^2/log n) faster as compared to the previous algorithms, where n is the number of nodes, and is polynomial even if r is given in binary. We also present lower bounds showing that the upper bound on the expected number of effective steps we present is asymptotically tight for undirected graphs. △ Less

Submitted 21 June, 2017; originally announced June 2017.

Socio-inspired ICT - Towards a socially grounded society-ICT symbiosis

Authors: A. Ferscha, K. Farrahi, J. van denHoven, D. Hales, A. Nowak, P. Lukowicz, D. Helbing

Abstract: Modern ICT (Information and Communication Technology) has developed a vision where the "computer" is no longer associated with the concept of a single device or a network of devices, but rather the entirety of situated services originating in a digital world, which are perceived through the physical world. It is observed that services with explicit user input and output are becoming to be replaced… ▽ More Modern ICT (Information and Communication Technology) has developed a vision where the "computer" is no longer associated with the concept of a single device or a network of devices, but rather the entirety of situated services originating in a digital world, which are perceived through the physical world. It is observed that services with explicit user input and output are becoming to be replaced by a computing landscape sensing the physical world via a huge variety of sensors, and controlling it via a plethora of actuators. The nature and appearance of computing devices is changing to be hidden in the fabric of everyday life, invisibly networked, and omnipresent, with applications greatly being based on the notions of context and knowledge. Interaction with such globe spanning, modern ICT systems will presumably be more implicit, at the periphery of human attention, rather than explicit, i.e. at the focus of human attention. Socio-inspired ICT assumes that future, globe scale ICT systems should be viewed as social systems. Such a view challenges research to identify and formalize the principles of interaction and adaptation in social systems, so as to be able to ground future ICT systems on those principles. This position paper therefore is concerned with the intersection of social behaviour and modern ICT, creating or recreating social conventions and social contexts through the use of pervasive, globe-spanning, omnipresent and participative ICT. △ Less

Submitted 6 April, 2013; originally announced April 2013.

Journal ref: Eur. Phys. J. Special Topics vol. 214, pp 401-434 (2012)

TTP: Tool for Tumor Progression

Authors: Johannes G. Reiter, Ivana Bozic, Krishnendu Chatterjee, Martin A. Nowak

Abstract: In this work we present a flexible tool for tumor progression, which simulates the evolutionary dynamics of cancer. Tumor progression implements a multi-type branching process where the key parameters are the fitness landscape, the mutation rate, and the average time of cell division. The fitness of a cancer cell depends on the mutations it has accumulated. The input to our tool could be any fitne… ▽ More In this work we present a flexible tool for tumor progression, which simulates the evolutionary dynamics of cancer. Tumor progression implements a multi-type branching process where the key parameters are the fitness landscape, the mutation rate, and the average time of cell division. The fitness of a cancer cell depends on the mutations it has accumulated. The input to our tool could be any fitness landscape, mutation rate, and cell division time, and the tool produces the growth dynamics and all relevant statistics. △ Less

Submitted 21 March, 2013; originally announced March 2013.

Comments: A conference version of the paper will appear in CAV 2013