Showing 1–50 of 85 results for author: Singh, S P

Structural transitions of a Semi-Flexible Polyampholyte

Authors: Rakesh Palariya, Sunil P. Singh

Abstract: Polyampholytes (PA) are charged polymers composed of positively and negatively charged monomers along their backbone. The sequence of the charged monomers and the bending of the chain significantly influence the conformation and dynamical behavior of the PA. Using coarse-grained molecular dynamics simulations, we comprehensively study the structural and dynamical properties of flexible and semi-fl… ▽ More Polyampholytes (PA) are charged polymers composed of positively and negatively charged monomers along their backbone. The sequence of the charged monomers and the bending of the chain significantly influence the conformation and dynamical behavior of the PA. Using coarse-grained molecular dynamics simulations, we comprehensively study the structural and dynamical properties of flexible and semi-flexible polyampholytes'. The simulation results demonstrate a flexible polyampholyte (PA) chain, displaying a transition from a coil to a globule in the parameter space of the charge sequence. Additionally, the behavior of the mean-square displacement (MSD), denoted as $<(Δr(t))^2>$, reveals distinct dynamics, specifically for the alternating and charge-segregated sequences. The MSD follows a power-law behavior, where $<(Δr(t))^2> \sim t^β$, with $β\approx 3/5$ and $β\approx 1/2$ for the alternating sequence and charge-segregated sequence in the absence of hydrodynamic interactions, respectively. However, when hydrodynamic interactions are incorporated, the exponent $β$ shifts to approximately 3/5 for the charge-segregated sequence and 2/3 for the well-mixed alternating sequence. For a semi-flexible PA chain, varying the bending rigidity and electrostatic interaction strength ($Γ_e$) leads to distinct, fascinating conformational states, including globule, bundle, and torus-like conformations. We show that PA acquires circular and hairpin-like conformations in the intermediate bending regime. The transition between various conformations is identified in terms of the shape factor estimated from the ratios of eigenvalues of the gyration tensor. △ Less

Submitted 28 August, 2024; originally announced August 2024.

Local vs Global continual learning

Authors: Giulia Lanzillotta, Sidak Pal Singh, Benjamin F. Grewe, Thomas Hofmann

Abstract: Continual learning is the problem of integrating new information in a model while retaining the knowledge acquired in the past. Despite the tangible improvements achieved in recent years, the problem of continual learning is still an open one. A better understanding of the mechanisms behind the successes and failures of existing continual learning algorithms can unlock the development of new succe… ▽ More Continual learning is the problem of integrating new information in a model while retaining the knowledge acquired in the past. Despite the tangible improvements achieved in recent years, the problem of continual learning is still an open one. A better understanding of the mechanisms behind the successes and failures of existing continual learning algorithms can unlock the development of new successful strategies. In this work, we view continual learning from the perspective of the multi-task loss approximation, and we compare two alternative strategies, namely local and global approximations. We classify existing continual learning algorithms based on the approximation used, and we assess the practical effects of this distinction in common continual learning settings.Additionally, we study optimal continual learning objectives in the case of local polynomial approximations and we provide examples of existing algorithms implementing the optimal objectives △ Less

Submitted 23 July, 2024; originally announced July 2024.

Comments: (10 pages, Will appear in the proceedings of CoLLAs 2024)

Active Polar Ring Polymer in Shear Flow -- An Analytical Study

Authors: Roland G. Winkler, Sunil P. Singh

Abstract: We theoretically study the conformational and dynamical properties of semiflexible active polar ring polymers under linear shear flow. A ring is described as a continuous Gaussian polymer with a tangential active force of a constant density along its contour. The linear but non-Hermitian equation of motion is solved using an eigenfunction expansion, which yields activity-independent, but shear-rat… ▽ More We theoretically study the conformational and dynamical properties of semiflexible active polar ring polymers under linear shear flow. A ring is described as a continuous Gaussian polymer with a tangential active force of a constant density along its contour. The linear but non-Hermitian equation of motion is solved using an eigenfunction expansion, which yields activity-independent, but shear-rate-dependent, relaxation times and activity-dependent frequencies. As a consequence, the ring's stationary-state properties are independent of activity, and its conformations as well as rheological properties are equal to those of a passive ring under shear. The presence of characteristic time scales by the relaxation and the frequency gives rise to a particular dynamical behavior. A tank-treading-like motion emerges for large relaxation times and high frequencies, specifically for stiffer rings, governed by the activity-dependent frequencies. In the case of very flexible polymers, the relaxation behavior dominates over tank-treading. Shear strongly affects the crossover from a tank-treading to a relaxation-time dominated dynamics and suppresses tank-treading. This is reflected in the tumbling frequency, which exhibits two shear-rate dependent regimes, with an activity-dependent plateau at low shear rates followed by a power-law regime with increasing tumbling frequency for large shear rates. △ Less

Submitted 3 July, 2024; originally announced July 2024.

Comments: 7 figures

Landscaping Linear Mode Connectivity

Authors: Sidak Pal Singh, Linara Adilova, Michael Kamp, Asja Fischer, Bernhard Schölkopf, Thomas Hofmann

Abstract: The presence of linear paths in parameter space between two different network solutions in certain cases, i.e., linear mode connectivity (LMC), has garnered interest from both theoretical and practical fronts. There has been significant research that either practically designs algorithms catered for connecting networks by adjusting for the permutation symmetries as well as some others that more th… ▽ More The presence of linear paths in parameter space between two different network solutions in certain cases, i.e., linear mode connectivity (LMC), has garnered interest from both theoretical and practical fronts. There has been significant research that either practically designs algorithms catered for connecting networks by adjusting for the permutation symmetries as well as some others that more theoretically construct paths through which networks can be connected. Yet, the core reasons for the occurrence of LMC, when in fact it does occur, in the highly non-convex loss landscapes of neural networks are far from clear. In this work, we take a step towards understanding it by providing a model of how the loss landscape needs to behave topographically for LMC (or the lack thereof) to manifest. Concretely, we present a `mountainside and ridge' perspective that helps to neatly tie together different geometric features that can be spotted in the loss landscape along the training runs. We also complement this perspective by providing a theoretical analysis of the barrier height, for which we provide empirical support, and which additionally extends as a faithful predictor of layer-wise LMC. We close with a toy example that provides further intuition on how barriers arise in the first place, all in all, showcasing the larger aim of the work -- to provide a working model of the landscape and its topography for the occurrence of LMC. △ Less

Submitted 23 June, 2024; originally announced June 2024.

Comments: ICML 2024 HiLD workshop paper

Collective dynamics of active dumbbells near a circular obstacle

Authors: Chandranshu Tiwari, Sunil P. Singh

Abstract: In this article, we present the collective dynamics of active dumbbells in the presence of a static circular obstacle using Brownian dynamics simulation. The active dumbbells aggregate on the surface of a circular obstacle beyond a critical radius. The aggregation is non-uniform along the circumference, and the aggregate size increases with the activity and the curvature radius. The dense aggregat… ▽ More In this article, we present the collective dynamics of active dumbbells in the presence of a static circular obstacle using Brownian dynamics simulation. The active dumbbells aggregate on the surface of a circular obstacle beyond a critical radius. The aggregation is non-uniform along the circumference, and the aggregate size increases with the activity and the curvature radius. The dense aggregate of active dumbbells displays persistent rotational motion with a certain angular speed, which linearly increases with the activity. Further, we show the strong polar ordering of the active dumbbells within the aggregate. The polar ordering exhibits a long-range correlation, with the correlation length corresponding to the aggregate size. Additionally, we show that the residence time of an active dumbbell on the obstacle surface grows rapidly with area fraction due to many-body interactions that lead to a slowdown of the rotational diffusion. The article further considers the dynamical behavior of a tracer particle in the solution of active dumbbells. Interestingly, the speed of the passive tracer particle displays a crossover from monotonically decreasing to increasing with the tracer particle's size upon increasing the dumbbells' speed. Furthermore, the effective diffusion of the tracer particle displays the non-monotonic behavior with area fraction; the initial increase of the diffusivity is followed by a decrease for larger area fraction. △ Less

Submitted 11 June, 2024; originally announced June 2024.

Comments: 14 pages, 23 figures

Journal ref: Soft Matter, 10/06/2024

The MESA Security Model 2.0: A Dynamic Framework for Mitigating Stealth Data Exfiltration

Authors: Sanjeev Pratap Singh, Naveed Afzal

Abstract: The rising complexity of cyber threats calls for a comprehensive reassessment of current security frameworks in business environments. This research focuses on Stealth Data Exfiltration, a significant cyber threat characterized by covert infiltration, extended undetectability, and unauthorized dissemination of confidential data. Our findings reveal that conventional defense-in-depth strategies oft… ▽ More The rising complexity of cyber threats calls for a comprehensive reassessment of current security frameworks in business environments. This research focuses on Stealth Data Exfiltration, a significant cyber threat characterized by covert infiltration, extended undetectability, and unauthorized dissemination of confidential data. Our findings reveal that conventional defense-in-depth strategies often fall short in combating these sophisticated threats, highlighting the immediate need for a shift in information risk management across businesses. The evolving nature of cyber threats, driven by advancements in techniques such as social engineering, multi-vector attacks, and Generative AI, underscores the need for robust, adaptable, and comprehensive security strategies. As we navigate this complex landscape, it is crucial to anticipate potential threats and continually update our defenses. We propose a shift from traditional perimeter-based, prevention-focused models, which depend on a static attack surface, to a more dynamic framework that prepares for inevitable breaches. This suggested model, known as MESA 2.0 Security Model, prioritizes swift detection, immediate response, and ongoing resilience, thereby enhancing an organizations ability to promptly identify and neutralize threats, significantly reducing the consequences of security breaches. This study suggests that businesses adopt a forward-thinking and adaptable approach to security management to stay ahead of the ever-changing cyber threat landscape. △ Less

Submitted 17 May, 2024; originally announced May 2024.

Journal ref: International Journal of Network Security & Its Applications (IJNSA) 2024

Post Quantum Cryptography and its Comparison with Classical Cryptography

Authors: Tanmay Tripathi, Abhinav Awasthi, Shaurya Pratap Singh, Atul Chaturvedi

Abstract: Cryptography plays a pivotal role in safeguarding sensitive information and facilitating secure communication. Classical cryptography relies on mathematical computations, whereas quantum cryptography operates on the principles of quantum mechanics, offering a new frontier in secure communication. Quantum cryptographic systems introduce novel dimensions to security, capable of detecting and thwarti… ▽ More Cryptography plays a pivotal role in safeguarding sensitive information and facilitating secure communication. Classical cryptography relies on mathematical computations, whereas quantum cryptography operates on the principles of quantum mechanics, offering a new frontier in secure communication. Quantum cryptographic systems introduce novel dimensions to security, capable of detecting and thwarting eavesdropping attempts. By contrasting quantum cryptography with its classical counterpart, it becomes evident how quantum mechanics revolutionizes the landscape of secure communication. △ Less

Submitted 28 March, 2024; originally announced March 2024.

Berezinskii-Kosterlitz-Thouless to BCS-like superconducting transition crossover driven by weak magnetic fields in ultra-thin NbN films

Authors: Meenakshi Sharma, Sergio Caprara, Andrea Perali, Surinder P. Singh, Sandeep Singh, Matteo Fretto, Natascia De Leo, Nicola Pinto

Abstract: The Berezinskii-Kosterlitz-Thouless (BKT) transition in ultra-thin NbN films is investigated in the presence of weak perpendicular magnetic fields. A jump in the phase stiffness at the BKT transition is detected up to 5 G, while the BKT features are smeared between 5 G and 50 G, disappearing altogether at 100 G, where conventional current-voltage behaviour is observed. Our findings demonstrate tha… ▽ More The Berezinskii-Kosterlitz-Thouless (BKT) transition in ultra-thin NbN films is investigated in the presence of weak perpendicular magnetic fields. A jump in the phase stiffness at the BKT transition is detected up to 5 G, while the BKT features are smeared between 5 G and 50 G, disappearing altogether at 100 G, where conventional current-voltage behaviour is observed. Our findings demonstrate that weak magnetic fields, insignificant in bulk systems, deeply affect our ultra-thin system, promoting a crossover from Halperin-Nelson fluctuations to a BCS-like state with Ginzburg-Landau fluctuations, as the field increases. This behavior is related to field-induced free vortices that screen the vortex-antivortex interaction and smear the BKT transition. △ Less

Submitted 16 July, 2024; v1 submitted 18 March, 2024; originally announced March 2024.

Comments: 10 pages, 13 figures

$Herschel$ investigation of cores and filamentary structures in L1251 located in the Cepheus flare

Authors: Divyansh Dewan, Archana Soam, Guo-Yin Zhang, Akhil Lasrado, Saikhom Pravash Singh, Chang Won Lee

Abstract: Context: Molecular clouds are the prime locations of star formation. These clouds contain filamentary structures and cores which are crucial in the formation of young stars. Aims: In this work, we aim to quantify the physical properties of structural characteristics within the molecular cloud L1251 to better understand the initial conditions for star formation. Methods: We applied the getsf algori… ▽ More Context: Molecular clouds are the prime locations of star formation. These clouds contain filamentary structures and cores which are crucial in the formation of young stars. Aims: In this work, we aim to quantify the physical properties of structural characteristics within the molecular cloud L1251 to better understand the initial conditions for star formation. Methods: We applied the getsf algorithm to identify cores and filaments within the molecular cloud L1251 using the Herschel multiband dust continuum image, enabling us to measure their respective physical properties. Additionally, we utilized an enhanced differential term algorithm to produce high-resolution temperature maps and column density maps with a resolution of ${13.5}''$. Results: We identified 122 cores in the region. Out of them, 23 are protostellar cores, 13 are robust prestellar cores, 32 are candidate prestellar cores (including 13 robust prestellar cores and 19 strictly candidate prestellar cores), and 67 are unbound starless cores. getsf also found 147 filament structures in the region. Statistical analysis of the physical properties (mass (M), temperature (T), size, and core brightness (hereafter, we are using the word luminosity (L)) for the core brightness) of obtained cores shows a negative correlation between core mass and temperature and a positive correlation between (M/L) and (M/T). Analysis of the filaments gives a median width of 0.14 pc and no correlation between width and length. Out of those 122 cores, 92 are present in filaments (75.4%) and the remaining were outside them. Out of the cores present in filaments, 57 (62%) cores are present in supercritical filaments ($M_{\rm line}>16M_{\odot }/{\rm pc}$). △ Less

Submitted 16 March, 2024; originally announced March 2024.

Comments: 15 pages, 20 figures, 2 tables, accepted for publication in JAA

Hallmarks of Optimization Trajectories in Neural Networks: Directional Exploration and Redundancy

Authors: Sidak Pal Singh, Bobby He, Thomas Hofmann, Bernhard Schölkopf

Abstract: We propose a fresh take on understanding the mechanisms of neural networks by analyzing the rich directional structure of optimization trajectories, represented by their pointwise parameters. Towards this end, we introduce some natural notions of the complexity of optimization trajectories, both qualitative and quantitative, which hallmark the directional nature of optimization in neural networks:… ▽ More We propose a fresh take on understanding the mechanisms of neural networks by analyzing the rich directional structure of optimization trajectories, represented by their pointwise parameters. Towards this end, we introduce some natural notions of the complexity of optimization trajectories, both qualitative and quantitative, which hallmark the directional nature of optimization in neural networks: when is there redundancy, and when exploration. We use them to reveal the inherent nuance and interplay involved between various optimization choices, such as momentum and weight decay. Further, the trajectory perspective helps us see the effect of scale on regularizing the directional nature of trajectories, and as a by-product, we also observe an intriguing heterogeneity of Q,K,V dynamics in the middle attention layers in LLMs and which is homogenized by scale. Importantly, we put the significant directional redundancy observed to the test by demonstrating that training only scalar batchnorm parameters some while into training matches the performance of training the entire network, which thus exhibits the potential of hybrid optimization schemes that are geared towards efficiency. △ Less

Submitted 24 June, 2024; v1 submitted 12 March, 2024; originally announced March 2024.

Comments: Preprint, 57 pages

Investigation of the Thermal Structure in the Atmospheric Boundary Layer During Evening Transition and the Impact of Aerosols on Radiative Cooling

Authors: Suryadev Pratap Singh, Mohammad Rafiuddin, Subham Banerjee, Sreenivas K R

Abstract: We have explored the evening transition using data from eighty days of observations across two fog seasons at the Kempegowda International Airport, Bengaluru (KIAB). Through field experiments and simulations integrating aerosol interaction in a radiation-conduction model, we elucidate the impact of aerosols on longwave cooling of the Atmospheric Boundary Layer (ABL). Field observations indicate th… ▽ More We have explored the evening transition using data from eighty days of observations across two fog seasons at the Kempegowda International Airport, Bengaluru (KIAB). Through field experiments and simulations integrating aerosol interaction in a radiation-conduction model, we elucidate the impact of aerosols on longwave cooling of the Atmospheric Boundary Layer (ABL). Field observations indicate that under calm and clear-sky conditions, the evening transition typically results in a distinct vertical thermal structure called the Lifted Temperature Minimum (LTM). We observe that the prevailing profile near the surface post-sunset is the LTM-profile. Additionally, the occurrence of LTM is observed to increase with decreases in downward and upward longwave flux, soil sensible heat flux, wind speed, and turbulent kinetic energy measured at two meters above ground level (AGL). In such scenarios, the intensity of LTM-profiles is primarily governed by aerosol-induced longwave heating rate (LHR) within the surface layer. Furthermore, the presence of clouds leads to increased downward flux, causing the disappearance of LTM, whereas shallow fog can enhance LTM intensity, as observed in both field observations and simulations. Usually, prevailing radiation models underestimate aerosol-induced longwave heating rate (LHR) by an order, compared to actual field observations. We attribute this difference to aerosol-induced radiation divergence. We show that impact of aerosol-induced LHR extends hundreds of meters into the inversion layer, affecting temperature profiles and potentially influencing processes such as fog formation. As the fog layer develops, LHR strengthens at its upper boundary, however, we highlight the difficulty in detecting this cooling using remote instruments such as microwave radiometer. △ Less

Submitted 11 March, 2024; originally announced March 2024.

Towards Meta-Pruning via Optimal Transport

Authors: Alexander Theus, Olin Geimer, Friedrich Wicke, Thomas Hofmann, Sotiris Anagnostidis, Sidak Pal Singh

Abstract: Structural pruning of neural networks conventionally relies on identifying and discarding less important neurons, a practice often resulting in significant accuracy loss that necessitates subsequent fine-tuning efforts. This paper introduces a novel approach named Intra-Fusion, challenging this prevailing pruning paradigm. Unlike existing methods that focus on designing meaningful neuron importanc… ▽ More Structural pruning of neural networks conventionally relies on identifying and discarding less important neurons, a practice often resulting in significant accuracy loss that necessitates subsequent fine-tuning efforts. This paper introduces a novel approach named Intra-Fusion, challenging this prevailing pruning paradigm. Unlike existing methods that focus on designing meaningful neuron importance metrics, Intra-Fusion redefines the overlying pruning procedure. Through utilizing the concepts of model fusion and Optimal Transport, we leverage an agnostically given importance metric to arrive at a more effective sparse model representation. Notably, our approach achieves substantial accuracy recovery without the need for resource-intensive fine-tuning, making it an efficient and promising tool for neural network compression. Additionally, we explore how fusion can be added to the pruning process to significantly decrease the training time while maintaining competitive performance. We benchmark our results for various networks on commonly used datasets such as CIFAR-10, CIFAR-100, and ImageNet. More broadly, we hope that the proposed Intra-Fusion approach invigorates exploration into a fresh alternative to the predominant compression approaches. Our code is available here: https://github.com/alexandertheus/Intra-Fusion. △ Less

Submitted 13 February, 2024; v1 submitted 12 February, 2024; originally announced February 2024.

Comments: Accepted as a Spotlight (top 5% of submissions) at the International Conference on Learning Representations (ICLR) 2024

Formation of nano and micro scale hierarchical structures in MgO and ZnO quantum dots doped LC media: The role of competitive forces

Authors: A. K. Singh, S. P. Singh

Abstract: In this paper, we have studied the effect of doping of ZnO and MgO nanoparticles (NPs) in 4-(trans-4-n-hexylcyclo-hexyl) isothiocyanatobenzoate. A thorough comparison of dielectric properties, optoelectronic properties, and calorimetric phase transition properties has been done for MgO and ZnO NP doped LC. We prepare their homogenous mixture of MgO and ZnO NPs in toluene and transfer into cells ma… ▽ More In this paper, we have studied the effect of doping of ZnO and MgO nanoparticles (NPs) in 4-(trans-4-n-hexylcyclo-hexyl) isothiocyanatobenzoate. A thorough comparison of dielectric properties, optoelectronic properties, and calorimetric phase transition properties has been done for MgO and ZnO NP doped LC. We prepare their homogenous mixture of MgO and ZnO NPs in toluene and transfer into cells made of glass and Indium Tin-Oxide (ITO) coated glass. The observed microstructures in the hybrid system can be classified into three main categories: grain like structures formed by aggregation of smaller size MgO nanoparticles while liquid crystal molecules anchor over the surfaces of nanoparticles, the grtu grain-like structures further integrate to form inorganic polymeric type of honeycomb-like mesostructures in presence of glass surface, and flower-like clusters of MgO nanoparticles on ITO surface. The smaller size nanoparticles can maintain the energy balance by allowing the anchoring of liquid crystal molecules over their surfaces whereas the larger size nanoparticles cannot compromise or maintain the energy balance with the liquid crystal molecules and are separated out to nucleate and form bigger size nanoaggregate or clusters. The energy preference of the substrate and nanoparticle's surface to liquid crystal molecules plays an important role in the formation of different types of hierarchical nano- and microstructures. We account the reasons for the formation of nano and micro scale hierarchical structures on the basis of the competition between the forces: NP-NP, LC-LC, NP-LC, Glass/ITO-NP, and Glass/ITO-LC interactions. We observed a considerable change in the dielectric properties, transition temperature, bandgap, and other parameters of LC molecules when MgO NPs are doped, but a minor change occurs when ZnO NPs are doped in LC. △ Less

Submitted 17 January, 2024; originally announced January 2024.

Comments: 22 pages, 20 figures, 5 tables

Journal ref: Condensed Matter Physics, 2023, vol. 26, No. 4, 43602

Characteristic features of self-avoiding active Brownian polymers under linear shear flow

Authors: Arindam Panda, Roland G. Winkler, Sunil P. Singh

Abstract: We present Brownian dynamics simulation results of a flexible linear polymer with excluded-volume interactions under shear flow in the presence of active noise. The active noise strongly affects the polymer's conformational and dynamical properties, such as the stretching in the flow direction and compression in the gradient direction, shear-induced alignment, and shear viscosity. In the asymptoti… ▽ More We present Brownian dynamics simulation results of a flexible linear polymer with excluded-volume interactions under shear flow in the presence of active noise. The active noise strongly affects the polymer's conformational and dynamical properties, such as the stretching in the flow direction and compression in the gradient direction, shear-induced alignment, and shear viscosity. In the asymptotic limit of large activities and shear rates, the power-law scaling exponents of these quantities differ significantly from those of passive polymers. The chain's shear-induced stretching at a given shear rate is reduced by active noise, and it displays a non-monotonic behavior, where an initial polymer compression is followed by its stretching with increasing active force. △ Less

Submitted 20 November, 2023; originally announced November 2023.

Comments: 13 pages, 13 figures

Journal ref: Soft Matter, 2023, 19, 8577-8586

Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers

Authors: Vukasin Bozic, Danilo Dordevic, Daniele Coppola, Joseph Thommes, Sidak Pal Singh

Abstract: This work presents an analysis of the effectiveness of using standard shallow feed-forward networks to mimic the behavior of the attention mechanism in the original Transformer model, a state-of-the-art architecture for sequence-to-sequence tasks. We substitute key elements of the attention mechanism in the Transformer with simple feed-forward networks, trained using the original components via kn… ▽ More This work presents an analysis of the effectiveness of using standard shallow feed-forward networks to mimic the behavior of the attention mechanism in the original Transformer model, a state-of-the-art architecture for sequence-to-sequence tasks. We substitute key elements of the attention mechanism in the Transformer with simple feed-forward networks, trained using the original components via knowledge distillation. Our experiments, conducted on the IWSLT2017 dataset, reveal the capacity of these "attentionless Transformers" to rival the performance of the original architecture. Through rigorous ablation studies, and experimenting with various replacement network types and sizes, we offer insights that support the viability of our approach. This not only sheds light on the adaptability of shallow feed-forward networks in emulating attention mechanisms but also underscores their potential to streamline complex architectures for sequence-to-sequence tasks. △ Less

Submitted 4 February, 2024; v1 submitted 17 November, 2023; originally announced November 2023.

Comments: Accepted at AAAI24(https://aaai.org/aaai-conference/)

P wave mesons emitting weak decays of bottom mesons

Authors: Maninder Kaur, Supreet Pal Singh, R C Verma

Abstract: This paper is the extension of our previous work entitled Searching a systematics for nonfactorizable contributions to and hadronic decays. Obtaining the factorizable contributions from the spectator quark model for a systematics has been identified among the isospin reduced amplitudes for the nonfactorizable terms among decay modes. This systematics helps us to derive a generic formula which assi… ▽ More This paper is the extension of our previous work entitled Searching a systematics for nonfactorizable contributions to and hadronic decays. Obtaining the factorizable contributions from the spectator quark model for a systematics has been identified among the isospin reduced amplitudes for the nonfactorizable terms among decay modes. This systematics helps us to derive a generic formula which assists to predict the branching fractions for Inspired by this observation, we extend our analysis to p wave meson emitting decays of which have similar isospin structure and make predictions for where the experimental measurements are not yet available. △ Less

Submitted 5 December, 2023; v1 submitted 7 November, 2023; originally announced November 2023.

Comments: 23 pages, 8 Figures

Controlled dissipation for Rydberg atom experiments

Authors: Bleuenn Bégoc, Giovanni Cichelli, Sukhjit P. Singh, Francesco Perciavalle, Davide Rossini, Luigi Amico, Oliver Morsch

Abstract: We demonstrate a simple technique for adding controlled dissipation to Rydberg atom experiments. In our experiments we excite cold rubidium atoms in a magneto-optical trap to $70$-S Rydberg states whilst simultaneously inducing forced dissipation by resonantly coupling the Rydberg state to a hyperfine level of the short-lived $6$-P state. The resulting effective dissipation can be varied in streng… ▽ More We demonstrate a simple technique for adding controlled dissipation to Rydberg atom experiments. In our experiments we excite cold rubidium atoms in a magneto-optical trap to $70$-S Rydberg states whilst simultaneously inducing forced dissipation by resonantly coupling the Rydberg state to a hyperfine level of the short-lived $6$-P state. The resulting effective dissipation can be varied in strength and switched on and off during a single experimental cycle. △ Less

Submitted 1 November, 2023; v1 submitted 31 October, 2023; originally announced October 2023.

Comments: 3 pages, 3 figures

Strategy Revision Phase with Payoff Threshold in the Public Goods Game

Authors: Marco Alberto Javarone, Shaurya Pratap Singh

Abstract: Commonly, the strategy revision phase in evolutionary games relies on payoff comparison. Namely, agents compare their payoff with the opponent, assessing whether changing strategy can be potentially convenient. Even tiny payoff differences can be crucial in this decision process. In this work, we study the dynamics of cooperation in the Public Goods Game, introducing a threshold $ε$ in the strat… ▽ More Commonly, the strategy revision phase in evolutionary games relies on payoff comparison. Namely, agents compare their payoff with the opponent, assessing whether changing strategy can be potentially convenient. Even tiny payoff differences can be crucial in this decision process. In this work, we study the dynamics of cooperation in the Public Goods Game, introducing a threshold $ε$ in the strategy revision phase. In doing so, payoff differences narrower than $ε$ entail the decision process reduces to a coin flip. Interestingly, with ordinary agents, results show that payoff thresholds curb the emergence of cooperation. Yet, the latter can be sustained by these thresholds if the population is composed of conformist agents, which replace the random-based revision with selecting the strategy of the majority. To conclude, agents sensible only to consistent payoff differences may represent 'real-world' individuals unable to properly appreciate advantages or disadvantages when facing a dilemma. These agents may be detrimental to the emergence of cooperation or, on the contrary, supportive when endowed with a conformist attitude. △ Less

Submitted 23 October, 2023; originally announced October 2023.

Comments: 7 pages, 4 figures, 1 table

Transformer Fusion with Optimal Transport

Authors: Moritz Imfeld, Jacopo Graldi, Marco Giordano, Thomas Hofmann, Sotiris Anagnostidis, Sidak Pal Singh

Abstract: Fusion is a technique for merging multiple independently-trained neural networks in order to combine their capabilities. Past attempts have been restricted to the case of fully-connected, convolutional, and residual networks. This paper presents a systematic approach for fusing two or more transformer-based networks exploiting Optimal Transport to (soft-)align the various architectural components.… ▽ More Fusion is a technique for merging multiple independently-trained neural networks in order to combine their capabilities. Past attempts have been restricted to the case of fully-connected, convolutional, and residual networks. This paper presents a systematic approach for fusing two or more transformer-based networks exploiting Optimal Transport to (soft-)align the various architectural components. We flesh out an abstraction for layer alignment, that can generalize to arbitrary architectures - in principle - and we apply this to the key ingredients of Transformers such as multi-head self-attention, layer-normalization, and residual connections, and we discuss how to handle them via various ablation studies. Furthermore, our method allows the fusion of models of different sizes (heterogeneous fusion), providing a new and efficient way to compress Transformers. The proposed approach is evaluated on both image classification tasks via Vision Transformer and natural language modeling tasks using BERT. Our approach consistently outperforms vanilla fusion, and, after a surprisingly short finetuning, also outperforms the individual converged parent models. In our analysis, we uncover intriguing insights about the significant role of soft alignment in the case of Transformers. Our results showcase the potential of fusing multiple Transformers, thus compounding their expertise, in the budding paradigm of model fusion and recombination. Code is available at https://github.com/graldij/transformer-fusion. △ Less

Submitted 22 April, 2024; v1 submitted 9 October, 2023; originally announced October 2023.

Comments: Appears at International Conference on Learning Representations (ICLR), 2024. M. Imfeld, J. Graldi, and M. Giordano are the first authors and contributed equally to this work

Towards guarantees for parameter isolation in continual learning

Authors: Giulia Lanzillotta, Sidak Pal Singh, Benjamin F. Grewe, Thomas Hofmann

Abstract: Deep learning has proved to be a successful paradigm for solving many challenges in machine learning. However, deep neural networks fail when trained sequentially on multiple tasks, a shortcoming known as catastrophic forgetting in the continual learning literature. Despite a recent flourish of learning algorithms successfully addressing this problem, we find that provable guarantees against catas… ▽ More Deep learning has proved to be a successful paradigm for solving many challenges in machine learning. However, deep neural networks fail when trained sequentially on multiple tasks, a shortcoming known as catastrophic forgetting in the continual learning literature. Despite a recent flourish of learning algorithms successfully addressing this problem, we find that provable guarantees against catastrophic forgetting are lacking. In this work, we study the relationship between learning and forgetting by looking at the geometry of neural networks' loss landscape. We offer a unifying perspective on a family of continual learning algorithms, namely methods based on parameter isolation, and we establish guarantees on catastrophic forgetting for some of them. △ Less

Submitted 2 October, 2023; originally announced October 2023.

Comments: 10 pages, 3 figures

On the curvature of the loss landscape

Authors: Alison Pouplin, Hrittik Roy, Sidak Pal Singh, Georgios Arvanitidis

Abstract: One of the main challenges in modern deep learning is to understand why such over-parameterized models perform so well when trained on finite data. A way to analyze this generalization concept is through the properties of the associated loss landscape. In this work, we consider the loss landscape as an embedded Riemannian manifold and show that the differential geometric properties of the manifold… ▽ More One of the main challenges in modern deep learning is to understand why such over-parameterized models perform so well when trained on finite data. A way to analyze this generalization concept is through the properties of the associated loss landscape. In this work, we consider the loss landscape as an embedded Riemannian manifold and show that the differential geometric properties of the manifold can be used when analyzing the generalization abilities of a deep net. In particular, we focus on the scalar curvature, which can be computed analytically for our manifold, and show connections to several settings that potentially imply generalization. △ Less

Submitted 10 July, 2023; originally announced July 2023.

Comments: 12 pages, 5 figures, preliminary work

The Hessian perspective into the Nature of Convolutional Neural Networks

Authors: Sidak Pal Singh, Thomas Hofmann, Bernhard Schölkopf

Abstract: While Convolutional Neural Networks (CNNs) have long been investigated and applied, as well as theorized, we aim to provide a slightly different perspective into their nature -- through the perspective of their Hessian maps. The reason is that the loss Hessian captures the pairwise interaction of parameters and therefore forms a natural ground to probe how the architectural aspects of CNN get mani… ▽ More While Convolutional Neural Networks (CNNs) have long been investigated and applied, as well as theorized, we aim to provide a slightly different perspective into their nature -- through the perspective of their Hessian maps. The reason is that the loss Hessian captures the pairwise interaction of parameters and therefore forms a natural ground to probe how the architectural aspects of CNN get manifested in its structure and properties. We develop a framework relying on Toeplitz representation of CNNs, and then utilize it to reveal the Hessian structure and, in particular, its rank. We prove tight upper bounds (with linear activations), which closely follow the empirical trend of the Hessian rank and hold in practice in more general settings. Overall, our work generalizes and establishes the key insight that, even in CNNs, the Hessian rank grows as the square root of the number of parameters. △ Less

Submitted 15 May, 2023; originally announced May 2023.

Comments: ICML 2023 conference proceedings

OriCon3D: Effective 3D Object Detection using Orientation and Confidence

Authors: Dhyey Manish Rajani, Surya Pratap Singh, Rahul Kashyap Swayampakula

Abstract: In this paper, we propose an advanced methodology for the detection of 3D objects and precise estimation of their spatial positions from a single image. Unlike conventional frameworks that rely solely on center-point and dimension predictions, our research leverages a deep convolutional neural network-based 3D object weighted orientation regression paradigm. These estimates are then seamlessly int… ▽ More In this paper, we propose an advanced methodology for the detection of 3D objects and precise estimation of their spatial positions from a single image. Unlike conventional frameworks that rely solely on center-point and dimension predictions, our research leverages a deep convolutional neural network-based 3D object weighted orientation regression paradigm. These estimates are then seamlessly integrated with geometric constraints obtained from a 2D bounding box, resulting in derivation of a comprehensive 3D bounding box. Our novel network design encompasses two key outputs. The first output involves the estimation of 3D object orientation through the utilization of a discrete-continuous loss function. Simultaneously, the second output predicts objectivity-based confidence scores with minimal variance. Additionally, we also introduce enhancements to our methodology through the incorporation of lightweight residual feature extractors. By combining the derived estimates with the geometric constraints inherent in the 2D bounding box, our approach significantly improves the accuracy of 3D object pose determination, surpassing baseline methodologies. Our method is rigorously evaluated on the KITTI 3D object detection benchmark, demonstrating superior performance. △ Less

Submitted 3 January, 2024; v1 submitted 27 April, 2023; originally announced April 2023.

Twilight SLAM: Navigating Low-Light Environments

Authors: Surya Pratap Singh, Billy Mazotti, Dhyey Manish Rajani, Sarvesh Mayilvahanan, Guoyuan Li, Maani Ghaffari

Abstract: This paper presents a detailed examination of low-light visual Simultaneous Localization and Mapping (SLAM) pipelines, focusing on the integration of state-of-the-art (SOTA) low-light image enhancement algorithms with standard and contemporary SLAM frameworks. The primary objective of our work is to address a pivotal question: Does illuminating visual input significantly improve localization accur… ▽ More This paper presents a detailed examination of low-light visual Simultaneous Localization and Mapping (SLAM) pipelines, focusing on the integration of state-of-the-art (SOTA) low-light image enhancement algorithms with standard and contemporary SLAM frameworks. The primary objective of our work is to address a pivotal question: Does illuminating visual input significantly improve localization accuracy in both semi-dark and dark environments? In contrast to previous works that primarily address partially dim-lit datasets, we comprehensively evaluate various low-light SLAM pipelines across obscurely-lit environments. Employing a meticulous experimental approach, we qualitatively and quantitatively assess different combinations of image enhancers and SLAM frameworks, identifying the best-performing combinations for feature-based visual SLAM. The findings advance low-light SLAM by highlighting the practical implications of enhancing visual input for improved localization accuracy in challenging lighting conditions. This paper also offers valuable insights, encouraging further exploration of visual enhancement strategies for enhanced SLAM performance in real-world scenarios. △ Less

Submitted 24 December, 2023; v1 submitted 21 April, 2023; originally announced April 2023.

Leveraging Neo4j and deep learning for traffic congestion simulation & optimization

Authors: Shyam Pratap Singh, Arshad Ali Khan, Riad Souissi, Syed Adnan Yusuf

Abstract: Traffic congestion has been a major challenge in many urban road networks. Extensive research studies have been conducted to highlight traffic-related congestion and address the issue using data-driven approaches. Currently, most traffic congestion analyses are done using simulation software that offers limited insight due to the limitations in the tools and utilities being used to render various… ▽ More Traffic congestion has been a major challenge in many urban road networks. Extensive research studies have been conducted to highlight traffic-related congestion and address the issue using data-driven approaches. Currently, most traffic congestion analyses are done using simulation software that offers limited insight due to the limitations in the tools and utilities being used to render various traffic congestion scenarios. All that impacts the formulation of custom business problems which vary from place to place and country to country. By exploiting the power of the knowledge graph, we model a traffic congestion problem into the Neo4j graph and then use the load balancing, optimization algorithm to identify congestion-free road networks. We also show how traffic propagates backward in case of congestion or accident scenarios and its overall impact on other segments of the roads. We also train a sequential RNN-LSTM (Long Short-Term Memory) deep learning model on the real-time traffic data to assess the accuracy of simulation results based on a road-specific congestion. Our results show that graph-based traffic simulation, supplemented by AI ML-based traffic prediction can be more effective in estimating the congestion level in a road network. △ Less

Submitted 9 December, 2023; v1 submitted 31 March, 2023; originally announced April 2023.

Comments: The paper was rejected by a journal publisher and we have advanced the research so need to re-write and re-publish in light of reviewers' comments and revised scope of research

Accelerating cosmological models in $f(Q)$ gravity and the phase space analysis

Authors: S. A. Narawade, Shashank P. Singh, B. Mishra

Abstract: The dynamical aspect of accelerating cosmological model has been studied in this paper in the context of modified symmetric teleparallel gravity, the $f(Q)$ gravity. Initially, we have derived the dynamical parameters for two well known forms of $f(Q)$ such as: (i) log-square-root form and (ii) exponential form. The equation of state (EoS) parameter for the dark energy in the $f(Q)$ gravity in bot… ▽ More The dynamical aspect of accelerating cosmological model has been studied in this paper in the context of modified symmetric teleparallel gravity, the $f(Q)$ gravity. Initially, we have derived the dynamical parameters for two well known forms of $f(Q)$ such as: (i) log-square-root form and (ii) exponential form. The equation of state (EoS) parameter for the dark energy in the $f(Q)$ gravity in both the models emerges into a dynamical quantity. At present model-I shows the quintessence behavior and behave like the $Λ$CDM at the late time whereas model-II shows phantom behaviour. Further, the dynamical system analysis has been performed to determine the cosmological behaviour of the models along with its stability behaviour. For both the models the critical points are obtained and analysed the stability at each critical points with phase portraits. The evolutionary behaviour of density parameters for the matter-dominated, radiation-dominated, and dark energy phases are also shown for both the models. △ Less

Submitted 17 July, 2023; v1 submitted 11 March, 2023; originally announced March 2023.

Comments: 13 pages, 9 figures

Journal ref: Physics of the Dark Universe 42 (2023) 101282

Some Fundamental Aspects about Lipschitz Continuity of Neural Networks

Authors: Grigory Khromov, Sidak Pal Singh

Abstract: Lipschitz continuity is a crucial functional property of any predictive model, that naturally governs its robustness, generalisation, as well as adversarial vulnerability. Contrary to other works that focus on obtaining tighter bounds and developing different practical strategies to enforce certain Lipschitz properties, we aim to thoroughly examine and characterise the Lipschitz behaviour of Neura… ▽ More Lipschitz continuity is a crucial functional property of any predictive model, that naturally governs its robustness, generalisation, as well as adversarial vulnerability. Contrary to other works that focus on obtaining tighter bounds and developing different practical strategies to enforce certain Lipschitz properties, we aim to thoroughly examine and characterise the Lipschitz behaviour of Neural Networks. Thus, we carry out an empirical investigation in a range of different settings (namely, architectures, datasets, label noise, and more) by exhausting the limits of the simplest and the most general lower and upper bounds. As a highlight of this investigation, we showcase a remarkable fidelity of the lower Lipschitz bound, identify a striking Double Descent trend in both upper and lower bounds to the Lipschitz and explain the intriguing effects of label noise on function smoothness and generalisation. △ Less

Submitted 14 May, 2024; v1 submitted 21 February, 2023; originally announced February 2023.

Complex phase-fluctuation effects correlated with granularity in superconducting NbN nanofilms

Authors: Meenakshi Sharma, Manju Singh, Rajib K. Rakshit, Surinder P. Singh, Matteo Fretto, Natascia De Leo, Andrea Perali, Nicola Pinto

Abstract: Superconducting nanofilms are tunable systems that can host a 3D-2D dimensional crossover, leading to the Berezinskii-Kosterlitz-Thouless (BKT) superconducting transition approaching the 2D regime. Reducing further the dimensionality, from 2D to quasi-1D, superconducting nanostructures with disorder can generate quantum and thermal phase slips (PS) of the order parameter. Both BKT and PS are compl… ▽ More Superconducting nanofilms are tunable systems that can host a 3D-2D dimensional crossover, leading to the Berezinskii-Kosterlitz-Thouless (BKT) superconducting transition approaching the 2D regime. Reducing further the dimensionality, from 2D to quasi-1D, superconducting nanostructures with disorder can generate quantum and thermal phase slips (PS) of the order parameter. Both BKT and PS are complex phase fluctuation phenomena of difficult experimental detection. Here, we have characterized superconducting NbN nanofilms thinner than 15 nm, on different substrates, by temperature dependent resistivity and current-voltage (I-V) characteristics. Our measurements have evidenced clear features related to the emergence of BKT transition and PS events. The contemporary observation in the same system of BKT transition and PS events and their tunable evolution in temperature and thickness, has been explained as due to the nano-conducting paths forming in a granular NbN system. In one of the investigated samples we have been able to trace and characterize the continuous evolution in temperature from quantum to thermal PS. Our analysis has established that the detected complex phase phenomena are strongly related to the interplay between the typical size of the nano-conductive paths and the superconducting coherence length. △ Less

Submitted 16 November, 2022; originally announced November 2022.

Comments: 22 pages, 14 figures

Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning

Authors: Elias Frantar, Sidak Pal Singh, Dan Alistarh

Abstract: We consider the problem of model compression for deep neural networks (DNNs) in the challenging one-shot/post-training setting, in which we are given an accurate trained model, and must compress it without any retraining, based only on a small amount of calibration input data. This problem has become popular in view of the emerging software and hardware support for executing models compressed via… ▽ More We consider the problem of model compression for deep neural networks (DNNs) in the challenging one-shot/post-training setting, in which we are given an accurate trained model, and must compress it without any retraining, based only on a small amount of calibration input data. This problem has become popular in view of the emerging software and hardware support for executing models compressed via pruning and/or quantization with speedup, and well-performing solutions have been proposed independently for both compression approaches. In this paper, we introduce a new compression framework which covers both weight pruning and quantization in a unified setting, is time- and space-efficient, and considerably improves upon the practical performance of existing post-training methods. At the technical level, our approach is based on an exact and efficient realization of the classical Optimal Brain Surgeon (OBS) framework of [LeCun, Denker, and Solla, 1990] extended to also cover weight quantization at the scale of modern DNNs. From the practical perspective, our experimental results show that it can improve significantly upon the compression-accuracy trade-offs of existing post-training methods, and that it can enable the accurate compound application of both pruning and quantization in a post-training setting. △ Less

Submitted 8 January, 2023; v1 submitted 24 August, 2022; originally announced August 2022.

Comments: Published at NeurIPS 2022

Capacity Management in a Pandemic with Endogenous Patient Choices and Flows

Authors: Sanyukta Deshpande, Lavanya Marla, Alan Scheller-Wolf, Siddharth Prakash Singh

Abstract: Motivated by the experiences of a healthcare service provider during the Covid-19 pandemic, we aim to study the decisions of a provider that operates both an Emergency Department (ED) and a medical Clinic. Patients contact the provider through a phone call or may present directly at the ED: patients can be COVID (suspected/confirmed) or non-COVID, and have different severities. Depending on the se… ▽ More Motivated by the experiences of a healthcare service provider during the Covid-19 pandemic, we aim to study the decisions of a provider that operates both an Emergency Department (ED) and a medical Clinic. Patients contact the provider through a phone call or may present directly at the ED: patients can be COVID (suspected/confirmed) or non-COVID, and have different severities. Depending on the severity, patients who contact the provider may be directed to the ED (to be seen in a few hours), be offered an appointment at the Clinic (to be seen in a few days), or be treated via phone or telemedicine, avoiding a visit to a facility. All patients make joining decisions based on comparing their own risk perceptions versus their anticipated benefits: They then choose to enter a facility only if it is beneficial enough. Also, after initial contact, their severities may evolve, which may change their decision. The hospital system's objective is to allocate service capacity across facilities so as to minimize costs from patient deaths or defections. We model the system using a fluid approximation over multiple periods, possibly with different demand profiles. While the feasible space for this problem can be extremely complex, it is amenable to decomposition into different sub-regions that can be analyzed individually, the global optimal solution can be reached via provably parsimonious computational methods over a single period and over multiple periods with different demand rates. Our analytical and computational results indicate that endogeneity results in non-trivial and non-intuitive capacity allocations that do not always prioritize high severity patients, for both single and multi-period settings. △ Less

Submitted 11 July, 2022; originally announced July 2022.

Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse

Authors: Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, Aurelien Lucchi

Abstract: Transformers have achieved remarkable success in several domains, ranging from natural language processing to computer vision. Nevertheless, it has been recently shown that stacking self-attention layers - the distinctive architectural component of Transformers - can result in rank collapse of the tokens' representations at initialization. The question of if and how rank collapse affects training… ▽ More Transformers have achieved remarkable success in several domains, ranging from natural language processing to computer vision. Nevertheless, it has been recently shown that stacking self-attention layers - the distinctive architectural component of Transformers - can result in rank collapse of the tokens' representations at initialization. The question of if and how rank collapse affects training is still largely unanswered, and its investigation is necessary for a more comprehensive understanding of this architecture. In this work, we shed new light on the causes and the effects of this phenomenon. First, we show that rank collapse of the tokens' representations hinders training by causing the gradients of the queries and keys to vanish at initialization. Furthermore, we provide a thorough description of the origin of rank collapse and discuss how to prevent it via an appropriate depth-dependent scaling of the residual branches. Finally, our analysis unveils that specific architectural hyperparameters affect the gradients of queries and values differently, leading to disproportionate gradient norms. This suggests an explanation for the widespread use of adaptive methods for Transformers' optimization. △ Less

Submitted 7 June, 2022; originally announced June 2022.

Compression of a confined semiflexible polymer under direct and oscillating fields

Authors: Keerthi Radhakrishnan, Sunil P. Singh

Abstract: The folding transition of biopolymers from the coil to compact structures has attracted wide research interest in the past and is well studied in polymer physics. Recent seminal works on DNA in confined devices have shown that these long biopolymers tend to collapse under an external field, contrary to the previously reported stretching. These long folded structures have a tendency to form knots t… ▽ More The folding transition of biopolymers from the coil to compact structures has attracted wide research interest in the past and is well studied in polymer physics. Recent seminal works on DNA in confined devices have shown that these long biopolymers tend to collapse under an external field, contrary to the previously reported stretching. These long folded structures have a tendency to form knots that has profound implications in gene regulation and various other biological functions. These knots have been mechanically induced via optical tweezers, nanochannel confinement, etc., until recently, where uniform field driven compression lead to self entanglement of DNA. In this work, we capture the compression of a confined semiflexible polymer under direct and oscillating fields, using a coarse-grained computer simulation model in the presence of long-range hydrodynamics. Within this framework, we show that subjected to direct field, chains in stronger confinements exhibit substantial compaction, contrary to the one in moderate confinements or bulk, where such compaction is absent. Interestingly, an alternating field within an optimum frequency can effectuate this compression even in moderate or no confinement. Additionally, we show that the bending rigidity has a profound influence on the chains folding favourability under direct and alternating fields. This field induced collapse is a quintessential hydrodynamic phenomenon, resulting in intertwined knotted structures, even for shorter chains, unlike DNA knotting experiments, where it happens exclusively for longer chains. △ Less

Submitted 11 April, 2022; originally announced April 2022.

Phenomenology of Double Descent in Finite-Width Neural Networks

Authors: Sidak Pal Singh, Aurelien Lucchi, Thomas Hofmann, Bernhard Schölkopf

Abstract: `Double descent' delineates the generalization behaviour of models depending on the regime they belong to: under- or over-parameterized. The current theoretical understanding behind the occurrence of this phenomenon is primarily based on linear and kernel regression models -- with informal parallels to neural networks via the Neural Tangent Kernel. Therefore such analyses do not adequately capture… ▽ More `Double descent' delineates the generalization behaviour of models depending on the regime they belong to: under- or over-parameterized. The current theoretical understanding behind the occurrence of this phenomenon is primarily based on linear and kernel regression models -- with informal parallels to neural networks via the Neural Tangent Kernel. Therefore such analyses do not adequately capture the mechanisms behind double descent in finite-width neural networks, as well as, disregard crucial components -- such as the choice of the loss function. We address these shortcomings by leveraging influence functions in order to derive suitable expressions of the population loss and its lower bound, while imposing minimal assumptions on the form of the parametric model. Our derived bounds bear an intimate connection with the spectrum of the Hessian at the optimum, and importantly, exhibit a double descent behaviour at the interpolation threshold. Building on our analysis, we further investigate how the loss function affects double descent -- and thus uncover interesting properties of neural networks and their Hessian spectra near the interpolation threshold. △ Less

Submitted 14 March, 2022; originally announced March 2022.

Comments: Published at ICLR 2022

A Deep Learning Approach for the Detection of COVID-19 from Chest X-Ray Images using Convolutional Neural Networks

Authors: Aditya Saxena, Shamsheer Pal Singh

Abstract: The COVID-19 (coronavirus) is an ongoing pandemic caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). The virus was first identified in mid-December 2019 in the Hubei province of Wuhan, China and by now has spread throughout the planet with more than 75.5 million confirmed cases and more than 1.67 million deaths. With limited number of COVID-19 test kits available in medical fa… ▽ More The COVID-19 (coronavirus) is an ongoing pandemic caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). The virus was first identified in mid-December 2019 in the Hubei province of Wuhan, China and by now has spread throughout the planet with more than 75.5 million confirmed cases and more than 1.67 million deaths. With limited number of COVID-19 test kits available in medical facilities, it is important to develop and implement an automatic detection system as an alternative diagnosis option for COVID-19 detection that can used on a commercial scale. Chest X-ray is the first imaging technique that plays an important role in the diagnosis of COVID-19 disease. Computer vision and deep learning techniques can help in determining COVID-19 virus with Chest X-ray Images. Due to the high availability of large-scale annotated image datasets, great success has been achieved using convolutional neural network for image analysis and classification. In this research, we have proposed a deep convolutional neural network trained on five open access datasets with binary output: Normal and Covid. The performance of the model is compared with four pre-trained convolutional neural network-based models (COVID-Net, ResNet18, ResNet and MobileNet-V2) and it has been seen that the proposed model provides better accuracy on the validation set as compared to the other four pre-trained models. This research work provides promising results which can be further improvise and implement on a commercial scale. △ Less

Submitted 24 January, 2022; originally announced January 2022.

Soft Actor-Critic with Cross-Entropy Policy Optimization

Authors: Zhenyang Shi, Surya P. N. Singh

Abstract: Soft Actor-Critic (SAC) is one of the state-of-the-art off-policy reinforcement learning (RL) algorithms that is within the maximum entropy based RL framework. SAC is demonstrated to perform very well in a list of continous control tasks with good stability and robustness. SAC learns a stochastic Gaussian policy that can maximize a trade-off between total expected reward and the policy entropy. To… ▽ More Soft Actor-Critic (SAC) is one of the state-of-the-art off-policy reinforcement learning (RL) algorithms that is within the maximum entropy based RL framework. SAC is demonstrated to perform very well in a list of continous control tasks with good stability and robustness. SAC learns a stochastic Gaussian policy that can maximize a trade-off between total expected reward and the policy entropy. To update the policy, SAC minimizes the KL-Divergence between the current policy density and the soft value function density. Reparameterization trick is then used to obtain the approximate gradient of this divergence. In this paper, we propose Soft Actor-Critic with Cross-Entropy Policy Optimization (SAC-CEPO), which uses Cross-Entropy Method (CEM) to optimize the policy network of SAC. The initial idea is to use CEM to iteratively sample the closest distribution towards the soft value function density and uses the resultant distribution as a target to update the policy network. For the purpose of reducing the computational complexity, we also introduce a decoupled policy structure that decouples the Gaussian policy into one policy that learns the mean and one other policy that learns the deviation such that only the mean policy is trained by CEM. We show that this decoupled policy structure does converge to a optimal and we also demonstrate by experiments that SAC-CEPO achieves competitive performance against the original SAC. △ Less

Submitted 21 December, 2021; originally announced December 2021.

The explicit characterization of counterion dynamics around a flexible polyelectrolyte

Authors: Keerthi Radhakrishnan, Sunil P. Singh

Abstract: The article presents a comprehensive study of counterion dynamics around a generic linear polyelectrolyte (PE) chain with the help of coarse-grained computer simulations. The ion-chain coupling is discussed in the form of binding time, mean-square-displacement (MSD) relative to the chain, local ion transport coefficient, and spatio-temporal correlations in the effective charge. We have shown that… ▽ More The article presents a comprehensive study of counterion dynamics around a generic linear polyelectrolyte (PE) chain with the help of coarse-grained computer simulations. The ion-chain coupling is discussed in the form of binding time, mean-square-displacement (MSD) relative to the chain, local ion transport coefficient, and spatio-temporal correlations in the effective charge. We have shown that a counterion exhibits sub-diffusive behavior $\langle δR^2 \rangle \sim t^δ$, $δ\approx0.9$ w.r.t. chain's centre of mass (COM). The MSD of ions perpendicularly outwards from the chain segment exhibits a smaller sub-diffusive exponent compared to the one relative to the chain's COM. Further, we confirm that the effective diffusion-coefficient of counterions is strongly coupled with the chain. The effective diffusivity of ion is the lowest in chain's close proximity, extending up to length-scale of the radius of gyration Rg. Beyond Rg at larger distances, they attain diffusivity of free ion with a smooth cross-over from the adsorbed regime to the free ion regime. We have shown that the effective diffusivity drastically decreases for the higher valent ions, while the crossover length scale remains the same. Conversely, with increasing salt concentration the coupling-length scale reduces, while the diffusivity remains unaltered. The effective diffusivity of adsorbed-ion reveals an exponential reduction with electrostatic interaction strength. We further corroborate this from the binding time of ions on the chain, which also grows exponentially with the coupling strength of the ion-polymer duo. Moreover, the binding time of ions exhibits a weak dependence with salt concentration for the monovalent salt, while for higher valent salts the binding time decreases dramatically with concentration. △ Less

Submitted 16 December, 2021; originally announced December 2021.

Comments: 14 pages, 12 figures, SI material

The CAT SET on the MAT: Cross Attention for Set Matching in Bipartite Hypergraphs

Authors: Govind Sharma, Swyam Prakash Singh, V. Susheela Devi, M. Narasimha Murty

Abstract: Usual relations between entities could be captured using graphs; but those of a higher-order -- more so between two different types of entities (which we term "left" and "right") -- calls for a "bipartite hypergraph". For example, given a left set of symptoms and right set of diseases, the relation between a set subset of symptoms (that a patient experiences at a given point of time) and a subset… ▽ More Usual relations between entities could be captured using graphs; but those of a higher-order -- more so between two different types of entities (which we term "left" and "right") -- calls for a "bipartite hypergraph". For example, given a left set of symptoms and right set of diseases, the relation between a set subset of symptoms (that a patient experiences at a given point of time) and a subset of diseases (that he/she might be diagnosed with) could be well-represented using a bipartite hyperedge. The state-of-the-art in embedding nodes of a hypergraph is based on learning the self-attention structure between node-pairs from a hyperedge. In the present work, given a bipartite hypergraph, we aim at capturing relations between node pairs from the cross-product between the left and right hyperedges, and term it a "cross-attention" (CAT) based model. More precisely, we pose "bipartite hyperedge link prediction" as a set-matching (SETMAT) problem and propose a novel neural network architecture called CATSETMAT for the same. We perform extensive experiments on multiple bipartite hypergraph datasets to show the superior performance of CATSETMAT, which we compare with multiple techniques from the state-of-the-art. Our results also elucidate information flow in self- and cross-attention scenarios. △ Less

Submitted 30 October, 2021; originally announced November 2021.

Comments: 18 pages, 9 figures, under review

Searching a systematics for nonfactorizable contribution to B-and B0 mesons

Authors: Maninder Kaur, Supreet Pal Singh, R. C. Verma

Abstract: Two-body weak decays / and are examined under isospin analysis to study nonfactorizable contributions. After extracting the strong phases and obtaining the factorizable contributions from spectator-quark diagrams for Nc=3, we determine nonfactorizable isospin amplitudes from the experimental data for these modes. Our results support the universality of ratio of nonfactorizable isospin reduced ampl… ▽ More Two-body weak decays / and are examined under isospin analysis to study nonfactorizable contributions. After extracting the strong phases and obtaining the factorizable contributions from spectator-quark diagrams for Nc=3, we determine nonfactorizable isospin amplitudes from the experimental data for these modes. Our results support the universality of ratio of nonfactorizable isospin reduced amplitudes for these decays within experimental errors. In order to show that this systematics is not coincidental, we also plot our results w. r. t. this ratio. △ Less

Submitted 9 October, 2021; v1 submitted 6 August, 2021; originally announced August 2021.

Journal ref: Chinese Physics C, Vol. 46, No. 7 (2022) 073105

Analytic Insights into Structure and Rank of Neural Network Hessian Maps

Authors: Sidak Pal Singh, Gregor Bachmann, Thomas Hofmann

Abstract: The Hessian of a neural network captures parameter interactions through second-order derivatives of the loss. It is a fundamental object of study, closely tied to various problems in deep learning, including model design, optimization, and generalization. Most prior work has been empirical, typically focusing on low-rank approximations and heuristics that are blind to the network structure. In con… ▽ More The Hessian of a neural network captures parameter interactions through second-order derivatives of the loss. It is a fundamental object of study, closely tied to various problems in deep learning, including model design, optimization, and generalization. Most prior work has been empirical, typically focusing on low-rank approximations and heuristics that are blind to the network structure. In contrast, we develop theoretical tools to analyze the range of the Hessian map, providing us with a precise understanding of its rank deficiency as well as the structural reasons behind it. This yields exact formulas and tight upper bounds for the Hessian rank of deep linear networks, allowing for an elegant interpretation in terms of rank deficiency. Moreover, we demonstrate that our bounds remain faithful as an estimate of the numerical Hessian rank, for a larger class of models such as rectified and hyperbolic tangent networks. Further, we also investigate the implications of model architecture (e.g.~width, depth, bias) on the rank deficiency. Overall, our work provides novel insights into the source and extent of redundancy in overparameterized networks. △ Less

Submitted 1 July, 2021; v1 submitted 30 June, 2021; originally announced June 2021.

LiMIIRL: Lightweight Multiple-Intent Inverse Reinforcement Learning

Authors: Aaron J. Snoswell, Surya P. N. Singh, Nan Ye

Abstract: Multiple-Intent Inverse Reinforcement Learning (MI-IRL) seeks to find a reward function ensemble to rationalize demonstrations of different but unlabelled intents. Within the popular expectation maximization (EM) framework for learning probabilistic MI-IRL models, we present a warm-start strategy based on up-front clustering of the demonstrations in feature space. Our theoretical analysis shows th… ▽ More Multiple-Intent Inverse Reinforcement Learning (MI-IRL) seeks to find a reward function ensemble to rationalize demonstrations of different but unlabelled intents. Within the popular expectation maximization (EM) framework for learning probabilistic MI-IRL models, we present a warm-start strategy based on up-front clustering of the demonstrations in feature space. Our theoretical analysis shows that this warm-start solution produces a near-optimal reward ensemble, provided the behavior modes satisfy mild separation conditions. We also propose a MI-IRL performance metric that generalizes the popular Expected Value Difference measure to directly assesses learned rewards against the ground-truth reward ensemble. Our metric elegantly addresses the difficulty of pairing up learned and ground truth rewards via a min-cost flow formulation, and is efficiently computable. We also develop a MI-IRL benchmark problem that allows for more comprehensive algorithmic evaluations. On this problem, we find our MI-IRL warm-start strategy helps avoid poor quality local minima reward ensembles, resulting in a significant improvement in behavior clustering. Our extensive sensitivity analysis demonstrates that the quality of the learned reward ensembles is improved under various settings, including cases where our theoretical assumptions do not necessarily hold. Finally, we demonstrate the effectiveness of our methods by discovering distinct driving styles in a large real-world dataset of driver GPS trajectories. △ Less

Submitted 3 June, 2021; originally announced June 2021.

Comments: Under review for NeurIPS 2021

Role of viscoelasticity on the dynamics and aggregation of chemically active sphere-dimers

Authors: Soudamini Sahoo, Sunil Pratap Singh, Snigdha Thakur

Abstract: The impact of complex media on the dynamics of active swimmers has gained a thriving interest in the research community for their prominent applications in various fields. This paper investigates the effect of viscoelasticity on the dynamics and aggregation of chemically powered sphere-dimers by using a coarse-grained hybrid mesoscopic simulation technique. The sphere-dimers perform active motion… ▽ More The impact of complex media on the dynamics of active swimmers has gained a thriving interest in the research community for their prominent applications in various fields. This paper investigates the effect of viscoelasticity on the dynamics and aggregation of chemically powered sphere-dimers by using a coarse-grained hybrid mesoscopic simulation technique. The sphere-dimers perform active motion by virtue of the concentration gradient around the swimmer's surface, produced by the chemical reaction at one end of the dimer. We observe that the fluid elasticity enhances translational and rotational motion of a single dimer, however for a pair of dimers, the clustering in a particular alignment is more pronounced. In case of multiple dimers, the kinetics of cluster formation along with their propulsive nature are presented in detail. The key factors influencing the enhanced motility and the aggregation of dimers are the concentration gradients, hydrodynamic coupling and the microstructures present in the system. △ Less

Submitted 4 February, 2021; originally announced February 2021.

Comments: 11 pages, 14 figures

Journal ref: Physics of Fluids, 33, 017120 (2021)

Detection and Prediction of Infectious Diseases Using IoT Sensors: A Review

Authors: Mohammad Meraj, Surendra Pal Singh, Prashant Johri, Mohammad Tabrez Quasim

Abstract: An infectious kind of disease affects a huge number of human beings. A lot of investigation being conducted throughout the world. There are many interactive hardware platform packages like IoT in healthcare including smart tracking, smart sensors, and clinical device integration available in the market. Emerging technology like IoT has a notable ability to hold patients secure and healthful and al… ▽ More An infectious kind of disease affects a huge number of human beings. A lot of investigation being conducted throughout the world. There are many interactive hardware platform packages like IoT in healthcare including smart tracking, smart sensors, and clinical device integration available in the market. Emerging technology like IoT has a notable ability to hold patients secure and healthful and also enhance how physicians supply care. Healthcare IoT also can bolster affected person pride by permitting patients to spend more time interacting with their medical doctors due to the fact docs aren't as taken with the mundane and rote aspects of their career. The most considerable advantage to IoT in healthcare is that it supports doctors in undertaking extra significant clinical work in a profession that already is experiencing a worldwide professional hard work shortage. This paper investigates the basis exploration of the applicability of IoT in the healthcare System. △ Less

Submitted 2 January, 2021; originally announced January 2021.

Comments: 7 pages, 2figures

Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms

Authors: Aaron J. Snoswell, Surya P. N. Singh, Nan Ye

Abstract: We provide new perspectives and inference algorithms for Maximum Entropy (MaxEnt) Inverse Reinforcement Learning (IRL), which provides a principled method to find a most non-committal reward function consistent with given expert demonstrations, among many consistent reward functions. We first present a generalized MaxEnt formulation based on minimizing a KL-divergence instead of maximizing an en… ▽ More We provide new perspectives and inference algorithms for Maximum Entropy (MaxEnt) Inverse Reinforcement Learning (IRL), which provides a principled method to find a most non-committal reward function consistent with given expert demonstrations, among many consistent reward functions. We first present a generalized MaxEnt formulation based on minimizing a KL-divergence instead of maximizing an entropy. This improves the previous heuristic derivation of the MaxEnt IRL model (for stochastic MDPs), allows a unified view of MaxEnt IRL and Relative Entropy IRL, and leads to a model-free learning algorithm for the MaxEnt IRL model. Second, a careful review of existing inference algorithms and implementations showed that they approximately compute the marginals required for learning the model. We provide examples to illustrate this, and present an efficient and exact inference algorithm. Our algorithm can handle variable length demonstrations; in addition, while a basic version takes time quadratic in the maximum demonstration length L, an improved version of this algorithm reduces this to linear using a padding trick. Experiments show that our exact algorithm improves reward learning as compared to the approximate ones. Furthermore, our algorithm scales up to a large, real-world dataset involving driver behaviour forecasting. We provide an optimized implementation compatible with the OpenAI Gym interface. Our new insight and algorithms could possibly lead to further interest and exploration of the original MaxEnt IRL model. △ Less

Submitted 4 June, 2021; v1 submitted 1 December, 2020; originally announced December 2020.

Comments: Published as a conference paper at the 2020 IEEE Symposium Series on Computational Intelligence (SSCI)

AutoKnow: Self-Driving Knowledge Collection for Products of Thousands of Types

Authors: Xin Luna Dong, Xiang He, Andrey Kan, Xian Li, Yan Liang, Jun Ma, Yifan Ethan Xu, Chenwei Zhang, Tong Zhao, Gabriel Blanco Saldana, Saurabh Deshpande, Alexandre Michetti Manduca, Jay Ren, Surender Pal Singh, Fan Xiao, Haw-Shiuan Chang, Giannis Karamanolakis, Yuning Mao, Yaqing Wang, Christos Faloutsos, Andrew McCallum, Jiawei Han

Abstract: Can one build a knowledge graph (KG) for all products in the world? Knowledge graphs have firmly established themselves as valuable sources of information for search and question answering, and it is natural to wonder if a KG can contain information about products offered at online retail sites. There have been several successful examples of generic KGs, but organizing information about products p… ▽ More Can one build a knowledge graph (KG) for all products in the world? Knowledge graphs have firmly established themselves as valuable sources of information for search and question answering, and it is natural to wonder if a KG can contain information about products offered at online retail sites. There have been several successful examples of generic KGs, but organizing information about products poses many additional challenges, including sparsity and noise of structured data for products, complexity of the domain with millions of product types and thousands of attributes, heterogeneity across large number of categories, as well as large and constantly growing number of products. We describe AutoKnow, our automatic (self-driving) system that addresses these challenges. The system includes a suite of novel techniques for taxonomy construction, product property identification, knowledge extraction, anomaly detection, and synonym discovery. AutoKnow is (a) automatic, requiring little human intervention, (b) multi-scalable, scalable in multiple dimensions (many domains, many products, and many attributes), and (c) integrative, exploiting rich customer behavior logs. AutoKnow has been operational in collecting product knowledge for over 11K product types. △ Less

Submitted 24 June, 2020; originally announced June 2020.

Comments: KDD 2020

Deep ConvLSTM with self-attention for human activity decoding using wearables

Authors: Satya P. Singh, Aimé Lay-Ekuakille, Deepak Gangwar, Madan Kumar Sharma, Sukrit Gupta

Abstract: Decoding human activity accurately from wearable sensors can aid in applications related to healthcare and context awareness. The present approaches in this domain use recurrent and/or convolutional models to capture the spatio-temporal features from time-series data from multiple sensors. We propose a deep neural network architecture that not only captures the spatio-temporal features of multiple… ▽ More Decoding human activity accurately from wearable sensors can aid in applications related to healthcare and context awareness. The present approaches in this domain use recurrent and/or convolutional models to capture the spatio-temporal features from time-series data from multiple sensors. We propose a deep neural network architecture that not only captures the spatio-temporal features of multiple sensor time-series data but also selects, learns important time points by utilizing a self-attention mechanism. We show the validity of the proposed approach across different data sampling strategies on six public datasets and demonstrate that the self-attention mechanism gave a significant improvement in performance over deep networks using a combination of recurrent and convolution networks. We also show that the proposed approach gave a statistically significant performance enhancement over previous state-of-the-art methods for the tested datasets. The proposed methods open avenues for better decoding of human activity from multiple body sensors over extended periods of time. The code implementation for the proposed model is available at https://github.com/isukrit/encodingHumanActivity. △ Less

Submitted 17 December, 2020; v1 submitted 2 May, 2020; originally announced May 2020.

Comments: 8 pages, 2 figures, 3 tables. IEEE Sensors Journal, 2020

WoodFisher: Efficient Second-Order Approximation for Neural Network Compression

Authors: Sidak Pal Singh, Dan Alistarh

Abstract: Second-order information, in the form of Hessian- or Inverse-Hessian-vector products, is a fundamental tool for solving optimization problems. Recently, there has been significant interest in utilizing this information in the context of deep neural networks; however, relatively little is known about the quality of existing approximations in this context. Our work examines this question, identifies… ▽ More Second-order information, in the form of Hessian- or Inverse-Hessian-vector products, is a fundamental tool for solving optimization problems. Recently, there has been significant interest in utilizing this information in the context of deep neural networks; however, relatively little is known about the quality of existing approximations in this context. Our work examines this question, identifies issues with existing approaches, and proposes a method called WoodFisher to compute a faithful and efficient estimate of the inverse Hessian. Our main application is to neural network compression, where we build on the classic Optimal Brain Damage/Surgeon framework. We demonstrate that WoodFisher significantly outperforms popular state-of-the-art methods for one-shot pruning. Further, even when iterative, gradual pruning is considered, our method results in a gain in test accuracy over the state-of-the-art approaches, for pruning popular neural networks (like ResNet-50, MobileNetV1) trained on standard image classification datasets such as ImageNet ILSVRC. We examine how our method can be extended to take into account first-order information, as well as illustrate its ability to automatically set layer-wise pruning thresholds and perform compression in the limited-data regime. The code is available at the following link, https://github.com/IST-DASLab/WoodFisher. △ Less

Submitted 25 November, 2020; v1 submitted 29 April, 2020; originally announced April 2020.

Comments: NeurIPS 2020

Conformation and dynamics of a self-avoiding active flexible polymer

Authors: Shalabh K. Anand, Sunil P. Singh

Abstract: We investigate conformations and dynamics of a polymer considering its monomers to be active Brownian particles. This active polymer shows very intriguing physical behavior which is absent in an active Rouse chain. The chain initially shrinks with active force, which starts swelling on further increase in force. The shrinkage followed by swelling is attributed purely to excluded-volume interaction… ▽ More We investigate conformations and dynamics of a polymer considering its monomers to be active Brownian particles. This active polymer shows very intriguing physical behavior which is absent in an active Rouse chain. The chain initially shrinks with active force, which starts swelling on further increase in force. The shrinkage followed by swelling is attributed purely to excluded-volume interactions among the monomers. In the swelling regime, chain shows a cross-over from the self-avoiding behavior to Rouse-behavior with scaling exponent $ν_a \simeq 1/2$ for end-to-end distance. The non-monotonicity in the structure is analysed through various physical quantities specifically, radial distribution function of monomers, scattering time, as well as various energy calculations. The chain relaxes faster than the Rouse chain in the intermediate force regime, with a cross-over in variation of relaxation time at large active force as given by a power-law $τ_r \sim Pe^{-4/3}$ (P e is Péclet number). △ Less

Submitted 10 April, 2020; v1 submitted 9 April, 2020; originally announced April 2020.

Journal ref: Physical Review E 101, 030501(R) (2020)

A phase separation of active colloidal suspension via Quorum-sensing

Authors: Francis Jose, Shalabh K. Anand, Sunil P. Singh

Abstract: We present the Brownian dynamics simulation of active colloidal suspension in two dimensions, where the self-propulsion speed of a colloid is regulated according to the local density sensed by it. The role of concentration-dependent motility on the phase-separation of colloids and their dynamics is investigated in detail. Interestingly, the system phase separates at a very low packing fraction (… ▽ More We present the Brownian dynamics simulation of active colloidal suspension in two dimensions, where the self-propulsion speed of a colloid is regulated according to the local density sensed by it. The role of concentration-dependent motility on the phase-separation of colloids and their dynamics is investigated in detail. Interestingly, the system phase separates at a very low packing fraction ($Φ\approx 0.125$) at higher self-propulsion speeds ($\text{Pe}$), which coexists with a homogeneous phase and attains long-range crystalline order beyond a transition point. The transition point is quantified here from the local density profiles, local and global-bond order parameters. We have shown that the phase diagram's characteristics are qualitatively akin to the active Brownian particle (ABP) model. Moreover, our investigation reveals that the density-dependent motility amplifies the slow-down of the directed speed, which facilitates phase-separation even at low packing fractions. The effective diffusivity shows a crossover from quadratic rise to a power-law behavior of exponent $3/2$ with $\text{Pe}$ in the phase-separated regime. Furthermore, we have shown that the effective diffusion decreases exponentially with packing fraction in the phase-separated regime while linear decrease in the single phase regime. △ Less

Submitted 28 January, 2021; v1 submitted 4 April, 2020; originally announced April 2020.

3D Deep Learning on Medical Images: A Review

Authors: Satya P. Singh, Lipo Wang, Sukrit Gupta, Haveesh Goli, Parasuraman Padmanabhan, Balázs Gulyás

Abstract: The rapid advancements in machine learning, graphics processing technologies and the availability of medical imaging data have led to a rapid increase in the use of deep learning models in the medical domain. This was exacerbated by the rapid advancements in convolutional neural network (CNN) based architectures, which were adopted by the medical imaging community to assist clinicians in disease d… ▽ More The rapid advancements in machine learning, graphics processing technologies and the availability of medical imaging data have led to a rapid increase in the use of deep learning models in the medical domain. This was exacerbated by the rapid advancements in convolutional neural network (CNN) based architectures, which were adopted by the medical imaging community to assist clinicians in disease diagnosis. Since the grand success of AlexNet in 2012, CNNs have been increasingly used in medical image analysis to improve the efficiency of human clinicians. In recent years, three-dimensional (3D) CNNs have been employed for the analysis of medical images. In this paper, we trace the history of how the 3D CNN was developed from its machine learning roots, we provide a brief mathematical description of 3D CNN and provide the preprocessing steps required for medical images before feeding them to 3D CNNs. We review the significant research in the field of 3D medical imaging analysis using 3D CNNs (and its variants) in different medical areas such as classification, segmentation, detection and localization. We conclude by discussing the challenges associated with the use of 3D CNNs in the medical imaging domain (and the use of deep learning models in general) and possible future trends in the field. △ Less

Submitted 13 October, 2020; v1 submitted 31 March, 2020; originally announced April 2020.

Comments: Published in Sensors Journal (https://www.mdpi.com/1424-8220/20/18/5097)

Journal ref: Sensors 2020, 20, 5097

Model Fusion via Optimal Transport

Authors: Sidak Pal Singh, Martin Jaggi

Abstract: Combining different models is a widely used paradigm in machine learning applications. While the most common approach is to form an ensemble of models and average their individual predictions, this approach is often rendered infeasible by given resource constraints in terms of memory and computation, which grow linearly with the number of models. We present a layer-wise model fusion algorithm for… ▽ More Combining different models is a widely used paradigm in machine learning applications. While the most common approach is to form an ensemble of models and average their individual predictions, this approach is often rendered infeasible by given resource constraints in terms of memory and computation, which grow linearly with the number of models. We present a layer-wise model fusion algorithm for neural networks that utilizes optimal transport to (soft-) align neurons across the models before averaging their associated parameters. We show that this can successfully yield "one-shot" knowledge transfer (i.e, without requiring any retraining) between neural networks trained on heterogeneous non-i.i.d. data. In both i.i.d. and non-i.i.d. settings , we illustrate that our approach significantly outperforms vanilla averaging, as well as how it can serve as an efficient replacement for the ensemble with moderate fine-tuning, for standard convolutional networks (like VGG11), residual networks (like ResNet18), and multi-layer perceptrons on CIFAR10, CIFAR100, and MNIST. Finally, our approach also provides a principled way to combine the parameters of neural networks with different widths, and we explore its application for model compression. The code is available at the following link, https://github.com/sidak/otfusion. △ Less

Submitted 16 May, 2023; v1 submitted 12 October, 2019; originally announced October 2019.

Comments: NeurIPS 2020 conference proceedings (early version featured in the Optimal Transport & Machine Learning workshop, NeurIPS 2019)