Statistics > Machine Learning
[Submitted on 29 Jul 2019 (v1), last revised 16 Jun 2021 (this version, v10)]
Title:LassoNet: A Neural Network with Feature Sparsity
View PDFAbstract:Much work has been done recently to make neural networks more interpretable, and one obvious approach is to arrange for the network to use only a subset of the available features. In linear models, Lasso (or $\ell_1$-regularized) regression assigns zero weights to the most irrelevant or redundant features, and is widely used in data science. However the Lasso only applies to linear models. Here we introduce LassoNet, a neural network framework with global feature selection. Our approach enforces a hierarchy: specifically a feature can participate in a hidden unit only if its linear representative is active. Unlike other approaches to feature selection for neural nets, our method uses a modified objective function with constraints, and so integrates feature selection with the parameter learning directly. As a result, it delivers an entire regularization path of solutions with a range of feature sparsity. On systematic experiments, LassoNet significantly outperforms state-of-the-art methods for feature selection and regression. The LassoNet method uses projected proximal gradient descent, and generalizes directly to deep networks. It can be implemented by adding just a few lines of code to a standard neural network.
Submission history
From: Ismael Lemhadri [view email][v1] Mon, 29 Jul 2019 04:23:21 UTC (82 KB)
[v2] Fri, 9 Aug 2019 03:35:46 UTC (84 KB)
[v3] Thu, 5 Sep 2019 12:32:49 UTC (84 KB)
[v4] Wed, 30 Oct 2019 06:05:03 UTC (201 KB)
[v5] Sat, 8 Feb 2020 09:17:50 UTC (797 KB)
[v6] Fri, 12 Jun 2020 01:33:22 UTC (1,112 KB)
[v7] Mon, 16 Nov 2020 02:17:14 UTC (1,507 KB)
[v8] Thu, 21 Jan 2021 20:25:22 UTC (1,506 KB)
[v9] Tue, 23 Feb 2021 16:57:45 UTC (1,507 KB)
[v10] Wed, 16 Jun 2021 04:43:38 UTC (2,968 KB)
Current browse context:
stat.ML
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.