Showing 1–13 of 13 results for author: Sinop, A K

Exphormer: Sparse Transformers for Graphs

Authors: Hamed Shirzad, Ameya Velingker, Balaji Venkatachalam, Danica J. Sutherland, Ali Kemal Sinop

Abstract: Graph transformers have emerged as a promising architecture for a variety of graph learning and representation tasks. Despite their successes, though, it remains challenging to scale graph transformers to large graphs while maintaining accuracy competitive with message-passing networks. In this paper, we introduce Exphormer, a framework for building powerful and scalable graph transformers. Exphor… ▽ More Graph transformers have emerged as a promising architecture for a variety of graph learning and representation tasks. Despite their successes, though, it remains challenging to scale graph transformers to large graphs while maintaining accuracy competitive with message-passing networks. In this paper, we introduce Exphormer, a framework for building powerful and scalable graph transformers. Exphormer consists of a sparse attention mechanism based on two mechanisms: virtual global nodes and expander graphs, whose mathematical characteristics, such as spectral expansion, pseduorandomness, and sparsity, yield graph transformers with complexity only linear in the size of the graph, while allowing us to prove desirable theoretical properties of the resulting transformer models. We show that incorporating Exphormer into the recently-proposed GraphGPS framework produces models with competitive empirical results on a wide variety of graph datasets, including state-of-the-art results on three datasets. We also show that Exphormer can scale to datasets on larger graphs than shown in previous graph transformer architectures. Code can be found at \url{https://github.com/hamed1375/Exphormer}. △ Less

Submitted 24 July, 2023; v1 submitted 10 March, 2023; originally announced March 2023.

Affinity-Aware Graph Networks

Authors: Ameya Velingker, Ali Kemal Sinop, Ira Ktena, Petar Veličković, Sreenivas Gollapudi

Abstract: Graph Neural Networks (GNNs) have emerged as a powerful technique for learning on relational data. Owing to the relatively limited number of message passing steps they perform -- and hence a smaller receptive field -- there has been significant interest in improving their expressivity by incorporating structural aspects of the underlying graph. In this paper, we explore the use of affinity measure… ▽ More Graph Neural Networks (GNNs) have emerged as a powerful technique for learning on relational data. Owing to the relatively limited number of message passing steps they perform -- and hence a smaller receptive field -- there has been significant interest in improving their expressivity by incorporating structural aspects of the underlying graph. In this paper, we explore the use of affinity measures as features in graph neural networks, in particular measures arising from random walks, including effective resistance, hitting and commute times. We propose message passing networks based on these features and evaluate their performance on a variety of node and graph property prediction tasks. Our architecture has lower computational complexity, while our features are invariant to the permutations of the underlying graph. The measures we compute allow the network to exploit the connectivity properties of the graph, thereby allowing us to outperform relevant benchmarks for a wide variety of tasks, often with significantly fewer message passing steps. On one of the largest publicly available graph regression datasets, OGB-LSC-PCQM4Mv1, we obtain the best known single-model validation MAE at the time of writing. △ Less

Submitted 23 June, 2022; originally announced June 2022.

Spectrally Robust Graph Isomorphism

Authors: Alexandra Kolla, Ioannis Koutis, Vivek Madan, Ali Kemal Sinop

Abstract: We initiate the study of spectral generalizations of the graph isomorphism problem. (a)The Spectral Graph Dominance (SGD) problem: On input of two graphs $G$ and $H$ does there exist a permutation $π$ such that $G\preceq π(H)$? (b) The Spectrally Robust Graph Isomorphism (SRGI) problem: On input of two graphs $G$ and $H$, find the smallest number $κ$ over all permutations $π$ such that… ▽ More We initiate the study of spectral generalizations of the graph isomorphism problem. (a)The Spectral Graph Dominance (SGD) problem: On input of two graphs $G$ and $H$ does there exist a permutation $π$ such that $G\preceq π(H)$? (b) The Spectrally Robust Graph Isomorphism (SRGI) problem: On input of two graphs $G$ and $H$, find the smallest number $κ$ over all permutations $π$ such that $ π(H) \preceq G\preceq κc π(H)$ for some $c$. SRGI is a natural formulation of the network alignment problem that has various applications, most notably in computational biology. Here $G\preceq c H$ means that for all vectors $x$ we have $x^T L_G x \leq c x^T L_H x$, where $L_G$ is the Laplacian $G$. We prove NP-hardness for SGD. We also present a $κ$-approximation algorithm for SRGI for the case when both $G$ and $H$ are bounded-degree trees. The algorithm runs in polynomial time when $κ$ is a constant. △ Less

Submitted 1 May, 2018; originally announced May 2018.

Comments: Extended version of a paper appearing in the proceedings of ICALP 2018

How to Round Subspaces: A New Spectral Clustering Algorithm

Authors: Ali Kemal Sinop

Abstract: A basic problem in spectral clustering is the following. If a solution obtained from the spectral relaxation is close to an integral solution, is it possible to find this integral solution even though they might be in completely different basis? In this paper, we propose a new spectral clustering algorithm. It can recover a $k$-partition such that the subspace corresponding to the span of its indi… ▽ More A basic problem in spectral clustering is the following. If a solution obtained from the spectral relaxation is close to an integral solution, is it possible to find this integral solution even though they might be in completely different basis? In this paper, we propose a new spectral clustering algorithm. It can recover a $k$-partition such that the subspace corresponding to the span of its indicator vectors is $O(\sqrt{opt})$ close to the original subspace in spectral norm with $opt$ being the minimum possible ($opt \le 1$ always). Moreover our algorithm does not impose any restriction on the cluster sizes. Previously, no algorithm was known which could find a $k$-partition closer than $o(k \cdot opt)$. We present two applications for our algorithm. First one finds a disjoint union of bounded degree expanders which approximate a given graph in spectral norm. The second one is for approximating the sparsest $k$-partition in a graph where each cluster have expansion at most $φ_k$ provided $φ_k \le O(λ_{k+1})$ where $λ_{k+1}$ is the $(k+1)^{st}$ eigenvalue of Laplacian matrix. This significantly improves upon the previous algorithms, which required $φ_k \le O(λ_{k+1}/k)$. △ Less

Submitted 19 October, 2015; v1 submitted 2 March, 2015; originally announced March 2015.

Comments: Appeared in SODA 2016

The Hardness of Approximation of Euclidean k-means

Authors: Pranjal Awasthi, Moses Charikar, Ravishankar Krishnaswamy, Ali Kemal Sinop

Abstract: The Euclidean $k$-means problem is a classical problem that has been extensively studied in the theoretical computer science, machine learning and the computational geometry communities. In this problem, we are given a set of $n$ points in Euclidean space $R^d$, and the goal is to choose $k$ centers in $R^d$ so that the sum of squared distances of each point to its nearest center is minimized. The… ▽ More The Euclidean $k$-means problem is a classical problem that has been extensively studied in the theoretical computer science, machine learning and the computational geometry communities. In this problem, we are given a set of $n$ points in Euclidean space $R^d$, and the goal is to choose $k$ centers in $R^d$ so that the sum of squared distances of each point to its nearest center is minimized. The best approximation algorithms for this problem include a polynomial time constant factor approximation for general $k$ and a $(1+ε)$-approximation which runs in time $poly(n) 2^{O(k/ε)}$. At the other extreme, the only known computational complexity result for this problem is NP-hardness [ADHP'09]. The main difficulty in obtaining hardness results stems from the Euclidean nature of the problem, and the fact that any point in $R^d$ can be a potential center. This gap in understanding left open the intriguing possibility that the problem might admit a PTAS for all $k,d$. In this paper we provide the first hardness of approximation for the Euclidean $k$-means problem. Concretely, we show that there exists a constant $ε> 0$ such that it is NP-hard to approximate the $k$-means objective to within a factor of $(1+ε)$. We show this via an efficient reduction from the vertex cover problem on triangle-free graphs: given a triangle-free graph, the goal is to choose the fewest number of vertices which are incident on all the edges. Additionally, we give a proof that the current best hardness results for vertex cover can be carried over to triangle-free graphs. To show this we transform $G$, a known hard vertex cover instance, by taking a graph product with a suitably chosen graph $H$, and showing that the size of the (normalized) maximum independent set is almost exactly preserved in the product graph using a spectral analysis, which might be of independent interest. △ Less

Submitted 11 February, 2015; originally announced February 2015.

Rounding Lasserre SDPs using column selection and spectrum-based approximation schemes for graph partitioning and Quadratic IPs

Authors: Venkatesan Guruswami, Ali Kemal Sinop

Abstract: We present an approximation scheme for minimizing certain Quadratic Integer Programming problems with positive semidefinite objective functions and global linear constraints. This framework includes well known graph problems such as Minimum graph bisection, Edge expansion, Sparsest Cut, and Small Set expansion, as well as the Unique Games problem. These problems are notorious for the existence of… ▽ More We present an approximation scheme for minimizing certain Quadratic Integer Programming problems with positive semidefinite objective functions and global linear constraints. This framework includes well known graph problems such as Minimum graph bisection, Edge expansion, Sparsest Cut, and Small Set expansion, as well as the Unique Games problem. These problems are notorious for the existence of huge gaps between the known algorithmic results and NP-hardness results. Our algorithm is based on rounding semidefinite programs from the Lasserre hierarchy, and the analysis uses bounds for low-rank approximations of a matrix in Frobenius norm using columns of the matrix. For all the above graph problems, we give an algorithm running in time $n^{O(r/ε^2)}$ with approximation ratio $\frac{1+ε}{\min\{1,λ_r\}}$, where $λ_r$ is the $r$'th smallest eigenvalue of the normalized graph Laplacian $\mathcal{L}$. In the case of graph bisection and small set expansion, the number of vertices in the cut is within lower-order terms of the stipulated bound. Our results imply $(1+O(ε))$ factor approximation in time $n^{O(r^\ast/ε^2)}$ where is the number of eigenvalues of $\mathcal{L}$ smaller than $1-ε$ (for variants of sparsest cut, $λ_{r^\ast} \ge \mathrm{OPT}/ε$ also suffices, and as $\mathrm{OPT}$ is usually $o(1)$ on interesting instances of these problems, this requirement on $r^\ast$ is typically weaker). For Unique Games, we give a factor $(1+\frac{2+ε}{λ_r})$ approximation for minimizing the number of unsatisfied constraints in $n^{O(r/ε)}$ time, improving upon an earlier bound for solving Unique Games on expanders. We also give an algorithm for independent sets in graphs that performs well when the Laplacian does not have too many eigenvalues bigger than $1+o(1)$. △ Less

Submitted 10 December, 2013; originally announced December 2013.

Comments: This manuscript is a merged and definitive version of (Guruswami, Sinop: FOCS 2011) and (Guruswami, Sinop: SODA 2013), with a significantly revised presentation. arXiv admin note: substantial text overlap with arXiv:1104.4746

Towards a better approximation for sparsest cut?

Authors: Sanjeev Arora, Rong Ge, Ali Kemal Sinop

Abstract: We give a new $(1+ε)$-approximation for sparsest cut problem on graphs where small sets expand significantly more than the sparsest cut (sets of size $n/r$ expand by a factor $\sqrt{\log n\log r}$ bigger, for some small $r$; this condition holds for many natural graph families). We give two different algorithms. One involves Guruswami-Sinop rounding on the level-$r$ Lasserre relaxation. The other… ▽ More We give a new $(1+ε)$-approximation for sparsest cut problem on graphs where small sets expand significantly more than the sparsest cut (sets of size $n/r$ expand by a factor $\sqrt{\log n\log r}$ bigger, for some small $r$; this condition holds for many natural graph families). We give two different algorithms. One involves Guruswami-Sinop rounding on the level-$r$ Lasserre relaxation. The other is combinatorial and involves a new notion called {\em Small Set Expander Flows} (inspired by the {\em expander flows} of ARV) which we show exists in the input graph. Both algorithms run in time $2^{O(r)} \mathrm{poly}(n)$. We also show similar approximation algorithms in graphs with genus $g$ with an analogous local expansion condition. This is the first algorithm we know of that achieves $(1+ε)$-approximation on such general family of graphs. △ Less

Submitted 11 April, 2013; originally announced April 2013.

Faster SDP hierarchy solvers for local rounding algorithms

Authors: Venkatesan Guruswami, Ali Kemal Sinop

Abstract: Convex relaxations based on different hierarchies of linear/semi-definite programs have been used recently to devise approximation algorithms for various optimization problems. The approximation guarantee of these algorithms improves with the number of {\em rounds} $r$ in the hierarchy, though the complexity of solving (or even writing down the solution for) the $r$'th level program grows as… ▽ More Convex relaxations based on different hierarchies of linear/semi-definite programs have been used recently to devise approximation algorithms for various optimization problems. The approximation guarantee of these algorithms improves with the number of {\em rounds} $r$ in the hierarchy, though the complexity of solving (or even writing down the solution for) the $r$'th level program grows as $n^{Ω(r)}$ where $n$ is the input size. In this work, we observe that many of these algorithms are based on {\em local} rounding procedures that only use a small part of the SDP solution (of size $n^{O(1)} 2^{O(r)}$ instead of $n^{Ω(r)}$). We give an algorithm to find the requisite portion in time polynomial in its size. The challenge in achieving this is that the required portion of the solution is not fixed a priori but depends on other parts of the solution, sometimes in a complicated iterative manner. Our solver leads to $n^{O(1)} 2^{O(r)}$ time algorithms to obtain the same guarantees in many cases as the earlier $n^{O(r)}$ time algorithms based on $r$ rounds of the Lasserre hierarchy. In particular, guarantees based on $O(\log n)$ rounds can be realized in polynomial time. We develop and describe our algorithm in a fairly general abstract framework. The main technical tool in our work, which might be of independent interest in convex optimization, is an efficient ellipsoid algorithm based separation oracle for convex programs that can output a {\em certificate of infeasibility with restricted support}. This is used in a recursive manner to find a sequence of consistent points in nested convex bodies that "fools" local rounding algorithms. △ Less

Submitted 4 September, 2012; v1 submitted 18 July, 2012; originally announced July 2012.

Comments: 30 pages, 8 figures

Constant Factor Lasserre Integrality Gaps for Graph Partitioning Problems

Authors: Venkatesan Guruswami, Ali Kemal Sinop, Yuan Zhou

Abstract: Partitioning the vertices of a graph into two roughly equal parts while minimizing the number of edges crossing the cut is a fundamental problem (called Balanced Separator) that arises in many settings. For this problem, and variants such as the Uniform Sparsest Cut problem where the goal is to minimize the fraction of pairs on opposite sides of the cut that are connected by an edge, there are lar… ▽ More Partitioning the vertices of a graph into two roughly equal parts while minimizing the number of edges crossing the cut is a fundamental problem (called Balanced Separator) that arises in many settings. For this problem, and variants such as the Uniform Sparsest Cut problem where the goal is to minimize the fraction of pairs on opposite sides of the cut that are connected by an edge, there are large gaps between the known approximation algorithms and non-approximability results. While no constant factor approximation algorithms are known, even APX-hardness is not known either. In this work we prove that for balanced separator and uniform sparsest cut, semidefinite programs from the Lasserre hierarchy (which are the most powerful relaxations studied in the literature) have an integrality gap bounded away from $1$, even for $Ω(n)$ levels of the hierarchy. This complements recent algorithmic results in Guruswami and Sinop (2011) which used the Lasserre hierarchy to give an approximation scheme for these problems (with runtime depending on the spectrum of the graph). Along the way, we make an observation that simplifies the task of lifting "polynomial constraints" (such as the global balance constraint in balanced separator) to higher levels of the Lasserre hierarchy. △ Less

Submitted 11 August, 2014; v1 submitted 27 February, 2012; originally announced February 2012.

Comments: 15 pages, 1 figure

Approximating Non-Uniform Sparsest Cut via Generalized Spectra

Authors: Venkatesan Guruswami, Ali Kemal Sinop

Abstract: We give an approximation algorithm for non-uniform sparsest cut with the following guarantee: For any $ε,δ\in (0,1)$, given cost and demand graphs with edge weights $C, D$ respectively, we can find a set $T\subseteq V$ with $\frac{C(T,V\setminus T)}{D(T,V\setminus T)}$ at most $\frac{1+ε}δ$ times the optimal non-uniform sparsest cut value, in time $2^{r/(δε)}\poly(n)$ provided $λ_r \ge Φ^*/(1-δ)$.… ▽ More We give an approximation algorithm for non-uniform sparsest cut with the following guarantee: For any $ε,δ\in (0,1)$, given cost and demand graphs with edge weights $C, D$ respectively, we can find a set $T\subseteq V$ with $\frac{C(T,V\setminus T)}{D(T,V\setminus T)}$ at most $\frac{1+ε}δ$ times the optimal non-uniform sparsest cut value, in time $2^{r/(δε)}\poly(n)$ provided $λ_r \ge Φ^*/(1-δ)$. Here $λ_r$ is the $r$'th smallest generalized eigenvalue of the Laplacian matrices of cost and demand graphs; $C(T,V\setminus T)$ (resp. $D(T,V\setminus T)$) is the weight of edges crossing the $(T,V\setminus T)$ cut in cost (resp. demand) graph and $Φ^*$ is the sparsity of the optimal cut. In words, we show that the non-uniform sparsest cut problem is easy when the generalized spectrum grows moderately fast. To the best of our knowledge, there were no results based on higher order spectra for non-uniform sparsest cut prior to this work. Even for uniform sparsest cut, the quantitative aspects of our result are somewhat stronger than previous methods. Similar results hold for other expansion measures like edge expansion, normalized cut, and conductance, with the $r$'th smallest eigenvalue of the normalized Laplacian playing the role of $λ_r$ in the latter two cases. Our proof is based on an l1-embedding of vectors from a semi-definite program from the Lasserre hierarchy. The embedded vectors are then rounded to a cut using standard threshold rounding. We hope that the ideas connecting $\ell_1$-embeddings to Lasserre SDPs will find other applications. Another aspect of the analysis is the adaptation of the column selection paradigm from our earlier work on rounding Lasserre SDPs [GS11] to pick a set of edges rather than vertices. This feature is important in order to extend the algorithms to non-uniform sparsest cut. △ Less

Submitted 17 December, 2012; v1 submitted 17 December, 2011; originally announced December 2011.

Comments: 16 pages

Lasserre Hierarchy, Higher Eigenvalues, and Approximation Schemes for Quadratic Integer Programming with PSD Objectives

Authors: Venkatesan Guruswami, Ali Kemal Sinop

Abstract: We present an approximation scheme for optimizing certain Quadratic Integer Programming problems with positive semidefinite objective functions and global linear constraints. This framework includes well known graph problems such as Minimum graph bisection, Edge expansion, Uniform sparsest cut, and Small Set expansion, as well as the Unique Games problem. These problems are notorious for the exist… ▽ More We present an approximation scheme for optimizing certain Quadratic Integer Programming problems with positive semidefinite objective functions and global linear constraints. This framework includes well known graph problems such as Minimum graph bisection, Edge expansion, Uniform sparsest cut, and Small Set expansion, as well as the Unique Games problem. These problems are notorious for the existence of huge gaps between the known algorithmic results and NP-hardness results. Our algorithm is based on rounding semidefinite programs from the Lasserre hierarchy, and the analysis uses bounds for low-rank approximations of a matrix in Frobenius norm using columns of the matrix. For all the above graph problems, we give an algorithm running in time $n^{O(r/ε^2)}$ with approximation ratio $\frac{1+ε}{\min\{1,λ_r\}}$, where $λ_r$ is the $r$'th smallest eigenvalue of the normalized graph Laplacian $\mathcal{L}$. In the case of graph bisection and small set expansion, the number of vertices in the cut is within lower-order terms of the stipulated bound. Our results imply $(1+O(ε))$ factor approximation in time $n^{O(r^\ast/ε^2)}$ where $r^\ast$ is the number of eigenvalues of $\mathcal{L}$ smaller than $1-ε$. For Unique Games, we give a factor $(1+\frac{2+ε}{λ_r})$ approximation for minimizing the number of unsatisfied constraints in $n^{O(r/ε)}$ time. This improves an earlier bound for solving Unique Games on expanders, and also shows that Lasserre SDPs are powerful enough to solve well-known integrality gap instances for the basic SDP. We also give an algorithm for independent sets in graphs that performs well when the Laplacian does not have too many eigenvalues bigger than $1+o(1)$. △ Less

Submitted 17 May, 2011; v1 submitted 25 April, 2011; originally announced April 2011.

Comments: 43 pages

Optimal Column-Based Low-Rank Matrix Reconstruction

Authors: Venkatesan Guruswami, Ali Kemal Sinop

Abstract: We prove that for any real-valued matrix $X \in \R^{m \times n}$, and positive integers $r \ge k$, there is a subset of $r$ columns of $X$ such that projecting $X$ onto their span gives a $\sqrt{\frac{r+1}{r-k+1}}$-approximation to best rank-$k$ approximation of $X$ in Frobenius norm. We show that the trade-off we achieve between the number of columns and the approximation ratio is optimal up to l… ▽ More We prove that for any real-valued matrix $X \in \R^{m \times n}$, and positive integers $r \ge k$, there is a subset of $r$ columns of $X$ such that projecting $X$ onto their span gives a $\sqrt{\frac{r+1}{r-k+1}}$-approximation to best rank-$k$ approximation of $X$ in Frobenius norm. We show that the trade-off we achieve between the number of columns and the approximation ratio is optimal up to lower order terms. Furthermore, there is a deterministic algorithm to find such a subset of columns that runs in $O(r n m^ω \log m)$ arithmetic operations where $ω$ is the exponent of matrix multiplication. We also give a faster randomized algorithm that runs in $O(r n m^2)$ arithmetic operations. △ Less

Submitted 4 January, 2012; v1 submitted 9 April, 2011; originally announced April 2011.

Comments: 8 pages

Improved Inapproximability Results for Maximum k-Colorable Subgraph

Authors: Venkatesan Guruswami, Ali Kemal Sinop

Abstract: We study the maximization version of the fundamental graph coloring problem. Here the goal is to color the vertices of a k-colorable graph with k colors so that a maximum fraction of edges are properly colored (i.e. their endpoints receive different colors). A random k-coloring properly colors an expected fraction 1-1/k of edges. We prove that given a graph promised to be k-colorable, it is NP-h… ▽ More We study the maximization version of the fundamental graph coloring problem. Here the goal is to color the vertices of a k-colorable graph with k colors so that a maximum fraction of edges are properly colored (i.e. their endpoints receive different colors). A random k-coloring properly colors an expected fraction 1-1/k of edges. We prove that given a graph promised to be k-colorable, it is NP-hard to find a k-coloring that properly colors more than a fraction ~1-O(1/k} of edges. Previously, only a hardness factor of 1-O(1/k^2) was known. Our result pins down the correct asymptotic dependence of the approximation factor on k. Along the way, we prove that approximating the Maximum 3-colorable subgraph problem within a factor greater than 32/33 is NP-hard. Using semidefinite programming, it is known that one can do better than a random coloring and properly color a fraction 1-1/k +2 ln k/k^2 of edges in polynomial time. We show that, assuming the 2-to-1 conjecture, it is hard to properly color (using k colors) more than a fraction 1-1/k + O(ln k/ k^2) of edges of a k-colorable graph. △ Less

Submitted 11 November, 2009; v1 submitted 12 October, 2009; originally announced October 2009.

Comments: 16 pages, 2 figures