Showing 1–6 of 6 results for author: Khalife, S

Sequence graphs realizations and ambiguity in language models

Authors: Sammy Khalife, Yann Ponty, Laurent Bulteau

Abstract: Several popular language models represent local contexts in an input text as bags of words. Such representations are naturally encoded by a sequence graph whose vertices are the distinct words occurring in x, with edges representing the (ordered) co-occurrence of two words within a sliding window of size w. However, this compressed representation is not generally bijective, and may introduce some… ▽ More Several popular language models represent local contexts in an input text as bags of words. Such representations are naturally encoded by a sequence graph whose vertices are the distinct words occurring in x, with edges representing the (ordered) co-occurrence of two words within a sliding window of size w. However, this compressed representation is not generally bijective, and may introduce some degree of ambiguity. Some sequence graphs may admit several realizations as a sequence, while others may not admit any realization. In this paper, we study the realizability and ambiguity of sequence graphs from a combinatorial and computational point of view. We consider the existence and enumeration of realizations of a sequence graph under multiple settings: window size w, presence/absence of graph orientation, and presence/absence of weights (multiplicities). When w = 2, we provide polynomial time algorithms for realizability and enumeration in all cases except the undirected/weighted setting, where we show the #P-hardness of enumeration. For a window of size at least 3, we prove hardness of all variants, even when w is considered as a constant, with the notable exception of the undirected/unweighted case for which we propose an XP algorithms for both (realizability and enumeration) problems, tight due to a corresponding W[1]-hardness result. We conclude with an integer program formulation to solve the realizability problem, and with dynamic programming to solve the enumeration problem. This work leaves open the membership to NP for both problems, a non-trivial question due to the existence of minimum realizations having exponential size on the instance encoding. △ Less

Submitted 13 February, 2024; originally announced February 2024.

Sample Complexity of Algorithm Selection Using Neural Networks and Its Applications to Branch-and-Cut

Authors: Hongyu Cheng, Sammy Khalife, Barbara Fiedorowicz, Amitabh Basu

Abstract: Data-driven algorithm design is a paradigm that uses statistical and machine learning techniques to select from a class of algorithms for a computational problem an algorithm that has the best expected performance with respect to some (unknown) distribution on the instances of the problem. We build upon recent work in this line of research by considering the setup where, instead of selecting a sin… ▽ More Data-driven algorithm design is a paradigm that uses statistical and machine learning techniques to select from a class of algorithms for a computational problem an algorithm that has the best expected performance with respect to some (unknown) distribution on the instances of the problem. We build upon recent work in this line of research by considering the setup where, instead of selecting a single algorithm that has the best performance, we allow the possibility of selecting an algorithm based on the instance to be solved, using neural networks. In particular, given a representative sample of instances, we learn a neural network that maps an instance of the problem to the most appropriate algorithm for that instance. We formalize this idea and derive rigorous sample complexity bounds for this learning problem, in the spirit of recent work in data-driven algorithm design. We then apply this approach to the problem of making good decisions in the branch-and-cut framework for mixed-integer optimization (e.g., which cut to add?). In other words, the neural network will take as input a mixed-integer optimization instance and output a decision that will result in a small branch-and-cut tree for that instance. Our computational results provide evidence that our particular way of using neural networks for cut selection can make a significant impact in reducing branch-and-cut tree sizes, compared to previous data-driven approaches. △ Less

Submitted 4 June, 2024; v1 submitted 3 February, 2024; originally announced February 2024.

The logic of rational graph neural networks

Authors: Sammy Khalife

Abstract: The expressivity of Graph Neural Networks (GNNs) can be described via appropriate fragments of the first order logic. Any query of the two variable fragment of graded modal logic (GC2) interpreted over labeled graphs can be expressed using a Rectified Linear Unit (ReLU) GNN whose size does not grow with graph input sizes [Barcelo & Al., 2020]. Conversely, a GNN expresses at most a query of GC2, fo… ▽ More The expressivity of Graph Neural Networks (GNNs) can be described via appropriate fragments of the first order logic. Any query of the two variable fragment of graded modal logic (GC2) interpreted over labeled graphs can be expressed using a Rectified Linear Unit (ReLU) GNN whose size does not grow with graph input sizes [Barcelo & Al., 2020]. Conversely, a GNN expresses at most a query of GC2, for any choice of activation function. In this article, we prove that some GC2 queries of depth $3$ cannot be expressed by GNNs with any rational activation function. This shows that not all non-polynomial activation functions confer GNNs maximal expressivity, answering a open question formulated by [Grohe, 2021]. This result is also in contrast with the efficient universal approximation properties of rational feedforward neural networks investigated by [Boullé & Al., 2020]. We also present a rational subfragment of the first order logic (RGC2), and prove that rational GNNs can express RGC2 queries uniformly over all graphs. △ Less

Submitted 13 August, 2024; v1 submitted 19 October, 2023; originally announced October 2023.

On the power of graph neural networks and the role of the activation function

Authors: Sammy Khalife, Amitabh Basu

Abstract: In this article we present new results about the expressivity of Graph Neural Networks (GNNs). We prove that for any GNN with piecewise polynomial activations, whose architecture size does not grow with the graph input sizes, there exists a pair of non-isomorphic rooted trees of depth two such that the GNN cannot distinguish their root vertex up to an arbitrary number of iterations. In contrast, i… ▽ More In this article we present new results about the expressivity of Graph Neural Networks (GNNs). We prove that for any GNN with piecewise polynomial activations, whose architecture size does not grow with the graph input sizes, there exists a pair of non-isomorphic rooted trees of depth two such that the GNN cannot distinguish their root vertex up to an arbitrary number of iterations. In contrast, it was already known that unbounded GNNs (those whose size is allowed to change with the graph sizes) with piecewise polynomial activations can distinguish these vertices in only two iterations. It was also known prior to our work that with ReLU (piecewise linear) activations, bounded GNNs are weaker than unbounded GNNs [ACI+22]. Our approach adds to this result by extending it to handle any piecewise polynomial activation function, which goes towards answering an open question formulated by [2021, Grohe] more completely. Our second result states that if one allows activations that are not piecewise polynomial, then in two iterations a single neuron perceptron can distinguish the root vertices of any pair of nonisomorphic trees of depth two (our results hold for activations like the sigmoid, hyperbolic tan and others). This shows how the power of graph neural networks can change drastically if one changes the activation function of the neural networks. The proof of this result utilizes the Lindemann-Weierstrauss theorem from transcendental number theory. △ Less

Submitted 8 May, 2024; v1 submitted 10 July, 2023; originally announced July 2023.

Neural networks with linear threshold activations: structure and algorithms

Authors: Sammy Khalife, Hongyu Cheng, Amitabh Basu

Abstract: In this article we present new results on neural networks with linear threshold activation functions. We precisely characterize the class of functions that are representable by such neural networks and show that 2 hidden layers are necessary and sufficient to represent any function representable in the class. This is a surprising result in the light of recent exact representability investigations… ▽ More In this article we present new results on neural networks with linear threshold activation functions. We precisely characterize the class of functions that are representable by such neural networks and show that 2 hidden layers are necessary and sufficient to represent any function representable in the class. This is a surprising result in the light of recent exact representability investigations for neural networks using other popular activation functions like rectified linear units (ReLU). We also give precise bounds on the sizes of the neural networks required to represent any function in the class. Finally, we design an algorithm to solve the empirical risk minimization (ERM) problem to global optimality for these neural networks with a fixed architecture. The algorithm's running time is polynomial in the size of the data sample, if the input dimension and the size of the network architecture are considered fixed constants. The algorithm is unique in the sense that it works for any architecture with any number of layers, whereas previous polynomial time globally optimal algorithms work only for very restricted classes of architectures. Using these insights, we propose a new class of neural networks that we call shortcut linear threshold networks. To the best of our knowledge, this way of designing neural networks has not been explored before in the literature. We show that these neural networks have several desirable theoretical properties. △ Less

Submitted 18 October, 2023; v1 submitted 15 November, 2021; originally announced November 2021.

Scalable graph-based individual named entity identification

Authors: Sammy Khalife, Michalis Vazirgiannis

Abstract: Named entity discovery (NED) is an important information retrieval problem that can be decomposed into two sub-problems. The first sub-problem, named entity recognition (NER), aims to tag pre-defined sets of words in a vocabulary (called "named entities": names, places, locations, ...) when they appear in natural language. The second subproblem, named entity linking/identification (NEL), considers… ▽ More Named entity discovery (NED) is an important information retrieval problem that can be decomposed into two sub-problems. The first sub-problem, named entity recognition (NER), aims to tag pre-defined sets of words in a vocabulary (called "named entities": names, places, locations, ...) when they appear in natural language. The second subproblem, named entity linking/identification (NEL), considers these entity mentions as queries to be identified in a pre-existing database. In this paper, we consider the NEL problem, and assume a set of queries (or mentions) that have to be identified within a knowledge base. This knowledge base is represented by a text database paired with a semantic graph. We present state-of-the-art methods in NEL, and propose a 2-step method for individual identification of named entities. Our approach is well-motivated by the limitations brought by recent deep learning approaches that lack interpratability, and require lots of parameter tuning along with large volume of annotated data. First of all, we propose a filtering algorithm designed with information retrieval and text mining techniques, aiming to maximize precision at K (typically for 5 <= K <=20). Then, we introduce two graph-based methods for named entity identification to maximize precision at 1 by re-ranking the remaining top entity candidates. The first identification method is using parametrized graph mining, and the second similarity with graph kernels. Our approach capitalizes on a fine-grained classification of entities from annotated web data. We present our algorithms in details, and show experimentally on standard datasets (NIST TAC-KBP, CONLL/AIDA) their performance in terms of precision are better than any graph-based method reported, and competitive with state-of-the-art systems. Finally, we conclude on the advantages of our graph-based approach compared to recent deep learning methods. △ Less

Submitted 26 November, 2018; originally announced November 2018.