Graph Neural Networks for Text Classification: A Survey

Some applications have explicit graph structures including constituency or dependency graphs (Tang et al., 2020), knowledge graphs (Ostendorff et al., 2019; Marin et al., 2014), social networks (Dai et al., 2022) without constructing graph structure and defining corresponding nodes and edges. However, for text classification, the most common graph structures are implicit, which means we need to define a new graph structure for a specific task such as designing a word-word or word-document co-occurrence graph. In addition, for text classification tasks, the graph structure can be generally classified into two types:

• •

  Corpus-level/Document-level. Corpus-level graphs intend to construct the graph to represent the whole corpus such as (Yao et al., 2019; Liu et al., 2020; Lin et al., 2021; Wu et al., 2019b), while the document-level graphs focus on representing the non-Euclidean relations existing in a single text body like (Chen et al., 2020; Nikolentzos et al., 2020; Zhang et al., 2020). Supposing a specific corpus $\mathcal{C}$ contains a set of documents (text bodies) $\mathcal{C}=\{D_{1},D_{2},...,D_{j}\}$ and each $D_{i}$ contains a set of tokens $D_{i}=\{t_{i_{1}},t_{i_{2}},...,t_{i_{k}}\}$. The vocabulary of $\mathcal{C}$ can be represented as $\mathcal{D}=\{t_{1},t_{2},...,t_{l}\}$, where $l$ is the length of $\mathcal{D}$. For the most commonly adopted corpus-level graph $G_{corpus}=(V_{corpus},E_{corpus})$, a node $v_{i}$ in $V_{corpus}$ follows $v_{i}\in\mathcal{C}\cup\mathcal{D}$ and the edge $e_{ij}\in E_{corpus}$ is one kind of relations between $v_{i}$ and $v_{j}$. Regarding the document level graph $G_{doc_{i}}=(V_{doc_{i}},E_{doc_{i}})$, a node $v_{i_{j}}$ in $V_{doc_{i}}$ follows $v_{i_{j}}\in D_{i}$.

After designing the graph-scale for the specific tasks, specifying the graph types is also important to determine the nodes and their relations. For text classification tasks, the commonly used graph construction ways can be summarized into:

• •

  Homogeneous/Heterogeneous Graphs: homogeneous graphs have the same node and edge type while heterogeneous graphs have various node and edge types. For a graph $G=(V,E)$, we use $\mathcal{N}^{v}$ and $\mathcal{N}^{e}$ to represent the number of types of $V$ and $E$. If $\mathcal{N}^{v}=\mathcal{N}^{e}=1$, $G$ is a homogeneous graph. If $\mathcal{N}^{v}>1$ or $\mathcal{N}^{e}>1$, $G$ is a heterogeous graph.

• •

  Static/Dynamic Graphs: Static graphs aim to use the constructed graph structure by various external or internal information to leverage to enhance the initial node representation such as dependency or constituency graph (Tang et al., 2020), co-occurrence between word nodes (Zhang et al., 2020), TF-IDF between word and document nodes (Yao et al., 2019; Wu et al., 2019b; Lei et al., 2021) and so on. However, compared with the static graph, the dynamic graph initial representations or graph topology are changing during training without certain domain knowledge and human efforts. The feature representations or graph structure can jointly learn with downstream tasks to be optimised together. For example, (Wang et al., 2020) proposed a novel topic-awared GNN text classification model with dynamically updated edges between topic nodes with others (e.g. document, word). Piao et al. (Piao et al., 2021) also designed a dynamic edge based graph to update the contextual dependencies between nodes. Additionally, (Chen et al., 2020) propose a dynamic GNN model to jointly update the edge and node representation simultaneously. We provide more details about above mentioned models in Section 3 and Section 4.

Another widely used pair of graph categories are directed or undirected graphs based on whether the directions of edges are bi-directional or not. For text classification, most of the GNN designs are following the unidirectional way. In addition, those graph type pairs are not parallel which means those types can be combined.

Based on the pre-defined graph structure and specified graph type, selecting the appropriate initial node representations is the key procedure to ensure the proposed graph structure can effectively learn node. According to the node entity type, the existing node representation approaches for text classification can be generally summarised into:

• •

  Word-level Representation: non-context word embedding methods such as GloVe (Pennington et al., 2014), Word2vec (Mikolov et al., 2013), FastText (Bojanowski et al., 2017) are widely adopted by many GNN-based text classification framework to numerically represent the node features. However, those embedding methods are restricted to capturing only syntactic similarity and fail to represent the complex semantic relationships between words, as well as, they cannot capture the meaning of out-of-vocabulary (OOV) words, and their representations are fixed. Therefore, there are some recent studies selecting ELMo (Peters et al., 2018), BERT (Devlin et al., 2018), GPT (Radford et al., 2018) to get contextual word-level node representation. Notably, even if one-hot encoding is the simplest word representation method, there are many GNN-based text classifiers using one-hot encoding and achieving state-of-the-art performance. Few frameworks use randomly initialised vectors to represent the word-level node features.

• •

  Document-level Representation: similar to other NLP applications, document level representations are normally acquired by aggregating the word level representation via some deep learning frameworks. For example, some researchers select by extracting the last-hidden state of LSTM or using the [CLS] token from BERT to numerically represent the input text body. Furthermore, it is also a commonly used document-level node representation way to use TF-IDF based document vectors.

Most GNN based text classification frameworks will compare the performance between different node representation methods to conduct quantitative analysis, as well as provide reasonable justifications for demonstrating the effectiveness of the selected initial node representation based on defined graph structure.

Well-defined edge features can effectively improve the graph representation learning efficiency and performance to exploit more explicit and implicit relations between nodes. Based on the predefined graph types, the edge feature types can be divided into structural features and non-structural features. The structural edge features are acquired from explicit relations between nodes such as dependency or constituency relation between words, word-word adjacency relations, etc. Those relations between nodes are explicitly defined and are also widely employed in other NLP applications. However, more commonly used edge features are non-structural features which implicitly existed between the nodes and specifically applied to specific graph-based frameworks. The typically non-structural edge features are firstly defined by (Kim, 2014) for GNNs-based text classification tasks including:

• •

  PMI measures the co-occurrence between two words in a sliding window $W$ and is calculated as:

  (4)		$$\displaystyle\text{PMI}(i,j)=log\frac{p(i,j)}{p(i)p(j)}$$
  (5)		$$\displaystyle p(i,j)=\frac{\#W(i,j)}{\#W}$$
  (6)		$$\displaystyle p(i)=\frac{\#W(i)}{\#W}$$

  where $\#W$ is the number of windows in total, and $\#W(i)$, $\#W(i,j)$ shows the number of windows containing word $i$ and both word $i$ and $j$ respectively.

• •

  TF-IDF is the broadly used weight of the edges between document-level nodes and word-level nodes.

Except for those two widely used implicit edge features, some specific edge weighting methods are proposed to meet the demands of particular graph structures for exploiting more information of input text bodies.

After specifying the graph structure and types, the graph representation learning tasks and training settings also need to be determined to decide how to optimise the designed GNNs. Generally, the graph representation learning tasks can be categorised into three levels including Node-level, Graph-level and Edge-level. Node-level and Graph-level tasks involve node or graph classification, clustering, regression, etc, while Edge-level tasks include link prediction or edge classification for predicting the relation existence between two nodes or the corresponding edge categories.

Similar to other deep learning model training settings, GNNs also can be divided into supervised, semi-supervised and unsupervised training settings. Supervised training provides labelled training data, while unsupervised training utilises unlabeled data to train the GNNs. However, compared with supervised or unsupervised learning, semi-supervised learning methods are broadly used by GNNs designed for text classification applications which could be classified into two types:

• •

  Inductive Learning adjusts the weights of proposed GNNs based on a labelled training set for learning the overall statistics to induce the general trained model for following processing. The unlabeled set can be fed into the trained GNNs to compute the expected outputs.

• •

  Transductive Learning intends to exploit labelled and unlabeled sets simultaneously for leveraging the relations between different samples to improve the overall performance.

TextGCN(Yao et al., 2019) Yao et al. (2019) builds a corpus-level graph with training document nodes, test document nodes and word nodes. Before constructing the graph, a common preprocessing method(Kim, 2014) has been applied and words shown fewer than 5 times or in NLTK(Bird et al., 2009) stopwords list have been removed. The edge value between the document node and the word node is TF-IDF and that between the word nodes is PMI. The adjacency matrix of this graph shows as follows.

A two-layer GCN is applied to the graph, and the dimension of the second layer output equals to the number of classes in the dataset. Formally, the forward propagation of TextGCN shows as:

where $\tilde{A}$ is the normalized adjacency of $A$ and $X$ is one-hot embedding. $W_{0}$ and $W_{1}$ are learnable parameters of the model. The representation on training documents is used to calculate the loss and that on test documents is for prediction. TextGCN is the first work that treats a text classification task as a node classification problem by constructing a corpus-level graph and has inspired many following works.

Based on TextGCN, several works follow the same graph construction method and node initialization but apply different graph propagation models.

SGC(Wu et al., 2019b) To make GCN efficient, SGC (Simple Graph Convolution) removes the nonlinear activation function in GCN layers, therefore, the K-layer propagation of SGC shows as:

which can be reparameterized into

and $K$ is 2 when applied to text classification tasks. With a smaller number of parameters and only one feedforward layer, SGC saves computation time and resources while improving performance.

S²GC(Zhu and Koniusz, 2020) To solve the oversmoothing issues in GCN, Zhu and Koniusz (2020) propose Simple Spectral Graph Convolution(S²GC) which includes self-loops using Markov Diffusion Kernel. The output of S²GC is calculated as:

and can be generalized into:

Similarly, $K$ = 2 on text classification tasks and $\alpha$ denotes the trade-off between self-information of the node and consecutive neighbourhood information. S²GC can also be viewed as introducing skip connections into GCN.

NMGC(Lei et al., 2021) Other than using the sum of each GCN layer in S²GC, NMGC applies min pooling using the Multi-hop neighbour Information Fusion (MIF) operator to address oversmoothing problems. A MIF function is defined as:

NMGC-K firstly applies a MIF($K$) layer then a GCN layer and K is 2 or 3. For example, when $K$ = 3, the output is:

NMGC can also be treated as a skip-connection in Graph Neural Networks which makes the shallow layer of GNN contribute to the final representation directly.

TG-Transformer(Zhang and Zhang, 2020) TextGCN treats the document nodes and word nodes as the same type of nodes during propagation, and to introduce heterogeneity into the TextGCN graph, TG-Transformer (Text Graph Transformer) adopts two sets of weights for document nodes and word nodes respectively. To cope with a large corpus graph, subgraphs are sampled from the TextGCN graph using PageRank algorithm(Page et al., 1999). The input embedding of is the sum of three types of embedding: pretrained GloVe embedding, node type embedding, and Weisfeiler-Lehman structural encoding(Niepert et al., 2016). During propagation, self-attention(Vaswani et al., 2017) with graph residual(Zhang and Meng, 2019) is applied.

BertGCN(Lin et al., 2021) To combine BERT(Kenton and Toutanova, 2019) and TextGCN, BertGCN enhances TextGCN by replacing the document node initialization with the BERT [CLS] output of each epoch and replacing the word input vector with zeros. BertGCN trains BERT and TextGCN jointly by interpolating the output of TextGCN and BERT:

where $\lambda$ is the trade-off factor. To optimize the memory during training, a memory bank is used to track the document input and a smaller learning rate is set to BERT module to remain the consistency of the memory bank. BertGCN shows that with the help of TextGCN, BERT can achieve better performance.

TensorGCN(Liu et al., 2020) Instead of constructing a single corpus-level graph, TensorGCN builds three independent graphs: Semantic-based graph, Syntactic-based graph, and Sequential-based graph to incorporate semantic, syntactic and sequential information respectively and combines them into a tensor graph.

Three graphs share the same set of TF-IDF values for the word-document edge but different values for word-word edges. Semantic-based graph extracts the semantic features from a trained Long short-term memory(LSTM)(Hochreiter and Schmidhuber, 1997) model and connects the words sharing high similarity. Syntactic-based graph uses Stanford CoreNLP parser(Manning et al., 2014) and constructs edges between words when they have a larger probability of having dependency relation. For Sequential-based graph, PMI value is applied as TextGCN does.

The propagation includes intra-graph propagation and inter-graph propagation. The model first applies the GCN layer on three graphs separately as intra-graph propagation. Then the same nodes on three graphs are treated as a virtual graph and another GCN layer is applied as inter-graph propagation.

ME-GCN(Wang et al., 2022b) To fully utilize the corpus information and analyze rich relational information of the graph, ME-GCN (Multi-dimensional Edge-Embedded GCN) builds a graph with multi-dimensional word-word, word-document and document-document edges. Word2vec and Doc2vec embedding is firstly trained on the given corpus and the similarity of each dimension of trained embedding is used to construct the multi-dimensional edges. The trained embedding also serves as the input embedding of the graph nodes. During propagation, GCN is firstly applied on each dimension and representations on different dimensions are either concatenated or fed into a pooling method to get the final representations of each layer.

HeteGCN(Ragesh et al., 2021) HeteGCN (Heterogeneous GCN) optimizes the TextGCN by decomposing the TextGCN undirected graph into several directed subgraphs. Several subgraphs from TextGCN graph are combined sequentially as different layers: feature graph (word-word graph), feature-document graph (word-document graph), and document-feature graph (document-word graph). Different combinations were tested and the best model is shown as:

where $\bm{A}_{w-w}$ and $\bm{A}_{w-d}$ show the adjacency matrix for the word-word subgraph and word-document subgraph. Since the input of HeteGCN is the word node embeddings without using document nodes, it can also work in an inductive way while the previous corpus-level graph text classification models are all transductive models.

InducT-GCN(Wang et al., 2022a) InducT-GCN (InducTive Text GCN) aims to extend the transductive TextGCN into an inductive model. Instead of using the whole corpus for building the graph, InducT-GCN builds a training corpus graph and makes the input embedding of the document as the TF-IDF vectors, which aligns with the one-hot word embeddings. The weights are learned following TextGCN but InducT-GCN builds virtual subgraphs for prediction on new test documents.

T-VGAE(Xie et al., 2021) T-VGAE (Topic Variational Graph Auto-Encoder) applies Variational Graph Auto-Encoder on the latent topic of each document to make the model inductive. A vocabulary graph $A_{v}$ which connects the words using PMI values is constructed while each document is represented using the TF-IDF vector. All the document vectors are stacked into a matrix which can also be treated as a bipartite graph $A_{d}$. Two graph auto-encoder models are applied on $A_{v}$ and $A_{d}$ respectively. The overall workflow shows as:

where $X^{v}$ is an Identity Matrix. The $\text{Encoder}_{GCN}$ and the decoders are applied following $VGAE$(Kipf and Welling, 2016) while $\text{Encoder}_{UDMP}$ is an unidirectional message passing variant of $\text{Encoder}_{GCN}$. The training objective is minimising the reconstruction error and $Z_{d}$ is used for the classification task.

HGAT(Linmei et al., 2019) HGAT (Heterogeneous GAT) applies LDA(Blei et al., 2003) to extract topic information for each document, top $P$ topics with the largest probabilities are selected as connected with the document. Instead of using the words directly, to utilize the external knowledge HGAT applies the entity linking tool TAGME¹¹1https://sobigdata.d4science.org/group/tagme/ to identify the entities in the document and connects them. The semantic similarity between entities using pretrained Word2vec with threshold is used to define the connectedness between entity nodes. Since the graph is a heterogeneous graph, a HIN (heterogeneous information network) model is implemented which propagates solely on each sub-graphs depending on the type of node. An HGAT model is applied by considering type-level attention and node-level attention. For a given node, the type-level attention learns the weights of different types of neighbouring nodes while node-level attention captures the importance of different neighbouring nodes when ignoring the type. By using the dual attention mechanism, HGAT can capture the information of type and node at the same time.

STGCN(Yan et al., 2013) In terms of short text classification, STGCN (Short-Text GCN) applies BTM to get topic information to avoid the data sparsity problem from LDA. The graph is constructed following TextGCN while extra topic nodes are included. The edge values of word-topic and document-topic are from BTM and a classical two-layer GCN is applied. The word embeddings learned from STGCN are concatenated with BERT embeddings and a bi-LSTM model is applied for final prediction.

DHTG(Wang et al., 2020) To capture different levels of information, DHTG (Dynamic Hierarchical Topic Graph) introduces hierarchical topic-level nodes in the graph from fine-grain to coarse. Poisson gamma belief network (PGBN)(Zhou et al., 2015) is used as a probabilistic deep topic model. The first-layer topics are from the combination of words, while deeper layers are generated by previous layers’ topics with the weights of PGBN, and the weights serve as the edge values of each layer of topics. For the topics on the same layer, the cosine similarity is chosen as the edge value. A two-layer GCN is applied and the model is learned jointly with PGBN, which makes the edge of the topics dynamic.

Text-Level-GNN(Huang et al., 2019) Text-Level-GNN applies a small sliding window and constructs the graph with a small number of nodes and edges in each graph, which saves memory and computation time. The edge value is trainable and shared across the graphs when connecting the same two words, which also brings global information.

Unlike corpus-level graph models, Text-Level-GNN applies a message passing mechanism (MPM)(Gilmer et al., 2017) instead of GCN for graph learning. For each node, the neighbour information is aggregated using max-pooling with trainable edge values as the AGGREGATE function and then the weighted sum is used as the COMBINE function. To get the representation of each graph, sum-pooling and an MLP classifier are applied as the READOUT function. The propagation shows as:

where $\bm{h}^{(l)}_{i}$ is $i$th word node presentation of layer $l$, $e_{ni}$ is edge weight from node $n$ to node $i$. A two-layer MPM is applied, and the input of each graph is pretrained GloVe vectors.

MPAD(Nikolentzos et al., 2020) MPAD (Message Passing Attention Networks) connects words within a sliding window of size 2 but also includes an additional master node connecting all nodes in the graph. The edge only shows the connectedness of each pair of word nodes and is fixed. A variant of Gated Graph Neural Networks is applied where the AGGREGATE function is the weighted sum and the COMBINE function is GRU(Chung et al., 2014). Self-attention is applied in the READOUT function.

To learn the high-level information, the master node is directly concatenated with the READOUT output, working as a skip connection mechanism. To get the final representation, each layer’s READOUT results are concatenated to capture multi-granularity information. Pretrained Word2vec is used as the initialization of word nodes input.

TextING(Zhang et al., 2020) To simplify MPAD, TextING ignores the master node in the document-level graphs, which makes the graph sparser. Compared with Text-Level-GNN, TextING remains fixed edges. A similar AGGREGATE and COMBINE function are applied under the concept of e Gated Graph Neural Networks(GGNN)(Li et al., 2016) with the weighted sum and GRU. However, for the READOUT function, soft attention is used and both max-pooling and mean-pooling are applied to make sure that ”every word plays a role in the text and the keywords should contribute more explicitly”.

MLGNN(Liao et al., 2021b) MLGNN (Multi-level GNN) builds the same graph as TextING but introduces three levels of MPM: bottom-level, middle-level and top-level. In the bottom-level MPM, the same method with Text-Level-GNN is applied with pretrained Word2vec as input embedding but the edge is non-trainable. In the middle level, a larger window size is adopted and Graph Attention Networks(GAT)(Veličković et al., 2018) is applied to learn long distant word nodes information. In the top-level MPM, all word nodes are connected and multi-head self-attention(Vaswani et al., 2017) is applied. By applying three different levels of MPM, MLGNN learns multi-granularity information well.

DADGNN(Liu et al., 2021a) DADGNN (Deep Attention Diffusion GNN) constructs the same graph as TextING but uses attention diffusion to overcome the oversmoothing issue. Pretrained word embedding is used as the input of each node and an MLP layer is applied. Then, the graph attention matrix is calculated based on the attention to the hidden states of each node. The diffusion matrix is calculated as

where $A$ is the graph attention matrix and $\epsilon$ is the learnable coefficients. $A^{n}$ plays a role of connecting $n$-hop neighbours and Liu et al. (2021a) uses $n\in[4,7]$ in practice. A multi-head diffusion matrix is applied for layer propagation.

TextSSL(Piao et al., 2021) To solve the word ambiguity problem and show the word synonymity and dynamic contextual dependency, TextSSL (Sparse Structure Learning) learns the graph using intra-sentence neighbours and inter-sentence neighbours simultaneously. The local syntactic neighbour is defined as the consecutive words and trainable edges across graphs are also included by using Gumbel-softmax . By applying sparse structure learning, TextSSL manages to select edges with dynamic contextual dependencies.

DAGNN(Wu et al., 2019a) To address the long-distance dependency, hierarchical information and cross-domain learning challenges in domain-adversarial text classification tasks, Wu et al. (2019a) propose DAGNN (Domain-Adversarial Graph Neural Network). Each document is represented by a graph with content words as nodes and PMI values as edge values, which can capture long-distance dependency information. Pretrained FastText is chosen as the input word embeddings to handle the out-of-vocabulary issue and a GCN model with skip connection is used to address the oversmoothing problem. The propagation is formulated as:

To learn the hierarchical information of documents, DiffPool(Ying et al., 2018) is applied to assign each document into a set of clusters. Finally, adversarial training is used to minimize the loss on source tasks and maximize the differentiation between source and target tasks.

ReGNN(Li et al., 2019) To capture both global and local information, ReGNN (Recursive Graphical Neural Network) uses PMI together with consecutive words as the word edges. And graph propagation function is the same as GGNN while additive attention(Bahdanau et al., 2015) is applied in aggregation. Pretrained GloVe is the input embedding of each word node.

GFN(Dai et al., 2022) GFN (Graph Fusion Network) builds four types of graphs using the word co-occurrence statistics, PMI, the similarity of pretrained embedding and Euclidean distance of pretrained embedding. Although four corpus-level graphs are built, the graph learning actually happens on subgraphs of each document, making the method a document-level GNN. For each subgraph, each type of graph is learned separately using the graph convolutional method and then a fusion method of concatenation is used. After an MLP layer, average pooling is applied to get the document representation.

Topic classification models aim to classify input text bodies from diverse sources into predefined categories. News categorization is a typical topic classification task to obtain key information from news and classify them into corresponding topics. The input text bodies normally are paragraphs or whole documents especially for news categorization, while there are still some short text classification datasets from certain domains such as micro-blogs, bibliography, etc. Some typical datasets are listed:

• •

  Ohsumed (Joachims, 1998) is acquired from the MEDLINE database and further processed by (Yao et al., 2019) via selecting certain documents (abstracts) and filtering out the documents belonging to multiple categories. Those documents are classified into 23 cardiovascular diseases. The statistics of (Yao et al., 2019) processed Ohsumed dataset is represented in Table 3, which is directly employed by other related works.

• •

  R8 / R52 are two subsets of the Reuters 21587 dataset ²²2 For the original Reuters 21587 dataset, please refer to this link http://www.daviddlewis.com/resources/testcollections/reuters21578 which contain 8 and 52 news topics from Reuters financial news services, respectively.

• •

  20NG is another widely used news categorization dataset that contains 20 newsgroups. It was originally collected by (Lang, 1995), but the procedures are not explicitly described.

• •

  AG News (Zhang et al., 2015) is a large-scale news categorization dataset compared with other commonly used datasets which are constructed by selecting the top-4 largest categories from the AG corpus. Each news topic contains 30,000 samples for training and 1900 samples for testing.

• •

  Database systems and Logic Programming (DBLP) is a topic classification dataset to classify the computer science paper titles into six various topics (Mei et al., 2021). Different from paragraph or document based topic classification dataset, DBLP aims to categorise scientific paper titles into corresponding categories, the average input sentence length is much lower than others.

• •

  Dbpedia (Lehmann et al., 2015) is a large-scale multilingual knowledge base that contains 14 non-overlapping categories. Each category contains 40000 samples for training and 5000 samples for testing.

• •

  WebKB (Craven et al., 1998) is a long corpus web page topic classification dataset.

• •

  TREC (Li and Roth, 2002) is a question topic classification dataset to categorise one question sentence into 6 question categories.

The purpose of sentiment analysis is to analyse and mine the opinion of the textual content which could be treated as a binary or multi-class classification problem. The sources of existing sentiment analysis tasks come from movie reviews, product reviews or user comments, social media posts, etc. Most sentiment analysis datasets target to predict the people’s opinions of one or two input sentences of which the average length of each input text body is around 25 tokens.

• •

  Movie Review (MR) (Pang and Lee, 2005) is a binary sentiment classification dataset for movie review which contains positive and negative data equally distributed. Each review only contains one sentence.

• •

  Stanford Sentiment Treebank (SST) (Socher et al., 2013) is an upgraded version of MR which contains two subsets SST-1 and SST-2. SST-1 provides five fine-grained labels while SST-2 is a binary sentiment classification dataset.

• •

  Internet Movie DataBase (IMDB) (Maas et al., 2011) is also an equally distributed binary classification dataset for sentiment analysis. Different from other short text classification dataset, the average number of words of each review is around 221.

• •

  Yelp 2014 (Tang et al., 2015) is a large scale binary category based sentiment analysis dataset for longer user reviews collected from Yelp.com.

Certain binary sentiment classification benchmark datasets are also used by GNN-based text classifiers. Most of them are gathered from shorter user reviews or comments (normally one or two sentences) from different websites including Amazon Alexa Reviews (AAR), Twitter US Airline (TUA), Youtube comments (SenTube-A and SenTube-T) (Uryupina et al., 2014).

There are some datasets targeting other tasks including hate detection, grammaticality checking, etc. For example, ArangoHate (Arango et al., 2019) is a hate detection dataset, a sub-task of intend detection, which contains 2920 hateful documents and 4086 normal documents by resampling the merged datasets from (Davidson et al., 2017) and (Waseem, 2016). In addition, (Founta et al., 2018) proposes another large scale hate language detection dataset, namely FountaHate to classify the tweets into four categories including 53,851, 14,030, 27,150, and 4,965 samples of normal, spam, hateful and abusive, respectively. Since there is no officially provided training and testing splitting radio for above datasets, the numbers represented in Table 3 are following the ratios (train/development/test is 85:5:10) defined by (Lu et al., 2020).

Since an obvious limitation of corpus-level GNN models has high memory consumption limitation (Zhang and Zhang, 2020; Huang et al., 2019; Ding et al., 2020), the datasets with a smaller number of documents and vocabulary sizes such as Ohsumed, R8/R52, 20NG or MR are widely used to ensure feasibly build and evaluate corpus-level graphs. For the document-level GNN based models, some larger size datasets like AG-News can be adopted without considering the memory consumption problem. From Table 3, we could find most of the related works mainly focus on the GNN applied in topic classification and sentiment analysis which means the role of GNNs in other text classification tasks such as spam detection, intent detection, abstractive question answering need to be further exploited. Another observed trend is short text classification are gained less attention compared with long document classification tasks. In this case, GNN in short text classification may be an .

In terms of evaluating and comparing the performance of proposed models with other baselines, accuracy and F1 are most commonly used metrics to conduct overall performance analysis, ablation studies and breakdown analysis. We use $TP$, $FP$, $TN$ and $FN$ to represent the number of true positive, false positive, true negative and false negative samples. $N$ is the total number of samples.

• •

  Accuracy and Error Rate: are basic evaluation metrics adopted by many GNN-based text classifiers such as (Li et al., 2021; Liu et al., 2016; Wang et al., 2020; Yao et al., 2019; Zhang and Zhang, 2020). Most of the related papers run all baselines and their models 10 times or 5 times to show the $mean\pm standard$ $deviation$ of accuracy for reporting more convincing results. It can be defined as:

  (29)

  $$Accuracy=\frac{(TF+TN)}{N},$$

  (30)

  $$ErrorRate=1-Accuracy=\frac{(FP+FN)}{N}.$$

• •

  Precision, Recall and F1: are metrics for measuring the performance especially for imbalanced datasets. Precision is used to measure the results relevancy, while recall is utilised to measure how many truly relevant results acquired. Through calculating the harmonic average of Precision and Recall could get F1. Those three measurements can be defined as:

  (31)

  $$Precision=\frac{TP}{(TP+FP)},$$

  (32)

  $$Recall=\frac{TP}{(TP+FN)},$$

  (33)

  $$F1=\frac{2\times Precision\times Recall}{(Precision+Recall)},$$