OAG-Bench: A Human-Curated Benchmark for
Academic Graph Mining

Entity Alignment. Given two entity sets $E_{1}$ and $E_{2}$, the goal of entity alignment is to generate entity matchings $L=\{(e_{1},e_{2})|e_{1}\in E_{1},e_{2}\in E_{2}\}$ such that $e_{1}$ and $e_{2}$ refer to the same real-world entity. Specifically, we consider three types of entities, i.e., authors, affiliations, and venues. As for dataset annotation, we randomly sample a venue set and an affiliation set, and then manually label venue pairs with high similarity calculated by the Jaccard Index, and construct affiliation alignment pairs by using aliases or former names derived from the information box of their Wikipedia entries. We utilize Wikipedia due to its high data quality. Meanwhile, it allows us to accurately obtain positive affiliation alignment pairs without the need for manual labeling. For author alignment, we sample top-viewed computer science authors from AMiner, and then manually pair them with DBLP authors according to their affiliations, published venues, and papers. As a result, we construct $1{\small,}200$ venue pairs, $5{\small,}000$ affiliation pairs, and $10000$ author pairs.

Author Name Disambiguation (AND). Aiming to disambiguate the same-name authors, AND is a key and challenging task in academic knowledge graph construction. We adopt the WhoIsWho (Chen et al., 2023b) dataset, a million-scale human-annotated dataset for author name disambiguation. WhoIsWho breaks down the task into three subtasks: (1) From-scratch Name Disambiguation (SND), (2) Real-time Name Disambiguation (RND), and (3) Incorrect Assignment Detection (IND). While existing research primarily concentrates on SND and RND, the IND task has received less attention despite its growing importance with the expansion of academic databases. Given an author profile with paper lists, IND aims to detect incorrectly assigned papers to this author. To address the IND challenge, if we were to randomly select author profiles for annotation of their paper assignments, there’s a high likelihood that we would encounter numerous profiles with a little ambiguity. Thus, we propose an effective cross-checking annotation strategy. Specifically, we utilize existing paper alignments and author alignments between AMiner and DBLP, and then gather inconsistent paper-author assignments for further expert checking. This strategy ensures that the profiles under checking have a high likelihood of inaccuracy (with a significant error rate exceeding 30% for these inconsistencies within AMiner). Subsequently, all papers associated with AMiner authors that have incorrect assignments are manually checked by experts. Finally, the refined IND dataset includes $1{\small,}691$ authors, $326738$ papers, with an assignment error rate of $11.32\%$ and reaching $1.5$ times the scale of papers of the IND task in WhoIsWho.

Scholar Profiling. Profiling scholars from big data is a vital task in scholar mining, and it becomes harder and harder due to data fragmentation, modeling lengthy texts, data noise, etc. Previous works on scholar profiling usually extract attributes from scholars’ homepages or search engines. In OAG-Bench, besides profiling scholars from search engines, we introduce a new complex setting — Multidimensional Scholar Profiling from Long Texts, which aims to extract multiple attributes in lengthy texts. Each attribute extraction includes the starting and ending positions in the text. Importantly, long attributes are also taken into consideration, such as work experience and education experience. These attributes can often exceed 100 tokens. Traditional scholar profiling or named entity recognition tasks seldom focus on extracting such long attributes. For data annotation, scholars with detailed biographical descriptions are randomly sampled. Then, we manually label the starting and ending positions of each attribute in the texts. Finally, we construct $2{\small,}099$ scholars with $12$ attributes.

Entity Tagging. Aiming at associating entities with concept labels, entity tagging is an important step in building semantic and hierarchical academic graphs. We introduce scholar interest extraction and paper topic classification to attach concepts to scholars and papers, respectively. Scholar interest extraction aims to extract scholars’ research interests from their publications. Derived from 2017 Open Academic Data Challenge⁸⁸8https://www.biendata.xyz/competition/scholar/, the dataset of this task contains manually annotated $789$ research interest tags for $11357$ scholars and their papers. Paper topic classification aims to classify papers into several topics based on the paper citation network. For dataset construction, based on the DBLP paper citation network⁹⁹9https://originalstatic.aminer.cn/misc/dblp.v12.7z, each paper is assigned one of nine topics¹⁰¹⁰10https://numbda.cs.tsinghua.edu.cn/~yuwj/TH-CPL.pdf. The topics include high-performance computing, computer networks, network and information security, theoretical computer science, system software and software engineering, database and data mining, artificial intelligence and pattern recognition, computer graphics and multimedia, human-computer interaction, and pervasive computing. related to computer science based on its publication venue.

Concept Taxonomy Completion. Concept taxonomies are typically manually created by experts, like defining “deep learning” falls under “machine learning”. The automatic construction of concept taxonomies is a critical challenge in the fast-evolving landscape of knowledge concepts, which is beneficial to the organization of knowledge in the realm of big data. Given an existing concept hierarchy tree (Taxonomy) $T_{0}$ and a set of new concepts $C$, the goal of concept taxonomy completion is to predict its hypernym $pa(c)\in T_{0}$ and hyponym $ch(c)\in T_{0}$ for each new concept $c\in C$ to complete and expand the existing concept hierarchy tree. We adopt two MAG taxonomies (Sinha et al., 2015) (MAG-Full and MAG-CS) as two taxonomy datasets. These two datasets are large-scale but not carefully verified by experts. Thus, we introduce a newly manually curated dataset covering AI sub-fields by AI researchers, with $1{\small,}335$ concepts and $1{\small,}283$ edges. The guidelines of edge construction refer to relevant textbooks and the ACM Computing Classification System.

Paper Recommendation. As the volume of papers surges, researchers face increasing challenges in locating relevant literature. Given a user-paper bipartite graph $G=\{U,P,R\}$, where $U$ is the user set, $P$ is the paper set, and $R$ signifies interactions (e.g., clicks) between users and papers, the goal of paper recommendation is to predict the next paper a user will interact with (Zhang et al., 2023c). We collect user behavior data based on the real AMiner system. AMiner provides a real-time paper recommendation service for researchers on the homepage. Researchers can offer several keywords to subscribe to relevant research papers. The back-end recommendation engine makes recommendations based on the users’ historical click records. This dataset includes $5{\small,}340$ users, $14967$ papers, and $163084$ interactions as of October 2021. To ensure quality, only users/papers with over $10$ clicks/be-clicked instances are included.

Reviewer Recommendation. As the volume of submissions to academic journals and conferences increases, reviewer recommendation becomes increasingly hard. Different from paper recommendations, reviewer recommendation aims to pair papers with proficient and willing reviewers. Given a paper submission set $S$, a reviewer set $A$, and known paper-reviewer matches $R\subseteq S\times A$, this task is to predict the reviewer $a\in A$ for a new submission record $s_{i}\in S$, Additional information, including paper metadata and reviewer expertise, is available. For data collection, we extract real paper-reviewing records from the open-access platform Frontiers. After processing, it includes $210069$ reviewers and $225478$ papers, with each paper having at least $2$ reviewers. Furthermore, we match reviewers to authors in OAG using names, affiliations, and research interests, linking approximately half of the reviewers to the OAG. These reviewers are associated with their respective publications.

Academic Question Answering. Traditional keyword-based information retrieval cannot satisfy professional knowledge retrieval in the era of artificial intelligence. For instance, consider the question, “Can neural networks be used to prove conjectures?”. How to retrieve answers and evidence from scholarly literature? Given an academic question $q$ and a paper set $P^{q}=\{p^{q}_{1},p^{q}_{2},..,p^{q}_{N}\}$, the goal of academic question answering is to select the most relevant papers from the candidate set $P^{q}$. We adopt OAG-QA (Tam et al., 2022) dataset, which is derived from academic question-answering platforms. We retrieve question posts from StackExchange and Zhihu websites, extract the paper URL mentioned in the answer, and match it with the paper in OAG (Zhang et al., 2019). It comprises $17948$ question-paper pairs. Questions cover $22$ disciplines and $87$ topics, forming a two-level hierarchical structure; that is, each topic belongs to a discipline. For each topic, $10000$ candidate papers, including the ground-truth papers in the answers, are included.

Paper Source Tracing (PST). Tracing the sources of papers is crucial for understanding technological essence and uncovering innovation patterns. Given a paper $p$ (including its full text) and its references, the goal of PST is to identify the most important references (termed ref-source) that largely inspired the paper $p$ in terms of ideas or methods. The source papers of a given paper are defined by the following principles: (1) the main idea of the paper $p$ is inspired by the reference; or (2) the main method of the paper $p$ comes from the reference. In other words, this paper would not come into being without these source papers. We carefully build a dataset PST-Bench for this task. Given the specialized knowledge required for paper source tracing, dozens of computer science graduate students were employed to mark the sources of papers in their familiar fields. The annotation process was organized in an online paper reading group, where each student needed to share two papers and mark their source papers each week. After collection, expert-checking, and preprocessing, $2{\small,}141$ labeled computer science papers were obtained. We conducted a human evaluation on the test set, with senior researchers double-checking 100 papers. The accuracy rate was 94%.

Academic Influence Prediction. Paper influence prediction aims to forecast a paper’s impact $\Delta yr$ years later based on its metadata and citation relationships. Regarding the “Test-of-Time Paper Award” (TOT award) as an indicator of high impact, we collate TOT awards in computer science venues¹¹¹¹11https://numbda.cs.tsinghua.edu.cn/~yuwj/TH-CPL.pdf. Awards with similar meanings include the Most Influential Award, Sustained Influence Award, etc. At present, a total of $1{\small,}063$ papers awarded by 2022 have been collected. Similarly, author influence prediction seeks to predict an author’s influence $\Delta yr$ years later using their papers and citation relationships. For this task, we provide two datasets. First, we adopt AuthPred-2017¹²¹²12https://www.biendata.xyz/competition/Tsinghua_course3/. This dataset contains a subset of AMiner authors and papers published by these authors before 2011, intending to predict the citations of these authors as of 2016. This dataset uses AMiner’s citation statistics and provides $1112931$ authors for training and $123823$ authors for testing. In addition, we construct a new dataset AuthPred-2022. This dataset contains a subset of AMiner’s authors in the field of computer science and papers published by these authors before 2017, with the goal of predicting the citations of these authors as of early 2022 (as calculated by Google Scholar). This dataset contains $26797$ AMiner authors with Google Scholar links.

Baselines. We adopt graph-based anomaly detection methods and LLM-based methods as baselines. For each author, graph-based methods first construct a paper similarity graph based on attribute similarity (e.g., co-authorship, co-organization) and then detect anomalies in the graph. (1) Logistic Regression (LR): injects top eigenvectors of each graph as features to perform node classification. (2) GCN (Kipf and Welling, 2016): employs graph convolutional networks as the encoder, and then uses fully-connected layers to classify normal/abnormal nodes. (3) GCCAD (Chen et al., 2023a): leverages graph contrastive learning and contrasts abnormal nodes with normal ones in terms of their distances to the global context. (4) ChatGLM (Du et al., 2022): finetunes ChatGLM-6B model by inputting each author’s paper list and asking the model whether one given paper is an anomaly or not.

Evaluation Metrics. Due to the imbalance between positive and negative instances, we adopt the widely-used metric AUC. Furthermore, we choose mean average precision (MAP) as another metric, which pays more attention to the rankings of incorrect instances. We take a macro average of each metric for each author.

Experimental Results. Table 3 shows the performance of incorrect assignment detection. We observe that graph neural network-based methods (GCN and GCCAD) outperform the traditional method (LR) based on eigenvalue decomposition. GCCAD explicitly contrasts abnormal paper nodes with other nodes, yielding better performance than GCN. Surprisingly, ChatGLM outperforms graph-based anomaly detection methods, indicating the potential of the attention mechanisms in LLMs to capture the complex correlations between the target paper and the overall author profile. The best performance of IND is not that satisfactory compared with that of SND and RND tasks (Chen et al., 2023b), suggesting that more attention should be paid to the IND task for author name disambiguation in the future.

Baselines. We select the latest NER methods based on pre-trained models: (1) Han et al. (Yan et al., 2023): use a Biaffine decoder to generate features for each start and end position and then employ CNN to classify locations based on spatial position dependence. (2) GlobalPointer (Su et al., 2022): uses a multiplicative attention mechanism to incorporate relative positional encodings of start and end positions and alleviates class imbalance via modified loss functions. (3) UIE (Lu et al., 2022): is a generative pre-trained model based extraction framework with structure extraction languages and template-specific prompts.

Evaluation Metrics. Precision, Recall, and F1 are computed by comparing predicted and annotated text segments for each attribute. These individual attribute results are then averaged to obtain the overall evaluation result.

Experimental Results. Table 4 display extraction results for long text-based scholar profiling. In the context of scholar profiling from long texts, Table 4 reveals that span-based methods like Han et al. surpass generation-based methods like UIE. We also conduct preliminary experiments by giving some demonstrations and calling GPT-4 (Achiam et al., 2023) API on a subset of test sets, achieving only less than $5\%$ F1 score. This performance disparity likely stems from the challenges that language models face when generating accurate lengthy texts directly for attributes such as education/work experiences. However, with the highest F1 score in Table 4 being 45.09%, there is still room for improvement in extraction performance. Exploring the fusion of large language models (LLMs) and span-based methods presents a promising research avenue.

Baselines. For scholar interest extraction, we employ competition-winning solutions and methods relying on pre-trained models. These approaches follow a common principle: they gauge the similarity between authors in the test and training sets, using the weighted interest tags of training authors for the authors in the test set. The variations among baselines lie in how they compute author similarity. (1) LSI (Deerwester et al., 1990) employs bag of words and TF-IDF for paper texts, reducing dimensions with the LSI model. (2) ACA¹⁴¹⁴14https://github.com/geekinglcq/aca: utilizes more paper attributes, including titles, citations, and venues, for a more nuanced author similarity calculation. (3) pre-training models: leverage models like Sentence-BERT (S-BERT) (Reimers and Gurevych, 2019), SimCSE (Gao et al., 2021), E5 (Wang et al., 2022), BERT (Devlin et al., 2019), GTE (Li et al., 2023), BGE¹⁵¹⁵15https://github.com/FlagOpen/FlagEmbedding, and Sentence-T5 (S-T5) (Ni et al., 2022) to encode paper texts for similarity measurement.

Evaluation Metrics. For scholar interest extraction, we calculate the overlap ratio between predicted and ground-truth tags.

where $N$ is the number of scholars, $T_{i}^{*}$ is the annotated interest set of the $i$-th scholar, and $T_{i}$ is the predicted interest set of the $i$-th scholar. We pick the $3$ closest tags to the author for evaluation. For paper topic classification, we measure multi-classification accuracy.

Experimental Results. Figure 3 presents the results of scholar interest extraction. Initial attempts to classify scholars’ paper texts using research interest tags as labels yielded unsatisfactory results, likely due to the large number of tags and limited training data. The methods compared in Figure 3 rely on author similarity to calculate interest tags, which proves more effective than text classification. Notably, encoding with pre-trained models directly is less effective than LSI, highlighting the effectiveness of shallow semantic models. Additionally, models focused on sentence embedding outperform general pre-trained models like BERT. The ACA method, which leverages various author attributes such as venues and citing papers, yields improved prediction results. However, the overall accuracy remains low, indicating a challenge in accurate classification with a large number of interest tags.

Baselines. We compare various recommendation algorithms: (1) TF-IDF, (2) Toronto paper matching system (TPMS) (Charlin and Zemel, 2013) for reviewer recommendation task, (3) Variational autoencoder (VAE)-based item-based collaborative filtering method Mult-VAE (Liang et al., 2018), (4) Graph filtered-based collaborative filtering method GF-CF (Shen et al., 2021), and (5) Graph neural networks (GNN)-based collaborative filtering methods NGCF (Wang et al., 2019) and LightGCN (He et al., 2020).

Evaluation Metrics. Like the standard recommendation task (Zhang et al., 2023c, 2024), we adopt Recall@20 and NDCG@20 as evaluation metrics for paper/reviewer recommendation.

Experimental Results. Table 5 shows the paper recommendation performance. GF-CF outperforms other methods, highlighting the effectiveness of graph filters. GNN-based methods exceed Mult-VAE, demonstrating the value of high-order graph structures. LightGCN performs better than NGCF, which confirms the redundancy of some GNN modules in NGCF. However, there is room for improvement in recommendation accuracy. How to use the attributes of papers and users to capture the dynamic interest changes of users is a difficult point.

Baselines. We adopt sparse and dense retrieval methods: (1) Sparse retrieval methods: BM25, (2) Dense retrieval methods: DPR-FT (full fine-tuning of Dense Passage Retriever (DPR) (Karpukhin et al., 2020)), DPR-PT2 (parameter-efficient fine-tuning of DPR with P-Tuning v2 (Liu et al., 2022a)), ColBERT-FT (full fine-tuning of ColBERT (Khattab and Zaharia, 2020)), ColBERT-PT2 (parameter-efficient fine-tuning of ColBERT with P-Tuning v2), and LLM-Embedder (Zhang et al., 2023b) (a fine-tuned LLM based on various retrieval-related tasks).

Evaluation Metrics. Hit@K is used to measure retrieval accuracy, reporting if the top $K$ retrieved papers contain the correct answer. The average Hit@K across all questions is reported.

Experimental Results. Figure 4 presents the results of the OAG-QA dataset. Generally speaking, dense retrieval methods outperform sparse retrieval methods. ColBERT-based methods are significantly better than DPR-based ones. This shows that by employing late interaction patterns and multi-vector representations, ColBERT models the correlation between questions and papers better. Interestingly, efficient parameter fine-tuning methods excel over full fine-tuning, possibly due to better knowledge retention and generalization from the pre-trained model. The effect of LLM-Embedder suggests there still exists noticeable gap between LLM and academic retrieval. Overall, these methods’ retrieval effects are suboptimal, suggesting room for improvement.

Baselines. We compare three types of methods. (1) Statistical methods: Rule (employing regular expressions to extract references appearing near signal words like “motivated by” or “inspired by”), and Random Forest (RF) (following (Valenzuela et al., 2015), extracting statistical features about citations, citing positions, text similarity, etc., and using RF to predict the importance of references). (2) Graph-based methods: LINE (Tang et al., 2015) and NetSMF (Qiu et al., 2019) train paper embeddings in citation networks and then calculates the cosine similarity between the paper embedding and the reference embedding to measure the importance of references. (3) Pre-training methods: extract the contextual text where each reference appears in the full texts, encode the text with the pre-training models, and use the reference annotation results in the training set for fine-tuning. The pre-training models considered include BERT (Devlin et al., 2019), SciBERT (Beltagy et al., 2019), Galactica-standard (Taylor et al., 2022), and GLM (Du et al., 2022). We also adopt three SOTA closed-source models: GPT-3.5 (OpenAI, 2022), GPT-4 (Achiam et al., 2023), and Claude-instant (Anthropic, 2023). For both open-source and closed-source LLMs, we input the context of a referenced paper and query the model to assess the reference’s significance. For instance, we ask, “Given the context …, is the current reference important?” Closed-source LLMs perform this task using zero-shot evaluation.

Evaluation Metrics. A paper may have one or more ref-sources. For each reference of the paper $p$, an importance score between $[0,1]$ needs to be output. For each paper $p$ to be traced, its reference list is encoded as 0-1 based on the labeling results (1 if it’s ref-source, 0 otherwise). By comparing the prediction result of each reference with its labeling result, we compute the Mean Average Precision (MAP). The average MAP across different papers serves as the evaluation metric.

Experimental Results. Figure 5 presents the results of paper source tracing. Among all methods, SciBERT delivers the best performance, indicating the efficacy of pre-trained language models. RF outperforms the Rule method, underscoring the effectiveness of feature engineering. Graph-based methods achieve average performance, possibly owing to the ignorance of the contextual information of references. The Rule-based approach’s performance is subpar, likely due to many important references lacking surrounding signal words like “inspired by”, resulting in a low recall. Surprisingly, finetuned SciBERT and BERT-base outperform larger models like GLM-2B, Galactica-standard, and closed-source LLMs. The reason may lie in two aspects. First, the training objective of the mask language model is more suitable for this context understanding task. Second, API-based models may not be well-trained on similar tasks. A potential future direction could involve merging graph-based and text-based methods for paper source tracing. Note that the current methods’ results are not yet satisfactory, indicating ample room for further exploration.

Baselines. We select the following methods: (1) Citation: is based on the paper citation number of known years; (2) Random Forest (RF) (Breiman, 2001): defines features as the paper citation number per year and the total number of citations; (3) GBDT (Friedman, 2001): uses the same features as RF; (4) PageRank (Page et al., 1999): calculates papers’ PageRank score based on paper citation networks; (5) GraphSAGE (Hamilton et al., 2017): performs semi-supervised classification on the paper citation network. Additionally, we consider graph-based node importance prediction methods: (6) GENI (Park et al., 2019) and (7) RGTN (Huang et al., 2021).

Evaluation Metrics. We predict for each venue to determine whether a paper would be awarded, with labels being $0$ oder $1$ indicating whether the paper is awarded or not. Mean Average Precision (MAP) is calculated by comparing the predicted probability of winning the award with the ground truth label, and the mean MAP across different venues is used as the evaluation metric.

Experimental Results. Table 7 presents results for paper influence prediction. Table 7 shows PageRank performing best, as it considers the influence of citing papers, unlike the citation method that treats each citing paper equally. Traditional classifiers (RF and GBDT) are inferior to methods using only total citations, indicating that total citations are a very important indicator. The features added by the classifier may dilute the effect of total citations. GraphSAGE’s poor performance may be due to its inability to capture paper-influence factors like citation count. GENI outperformed GraphSAGE, but both methods were less effective than Citation and PageRank methods. This could be due to their implicit incorporation of citation statistics and the severe class imbalance problem (positive vs. negative $<1:100$). Thus, identifying factors beyond citation count remains a challenge in predicting papers’ breakthrough innovation.