Applying BioBERT to Extract Germline Gene-Disease Associations for Building a Knowledge Graph from the Biomedical Literature

In medicine, biomedical knowledge can be divided into many subdomains, such as genes, chemical compounds, diseases, organs, symptoms, and syndromes. The purpose of biomedical ontology goes beyond collecting names of entities, a dictionary of terms, and controlled vocabulary for a variety of entities. It defines biological classes of entities and the relations among them for building a knowledge base (Bodenreider et al., 2005). A well-defined ontology is essential for the creation of a biomedical knowledge graph because the ontology enables complex reasoning about biomedical knowledge. Some of these KGs, e.g. GARD (Zhu et al., 2020) and GenomicKB (Feng et al., 2022) have made significant contributions to the integration and utilization of existing biomedical knowledge regarding rare disease information sources, human genome, epigenome, transcriptome, and 4D nucleome to provide patients with the latest health information to answer human genomics-related questions.

BioPortal (Whetzel et al., 2011), an open repository of biomedical ontologies, has more than 1,000 ontologies and 15 million classes of entities. These ontologies have been designed and developed by the community of research teams to summarize and organize information. Maintaining an ontology through its life cycle is infeasible for a human expert since it is expensive and time-consuming. In addition, the difficulty is compounded by the fact that high-quality and scalable ontologies require reusing parts of other ontologies and applying automated quality control testing that guarantees best practices for software development (Matentzoglu et al., 2022).

Managing the increased rate of publications via manual curation is infeasible, requiring approaches that can automate part or all of the process. Natural Language Processing is commonly used to extract entities and their relations from biomedical text. Therefore, NLP can facilitate and automate knowledge graph construction. NER and NEN approaches have been developed to find relevant entities and connect these entities to meet the agreed data model (Milošević and Thielemann, 2023). Biomedical named entity recognition and named entity resolution techniques have been studied since the late 1990s (Fukuda et al., 1998), and different approaches have been proposed and developed to solve NER systems. These approaches can be classified into (1) rule-based, which relies on linguistic experts designing accurate rules, (2) machine learning-based, such as Hidden Markov Models (HMM) and Conditional Random Fields (CRF), (3) deep learning-based such as RNN, LSTM, CNN, and pre-trained language models, and (4) hybrid approaches (Li et al., 2018; Pawar et al., 2017; Yang et al., 2021a; Devlin et al., 2018).

Due to significant advances in deep learning, pre-training allows the model to incorporate domain-specific knowledge, which can further improve the ability of the pre-trained base model to achieve high performance on various tasks (Devlin et al., 2018). The model can be fine-tuned on task-specific datasets to perform tasks, such as named entity recognition and relation extraction (Lee et al., 2019; Beltagy et al., 2019; Chithrananda et al., 2020), which are two critical tasks to construct domain-specific knowledge graphs. In contrast with a manual curation approach, pre-trained models can reduce computing costs, and save time and resources. For example, BERT-based models can be used to generate scalable knowledge graphs from new corpora that include undiscovered knowledge (Verma et al., 2023).

A recent study, HerbKG (Zhu et al., 2022), a knowledge graph that bridges herbal and molecular medicine, uses text mining techniques, such as PubTator Central (PTC) NER model and a custom BERT-based RE model to produce a list of identified relation triplets, which are used for the HerbKG construction. The constructed HerbKG supports multiple downstream applications, such as descriptive analysis, evidence-based graph query, similarity analysis, and drug repurposing. Other studies (Milošević and Thielemann, 2023; Yang et al., 2021b) have attempted to apply machine learning algorithms and BERT-based models to extract relationships, such as Drug-Gene, Drug-Disease, and Gene-Disease, and have a clearer understanding of diseases, symptoms, and gene mutations.

Due to the complexity of properly defining and categorizing a large number of biomedical terms, we rely on two home-grown dictionaries (MUSC), one includes a list of diseases, and the other a list of genes. These dictionaries are relevant for mapping genes and diseases to a master term. The dictionary of diseases contains around 125 disease names and 452 synonyms, and the dictionary of genes contains around 336 gene names and 1,310 synonyms.

Tokenization is the process of breaking down unstructured data and natural language text into smaller units of information (Webster and Kit, 1992). For instance, sentences, punctuation marks, words, and numbers can be considered tokens. Large input sizes for machine learning models are not recommended, especially BERT-based models (Devlin et al., 2018), which have an input size restriction of 512 characters. Considering an abstract of a research paper is usually a paragraph of 300 words or less, an abstract may have over 512 characters with spaces included in the character count. To solve this problem, we used the PunktSentenceTokenizer (Marcus et al., 1993) method from the NLTK python library, which is trained on the Penn Treebank corpus and uses regular expressions to parse sentences and detect the sentence boundaries.

We used a fine-tuned BioBERT model pre-trained on NCBI-disease corpus (Abreu Vicente, 2022b) to extract diseases from germline abstracts. The Natural Center for Biotechnology Information Disease (NCBI) (Doğan et al., 2014) disease corpus is a collection of 793 PubMed articles with 6,892 manually annotated disease mentions. For extracting genes from germline abstracts, we used a fine-tuned BioBERT model pre-trained on the BC2GM corpus (Abreu Vicente, 2022a). The BioCreative II Gene Mention (BC2GM) corpus (Smith et al., 2008) consists of sentences from PubMed abstracts with manually labeled gene and alternative gene entities.

After extracting single and multi-word phrases from texts, Named Entity Normalization (Cho et al., 2017) is performed, which allocates suitable tags to the recognized entities. In biomedical articles, named entity normalization is a challenging task because biological terms, such as genes and diseases have multiple synonyms, and term variations, and are often referred to using abbreviations (Leaman et al., 2015). To resolve these ambiguities, machine-learning approaches (Neves et al., 2010) have been investigated. However, many normalization tools rely on domain-specific ontologies, dictionaries, or rules. Domain-specific dictionaries can differentiate between synonyms, abbreviations, and punctuation marks.

We used a dictionary-lookup approach using our two manually curated dictionaries and an approximate string-matching algorithm. The algorithm converts identified entities from text to lexical variations, such as lowercasing and removing whitespace and punctuations, and then maps them to specific master terms. To reduce the complexity of our Algorithm 1, we assume the BioBERT NER task classifies a single entity following the format BIO encoding scheme to represent the tokens and each entity that may be composed of several words must be unique. For example, the input sequence “BRCA1 and BRCA2” should be classified with the labels B-GENE, I-GENE, and B-GENE. This assumption allows us to map only one master term from the dictionary for each entity. However, an entity may be mapped to more than one master term because the BioBERT NER task may classify the previous example with the labels B-GENE, I-GENE, and I-GENE.

Given a pair of entities, such as a gene and disease, a semantic relation consists of identifying the relation type between them. An important semantic relation for many applications is the part-whole relation (Girju et al., 2006). Let us notate the part–whole relation as PART (¡Tail Entity¿, ¡Head Entity¿), where ¡Tail Entity¿ is part of ¡Head Entity¿. For example, the phrase “genes are found on tiny structures called chromosomes” contains the part-whole relation PART (genes, chromosomes). More recent studies, such as the SemEval 2018 Task 7 (Gábor et al., 2018), proposed a task on semantic relation extraction and classification in scientific paper abstracts that are practical for working on extracting specialized knowledge from domain corpora, such as biomedical information extraction.

Successful entity-relation linking requires detecting both the entity mentions in the abstracts, along with their respective entity types from the gene-disease dictionaries, and determining the type of relationship that exists between them. Based on psycholinguistic experiments and how the entities contribute to the structure of the part-whole relationship, we determined that the part-whole relationship from SemEval 2018 Task 7 can help us better identify and connect our entities to build the knowledge graph. SemEval 2018 Task 7 provides three comprehensive sets of classification rules, (1) composed of, (2) data source, and (3) phenomenon (Gábor et al., 2018).

Our main dataset contains germline abstracts and their PubMed ID. An abstract can include more than one gene and disease mentioned per sentence. Because germline mutations are passed on from parents to offspring, it is complicated to establish causal relationships in the germline association between genes and diseases, and thus they are not well-defined in the literature (Bonifaci et al., 2010). Therefore, we use a data source relationship in the form of PUBMED_ID-GENES_IN-GENE and PUBMED_ID-DISEASES_IN-DISEASE. Our approach matches all given genes and diseases to their given PubMed ID.

Our experiments are conducted on germline corpora, which contain 11,261 abstracts from PubMed, and 114,311 sentences after tokenization. The BioBERT-NER approach identified 19,751 gene entities and 19,135 disease entities. We also detected that most of the gene and disease entities are synonyms and therefore, are part of the same entity. To eliminate the data ambiguities, we applied an ad hoc mapping and filtering procedure (Algorithm 1). We also eliminated entities that represent objects that did not match our ontology. Then, we formally defined a semantic relation type to be a pair consisting of a domain class of type PART-DATASOURCE (Gene, PubMed ID) and PART-DATASOURCE (Disease, PubMed ID). We defined the semantic relation as “GENES_IN”, and “DISEASES_IN” to capture the connection between a PubMed ID and genes and/or diseases. Once we identified the disambiguated entity types and relationships, we linked them together and built the knowledge graph. The knowledge graph was built with the Neo4j graph platform and contains 46,747 triples with 9,414 entities, including 8,987 PubMed IDs, 297 genes, and 130 diseases.

BioPortal (Whetzel et al., 2011), an open repository of biomedical ontologies, has more than 1,000 ontologies and 15 million classes of entities. These ontologies have been designed and developed by the community of research teams to summarize and organize information. Maintaining an ontology through its life cycle is infeasible for a human expert since it is expensive and time-consuming. In addition, the difficulty is compounded by the fact that high-quality and scalable ontologies require reusing parts of other ontologies and applying automated quality control testing that guarantees best practices for software development (Matentzoglu et al., 2022).

So far, SimpleGermKG which covers germline abstracts from PubMed has been constructed with an integrated ontology of genes and diseases. We store the SimpleGermKG in the Neo4j graph database, which allows researchers and clinicians to find relevant information and facilitates the navigation of biomedical data. To prove the visual management and ease-to-query of Neo4j, we show the search results of two queries through the Neo4j Cypher graph query language in Figure 2(b). The blue node represents “PubMed ID” (abstract ID), the green node is “gene” (disambiguated gene names), and the red node is “disease” (disambiguated diseases) mentioned in the text. The “gene” and “disease” nodes can be also identified with the relationship (edge) “GENES_IN” and “DISEASES_IN” respectively. Figure 2(a) shows articles that mentioned the disease “Teeth (Benign)” and all gene entities mentioned in the abstracts. Figure 2(b) shows genes and diseases mentioned in the abstract of the article “9024708”.