Classification of geological borehole descriptions using a domain adapted large language model

Emerging advances in the development of large language models have significantly impacted the field of natural language processing (NLP) by enabling machines to generate and comprehend human-like texts. In particular, recent studies have been exploring the potential of NLP to capture geological information from unstructured text. Both non contextual models with static word embeddings such as Word2Vec and GloVe are employed [Lawley et al., 2022, Padarian and Fuentes, 2019] to this aim, as well as more advanced contextual transformer architectures [Vaswani et al., 2017b] (e.g., BERT [Devlin et al., 2018]). Transformers have reshaped the NLP domain by introducing an innovative architecture that outperforms traditional methods like recurrent neural networks (RNNs) both in terms of accuracy and scalability [Vaswani et al., 2017a, Howard and Ruder, 2018]. Unlike RNNs, which process inputs sequentially, transformers employ a self-attention mechanism that allows them to dynamically weigh the importance of different words in a sequence. This mechanism facilitates the generation of contextual embeddings, allowing the model to concentrate on various segments of the input sequence when producing output representations. Consequently, transformers capture long-range dependencies more effectively than RNNs. These advancements have democratized access to pretrained contextual large language models and catalyzed their adoption across diverse domains (including geosciences).

The main goal of many of the NLP studies within the field of geology is to perform text mining on geological archives and annotate them with meaningful semantic entities. For this purpose, named entity recognition (NER) pathways are frequently developed [Heaven et al., 2020, Enkhsaikhan et al., 2021b, Morgenthaler et al., 2022, Qiu et al., 2019]. In addition, LLM’s have been applied to a range of downstream tasks, such as article summarization [Ma et al., 2022], translation [Gomes et al., 2021], prospectivity modeling [Lawley et al., 2023] and mineral exploration [Enkhsaikhan et al., 2021a]. Although English remains the reference language in much of this research, similar applications can be found in other languages such as Chinese [Li et al., 2021, Qiu et al., 2018] and Portuguese [Consoli et al., 2020, Gomes et al., 2021].

As far as the classification of borehole descriptions is concerned, the study by Fuentes et al. [2020] is of specific relevance. In their study, a GloVe model trained on a large corpus of articles in the geosciences domain is used to obtain embeddings of drill core descriptions. Next, a multilayer perceptron neural network is trained to classify them. Finally, they created 3D maps by interpolating the embeddings. Fuentes et al. [2020] utilize a non-contextual model with static word embeddings.

Research has shown that LLMs like BERT, primarily trained on general English language corpora, face inherent limitations when applied to domain-specific classification tasks involving texts in other languages. The primary reason is that these models have not been exposed to sufficient examples of technical language in the target language. This limits their effectiveness to recognize and process such texts. Monolingual models such as Camembert [Martin et al., 2019] for French and BERTje [De Vries et al., 2019] for Dutch outperform the multilingual BERT model on downstream NLP tasks [De Vries et al., 2019, Martin et al., 2019]. More recently, ChatGPT represents a significant advancement in the field of natural language processing with its advanced generative capabilities and ease of accessibility. Its design allows for flexible interaction through prompt engineering: a method that requires considerably less technical expertise compared to the complexities involved in finetuning models like BERT. This accessibility makes ChatGPT an attractive option for scientists who may not specialize in machine learning or who do not have access to specialized GPUs. This provides a more user-friendly approach to harness the power of large language models for domain-specific tasks [OpenAI et al., 2024].

In this study we develop GEOBERTje: a domain adapted version of BERTje [De Vries et al., 2019] (a pretrained Dutch BERT model). The domain adaption is achieved by transfer learning on a large corpus of $283000$ unlabeled Flemish borehole descriptions. Subsequently, we use a much smaller set of approximately $2500$ labeled samples to fine tune this model to a lithology classifier that is able to extract multiple lithology classes from borehole descriptions. We compare the performance of the classifier based on GEOBERTje with those obtained using (a) traditional rule-based scripting and (b) GPT-4 of OpenAI through prompt engineering. Our study exemplifies how the adoption of domain-specific LLMs can provide significant added value in the field of geology by enhancing the efficiency and accuracy with which unstructured geological datasets can be analyzed.

The majority of Flemish borehole descriptions reside in Databank Ondergrond Vlaanderen (DOV²²2https://www.dov.vlaanderen.be), which is a partnership among various government entities concerned with Flanders’ subsurface. Initiated in 1996, it is now the main data holder for open Flemish data and information concerning geology, natural resources, soil, hydrogeology, geotechnical characteristics and groundwater licenses. It contains digitized archives of several federal, regional and research institutes and is constantly updated with new data [De Nil et al., 2020]. DOV offers unparalleled access to Flanders’ subsurface information with a relational database backbone, a web service for interactive querying and visualization and an API that supports machine-based extraction [De Nil et al., 2016, Haest et al., 2018]. In addition to serving as an information platform, it also acts as a prime data source for regional and cross-border 3D geological models [Deckers et al., 2019, Vernes et al., 2018]. A model type specifically in need of large volumes of subsurface data are the geological voxel models, composed of volumetric pixels that contain modeled data on the load of a specific lithology (peat, clay, gravel, etc.) [Van Haren et al., 2023]. At the base of these voxel models lie tens of thousands detailed standardized borehole descriptions. While the borehole descriptions are included in a structured format with the DOV database³³3For example, see https://www.dov.vlaanderen.be/data/interpretatie/2016-252085, the description intervals themselves are generally composed of unstructured text. In-house rule-based scripts were developed to classify them into detailed categories to enable direct usage of these descriptions into models. These rule-based scripts employ dictionaries and regular expressions to transform the descriptions into a primary, secondary and tertiary lithology class, as is displayed in Table 1 [van Haren et al., 2016, Van Haren et al., 2023]. While admixtures are generally captured as well, they fall outside the scope of this paper, as they are easily extractible using traditional rule-based text mining methods.

The geology of Flanders is characterized by a relatively flat landscape dominated by Quaternary deposits of sands, gravels, and clays, along with localized peat bogs. In the southern part of Flanders, a Pleistocene loess belt is present [van Haren et al., 2016]. These deposits overlie Neogene or Paleogene sediments, which are rich in sand and clay layers, deposited in various marine and continental Neogene and Paleogene environments. Beneath these, Mesozoic and Paleozoic deposits feature a complex assembly of older marine, peri-marine and continental rock formations, including limestone and shale. The shallow subsurface is hence almost entirely composed of soft sediments, with a predominance of sandy and clayey deposits. The lithology classes applied to categorise these soft sediments are adapted from Wentworth [1922] and given in Table 2. In this table different grain size classes of sand are distinguished, as each of these classes have different geotechnical characteristics and contrasting applications as raw materials [Flemish authorities, 2010]. In addition, a class sand without further grain size specification is included. Although the presence of this class poses challenges for subsequent lithological modelling tasks, it has to be present as many of the lithological descriptions do not mention grain size information. For example, green sand with clay and some gravel at the base.

The severe imbalance in the occurrence of these different lithology classes in Flanders can be visualized by inspecting the distribution of each lithology class for the main lithology or secondary lithologies as determined by the rule-based scripting. Figure 2(a) shows that clay (klei), and the different sand fractions (zand, fijn zand, middelmatig zand and grof zand) are the predominant main lithologies. It further reveals that secondary lithologies are less commonly documented, with tertiary lithologies being even rarer. This is logical given the fact that only the most detailed descriptions document up to three lithologies within a single lithological interval. Nonetheless, capturing these different levels of detail can be essential to fully understand the complexity of the shallow subsurface, as the detailed descriptions in Table 1 illustrate.

We selected a total of $341000$  borehole description intervals originating from $23000$  boreholes spatially spanning the entire territory of Flanders. Figure 1 shows the spatial distribution of the sampled borehole descriptions in Flanders and Brussels. Although boreholes are spread throughout Flanders, a larger concentration can be observed in more densely populated areas such as Brussels, Antwerp, Ghent, etc. This initial dataset contains descriptions in both Dutch and French.

The following preprocessing steps were applied to improve the quality of the dataset: (i) removing all French descriptions; (ii) removing descriptions pertaining to multiple depths of the borehole; (iii) replacing idem with the borehole description immediately preceding it. The final dataset contains $283000$  borehole descriptions. The average length of the lithological descriptions is $19$ words with a standard deviation of $26$. The minimum and maximum number of words per description is $1$ and $336$.

Labeled data are required to develop the lithology classifier. Rule-based scripts are available to directly classify this dataset into a main and secondary lithologies (i.e., second and third) [Van Haren et al., 2023]. However, directly using these labels in training of a classification model would at best only hope to achieve a performance on par to these rule-based scripts. Therefore, a ground truth needs to be established through manual labeling by experts. In order to make most efficient use of their time, a limited subset needed to be selected for manual labeling. However, random sampling from a dataset with such an imbalanced distribution would replicate this bias leading to classifiers that can only perform well on the majority classes because they have not seen enough training observations of the minority classes (Figure 2(a)). Therefore, we first eliminated duplicate observations and then grouped the remaining data by (rule-based) lithology class to select descriptions for manual labeling. Subsequently, we randomly chose a similar number of samples from each of these groups. This resulted in $2671$ borehole descriptions that were manually labeled. Finally, we partitioned this dataset into three subsets: $15$% for testing, $15$% for validation, and the remaining $70$% for training.

Figure 2(b) shows the lithology class occurrence (as labeled by the rule-based script) of the selected subset. The class distribution is still skewed but far less so than before. This subset was subsequently labeled manually by experts.

We start from BERTje of De Vries et al. [2019].⁴⁴4The model is available here on the Hugging Face platform. BERTje uses the same model architecture as the original BERT model of Devlin et al. [2018]. BERT is trained on a multilingual text corpus. While only Wikipedia articles in Dutch belong to the Dutch training corpus of multilingual BERT, BERTje was trained on a large and more diverse corpus of 2.4 billion Dutch tokens originating from Dutch books, news, etc [De Vries et al., 2019]. Intuitively, BERTje has an improved performance for the Dutch language compared to multilingual BERT.

We could directly train BERTje with labeled data to perform the classification task. However, since BERTje’s training data is not tailored specifically to geology, refining its understanding of geological subtleties can be achieved through additional training on an unlabeled dataset focused on lithological descriptions. This method is known as domain adaptation (DA). Instead of training a LLM from scratch, it allows further training on data from a particular domain [Guo and Yu, 2022]. This pre-training step enhances the model’s performance by improving the vector embeddings. This aligns the embeddings more with the downstream classification task (see subsection 4.1).

DA techniques prove particularly advantageous in scenarios where there is a scarcity of labeled data for a target task but unlabeled data is abundantly present. Training in this approach relies on the masked language model (MLM) objective and does not require labeled data. It involves randomly masking words in the descriptions and predicting the masked word using the model. The loss function value is then calculated using the cross-entropy between the labels and logits. Consequently, the model becomes adept at predicting masked words from their context and recognizing underlying patterns and long-range dependencies within the lithological descriptions. This step effectively leverages the vast amounts of unlabeled lithological descriptions available, enriching the model’s training process [De Vries et al., 2019]. Table 3 illustrates the process of applying random masking to an example sentence, “weinig fijn zand met grindelementen” ⁵⁵5slightly fine sand with shell gravel elements.. The tokens resulting from this process are subsequently utilized in masked language modeling. It is important to note that tokens with a label value of $-100$ are disregarded by the loss function. Thus, they do not contribute to the training process and ensure that the focus remains solely on the missing words in the sentence.

We employed a masking probability of $0.15$, which was randomly applied to input sample data on-the-fly, with training batch sizes of $32$ and a learning rate of $0.0005$.

Figure 4 illustrates how the loss values for both the training and validation sets evolve in relation to the number of epochs (i.e., training iterations) during the MLM training. Initially as the model learns from the data, there is a notable decline in loss values: this indicates rapid improvement in predicting masked tokens within the training data. This decline stabilizes as the training progresses: this reflects the model’s convergence towards optimal performance. The loss values for the validation set exhibits a similar trend, albeit with occasional fluctuations. It must be noted that while training loss reflects the model’s fit to the training data, validation loss assesses its ability to generalize to unseen data. If validation loss stagnates or starts to increase, indicating potential overfitting, then early stopping mechanisms are employed. This involves halting training if there is no improvement in validation loss over a predefined number of epochs. This prevents the model from overlearning the training data and ensuring better generalization performance. The inset in Figure 4 displays the progression of the learning rate during training. Initially, we implemented a warm-up learning rate to ensure the model’s stability and mitigating the potential loss of previously acquired knowledge from the BERTje base model. Subsequently, we applied learning rate annealing to enhance the model’s final performance. Following this step, we possess a pre-trained base language model for Dutch texts that is already proficient in lithological descriptions. This serves as a foundation model for the next finetuning stage and is particularly tailored to the lithology classifications task using labeled data.

After the initial training phase of domain adaptation, the subsequent step involved finetuning the domain adapted model using a limited set of labeled data [Howard and Ruder, 2018]. For the classification task, each lithological description accordingly needs to be assigned three labels: a main lithology and up to two secondary lithologies. “None” is used if no label was present. We chose a simple approach by training three separate classification models for each of the three label groups (illustrated by the yellow parallel boxes in “Stage 2: Finetuning” of Figure 3). This means that, starting from the same pretrained base model, all three classification models operate independently (and parallel) from each other. For each classification model a classification head is attached to the pretrained base model by adding a linear layer on top. The three models are further finetuned on the labeled training data to minimize the cross-entropy loss. In order to account for potential class imbalances in the labeled data, we utilized class weights separately for each of the three target labels (see Equation 1). Because class weights assign higher importance to underrepresented classes during training, they reduce the impact of class imbalances and ensure that the model learns to generalize evenly well across all classes.

The cross-entropy loss function with class weights is defined as:

where $N$ spans minibatch dimension, $C$ is the number of classes, $y_{ij}=1$ if the sample $i$ belongs to class $j$, otherwise $y_{ij}=0$. $\hat{y}_{ij}$ is the softmax predicted probability of sample $i$ belonging to class $j$. $w_{j}$ is the weight assigned to class $j$ [Bengio, 2016]. In our experience, using class weights yields more accurate results compared to oversampling or undersampling methods to deal with uneven data distributions. Figure 5 sets out the cross-entropy loss on the training and validation set as a function of the number of epochs for the main, second and third lithology model. While the training loss exhibits an ever decreasing trend with the number of epochs, the validation loss first decreases up to an inflection point before increasing again. This illustration of the bias-variance trade-off shows that the models learn from the data until at a certain point they start overfitting on the training data and their generalization ability starts to diminish. Therefore, training is stopped as soon as the loss on the validation set starts to increase. It must be noted that we reused the same learning rate strategy throughout, incorporating both warm-up and linear annealing techniques, with a maximum learning rate value set at $0.0005$. This process results in the final classification models tailored to the lithology classification task.

Two postprocessing steps were introduced to further improve the classifiers’ overall accuracy and to correct for the fact that the three parallel classifiers are not aware of each other’s prediction. First, a lithology class can occur only once in each classification result. If the predicted secondary (tertiary) lithology equals the predicted main (secondary) lithology, then these duplicate class predictions from the secondary lithologies are removed and replaced by the subsequent most probable class within the group. Additionally, only model predictions surpassing a confidence threshold of $\tau=0.1$ are considered valid. This threshold $\tau$ is determined so that it maximizes the accuracy score on the validation set, averaged over each classifier.⁶⁶6This value did not change when maximizing a different metric such as balanced accuracy or matthews correlation coefficient. If no classes meet this criterion, then “None” is predicted with confidence score $-1$.

We compare the performance of our domain adapted and finetuned GEOBERTje classifier to classification using the original rule-based script and GPT-4 through prompt engineering. Appendix A describes both approaches in more detail.

We finetuned the classification model directly from the generic BERTje base model (i.e., without domain adaptation) to asses the effect of domain adaptation on accuracy. We subsequently compare the performance of this approach to GEOBERTje. Figure 6 compares the classification accuracy of the final finetuned model using two different base models: GEOBERTje and the original BERTje. The results are presented separately for each label. This figure shows a negligible difference in the accuracy score for the main lithology. This suggests that geological context may not be crucial for predicting the main lithology. However, the domain adapted model demonstrates an improved accuracy for the secondary ($0.84$ vs. $0.8$ with BERTje as base model) and tertiary lithology ($0.77$ vs. $0.73$ with BERTje as base model). This highlights the benefits of employing a domain adapted model over a generic one in order to achieve better performance in downstream applications. The observed improvement is attributed to the domain adaptation process which enhances the model’s understanding of the geological context by taking advantage of unlabeled data. This adaptation is especially beneficial to more accurately classify secondary and tertiary lithologies.

We evaluate the efficacy of our finetuned GEOBERTje classifier against the traditional rule-based method and GPT-4 on lithology classification. First, we consider the classification accuracy – defined as the ratio of correctly classified observations over the total number of observations – on an unseen test set for the main and secondary lithologies. Figure 7 demonstrates that the GEOBERTje classifier outperforms the other models in all categories. It achieves a very high classification accuracy of $0.94$ on the main lithology. It exhibits much better performance over the rule-based and GPT-4 models by margins of $9$, $12$ and $6\%$ compared to the runner-up for the main, secondary, and tertiary lithologies respectively. While GPT-4 competes closely with the rule-based approach in main lithology classification and exceeds it in classifying the secondary lithology, it falls behind in accurately classifying tertiary lithology. A noticeable pattern is the diminution of accuracy across all models from main to tertiary lithology.

The confusion matrix in Figure 8 facilitates a more detailed analysis of the classification performance for the individual lithology classes. Diagonal entries reveal normalized classification accuracy on a class level, off-diagonal values provide insights on the classes with whom the classifier “confuses” the correct label. Figure 10 in Appendix B also contains the confusion matrices for the rule-based scripting and the GPT-4 model.

We investigated the impact of the labeled training set size on classification accuracy because the availability of sufficient ground truth labels represents an important challenge for the potential of LLMs to act as an alternative to more traditional classification approaches. This provides insights in whether we possess adequate labeled data for the model finetuning and how many additional samples are required to attain specific performance levels. Figure 9 represents the relation between the number of training data and the ability of the three models to correctly predict the main, second, and third lithologies. These results show that a relatively small number of training samples (e.g., $500$) suffice for GEOBERTje to achieve a reasonable accuracy of $0.90$ in predicting the main lithology. This suggests again that extensive geological context may not be necessary for main lithology prediction and elucidates why the zero-shot GPT-4 model yields a relatively high accuracy of $0.85$ in Figure 7.

The accuracy of predicting the main lithology gradually increases by adding more training data before plateauing at around $1500$ training observations. However, this does not apply to the secondary lithologies as the accuracies continue to exhibit a significant improvement. We found that increasing the training samples from $500$ to $2270$ yields a gain of approximately $20\%$ in accuracy. This underscores the significance of labeled data in attaining higher accuracy, especially for the second and third lithologies. Our results, based on the extrapolation of these logarithmic trends using $accuracy=\log size$ for the different lithologies, suggest an accuracy of $89.3\%$ would be achieved for $3000$ training observations for the second and $83.5\%$ at $3500$ training observations for the third lithology.

Future research can further enhance the model we developed. GEOBERTje’s domain adaptation would improve further with training on a larger dataset. Additionally, we illustrated that the classifier would benefit from additional labeled data specifically focusing on phrases describing at least two different lithologies. Finally, linking the three classifiers to make them aware of each other’s predictions could potentially improve the results. Our modeling approach can also easily be transferred to different languages. Moreover, the potential of GEOBERTje is broader than the lithological classification for which it is currently developed. The domain adaptation can be easily expanded by including a more diverse corpus of geological data (e.g. stratigraphical borehole interpretations or geological reports). This will result in improved embeddings that reflect geological context and semantic relations on a more holistic level. This would also open the door for a broader range of downstream applications such as lithostratigraphic mapping or resource estimation.