²²institutetext: FH Aachen – University of Applied Sciences, 52428 Jülich, Germany ²²email: {p.kohl,y.kraemer,kraft}@fh-aachen.de ²²institutetext: University of Kassel, 34121 Kassel, Germany ²²email: [email protected]

Scoping Review of Active Learning Strategies and their Evaluation Environments for Entity Recognition Tasks

Philipp Kohl 11 0000-0002-5972-8413 Yoka Krämer 11 0009-0006-7326-3268 Claudia Fohry 22 Bodo Kraft 11

Abstract

We conducted a scoping review for active learning in the domain of natural language processing (NLP), which we summarize in accordance with the PRISMA-ScR guidelines as follows:

Objective: Identify active learning strategies that were proposed for entity recognition and their evaluation environments (datasets, metrics, hardware, execution time).

Design: We used Scopus and ACM as our search engines. We compared the results with two literature surveys to assess the search quality. We included peer-reviewed English publications introducing or comparing active learning strategies for entity recognition.

Results: We analyzed 62 relevant papers and identified 106 active learning strategies. We grouped them into three categories: exploitation-based (60x), exploration-based (14x), and hybrid strategies (32x). We found that all studies used the F1-score as an evaluation metric. Information about hardware (6x) and execution time (13x) was only occasionally included. The 62 papers used 57 different datasets to evaluate their respective strategies. Most datasets contained newspaper articles or biomedical/medical data. Our analysis revealed that 26 out of 57 datasets are publicly accessible.

Conclusion: Numerous active learning strategies have been identified, along with significant open questions that still need to be addressed. Researchers and practitioners face difficulties when making data-driven decisions about which active learning strategy to adopt. Conducting comprehensive empirical comparisons using the evaluation environment proposed in this study could help establish best practices in the domain.

Keywords:

Scoping Review Active Learning Selective Sampling Entity Recognition Span Labeling Annotation Effort Annotation Costs NLP.

1 Introduction

Recent years showed significant advancements [98, 7, 19] in natural language processing (NLP): Large language models (LLMs) emerged [8], facilitating new methodologies by describing tasks in natural language without a strong formalism. Because resource-intensive LLMs are not always superior [112], smaller, supervised learning-based models are still highly relevant for specialized domains or use cases that require rapid inference or are constrained by hardware limitations (such as mobile devices or offline scenarios) [34].

One of these domains is entity recognition [73]. Entity recognition (ER) describes the task of assigning a label to a sequence of words (e.g. to extract a person, a date or any other predefined label). To apply supervised learning to ER, data must be annotated. The manual annotation process, in which humans annotate data points with these predefined labels, is time-intensive and expensive [106]. Its output is an annotated dataset, which is also called corpus (pl. corpora) in the NLP domain. We use the terms interchangeably in this paper.

Researchers have been exploring supporting methods to reduce the annotation effort, such as semi-supervised learning [93, 31], self-learning [94, 108], and active learning [4, 72]. Active learning (AL) is an approach to strategically or heuristically select data points for human annotation. This methodology can reduce the number of data points required to achieve competitive model performance compared to the classical annotation and training process. Thus, AL can decrease the time and cost of the annotation and training processes. However, the selection of an appropriate AL strategy is crucial. Selecting an inappropriate strategy can lead to lower performance compared to random data selection [10, 38].

Over the past two decades, researchers have developed many active learning strategies in the field of NLP for various scenarios. However, it is still challenging for researchers and practitioners to select a promising strategy for a given use case. While existing AL surveys provide taxonomies [72, 105, 106], there is still a lack of comprehensive performance analyses. Towards closing this gap, an overview of the domain can support researchers in conducting such analyses. Therefore, we executed a scoping review focusing on active learning strategies and their evaluation environments limited to the entity recognition task in NLP. We concentrated our review on model-agnostic strategies so researchers can use our results for a broad range of models. Our review answers the following review questions:

1.

Which model-agnostic AL strategies have been applied to ER?
2.
How did the researchers evaluate their strategies?
1. (a)
  
  Which datasets did they use?
2. (b)
  
  Which metrics did they use to compare AL strategies?
3. (c)
  
  How much time do the AL strategies need for initialization, proposing new data points to annotators, and model retraining (in case of exploitation) depending on the hardware?

We chose the ER task due to its complexity in the annotation process [17] and AL [72]. The complexity results from the ER model, which makes decisions for every token (e.g., word). Many AL strategies ( $>$ 80) compute the relevance of a data point based on these individual decisions.

For our review, we selected the format of a scoping review [25, 61] because we give an overview of the domain by identifying the research field’s available strategies, datasets, metrics, execution times, and hardware used. We also identified research gaps and open questions, which enables other researchers to conduct a systematic review with precisely defined research questions in the field of AL and ER based on our work¹¹1The preparation of systematic reviews is a strong hint for performing a scoping review..

We adhere closely to the PRISMA-ScR [95] reporting schema and checklist. The schema provides detailed guidelines for conducting a scoping review (databases, criteria, search, data charting, …) and writing a paper with all necessary information. We publish our additional materials (exhaustive lists of all information regarding the review questions) publicly on GitHub²²2https://github.com/philipp-kohl/scoping-review-active-learning-er.

Section 2 defines the entity recognition task and active learning and outlines established taxonomies. Section 3 details our review process, ensuring reproducibility and extensibility. Section 4 presents the aggregated findings on AL strategies and their evaluation environments alongside open research questions. Section 5 discusses related work on reducing annotation efforts besides AL. Section 6 addresses ethical considerations, and we conclude by summarizing our findings and highlighting future directions in Section 7.

2 Definitions

Our scoping review focuses on the application of AL to ER. These terms are not used uniformly in the literature. To make our work reproducible and comprehensible, this section starts with precise definitions of both concepts concentrated on the NLP domain, as they are assumed throughout this paper. We also use the definitions as eligibility criteria (Subsection 3.3) for our scoping review.

2.1 Entity Recognition

Entity recognition (ER) describes the NLP task of using a machine learning model to find entities automatically (e.g., persons or organizations) in a text. ER works with an arbitrary predefined label set. A specialized type of ER is named entity recognition [75], which focuses on proper nouns with a label set of person, organization, location, and dates.

ER splits the text into tokens (in a simplified format into words) and assigns a label to each token. This type is called sequence labeling. The literature distinguishes between sequence [75] and span labeling approaches [84, 104]. For sequence labeling approaches, it is challenging to label overlapping entities because every token receives only one label. Span labeling closes this gap. Spans represent $n$ consecutive tokens. Span labeling enumerates all spans with length $1-n$ and classifies each span. Thus, span labeling has to make more decisions, which makes it more complex.

In this paper, we use the term entity recognition for sequence and span labeling approaches with an arbitrary predefined label set.

2.2 Active Learning

Refer to caption — Figure 1: Flow diagram for the pool-based active learning cycle following [72, 38].

Active learning (AL) reduces the annotation effort by selecting data points from an unannotated corpus with an AL strategy. Figure 1 shows the AL cycle: 1) We start with a pool of unlabeled data points. The AL strategy selects data points from the pool and passes them to the human annotator. 2) Once the annotator has enriched the data points with the labels, the new labeled batch is added to the already labeled dataset. 3) When the amount of newly added data points reaches a threshold, the (re-)training of the NLP model will be triggered. 4) A new annotation cycle is started if no stopping criterion³³3Stopping criteria are e.g., desired model performance level or the unavailability of unlabeled data points. is fulfilled. 5) This cycle repeats until a stopping criterion is met.

AL assumes that data points are not equally valuable for the amount of knowledge the NLP model gains [69]. AL algorithms pick the data points for annotating and, therefore, also for training to maximize the knowledge gained per annotated data point [69, 58, 110]. Ideally, this process reduces annotation effort and cost compared to sequential or random data selection [110, 69]. However, using an unsuitable strategy can even lower performance compared to random selection [10, 38].

We address pool-based AL, where we have a (large) unlabeled dataset, and select the most promising data points by the strategy. An alternative approach is stream-based AL [53]. The data is passed one by one to the strategy. The strategy decides whether to propose the single data point to the annotator without incorporating information from other data points. Users apply stream-based AL e.g., due to limited hardware settings.

As a basis for our own work, we adopted the well-known taxonomies from surveys [56, 5, 106, 72] to categorize active learning strategies. Thereby, we modified the terminology to consistently use the same term for semantically similar concepts, improving our work’s clarity and readability. On the top level of our categorization of AL strategies, we follow the distinction into exploitation-based, exploration-based, and hybrid strategies:

Exploitation methods leverage feedback from the ER model to assess the potential value of a data point in the learning process [56]. These methods typically use uncertainty scores, disagreement among multiple weak learners, or performance predictions as their basis. Exploitation strategies’ intuition wants to enhance the decision boundaries, although they tend to focus on outliers [106]. The authors of [106] use the term informativeness, which is very similar to the understanding of exploitation. Hence, we combine the terms to have a consistent naming throughout this work. Informativeness strategies consider a single independent instance without assessing the relation to other data points. They do not explicitly state the need for model feedback, although all the stated strategies use model information. Thus, the definition aligns well with exploitation.

In the case of AL for ER, we have to consider that ER works on token- or span-level. This gives us single feedback information for each token, which AL strategies have to interpret to measure the usefulness of the whole data point. For this purpose, different aggregation methods are known in the literature [71]: e.g., total sum, average, or single most uncertainty.

Exploration methods select data points independently of model feedback. They use vector representations combined with clustering approaches based on density, diversity, and discriminative to determine a data point’s relevance. Exploration strategies aim to cover the vector space holistically [56]. [106] uses the term representativeness for exploration. Similar to exploitation and informativeness, we use the term exploration in this work. The authors use representativeness for strategies that include information on multiple data points to select a subset.

Hybrid methods combine exploitation and exploration methods [106]. They use several strategies sequentially, in parallel, or combine them in weighted aggregation. This way, strategies compensate for other strategies’ drawbacks (e.g., selecting outliers). Because hybrid strategies incorporate exploitation strategies, we must also consider aggregation methods.

We use this taxonomy to group the AL strategies of our screened papers in Subsection 4.1.

3 Methodology

Our scoping review follows the procedure proposed by PRISMA-ScR [95]. We have selected and analyzed papers introducing or modifying active learning strategies applied to ER. For these papers, we list and group the active learning strategies and evaluation datasets. We inspected the evaluation environment to see which metrics researchers use and if they describe the used hardware and give information about execution times.

3.1 Search Engines

We based the search engine selection on [26]. We chose Scopus⁴⁴4https://www.scopus.com/ as our primary search engine because of its literature coverage, advanced searching, and filter features. As a secondary search engine, we used ACM Digital Library⁵⁵5https://dl.acm.org/ to challenge Scopus and enhance the literature coverage.

Scopus is a multidisciplinary, international database and search engine⁶⁶6https://www.elsevier.com/products/scopus/content. It allows the downloading of search results in bulk and supports repeatable queries, guaranteeing reproducibility and maintainability [26]. Scopus was highlighted in [26, 9] as an appropriate choice due to its robust functionalities and extensive database containing more than 14,000 scientific journals. Regarding our specific area of computational linguistics, a manual search for 20 prominent conferences and journals⁷⁷7https://scholar.google.com/citations?view_op=top_venues&hl=en&vq=eng_computationallinguistics confirmed that Scopus indexes all of them. The database encompasses 2689 sources (conference proceedings, journals, book series, …) in computer science and 364 in artificial intelligence, highlighting its broad scope.

We used advanced searching with Scopus and the following query:

{minted}

[fontsize=]python TITLE-ABS-KEY ( (”Active Learning” OR ”Selective Sampling”) AND # End Group 1 (”Sequence Labeling” OR ”Span Categorization” OR ”Entity Classification” OR ”Named Entity” OR ”Entity Recognition” OR ”Span Labeling” OR ”Information Extraction” OR ”Sequence Tagging”)) AND # End Group 2 ( LIMIT-TO ( LANGUAGE,”English” ) ) AND ( LIMIT-TO ( DOCTYPE,”cp” ) OR LIMIT-TO ( DOCTYPE,”ar” ) OR LIMIT-TO ( DOCTYPE,”ch” ) ) AND ( LIMIT-TO ( PUBSTAGE,”final” )) # End Group 3

This search query covers articles’ titles, abstracts, and keywords, looking for relevant matches. It is structured with three groups of terms linked by a logical ’AND’, which are indicated with the comments End Group n. These groups include synonyms for active learning, entity recognition, and filtering criteria⁸⁸8We included only papers written in English that are either book chapters (ch), articles (ar) or conference papers (cp).. The terms in the second group are not strict synonyms. However, in various sources, they are commonly used to describe ER. The selection of these synonyms has evolved iteratively: beginning with ’Entity Recognition’, we then expanded the list by analyzing keywords in articles found through Scopus and literature surveys such as [69, 106], adding relevant terms gradually.

ACM Digital Library serves as a secondary search engine to complement our primary database search with Scopus. Its ACM Guide to Computing Literature database indexes over 2.8 million records⁹⁹9https://libraries.acm.org/digital-library/acm-guide-to-computing-literature, emphasizing conference proceedings, a key source of current research.

ACM does not offer to search titles, abstracts and keywords as a whole. Thus, we adapted our Scopus search string and applied it only to abstracts, leaving out our Scopus-specific filtering criteria. We hypothesized that research papers that address AL and ER would likely include relevant keywords in their abstracts.

3.2 Review Process

Figure 2 gives an overview of our review process: First, for the Identification of relevant papers (records), we entered the search strings stated in Subsection 3.1 in the advanced search fields on Scopus and ACM. We last updated the results of this search on the 12th of January, 2024. This search yielded 260 papers at Scopus and 105 papers at ACM. To further secure the comprehensiveness of our results, we manually compared the results with two AL literature surveys [69, 106]. Together, these two surveys yielded 121 papers: 103 papers from [69] and 18 papers from [106]¹⁰¹⁰10We focused our manual search on the relevant sections to avoid including papers not targeting our topic. We extracted references from Section 3 and 4.2.4 of [69]. From [106], we took the sources listed in appendix A for (named) ER..

We imported the results from Scopus, ACM, and the literature surveys (486 papers) to the screening and documentation tool rayyan.ai¹¹¹¹11https://www.rayyan.ai/. This tool facilitates documenting the results of the automatic Pre-screening and the following manual Screening rounds 1 + 2 by recording the reviewers’ decisions and exclusion reasons. During Pre-Screening, we excluded duplicates and conference proceedings. Then, in Screening Round 1, one reviewer screened all documents and excluded only those fulfilling at least one of the obvious exclusion criteria (Subsection 3.3). Thereby, the reviewer excluded 248 documents. Then, in Screening Round 2, two reviewers screened the remaining 124 papers independently and analyzed them regarding their fit to our detailed exclusion criteria (Subsection 3.3). In this stage, we excluded 2 of the 3 remaining papers of our manual search that had surpassed the screening process so far. All other 118 papers had already been excluded in earlier stages: 25 were excluded during Pre-Screening, 93 during Screening Round 1. The resulting exclusion of more than 99% of the manually added records indicates a high coverage of our Scopus and ACM search results.

Finally, we analyzed the remaining 62 papers and created the results (Section 4): One reviewer extracted the AL strategies, the datasets, metrics, used hardware, and execution times. The second reviewer verified these results to improve the outcome’s quality and coverage. Overall, we identified 106 AL strategies applied to 57 datasets. See our GitHub repository for details.

The distinction between different AL strategies in the context of ER is not trivial. The scores are often calculated at the token level, which must be aggregated to select entire documents. We consider two AL strategies as different if they differ on at least one level: e.g. if the Least Confidence (LC) score is calculated on the token level, some authors average all token scores to a document-level LC score. Others use the value of the token with minimal confidence. This difference is represented in our analysis by identifying two separate AL strategies.

We categorize the identified strategies according to the taxonomy already described in Subsection 2.2.

3.3 Eligibility Criteria

We defined inclusion and exclusion criteria based on our review questions in Section 1. We used these criteria to perform our scoping review. Furthermore, they help other researchers reproduce or update this scoping review. We did not apply any restrictions on the papers’ publication year. All other criteria are listed below.

A paper had to match the following inclusion criteria holistically in order to be included in the review. The paper had to:

•

apply a pool-based AL method as defined in Subsection 2.2.
•

apply AL strategies to ER as defined in Subsection 2.1.
•

be written in English to ensure it addresses the global community.
•

be peer-reviewed, which represents a successful prior quality assessment.
•

use at least one model-agnostic AL strategy. We want to investigate strategies that can be applied to a broad spectrum of use cases and models.

We defined two groups of exclusion criteria to structure our review process (Figure 2): Obvious exclusion criteria contain more formal and less complex decisions that one reviewer can make based on the abstract and, if necessary, an additional short screening of the full text. Detailed exclusion criteria require a more detailed content analysis and were therefore assessed by two reviewers independently. In this case, both reviewers read the paper carefully and analyzed its contents to make a decision.

If a paper matched one of the following obvious exclusion criteria, we excluded it from the review:

•

The paper was a duplicate. Duplicates could occur because we used several search strategies and included all results in the first step.
•

It was not possible to access a full-text version with free access, IEEE or Scopus subscription.
•

The record was a complete conference proceeding. Conference proceedings were excluded because the relevant individual papers should also be contained in our search results.
•

The paper conducted a survey or a systematic/scoping review. We excluded them due to the same reason as conference proceedings.
•

The paper evaluated their AL strategies only on datasets that do not follow a language based on the Latin writing system. This creates a language group with a common base, which is essential for the model selection [16].
•

The record did not report on an ER task.
•

The paper did not use AL.
•

AL was not applied to an ER task.
•

The paper used a stream-based AL procedure.

If a paper matched one of the detailed exclusion criteria, we excluded it from our review:

•

Used AL strategies were not identifiable. In that case, the paper does not focus on AL as a main topic, which does not align with our objective.
•

None of the AL strategies presented were model-agnostic.
•

AL was applied to multiple NLP tasks in a non-separable manner.
•

AL was combined with other methods¹²¹²12Data augmentation [43], weak [24] and distant [40] supervision, proactive learning [42], over-labeling [59], semi supervised learning [93], self learning [62], self-training [108], multi-task AL [109], pre-tagging [54], cross-lingual transfer learning [11], and imitation learning [50] to reduce the annotation costs. We excluded the combination because these methods introduce different changes to the AL cycle (Subsection 2.2).
•

The paper’s entity recognition task did not match our definition from Subsection 2.1. Observed modifications were transfer knowledge (e.g., using a source corpus to transfer knowledge onto a target corpus [47, 86]) or selecting subsequences instead of whole samples [68, 52].

4 Results

The following sections discuss and summarize our results. For comprehensive lists of papers, AL strategies, corpora, and evaluation environments with detailed information, please consult our GitHub repository¹³¹³13https://github.com/philipp-kohl/scoping-review-active-learning-er. The analyses presented in the following answer our review questions from Section 1. Furthermore, we identify research gaps, which we provide after our observations.

All papers identified through the procedure described in Subsection 3.2 were published between 2004 and 2023 (compare Figure 3). As shown in the figure, the interest in AL for ER is on the rise. Almost half of the papers (30 out of 62) were published in the past five years.

When looking at our results, our eligibility criteria must be kept in mind. We selected papers presenting research on developing or modifying AL strategies for ER. This introduces a bias that hinders the transfer of our results outside of this scientific scope.

Table 1: Overview of the AL strategies identified divided into categories following [106, 73, 72]. More details concerning the concrete selection strategies can be found in the references and in our GitHub repository.

method

Specification

# of AL

strategies

# of

usages

Papers

Exploitation

Uncertainty

[51, 44, 56, 103, 79, 1, 12, 91, 64, 92, 107]

[71, 2, 17, 18, 107, 83, 39, 64, 20, 57, 55]

[73, 85, 74, 29, 87, 63, 37, 45, 46, 97, 60]

[80, 10, 111, 76, 77, 48, 81, 14, 70, 78, 36]

[67, 77, 101, 13]

Disagreement

[103, 78, 80, 76, 10, 77, 28, 90, 66, 73, 17]

[65, 27, 89]

Performance

Prediction

[10, 29, 63, 48, 73]

Variance Reduction

[73]

Exploration

Density

[10, 13, 29, 111, 14, 97]

Discriminative

[44, 10, 13]

Density &

Discriminative

[56, 36]

Hybrid

Uncertainty

& Density

[37, 97, 48, 35, 12, 102, 73, 56, 14, 111, 100]

Uncertainty

& Discriminative

[78, 10, 35, 36, 71]

Uncertainty

& Other

[36, 87, 6]

Disagreement &

Discriminative

[23]

Table 2: Uncertainty-based AL strategies with their heuristic.

Heuristic	# of strategies	# of usages
Least Confidence	11	35
Entropy	9	22
Margin	4	14
Count	4	4
Round Robin	3	3
Max. Norm. Log-Probability	1	15
Other	4	4
Sum	36	97

4.1 Active Learning Strategies for ER (Review Question 1)

In total, we identified 106 AL strategies with ER applications in our scoping review. Table 1 lists the total number of AL strategies using the different methods (exploitation, exploration, and hybrid) and their specification following [106, 73, 72].

We list our observations of the results regarding the AL strategies in the following:

Focus on Exploitation-based Approaches: We identified 60 exploitation-based AL strategies, which were used 131 times in our 62 analyzed papers. Uncertainty-based AL strategies represent the majority (Table 1): 60% of the exploitation-based strategies belong to uncertainty-based approaches, which are used 74% of the time. Table 2 shows the number of strategies and their usages of the uncertainty-based AL strategies grouped by the different scoring approaches, which we call heuristics. Least confidence approaches were developed and used most.
Infrequently Used Exploration-based Approaches: We identified 14 exploration-based strategies. They are applied less often in isolation (15 times) than in hybrid settings (37 times). This is noteworthy because the implementation of hybrid approaches is more complex. Exploitation-based strategies, in contrast, are used extensively on their own.
Distribution of AL Strategies Over Domains: As depicted in Table 3, the three most used domains are bio-medicine, medicine, and newspaper. We observed that significantly more exploitation than exploration approaches are applied in all domains. Interestingly, hybrid strategies are on par with exploitation strategies in the medical domain. The other two domains use less than half as often hybrid strategies as exploitation strategies.

Table 3: Number of strategies applied to the three main domains.

Approach	# of strategies	Medicine	Biomedicine	News-corpora
Exploitation	60	17	29	44
Exploration	14	5	3	4
Hybrid	32	17	13	14
Sum	106	39	45	62

4.2 Corpora (Review Question 2a)

We identified 57 corpora from more than 9 domains. Most corpora belong to the domain of bio-medicine (12), medicine (9), and newspaper articles (7). The others hold three or fewer corpora¹⁴¹⁴14Cybersecurity (3), Scientific Papers (3), Twitter Posts (2), Wikipedia Articles (3), Instructions (2), E-Mail (2), and we group the single domain corpora into an other category (14). See GitHub repository for an exhaustive list..

Figure 4 shows the corpora usage per domain. 35 out of 62 papers use newspaper articles to investigate AL performance. Second and third are bio-medicine and medicine, with 23 usages each. The bio-medicine and medicine domains have the highest number of corpora, but researchers use newspaper articles more frequently.

The CoNLL [88] corpora (2003 and 2002) based on newspaper articles were the most used corpora with 30 usages. Other often used corpora were i2b2/VA 2010 (medicine) [96] with 7 usages and JNLPBA (bio-medicine) [15] with 6 usages.

We investigated public access to corpora and prepared a list of accessible datasets for further research. We consider a corpus open access if the researchers provide a link to the dataset or to a reference that introduced and published the corpus. Also, we consulted the authors’ web pages to find corpora for their publications when necessary. If a corpus has to be requested and is only licensable for research and academic purposes, we consider it as open access. 26 out of 57 corpora follow the reproducible research requirement to publish their datasets with open access.

Researchers do not always publish their datasets or make their annotations freely available. 26 of 57 corpora are private or can be licensed against a fee. We classify them as not open access. For 5 corpora, we could not draw a reasonable decision due to invalid or moved internet resources. Thus, we consider them as not openly accessible.

We list our observations of the results regarding the corpora in the following.

High Usage of Newspaper Articles: Newspaper articles show the highest usage (Figure 4) across the domains. More than half of the identified papers evaluate their active learning strategies on newspaper articles, although the medicine and bio-medicine corpora have more corpora. We hypothesize that newspaper articles are often freely available, and the annotation process does not need the same level of expertise as for (bio-)medicine.
Most Corpora for Bio-medicine and Medicine Domain: The annotation process for the bio-medicine and medicine corpora can be very costly due to highly educated staff. 7 out of 21 bio-medicine and medicine corpora are accessible under open-access licensing (see GitHub). We hypothesize that the number of corpora indicates that the domain sees great potential to reduce the annotation effort with AL. With their contribution of publicly available corpora, which were annotated by highly educated staff, they probably want to facilitate more research.

We provide a list of publicly accessible corpora designated for ER, enabling researchers to investigate and advance the development of ER and AL methods.

4.3 Metrics, Hardware, and Execution Times (Review Questions 2b, 2c)

Our examination reveals a uniformity in the metrics used across studies, suggesting a consensus on their effectiveness. Regarding the hardware, only 6 out of 62 papers detail the hardware used for experiments, which we consider a critical oversight given that the hardware can significantly influence the training and inference times. This impacts the annotation process: The time for retraining models and determining new data points for the annotators results in waiting times [97]. The hardware reported ranged from personal computers to small server instances and workstations. No information was found on the usage of distributed clusters. Our analysis of execution times identified 13 papers reporting on training duration, annotator wait times, inference speeds, and annotation time.

We made the following observations:

AL Performance Comparison Metrics: Due to the differences in the implementations, parameters, and environments, a direct comparison of the performance of different AL strategies is unrealizable. However, the findings of the records offer information about the metrics used to evaluate AL strategies. Frequently it is F1-score (60), precision (16), and recall (16). Rarely ( $<$ 4 times), it is accuracy, annotation time, and error rate.
AL Execution Times and Used Hardware: Little attention is paid to these aspects. 13 out of 62 papers reported any kind of timing information. Only 6 stated their used hardware. Papers presenting real-world applications of AL to ER tasks [97] mention the relevance of short retraining times for the AL model because they correlate with the waiting time for annotators. We could not find information about the duration of the initialization of an AL strategy, nor did we find information about the time a strategy needs to propose the new data points to the annotator.

4.4 Research Gaps

Based on our observations from the last sections, we formulate open topics as questions, which can guide future work in the field of AL and ER:

General: How do AL strategies perform in various domains in terms of their performance and execution time on specified hardware conditions? Is there a universally effective active learning strategy independent of the use case?
Exploitation Approaches: What are the reasons for the quantitative dominance of exploitation-based, especially uncertainty-based, AL strategies? Are these strategies also outstanding qualitatively? Are they used as a solution for ER, or are they primarily used as baselines for comparison with other strategies?
Exploration and Hybrid Approaches: Is the isolated usage of exploration-based AL strategies less beneficial than solely exploitation-based applications? Do hybrid approaches outperform exploration but not exploitation-based strategies?
Domain Research: Why does medical research for ER focus equally on exploitation and hybrid approaches while other domains favor exploitation-based strategies? Does the intense focus on evaluating AL strategies on newspaper articles reveal well-performing strategies? Do newspaper articles function as a baseline? Do AL strategies perform on specialized domains such as medicine as well as for broad domains like newspaper articles? Do the corpora and label sets differ in number and complexity? Is there a universally effective active learning strategy, or are certain strategies more effective in specific domains?

4.5 Evaluation Environment

According to the results presented, we establish a set of criteria for evaluating the effectiveness of Active Learning (AL) strategies in future works. An evaluation framework like ALE [38] and a reasoned selection of AL strategies and datasets enable a fair comparison. The evaluation environment should consider the following aspects:

Strategies: The comparison of strategies should include at least one strategy of each specification and heuristic (see Table 1 and Table 2). To assess the impact of the aggregation method, the permutation should be considered for exploitation and hybrid strategies.
Dataset: The strategies should be tested on a diverse set of corpora. This assesses the strategy’s robustness and allows to investigate the existence of an overall high-performing AL strategy. Based on the open access corpora (Figure 4), several domains can be tested (news, (bio-) medicine, scientific, and social media). The corpora may differ in size, language, and label complexity, which introduces different challenges.
Hardware and Execution Time: It is important to consider the time constraints of AL strategies as they can affect the annotation process [33]: The time required for proposing new data points and retraining the model can impact the waiting time for annotators. Therefore, it is essential to record the timings for initializing the AL strategy, proposing data points, and retraining the model. The timing information is strongly dependent on the hardware used. Therefore, it is crucial to provide information about the used hardware.
Evaluation metric: Based on this scoping review, the F1-Score should be in the list of reporting metrics.
Bias: Tracking bias reinforcement can help identify strategies that mitigate bias instead of amplifying it (see Section 6).

5 Related Work

Besides AL, researchers developed other approaches to reduce the annotation effort. Semi-supervised [82] and weak supervision [49] techniques rely on an initial dataset from which they derive rules or heuristics, enabling the automatic annotation of a larger dataset with reduced manual effort. These approaches might introduce noise into the data due to less precise heuristics.

Distant supervision [32], on the other hand, leverages external resources to generate positive examples for specific tasks, which is especially useful when external knowledge bases can provide substantial input. Data augmentation [22] complements these methods by creating new instances from already labeled data through various linguistic transformations.

AL and the stated methods can be used together to further reduce the manual effort [93, 24, 40, 43].

Surveys such as those by [69, 106] are most closely related to our work. Thereby, [69] considers deep learning techniques with AL in several areas (such as computer vision), while [106] focuses solely on NLP. Both surveys categorize the strategies found, many of which cannot be directly applied to ER.

6 Ethical Consideration

AL must be ethically scrutinized in the general context of NLP [41]. Specifically, AL as a data selection method can insert or enforce statistical bias following [21, 30]. The authors propose possible reasons for this unwanted effect, such as the distribution of data points in the seed and train dataset. Furthermore, AL errors can cause the model to become wrongly confident, making it difficult to correct the learned structure. As a result, data may stop being proposed for this concept because the model does not show room for improvement [30]. Another bias may be transferred from transformer models [99] during the pre-training phase of the AL model [30].

These bias-introducing and enforcing effects are especially alarming considering the focus of AL research on the medical domain. Existing approaches optimize for fairness metrics [3] or vary the error rate in each iteration of adaptive clustering to reduce bias [30] for classification tasks. These methods should also be tested for ER.

7 Conclusion and Future Work

We conducted a scoping review to provide an overview of active learning strategies, metrics, datasets, execution times, and hardware used for the entity recognition task.

Our results as comprehensive lists can be found in the provided GitHub repository: We identified 106 AL strategies and 57 datasets in 62 papers. A large share of the strategies follows the exploitation-based (60) approach. 36 of them use uncertainty-based sampling. Furthermore, we noted fewer exploration-based AL strategies than hybrid ones. For evaluation purposes, the F1-score is the dominant metric to demonstrate the performance of an AL strategy. Unfortunately, very few researchers report the execution time and used hardware for their experiments. We examined the 57 datasets on their availability and found 26 publicly accessible corpora. The most frequently used corpora are from the newspaper, bio-medical, and medical domains. We created an evaluation environment based on our observations. Additionally, we have identified research gaps in the field, which researchers can use as an outline for further research.

We plan to conduct comprehensive performance tests on a subset of the AL strategies and datasets found in this scoping review based on our evaluation environment.

References

[1] Agrawal, A., Tripathi, S., Vardhan, M.: Active learning approach using a modified least confidence sampling strategy for named entity recognition. Progress in Artificial Intelligence 10(2), 113–128 (2021). https://doi.org/10.1007/s13748-021-00230-w
[2] Agrawal, A., Tripathi, S., Vardhan, M.: Multicore based least confidence query sampling strategy to speed up active learning approach for named entity recognition. Computing 105(5), 979–997 (2023). https://doi.org/10.1007/s00607-021-01000-1
[3] Anahideh, H., Asudeh, A., Thirumuruganathan, S.: Fair Active Learning. arXiv:2001.01796 [cs, stat] (Mar 2021)
[4] Arora, S., Agarwal, S., Students, M.: Active Learning for Natural Language Processing. Language Technologies Institute School of Computer Science Carnegie Mellon University 2 (2007)
[5] Bondu, A., Lemaire, V., Boullé, M.: Exploration vs. exploitation in active learning : A Bayesian approach. In: The 2010 International Joint Conference on Neural Networks (IJCNN). pp. 1–7 (Jul 2010). https://doi.org/10.1109/IJCNN.2010.5596815
[6] Brent, P., Green, N., Breimyer, P., Krishnamurthy, R., Samatova, N.: Systematic evaluation of convergence criteria in iterative training for NLP. In: Proceedings of the 22nd International Florida Artificial Intelligence Research Society Conference, FLAIRS-22. pp. 15–20 (2009)
[7] Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language Models are Few-Shot Learners (Jul 2020). https://doi.org/10.48550/arXiv.2005.14165
[8] Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language models are few-shot learners. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. pp. 1877–1901. NIPS’20, Curran Associates Inc., Red Hook, NY, USA (Dec 2020)
[9] Burnham, J.F.: Scopus database: A review. Biomedical Digital Libraries 3(1), 1–8 (Dec 2006). https://doi.org/10.1186/1742-5581-3-1
[10] Chang, H.S., Vembu, S., Mohan, S., Uppaal, R., McCallum, A.: Using error decay prediction to overcome practical issues of deep active learning for named entity recognition. Machine Learning 109(9-10), 1749–1778 (2020). https://doi.org/10.1007/s10994-020-05897-1
[11] Chaudhary, A., Xie, J., Sheikh, Z., Neubig, G., Carbonell, J.: A little annotation does a lot of good: A study in bootstrapping low-resource named entity recognizers. In: EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference. pp. 5164–5174 (2019)
[12] Chen, Y., Lask, T., Mei, Q., Chen, Q., Moon, S., Wang, J., Nguyen, K., Dawodu, T., Cohen, T., Denny, J., Xu, H.: An active learning-enabled annotation system for clinical named entity recognition. BMC Medical Informatics and Decision Making 17 (2017). https://doi.org/10.1186/s12911-017-0466-9
[13] Chen, Y., Lasko, T.A., Mei, Q., Denny, J.C., Xu, H.: A study of active learning methods for named entity recognition in clinical text. Journal of Biomedical Informatics 58, 11–18 (Dec 2015). https://doi.org/10.1016/j.jbi.2015.09.010
[14] Claveau, V., Kijak, E.: Strategies to select examples for active learning with conditional random fields. In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 10761 LNCS, pp. 30–43 (2018). https://doi.org/10.1007/978-3-319-77113-7_3
[15] Collier, N., Ohta, T., Tsuruoka, Y., Tateisi, Y., Kim, J.D.: Introduction to the Bio-entity Recognition Task at JNLPBA. In: Collier, N., Ruch, P., Nazarenko, A. (eds.) Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and Its Applications (NLPBA/BioNLP). pp. 73–78. COLING, Geneva, Switzerland (Aug 2004)
[16] Conneau, A., Lample, G.: Cross-lingual language model pretraining. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 7059–7069. No. 634, Curran Associates Inc., Red Hook, NY, USA (Dec 2019)
[17] Culotta, A., Kristjansson, T., McCallum, A., Viola, P.: Corrective feedback and persistent learning for information extraction. Artificial Intelligence 170(14-15), 1101–1122 (2006). https://doi.org/10.1016/j.artint.2006.08.001
[18] Culotta, A., McCallum, A.: Reducing labeling effort for structured prediction tasks. In: Proceedings of the National Conference on Artificial Intelligence. vol. 2, pp. 746–751 (2005)
[19] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. https://arxiv.org/abs/1810.04805v2 (Oct 2018)
[20] Esuli, A., Marcheggiani, D., Sebastiani, F.: Sentence-based active learning strategies for information extraction. In: CEUR Workshop Proceedings. vol. 560, pp. 41–45 (2010)
[21] Farquhar, S., Gal, Y., Rainforth, T.: On Statistical Bias In Active Learning: How and When To Fix It (May 2021)
[22] Feng, S.Y., Gangal, V., Wei, J., Chandar, S., Vosoughi, S., Mitamura, T., Hovy, E.: A Survey of Data Augmentation Approaches for NLP (Dec 2021). https://doi.org/10.48550/arXiv.2105.03075
[23] Gao, N., Karampatziakis, N., Potharaju, R., Cucerzan, S.: Active entity recognition in low resource settings. In: International Conference on Information and Knowledge Management, Proceedings. pp. 2261–2264 (2019). https://doi.org/10.1145/3357384.3358109
[24] Gonsior, J., Thiele, M., Lehner, W.: WeakAL: Combining Active Learning and Weak Supervision. In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 12323 LNAI, pp. 34–49 (2020). https://doi.org/10.1007/978-3-030-61527-7_3
[25] Grant, M.J., Booth, A.: A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal 26(2), 91–108 (2009). https://doi.org/10.1111/j.1471-1842.2009.00848.x
[26] Gusenbauer, M., Haddaway, N.R.: Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Research Synthesis Methods 11(2), 181–217 (2020). https://doi.org/10.1002/jrsm.1378
[27] Hachey, B., Alex, B., Becker, M.: Investigating the effects of selective sampling on the annotation task. In: CoNLL 2005 - Proceedings of the Ninth Conference on Computational Natural Language Learning. pp. 144–151 (2005). https://doi.org/10.3115/1706543.1706569
[28] Hahn, U., Beisswanger, E., Buyko, E., Faessler, E.: Active Learning-based corpus annotation–the PathoJen experience. AMIA … Annual Symposium proceedings / AMIA Symposium. AMIA Symposium 2012, 301–310 (2012)
[29] Han, X., Kwoh, C., Kim, J.J.: Clustering based active learning for biomedical Named Entity Recognition. In: Proceedings of the International Joint Conference on Neural Networks. vol. 2016-October, pp. 1253–1260 (2016). https://doi.org/10.1109/IJCNN.2016.7727341
[30] Hassan, S., Alikhani, M.: D-CALM: A Dynamic Clustering-based Active Learning Approach for Mitigating Bias
[31] Hassanzadeh, H., Keyvanpour, M.: A two-phase hybrid of semi-supervised and active learning approach for sequence labeling. Intelligent Data Analysis 17(2), 251–270 (2013). https://doi.org/10.3233/IDA-130577
[32] Hedderich, M.A., Lange, L., Klakow, D.: ANEA: Distant Supervision for Low-Resource Named Entity Recognition (Apr 2021). https://doi.org/10.48550/arXiv.2102.13129
[33] Herde, M., Huseljic, D., Sick, B., Calma, A.: A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification. IEEE Access 9, 166970–166989 (2021). https://doi.org/10.1109/ACCESS.2021.3135514
[34] Jayakumar, T., Farooqui, F., Farooqui, L.: Large Language Models are legal but they are not: Making the case for a powerful LegalLLM. In: Preo\textcommabelowtiuc-Pietro, D., Goanta, C., Chalkidis, I., Barrett, L., Spanakis, G.J., Aletras, N. (eds.) Proceedings of the Natural Legal Language Processing Workshop 2023. pp. 223–229. Association for Computational Linguistics, Singapore (Dec 2023). https://doi.org/10.18653/v1/2023.nllp-1.22
[35] Kholghi, M., De Vine, L., Sitbon, L., Zuccon, G., Nguyen, A.: Clinical information extraction using small data: An active learning approach based on sequence representations and word embeddings. Journal of the Association for Information Science and Technology 68(11), 2543–2556 (2017). https://doi.org/10.1002/asi.23936
[36] Kholghi, M., Sitbon, L., Zuccon, G., Nguyen, A.: External knowledge and query strategies in active learning: A study in clinical information extraction. In: International Conference on Information and Knowledge Management, Proceedings. vol. 19-23-Oct-2015, pp. 143–152 (2015). https://doi.org/10.1145/2806416.2806550
[37] Kim, S., Song, Y., Kim, K., Cha, J.W., Lee, G.: MMR-based active machine learning for bio named entity recognition. In: HLT-NAACL 2006 - Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, Short Papers. pp. 69–72 (2006)
[38] Kohl, P., Freyer, N., Krämer, Y., Werth, H., Wolf, S., Kraft, B., Meinecke, M., Zündorf, A.: ALE: A Simulation-Based Active Learning Evaluation Framework for the Parameter-Driven Comparison of Query Strategies for NLP. In: Conte, D., Fred, A., Gusikhin, O., Sansone, C. (eds.) Deep Learning Theory and Applications. pp. 235–253. Communications in Computer and Information Science, Springer Nature Switzerland, Cham (2023). https://doi.org/10.1007/978-3-031-39059-3_16
[39] Laws, F., Scheible, C., Schütze, H.: Active learning with amazon mechanical turk. In: EMNLP 2011 - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference. pp. 1546–1556 (2011)
[40] Lee, S., Song, Y., Choi, M., Kim, H.: Bagging-based active learning model for named entity recognition with distant supervision. In: 2016 International Conference on Big Data and Smart Computing, BigComp 2016. pp. 321–324 (2016). https://doi.org/10.1109/BIGCOMP.2016.7425938
[41] Leidner, J.L., Plachouras, V.: Ethical by Design: Ethics Best Practices for Natural Language Processing. In: Proceedings of the First ACL Workshop on Ethics in Natural Language Processing. pp. 30–40. Association for Computational Linguistics, Valencia, Spain (2017). https://doi.org/10.18653/v1/W17-1604
[42] Li, M., Nguyen, N., Ananiadou, S.: Proactive Learning for Named Entity Recognition. In: BioNLP 2017 - SIGBioMed Workshop on Biomedical Natural Language Processing, Proceedings of the 16th BioNLP Workshop. pp. 117–125 (2017)
[43] Li, Q., Huang, Z., Dou, Y., Zhang, Z.: A Framework of Data Augmentation While Active Learning for Chinese Named Entity Recognition. In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 12816 LNAI, pp. 88–100 (2021). https://doi.org/10.1007/978-3-030-82147-0_8
[44] Li, W., Du, Y., Li, X., Chen, X., Xie, C., Li, H., Li, X.: UD_BBC: Named entity recognition in social network combined BERT-BiLSTM-CRF with active learning. Engineering Applications of Artificial Intelligence (2022). https://doi.org/10.1016/j.engappai.2022.105460
[45] Li, Y., Yue, T., Zhenxin, W.: IEKM-MD: An intelligent platform for information extraction and knowledge mining in multi-domains. In: CEUR Workshop Proceedings. vol. 2658, pp. 73–78 (2020)
[46] Lin, B., Lee, D.H., Xu, F., Lan, O., Ren, X.: AlpacaTag: An active learning-based crowd annotation framework for sequence tagging. In: ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of System Demonstrations. pp. 58–63 (2019)
[47] Lin, S., Gao, J., Zhang, S., He, X., Sheng, Y., Chen, J.: A continuous learning method for recognizing named entities by integrating domain contextual relevance measurement and Web farming mode of Web intelligence. World Wide Web 23(3), 1769–1790 (2020). https://doi.org/10.1007/s11280-019-00758-x
[48] Linh, L., Nguyen, M.T., Zuccon, G., Demartini, G.: Loss-based Active Learning for Named Entity Recognition. In: Proceedings of the International Joint Conference on Neural Networks. vol. 2021-July (2021). https://doi.org/10.1109/IJCNN52387.2021.9533675
[49] Lison, P., Barnes, J., Hubin, A.: Skweak: Weak Supervision Made Easy for NLP. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations. pp. 337–346 (2021). https://doi.org/10.18653/v1/2021.acl-demo.40
[50] Liu, M., Buntine, W., Haffari, G.: Learning how to actively learn: A deep imitation learning approach. In: ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers). vol. 1, pp. 1874–1883 (2018). https://doi.org/10.18653/v1/p18-1174
[51] Liu, M., Tu, Z., Zhang, T., Su, T., Xu, X., Wang, Z.: LTP: A New Active Learning Strategy for CRF-Based Named Entity Recognition. Neural Processing Letters 54(3), 2433–2454 (2022). https://doi.org/10.1007/s11063-021-10737-x
[52] Liu, Y., Hu, J., Chen, Z., Wan, X., Chang, T.H.: EASAL: Entity-Aware Subsequence-Based Active Learning for Named Entity Recognition. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence, AAAI 2023. vol. 37, pp. 8897–8905 (2023)
[53] Loy, C.C., Hospedales, T.M., Tao Xiang, Shaogang Gong: Stream-based joint exploration-exploitation active learning. 2012 IEEE Conference on Computer Vision and Pattern Recognition pp. 1560–1567 (Jun 2012). https://doi.org/10.1109/CVPR.2012.6247847
[54] Marcheggiani, D., Artières, T.: An experimental comparison of active learning strategies for partially labeled sequences. In: EMNLP 2014 - 2014 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference. pp. 898–906 (2014). https://doi.org/10.3115/v1/d14-1097
[55] Mejer, A., Crammer, K.: Confidence in structured-prediction using Confidence-Weighted models. In: EMNLP 2010 - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference. pp. 971–981 (2010)
[56] Mendonça, V., Sardinha, A., Coheur, L., Santos, A.L.: Query Strategies, Assemble! Active Learning with Expert Advice for Low-resource Natural Language Processing. In: 2020 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE). pp. 1–8 (Jul 2020). https://doi.org/10.1109/FUZZ48607.2020.9177707
[57] Miller, S., Guinness, J., Zamanian, A.: Name tagging with word clusters and discriminative training. In: HLT-NAACL 2004 - Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference. pp. 337–342 (2004)
[58] Mirończuk, M.M., Protasiewicz, J.: A recent overview of the state-of-the-art elements of text classification. Expert Systems with Applications 106, 36–54 (2018)
[59] Mo, Y., Scott, S., Downey, D.: Learning hierarchically decomposable concepts with active over-labeling. In: Proceedings - IEEE International Conference on Data Mining, ICDM. pp. 340–349 (2017). https://doi.org/10.1109/ICDM.2016.165
[60] Moniz, J., Patra, B., Gormley, M.: On Efficiently Acquiring Annotations for Multilingual Models. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics. vol. 2, pp. 69–85 (2022)
[61] Munn, Z., Peters, M.D.J., Stern, C., Tufanaru, C., McArthur, A., Aromataris, E.: Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology 18(1), 143 (Nov 2018). https://doi.org/10.1186/s12874-018-0611-x
[62] Neto, J., Faleiros, T.: Deep Active-Self Learning Applied to Named Entity Recognition. In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 13074 LNAI, pp. 405–418 (2021). https://doi.org/10.1007/978-3-030-91699-2_28
[63] Nguyen, V., Lee, W., Ye, N., Chai, K., Chieu, H.: Active learning for probabilistic hypotheses using the maximum Gibbs error criterion. In: Advances in Neural Information Processing Systems (2013)
[64] Ni, J., Delaney, B., Florian, R.: Fast Model Adaptation for Automated Section Classification in Electronic Medical Records. In: Studies in Health Technology and Informatics. vol. 216, pp. 35–39 (2015). https://doi.org/10.3233/978-1-61499-564-7-35
[65] Olsson, F.: On privacy preservation in text and document-based active learning for named entity recognition. In: International Conference on Information and Knowledge Management, Proceedings. pp. 53–60 (2009). https://doi.org/10.1145/1651449.1651460
[66] Olsson, F., Tomanek, K.: An intrinsic stopping criterion for committee-based active learning. In: CoNLL 2009 - Proceedings of the Thirteenth Conference on Computational Natural Language Learning. pp. 138–146 (2009). https://doi.org/10.3115/1596374.1596398
[67] Pradhan, A., Todi, K., Selvarasu, A., Sanyal, A.: Knowledge Graph Generation with Deep Active Learning. In: Proceedings of the International Joint Conference on Neural Networks (2020). https://doi.org/10.1109/IJCNN48605.2020.9207515
[68] Radmard, P., Fathullah, Y., Lipani, A.: Subsequence Based Deep Active Learning for Named Entity Recognition. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 4310–4321. Association for Computational Linguistics, Online (Aug 2021). https://doi.org/10.18653/v1/2021.acl-long.332
[69] Ren, P., Xiao, Y., Chang, X., Huang, P.Y., Li, Z., Gupta, B.B., Chen, X., Wang, X.: A Survey of Deep Active Learning. ACM Computing Surveys 54(9), 1–40 (Dec 2022). https://doi.org/10.1145/3472291
[70] Saha, S., Ekbal, A., Verma, M., Sikdar, U., Poesio, M.: Active learning technique for biomedical named entity extraction. In: ACM International Conference Proceeding Series. pp. 835–841 (2012). https://doi.org/10.1145/2345396.2345532
[71] Şapci, A., Kemik, H., Yeniterzi, R., Tastan, O.: Focusing on potential named entities during active label acquisition. Natural Language Engineering (2023). https://doi.org/10.1017/S1351324923000165
[72] Settles, B.: Active Learning Literature Survey p. 67 (2009)
[73] Settles, B., Craven, M.: An analysis of active learning strategies for sequence labeling tasks. In: Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. pp. 1070–1079 (2008)
[74] Shardlow, M., Ju, M., Li, M., O’Reilly, C., Iavarone, E., McNaught, J., Ananiadou, S.: A Text Mining Pipeline Using Active and Deep Learning Aimed at Curating Information in Computational Neuroscience. Neuroinformatics 17(3), 391–406 (2019). https://doi.org/10.1007/s12021-018-9404-y
[75] Sharma, A., Amrita, Chakraborty, S., Kumar, S.: Named Entity Recognition in Natural Language Processing: A Systematic Review. In: Gupta, D., Khanna, A., Kansal, V., Fortino, G., Hassanien, A.E. (eds.) Proceedings of Second Doctoral Symposium on Computational Intelligence. pp. 817–828. Advances in Intelligent Systems and Computing, Springer, Singapore (2022). https://doi.org/10.1007/978-981-16-3346-1_66
[76] Shelmanov, A., Puzyrev, D., Kupriyanova, L., Belyakov, D., Larionov, D., Khromov, N., Kozlova, O., Artemova, E., Dylov, D.V., Panchenko, A.: Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates (Feb 2021)
[77] Shen, Y., Yun, H., Lipton, Z., Kronrod, Y., Anandkumar, A.: Deep active learning for named entity recognition. In: Proceedings of the 2nd Workshop on Representation Learning for NLP, Rep4NLP 2017 at the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017. pp. 252–256 (2017)
[78] Shen, Y., Yun, H., Lipton, Z.C., Kronrod, Y., Anandkumar, A.: Deep Active Learning for Named Entity Recognition (Feb 2018)
[79] Shrivastava, A., Heer, J.: ISeqL. In: International Conference on Intelligent User Interfaces, Proceedings IUI. pp. 43–54 (2020). https://doi.org/10.1145/3377325.3377503
[80] Siddhant, A., Lipton, Z.C.: Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study (Sep 2018)
[81] Simpson, E., Gurevych, I.: A Bayesian approach for sequence tagging with crowds. In: EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference. pp. 1093–1104 (2019)
[82] Sintayehu, H., Lehal, G.S.: Named entity recognition: A semi-supervised learning approach. International Journal of Information Technology 13(4), 1659–1665 (Aug 2021). https://doi.org/10.1007/s41870-020-00470-4
[83] Skeppstedt, M., Rzepka, R., Araki, K., Kerren, A.: Visualising and evaluating the effects of combining active learning with word embedding features. In: Proceedings of the 15th Conference on Natural Language Processing, KONVENS 2019. pp. 91–100 (2020)
[84] Son, N.H., Yu, H.M., Nguyen, T.A.D., Nguyen, M.T.: Jointly Learning Span Extraction and Sequence Labeling for Information Extraction from Business Documents. In: 2022 International Joint Conference on Neural Networks (IJCNN). pp. 1–8 (Jul 2022). https://doi.org/10.1109/IJCNN55064.2022.9892779
[85] Tang, S., Liu, H., Almatared, M., Abudayyeh, O., Lei, Z., Fong, A.: Towards Automated Construction Quantity Take-Off: An Integrated Approach to Information Extraction from Work Descriptions. Buildings 12(3) (2022). https://doi.org/10.3390/buildings12030354
[86] Tang, X., Wu, S., Chen, G., Chen, K., Shou, L.: Learning to Label with Active Learning and Reinforcement Learning. In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 12682 LNCS, pp. 549–557 (2021). https://doi.org/10.1007/978-3-030-73197-7_36
[87] Tchoua, R., Ajith, A., Hong, Z., Ward, L., Chard, K., Audus, D., Patel, S., De Pablo, J., Foster, I.: Active learning yields better training data for scientific named entity recognition. In: Proceedings - IEEE 15th International Conference on eScience, eScience 2019. pp. 126–135 (2019). https://doi.org/10.1109/eScience.2019.00021
[88] Tjong Kim Sang, E.F., De Meulder, F.: Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition. In: Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003. pp. 142–147 (2003)
[89] Tomanek, K., Hahn, U.: Approximating learning curves for active-learning-driven annotation. In: Proceedings of the 6th International Conference on Language Resources and Evaluation, LREC 2008. pp. 1319–1324 (2008)
[90] Tomanek, K., Hahn, U.: Reducing class imbalance during active learning for named entity annotation. In: K-CAP’09 - Proceedings of the 5th International Conference on Knowledge Capture. pp. 105–112 (2009). https://doi.org/10.1145/1597735.1597754
[91] Tomanek, K., Hahn, U.: Annotation time stamps - Temporal metadata from the linguistic annotation process. In: Proceedings of the 7th International Conference on Language Resources and Evaluation, LREC 2010. pp. 2516–2521 (2010)
[92] Tomanek, K., Laws, F., Hahn, U., Schütze, H.: On proper unit selection in active learning: Co-selection effects for named entity recognition. In: Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing. pp. 9–17. HLT ’09, Association for Computational Linguistics, USA (2009)
[93] Tran, V., Hoang, D., Nguyen, N., Hwang, D.: A hybrid method for named entity recognition on tweet streams. In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 10191 LNAI, pp. 258–268 (2017). https://doi.org/10.1007/978-3-319-54472-4_25
[94] Tran, V., Nguyen, N., Fujita, H., Hoang, D., Hwang, D.: A combination of active learning and self-learning for named entity recognition on Twitter using conditional random fields. Knowledge-Based Systems 132, 179–187 (2017). https://doi.org/10.1016/j.knosys.2017.06.023
[95] Tricco, A.C., Lillie, E., Zarin, W., O’Brien, K.K., Colquhoun, H., Levac, D., Moher, D., Peters, M.D., Horsley, T., Weeks, L., Hempel, S., Akl, E.A., Chang, C., McGowan, J., Stewart, L., Hartling, L., Aldcroft, A., Wilson, M.G., Garritty, C., Lewin, S., Godfrey, C.M., Macdonald, M.T., Langlois, E.V., Soares-Weiser, K., Moriarty, J., Clifford, T., Tunçalp, Ö., Straus, S.E.: PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Annals of Internal Medicine 169(7), 467–473 (Oct 2018). https://doi.org/10.7326/M18-0850
[96] Uzuner, Ö., South, B.R., Shen, S., DuVall, S.L.: 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association : JAMIA 18(5), 552–556 (2011). https://doi.org/10.1136/amiajnl-2011-000203
[97] Van Nguyen, M., Ngo, N., Min, B., Nguyen, T.: FAMIE: A Fast Active Learning Framework for Multilingual Information Extraction. In: NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Demonstrations Session. pp. 131–139 (2022)
[98] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention Is All You Need (Aug 2023). https://doi.org/10.48550/arXiv.1706.03762
[99] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention Is All You Need (Aug 2023). https://doi.org/10.48550/arXiv.1706.03762
[100] Veerasekharreddy, B., Rao, K., Koppula, N.: Named Entity Recognition using CRF with Active Learning Algorithm in English Texts. In: 6th International Conference on Electronics, Communication and Aerospace Technology, ICECA 2022 - Proceedings. pp. 1041–1044 (2022). https://doi.org/10.1109/ICECA55336.2022.10009592
[101] Verma, M., Sikdar, U., Saha, S., Ekbal, A.: Ensemble based active annotation for biomedical named entity recognition. In: Proceedings of the 2013 International Conference on Advances in Computing, Communications and Informatics, ICACCI 2013. pp. 973–978 (2013). https://doi.org/10.1109/ICACCI.2013.6637308
[102] Wei, Q., Chen, Y., Salimi, M., Denny, J., Mei, Q., Lasko, T., Chen, Q., Wu, S., Franklin, A., Cohen, T., Xu, H.: Cost-aware active learning for named entity recognition in clinical text. Journal of the American Medical Informatics Association 26(11), 1314–1322 (2019). https://doi.org/10.1093/jamia/ocz102
[103] Yao, J., Dou, Z., Nie, J., Wen, J.: Looking Back on the Past: Active Learning with Historical Evaluation Results. IEEE Transactions on Knowledge and Data Engineering (2020). https://doi.org/10.1109/TKDE.2020.3045816
[104] Zaratiana, U., Tomeh, N., Holat, P., Charnois, T.: GNNer: Reducing Overlapping in Span-based NER Using Graph Neural Networks. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop. pp. 97–103. Association for Computational Linguistics, Dublin, Ireland (2022). https://doi.org/10.18653/v1/2022.acl-srw.9
[105] Zhan, X., Wang, Q., Huang, K.h., Xiong, H., Dou, D., Chan, A.B.: A Comparative Survey of Deep Active Learning (Jul 2022)
[106] Zhang, Z., Strubell, E., Hovy, E.: A survey of active learning for natural language processing. arXiv preprint arXiv:2210.10109 (2022)
[107] Zheng, G., Mukherjee, S., Dong, X., Li, F.: OpenTag: Open aribute value extraction from product profiles. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 1049–1058 (2018). https://doi.org/10.1145/3219819.3219839
[108] Zhong, Z., Liu, F., Wu, Y., Wu, J.: Chinese named entity recognition combined active learning with self-training. Guofang Keji Daxue Xuebao/Journal of National University of Defense Technology 36(4), 82–88 (2014). https://doi.org/10.11887/j.cn.201404015
[109] Zhou, B., Cai, X., Zhang, Y., Guo, W., Yuan, X.: MTAAL: Multi-Task Adversarial Active Learning for Medical Named Entity Recognition and Normalization. In: 35th AAAI Conference on Artificial Intelligence, AAAI 2021. vol. 16, pp. 14586–14593 (2021)
[110] Zhou, M., Duan, N., Liu, S., Shum, H.Y.: Progress in neural NLP: Modeling, learning, and reasoning. Engineering 6(3), 275–290 (2020)
[111] Zhou, S., Liang, S., Yang, Q., Jiang, W., He, Y., Li, Y.: Active Learning Based Labeling Method for Fault Disposal Pre-plans. In: Advances and Trends in Artificial Intelligence. Theory and Applications. pp. 377–382 (2023)
[112] Zhuo, T.Y., Huang, Y., Chen, C., Xing, Z.: Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity (May 2023). https://doi.org/10.48550/arXiv.2301.12867