We include all entities for i2b2 2010, ClinicalIE, and CLEF 2014. For i2b2 2012, we found that UniversalNER and GNER performed poorly on the last two entities (e.g., evidence and occurrence) and decided to exclude them. We attribute this to them consisting mostly of verb phrases, while the training dataset consists mainly of noun entities. For i2b2 2018 Task 2, we test our method on a more challenging setup, extracting adverse drugs and adverse drug events [henry20202018].

We provide the precisions and recalls for each dataset and entity type from \autoreftable_main in \autoreftable_precision and \autoreftable_recall respectively. We observe a similar trend for both metrics. Furthermore, we observe that UniNER benefits more from precision and GNER on recalls using our framework.

We experiment with different ways to prompt in \autorefsubsubsec_result_filter_prompt and provide the specific instructions here.

Here, we tried including some annotated samples in our framework and compared the approach to standard in-context learning. We randomly sample from the annotation guidelines and add them to the UniversalNER prompt. We also guarantee that there is at least one sample without entities of interest (e.g., sentence does not contain treatments or medical problems). Interestingly, we observe performance degradation across entity types the more samples we use. These contrastive results to general LLMs [xie2023empirical] further justify that open NER LLMs cannot be treated similarly to them. Furthermore, we observe that this also applies to our framework, although it is not as severe as standard in-context learning. We remark that performance drops on in-context learning are not uncommon. Previous works [zhao2021calibrate, zhu2023promptbench] show instability in performance for in-context learning. In addition, few-shot experiments are uncommon for zero-shot NER task [zhou2023universalner, ding2024rethinking, zaratiana2023gliner], even if they use LLMs. Our work reveals that open NER LLMs may not benefit from in-context learning and are different from general LLMs. We leave further investigation to future works.

We use GLiNER [zaratiana2023gliner], a BERT-based model for open-named entity recognition. Note that previous prompt engineering methods cannot be applied here. We conduct the experiment similar to UniNER and GNER, with Xie et.al. [xie2023empirical] as our baseline. We present the results in \autoreftable_gliner. We observe the same trend as in \autorefsubsec_overall_perf with an average of $7.29\%$ F1-score improvement.

We observe significant recall drops to “clinical department” entities across all models. Here, we posit that some entities may not necessarily conform to the clinical department in the clinical domain. For instance, some entities are hospitals; thus, if a filter is prompted with our template (e.g., “Can hospitals be considered as a clinical department?”), then it is likely to reject them. One possible solution is using a clear entity description. As illustrated in \autoreftable_filter_ent_desc, our framework outperforms the baseline (e.g. $45.97\%$ vs $38.66\%$ F1-score respectively) when using entity description.

We analyze how filtering can be made more or less strict to achieve better trade-offs. We use the filter output probability to determine whether the entity is rejected or not. Concretely, rather than directly rejecting them if the filter outputs “No”, we first look at the token probability. If it is less than a certain threshold, we then reject them. Our framework is simplified to entity decomposition if the threshold is $1$. We provide the results in \autoreffig_filter_tradeoff. Overall, increasing the threshold leads to decreased precision and improved recall. Interestingly, better thresholding can improve the F1 Score in “clinical department” entities. This might be due to the noises for the entities as described in \autorefapp_drop_cd.

Our experiments involve large language models, which are often trained with specific templates. We use their default templates (except Llama2) throughout the experiments and present them here.

We use the default hyperparameters for each model. For UniNER and GNER, we use greedy search. For Asclepius and Llama2, we use temperature $0.2$ and top $P$ probability $0.95$.