Domain-Hierarchy Adaptation via Chain of Iterative Reasoning for Few-shot Hierarchical Text Classification

This paper mainly focuses on few-shot HTC tasks. Different from previous works that assume a rich labeled dataset, we adopt the few-shot setting, i.e., only a limited number of samples are available for fine-tuning which is more practical as it assumes minimal resources. We follow the greedy sampling algorithm proposed in Ji et al. (2023) to obtain the support set S from the original HTC datasets, ensuring that each label path has exactly K-shot samples. Formally, the K-shot HTC task is defined as follows: given a text x and a K-shot support set S for the target mandatory-leaf path set $\textit{C}_{\mathcal{T}}$, the goal is to predict all golden paths on the label hierarchy tree, where $\textit{C}_{\mathcal{T}}$ consists of all paths from the root node to the leaf nodes on the label hierarchy tree $\mathcal{H}$. In our settings, there is only one correct label for each layer.

Figure 2 shows the overall architecture of our proposed HierICRF. Since the label structure that this paper focuses on is a tree, we view the HTC task as a process of a chain of thoughts via iterative CRF. Specifically, in the first stage, we construct a hierarchy-aware reasoning chain $T_{chain}$ as our prompt to deepen the process of hierarchy-aware reasoning. During the second stage, we use $T_{chain}$ to guide the generator to generate hierarchically repeated series t, the t will be fed into a verbalizer to obtain their MLM logits $z$ in the label hierarchy tree. Finally, during layer-to-layer transfer over the MLM logits $z_{i}$ of the series $\textbf{{t}}_{i}$, we use a hierarchical iterative CRF (ICRF) to model its path routing by incorporating hierarchical information into the transition matrix of ICRF.

Due to the complex label dependency in HTC task, it is hard to adapt flat prior knowledge in PLMs to downstream hierarchical tasks. Besides, hierarchical dependency information between labels at different layers or the same layers may be implicit. To address the above issues, inspired by the chain of thought (COT) Wei et al. (2022) and math Zhu et al. (2023) that solve mathematical or planning-related problems by decomposing complex problems into sub-processes, we propose a simple but effective method named hierarchy-aware reasoning chain to elegantly flattens HTC into a hierarchical loop-based language modeling process, thus allowing it to fully exploit the capabilities of PLM. The details of obtaining the prompt of the hierarchy-aware reasoning chain named $T_{chain}$ are shown in Algorithm 1. For example, when the depth of the hierarchy tree is 2 and the number of iterations $T_{chain}$ is 2, the reasoning chain template $T_{chain}$ is simply like ”x. It was 1 level: [MASK] 2 level: [MASK] 1 level: [MASK] 2 level: [MASK] 1 level: [MASK]. ”. The $T_{chain}$ will later be used to guide the model to generate hierarchically repeated series.

To verify the ability of our method for few-shot domain-hierarchy adaptation, we implement our method on both BERT and T5. We first feed the input text $x$ wrapped with template $T_{chain}$ into the encoder to obtain the hidden states $\textbf{{h}}_{1:n}$:

where $\textbf{{h}}_{1:n}\in\mathbb{R}^{n\times r}$, and r denotes the hidden state dimension of encoder and n represents the length of $T_{chain}(x)$. We then obtain target hierarchically repeated series $\textbf{{t}}_{1:l}=\{\textbf{{t}}_{i}\}$ through:

For encoder-only LMs: Generator($\cdot$) means to directly extract a subset $\{\textbf{{h}}_{j}\}$ consisting of hidden state vectors corresponding to all [MASK] tokens from $\textbf{{h}}_{1:n}$ as our $\textbf{{t}}_{1:l}$. For encoder-decoder LMs: Generator($\cdot$) represents feeding $\textbf{{h}}_{1:n}$ into and prompt its decoder to obtain our final $\textbf{{t}}_{1:l}$.

Furthermore, we construct a flat-verbalizer V based on all labels on the hierarchical tree for label mapping learning. The verbalizer is represented as a continuous vector $\textbf{{W}}_{V}\in\mathbb{R}^{r\times m}$, where $m$ signifies the number of labels. The embedding $\textbf{{W}}_{V}$ is initialized by averaging the embeddings of its corresponding label tokens. We feed the series {$\textbf{{t}}_{i}$} into the verbalizer to get the emission probabilities $z=\{z_{1},...,z_{l}\}$.

After obtaining the label distribution of each step in the hierarchy-aware reasoning chain, instead of directly classifying, we regard this hierarchically repeated series as a process of hierarchical path routing step-by-step. Inspired by CRF Sutton et al. (2012) widely used to model the transition of state space of time series in named entity recognition Nadeau and Sekine (2007), here we model this sequencing process using hierarchical iterative CRF, injecting hierarchical constraint by optimizing transition matrices.

Formally, given a sentence x, we use z = {$\textit{z}_{1},...,\textit{z}_{l}$} to represent a generic series where $z_{i}$ is the emission probability of the i-th routing step. $y=\{y_{1},...,y_{n}\}$ represents the golden labels for z. The probabilistic model of a sequence CRF defines a set of conditional probabilities $p(y|z;\textbf{{W}},b)$ over all possible label sequences y given z as:

where $\psi_{i}(y^{\prime},y,z)=\exp({\textbf{{W}}_{y^{\prime},y}^{T}z_{i}+b_{y^{\prime},y}}$) are potential functions, and $\textbf{{W}}_{y^{\prime},y}^{T}$ and $b_{y^{\prime},y}$ denote the weight and bias corresponding to label pair $(y^{\prime},y)$, respectively. The final training objective is calculated as:

Additionally, we initialize the transition scores between non-adjacent layers to a minimum to avoid erroneous cross-layer transfers. The transition score between Earthquake and Species in Figure 2 is initialized to 0 before training.

After training, the decoding is to search for the label sequence $y^{*}$ with the maximum conditional probability:

where we pick out the outputs of the last path routing iteration from $y^{*}$ as our final predictions.

Main experimental results are reported in Table 2. By optimizing from a path routing perspective, HierICRF outperforms overall comparison baselines under most situations.

We first find out that prompt-based methods outperform vanilla FT by a dramatic margin in the case of no more than 4-shot settings. On average, 67.01%, 13.30%, and 10.01% Micro-F1 improvements are achieved by HierICRF-BERT compared to the vanilla FT. However, although HPT is designed as a prompt-based method, its few-shot results are not satisfactory. The reason is obvious since the overfitting problem of the GNN layers becomes serious when labeled data used for fine-tuning is limited.

Second, HierICRF-BERT achieves 1.53%, 1.14%, and 2.28% Macro-F1 improvements from the best baselines on 1, 2, and 4-shot on WOS, respectively. Besides, HierICRF-T5 has a substantial improvement with an average Macro-F1 of 2.08% on WOS and 0.46% on DBpedia compared to the previous SOTA HierVerb while keeping competitive with HierICRF-BERT on Micro-F1.

Additionally, it is clear that both the HierICRF-BERT and HierICRF-T5’s Micro-F1 and Macro-F1 change very slightly from 1 to 16 shots on DBpedia while other models except HierVerb are particularly dependent on the increase of labeled training samples. For example, as the shots become fewer, the HierICRF-BERT’s Micro-F1 changes from 96.22% to 92.05% while HGCLR’s Micro-F1 changes from 95.49% to 15.73%. The results indicate that our method can efficiently mine the prior knowledge in pre-training for hierarchical tasks in the case of extremely few samples instead of relying too much on limited training samples to optimize the model.

Table 3 further studies the consistency performance. Our method still maintains SOTA consistency performance in the absence of labeled training corpora. It is clear that HGCLR and BERT (Vanilla FT) which uses the direct fitting method only achieve 0 points in PMicro-F1 and PMacro-F1 under the 1-shot setting. Compared with the best-performing baseline HierVerb, on average, HierICRF-BERT outperforms 7.08% and 4.25% PMicro-F1 scores, and 6.26% and 3.92% PMacro-F1, and 4.38% and 1.79% CMicro-F1, and 9.3% and 4.38% CMacro-F1 scores on WOS, DBpedia, respectively. As for HPT and HierVerb, directly extra embedding injection to the pre-trained LM pays less attention to the hierarchy consistency. The results highlight that the proposed paradigm allows model to optimize from a path routing perspective, more consideration is given to the label dependency during the process of iteratively transiting between layers on the hierarchically repeated series to better deal with the hierarchical inconsistency problem. Surprisingly, when more training data are given, the performance gap between HierICRF and all other baselines gradually decreases as we may hypothesize. We conjecture that it is because although all baseline methods lack domain-hierarchy adaptation in the paradigm, data-hungry based methods such as GNN can still learn hierarchical consistency dependency through directly overfitting, thereby gradually improving consistency performance.

To illustrate the effect of our proposed mechanisms, we conduct ablation studies on WOS, as shown in Table 4. When both ICRF and CHR are removed, HierICRF is equivalent to Vanilla SoftVerb. Removing ICRF results in significant performance degradation, with an average of 1.72% Micro-F1 and 2.38% Macro-F1 on BERT and 1.72% Micro-F1 and 2.37% Macro-F1 on T5, meaning that ICRF plays an important role in incorporating hierarchy dependencies. Furthermore, the performance decreases even more when both ICRF and CHR are removed, e.g., in the 4-shot case, Macro-F1 even drops by 6.57% compared to the full method deployed on BERT. This firmly highlights the significance of combining chain of hierarchy-aware reasoning and hierarchical iterative CRF allows HierICRF to gradually inject label hierarchy information by feeding back the accumulated error transition of label nodes in the process of path routing step-by-step.

We conduct experiments using a full-shot setting and use the hyperparameter set directly from the few-shot settings. For the baseline models, we reproduce their experiments based on the settings in their original paper. While the main focus of our work is on the performance under few-shot settings, it is worth noting that HierICRF surprisingly outperforms all baselines, such as HGCLR, HPT, and HierVerb, in the full-shot setting. As shown in Table 5, our Micro-F1 is slightly higher than HGCLR, HPT, and HierVerb, while Macro-F1 significantly beats them by 0.28% on average. Besides, HierICRF significantly outperforms BERT (Vanilla FT), HiMatch, and SoftVerb.

To further study the effects of the hierarchy-aware reasoning chain, we conduct experiments on the iteration length of the reasoning chain. As we can see in Table 6, the performances on both WOS and DBpedia degrade substantially as $I_{chain}$ decreases, with an average Micro-F1 of 1.82% and 0.54% on WOS and DBpedia. The results show that our method can correct more hierarchical inconsistencies between time steps and feedback to model optimization when performing hierarchical transitions in longer hierarchically repeated series.

To further study the ability of HierICRF to utilize the prior knowledge of the PLMs, we conduct experiments on BERT-large-uncased and T5-large. Table 7 demonstrates that HierICRF consistently outperforms all baseline models in all shot settings. We find that the gap is even significantly larger for HierICRF and all other baseline models compared to using BERT-base-uncased. For example, compared with HierVerb, HierICRF-BERT(BERT-large-uncased) achieves a 1.53% Macro-F1 and a 0.45% Micro-F1 scores increase under 1-shot setting. But the improvements of Macro-F1 and Micro-F1 are 1.81% and 0.81% under BERT-base-uncased, respectively. Surprisingly, when using T5-large, we find that its performance improvement relative to T5-base is greater than that of the improvement obtained by all all BERT-based methods (including HierICRF-BERT) from BERT-base-uncased to BERT-large-uncased, and even achieve SOTA under no more than 2-shot settings. The findings further underscore that HierICRF outperforms all baseline models in effectively leveraging the prior knowledge embedded within larger language models. This advantage becomes even more pronounced as the scale of the language model increases, highlighting the significant impact of HierICRF’s ability to harness this expansive prior knowledge.