Unified Interpretation of Smoothing Methods for Negative Sampling Loss Functions in Knowledge Graph Embedding

KGC is a research topic for automatically inferring new links in a KG that are likely but not yet known to be true. To infer the new links by KGE, we decompose KGs into a set of triplets (links). By using entities $e_{i}$, $e_{j}$ and their relation $r_{k}$, we represent the triplet as $(e_{i},r_{k},e_{j})$. In a typical KGC task, a KGE model receives a query $(e_{i},r_{k},?)$ or $(?,r_{k},e_{j})$ and predicts the entity corresponding to $?$ as an answer.

In KGE, a KGE model scores a triplet $(e_{i},r_{k},e_{j})$ by using a scoring function $s_{\mathbf{\theta}}(x,y)$, where $\mathbf{\theta}$ denotes model parameters. Here, using a softmax function, we represent the existence probability $p_{\mathbf{\theta}}(y|x)$ for an answer $y$ of the query $x$ as follows:

where Y is a set of entities.

To train $s_{\mathbf{\theta}}(x,y)$, we need to calculate losses for the observables $D=\{(x_{1},y_{1}),\cdots,(x_{n},y_{n})\}$ that follow $p_{d}(x,y)$. Even if we can represent KGC by Eq. (1), it does not mean we can tractably perform KGC due to the large number of Y in KGs. For the reason of the computational cost, the NS loss (Mikolov et al., 2013) is used to approximate Eq. (1) by sampling false answers.

By modifying that of Mikolov et al. (2013), the following NS loss (Sun et al., 2019; Ahrabian et al., 2020) is commonly used in KGE:

where $U$ is the noise distribution that follows uniform distribution, $\sigma$ is the sigmoid function, $\nu$ is the number of negative samples per positive sample $(x,y)$, and $\tau$ is a margin term to adjust the value range decided by $s_{\mathbf{\theta}}(x,y)$.

As shown in Figure 1, KGC needs to deal with the sparsity problem caused by low frequent queries and answers in KGs. Imposing smoothing on the appearance frequencies of queries and answers can mitigate this problem. The following subsections introduce subsampling (Mikolov et al., 2013; Sun et al., 2019; Kamigaito and Hayashi, 2022a) and SANS (Sun et al., 2019), the conventional smoothing methods for the NS loss in KGE.

TANS is fundamentally different from SANS, with SANS only taking into account the conditional probability of negative samples and TANS being a loss function that considers the joint probability of the pair of queries and their answers.

Table 1 shows the characteristics of TANS and the conventional smoothing methods of the NS loss for KGE introduced in §2.3. These characteristics are based on the decomposition of $p_{d}(x,y)$, the appearance probability for the triplet $(x,y)$, into that of its answer $p_{d}(y|x)$ and query $p(x)$:

In Eq. (10), smoothing both $p_{d}(y|x)$ and $p_{d}(x)$ is similar to smoothing $p_{d}(x,y)$. However, smoothing $p_{d}(x,y)$ does not ensure smoothing both $p_{d}(x)$ and $p_{d}(y|x)$ considering the case of only one of them being smoothed, and the left one being still sparse. Similarly, smoothing only $p_{d}(x)$ or $p_{d}(y|x)$ does not ensure $p_{d}(x,y)$ being smoothed due to the case where one of them is still sparse. In Table 1, we denote such a case where the method can influence the probability, but no guarantee of the probability be smoothed as $\triangle$.

In TANS, we aim to smooth $p_{d}(x,y)$ by smoothing both $p_{d}(y|x)$ and $p_{d}(x)$ based on Eq. (10).

Here, we induce TANS from SANS with targeting to smooth $p_{d}(x,y)$ by smoothing both $p_{d}(y|x)$ and $p_{d}(x)$. First, we assume a simple replacement from $p_{\theta}(y|x)$ to $p_{\theta}(x,y)$ in $\ell_{\text{SANS}}(\mathbf{\theta})$ of Eq. (9):

However, using Eq. (11) causes an imbalanced loss between the first and second terms since the sum of $p_{\theta}(x,y_{i})$ on all negative samples is not always 1. Thus, Eq. (11) is impractical as a loss function.

As a solution, we focus on the decomposition $p_{\mathbf{\theta}}(x,y)=p_{\mathbf{\theta}}(y|x)p_{\mathbf{\theta}}(x)$ and the fact that the sum of $p_{\mathbf{\theta}}(y|x)$ of all negative samples is always 1. By using $p_{\mathbf{\theta}}(x)$ to make a balance between the first and second loss term, we can modify Eq. (11) and induce our TANS as follows:

where $\gamma$ is a temperature to smooth the frequency of queries. Since TANS uses a noise distribution decided by $p_{\theta}(x;\gamma)$ and $p_{\theta}(y_{i}|x;\beta)$, it does not propagate gradients through probabilities for negative samples, and thus, memory usage is not increased.

In this subsection, we discuss the difference and similarities among TANS and other smoothing methods for the NS loss in KGE. As shown in Table 1, the subsampling methods, Base and Freq, can smooth triplet frequencies similar to our TANS. To investigate TANS from the view point of subsampling, we reformulate Eq. (12) as follows:

Apart from the temperature terms, $\alpha$, $\beta$, and $\gamma$, we can see that the general formulation of subsampling in Eq. (3) and the above Eq. (14) has the same formulation. Thus, TANS is not merely an extension of SANS but also a novel subsampling method.

Even though their similar characteristic, TANS and subsampling have an essential difference: TANS smooths the frequencies by model-predicted distributions as in Eq. (13), and the subsampling methods smooth them by counting appearance frequencies on the observed data as in Eq. (4), (5), (6), and (7). For instance, TANS can work even when the entity or relations included in the target triplet appear more than once, which is theoretically different from conventional approaches.

Since the superiority of using either model-based or count-based frequencies depends on the model and dataset, we empirically investigate this point through our experiments.

We formulate the unified loss function by introducing subsampling (Eq. (3)) into our TANS (Eq. (12)) as follows:

where $\eta$ is a hyperparamter that can be any value to absorb the difference among the three different subsampling methods, Base, Uniq, and Freq.

Here, we can induce the NS losses shown in our paper from Eq. (16) by changing the temperature parameters $\alpha$, $\beta$, and $\gamma$. Table 2 shows the induced losses from our $\ell_{\text{UNI}}(\mathbf{\theta})$. Note that since $p_{\theta}(x;\gamma)$ only appears in our TANS, canceling $p_{\theta}(x;\gamma)$ by $\gamma=0$ induces an inequivalent but a proportional relationship to the conventional NS loss.

As shown in Table 2, TANS w/ subsampling has characteristics of all smoothing methods for the NS loss in KGE introduced in this paper. Therefore, we can expect higher performance of TANS w/ subsampling than the combination of conventional methods, the basic NS, SANS, and subsampling. However, because TANS w/ subsampling uses subsampling in §2.3, we need to choose the one from Base, Uniq, and Freq for TANS w/ subsampling. Since this part is out of the scope of theoretical interpretation, we investigate this in the experiments.

Since the result tables are large⁷⁷7The full experimental results are listed in Appendix B. The scores are included in Table 5, 6, and 7 of Appendix B.1. The training loss curves and validation MRR curves for each smoothing method are in Figure 6, 7, and 8 of Appendix B.2., we discuss them individually, focusing on important information in the following subsections.