Deep Bag-of-Words Model: An Efficient and Interpretable Relevance Architecture for Chinese E-Commerce

The text matching task takes textual sequences as input and predicts a numerical value or a category indicating their relationship. Text matching is a long-stand problem and a hot research topic as it’s important in information retrieval and search system. The e-commerce relevance learning can be regarded as a text-matching task. Early work mostly performs keyword-based matching that relies on manually defined features, such as TF-IDF similarity and BM25 (Robertson et al., 1994). These methods cannot effectively utilize raw text features and usually fail to evaluate semantic relevance.

Recently with the development of deep learning, neural-based text-matching models have been employed to solve the semantic matching task and have achieved promising performance. The architecture of the neural-based text-matching model can be roughly divided into the interaction-based model and the representation-based (two-tower) model. The interaction-based model (Hu et al., 2014; Pang et al., 2016, 2016; Wan et al., 2016; Parikh et al., 2016) usually puts all candidate text together as the input. The model can employ the full textual feature to calculate the matching feature as the low layer, and then aggregate the partial evidence of relevance to make the final decision. So interaction-based model can leverage sophisticated techniques in the aggregation procedure and achieve better performance. More recent studies are built upon the pre-trained language model like BERT (Devlin et al., 2019). With an extremely large corpus for pre-training, these methods can achieve new state-of-the-art performance on various benchmarks. The architecture of these models is the pre-trained bidirectional Transformer (Vaswani et al., 2017), which can also be regarded as an interaction-based model. The typical paradigm of the BERT-based relevance model is to feed text pair into BERT and then build a non-linear classifier upon BERT’s [CLS] output token to predict the relevance score (Nogueira et al., 2019; Nogueira and Cho, 2020).

Although having excellent performance in the text-matching task, interaction-based models are still hard to be deployed to practical online service as they are mostly time-consuming, and the features of queries and documents cannot be pre-computed offline. Two-tower models are widely used in many online search systems. The two-tower model consists of two identical neural networks, each taking one of the two inputs. DSSM (Huang et al., 2013; Shen et al., 2014) is a two-tower model that employs two separate deep full-connected networks to encode the candidate texts. Meanwhile, more sophisticated architectures can be adopted to enhance the ability of learning semantic representations. LSTM-DSSM (Palangi et al., 2015) and LSTM-RNN (Palangi et al., 2016) use RNNs to explicitly model word dependencies in the sentences. Typically dot-product, cosine, or parameterized non-linear layers are used to measure the similarity between representations of all candidate texts. Since individually encoding both the queries and documents, the embeddings of them can be pre-computed offline. Therefore, representation-based methods are online efficient and are widely used in industrial search engines. However, the encoding procedure of two inputs is independent with each other, making the final classifier hard to predict their relationship.

Relevance in search engine is a special sub-task of the text-matching which computes the relevance score between the query and the product (as the document). Different from the typical text-matching task which all input texts are semantically similar and homogeneous (i.e. having comparable lengths), the length of query may be much shorter than the length of document. Query only needs to match the partial semantics in the document.

A large number of models are proposed for conducting matching in search. Neural Tensor Network (NTN) (Socher et al., 2013) is originally proposed to explicitly model multiple interactions of relational data. It achieves powerful representation ability that can represent multiple similarity functions, including cosine similarity, dot product, and bilinear product, etc. Qiao et al. (2019) apply the BERT model to ad-hoc retrieval and passage retrieval. Reimers and Gurevych (2019b) propose Sentence-BERT for reducing the computational overhead for text matching. Bai et al. (2020) conduct a pilot study to map the frequency-based and BoW representation of a document to a sparse term importance distribution for text retrieval.

E-commerce search is a special scenario of the Web search system. Both tasks model the semantic matching between query and candidate and require high efficiency and low latency in the online search system. Differently, in Web search the query and document are usually very different in length, making most methods not feasible for the e-commerce relevance task. Currently, there is not a commonly-used public benchmark for the Chinese e-commerce relevance task, so previous works usually evaluate their models on the online service and the real-world dataset constructed from the online platforms. Guo et al. (2016) introduce a typical framework for e-commerce relevance learning. A Siamese network is adopted to learn pair-wise relevance of two products to a query. They investigate training the model with user clicks and batch negatives, followed by finetuning with human supervision to calibrate the score by pair-wise learning. Xiao et al. (2019) propose a co-training framework to address the data sparseness problem by investigating the instinctive connection between query rewriting and semantic matching. Yao et al. (2021) propose to learn a two-tower relevance model from click-through data in e-commerce by designing a point-wise loss function. Zhang et al. (2019) also find the weakness of training with click signals, and address this problem by proposing a multi-task learning framework of query intent classification and semantic textual similarity to improve semantic matching efficiency. Nigam et al. (2019) introduce a 3-part hinge loss to differentiate multiple types of training data. They classified training instances into three categories: random negative examples, impressed but not purchased examples, and purchased items. Recently Yao et al. (2022) propose ReprBERT, which has the advantages of both excellent performance and low latency, by distilling the interaction-based BERT model to a representation-based architecture. This framework is taken as the baseline of our model.

The proposed DeepBoW model is based on the two-tower architecture, which encodes the query and document separately and computes the semantic relevance score with the representations of query and document. Different from other text relevance models with dense embeddings, our model encodes the query and document into the Bag-of-Words vectors and calculates the relevance score from sparse BoW representations.

In this section, we introduce the components of our model in detail. We first describe the multi-granularity encoder to aggregate the character-grained feature and word-grained feature. Next, we introduce two different sparse BoW representations including the term-weighting BoW and the synonym-weighting BoW. Then, we show how to use N-gram hashing to reduce the semantic loss from word segmentation and enhance the quality of the sparse BoW representation. Finally, we describe the training process of our model in detail and show the deployment of our DeepBoW model to the online e-commerce search system. Figure 1 shows an overview of the DeepBoW model.

The text encoder aims to obtain the input text’s contextual representations. We choose Transformer encoder (Vaswani et al., 2017) as our sentence encoder because of its excellent performance in many tasks. The Transformer encoder is a stack of $L$ identical layers, and each layer includes a multi-head self-attention and a fully connected feed-forward network. For the input senquence $S$, we obtain the output encoding matrix of $i$-th layer as $H^{i}=\{h^{i}_{1},h^{i}_{2},\cdots,h^{i}_{l}\}$, where $h^{i}_{j}\in\mathbb{R}^{d}$ is the word embedding vector and $l$ is the number of words in $S$. Same to ReprBERT (Yao et al., 2022), we aggregate the output of each layer as the text encoding representation according to:

Usually the character-based model performs better than the word-based model in Chinese NLP tasks (Li et al., 2019), and most Chinese-BERT models are character-based Transformer architecture. However, since our model intends to catch the relationship between the semantic and the word of the sentence, we propose to encode not only the character-segmentation sequence but also the word-segmentation sequence separately. For convenience, we denote the character-segmentation sequence and the word-segmentation sequence of the input text as $S_{c}=\{c_{1},c_{2},\cdots,c_{m}\}$ and $S_{w}=\{w_{1},w_{2},\cdots,w_{n}\}$, respectively, where $c_{i}$ and $w_{i}$ are the indices of the token in the vocabulary. The output encoding matrices of the character-segmentation sequence and the word-segmentation sequence are $H_{c}=\{h_{c}^{1},h_{c}^{2},\cdots,h_{c}^{m}\}$ and $H_{w}=\{h_{w}^{1},h_{w}^{2},\cdots,h_{w}^{n}\}$, where $m$ and $n$ are the lengths of the character-segmentation and the word-segmentation sequence. The text encoding representation of these two sequences are $h_{c}$ and $h_{w}$.

Unlike traditional two-tower architecture models that encode text into the ”embedding” which is a dense distributed representation, our model encodes the text into the sparse BoW representation. The sparse BoW representation is a set of word-weight pairs, where each word corresponds to a weight that indicates the importance or the relevance of this word to the input text. In this section, we introduce two different sparse BoW representations: term-weighting BoW representation and synonym-weighting BoW representation, and describe the module to generate these two sparse BoW representations in detail.

In the preceding section, we describe the sparse BoW representation in detail. Unfortunately, due to the limitation of model’s parameter size, we can only leverage the vocabulary within a limited number of words. Using the “[UNK]” to replace all Out-Of-Vocabulary (OOV) words may lead to significant semantic loss. To mitigate this issue, we introduce an ensemble of hashing tokens into the vocabulary, where the OOV word can be replaced with its hashing tokens²²2For the word $w$, we leverage $\operatorname{MD5}(w)\%B$ as its hashing token, where $B$ is the hashing bucket number..

Semantic loss may occur between the raw text and its BoW representation, particularly when syntactically cohesive phrases are fragmented during the word segmentation process. This issue can lead to misalignment for the essential semantics such as product types, entity names, or brand identifiers in query/product. For example, the brand name “L’ORÉAL Paris” could be inaccurately divided into separate tokens during word segmentation. To address this problem, we introduce an N-gram hashing vocabulary strategy. Concretely, N-gram phrases are incorporated into the text’s BoW and are subsequently replaced with their respective hashing tokens, analogous to the treatment of OOV words. The significance of a particular N-gram phrase is directly proportional to the frequency of its occurrence within relevant query-product pairs in the corpora. Our model is equipped to ascertain the importance of these N-gram hashing tokens through the analysis of large-scale corpora. Consequently, the semantics of these N-gram phrases are retained within the sparse BoW representation.

In this section, we describe the method to compute the relevance score between the query and the product from the sparse BoW representations. Note that in the search engine scenario the product should match all the semantics of query, while conversely the query does not need to match all the semantics of the product. Accordingly, we encode the query as the term-weighting BoW representation while encode the product as the synonym-expansion BoW representation. The relevance score of the query/product can be calculated as follows:

We leverage the cross-entropy loss between the output score and the ground truth to train our model. In addition, we also optimize the L2 norm of $\operatorname{BoW_{SE}}(D)$ to enhance the sparsity of the synonym-expansion BoW representation, so that only the most relevant words can get high scores. The loss function is as follows:

For both $loss_{t}$ and $loss_{s}$, we optimize the difference between the query’s sparse BoW representation and the product’s sparse BoW representation. This can align the vocabulary between the query and the product.

Since most online search systems have strict latency limitations, we pre-compute the sparse BoW representation of the queries and the products offline. We discard the word-weight pairs in the sparse BoW representation whose weights are lower than the given threshold to optimize memory usage. We sort the word-weight pairs offline by the word index. The relevance score of two sparse BoW representations can be calculated by using the two-pointer algorithm. Then the time complexity of Eq.5 and Eq.7 can be optimized to $\mathcal{O}(N)$, which is much faster than the state-of-the-art deep relevance model (Yao et al., 2022). Although some deep relevance models with cosine similarity can achieve comparable efficiency, the performance of these models is much lower than our model as shown in the section 4.5.

There is no public dataset and benchmark for the Chinese e-commerce relevance task, so we conduct experiments on three different types of industrial datasets to learn the DeepBoW model. The first is a large-scale Human-Annotation dataset which contains query-product pairs sampled from the Taobao search logs. Each query-product pair is labeled as Good (relevant) or Bad (irrelevant) by experienced human annotators. This is a daily task running in Taobao, which has accumulated more than 8 million labeled samples. We split the human-annotated datasets into training, validation and test sets, as detailed in Table 1.

The second dataset for training is built by knowledge distillation, similar to Yao et al. (2022). We leverage the training set of the human-annotation dataset to finetune the StructBERT (Wang et al., 2020) model, which results in an interaction-based teacher model with strong performance. Then the teacher model predicts the relevance scores of the large unlabeled query-product pairs sampled from the search logs of Taobao within a year. This training dataset is denoted as “Search-Logs” in Table 2. Third, we also sample click-through data from search logs and investigate the performance of our model on this training set. We denote this dataset as “Click-Through” in Table 2. Although these are training datasets of different sources, we all use the human-annotation validation and test dataset to evaluate the model performance.

We employ Transformer as both the character-based encoder and the word-based encoder. We reduce the total 12-layer encoder to improve efficiency. After balancing the effectiveness and efficiency, our model adopts 2 layers that can still achieve competitive performance. We select the top 50000 words as the vocabulary $\mathbb{V}$ according to the word’s frequency in corpora, and we also add another 10000 hashing tokens into the vocabulary for the OOV words and the N-gram phrases.

We use “PyTorch” to implement our model and train the model with Adam optimizer. The hyper-parameters of Adam optimizer are $\beta_{1}=0.9,\beta_{2}=0.999,\epsilon=10^{-8}$ and the learning rate is set to 0.0001. Query-document pairs are batched together by approximate sequence length. Each training batch contains a set of sentence pairs with about 50000 tokens. The hyper-parameters and the best model are chosen from the experimental results on the validation set. We train our model on 2 Tesla V100 GPU and it usually takes about 3 days for the model to converge. The convergence is reached when the ROC-AUC does not improve on the validation set.

We explore the performance of DeepBoW(Q-Weight) and DeepBoW (Q-Synonym) respectively. The main difference between the two methods is for DeepBoW(Q-Synonym) we leverage the synonym-expansion BoW representation to replace the term-weighting BoW representation for the query. To reduce memory usage and computation, we truncate the BoW representation to make it sparse. There are two methods to truncate the sparse BoW representation, one is to keep the $k$ largest words according to their respective values, and the other is to discard the terms whose values are smaller than the giving threshold.

In addition, we adopt several state-of-the-art methods for comparison. BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) and StructBERT (Wang et al., 2020) belong to the interaction-based architecture which is also known as the cross-encoder architecture. Siamese BERT (Jiang et al., 2019b), MASM (Yao et al., 2021), Poly-encoders (Humeau et al., 2019) and ReprBERT (Yao et al., 2022) belong to the two-tower architecture which is also known as the bi-encoder architecture. Besides, we investigate the performance of ReprBERT with cosine similarity score instead of MLP for online computation of relevance from query/product embeddings. For fair comparison, we also leverage the pre-trained model like RoBERTa and StructBERT as the encoder of the two-tower model. These models are also baselines in our experiment, which denote as DSSM RoBERTa and DSSM StructBERT .

We evaluate our model on both offline and online metrics. In offline evaluation, since the human annotation is binary, the task is evaluated as a classification task. The Receiver Operator Characteristic Area Under Curve (ROC-AUC) is widely adopted in text relevance tasks (Jiang et al., 2019a; Wan et al., 2016; Davis and Goadrich, 2006). Note that in the e-commerce relevance scenario, most instances are positive and we are more concerned about negative instances. Therefore the PR-AUC used in this paper is the negative PR-AUC that treats Bad as 1 and Good as 0 following Yao et al. (2021, 2022). This metric is denoted as Neg PR-AUC.

Besides, we also evaluate the different model complexity of parameters and online computation efficiency. The FLOPs / token is computed according to Molchanov et al. (2017) which shows the floating-point operations per second (FLOPs) when there is only 1 token being considered. The plus sign separates the online and offline calculation FLOPs, which means the former part of computation can be pre-computed offline. Memory indicates the online memory overhead for storing pre-computed query and product vectors where we use vector size for comparison. In online evaluation, we use the rate of Good annotated by human annotators and the number of transactions as the evaluation metrics. The query-product pairs for human relevance judgment are randomly sampled from the online search logs according to the amount of Page View (PV) as the sample weight.

Table 2 presents an evaluative comparison across various models. Our DeepBoW model demonstrates robust performance across three different training sets. For human-annotation dataset, StructBERT has the best performance, and interaction-based models outperform two-tower models. Unfortunately, it is infeasible to deploy the interaction-based model in industrial system because of prohibitive computation and resource requirements. The ROC-AUC and Neg PR-AUC of two-tower models are much lower than the pre-trained model, because the human-annotation data is insufficient and the pre-trained model can introduce extra knowledge. Nonetheless, our DeepBoW model still outperforms other two-tower models.

The data enhancement sampled from the search logs and labeled by the teacher model can greatly improve the performance of two-tower models. Our model achieves the best performance around the two-tower models in search-logs training sets. Click-through data is used to train the relevance model in some cases where lack of human-annotation training data. The models trained on the click-through data get weak performance. The main reason is that the click-through data in e-commerce is much more noisy and misleading, which is not only affected by the query-product relevance but also by many factors including price, attractive titles, or images. Even so, our model also performs better than the other models since it explicitly encodes the semantics to the sparse bag-of-words while the other models may capture the personalized information beyond the textual feature.

Our proposed architecture can truncate the sparse BoW representation to reduce memory usage in the online search system. We can either truncate the sparse BoW representation to the fixed length or discard the word-weight pairs whose values are smaller than a given threshold. Benefiting from the interpretability of our sparse BoW representation, both truncation methods achieve competitive performance and only produce a slight loss of performance. We further explore the distribution of the sparse BoW representation as case study in Appx. A.

Table 3 shows the parameter, computation and memory consumption of each model. While our model has a considerable number of parameters since we project the dense vector into the vocabulary (78.8% parameters about 126M come from $\mathbf{W_{c}}$ and $\mathbf{W_{w}}$ in Eq.LABEL:projection), DeepBoW is the most efficient at online inference. Our model only uses CPU to compute the relevance score while the other models need GPU to speed the inference. Table 4 shows the inference time of each model. The experiments are performed on a local CPU platform. We report the average inference time of the model to score 1000 products per query. The results show that our model is much faster than the ReprBERT (Yao et al., 2022) which has been deployed in the Taobao search system.

We perform the ablation study on the human-annotation dataset to investigate the influence of different modules in our model. We investigate the performance of the 2-layer encoder and the 6-layer encoder. Besides, we remove the word encoder and the character encoder separately to show the importance of encoding at both word-level and character-level. We also employ different capacities of vocabulary and hashing tokens in the sparse BoW representation. To further explore the sparsity of BoW representation, we remove the $l_{2}$ norm from the loss function. This may lead to a sparsity deterioration in the sparse BoW representation.

Table 5 shows the results of the ablation study. We can see that each module in our model does contribute to the overall performance. The performance of our model has a significant deterioration if either the word encoder or the character encoder is removed. Increasing the capacity of vocabulary or hashing tokens cannot improve the model’s performance, because it may lead to insufficient training for each token. Optimizing our model without the $l_{2}$ norm loss can lead to slight performance decline. There is almost no difference between the 2-layer and 6-layer encoders in our model.

Online A/B testing is also conducted to evaluate our DeepBoW model, by replacing the online ReprBERT model with the DeepBoW model for comparison. Both experiments take about 2.5% proportion of Taobao search traffic, and the A/B testing lasts for a month. As a result, DeepBoW improves the number of transactions by about 0.4% on average. The daily human annotation results show that DeepBoW also improves the rate of relevance by 0.5%. Online A/B testing verifies the proposed DeepBoW is superior to previous state-of-the-art models, and can achieve significant online profit considering the extremely large traffic of Taobao every day.

Our DeepBoW model has already served the entire Taobao search traffic. After pre-computing the representations of queries and products, the online serving latency can be optimized to as low as 4ms on the distributed computing system with CPUs. This is much faster than the previous online relevance serving model ReprBERT (Yao et al., 2022) of 10ms with GPUs and can satisfy the extremely large traffic of Taobao.