Using LLMs for the Extraction and Normalization of Product Attribute Values

The goal of product attribute value extraction is to identify attribute-value pairs in product descriptions. To enable features like faceted product filtering and product recommendation, the extracted attribute values must be normalized to be represented on single, attribute-specific scales. Figure 1 shows an example of a product offer together with attribute-value pairs that have been extracted from the product title. The attribute values are shown as directly extracted values and on the right as normalized values. Existing methods for product attribute value extraction often require large amounts of domain-specific training data to learn extraction rules [3, 8, 13], label attribute value sequences [5, 15, 19] or extract attribute values using question answering [10, 16]. The methods focus on extracting sequences of tokens that form attribute values but do not cover the normalization of the extracted values, despite value normalization being required in many use cases [4].

Motivated by the success of large language models (LLMs) in related NLP tasks [12, 18] and other information extraction use cases [1, 2, 6, 9, 11], this paper explores the potential of LLMs for product attribute-value extraction for two scenarios: (i) direct extraction and (ii) extraction with normalization. In the direct extraction scenario, the objective is to extract the exact surface form of attribute values from product titles and descriptions [10, 14, 15, 16]. In the extraction with normalization scenario, the goal is to extract and normalize attribute-value pairs. As current benchmarks for extracting product attribute values are solely designed for measuring extraction quality and do not cover the normalization of the extracted data [16, 14, 17], we introduce a new dataset, WDC Product Attribute-Value Extraction (WDC PAVE). WDC PAVE consists of 1,420 product offers that have been annotated by 87 websites using the schema.org vocabulary. The 24,583 attribute-value pairs have been verified manually. The attribute-value annotations are available in two formats: (i) extraction only and (ii) extraction with normalization. To normalize the values, the systems need to perform name expansion, generalization, unit of measurement conversion, and string wrangling. In summary, we make the following contributions:

1. 1.

   We propose prompt templates for instructing LLMs to extract and normalize attribute-value pairs from product titles and descriptions. The templates cover use cases with and without training data.

2. 2.

   We introduce the new dataset WDC PAVE containing 1,420 heterogeneous product offers and 24,583 manually verified attribute-value pairs, which are prepared for the two scenarios: (i) extraction and (ii) extraction with normalization. In contrast to existing benchmarks [16, 14, 17], WDC PAVE covers value extraction and value normalization.

3. 3.

   We experimentally compare the extraction performance of GPT-3.5 and GPT-4 and PLM-based methods on WDC PAVE. GPT-4 achieves the overall best results with an F1-score of 91% and outperforms PLM-based methods by 10%.

4. 4.

   We experiment with extracting and normalizing attribute values using GPT-3.5 and GPT-4. When using training data for example value and demonstration selection, GPT-4 performs particularly well for string wrangling and name expansion with F1-scores of 95% and 98% respectively. In contrast, existing work on attribute value normalization using LLMs [4] does not consider the attribute value context for the normalization.

This paper is structured as follows: Section 2 introduces the new dataset WDC PAVE. Section 3 describes the experimental setup. Section 4 and Section 5 discuss the results of the experiments for both scenarios: (i) extraction and (ii) extraction and normalization of product attribute values. The dataset WDC PAVE and the code for replicating all our experiments are available online¹¹1https://github.com/wbsg-uni-mannheim/wdc-pave.

This section introduces the WDC PAVE dataset. First, we describe the collection of product offers and attribute-value pairs using schema.org annotations and specification tables. Second, the statistics of WDC PAVE are profiled. Third, the normalization operations name expansion, generalization, unit of measurement normalization, and string wrangling are introduced.

For our experiments, we split WDC PAVE into a training and a test set with a ratio of 75:25, stratified by product category. A random subset of 20% of the training records per product category is used to select in-context demonstrations and example values to simulate a scenario with a few labeled attribute-value pairs. Attributes that do not require value normalization are removed from both the training and test sets. We access GPT-3.5-turbo-0613 referred to as GPT-3.5 and GPT-4-0613 referred to as GPT-4 through the OpenAI API. The temperature parameter of the LLMs is set to 0 to reduce the randomness. As baselines for the experiments, we finetune the PLM-based extraction methods SU-OpenTag [14], AVEQA [10] and MAVEQA [16] on the training set. For the evaluation, we follow related work and calculate F1-scores based on the exact match between the predicted and the ground truth attribute values [2, 10, 14, 15, 16].

In this section, we evaluate prompt templates for extracting attribute-value pairs from product offers in WDC PAVE. Following Brinkmann et al. [2], the prompts ask the LLM to extract all attributes in the target schema in a single step. Each template consists of up to six building blocks that are shown in Figure 2. The role description (blue) defines the overall goal of the LLM as well as the target schema for the extraction including attribute descriptions and example values. The target schema is encoded as a JSON-schema because this this representation proved to be most successful in related work [2]. The task description (blue) contains instructions for the attribute-value extraction. The task input (green) consists of the product title and the product description. The task output (orange) contains the LLM’s extracted attribute-value pairs in JSON format. The demonstration task input (green) and output (yellow) contain demonstrations with product offers that are semantically similar to the target product offer and are selected from the training set. Figure 2 contains the chat message building blocks for both scenarios: extraction and extraction with normalization. The red text is added to the prompts for the extraction and normalization scenario.

We assess the impact of zero-shot and few-shot prompt template configurations on the performance of GPT-3.5 and GPT-4. Table 3 shows the F1-scores of the prompt template configurations with different amounts of example values (val.) and the combination of 10 example values and different amounts of demonstrations (dem.). The highest F1-score with and without demonstrations is marked in bold. The results show that both GPT-3.5 and GPT-4 benefit from ten example values, achieving F1-scores of around 80%. Demonstrations further improve the performance of GPT-3.5 and GPT-4. Ten demonstrations allow GPT-4 to achieve the highest F1-score of 91%. We compare the performance of GPT-3.5 and GPT-4 with the PLM-based baselines SU-OpenTag [14], AVEQA [10] and MAVEQA [16] fine-tuned on the same training set that is used to select example values and task demonstrations. The GPT-4 using the 10-dem prompt template outperforms the best PLM-based baseline AVEQA by 10%.

This section evaluates prompt templates for the scenario in which attribute values are extracted from product offers and normalized to be comparable to other attribute values of the same attribute. The prompt template in Figure 2 is extended by the text in red. The extensions add normalization guidelines to the task description, mappings of example values to their normalized forms to the target schema in the role description, and use normalized values in the task output of the demonstrations.

As in Section 4, we assess how zero, three, five and ten example values without demonstrations and ten example values with three, five and ten demonstrations selected from the training set impact the performance of GPT-3.5 and GPT-4. Table 4 shows the F1-scores of the experiments. The highest F1-scores with and without demonstrations are marked in bold. GPT-4 benefits from normalized example values and reaches an F1-score of 86%, which is 12% better than the zero-shot configuration. In contrast to the extraction scenario, example values only marginally improve the extraction and normalization performance of GPT-3.5. However, adding five demonstrations enhances the performance of both GPT-3.5 and GPT-4, with GPT-4 achieving an F1-score of 91% and outperforming GPT-3.5 by 5%.

We now analyze the normalization operations in detail. Table 5 shows the F1-scores for the best configuration with and without demonstrations per normalization operation. Previous research has demonstrated that GPT-3’s performance is weaker in tasks that require reasoning or calculations, as opposed to tasks that involve manipulating free text or names [4]. Our results support these observations, GPT-3.5 and GPT-4 achieve the best zero-shot result for the string wrangling operation. With ten demonstrations, GPT-3.5 and GPT-4 are particularly strong at name expansion and string wrangling. GPT-4 achieves F1-scores of 98% and 95%. Unit of measurement normalization requires calculations and is the most challenging operation for GPT-3.5 and GPT-4. Example values and demonstrations improve GPT-4’s zero-shot performance by 22%, leading to an F1-score of 84%. Compared to pure extraction, we observe that the generalization of attribute values simplifies the task for GPT-4, likely because it can utilize its background knowledge for the generalization while making fewer errors in choosing the correct attribute value boundaries. For generalization attributes like the ’Product Type’, we observe across the five product categories that GPT-4 benefits from the generalization on average by 7% if example values and demonstrations from the training set are provided.

In this paper, we investigate the ability of GPT-3.5 and GPT-4 to extract product attribute values from product offers and normalize the attribute values to enable downstream e-commerce applications like faceted product filtering. For the evaluation, we introduce the WDC PAVE dataset, featuring manually verified ground truth values for both tasks, extraction and extraction with normalization. We experiment with different prompt designs that utilize example values and demonstrations selected from a training set.GPT-4 achieves the best F1-score of 91% in the extraction scenario, surpassing the best PLM baseline by 10%, and shows similar performance for the extraction with normalization scenario. GPT-4 excels in wrangling strings and in expanding names, where it can utilize its background knowledge. Normalization tasks that require reasoning such as the generalization of attribute values or performing calculations, such as the unit of measurement normalization, are most challenging for both GPT-3.5 and GPT-4. To improve the unit of measurement normalization, a compelling avenue for future research is giving LLMs access to functions or a code interpreter for calculating the unit of measurement conversions.