Using LLMs for the Extraction and Normalization of Product Attribute Values

Data collection. The Web Data Commons (WDC)³³3https://webdatacommons.org/ project extracts structured data from the Common Crawl⁴⁴4https://commoncrawl.org/ and provides the extracted data for public download. The WDC Product Data Corpus (WDC LSPM)⁵⁵5https://webdatacommons.org/largescaleproductcorpus/v2/ [13] is one of the extracted datasets. It consists of over 26 million product offers originating from 79 thousand different websites which employ the schema.org vocabulary to annotate structured product data within their HTML pages. The offers are classified into 26 product categories. In addition to schema.org annotations, WDC LSPM extracts attribute-value pairs from specification tables found in the web pages. The attributes in these pairs are product category-specific, such as the number of processor cores of a computer. Category-specific attributes are not part of the schema.org vocabulary and therefore are not explicitly annotated in the web pages. We clean the product offers and attribute-value pairs in the WDC LSPM corpus, omitting those with missing titles, descriptions, or specification tables, and those with descriptions exceeding 1,000 characters. In addition, HTML and language tags are stripped away, and only product offers in English are kept. We select the five categories ’Computers and Accessories’, ’Jewelry’, ’Grocery and Gourmet Food’, ’Office Products’, and ’Home and Garden’ for WDC-PAVE because they contain a large number of product offers and attribute-value pairs after pre-processing. Subsequently, a random sample of product offers is drawn for each category, with the objective to manually verify their attribute-value pairs. Based on the sampled product offers a fixed set of attributes per category is determined. As the attribute-value pairs in the specification tables are heterogeneously annotated on the different websites, a human annotator is required to verify that each attribute value is a sub-string of the title or the description, and that it semantically fits the attribute. Additionally, the human annotator adds attribute values that are not contained in the specification tables but are mentioned in the product offer to the gold standard. If an attribute is not referenced in the title or the description, the value "n/a" is assigned to this attribute.

Dataset Statistics. Table 1 shows profiling statistics describing the WDC-PAVE benchmark dataset. The dataset consists of 4,687 attribute-value pairs from 565 product which originate from 59 different websites. Overall, 45% of the attribute-value pairs hold the attribute value "n/a" meaning the attribute is neither mentioned in the title nor in the description. The dataset contains 2,011 unique attribute values so that each attribute has on average 54 unique values.

Data Normalization. Each of the 37 attribute in the dataset requires to be normalized before being usable for applications such as faceted product search. We have identified four normalization operations. Table 2 illustrates the normalization operations with examples for selected attributes. Name Expansion deals with the expansion of abbreviated attribute values such as "HP" into their non-abbreviated form, e.g. "Hewlett-Packard". Generalization assigns attribute values to broader categories, e.g. the color "Neon Lime Green" to the more general category "Green". Unit of Measurement Normalization converts an attribute value to an attribute-specific target unit of measurement and format, such as the weight value "20-lb." to "9.06", which represents the weight in kilograms (kg). String Wrangling normalizes attribute values to a specific format by for example replacing words with numbers or removing non-alphanumeric characters, e.g. the value "CTW-4M(208)" would be normalized to "CTW4M208". Each attribute is assigned to one of the normalization operations. After normalizing the 2,011 unique values in WDC-PAVE, the dataset contains 1,244 unique normalized attribute values.

Prompt Templates. Each prompt template consists of up to six building blocks visualized by different background colors. The building blocks are role description (blue), task description (blue), task input (green), and task output (orange), demonstration input (green) and demonstration output (yellow). The role description is a system message that defines the overall goal of the LLM as well as the target schema for the extraction including attribute descriptions and example values. The target schema is encoded using JSON-schema⁷⁷7https://json-schema.org/ as this representation proved to be most effective in related work [3]. The task description is a user message that provides instructions for the attribute-value extraction. The task input is a user message containing the product title and description. The task output contains the LLM’s response with the extracted attribute-value pairs in JSON format. The input and output of the demonstrations are user and assistant messages containing product offers that are semantically similar to the target product offer in the task input. To calculate the semantic similarity, the target product offer and the training demonstrations are embedded using OpenAI’s embedding model text-embedding-ada-002⁸⁸8https://platform.openai.com/docs/guides/embeddings/. The training demonstrations with the highest cosine similarity to the product offer are added into the prompt. Figure 2 contains instructions for extraction and normalization. The red text is added to the prompts for the extraction with normalization task and the normalization task, which are discussed in Section 5 and Section 6.

Discussion of Results. We assess the impact of zero-shot and few-shot prompt template configurations on GPT-3.5 and GPT-4 performance. Table 3 shows the F1-scores, the average number of tokens per prompt, and the cost in $ per 1,000 extracted attribute values of the prompt template configurations with different amounts of example values (Val.) and the combination of 10 example values and different amounts of demonstrations (Dem.). The highest F1-score with and without demonstrations is marked in bold. For the cost calculation, we use the OpenAI prices as of April 2024⁹⁹9https://openai.com/pricing. The results show that GPT-3.5 and GPT-4 benefit from ten example values, achieving F1-scores of around 80%. Demonstrations further improve the performance of GPT-3.5 and GPT-4. Ten demonstrations allow GPT-4 to achieve the highest F1-score of 91%. Both, example values and demonstrations implicitly guide the LLMs to extract exactly the surface form of the attribute values that is used in the offer and is expected by the ground truth. The usage of example values and demonstrations significantly increases the length and the cost of the prompts. Adding three demonstrations increases the cost of extracting 1,000 attribute values with GPT-4 by 1$. In order to enhance GPT-4’s performance by an additional 1.6%, an additional 2$ per 1,000 extracted attribute values must be spent.

Comparison of LLMs and PLMs. We now compare the performance of GPT-3.5 and GPT-4 with the prompt template that uses ten example values and ten demonstrations to the PLM baselines SU-OpenTag [23], AVEQA [21] and MAVEQA [26]. The baselines are fine-tuned on the same training set used to select example values and task demonstrations for the LLM prompt templates. The results in Table 4 show that GPT-4 outperforms the best PLM baseline AVEQA by 10% F1. It is important to mention that the training set with 1,750 attribute-value pairs is small. The training sets of the existing benchmark datasets MAVE [26] and AE-110k [23] contain 3.8 million and 84.7 thousand attribute-value pairs. As shown by Brinkmann et al. [3], it can be expected that with additional training data the performance of the PLM baselines increases whereas the performance of the LLMs only marginally improves.

Discussion of Results. As in Section 4, we measure how adding zero, three, five and ten example values without demonstrations and ten example values with three, five and ten demonstrations to the prompt affects the performance of GPT-3.5 and GPT-4. Table 5 shows the results of the experiments. The best F1-scores per model are marked in bold. Without demonstrations, GPT-4 benefits from normalized example values and reaches an F1-score of 86%, which is 12% better than the zero-shot configuration. In contrast to the extraction task, example values only marginally improve the extraction and normalization performance of GPT-3.5. However, adding five demonstrations improves the performance of both LLMs, with GPT-4 achieving an F1-score of 91% and outperforming GPT-3.5 by 5%. The prompts for the extraction with normalization task are longer than the prompts for the extraction task because of the additional normalization instructions. At the same time, the best performance is achieved with 5 instead of 10 demonstrations, resulting in a cost reduction of 1$ per 1,000 extracted attribute values for GPT-4 (see rightmost column in Table 5).

Analysis of Normalization Operations. We now analyze the performance of the LLMs on the normalization operations in detail. Table 8 shows the F1-scores per normalization operation for the prompt templates zero-shot , with ten example values and with ten example values and five demonstrations. Previous research has shown that GPT-3.5 performance is weaker on tasks requiring reasoning or calculation than on tasks involving manipulation of free text or names [8]. Our results support these observations. GPT-3.5 and GPT-4 are particularly strong at name expansion and string wrangling. GPT-4 achieves F1-scores of 98% and 97%. Unit of measurement conversion requires calculations and is the most challenging operation for GPT-3.5 and GPT-4. Example values and demonstrations improve GPT-4’s zero-shot performance by 22%, leading to an F1-score of 83.5%. Compared to the extraction task, we observe that generalizing attribute values simplifies the task for GPT-4, possibly because it can use its background knowledge to generalize the values. For attributes like ’Product Type’, we observe that GPT-4 benefits from the generalization by an average of 7% across all product categories if example values and demonstrations from a training set are provided.

Discussion of Results. Like in Section 4 and Section 5, we assess how zero, three, five and ten example values without demonstrations and ten example values with three, five and ten demonstrations selected from the training set impact the performance of GPT-3.5 and GPT-4. Table 7 shows the results of the experiments. The highest F1-scores are marked in bold. Similar as in the previous tasks, we observe that both example values and demonstrations improve the F1-scores of GPT-3.5 and GPT-4. The effect of adding demonstrations for GPT-3.5 is marginal while GPT-4 gains 4% F1-score.

Analysis of Normalization Operations. We now analyze the normalization operations in detail. Table 8 shows the F1-scores per normalization operation for the prompt templates zero-shot, with ten example values and with ten example values and five demonstrations. The results show that string wrangling can be well handled by LLMs if the attribute values have already been extracted. The other normalization operations require example values and demonstrations to reach high F1-scores. Compared to the extraction with normalization task, the unit of measurement conversion results for GPT-3.5 and GPT-4 improve by 17% and 14% if the values have already been extracted. This shows how challenging the combination of extraction and normalization for the LLMs is given that unit of measurement conversions need to be performed. In contrast, the generalization operation remains as challenging as in the extraction with normalization task.

Comparison across Tasks. We now analyze how the F1-scores change between (i) direct extraction, (ii) extraction with normalization and (iii) normalization. Table 9 shows the F1-scores of GPT-3.5 and GPT-4 for the zero-shot scenario and the few-shot scenario with ten example values and five demonstrations. In addition, the deltas between the extraction task and the extraction with normalization task ($\Delta_{1}$) and the delta between the extraction task and the normalization task ($\Delta_{2}$) are shown. The results indicate that extraction is more challenging for LLMs than normalization. Zero-shot both GPT-3.5 and GPT-4 achieve 12% higher F1-scores if no extraction is required. With example values and demonstrations, GPT-3.5 and GPT-4 reach F1-scores that are 4% and 8% higher if no extraction is required.

Product Attribute Value Extraction. Early research on PAVE used domain-specific rules to extract attribute-value pairs [20, 27, 11] from product descriptions. The initial learning-based methods required extensive feature engineering and did not generalize to unknown attributes and values [6, 14]. Recent works have adopted BiLSTM-CRF architectures [10, 29] to tag attribute values in product titles. SU-OpenTag [23] builds upon OpenTag [29] by encoding both a target attribute and the product title using the PLM BERT [4]. [17, 18, 21, 26] approach PAVE as a question-answering task, using different PLMs to encode target attribute, category, and title. The PLM-based methods SU-OpenTag [23], AVEQA [21], and MAVEQA [26] serve as baselines for the direct extraction task.

LLMs for Attribute Value Extraction. LLMs have successfully been used for information extraction in various domains [1, 7, 12]. In the context of PAVE recent works experiment with different prompt designs for PAVE using LLMs [3, 5]. We use the prompt templates from [3] as a role model for our prompt templates. In contrast to related work  [2, 16, 25], we do not fine-tune the LLMs, but instead rely on in-context learning via demonstrations and example values. Early work on attribute value extraction and normalization applied domain-specific normalization rules [15, 19]. Jaimovitch-López et al. [8] experiment with using GPT-3.5 for attribute value normalization but present the values to be normalized without any context to the model. In contrast, we include the original titles and descriptions into the prompts in order to provide context for the normalization.

Benchmarks for Attribute Value Extraction and Normalization. The benchmarks MAVE [26] and AE-110k [23] are widely used to evaluate methods for PAVE. MAVE relies on an ensemble of five fine-tuned PLMs for determining ground truth annotates. AE-110k [23] uses values from product specification tables as ground truth. In contrast to these benchmarks, all attribute-value annotations in WDC-PAVE are manually verified. To our knowledge, OA-Mine [28] is the only other publicly available benchmark offering human-verified attribute-value annotations. MAVE, AE-110k, and OA-Mine address value extraction and do not consider value normalization. WDC-PAVE covers both tasks. [8] propose an attribute value normalization benchmark including operations such as transforming dates, units of measurement, or names. Unlike WDC-PAVE, their benchmark presents the values to be normalized without any context that can be exploited by the methods.