NutriBench: A Dataset for Evaluating Large Language Models in Carbohydrate Estimation from Meal Descriptions

The FoodData Central (FDC) [2] is the food composition information center of the US Department of Agriculture (USDA) [14]. It is a semi-annually updated data repository and we obtained the most recent version of the data at the time of writing (April 2024) to compile food items and their nutrient labels in our dataset. We retained entries that provided complete macro-nutrient information, including carbohydrates, proteins, fats, and calories. Overall, we retained a total of 1,788,981 entries from the database containing all macro-nutrients. Among these, 465,059 items had unique food names, and 1,286,036 items were associated with a specific brand.

We conducted two rounds of outlier-based filtering to clean the data, with a primary focus on carbohydrates. In the first round, we eliminated variability among entries with the same food name and brand. Since we expect these entries to have near-similar nutrition estimates, we applied a strict z-score-based filtering approach, removing estimates with a score greater than 1, and selected the median of the remaining estimates as the final carbohydrate label. For same-brand items with only two entries, we kept their mean as the final label if their relative difference was less than 0.1. The second round filtered out extreme outliers for the same food name across different brands. In this step, we allowed for more variability and increased the z-score and relative difference thresholds to 2 and 0.3 respectively. Finally, we were left with 756,594 entries including 451,555 unique food names.

The standard nutrition measurement in the FDC database is based on a normalized quantity of 100g, denoted as “Metric Serving”. However, typical meal descriptions in natural language rarely include such precise measurements. For instance, it is more common to refer to “1 cup of rice" rather than “100g of rice" in day-to-day life. We extracted such natural servings from the FDC database, denoted as “Natural Serving” in this paper. In summary, our cleaned FDC database has 756,594 food items with varying brands and sources. Among them, 13,048 food items have additional natural serving descriptions with their corresponding weight in grams.

Next, we use the cleaned and filtered FDC database to create our retrieval database Retri-DB  and our evaluation benchmark NutriBench . The data sizes are in Table 1.

For our experiments with Retrieval-Augmented Generation, we construct a retrieval database, Retri-DB, to provide relevant external nutrition knowledge. For each food item, we compile the nutrition information available across different brands and for different servings into a comprehensive list of facts. If multiple brands provide estimates for a specific serving of a food item, we use the median as the reference value. Since models process unstructured text information more efficiently than tabular data [11], we use a rule-based transformation to convert the context to natural language. Specifically, for an entry containing a $serving\ amount$ and $nutrition\ amount$ for a particular $nutrient$, we convert it to the string "$serving\ amount$ has $nutrition\ amount$ $nutrient$". Figure 1 shows an example of retrieved context for the two food items in the input query.

To construct NutriBench , we generate 5,000 natural language meal descriptions from the cleaned FDC nutrition dataset, divided into 15 subsets. These subsets vary in the number of food items, serving sizes, and natural vs. metric serving descriptions and include both directly and indirectly retrievable food items to enable a comprehensive assessment of LLMs’ capabilities on this task. We also include both common and uncommon food items in NutriBench. The overall construction pipeline for NutriBench is depicted in Figure 3, with a summary of each subset sizes presented in Table 2. More detailed examples are available in the Appendix.

In this work, we conduct a comprehensive evaluation of seven state-of-the-art large language models (LLMs) using our proposed NutriBench. The evaluation spans models of varying sizes, ranging from small-scale models with 7 billion parameters to large-scale models with 175 billion parameters. Additionally, we compare models with integrated general medical knowledge to those without such specialized information. The evaluated LLMs are introduced as follows:

• •

  GPT-3.5 [7]: GPT-3.5-Turbo from OpenAI is a closed-source model, accessible via an API.

• •

  Llama2-7B and Llama2-70B [38]: Llama-2-7B-chat and Llama-2-70B are open-source instruction-tuned models from Meta.

• •

  Llama3-8B and Llama3-70B [3]: We also evaluate the advanced Llama models, Llama-3-8B-Instruct and Llama-3-70B. Notably, Llama-3-70B competes with GPT-3.5 in human evaluations.

• •

  Alpaca-7B [33]: Alpaca-7B, developed by Stanford, fine-tunes Llama-7B with 52K instruction-following examples

• •

  MedAlpaca-7B [17]: In comparison to Alpaca-7B, MedAlpaca-7B fine-tunes Llama with medical data including established medical NLP tasks as well as various internet resources.

In this section, we introduce how we adapt four existing prompting methods with carefully designed prompts tailored for carbohydrate estimation. They are:

• •

  Base: The first baseline involves instructing LLMs to estimate the carbohydrate content based on the meal description provided in the query with basic instructions.

• •

  Chain-of-Thought (CoT) [39]: Since our data includes complex queries with multiple items in varying quantities for a meal description, we hypothesize that the step-by-step reasoning induced by chain-of-thought prompting would reduce model errors by enabling the model to identify and reason about individual query components required to make the overall decision.

• •

  Retrieval-Augmented Generation (RAG) [21]: To further enhance the reliability of LLM, we use RAG to ground their predictions with nutrition knowledge retrieved from Retri-DB. First, for a given meal query, we prompt the model to parse it into individual food components. Next, we retrieve nutrition information about each food item in the query through a nearest neighbor semantic similarity search and concatenate the results to form a comprehensive set of facts about the food components in the query. Finally, we provide this retrieved context along with the original prompts for LLMs.

• •

  RAG+CoT: We combine the nutrition retrieval capability of RAG with step-by-step reasoning in CoT by concatenating the retrieved nutrition context with the CoT prompting for LLMs.

In all the cases mentioned above, we instruct the models to respond with ‘-1’ if they don’t know the answer to reduce the risk of potentially harmful predictions. Figure 1 shows different outputs of GPT-3.5 using the four different paradigms. We apply RAG and RAG+CoT on GPT-3.5, Llama3-8B, and Llama3-70B only due to computation constrain. The prompts for each method are in the Appendix.

We calculate the mean absolute error (MAE) to measure the deviation of the model responses from the true carbohydrates. In addition, we report accuracy ([email protected]) by considering the model output as ‘correct’ if the predicted value is within ±7.5g of the ground truth. This is based on the insulin-to-carbohydrate ratio, which indicates the grams of carbohydrates one unit of insulin can cover. While this ratio varies among individuals, it is generally considered 1:15 as a rule of thumb ²²2https://www.tidepool.org/blog/optimizing-insulin-to-carb-ratios. Since we aim to improve insulin management and avoid even half-unit insulin dosage errors, we maintain a conservative threshold of 7.5g on the absolute error to measure accuracy.

Finally, since we allow the models not to provide an estimate if uncertain, we also report the answer rate (AR), indicating the percentage of answered questions. Overall, the models should have a high AR and [email protected], and a low AE.

In this section, we investigate how external nutrition knowledge retrieved by RAG aids LLMs in making predictions. We compare results for natural and metric servings in Figure 4. Surprisingly, despite direct retrieval providing nutrition information with exact food names for different serving descriptions, it still degrades performance. This suggests that LLMs struggle to convert one serving description to another, particularly from metric to natural serving, even when our designed prompt explicitly instructs the model to convert the servings. In contrast, RAG consistently improves both [email protected] and AR for metric serving queries, even without directly retrieving the exact food names from Retri-DB. Considering most food items in Retri-DB only have metric serving descriptions, we conclude that RAG can provide LLMs with useful nutrition knowledge by retrieving similar, but not necessarily identical, food items, as long as the serving descriptions are aligned. Based on these findings, a promising future direction is to augment Retri-DB with more natural servings for RAG to support knowledge-grounded predictions with LLMs.

We examined the relationship between the prediction error and the true carbohydrate content. Figure 5 shows the histogram of the true carbohydrate content for single-item, single-serving queries. For each bin, we measure and plot the average MAE (in red). We observe a positive correlation: the MAE increases as the ground truth carbohydrate content rises. This indicates that LLM predictions are likely to be more accurate for individuals with a generally low-carbohydrate diet compared to those with a high-carbohydrate diet.

We conduct a voluntary human study on carbohydrate estimation involving 10 non-expert laypersons and 1 nutritionist. Among the ten laypersons, we include 1 patient with Type-1 diabetes for over 10 years, individuals with a general understanding of nutrition (including calorie awareness but not carbohydrates), and others without any particular focus on nutrition knowledge. We randomly sample 6 meal descriptions from each subset to create a final test set of 90 queries. To make a fair comparison with LLMs, we explicitly instruct the participants: Do not search online or use nutrition apps for carbohydrates. Additionally, we provide all participants with three meal descriptions with corresponding carbohydrates, identical to the few-shot examples given in the LLM prompts.

The results show an interesting finding: the professional nutritionist could not outperform advanced LLMs in carbohydrate estimation,⁴⁴4LLMs outperforms humans when evaluating the exactly same 90 queries. as illustrated in Figure 2. In addition, it takes the nutritionist in total of 50 minutes to complete all 90 queries ⁵⁵5This potentially includes time spent searching for information on uncommon foods. However, LLMs can answer all 90 queries within minutes , e.g., GPT-3.5 completed them in 2 minutes. Lastly, when the meal description becomes more complicated, participants experience significantly heightened stress in processing the information. In contrast, there is no difference when LLMs process longer meal descriptions. Taken together, we conclude that LLMs exhibit significant potential in addressing this challenging yet critical task.