Combining Constraint Programming Reasoning with Large Language Model Predictions

Large Language Models (LLMs), such as the GPT series, generate text by predicting the next token (word or character) given the history of previously generated words. Decoding in LLMs refers to the strategy used to select the next words to be generated.

The simplest decoding strategy is greedy decoding. Here, the LLM selects the words with the highest probability at each time step. Although simple and efficient, this approach does not guarantee the best overall sequence, as it does not consider the effect of the current selection on future tokens.

Beam Search (BS) [30, 10, 11] is a refined version of greedy decoding. A beam is a candidate sequence of words. Instead of selecting the single best token at each time step, it usually keeps track of the $k$ most likely sequences (beams) at each step.

Although BS usually achieves better results than greedy decoding, it assumes that high-ranking token sequences consist of high-ranking tokens, which may only sometimes be the case. For a more stochastic and diverse output, top-$k$ sampling and top-$p$ sampling (also known as nucleus sampling) are used. In top-$k$ sampling, the model selects from the top $k$ highest probability predictions, while in top-$p$ sampling, it dynamically selects the number of top predictions to cover $p$ percent of the total probability mass.

Perplexity is an entropy metric derived from Shannon’s information theory [31]. Since an LLM computes the probability of text, then it can compute text perplexity. It can be expressed as the geometric mean of the inverse conditional likelihood of the sequence [32]. Let $S_{n}$ be the sequence of a succession of words of size $n$: $S_{n}=w_{1}w_{2}..w_{n}$. The perplexity (PPL) of $S_{n}$ is computed as follows:

where probability $P(\cdot)$ is given by the LLM. PPL can be interpreted as the “how likely a text is generated by a given model” [8]. Usually, it is used to evaluate the LLM itself by checking that good samples are recognized as such (i.e., low PPL values).

In NLP, the evaluation of text is still an open problem, and human evaluation remains the gold standard. Numerous metrics have been developed to address this issue. Among them, PPL remains an objective criterion associated with text produced by a given model. PPL is also much more convenient to use than pure probability. Its range is $[1;+\infty[$ . The lower, the better.

In traditional CP, for the task of text generation under constraints, a variable represents a word. Since the domains of variables have to be set beforehand, they will be of enormous size, containing every word/declination of words for a given language. Furthermore, constraints between succession of words may lead to a combinatorial explosion. Since traditional CP is not well suited, this work focuses on OTFS, a CP based method recently introduced by Régin and De Maria [29]. Instead of having the variables/domains/constraints set before the search, OTFS generates the variables/domains/constraints during the search for a solution, avoiding the problem stated above and being expendable to permit the integration of an LLM. The new version of OTFS is called GenCP.

The set of generative functions $G=\{genV,genD,genC\}$ is such that:

• •

  $genV$ generates a new variable with an empty domain and adds it to $X$.

• •

  $genD$ calls the LLM with the current sentence formed by the model and sets the domain of the previously generated variable to the output.

• •

  $genC$ generates the constraint(s) relevant to the current variables of the model to $C$. The constraints generated depend on the problem (e.g., generate a sentence that does not contain the letter “e”).

A variable is generated with an empty domain. To generate the domain of variables, $genD$ calls an LLM using $callLLM(sentence,parameters,k)$, where:

• •

  $sentence$ is the current sentence represented by the variables of the model.

• •

  $parameters$ represents sampling parameters ($top\_k,top\_p$…). For this paper, $top\_k$ is used exclusively for both GenCP and BS: the LLM answers $k$ words ranked by probability, highest to lowest.

• •

  $k$ is the number of words asked to the LLM.

Since the parameters and $k$ are not modified after the definition of the model,
$callLLM(sentence,parameters,k)$ will be simply referred to as $callLLM(sentence)$.

Helping functions represent implicit constraints, like avoiding infinite loops. In our current implementation, the set of functions $H$ contains the following functions:

• •

  $H_{o}$: it orders the domain of variables depending on the problem.

• •

  $H_{onlyWords}$: it ensures that any word predicted by the LLM is a complete word and not the suffix or prefix of a word and it filters out any symbol or special character.

The main steps of GenCP are depicted in Fig. 3 and Algorithm 1:

1. 1.

   GenCP begins with an initial state. If the initial state is empty, the generative functions are called (2.), otherwise the helping functions are called (3.).

2. 2.

   The generative functions $genV/genD/genC$ are called ($genD$ calls the LLM).

3. 3.

   The helping functions are called to manage implicit constraints, backtracking if necessary (e.g., if the LLM generated an empty domain).

4. 4.

   The current state of the model $\mathcal{M}$ is saved.

5. 5.

   The propagation is called, if it fails the model backtracks (8.), else it calls the boolean functions (6.).

6. 6.

   The Boolean functions are called to check if a solution has been found. If a solution is found, it is saved (7.) and the model backtracks (8.), otherwise the model calls the generative functions (2.).

7. 7.

   The current sentence formed by the variables is saved as a solution.

8. 8.

   GenCP backtracks to a previously saved state (4.) of the model and changes the choices made during propagation (5.). If no previous state was saved, then backtracking fails (9.).

   • •

     When backtracking to a previously saved state, the model deletes all variables, their respective domains, and the constraints associated with them, that are not present in the previously saved state.

9. 9.

   GenCP outputs the solution(s) that were saved or it indicates that no solution was found.

Variability between two sentences is the number of words that are not equal at each position, for example:

• •

  “The little boy is” and “The little cat is” have a variability of 1.

• •

  “My name is John” and “John is my name” have a variability of 4.

To force a greater variability between solutions (greater than $2$), a special backtrack called $backtrackTo$($n$) is used. Let the set of variables $X=\{x_{1},\dots,x_{n},x_{n+1}\dots,x_{m}\}$. The function $backtrackTo$($n$) deletes the variables $x_{n+1}$ to $x_{m}$ and causes a backtrack. For example, consider the sentence “I like to swim in the summer.”. With $backtrackTo$(2), “to swim in the summer.” is deleted and the value of variable $x2=``like"$ is changed. The next solution might be “I want to break free.”.

For some tasks, not following the ordering strategies of the LLM (like top-$k$ and top-$p$) can lead to better/faster solutions. Two other orderings are considered: PPL valuation and length of a word (depending on the average word length in the given language).

The experiments presented by Yao et al. are partially reproduced [7]. In particular, the constrained sentence generation tasks described in Tab. 1. Five LLMs were selected: GPT4, GPT4-O, Mistral Next, Claude 3.5, and Gemini. The four LLMs are prompted with the same example command given in [7]. For example, “Please create a sentence of exactly 82 characters.” for the Sent-1 task¹¹1https://chatgpt.com/share/b2834735-f7d8-468a-ba54-7da19dd6723c. Tab. 2 gives an overview of the performance of the five LLMs on the four tasks. The satisfaction rate is based on ten trials per task per model. In addition, Tab. 2 also shows that the LLMs perform well on the lexically constrained scenario task-4 with a 90+% satisfaction rate over ten trials. Also, as Yao et al. previously showed in their paper, LLMs struggle to produce constrained sentences involving counting (e.g., words and characters). They provide a nice picture of current LLM satisfaction capabilities by introducing new benchmarks. Unfortunately, the Yao et al. article only provides the benchmarks and some hints on reproducing them. However, it does not give any hints on how to solve the tasks associated with the benchmarks (see the original article for more details [7]).

The experiments were performed on a laptop with Windows 10 Professional, 32 GB RAM, and Intel 16 CPU cores. The approach and the BS are implemented in Java 17 for easier comparisons.

LLaMa [6] is responsible for the predictions of words as domains for the variables, mainly because an efficient implementation in C++ was recently released²²2https://github.com/ggerganov/llama.cpp. It allows running a model on a personal laptop and CPU (only) efficiently. Thanks to quantization [33] (model weight compression), the 7B parameters model (in Float16) of 13GB original size, in 4-bit quantization (Q4) drops to 3.9GB of RAM. However, the biggest model of LLama 65B (120GB), even in Q4, needs 38.5 GB of RAM. Thus, the LLaMa v1 model used in the experiments is LLaMa 7B Q4 with low temperature (i.e., $\leq 1$, temp = 0.8).

When asked for $k$ words, this version of LLaMa will take the same amount of time to ouput $1$ word and $1000$ words. To minimize the importance of $k$, $callLLM$ outputs more than $k$ words, a beam/variable only keeps $k$ “valid” words. A “valid” word is a word that does not violate a constraint on its own. For example, a word that does not violate the constraint “does not contain the letter e”.

In the current implementations two halting conditions are defined for BS:

• •

  First solution: when the current beam contains at least one solution, BS is stopped and output the solutions.

• •

  All solutions: when the current beam contains at least one solution but another beam can continue to generate words without violating a constraint (for instance, it does not contain enough characters to satisfy a length constraint), the beam solutions are saved and BS continues with the remaining beams.

BS and GenCP are compared on some recent benchmarks described in Sec. 4.1.1.

To guarantee GenCP and BS to be judged on the generation of sentences of the same quality, a solution is a sentence that satisfies all the constraints of the current task and, when given this sentence, the LLM predicts a period (“.”). Not to alter BS too much, words are ordered by probability (PPL is not used) and, since BS sentences have low variability, GenCP is used without $backtrackTo$($n$).

BS and GenCP are compared on the following criteria:

• •

  Time in seconds.

• •

  Number of solutions. GenCP was stopped when it found the same number of solutions as BS on a task. $0/1$ means that BS found no solution while GenCP found one solution.

• •

  The ratio solutions/outputs as a constraint satisfaction rate.

• •

  For BS only, the number of bad outputs (number of outputs that are not a solution).

• •

  For GenCP only, the number of backtracks.

BS and GenCP are compared in Tab. 3. In all tables, the number of backtracks is denoted by #bk. BS is slower than GenCP and has lower satisfaction rate (number of outputs that are solutions / total number of outputs), denoted by %sat. This is due to multiple facts:

1. 1.

   Beam Search can not guarantee to find every solution.

2. 2.

   Beam Search chooses the next word depending on the probability of the LLM.

3. 3.

   At each step, BS considers $k$ sentences, each sentence asks $k$ words to the LLM, so each step considers $k^{2}$ words. BS orders these words decreasingly by probability and only keeps the $k$ first.

Facts 2 and 3 explain why increasing $k$ does not guarantee to find the same/more solutions, it might even cause BS to find less solutions.

Let us suppose $k$ = 5, BS found one solution, and at depth 4, the candidate needed to find this solution was ranked 5 out of 25. Let us suppose now $k$ is increased to 6: at each step BS will consider 36 candidates and take the 6 best ones. BS considers 11 more candidates than with $k$ = 5; if at depth 4, the candidate needed to find the previous solution is now ranked 7 instead of 5, BS will not consider it and $k$ = 6 will not find the solution found with $k$ = 5.

Tab. 4 shows the capability of GenCP to generate more solutions than BS. GenCP is given the same time as BS for the same task and $k=50$, GenCP obtains more solutions than BS. Note that for sent-1, without $backtrackTo$ GenCP only obtains 2 solutions in 1123 seconds, while with $backtrackTo(6)$ GenCP obtains 11 solutions in 1123 seconds.

The LLM-enhanced GenCP avoids the drawbacks of BS and proposes an alternative approach to text generation under constraints for the following reasons:

• •

  GenCP can guarantee to find every solution (if any). Increasing $k$ guarantees to find at least the same solutions previously found and potentially finds new solutions. Furthermore, it can offer more solutions than BS.

• •

  All the outputs answered by GenCP are solutions (all the constraints are satisfied).

• •

  GenCP offers more options for improvement, for example to ensure better variability ($backtrackTo$ explained in 3.1.5 can be used) or other orderings than probability (3.1.6).

Tab. 5 demonstrates the importance of enforcing variability and perplexity. When GenCP generated solutions for Tab. 3 and 4, the maximum variability was 4. Tab. 5 shows that with $backtrackTo(2)$/$backtrackTo(3)$, sentences generated are almost completely different thanks to high variability (10+ for sent-3 for example).

Tab. 5 purposefully contains sentences with high perplexity to illustrate that this leads to a degradation in the sentence quality (i.e., low meaning).

All the sentences generated for sent-4 had the words “soft”, “beach” and “math” next to each other. To showcase the capability of GenCP to improve sentences, sent-4* was created: it is the same as sent-4 except that “soft”, “beach” and “math” must contain at least three words between them.