Planning with Logical Graph-based Language Model
for Instruction Generation

To construct logical graph, we design a preprocessing module, which refines sophisticated logical relations in natural language instructions (upper left part of Figure 2). In detail, the logical relation between action A and action B means that action B can only be executed when action A has been committed.

If we directly adopt natural language instructions to mine rules, redundant and incorrect information may affect relation extraction. Therefore, for extracting refined logical relations, we first use semantic parser spaCy³³3https://spacy.io/ to extract the core verbs and objects in the domain. Then, we leverage pre-trained RoBERTa model to calculate the semantic vectors of extracted objects and tasks [13]. After that, we use K-means to cluster extracted objects and tasks according to the the semantic vectors. An object detector then substitutes objects with clustered object class. Since policy selection considers multiple previously executed actions in reinforcement learning, we then analyze relations between previous $k(k\in Z,k\geq 1)$ actions with current action for extracting logical relations. The frequency of action pair within k time steps is calculated and compared with the threshold $\epsilon(\epsilon\in R,0\leq\epsilon\leq 1)$. The threshold $\epsilon$ depends on the specific action (Equation 1). For each action, we calculate the frequency mean of previous actions plus the unbiased estimation of frequency variance within k time steps. The action pairs with higher frequency than $\epsilon$ are retained. In this way, we finally acquire refined logical rules for constructing logical graphs.

where $t$ is the number of previous adjacent actions of the current action. $p_{i}$ denotes the frequency of each previous action and $\bar{p}$ is the average frequency of all previous actions.

After acquiring refined logical rules, we then construct a logical probability graph to indicate the logical relations implied in the text. However, the natural language logical rules may contain much redundant information, so we first structuralize refined logical rules using syntax from AI planning [14, 15, 16]. Consistent with Planning Domain Definition Language (PDDL), we define the lifted action node as $action(?x-type)$ to reserve core verbs and objects, where $action$ denotes core verb and $?x-type$ denotes the object type without instantiation [12]. The lifted action node can be visited and then grounded into $action(instance)$. For example, $grab(?x-fruit)$ can be grounded as specific $grab(apple)$. Following the same source node, we add lifted action nodes and task type node of each action sequence in order. If the path from source node to the current node exists, we just add next node following the current node. In this way, we internalize the domain logic into the graph skeleton. Motivated by the wide use of bayes networks, we endow each edge with transition probability to learn causal relationships [17]. In detail, we represent the transition probability using the corresponding frequency of action pair in training data. In this sense, we construct a logical probability graph which contains refined domain logic.

The language model has better generalization abilities than traditional planning [18]. However, even large language models only work on simple reasoning tasks and fail to solve sophisticated reasoning problems [5].

Therefore, we design a logical probability graph search module, which generates instructional skeletons under the guidance of both logical probability graphs and language models. To guide the search across the constructed logical graph, we design a heuristic value calculation module. The total heuristic value is calculated based on four factors:

where $\mathcal{H}$ stands for the total heuristic value. $h_{dis}$, $h_{len}$, $h_{bayes}$ and $h_{lm}$ stand for distance, expected program length, bayes, and language model heuristic values. We define the program length as the number of sentences in instructional texts. $\omega_{1}$, $\omega_{2}$ and $\omega_{3}$ are positive hyper-parameters. The detailed heuristic value calculation is formulated as follows and the candidate node with the highest heuristics can be selected :

where $h_{dis}$ implies whether the current node can reach the target task type node. $y_{n}$ denotes action type node at current step and $x$ the target task type node, respectively. $\text{path\_exist}(y_{n},x)$ denotes whether exists one path from the current node to the target task type node. If there exists no path between $y_{n}$ and $x$ in the logical graph, this means the current action node is probably irrelevant to the task. When searching the graph, we give priority to nodes which can reach the target task node.

where $y_{[1:n]}$ denotes the previously $n-1$ selected action type nodes concatenated a current candidate action type node. $y_{n}$ denotes the current candidate node. $\hat{l}$ stands for the expected number of instructional sentences. $avg\_len(.)$ calculates the average length of all paths from the current candidate node to the target node. We use current steps plus $avg\_len(.)$ as the estimated length of our generated action sequence. Then, we calculate the difference between our estimated length and the expected length. The shorter the difference between our estimated instructional length and the ideal length, the larger the value of $-\omega_{1}h_{len}(.)$. In this way, the length of our instructions can be adjusted according to the expected length.

where $\rightarrow$ stands for the probability sum of all possible paths from the visited path to the target node. $sons$ denote the action nodes which can be visited from the last action node. Based on the already visited action nodes and target type nodes, we calculate the score of one action node among all candidates.

where $\overline{y}_{n}$ stands for all possible grounded actions at the current step. For instance, $grab(?x-fruit)$ can be grounded into $grab(mango)$, $grab(bananas)$ and other possible specific actions. $\overline{y}_{[1:n-1]}$ stands for grounded action node sequences at previous $n-1$ time steps. $h_{lm}$ calculates the probability of generating one possible grounded action given the texts of task and all previously grounded actions. The detailed probability calculation for each grounded action by the LM can be seen in Equation (7).

where $\gamma$ denotes the language model for generating the next action. In detail, the texts of the task $T_{x}$ and previous actions $T_{\bar{y}_{[1:n-1]}}$ serve as the input, and we acquire the probability of LM generating a grounded action $T_{y_{n}}$.

We then use heuristic values to guide the logical graph search. As illustrated in Figure 3, the graph search algorithm moves towards the task goal and the heuristic value guides the node selection. In detail, we first select the target task type node (yellow nodes) which has the highest semantic similarity with the current task text. Then, we start from the source node (blue node) and set it as the current node. At each step, we calculate the heuristic values of all successor nodes from the current node. For instance, the heuristic value of Action 1 is $v_{0}+p_{0}+q_{0}+l_{0}$, which corresponds to four heuristics defined in Equation (2). The nodes with the highest heuristic values (green nodes) are selected as the current nodes and grounded into $action(instance)$. The node selection continues until reach the target task type node. Finally, we concatenate all grounded actions and output the generated instructional texts.

The generated instructional texts from logical graphs internalize domain logic. Together with label instructions, we thus use generated instructional texts to guide language model training. The trained language model also guides the logical graph search. In this sense, logical graphs and language models guide each other to learn domain logic and make policies.
Specifically, each input to the language model is comprised of task description $x$ and label instructional text $\hat{y}$. Since both label instructions and generated instructional texts contain domain logic, we define the cross-entropy loss function $\mathcal{L}(\theta)$ as follows:

where $\mathcal{L}(.)$ is the loss function. $\mathcal{H}(.)$ stands for the cross entropy between two vectors. $f(.)$ is the embedding function. $\hat{y}_{[1:t]}$ is the label instructions and $y_{[1:t]}$ is the generated texts from language model. We use the above loss function to update parameters by stochastic gradient descent (SGD) [19]. In this sense, the parameters of language model are updated to make the language model internalize logic contained in both label instructions and generated instructional texts from the logical graph. The trained language model then guides the logical graph search. In this way, language model and logical graph guide each other and better internalize domain logic.

The overall algorithm is presented in Algorithm 1. We first initialize parameters $\theta$ of the language model randomly (Line 1). After that, we use the processed annotated task tuples $\Phi=<x,\hat{y}>$ to construct a logical graph (Line 2). We then search the logical graph to generate instructional texts under the guidance of the language models (Line 5). After that, we update parameters $\theta$ of the language model by optimizing $\mathcal{L}(\theta)$ (Line 6). We repeat the above procedure until the stop requirement is satisfied and output the language model parameters $\theta$.

To evaluate the effectiveness of the proposed model, we investigate the textual reasonability, executability and correctness of generated texts. To evaluate the executability, we need an environment where textual interactions are supported. Most existing embodied environments except VirtualHome are unsuitable. We thus choose VirtualHome as our environment [20].

In total, there are 5048 pieces of available instructional texts for 202 different tasks. A vivid example can be seen in Figure 4.We divide the tasks into training tasks and testing tasks with a ratio of 3:1. In detail, all instructional texts of 152 tasks are taken as training data, and texts of the remaining 50 tasks as testing data.

Our experiments evaluated the text quality and practicability of generated instructional texts. We will examine our Logical-GLM in the following aspects:

• 1

  We first evaluate Logical-GLM on classic NLP metrics like BLEU and ROUGE, which measures the precision and recall of generated instructional texts with label texts and reflects the text quality and reasonability to some degree.

• 2

  We then evaluate Logical-GLM on executability and correctness of instructional texts. We judge the executability and correctness by the completion of the task and human, respectively.

• 3

  Additionally, we evaluate the effects of language model guidance on Logical-GLM by modifying training epochs and evaluating the generated instructions.

• 4

  We adjust the weights of different heuristic values in graph search. We aim to see the impacts of heuristic values and find the optimal hyperparameter pair.

• 5

  Shorter instructional texts often have higher executability, which may disturb the evaluation of text quality. To validate the robustness of our model, we finally evaluate the executability of generated instructional texts with varying program lengths.

• 6

  Finally, we commit a case study by analyzing instructional texts of one specific task generated from Logical-GLM and all baselines.

Text quality of generated instructional texts

The BLEU and ROUGE score of generated instructional texts with label texts can reflect the text quality to some degree. In detail, we leverage ground truth instructional texts provided by [22]. We then compare BELU and ROUGE score among Logical-GLM and all fine-tuned baselines. Table (1) shows BLEU and ROUGE results from different language models. From Table (1), we can observe that our Logical-GLM outperforms all baselines and achieves state-of-the-art performance on both BLEU and ROUGE score with the least parameter amount, which demonstrates the reasonability and submissiveness to label texts of our generated instructional texts.

Executability and correctness of generated instructional texts

The final goal of our Logical-GLM is to complete tasks in sophisticated open environments. We thus compare the executability and correctness of generated instructional texts. In this experiment, executability indicates each action from instructional texts is effective and feasible. We judge correctness by humans and it implies instructional texts can ensure the completion of tasks. Specifically, we use the same questionnaires as Huang et al. [22], but we take the average correctness of evaluated tasks over all human evaluators for each model. The results together with the average program lengths are shown in Figure 5. Considering the great expenses of human evaluation, we randomly pick 20 tasks from evaluated tasks for human evaluation. In Figure 5, we see that our model performs better than all comparison models including ChatGPT. It is not surprising because our Logical-GLM generates instructional texts based on domain-specific logical graph. Also, we note that GPT2-small has the highest executability among all models except our Logical-GLM. The reason we believe is that the program length of instructional texts from GPT2-small is quite small than all other models and shorter instructional texts tend to have higher executability. In comparison, the program length of instructional texts from our Logical-GLM is between those of GPT2-medium and GPT2-large but still performs better than almost all comparison models. Moreover, it can be observed that the fine-tuned GPT2 generates unexecutable instructional texts even if the original GPT2 small produced some executable output. We attribute this to that the scarce training data and finetuning LLMs may damage the safety strategies of LLMs. Note that GPT4 gets higher correctness but lower executability compared to our Logical-GLM. The higher correctness is not surprising because GPT4 has 1 trillion parameters and the enormous training corpus accidentally contains the required world knowledge known by humans. However, GPT4 may lack the required strict domain logic, resulting in lower executability. Moreover, our Logical-GLM surpasses both the beam search and greedy search of SayCanPay. The reason we believe is that SayCanPay essentially internalizes domain logic using language models, which calls for sufficient expert data. Intuitively, our Logical-GLM works better for generating instructional texts when domain-specific training data is scarce.

To evaluate the effects of language model guidance on Logical-GLM, we vary the training epochs of language models for different quality of language model guidance. We compare generated texts from Logical-GLM on evaluated tasks with varying quality of language model guidance. As shown in Figure 6(a), both the BLEU score and ROUGE score rise first and then decline with the improvement of training epochs. The same trend of executability with varying quality of language models is also depicted in Figure 6(b). The early rising trend shows that our language model guidance influences Logical-GLM and higher training epochs indicate better guidance. Although higher epochs indicate better language model guidance, we should note that overlarge training epochs result in over-fitting, which illustrates the latter declining trend. Note that the over-fitting is unavoidable since our lightweight Logical-GLM applies to domains with few expert samples. Thus, we divide the expert data into training data and validation by the ratio 8:2 and we leverage the effects of validation samples to break off training in advance. In Figure 6(b), we also see that the executability is still high at epoch 0. This is not surprising because our Logical-GLM is also guided by bayes heuristic value, which contains domain logic.

To see the impact of different hyper-parameters, we evaluate the executability of generated instructional texts on evaluated tasks with respect to different weights of heuristic values(i.e., $w_{1}$, $w_{2}$, and $w_{3}$ in Equation (2)). The detailed heatmap about how hyper-parameters influence executability is depicted in Figure 7. We can fix the value of one axis and study how executability changes with another axis value. In most cases, we can see that executability of generated instructional texts drops with $w_{1}$ increasing. A big value of $w_{1}$ means to focus on generated instructional text length and thus ignore text logic. When $w_{3}:w_{2}=100:1$, the executability of generated texts is the highest for the same $w_{1}:w_{2}$. The reason we believe is that the heuristic value $h_{lm}(.)$ is much smaller than $h_{bayes}(.)$. The $h_{lm}(.)$ can have considerable effects as $h_{bayes}(.)$ only when the weight of $h_{lm}(.)$ is much higher than that of $h_{bayes}(.)$.

We next evaluate controllable program length ability of our model and the performance with regard to the answer length. Since the graph search uses the expected program length as heuristic guidance, the program length of generated instructions is human controllable. In Figure 8, we see that the program length (blue lines) truly changes with different program length. However, the generated program length change is not obvious. The small change range is because the training instances for constructing the logical graph have short program lengths and there exist no long paths in the logical graph. To validate the robustness of our Logical-GLM, we also compare executability of generated program length as the expected program length changes. From purple lines in Figure 8, we can observe that the executability soars as the program length increases. In the future, we hope to construct logical graph with longer paths and further explore the program length robustness of our Logical-GLM.

To vividly demonstrate the difference between generated instructional texts from different language models, we design an experiment to analyze the outputs of Logical-GLM and other language models. In detail, we choose ChatGPT and both Vanilla and fine-tuned GPT2-xl which has the higher parameter amount among GPT2 family. We sample the task "Watch movie" and demonstrate instructional texts from different models in Figure 9. It is intriguing to note that our model can handle the task well.

From Figure 9, we can observe that both our Logical-GLM and ChatGPT can generate reasonable instructional texts relevant to the given task, which indicates that our Logical-GLM can achieve comparative performance as ChatGPT with the guidance of domain-specific logical graph. It is obvious that the instructional texts generated from our Logical-GLM are also simple and have higher consistency between sentences. Moreover, We can observe that both Vanilla GPT2-xl and Finetune GPT2-xl generate much more sophisticated texts which lack consistency between sentences and include many non-critical objects in the specific task.