AutoGRAMS: Autonomous Graphical Agent Modeling Software
Abstract
We introduce the AutoGRAMS framework for programming multi-step interactions with language models. AutoGRAMS represents AI agents as a graph, where each node can execute either a language modeling instruction or traditional code. Likewise, transitions in the graph can be governed by either language modeling decisions or traditional branch logic. AutoGRAMS supports using variables as memory and allows nodes to call other AutoGRAMS graphs as functions. We show how AutoGRAMS can be used to design highly sophisticated agents, including self-referential agents that can modify their own graph. AutoGRAMS’s graph-centric approach aids interpretability, controllability, and safety during the design, development, and deployment of AI agents. We provide our framework as open source at https://github.com/autograms/autograms.
Inhalt
- 1 Introduction
- 2 AutoGRAMS Framework
- 3 AutoGRAMS graph compiler
- 4 AutoGRAMS interpreter
- 5 Self-modifying and meta autograms
- 6 Related Work
- 7 Discussion
- A Prompt formation
- B Language models in AutoGRAMS
- C Interjection Nodes
- D Simulating a user in conversational autograms
- E Wild card transition visualization
- F Additional details of the AutoGRAMS graph compiler
- G The reply algorithm
- H A simple self-referential autogram
1 Introduction
Artificial intelligence (AI) agents driven by large language models (LLMs) can tackle complex, multi-faceted tasks. LLMs pretrained on large datasets generate human-like text and display knowledge of various domains Radford et al. (2019); Brown et al. (2020). This enables them to perform a range of tasks and learn from training examples given in their prompt.
Training these language models to follow instructions enhances their utility, allowing them to generate outputs that align with given instructions Ouyang et al. (2022). This instruction-following behavior is crucial for the development of AI agents that provide useful and contextually relevant responses across a variety of applications (Yao et al., 2022; Paranjape et al., 2023; Khattab et al., 2022). It also enables these agents to drive autonomous operations, such as making decisions, accessing external information, and engaging in conversations. Despite their potential, current approaches often struggle with maintaining coherent behavior over extended interactions, adapting to unexpected user inputs, and providing designers with intuitive and flexible control over the agent’s decision-making process.
To overcome these limitations, we introduce the AutoGRAMS (Autonomous Graphical Agent Modeling Software) framework to empower designers to orchestrate intricate interactions with LLMs. AutoGRAMS enables the creation of sophisticated AI agents and chatbots by representing their behavior as a series of interconnected nodes, each with distinct actions and transition rules. This graphical representation provides a programming language for agent development, where the execution path dynamically adapts based on language model predictions or predefined conditions.
With AutoGRAMS, designers gain the ability to:
-
•
Design branched multi-step interactions with a language model: Define a series of interconnected nodes that represent the steps involved in language model generation. This graph outlines a pre-defined procedure, and its nodes may include multiple branches and decision points. Interaction is defined by graph traversal, by iteratively stepping through the nodes of the graph.
-
•
Design complex conversational flows: Define a graph of conversational reply steps with prompts that govern how the agent gives conversational replies and makes decisions about which conversational branches to take.
-
•
Control interaction steps with with node-specific prompts: Craft prompts tailored to each node, guiding the language model’s responses.
-
•
Leverage language model predictions in transitions: Define how language model predictions are used to navigate branch points and guide the agent’s behavior.
-
•
Incorporate conditional logic: Define specific conditions under which different actions or transitions occur.
-
•
Use variables to control memory: Define how variables in memory are set and used by a language model
-
•
Integrate code: Define Python statements to manipulate variables or interact with external APIs at any node of the interaction
-
•
Call agent modules as functions: Define callable agent systems that can be called at points in the interaction, allowing an interaction to return to a previous point, and giving additional control over prompt scopes.
-
•
Visualize agent behavior: Gain a clear understanding of the agent’s decision-making process and current state through an intuitive graphical interface.
-
•
Design agent interactions in spreadsheets: Enable the design of complex agents using spreadsheet software such as Microsoft Excel, Google Sheets, etc.
The designer does not necessarily need to be a person–the high level of flexibility of AutoGRAMS makes it possible for an AutoGRAMS agent to design other AutoGRAMS agents. AutoGRAMS also contains functionality to allow agents to modify their own graph directly, opening up possibilities for agents that learn by modifying their own behavior.
By bridging the gap between intuitive design and complex agent behavior, AutoGRAMS opens up new possibilities for developing AI agents that can effectively navigate diverse scenarios or tasks, adapt to user input, and maintain coherent, goal-oriented conversations. Specifically, by enabling designers to define the behavior of individual nodes and transitions flexibly and easily within a graphical framework, AutoGRAMS facilitates precise control over multi-step interactions with language models. This versatile tool serves as both a high-level programming language and a comprehensive framework for developing language model-based agents and chatbots.
The layout of this paper is as follows: We cover the AutoGRAMS framework in Section 2. We demonstrate the mapping from python-like code and AutoGRAMS, along with the AutoGRAMS compiler–in Section 3. We introduce how AutoGRAMS agents are executed in Section 4. And finally, we discuss self-modifying agents and agents that can design agents in Section 5.
2 AutoGRAMS Framework
We introduce AutoGRAMS and go over the process of defining a set of nodes and associated fields to create agents in Subsection 2.1. We then delve into the key concepts and mechanics of the AutoGRAMS frameworks, focusing on; nodes and node types in Subsection 2.2, transitions in Subsection 2.3, variables and memory in Subsection 2.4, functions in Subsection 2.5, methods of designing autograms in Subsection 2.6, and finally, autogram configurations and settings in Subsection 2.7. Some other details of the framework are included in the appendix, including how prompts are formed (Appendix A), and details of how language models are used (Appendix B), a description of interjection nodes that help model unexpected user replies (Appendix C), and simulating user replies in conversational autograms (Appendix D),
2.1 General overview
AutoGRAMS allows for a unified representation between graphical chatbots that use states and transitions, general code, and approaches that leverage LLMs internal reasoning abilities. An AutoGRAMS program, which we refer to as an autogram (autonomous program) 111An “autogram” can also refer to a sentence that describes itself by providing an inventory of its own characters. Likewise, autograms in this framework are also self-referential–they contain a reference to their own object that can be used to self-modify., is defined by a collection of nodes and the transition functions between them. Within the nodes, there are associated fields that define instructions to be executed and transition behaviors/functions to other nodes.
![Refer to caption](extracted/5729962/figures/qa.png)
An autogram is represented by a set of nodes and their associated fields. The nodes are used to form a data structure representing a graph, and execution of the autogram consists of taking a trajectory of one or more steps along this graph. In a conversational setting, one of the main goals is to model different conversational trajectories, where each node represents a chatbot’s reply, and each edge represents a user’s reply. For instance, consider the conversational graph in Figure 1–An AI agent designed to quiz the user on a subject may start by asking a question, and depending on whether the answer is correct, will continue by either going over the answer with the user, or asking them to try again. If a user repeatedly answers incorrectly, the agent may provide an explanation and then proceed to a new question after addressing any further inquiries. This type of agent can be modeled as a set of nodes, each with a unique set of instructions and transition behaviors.
There are many possible fields than can be defined for each node, but some of the most important fields are:
-
•
instruction: This field determines the node’s behavior and is interpreted differently depending on the node type. The instruction could be a language model prompt, executable code, or a call to another AutoGRAMS function.
-
•
action: This field determines the node type and how the instruction is processed. In simpler scenarios, such as Figure 1, all nodes can be designated as chat type, indicating that each node generates a response.
-
•
name: This provides a unique identifier for the node, allowing other nodes to reference it during transitions.
-
•
transition_question: This is a question posed to the language model to help determine the next node to transition to. Common examples include “What did the user say?” or “Does the user want X?”
-
•
transition_choices: This is a multiple-choice list of potential answers to the transition_question. For example, it could be [“The user said X“, “The user said Y”, “The user said Z”], [“yes”, “no”], or similar. The language model uses the transition_question and conversation history to predict the most fitting transition_choices.
-
•
transitions: This is a list of node names that the agent can transition to, depending on the predicted answer to the transition_question. For instance, if transitions is [“node1”, “node2”], transition_question is “Does the user want X?”, and transition_choices is [“yes”, “no”], then the autogram transitions to “node1” if the model predicts “yes” and to “node2” if it predicts “no”. The lists are order-dependent.
Each node in chatbots like the one described in Figure 1 performs the following steps:
-
1.
Generate a Contextually Relevant Reply: This is achieved by passing node-specific instructions to an instruction-following language model, ensuring the reply aligns with the node’s intended purpose.
-
2.
Determine the Next Node: In conversational autograms, this decision may depend on the user’s new reply. To accomplish this, the designer may:
-
•
Define a list of permissible next nodes within the graph structure.
-
•
Define a multiple-choice question about the user’s reply, with each answer choice corresponding to one of the possible next nodes. This question is then presented to a language model, and the predicted answer determines the node transition.
-
•
Nodes come in different types as defined by their actions (see Section 2.2), and in conversational settings (e.g. chatbots), chat-type nodes pause to return a reply to the user, pausing the trajectory through the graph until the user replies. When the user replies, the autogram can decide which of the possible transitions to select based on the user’s reply. So for instance, a node that provides a reply that asks a yes or no question may transition to a different node depending on whether the user answers affirmatively or negatively. This allows the designer of the autogram, which could be a person or itself an autogram, to pre-program proactive responses that lead conversations in well thought out ways towards objectives such as obtaining information from a user, troubleshooting a problem step by step, or completing a task.
Other types of nodes can perform other functions; e.g. use a language model to generate thoughts, call external APIs, or even execute arbitrary Python code, allowing for AutoGRAMS to be used for more general applications. Memory in an AutoGRAMS is set by variables (Section 2.4) defined at the execution of nodes, which can be referenced in later instructions or code executed by nodes at a future point. The AutoGRAMS framework is Turing complete with respect to its node definitions; the designer can implement nodes that result in loops and conditionals using graph logic, and individual nodes can be configured to execute basic code statements and variable assignments. We show that most functional Python programs can be represented using an autogram, and we implement an AutoGRAMS compiler that can convert vanilla Python code (Section 3). It also enables Python code and AutoGRAMS specific nodes to be easily integrated together. AutoGRAMS subgraphs can also be called as functions, allowing for AutoGRAMS modules to be reused (Section 2.5). An autogram can also reference its own object, allowing an autogram to be fully self referential and self-modifying (Section 5.3).
2.2 Node types
An autogram is defined by a set of nodes, where each node performs a single step of execution. An execution step could contain anything from calling a language model with a prompt to generate text, to executing arbitrary code. Each node has an instruction that determines what will be executed at that step. Each node also has an ’action’ which defines what type of node it is, and how that instruction will be interpreted.
The main actions in AutoGRAMS, which also correspond to types of nodes, are as follows
-
•
Chat actions:
-
-
chat: In these nodes, the instruction serves as a turn-specfic conversational prompt for the language model, incorporating the current memory state of the autogram. The autogram returns both the language model’s response and the updated memory state, allowing the system to pause and wait for further user input.
-
-
chat_exact: This action bypasses the language model’s response generation. Instead, it directly outputs the instruction as a pre-programmed conversational reply, providing more precise control over specific responses.
-
-
-
•
Thought actions: These nodes utilize the instruction as a text generation prompt for the language model. Unlike chat nodes, there is no pause for user input after execution; instead, the autogram immediately transitions to the next node. This type of node is useful for internal reasoning steps.
-
•
Python actions: In this case, the instruction is treated as Python code and executed directly by the Python interpreter, enabling custom logic and functionality within the autogram.
-
•
Function actions: These actions trigger the execution of a separate subgraph within the AutoGRAMS graph. The instruction contains a reference to a callable AutoGRAMS node (potentially with arguments) to initiate this subgraph, returning control to the calling node upon completion. (For more details, refer to Section 2.5 on AutoGRAMS functions.)
-
•
Prompt actions: This action type focuses on modifying the initial prompt presented to the language model. Language models receive both a turn-specific prompt (the instructions for chat and thought nodes) and an initial, overarching prompt. This action provides the capability to reset or alter this initial prompt, influencing the model’s behavior across the entire conversation.
-
•
Transition actions: These actions are primarily structural and do not utilize the instruction field. They serve as placeholders within the AutoGRAMS graph, allowing for additional branching points and greater control over the flow of the conversation without requiring specific instructions.
We describe how AutoGRAMS instructions are executed depending on node type in Figure 2.
![Refer to caption](extracted/5729962/figures/autograms_actions.png)
2.3 Transitions
After executing the instructions within a node, the autogram must select the next node to execute. Each node contains fields that govern this selection process.
Each node has a name field and a transitions field, which stores a list of strings. In the simplest case, each string in transitions directly references the name of another node in the graph. A node with only one possible transition would list a single name, while nodes with branching possibilities would list multiple names.
In other cases, a string in transitions may not directly refer to a node name. For example:
-
•
Wildcard Transitions: A transition string with a “.” suffix (e.g., “mynode.*”) indicates a conditional transition. This assumes the existence of nodes named “mynode.a,” “mynode.b,” etc., each with a defined boolean_condition field containing a Python statement that may reference variables in memory (see Section 2.4 for more on Variables). If-elseif-else logic is then used to select the appropriate node based on the evaluation of these boolean conditions. (see Algorithm 1 for a pseudo-code example, and Appendix E for a visualization)
-
•
Return Transitions: A transition string of “return” (or “return variable_name” to return a variable) signifies a return from a function call within an AutoGRAMS function, transitioning back to the calling node.
-
•
Variable Transitions (advanced use case): A transition string can also reference a variable, using its value in memory to determine the next node. This requires careful validation to ensure the variable’s value corresponds to a valid node name. A variable transition could become a return or wildcard transition, since a variable could specify any string–which could correspond to any transition type.
Function-type nodes override this standard transition behavior. They transition directly to the node specified in their instruction, returning to the original node upon function completion. (See Section 2.5 for details on AutoGRAMS functions.)
Pseudo-code showing how wildcard transitions implement if/else logic. Assume selected transition is “mynode.*”, and there is a dictionary called nodes which contains nodes with names “mynode.a”,“mynode.b”,“mynode.c”
Figure 3 presents a flowchart of how nodes are selected from the previous node.
![Refer to caption](extracted/5729962/figures/autograms_apply_transition.png)
2.4 Variables
AutoGRAMS allows the use of variables to store and manipulate memory, enabling dynamic behavior within an autogram. Variables in AutoGRAMS are managed as Python variables and can be created, referenced, and modified throughout the execution of an autogram.
2.4.1 Variable Assignment and references
Variables are assigned using the ‘=‘ sign within the instructions of nodes. The node’s instruction (with the assignment parsed out) will be executed in accordance with the node’s action, as previously described in Section 2.2. This means that the type of node will also influence what value is assigned to the variable.
-
•
Python Nodes: Execute Python code directly. For example, a node with the instruction,
x = [0, 1, 2]
initializes a variable x to a list.
-
•
Chat and Thought Nodes: Use the language model to generate text. For example, an instruction like,
x = summarize the recent conversation history
will store the generated text in the variable x. In this specific example, the text stored in the variable x will be a language model summary of the recent conversation history. i.e. at run time, this instruction will be executed and its results stored as a Python string in the variable x.
-
•
Function Nodes: When a function node is applied, it can pass variables as arguments to the called AutoGRAMS function. The calling node’s instruction specifies the variables being passed. Within the called function, these variables are assigned and can be used or modified. For instance, a function call,
summary = summarize(text)
will have the variable text accessible within the subgraph of the summarize() node. When that subgraph encounters a return transition, it will return a value that will be stored in the variable summary. See Section 2.5 for more details on how this works.
Although it is technically possible to assign variables in other node types, it is not recommended and currently has no utility.
Variables can also be referenced in two ways:
-
1.
Direct Use: Variables are used directly in Python code within the instructions of Python or Function-type nodes, or in the boolean_condition attribute of a node (used for wildcard-transitions). For instance:
sorted(list1)
could be used as an instruction for a Python node to sort a list called list1 that was assigned as a variable at a previous node.
-
2.
$ Variable Embedding ($ syntax): Variables are embedded as strings within instructions or other node fields using $ syntax. This is useful in chat and thought-type nodes. For example:
Here is a summary of what was previously discussed: $summary. Reply to the user and remind them what was previously discussed.
$ variable embedding can also be applied in the transition_question attribute of a node to allow the multiple transition question to change dynamically. They can also be used as transitions directly, although this requires care. For example, setting
transitions = ["node1", "node2", "$variable_node"]
, for a particular node would allow the third transition to be dynamically dependant on the value of variable_node, however this will not execute successfully if variable_node is not a valid transition.
2.4.2 Variable scopes
The memory in AutoGRAMS is managed through a memory object, which stores variables and the conversation history. This object is structured as a stack, where each function call adds a new layer, and each return removes the top layer. If no functions are used, a variable can be accessed anywhere within an autogram. However, local functions are unable to view their calling scope, including any variables in that calling scope. See Section 2.5 for more details.
2.4.3 Example of Variable Usage
Consider a set of nodes that demonstrate variable assignment and referencing:
name: "write_topics" instruction: "topics = List the topics covered in today’s lesson." action: "thought" transitions: ["append_topics"]
name: "append_topics" instruction: "all_topics.append(topics)" action: "python_function" transitions: ["tell_topics"]
name: "tell_topics" instruction: "Inform the user about the topics covered today, which were: $topics" action: "chat"
In this example:
-
•
The write_topics node initializes calls a language model to write down a list of topics and store it in a variable called topics, which will be a string.
-
•
The append_topics node applies its instruction as Python code, appending the variable topics to a list called all_topics that is presumed to have been defined previously.
-
•
The tell_topics node uses $ variable embedding to dynamically insert the topics variable within the instruction string to get a response.
By utilizing variables, AutoGRAMS facilitates memory management and dynamic interactions, enabling the creation of sophisticated and adaptable AI agents.
![Refer to caption](extracted/5729962/figures/autograms_variables.png)
2.5 Function calls and scopes
AutoGRAMS introduces the ability to call graphical modules as functions, allowing subgraphs to be called similarly to functions in programming. This enables modularity, reusability, and control over variable scopes.
2.5.1 Function Nodes and Calls
Function nodes allow for the execution of separate subgraphs within the main AutoGRAMS graph. These nodes can call other nodes within the graph or external programs. The process involves:
-
•
Defining the Function Node: A node designated as a function node will have its instruction specify the function to be called, such as,
summary = summarize(document).
-
•
Passing Arguments: Variables can be passed as arguments to the called function. Within the function, these variables are assigned and can be used or modified.
-
•
Defining a callable node: A node can be made callable by its name. Node name with parentheses are callable. So a node named summarize cannot be called a s function, a node named summarize() can be called as as function with no arguments. A node named summarize(text1,text2) can be called with 2 arguments. The arguments specified by the node name determine what the variables will be named in the scope of the called node.
-
•
Defining the Subgraph of the callable node: The callable node, which acts as the root node of the subgraph, can have transitions to other nodes that perform other operations. Transitions use the string “return” or “return varname” can be added to specify that the subgraph should end.
-
•
Executing the Subgraph of the callable node: The function node transitions to the callable node specified in the instruction. The autogram executes the subgraph of the callable node until it reaches a return transition, which may optionally return specific information back to the calling node.
2.5.2 Return Transitions
Return transitions mark the end of a function’s execution and return control to the calling node. They can also return specific values to be used by the calling node. For example, return topics_summary will pass the value of topics_summary back to the calling node.
2.5.3 Scopes in Function Calls
The scope of variables in function calls is crucial for maintaining the integrity of the memory and ensuring correct execution. AutoGRAMS supports different scope types for function calls:
-
•
Local Scope: The function can only access variables passed as arguments. Variables within the function do not affect the calling scope. This is specified using the local_function action of the calling node.
-
•
Global Scope: The function can access and modify all variables and conversation turns in the calling scope. This is specified using the global_function action of the calling node.
-
•
Mixed Scope: The function can read all variables and conversation turns from the calling scope, but the variable and conversation turns set during execution are erased after returning. This is specified using the function action of the calling node.
2.5.4 Example of Function Usage
Consider a set of nodes that demonstrate the use of function calls and scopes:
name: "call_summarize_docs" instruction: "summary = summarize_and_combine(document1,document2)" action: "local_function" transitions=["process_summary"]
name: "summarize_and_combine(text1,text2)" instruction: "summary1=write a summary of the following text: $text1" action: "thought" transitions: ["summarize_second"]
name: "summarize_second" instruction: "summary2=write a summary of the following text: $text2" action: "thought" transitions: ["combine_summaries"]
name: "combine_summaries" instruction: "combined_summary=Write a summary combining summary 1 -- $summary1 summary 2 -- $summary2" action: "thought" transitions: ["return combined_summary"]
In this example:
-
•
The call_summarize_topics node calls the summarize_and_combine() function using its instruction. The document1 and document2 variables are passed as arguments.
-
•
summarize_and_combine() function starts at the node named summarize_and_combine(text1,text2). The variable document1 in the calling namespace will be mapped to text1 in the called namespace, and the variable document2 in the calling name space will be mapped to text2 in the called namespace.
-
•
After the function call, the autogram will execute the summarize_and_combine() node, followed by the summarize_second and combine_summaries nodes.
-
•
After executing the combine_summaries node, it encounters a return transition specifying return combined_summary. This will cause the autogram to return to the calling node, which was call_summarize_docs. The variable summary defined by the instruction of call_summarize_docs will be set equal to the returned variable combined_summary.
-
•
after the return is complete, the autogram will transition to to a node called process_summary (not defined above) specified by the transitions of the call_summarize_docs node.
2.5.5 Visualization of Function Calls
Figures 5(a) and 5(b) illustrate how function calls are managed within AutoGRAMS, including the transitions and scope handling.
![Refer to caption](extracted/5729962/figures/chat_fun.png)
![Refer to caption](extracted/5729962/figures/thought_fun.png)
2.6 Methods of Implementation and Visualization
At the time of writing, there are three ways to implement an autogram:
-
1.
Spreadsheet-Based Design:
-
•
Each row in the spreadsheet corresponds to a different node in the AutoGRAM.
-
•
Each column corresponds to a different node field, determined from the heading of the column.
-
•
This method allows autograms to be defined and managed using a familiar spreadsheet interface.
-
•
-
2.
Pure Python Implementation:
-
•
Nodes of an autogram can be defined in Python by initializing an autogram object and using the autogram.add_node() method to create new nodes with fields corresponding to the arguments provided.
-
•
This is the most direct way to design an autogram, as all other methods of design are mapped to this.
-
•
The main limitation is that it can be inconvenient for autograms that need to execute Python code within the autogram. Python code can be passed in as a string instruction, but this makes the code more difficult to read.
-
•
Standard conditionals and loops can theoretically be implemented using wildcard transitions and loops in the graph with exiting branches, but these can be inconvenient to define graphically.
-
•
-
3.
AutoGRAMS Compiled from Python:
-
•
This method works best for autograms deeply integrated with Python code.
-
•
It is technically a new language that closely relates to Python but has some differences that allow Python code to be integrated with AutoGRAMS nodes using Python syntax.
-
•
AutoGRAMS is general enough to execute most functional Python programs using a combination of Python and transition-type nodes along with the right graph structure and wildcard transitions for branching.
-
•
AutoGRAMS compiled from Python applies this conversion automatically, while also allowing autogram nodes with non-standard Python behaviors to be defined using a special function called exec_node, which can also be written in the code to define where in the code those nodes should be applied.
-
•
Any autogram can be fully visualized using a graphical interface. An interactive HTML file can be automatically generated for any autogram, displaying all the fields associated with each node. By clicking on a node, the code for that node is shown in a panel to the right. For more complex autograms, it is possible to specify subcategories of nodes to visualize, making the graph easier to understand, provided the category attribute is used in the nodes defined in the autogram. An interactive graph of an autogram is shown in Figure 6. Future versions of AutoGRAMS will enable users to directly design and edit autograms from a similar interface.
![Refer to caption](extracted/5729962/figures/qa_html.png)
2.7 AutoGRAMS Configurations
AutoGRAMS configurations are essential for controlling various settings of an autogram. Key aspects of these configurations include:
-
•
Model Selection: Defining which models will be used within the autogram.
-
•
Prompt Templates: Establishing templates to guide the language model responses.
-
•
Python Imports and APIs: Specifying the Python modules and APIs accessible from the AutoGRAMS code.
Configurations are typically managed through a JSON file, with the exception of the Python modules field. This field requires actual Python modules containing code, which are passed to the autogram, allowing the use of any Python-defined function within Python function nodes.
For further details:
These settings ensure that the autogram operates correctly, utilizing the appropriate models, prompts, and Python functionalities.
3 AutoGRAMS graph compiler
AutoGRAMS gives the ability to embed general code within graphical AI agents and chatbots. One way to design autograms is to define nodes one-by-one, with transitions to other nodes specified in node definitions. This is especially useful for chatbots, since conversations often have certain states that can be modeled as a graph or tree. AutoGRAMS also allows nodes in this graph to execute code, rather than give a reply. However, this requires the code that executes at a node to be passed as a string in a node’s instruction. This graphical representation also means that loops and conditionals in the AutoGRAMS graph need to be defined using the appropriate transitions. We were motivated to allow deeper integration between general code statements and the graphical representations used by autograms. To do this, we implemented the AutoGRAMS compiler, which can map a combination of Python code and statements that define AutoGRAMS nodes, to an autogram that can be executed by the AutoGRAMS interpreter (Section 4). It does this by mapping the code in a file to a set of nodes with the appropriate attributes to perform the computation specified in the code file.
AutoGRAMS is general enough to execute functional programs that can be represented common features such as loops, conditionals, functions and variables. This is possible by combining python-type nodes with the desired flowchart needed to execute a program. For instance, consider a program that computes the nth element Fibonacci sequence recursively (While this is inefficient, we use it for illustrative purposes). Python code for this is given in Figure 7(a).
![Refer to caption](extracted/5729962/figures/fib_rec_light.png)
![Refer to caption](extracted/5729962/figures/fib_full_graph.png)
The function accepts a number, returns 0 of the number is 1, returns 1 if the number is 2, and returns the sum of and if the number is greater than 2. this function can be represented completely in AutoGRAMS with a combination of python-type nodes, function-calling nodes, transition nodes, return transitions, and wild card transitions. The first node of the graph must be a callable node named “fibonacci(n)”. The parentheses in the name means that it can be called function calling nodes, and using n means that the first argument passed can be referenced with the variable name “n”. A node with an action and instruction of can be used to call this function as set the result to a variable called “x”.
The first conditional of the program is implemented with a wildcard transition, where the root node of the conditional applies a transition to , which means that the next node will either be , , etc. In this specific case, since there are 4 possible branches, it is necessary to define nodes named and . Each of these nodes has a attribute that can be used to specify the condition at which this branch can be executed. Boolean conditions are passed to the Python interpreter in AutoGRAMS so they can include any Python code or references to any variables visible at the current scope. does not need a boolean conditional because it is the last node in alphabetical order. If the oder condition are reached, the appropriate value can be returned by using a python-type that simply includes the desired number, and then applying a return transition immediately after, which returns the result of the previous node if no return variable is specified. If the else condition is met reaching the node , the autogram can use two nodes that call the Fibonacci graph again with and , sum the result, and return the result. The AutoGRAMS graph, which was compiled automatically from the Python code using the AutoGRAMS compiler, is given in Figure 7(b).
The AutoGRAMS compiler is used to convert code with python-like syntax into a set of AutoGRAMS nodes that can be executed as an autogram. It is also possible to code any autogram in pure Python (or a spreadsheet) by defining nodes one by one, in which case the AutoGRAMS compiler is not used. However, for autograms that contain Python statements or programmatic features like loops and conditionals, the compiled version has several advantages:
-
•
Python statements can be included directly as code instead of as strings in the instruction of python_function nodes
-
•
Python style variable assignments can be be used instead of assignment variable in an instruction
-
•
AutoGRAMS function calls and functions can be defined using Python-like syntax. Note that functions are by default treated as local AutoGRAMS functions, but this behavior can be changed using the function decorator ‘@global_function” or “@function” above the function definition.
-
•
the order of node chains to be inferred automatically from the order they appear in code, even if names and transitions aren’t defined.
-
•
Python loops and if/else conditionals to compile into a graph automatically
Compared with standard Python programs, AutoGRAMS compiled from Python allows for exec_node() statements to define AutoGRAMS nodes at any point in the code. These nodes can connect with other nodes defined by exec_node() by specifying those nodes in the transitions argument, in which case these transitions will be governed by multiple choice questions asked to the language model. Chat type nodes defined in exec_node() temporarily return a serializable object defining the entire state of the autogram (see Section 4.2), allowing a conversation to continue from a specific place in the code after receiving a new user reply. The graphical representations used by AutoGRAMS also give it certain advantages for self-modifying code, as discussed in Section 5.
The AutoGRAMS compiler recursively traverses the Python abstract syntax tree representation of the code to add AutoGRAMS nodes to the graph. A overview of how the Autograms compiler converts code into AutoGRAMS nodes is given in Appendix F.
4 AutoGRAMS interpreter
Autograms are executed via an iterative process that selects nodes and executes them. Within the AutoGRAMS framework, an autogram can be applied using the reply method, which is for conversational modules, and the apply_fn method, which allows an AutoGRAMS function to be called directly from Python, and it meant for non-conversational modules. Both work in a very similar fashion, other than the way they terminate their main loop. The reply method terminates and returns a result when it encounters a chat node, and is designed to restart from the node it left off at when it receives another user reply. The apply_fn method terminates when it encounters a return statement–meaning the AutoGRAMS function is done executing and is returning a result. An illustration of the outer loop of the AutoGRAMS reply method is given in Figure 8.
4.1 Main steps for obtaining conversational replies
The outer loop of autogram.reply() has 6 main steps:
-
1.
get the variable output of previous node (skip this step if no previous node). This step calls a node specific method called get_variable_output() which returns the variable output of the node. The specific behavior of this will depend on the node type, but this is often the text generated by the chatbot.
-
2.
assign variables assigned in previous node to memory (skip this step if no previous node). If any variable outputs were in the previous node’s instruction, the memory object (Section 4.2) assigns the nodes variable output to a variable with the name defined in that instruction. This goes in the top level of the memory objects stack.
-
3.
apply transition function from previous node (skip this step if no previous node). This calls a node specific method called apply_transition() to get an unprocessed new node id. The behavior of this method will can be different for different types of nodes, but is generally similar for non-function calling nodes. If there is only 1 possible transition in the transitions list of the node, then the result will usually this transition.If there are multiple transitions, the transition will depend on what the classifier predicts.There also may be interjection transitions if the node is a chat type. If the node is calling a function, the new node id will be the root node being called.
-
4.
post-process the new node id (skip this step if no previous node). If the new node id output by apply_transition() corresponds to another node in the graph, and the next step will be to simply select that node. However, if the new node id corresponds to a return statement or a wildcard transition, additional post processing will be needed. A return statement will require using the memory object to find the previous function calling node, and transition back to this node. A wild card transition requires finding the nodes with the matching prefix and applying if/else logic to determine which node to select.
-
5.
get the new node and apply its instruction. This calls a node specific method called to execute the instruction of the node. This can very greatly for different node types. Apply instruction will always strip any variable assignments on the left of the node’s instruction string. It will then render the string for any $ variables in the instruction. The for chat and thought type nodes will then call the language model with the instruction and conversation history used in the prompt. In python-type nodes, the instruction will be executed as code by the Python eval function, with all in scope variables and Python modules set by the autogram config included as local variables.
-
6.
if the new node is a chat node, return a result with the models reply and the memory object storing the state of the program, continue the loop otherwise
![Refer to caption](extracted/5729962/figures/autograms_loop.png)
4.2 Managing Memory
![Refer to caption](extracted/5729962/figures/memory_object.png)
AutoGRAMS uses a “memory object” to keep track of the state of the program, including all the variables set by AutoGRAMS nodes and conversation turns. The memory object stores a stack of all of the layers of functions in an autogram. Each function call adds a layer to the stack, and each function return removes a layer from the stack. The type of function determines which variables and language model conversation turns are considered to be in scope at a certain layer of the stack. Within a global function or regular function, variables and conversation turns at the calling layer are visible, whereas in a local function these are not visible. Methods for obtaining variables and language model conversation turns within the memory object class add all variables or turns until they encounter a layer corresponding to a local function, which cuts off access to everything below the local function in the stack. When returning from a function, a layer in the stack is deleted. If the function was a global function, all variables and conversation turns are added to the calling layer. Otherwise these fields are erased, and only variables modified or returned by the function will be remembered. The stack of the memory object is illustrated in Figure 9.
In addition to storing the stack, the memory object is also used to log model turns and nodes visited. The entire state of an autogram can be recovered from the memory object, allowing autograms to continue from where they left off after executing chat-type nodes that wait for a user reply. The memory object and user reply can be passed to the autogram to obtain the model’s reply.
4.3 Interpreting Python statements
Python statements (as well as the ‘boolean_condition‘ field for wild card transitions, and arguments to AutoGRAMS functions) are interpreted by the Statement Interpreter. At initialization, the Statement Interpreter uses the Autogram Config to load all Python imports and modules that will be allowed within the scope of the program. It also overrides any Python builtins that are not explicitly listed in the Autogram Config to prevent them from being called. These imports are effectively treated as global variables that can be accessed anywhere from within the autogram. When the Statement Interpreter called to execute code, it uses the Python eval command–it includes all variables in scope in the memory object, as well as Python modules and imports passed from the Autogram Config in the scope of the code execution during the call to eval.
5 Self-modifying and meta autograms
5.1 Introduction
AutoGRAMS allows a designer to specify a series of instructions and transitions that form a program that control this process. However, the space of possible useful processes like this that can be formed is extremely large. For instance, there are likely many processes in AutoGRAMS that would be useful for solving certain problems, or handling certain conversational scenarios. It could be difficult for the designer to specify every possible process that the agent may need to handle. It would therefore be very useful to have a “meta-process” that can design large sets of these processes in advance, or be able to define new processes on the fly when encountering new scenarios.
AutoGRAMS is designed in such a way that the meta-process can be of the same form (an AutoGRAMS graph) as the processes that the meta-process designs. This makes it possible to design meta-processes that can self-modify, giving greater flexibility as compared with a system where the meta-process is of a different form from the process. Within AutoGRAMS, we refer to these meta-processes as meta-autograms.
![Refer to caption](extracted/5729962/figures/meta_agent.png)
5.2 Meta autograms
One of the goals of AutoGRAMS was to design an agent framework general enough that an agent could be used to design another agent of the same form. AutoGRAMS provides a simple working example of an AutoGRAMS function that can be used to design a chatbot autogram. The function, which we refer to as a meta-autogram, takes in a prompt as an argument. It applies the following steps
-
•
apply a thought node that asks the language model to design an outline of the possible ways the conversation can go from this prompt
-
•
apply a thought node that asks the language model to design a graph using the dot (graphviz) 222https://graphviz.org/ language, where each node should be labeled by how the agent should reply, and each edge should be labeled by how the user should reply.
-
•
call an external Python module to parse the nodes from the graphviz graph, go back to step 2 if problems with the graph are detected
-
•
For each node parsed from the graphviz graph, call thought type nodes to generate the attributes associated with that node. The prompt for the thought node conditions on the graph labels for that node from step 2 to guide it. All the attributes are saved as variables and used to form a diction ary of arguments for each nodes
-
•
return the arguments needed to initialize a new autogram
This process is visualized in Figure 10.
5.3 Self-modifying autograms
One advantage of the graphical representation of AutoGRAMS is that a node at one point in the graph can perform operations that modify nodes at other points in the graph. For instance, a series of nodes could be configured to perform the following operations
-
1.
select a node in the graph
-
2.
call a language model to decide what the new attributes of that node should be, and save the arguments to define this node in variables
-
3.
if any new nodes need to be defined (because transitions of original node changed), loop through and define new nodes using the process in step 2. Continue to define child nodes until all nodes are connected to the graph (the system will eventually need to connect all nodes back to existing nodes so that it doesn’t get stuck in an infinite loop).
-
4.
loop through to initialize/modify nodes using the arguments saved in variables during steps 2 and 3
This is one of many possible ways an autogram could self-modify by using nodes to modify other nodes or adding new nodes.
We provided the ability for autograms to directly access their own data structure during their execution when running in “self-referential” mode. An autogram is implemented as an instance of the Autogram class in Python, and when self-referential mode is enabled in the autogram config, nodes are able to reference a variable called self which refers to the autogram’s own object that interprets the autogram. This potentially allows any of the autograms nodes, methods, or other attributes to be accessed during the execution of the autogram. One potential usecase for this is dynamic modification of the autogram. For instance, it is possible for an autogram to add new node to itself by calling the self.add_node() method with the appropriate arguments from a python-type node. We implemented a simple proof of concept of this where an autogram simply adds and executes a new node at each turn, instead of having the nodes defined before hand. This works by having an AutoGRAMS function to generate all the attributes of a new node and add it to the autogram, and then when the function returns, apply a dynamically defined transition with a variable name that specifies the name of the new node. The new node then transitions back to the node that calls the function to design a node, allowing the chatbot to continue indefinitely. The full set of nodes needed to perform this algorithm along with explanations are provided in Appendix H.
5.4 Future Directions
Our future work will focus on developing software libraries of self-modifying autograms capable of defining useful autogram processes and learning from their interactions with the environment. In this paper, we demonstrated an autogram that can design an AutoGRAMS chatbot based on a prompt describing the chatbot’s required functionality. While the scope of the present paper was to develop useful representations for designing such algorithms, future work will aim to implement autograms that perform these functions.
Future applications could include autograms that process conversations conducted by a human agent and generate new autograms to replicate the human agent’s logic. Additionally, an autogram could be programmed to analyze its own interactions, identify where it went wrong due to unexpected user input, and modify its nodes accordingly.
Advanced autograms could also retrieve information on complex topics and adjust their reasoning structures to better integrate this material into their conversational or reasoning routines. This has significant implications for using large language models or generative AI in robotics. By fine-tuning language models to better model autograms, we hope to enhance the ability of autograms to design other autograms and self-modify.
6 Related Work
The development of AutoGRAMS is situated within a broader landscape of research encompassing natural language processing (NLP), AI agents, and programming languages for AI. This section reviews key advancements and their contributions to the field, highlighting how AutoGRAMS builds on and extends these innovations.
Significant progress in NLP and neural language modeling (Bengio et al., 2000) has been driven by transformer-based architectures (Vaswani et al., 2017), such as BERT (Bidirectional Encoder Representations from Transformers) Devlin et al. (2019), GPT-3 (Generative Pre-trained Transformer 3) Brown et al. (2020), and T5 (Text-To-Text Transfer Transformer) Raffel et al. (2020). These models have shown exceptional capabilities in understanding and generating human-like text, providing a robust foundation for developing advanced conversational agents. Notable conversational agents include Google’s Meena Adiwardana et al. (2020), Facebook’s BlenderBot Roller et al. (2020), and Microsoft’s DialoGPT Zhang et al. (2019), which leverage large-scale pre-training and fine-tuning on diverse datasets to perform a variety of conversational tasks. AutoGRAMS relies heavily on the ability of pretrained Transformer models to generate realistic text that can be used in conversational replies and apply reasoning steps.
The intersection of programming languages and AI has been extensively explored to facilitate the integration and development of AI models. Python, along with frameworks like TensorFlow (Abadi et al., 2016) and PyTorch (Paszke et al., 2019), has become integral to AI research, providing comprehensive libraries for model development, training, and deployment. Domain-specific languages (DSLs) like Keras for neural network modeling (Gulli & Pal, 2017) and Rasa for building conversational AI (Bocklisch et al., ) are designed to simplify AI programming and make it accessible to a broader audience. Code language models (Chen et al., 2021; Nijkamp et al., 2022; Roziere et al., 2023) have enabled AI to automatically generate code that can solve complex tasks, allowing for a deeper integration between programming languages and AI. In the AI agent space, Flows Josifoski et al. (2023) is conceptual framework that facilitates structured reasoning and collaboration among AI systems through modular, message-based interactions. DSPyKhattab et al. (2023) is a programming model that abstracts language model pipelines as text transformation graphs.Jojic et al. (2023) demonstrate the Iterations by Regimenting Self-Attention technique, which manipulates the self-attention mechanism in language models to trigger and control iterative behaviors for executing algorithmic tasks. Building on the integration of natural language with programming, the AIOS Compiler (Xu et al., 2024) is a system that leverages LLMs to interpret and execute instructions in natural language. AutoGRAMS integrates programming languages and language models by combining graphically represented agent nodes with graphically represented program nodes. computer programs essentially model flowcharts (Goldstine & von Neumann, 1947), where each node in the flow chart executes and instruction and each transition is facilitate by a branch. In AutoGRAMS, the idea of using a flow chart for programming is combined with the idea of using a flowchart to model multi-step language model interactions. The definition of explicit nodes, which can represent conversational states, and transitions between nodes make AutoGRAMS especially well suited for conversational AI applications Transition operations can optionally be governed by multiple choice questions for a language model, and instructions can optionally be governed by prompts to a language model, allowing language models to be deeply embedded in the execution of a program. The visual programming component of an autogram also gives added interpretability, as any autogram can easily be visualized as a graph.
AutoGRAMS is motivated by previous work showing that language models can reason by breaking difficult problems into smaller steps. Chain of thought prompting (Wei et al., 2022) showed that prompting LLMs to generate reasoning steps can significantly improve their problem-solving performance of large language models on complex reasoning tasks. Kojima et al. (2022) further show how a simple prompting strategy, ”Let’s think step by step,” can significantly enhance zero-shot reasoning across various complex tasks. Tree of Thoughts (Yao et al., 2024) and Graph of Thoughts framework (Besta et al., 2024) models allow for more complex dependencies of thought units. Demonstrate-Search-Predict (Khattab et al., 2022) incorporates sophisticated interaction pipelines between language models and retrieval models. ReAct (Yao et al., 2022) integrates reasoning and action generation in language models. AutoGRAMS gives the flexibility for the designer, which could itself be an autogram, to design specific reasoning steps using thought-type nodes to help the language model solve complex programs. A graph of thought type nodes in combination with multiple choice transitions allows an autogram to represent highly complex thought processes that can be used to solve problems.
Like many previous works, AutoGRAMS enables the ability to incorporate external knowledge tools into an LLM, as well allowing for the LLM to make decisions. Retrieval-Augmented Generation (RAG) models Lewis et al. (2020), allow models to access non-parametric memory systems. Toolformer (Schick et al., 2024) is a model that autonomously learns to interact with external APIs . The Automatic Reasoning and Tool-use (ART) framework Paranjape et al. (2023) automates the generation of intermediate reasoning steps and integrates external tool use. AutoGRAMS gives a large amount of flexibility in how external tools are used. Python function nodes can specify the use of an external tool if they call a Python API that applies that tool. The arguments to the Python API can be variables that were set by a language model or another Python API at a previous turn. The node-specific transitions and transition multiple choice questions of AutoGRAMS give additional control over decisions about which tools are selected for use.
AutoGRAMS relates to previous works that use LLMs to engineer prompts that control the execution of tasks.Automatic Prompt Engineer (Zhou et al., 2022) automates the generation and optimization of instructions for LLMs and systematically refines prompts through an LLM-driven search and evaluation process. Active-Prompt (Diao et al., 2023) uses uncertainty metrics within an active learning framework to optimize chain-of-thought reasoning prompts for LLMs. Promptbreeder (Fernando et al., 2023) uses self-referential self-improvement to evolve and adapt prompts for specific tasks. Likewise-the prompts in AutoGRAMS are also able to result in actions that lead to the modification of an autogram. AutoGRAMS allows any part of an autogram to be modified by the autogram itself, not just the prompts.
Previous approaches have considered meta-learning (Hochreiter et al., 2001; Finn et al., 2017), learning to learn, or meta-programming. Meta-genetic programming (Schmidhuber, 1987) relates to our motivation of developing program modifying programs. Unsupervised meta-learning (Metz et al., 2018) relates to the long term vision of of AutoGRAMS to develop agents that can learn from interactions with their environment in unsupervised ways. Shinn et al. (2024) considers agents that self-reflect on their own mistakes to learn. Self-Refine Madaan et al. (2023) leverages iterative feedback by enabling an LLM to act as both the generator and refiner. MetaGPT Hong et al. (2023) is meta-programming framework that enables LLM-based multi-agent collaborations by embedding standardized operating procedures into prompts. While AutoGRAMS itself is a programming language rather than a specific algorithm, its features such the flexibility to fully self-modify, the graphical structure, Turing completeness, and easy integration of language models for executing instructions and determining transitions, make AutoGRAMS for well-suited meta-programming applications. We demonstrate an autogram can be used to design an autogram chatbot from a prompt. AutoGRAMS gives the potential for the development of highly complex meta-agents and self-modifying agents by enabling autograms to design new AutoGRAMS graphs or modify their own graphs.
AutoGRAMS also relates to other approaches for controlling text generation. Previous work has used discriminators (Dathathri et al., 2019; Krause et al., 2020; Yang & Klein, 2021; Qin et al., 2022), special input tokens (Keskar et al., 2019), prompts (Reynolds & McDonell, 2021), instruction following (Ouyang et al., 2022; Longpre et al., 2023) or finetuning using a reward signal (Ziegler et al., 2019; Rafailov et al., 2024) to better constrain the outputs of language models for improved controllability and safety. AutoGRAMS allows a complex network of prompts to be defined to control multi-turn or multi-step interactions.
7 Discussion
This work presented the AutoGRAMS, a novel framework and high level programming language for controlling multi-step interactions with an LLM. AutoGRAMS gives the designer the ability to create branched multi-step interactions with an LLM in a way that is analogous to how programming languages allow for the design of branched multi-step interactions with a processor. The designer can define these steps of the interaction in advance by writing prompts for a language model at each step. AutoGRAMS also allows language models to exert direct control over transitions in the interaction by enabling the designer to write multiple choice questions that determine which branch to take. This structure enables the designer to use natural language to effectively govern the flow of the of steps that an AutoGRAMS takes. The graphical structure of an autogram allows it to be implemented and visualized in an intuitive way.
AutoGRAMS is especially useful for designing complex conversational flows. AutoGRAMS allows for agents be designed to behave in more predictable ways over longer conversations, giving added control-ability and safety to conversational agents. It accomplishes this by allowing the designer to define a prompt for how to reply to the user for every possible turn in the conversation in advance. The designer has to anticipate the users response, for which The designer can then also design a series of multiple choice questions for a language model that govern which turns will actually be implemented based on that response. Additionally, if the conversational agent needs more complex intermediate reasoning steps or memory, these can be implemented in AutoGRAMS as well. The full state of the autogram can be saved in a serializable object, making it easy to reload the complete state and memory of a complex conversational agent from a database.
By allowing conditionals and code statements, and variables, AutoGRAMS allows the functionality of a Turing complete programming language to be integrated into the graphs that define the steps of the language model interaction. We demonstrate how Python programs can be mapped to an AutoGRAMS graph of python-type nodes and embedded with other types of nodes. This enables the designer to define data structures to store and access memories, which could be text generated by a language model or variables created in code. These variables can be embedded into prompts, transition questions, referenced in code, or even used to create dynamically executing code or dynamic transitions in the graph. This high level of flexibility makes it possible for AutoGRAMS to be used for very advanced applications of AI agents.
We also demonstrate mechanisms for AutoGRAMS to design other AutoGRAMS. We also demonstrate their ability for self-adaptation and modification. AutoGRAMS allows an agent to be fully self-referential with respect to the nodes that define the agent. We anticipate self-modifying AI agents to be useful for generalizing agents to more situations and allowing agents to learn from experiences and information, and AutoGRAMS provides a useful framework to build such agents.
Acknowledgments
The authors thank Subu Subramaniam for helpful discussions on how autograms could be applied and for helping to test the AutoGRAMS repository.
References
- Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. TensorFlow: a system for Large-Scale machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pp. 265–283, 2016.
- Adiwardana et al. (2020) Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977, 2020.
- Bengio et al. (2000) Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances in neural information processing systems, 13, 2000.
- Besta et al. (2024) Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17682–17690, 2024.
- (5) Tom Bocklisch, Joey Faulkner, Nick Pawlowski, and Alan Nichol. Rasa: Open-source language understanding and dialogue.
- Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
- Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
- Dathathri et al. (2019) Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv:1912.02164, 2019.
- Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423.
- Diao et al. (2023) Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. Active prompting with chain-of-thought for large language models. arXiv preprint arXiv:2302.12246v4, 2 2023. URL http://arxiv.org/pdf/2302.12246v4.
- Fernando et al. (2023) Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797v1, 9 2023. URL http://arxiv.org/pdf/2309.16797v1.
- Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126–1135. PMLR, 2017.
- Goldstine & von Neumann (1947) Herman H. Goldstine and John von Neumann. Planning and coding of problems for an electronic computing instrument. Technical Report Part II, Volume 1, Institute for Advanced Study, Princeton, NJ, 1947.
- Gulli & Pal (2017) Antonio Gulli and Sujit Pal. Deep learning with Keras. Packt Publishing Ltd, 2017.
- Hochreiter et al. (2001) Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In Artificial Neural Networks—ICANN 2001: International Conference Vienna, Austria, August 21–25, 2001 Proceedings 11, pp. 87–94. Springer, 2001.
- Hong et al. (2023) Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for a multi-agent collaborative framework. arXiv preprint arXiv:2308.00352v5, 8 2023. URL http://arxiv.org/pdf/2308.00352v5.
- Jojic et al. (2023) Ana Jojic, Zhen Wang, and Nebojsa Jojic. Gpt is becoming a turing machine: Here are some ways to program it. arXiv preprint arXiv:2303.14310v1, 3 2023. URL http://arxiv.org/pdf/2303.14310v1.
- Josifoski et al. (2023) Martin Josifoski, Lars Klein, Maxime Peyrard, Nicolas Baldwin, Yifei Li, Saibo Geng, Julian Paul Schnitzler, Yuxing Yao, Jiheng Wei, Debjit Paul, and Robert West. Flows: Building blocks of reasoning and collaborating ai. arXiv preprint arXiv:2308.01285v3, 8 2023. URL http://arxiv.org/pdf/2308.01285v3.
- Keskar et al. (2019) Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858, 2019.
- Khattab et al. (2022) Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp. arXiv preprint arXiv:2212.14024v2, 12 2022. URL http://arxiv.org/pdf/2212.14024v2.
- Khattab et al. (2023) Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714v1, 10 2023. URL http://arxiv.org/pdf/2310.03714v1.
- Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916v4, 5 2022. URL http://arxiv.org/pdf/2205.11916v4.
- Krause et al. (2020) Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.06367, 2020.
- Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv preprint arXiv:2005.11401v4, 5 2020. URL http://arxiv.org/pdf/2005.11401v4.
- Longpre et al. (2023) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pp. 22631–22648. PMLR, 2023.
- Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651v2, 3 2023. URL http://arxiv.org/pdf/2303.17651v2.
- Metz et al. (2018) Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Meta-learning update rules for unsupervised representation learning. arXiv preprint arXiv:1804.00222, 2018.
- Nijkamp et al. (2022) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474, 2022.
- Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.
- Paranjape et al. (2023) Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014v1, 3 2023. URL http://arxiv.org/pdf/2303.09014v1.
- Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
- Qin et al. (2022) Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy-based constrained text generation with langevin dynamics. Advances in Neural Information Processing Systems, 35:9538–9551, 2022.
- Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
- Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020.
- Reynolds & McDonell (2021) Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended abstracts of the 2021 CHI conference on human factors in computing systems, pp. 1–7, 2021.
- Roller et al. (2020) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M Smith, et al. Recipes for building an open-domain chatbot. arXiv preprint arXiv:2004.13637, 2020.
- Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023.
- Schick et al. (2024) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.
- Schmidhuber (1987) Jürgen Schmidhuber. Evolutionary Principles in Self-Referential Learning, or on Learning How to Learn: The Meta-Meta-… Hook. PhD thesis, Technische Universität München, 1987. PhD thesis.
- Shinn et al. (2024) Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
- Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
- Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903v6, 1 2022. URL http://arxiv.org/pdf/2201.11903v6.
- Xu et al. (2024) Shuyuan Xu, Zelong Li, Kai Mei, and Yongfeng Zhang. Aios compiler: Llm as interpreter for natural language programming and flow programming of ai agents. arXiv preprint arXiv:2405.06907v2, 5 2024. URL http://arxiv.org/pdf/2405.06907v2.
- Yang & Klein (2021) Kevin Yang and Dan Klein. Fudge: Controlled text generation with future discriminators. arXiv preprint arXiv:2104.05218, 2021.
- Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629v3, 10 2022. URL http://arxiv.org/pdf/2210.03629v3.
- Yao et al. (2024) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
- Zhang et al. (2019) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536, 2019.
- Zhou et al. (2022) Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910v2, 11 2022. URL http://arxiv.org/pdf/2211.01910v2.
- Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.
Appendix A Prompt formation
Language models in AutoGRAMS take in a list of input turns (corresponding to user turns or instructions), a list of output turns (corresponding to assistant turns/model outputs), as well as an optional prefix for how the reply should start (for internal models only).
When preparing the prompt to get a new language model output at either a thought node or a chat node, the autogram re-iterates over past turns to get these input and output turns.
For every previously visited node that is within the current scope (depending functions and function types if used), if that node is a thought node or a chat node, it adds a turn to the conversation history that the language model sees. the inputs and outputs are determined as follows:
for each past thought or chat node visited:
-
•
if it is a thought node
-
-
if the thought node came immediately after a user reply, then both the instruction and user reply will appear in the input for that turn
-
-
otherwise, only the instruction will appear in the input for that turn
-
-
-
•
if it is a chat node
-
-
if the chat node came immediately after a user reply, then only user reply will appear in the input for that turn. The old instruction is discarded for chat nodes to avoid long prompts. We may later add an option to retain chat instructions in the autogram config
-
-
otherwise,the chat node’s instruction will appear in the input for that turn
-
-
The initial prompt is pre-appended to the first input.
For the last input, corresponding to the end of the prompt, the string is determined by the instruction template. The default instruction template is:
<last_response> Instruction for <agent_name>: <instruction>, where <last_response>, <agent_name>, and <instruction> are special place holder tokens for the instruction, agent name, and last response. Sometimes the last response will be the empty string if the last user response is already in one of the past inputs (which will happen if there is at least 1 thought node before replying with a chat node).
Lastly, replies have a reply start, which is “Agent’s reply:” if the default start template and agent name are used. Depending on whether the start type is suffix or prefix, the reply start either appears at the end of the last input, or at the start of the model’s reply.
Appendix B Language models in AutoGRAMS
Every autogram has 3 language models which can share weights or use the same API. The are referred to as the chatbot, classifier, and userbot. The chatbot executes the instruction of thought and chat nodes, the classifier predicts transitions, and the userbot can be used simulate the user in a conversational autogram.
The Chatbot is the language model that generates the output text of chat and thought nodes, corresponding to chain of thought and replies to the user. The chatbot is assumed to be a chat completion style model. As arguments, it takes a list in inputs (treated as user side inputs) and outputs (treated as previous model outputs). Depending on the nodes used, some of the inputs may be instructions or combinations of instructions and user replies, instead of just user replies.
The classifier model is a language model that predicts the answers to transition questions, and is used to predict transitions. The classifier model only needs to generate a single token, and uses a large positive logit bias on the allowable answer choices (which are always multiple choice A-Z, or yes/no) to ensure the model generates a valid answer. Without a logit bias, the model would likely sometimes generate things like “The answer is C.” instead of just “C”, which would make determining the predicted answer more difficult. Unlike the chatbot, which receives previous inputs and output turns, the conversation history to the classifier is all concatenated to a single turn.
The userbot of an autogram is by default set to be the same object as the chatbot, meaning that it will share the same API or the same weights if running on the local machine. The prompts to the userbot are what distinguish its behavior, since the instructions will typically be along the lines of “reply as the user and say x”
Appendix C Interjection Nodes
“Interjection nodes” are nodes that can be reached after any chat node. They are mostly meant for scenarios where the user does or says something unexpected and it may be necessary to leave the main conversational trajectory. Interjection nodes can be any action type–so a thought node could also be an interjection node. If interjection nodes are used, an additional multiple choice question will be applied after every chat node before applying the transition question. The autogram may override the main transition behavior depending on the answer predicted to that question.
Any node can be made an interjection node by defining a field when making a node. The default interjection question is “Which of the following is True?”. The last answer, by default, is “None of the above.”. The other answer choices correspond to the set by each of the interjection nodes. These defaults can be changed in the Autogram Config. If any interjection nodes are defined, the classifier will predict the answer to the interjection question after every conversation turn. If the classifier predicts that answer is the default (“none of the above”), the autogram will proceed as normal to ask the language model the transition question. However, if the classifier predicts an answer that corresponds to a , the normal transition behavior will be overridden and the autogram will jump to the interjection node and execute that next. It is important to define the answers for interjection transitions carefully, since if the model incorrectly predicts and interjection transition it can lead to large breaks in the conversational flow.
It would be possible to obtain a similar result to interjection nodes by using additional transition nodes after every chat node. The chat node could ask the interjection question as its transition question and map the answers to the interjection nodes. An additional transition node could be added for when the answer is the default or “none of the above”, and this transition node could ask the original transition question of the chat node and map to the original transitions. Interjection nodes are essentially a shortcut to achieve this result more conveniently without needing to define an additional transition node after every chat node.
Appendix D Simulating a user in conversational autograms
We built the ability to simulate the user’s end into conversational autograms. For nodes with multiple transitions, a list of user prompts corresponding to each transition can be provided. The autogram can then be used to simulate entire conversations, using the reply method to simulate the agents end, and the simulate_user method to reply automatically on the users behalf. Simulate user samples a transition at random and then generates text using the user prompt corresponding to the sampled transition. This simulation ability has several potential usecases. Firstly, it allows a conversational autogram to quickly be tested by generating full conversations automatically that can be inspected to see if the agent is replying in the desired way. Secondly, it can act as a check to be sure the classifier language model used for transitions is predicting the correct output. If the user simulation and classifier both work correctly, then the state sampled by the user simulation should match the transition predicted by the classifier. Lastly, we envision user simulation to be useful for finetuning the underlying language models from human or AI feedback–since it allows for full trajectories to be generated without the need for human input at every step.
Appendix E Wild card transition visualization
We give the flow chart of nodes to illustrate can wildcard transitions can use if/else-if/else logic to facilitate transitions in figure 11.
![Refer to caption](extracted/5729962/figures/autograms_wildcard_transitions.png)
Appendix F Additional details of the AutoGRAMS graph compiler
The AutoGRAMS compiler translates code with python-like syntax into a representation that the AutoGRAMS interpreter can understand. It allows for Python statements, conditionals, loops, variable assignments, and function calls, to be combined with special statements that create AutoGRAMS nodes at specific points in the program. T AutoGRAMS compiler processes code down to the level of individual nodes and statements, and to form an AutoGRAMS graph around these statements. It uses the Python abstract syntax tree (AST) to parse the code, and recursively traverses the AST until it reaches code that can be represented as a single statement. It then uses this statement to define a new AutoGRAMS node. If the statement is a Python statement, the compiler will define a python-type node, whereas if it is a special statement it will be a node type defined by the arguments given to
The high level steps implemented by the AutoGRAMS compiler are:
-
1.
Convert code to Python AST representation
-
2.
Extract the name of every function definition in the code and the subtree of the AST associated with that function
-
3.
Recursively traverse the subtree of the AST for each function to get the AutoGRAMS graph for that function
-
4.
Recursively traverse the AST for code that is outside of a function to get the AutoGRAMS graph of the outer scope/main program
Any nodes that aren’t explicitly named by exec_node statements need to be named automatically by the AutoGRAMS compiler. Node names need to be unique, so the compiler assigns names with prefixes based on functions and statement bodies that they reside in, and using a numbering system that increments for each statement type and node within each prefix. All automatically named names start with the’_’ character to reduce the risk of conflicts between nodes named by the designer and nodes named automatically by the compiler.
As we define the operations performed during AST traversal, we will use “AST node” to refer to a node in the AST tree and “AutoGRAMS node” to refer to e anode in the AutoGRAMS graph. The AutoGRAMS compile recursively traverses the AST tree, starting at the root AST node, and recursively traversing the subtree of the children AST nodes.
When encountering an AST node representing a conditional (if/else-if/else), the compiler creates a new independent chain object for each branch to recursively processes each branch of the conditional. Each chain returns with the AutoGRAMS nodes needed to implement that branch. The subgraphs in each branch are then combined by defining new AutoGRAMS transition nodes at the root of the conditional, and at the start of each branch of the conditional. The AutoGRAMS node at the root of the conditional is given an automatically defined wildcard style transition, and the AutoGRAMS nodes at the start of each branch are given names that match the name defined in the wildcard style transition, and a alphabet character suffix corresponding to their order in the branch logic. These AutoGRAMS nodes are also defined with boolean conditions that match the boolean condition defined in the Python conditional statement. This combination of a wild card transition, node names, and boolean conditions allows them to implement the exact transition logic of the if/else statement defined in Python.
When encountering an AST node corresponding to a while loop, the compiler first collects the condition or the while loop. It then creates a new independent chain object that recursively processes the body of the while loop to obtain the AutoGRAMS nodes in the body.The compiler can then follow the process to design conditionals described above, and can connect the final AutoGRAMS node in the body of the while loop with root AutoGRAMS node of the conditional, allowing the loop of the graph to be repeatedly traversed until the exit condition is met.
When encountering an AST node corresponding to a while loop, the compiler can execute a similar process to a while loop, while also creating an AutoGrams node to make node before the loop to define an iterator variable, as well as AutoGrams nodes in the forloop that extracts elements from the iterable object defined in the forloop, and increment the iterator variable. The forloop is given an exit condition that causes it to exit the graph when the iterator variable is equal to the length of the iterable object.
When encountering AST nodes corresponding to single Python statements, the autograms compiler adds a python-type node to cause that statement to be executed at that point in the graph. If the statement contains a varaible assignment, this is also added to the instruction off the corresponding AutoGRAMS node.
When encountering statements, the autograms compiler defines a new node with the arguments given in the statement. If the exec node statement contains a pythnon style variable assignment, this assignment is added to the node’s instruction string. If the statement contains no defined transitions, it is assumed that it will transition to the next node encountered in the code. It is also possible to connect an statement with nodes defined by other statements, however jumping into a loop or function from an external point will lead to errors.
Appendix G The reply algorithm
We present high-level pseudo-code to show the operations performed by the AutoGRAMS reply method to get a conversational response and updated memory from an autogram.
When only using chat style nodes with only standard transitions, the algorithm for replying to a user becomes simplified, allowing it to be more fully explained in a single block of pseudo-code. This pseudo-code assumes that it is not the first step in the conversation (if the model goes first, it will reply from a predetermined node).
Appendix H A simple self-referential autogram
We demonstrate a very simple proof-of-concept of a self-referential autogram below. This autogram creates a new node at every conversation turn to handle the user’s reply. the autogram was implemented in AutoGRAMS compiled from Python and automatically compiled into AutoGRAMS nodes. The fields for the nodes that make up the autogram are given below.
The first node sets a variable to hold an instruction we will use later.
name: "_node1" action: "python_function" transitions: [’intro’] instruction: "meta_inst = \"We need to decide how to deal with the last user reply. Rather than replying directly, write an instruction for another language model to reply.The instruction could be along the lines of ’respond to the user and tell them xyz’\""
the next node initiates a conversation turn with the user with the exact text ”Hi there, What can i help you with?”.
name: "intro" action: "chat_exact" instruction: "Hi there, What can i help you with?" transitions: [’dynamic_node’]
The next node calls a function that will do the majority of the heavy lifting. The function add_new_node() will add a new node to the autogram. add_new_node() also returns the string of the next node, which will be saved in the variable next_node. Since transitions are executed after instructions, the transition $next_node will cause the autogram to transition to the node defined by the function.
name: "dynamic_node" action: "function" transitions: [’$next_node’] instruction: "next_node=add_new_node(meta_inst)"
The next node defines a callable node that starts the add_new_node function, with the name field defining the function name and arguments.
name: add_new_node(meta_inst) action: transition transitions: [’_add_new_node_node2’]
The next node is a thought node that uses the language model to generate an instruction for the new node are are designing. It conditions on the variable meta_inst as a prompt. The resulting generated text is saved in a new variable called new_inst.
name: _add_new_node_node2 action: thought instruction: new_inst=$meta_inst transitions: [’_add_new_node_node3’]
The next node is a thought node that uses the language model to name the new node being created. It conditions on the new instruction generated at the previous node, which is embeded in the prompt. The result is saved in a variable called new_node
name: _add_new_node_node3 action: thought instruction: new_name=We need to come up with a short name for a node that executes this instruction: $new_inst The name must be all lower case letters and underscore (’_’). Reply with the name only with no spaces transitions: [’_add_new_node_whileloop1_start’]
the next node is an automatically generated node to facilitate the start of a while loop by applying a wildcard transition to check the exit conditions, which will be set in the boolean_condition attributes of 2 later nodes
name: _add_new_node_whileloop1_start action: transition transitions: [’_add_new_node_whileloop1.*’]
The first condition referenced by the wildcard transition. This will enter the loop if the node name we have defined not valid.
name: _add_new_node_whileloop1.a action: transition transitions: [’_add_new_node_whileloop1_node1’] boolean_condition: not meta_utils.check_node_name(new_name)
This node regenerates node name if node name from before was invalid.
name: _add_new_node_whileloop1_node1 action: thought instruction: new_name=We need to come up with a short name for a node that executes this instruction: $new_inst The name must be all lower case letters and underscore (’_’). The last name was invalid, please be sure to reply with the name only and obey the rules about characters. transitions: [’_add_new_node_whileloop1.*’]
this node is to exit the while loop when the boolean condition set in _add_new_node_whileloop1.a is False.
name: _add_new_node_whileloop1.b action: transition transitions: [’_add_new_node_node8’]
This node adds the node that we just defined the arguments for to the autogram interpreter object that is executing the graph, which is referenced using self. We set the action of this node to chat, since we will use it to give conversational replies. Notice that it transitions back to dynamic_node which we defined earlier. This ensures that the autogram will continue–after the generated node is executed, the autogram will transition back to dynamic_node and design another node to handle the new conversation turn.
name: _add_new_node_node8 action: python_function instruction: self.add_node(action=’chat’, instruction=new_inst, name=new_name, transitions=[’dynamic_node’]) transitions: [’_add_new_node_node9’]
Lastly, a node with a return transition returns the new_name variable. This returns the graph to dynamic_node, which called the function.
name: _add_new_node_node9 action: transition transitions: [’return new_name’]
After the function returns, dynamic_node will then transition to the new node we just defined (since it’s name is saved in the new_name variable and dynamic_node uses $new_name as its only transition. The new node will execute to give a conversational reply, and then transition back to dynamic_node, which will call the function again to design another node.
![Refer to caption](extracted/5729962/figures/self_ref_light.png)