AutoGRAMS: Autonomous Graphical Agent Modeling Software

Artificial intelligence (AI) agents driven by large language models (LLMs) can tackle complex, multi-faceted tasks. LLMs pretrained on large datasets generate human-like text and display knowledge of various domains Radford et al. (2019); Brown et al. (2020). This enables them to perform a range of tasks and learn from training examples given in their prompt.

Training these language models to follow instructions enhances their utility, allowing them to generate outputs that align with given instructions Ouyang et al. (2022). This instruction-following behavior is crucial for the development of AI agents that provide useful and contextually relevant responses across a variety of applications (Yao et al., 2022; Paranjape et al., 2023; Khattab et al., 2022). It also enables these agents to drive autonomous operations, such as making decisions, accessing external information, and engaging in conversations. Despite their potential, current approaches often struggle with maintaining coherent behavior over extended interactions, adapting to unexpected user inputs, and providing designers with intuitive and flexible control over the agent’s decision-making process.

To overcome these limitations, we introduce the AutoGRAMS (Autonomous Graphical Agent Modeling Software) framework to empower designers to orchestrate intricate interactions with LLMs. AutoGRAMS enables the creation of sophisticated AI agents and chatbots by representing their behavior as a series of interconnected nodes, each with distinct actions and transition rules. This graphical representation provides a programming language for agent development, where the execution path dynamically adapts based on language model predictions or predefined conditions.

With AutoGRAMS, designers gain the ability to:

• •

  Design branched multi-step interactions with a language model: Define a series of interconnected nodes that represent the steps involved in language model generation. This graph outlines a pre-defined procedure, and its nodes may include multiple branches and decision points. Interaction is defined by graph traversal, by iteratively stepping through the nodes of the graph.

• •

  Design complex conversational flows: Define a graph of conversational reply steps with prompts that govern how the agent gives conversational replies and makes decisions about which conversational branches to take.

• •

  Control interaction steps with with node-specific prompts: Craft prompts tailored to each node, guiding the language model’s responses.

• •

  Leverage language model predictions in transitions: Define how language model predictions are used to navigate branch points and guide the agent’s behavior.

• •

  Incorporate conditional logic: Define specific conditions under which different actions or transitions occur.

• •

  Use variables to control memory: Define how variables in memory are set and used by a language model

• •

  Integrate code: Define Python statements to manipulate variables or interact with external APIs at any node of the interaction

• •

  Call agent modules as functions: Define callable agent systems that can be called at points in the interaction, allowing an interaction to return to a previous point, and giving additional control over prompt scopes.

• •

  Visualize agent behavior: Gain a clear understanding of the agent’s decision-making process and current state through an intuitive graphical interface.

• •

  Design agent interactions in spreadsheets: Enable the design of complex agents using spreadsheet software such as Microsoft Excel, Google Sheets, etc.

The designer does not necessarily need to be a person–the high level of flexibility of AutoGRAMS makes it possible for an AutoGRAMS agent to design other AutoGRAMS agents. AutoGRAMS also contains functionality to allow agents to modify their own graph directly, opening up possibilities for agents that learn by modifying their own behavior.

By bridging the gap between intuitive design and complex agent behavior, AutoGRAMS opens up new possibilities for developing AI agents that can effectively navigate diverse scenarios or tasks, adapt to user input, and maintain coherent, goal-oriented conversations. Specifically, by enabling designers to define the behavior of individual nodes and transitions flexibly and easily within a graphical framework, AutoGRAMS facilitates precise control over multi-step interactions with language models. This versatile tool serves as both a high-level programming language and a comprehensive framework for developing language model-based agents and chatbots.

The layout of this paper is as follows: We cover the AutoGRAMS framework in Section 2. We demonstrate the mapping from python-like code and AutoGRAMS, along with the AutoGRAMS compiler–in Section 3. We introduce how AutoGRAMS agents are executed in Section 4. And finally, we discuss self-modifying agents and agents that can design agents in Section 5.

We introduce AutoGRAMS and go over the process of defining a set of nodes and associated fields to create agents in Subsection 2.1. We then delve into the key concepts and mechanics of the AutoGRAMS frameworks, focusing on; nodes and node types in Subsection 2.2, transitions in Subsection 2.3, variables and memory in Subsection 2.4, functions in Subsection 2.5, methods of designing autograms in Subsection 2.6, and finally, autogram configurations and settings in Subsection 2.7. Some other details of the framework are included in the appendix, including how prompts are formed (Appendix A), and details of how language models are used (Appendix B), a description of interjection nodes that help model unexpected user replies (Appendix C), and simulating user replies in conversational autograms (Appendix D),

AutoGRAMS gives the ability to embed general code within graphical AI agents and chatbots. One way to design autograms is to define nodes one-by-one, with transitions to other nodes specified in node definitions. This is especially useful for chatbots, since conversations often have certain states that can be modeled as a graph or tree. AutoGRAMS also allows nodes in this graph to execute code, rather than give a reply. However, this requires the code that executes at a node to be passed as a string in a node’s instruction. This graphical representation also means that loops and conditionals in the AutoGRAMS graph need to be defined using the appropriate transitions. We were motivated to allow deeper integration between general code statements and the graphical representations used by autograms. To do this, we implemented the AutoGRAMS compiler, which can map a combination of Python code and statements that define AutoGRAMS nodes, to an autogram that can be executed by the AutoGRAMS interpreter (Section 4). It does this by mapping the code in a file to a set of nodes with the appropriate attributes to perform the computation specified in the code file.

AutoGRAMS is general enough to execute functional programs that can be represented common features such as loops, conditionals, functions and variables. This is possible by combining python-type nodes with the desired flowchart needed to execute a program. For instance, consider a program that computes the nth element Fibonacci sequence recursively (While this is inefficient, we use it for illustrative purposes). Python code for this is given in Figure 7(a).

The function $fibonacci(n)$ accepts a number, returns 0 of the number is 1, returns 1 if the number is 2, and returns the sum of $fibonacci(n-1)$ and $fibonacci(n-2)$ if the number is greater than 2. this function can be represented completely in AutoGRAMS with a combination of python-type nodes, function-calling nodes, transition nodes, return transitions, and wild card transitions. The first node of the graph must be a callable node named “fibonacci(n)”. The parentheses in the name means that it can be called function calling nodes, and using n means that the first argument passed can be referenced with the variable name “n”. A node with an action $local\_function$ and instruction of $x=fibonacci(n)$ can be used to call this function as set the result to a variable called “x”.

The first conditional of the program is implemented with a wildcard transition, where the root node of the conditional applies a transition to $fibonacci\_conditional.*$, which means that the next node will either be $fibonacci\_conditional.a$, $fibonacci\_conditional.b$, etc. In this specific case, since there are 4 possible branches, it is necessary to define nodes named $fibonacci\_conditional.c$ and $fibonacci\_conditional.d$. Each of these nodes has a $boolean\_condition$ attribute that can be used to specify the condition at which this branch can be executed. Boolean conditions are passed to the Python interpreter in AutoGRAMS so they can include any Python code or references to any variables visible at the current scope. $fibonacci\_conditional.d$ does not need a boolean conditional because it is the last node in alphabetical order. If the $n==1$ oder $n==2$ condition are reached, the appropriate value can be returned by using a python-type that simply includes the desired number, and then applying a return transition immediately after, which returns the result of the previous node if no return variable is specified. If the else condition is met reaching the node $fibonacci\_conditional.d$, the autogram can use two $local\_function$ nodes that call the Fibonacci graph again with $n-1$ and $n-2$, sum the result, and return the result. The AutoGRAMS graph, which was compiled automatically from the Python code using the AutoGRAMS compiler, is given in Figure 7(b).

The AutoGRAMS compiler is used to convert code with python-like syntax into a set of AutoGRAMS nodes that can be executed as an autogram. It is also possible to code any autogram in pure Python (or a spreadsheet) by defining nodes one by one, in which case the AutoGRAMS compiler is not used. However, for autograms that contain Python statements or programmatic features like loops and conditionals, the compiled version has several advantages:

• •

  Python statements can be included directly as code instead of as strings in the instruction of python_function nodes

• •

  Python style variable assignments can be be used instead of assignment variable in an instruction

• •

  AutoGRAMS function calls and functions can be defined using Python-like syntax. Note that functions are by default treated as local AutoGRAMS functions, but this behavior can be changed using the function decorator ‘@global_function” or “@function” above the function definition.

• •

  the order of node chains to be inferred automatically from the order they appear in code, even if names and transitions aren’t defined.

• •

  Python loops and if/else conditionals to compile into a graph automatically

Compared with standard Python programs, AutoGRAMS compiled from Python allows for exec_node() statements to define AutoGRAMS nodes at any point in the code. These nodes can connect with other nodes defined by exec_node() by specifying those nodes in the transitions argument, in which case these transitions will be governed by multiple choice questions asked to the language model. Chat type nodes defined in exec_node() temporarily return a serializable object defining the entire state of the autogram (see Section 4.2), allowing a conversation to continue from a specific place in the code after receiving a new user reply. The graphical representations used by AutoGRAMS also give it certain advantages for self-modifying code, as discussed in Section 5.

The AutoGRAMS compiler recursively traverses the Python abstract syntax tree representation of the code to add AutoGRAMS nodes to the graph. A overview of how the Autograms compiler converts code into AutoGRAMS nodes is given in Appendix F.

Autograms are executed via an iterative process that selects nodes and executes them. Within the AutoGRAMS framework, an autogram can be applied using the reply method, which is for conversational modules, and the apply_fn method, which allows an AutoGRAMS function to be called directly from Python, and it meant for non-conversational modules. Both work in a very similar fashion, other than the way they terminate their main loop. The reply method terminates and returns a result when it encounters a chat node, and is designed to restart from the node it left off at when it receives another user reply. The apply_fn method terminates when it encounters a return statement–meaning the AutoGRAMS function is done executing and is returning a result. An illustration of the outer loop of the AutoGRAMS reply method is given in Figure 8.

The development of AutoGRAMS is situated within a broader landscape of research encompassing natural language processing (NLP), AI agents, and programming languages for AI. This section reviews key advancements and their contributions to the field, highlighting how AutoGRAMS builds on and extends these innovations.

Significant progress in NLP and neural language modeling (Bengio et al., 2000) has been driven by transformer-based architectures (Vaswani et al., 2017), such as BERT (Bidirectional Encoder Representations from Transformers) Devlin et al. (2019), GPT-3 (Generative Pre-trained Transformer 3) Brown et al. (2020), and T5 (Text-To-Text Transfer Transformer) Raffel et al. (2020). These models have shown exceptional capabilities in understanding and generating human-like text, providing a robust foundation for developing advanced conversational agents. Notable conversational agents include Google’s Meena Adiwardana et al. (2020), Facebook’s BlenderBot Roller et al. (2020), and Microsoft’s DialoGPT Zhang et al. (2019), which leverage large-scale pre-training and fine-tuning on diverse datasets to perform a variety of conversational tasks. AutoGRAMS relies heavily on the ability of pretrained Transformer models to generate realistic text that can be used in conversational replies and apply reasoning steps.

The intersection of programming languages and AI has been extensively explored to facilitate the integration and development of AI models. Python, along with frameworks like TensorFlow (Abadi et al., 2016) and PyTorch (Paszke et al., 2019), has become integral to AI research, providing comprehensive libraries for model development, training, and deployment. Domain-specific languages (DSLs) like Keras for neural network modeling (Gulli & Pal, 2017) and Rasa for building conversational AI (Bocklisch et al., ) are designed to simplify AI programming and make it accessible to a broader audience. Code language models (Chen et al., 2021; Nijkamp et al., 2022; Roziere et al., 2023) have enabled AI to automatically generate code that can solve complex tasks, allowing for a deeper integration between programming languages and AI. In the AI agent space, Flows Josifoski et al. (2023) is conceptual framework that facilitates structured reasoning and collaboration among AI systems through modular, message-based interactions. DSPyKhattab et al. (2023) is a programming model that abstracts language model pipelines as text transformation graphs.Jojic et al. (2023) demonstrate the Iterations by Regimenting Self-Attention technique, which manipulates the self-attention mechanism in language models to trigger and control iterative behaviors for executing algorithmic tasks. Building on the integration of natural language with programming, the AIOS Compiler (Xu et al., 2024) is a system that leverages LLMs to interpret and execute instructions in natural language. AutoGRAMS integrates programming languages and language models by combining graphically represented agent nodes with graphically represented program nodes. computer programs essentially model flowcharts (Goldstine & von Neumann, 1947), where each node in the flow chart executes and instruction and each transition is facilitate by a branch. In AutoGRAMS, the idea of using a flow chart for programming is combined with the idea of using a flowchart to model multi-step language model interactions. The definition of explicit nodes, which can represent conversational states, and transitions between nodes make AutoGRAMS especially well suited for conversational AI applications Transition operations can optionally be governed by multiple choice questions for a language model, and instructions can optionally be governed by prompts to a language model, allowing language models to be deeply embedded in the execution of a program. The visual programming component of an autogram also gives added interpretability, as any autogram can easily be visualized as a graph.

AutoGRAMS is motivated by previous work showing that language models can reason by breaking difficult problems into smaller steps. Chain of thought prompting (Wei et al., 2022) showed that prompting LLMs to generate reasoning steps can significantly improve their problem-solving performance of large language models on complex reasoning tasks. Kojima et al. (2022) further show how a simple prompting strategy, ”Let’s think step by step,” can significantly enhance zero-shot reasoning across various complex tasks. Tree of Thoughts (Yao et al., 2024) and Graph of Thoughts framework (Besta et al., 2024) models allow for more complex dependencies of thought units. Demonstrate-Search-Predict (Khattab et al., 2022) incorporates sophisticated interaction pipelines between language models and retrieval models. ReAct (Yao et al., 2022) integrates reasoning and action generation in language models. AutoGRAMS gives the flexibility for the designer, which could itself be an autogram, to design specific reasoning steps using thought-type nodes to help the language model solve complex programs. A graph of thought type nodes in combination with multiple choice transitions allows an autogram to represent highly complex thought processes that can be used to solve problems.

Like many previous works, AutoGRAMS enables the ability to incorporate external knowledge tools into an LLM, as well allowing for the LLM to make decisions. Retrieval-Augmented Generation (RAG) models Lewis et al. (2020), allow models to access non-parametric memory systems. Toolformer (Schick et al., 2024) is a model that autonomously learns to interact with external APIs . The Automatic Reasoning and Tool-use (ART) framework Paranjape et al. (2023) automates the generation of intermediate reasoning steps and integrates external tool use. AutoGRAMS gives a large amount of flexibility in how external tools are used. Python function nodes can specify the use of an external tool if they call a Python API that applies that tool. The arguments to the Python API can be variables that were set by a language model or another Python API at a previous turn. The node-specific transitions and transition multiple choice questions of AutoGRAMS give additional control over decisions about which tools are selected for use.

AutoGRAMS relates to previous works that use LLMs to engineer prompts that control the execution of tasks.Automatic Prompt Engineer (Zhou et al., 2022) automates the generation and optimization of instructions for LLMs and systematically refines prompts through an LLM-driven search and evaluation process. Active-Prompt (Diao et al., 2023) uses uncertainty metrics within an active learning framework to optimize chain-of-thought reasoning prompts for LLMs. Promptbreeder (Fernando et al., 2023) uses self-referential self-improvement to evolve and adapt prompts for specific tasks. Likewise-the prompts in AutoGRAMS are also able to result in actions that lead to the modification of an autogram. AutoGRAMS allows any part of an autogram to be modified by the autogram itself, not just the prompts.

Previous approaches have considered meta-learning (Hochreiter et al., 2001; Finn et al., 2017), learning to learn, or meta-programming. Meta-genetic programming (Schmidhuber, 1987) relates to our motivation of developing program modifying programs. Unsupervised meta-learning (Metz et al., 2018) relates to the long term vision of of AutoGRAMS to develop agents that can learn from interactions with their environment in unsupervised ways. Shinn et al. (2024) considers agents that self-reflect on their own mistakes to learn. Self-Refine Madaan et al. (2023) leverages iterative feedback by enabling an LLM to act as both the generator and refiner. MetaGPT Hong et al. (2023) is meta-programming framework that enables LLM-based multi-agent collaborations by embedding standardized operating procedures into prompts. While AutoGRAMS itself is a programming language rather than a specific algorithm, its features such the flexibility to fully self-modify, the graphical structure, Turing completeness, and easy integration of language models for executing instructions and determining transitions, make AutoGRAMS for well-suited meta-programming applications. We demonstrate an autogram can be used to design an autogram chatbot from a prompt. AutoGRAMS gives the potential for the development of highly complex meta-agents and self-modifying agents by enabling autograms to design new AutoGRAMS graphs or modify their own graphs.

AutoGRAMS also relates to other approaches for controlling text generation. Previous work has used discriminators (Dathathri et al., 2019; Krause et al., 2020; Yang & Klein, 2021; Qin et al., 2022), special input tokens (Keskar et al., 2019), prompts (Reynolds & McDonell, 2021), instruction following (Ouyang et al., 2022; Longpre et al., 2023) or finetuning using a reward signal (Ziegler et al., 2019; Rafailov et al., 2024) to better constrain the outputs of language models for improved controllability and safety. AutoGRAMS allows a complex network of prompts to be defined to control multi-turn or multi-step interactions.

This work presented the AutoGRAMS, a novel framework and high level programming language for controlling multi-step interactions with an LLM. AutoGRAMS gives the designer the ability to create branched multi-step interactions with an LLM in a way that is analogous to how programming languages allow for the design of branched multi-step interactions with a processor. The designer can define these steps of the interaction in advance by writing prompts for a language model at each step. AutoGRAMS also allows language models to exert direct control over transitions in the interaction by enabling the designer to write multiple choice questions that determine which branch to take. This structure enables the designer to use natural language to effectively govern the flow of the of steps that an AutoGRAMS takes. The graphical structure of an autogram allows it to be implemented and visualized in an intuitive way.

AutoGRAMS is especially useful for designing complex conversational flows. AutoGRAMS allows for agents be designed to behave in more predictable ways over longer conversations, giving added control-ability and safety to conversational agents. It accomplishes this by allowing the designer to define a prompt for how to reply to the user for every possible turn in the conversation in advance. The designer has to anticipate the users response, for which The designer can then also design a series of multiple choice questions for a language model that govern which turns will actually be implemented based on that response. Additionally, if the conversational agent needs more complex intermediate reasoning steps or memory, these can be implemented in AutoGRAMS as well. The full state of the autogram can be saved in a serializable object, making it easy to reload the complete state and memory of a complex conversational agent from a database.

By allowing conditionals and code statements, and variables, AutoGRAMS allows the functionality of a Turing complete programming language to be integrated into the graphs that define the steps of the language model interaction. We demonstrate how Python programs can be mapped to an AutoGRAMS graph of python-type nodes and embedded with other types of nodes. This enables the designer to define data structures to store and access memories, which could be text generated by a language model or variables created in code. These variables can be embedded into prompts, transition questions, referenced in code, or even used to create dynamically executing code or dynamic transitions in the graph. This high level of flexibility makes it possible for AutoGRAMS to be used for very advanced applications of AI agents.

We also demonstrate mechanisms for AutoGRAMS to design other AutoGRAMS. We also demonstrate their ability for self-adaptation and modification. AutoGRAMS allows an agent to be fully self-referential with respect to the nodes that define the agent. We anticipate self-modifying AI agents to be useful for generalizing agents to more situations and allowing agents to learn from experiences and information, and AutoGRAMS provides a useful framework to build such agents.