Making New Connections: LLMs as Puzzle Generators for The New York Times’ Connections Word Game

While LLMs are a relatively recent invention, there is already burgeoning literature on applying LLMs to games, as catalogued by (Gallotta et al. 2024). This includes using LLMs to play games, generate games, generate dialogue, implement game mechanics, and more.

Several studies have investigated the application of LLMs to level generation. MarioGPT (Sudhakaran et al. 2023) fine-tunes GPT-2 to generate levels in the side-scrolling platformer Super Mario Bros., conditioned on target characteristics of the generated level such as number of jumps, coins and enemies. Similarly, Todd et al. fine-tune GPT-2 and GPT-3 to generate Sokoban levels. In both cases, the resulting models are capable of generating novel and playable levels.

These prior LLM-based level-generation works above focus on embodied grid-worlds. Though these representations can be thought of as existing at a sub-linguistic level, results seem to suggest that pre-training on vast amount of natural language data seems to nonetheless result in useful priors for this task. In this paper, we investigate generating “levels” in a game that operates instead directly at the level of language. Unlike in grid-worlds, training a model from scratch strictly on human-generated levels would be a virtually infeasible approach (due to the sparsity of level data relative to the vast semantic complexity of human language), and pre-training on a vast corpus of human knowledge is a necessary pre-requisite for generating any novel puzzles at all.

Though generation of Connections puzzles is an unexplored domain, recent research has studied solving language-based games. Jaramillo et al. implement and evaluate various AI agents for the social board game Codenames. Codenames, similar to Connections, uses semantic clustering of seemingly unrelated words as a core game mechanic. They find that a GPT-2 word embedding based bot outperforms non-vector approaches such as TF-IDF and Naive-Bayes. We continue this exploration of Language Models for word association games, utilizing the latest GPT models.

Recent work has studied LLMs in the domain of Connections. Todd et al. (2024) first study the task of solving NYT Connections puzzles, finding that GPT-4 is able to solve $38.93\%$ of puzzles using Chain of Thought prompting. They find that sentence embedding models, using a cosine similarity guessing strategy, are able to solve $11.6\%$ of puzzles, outperforming the GPT-3.5 model. Later work further explores Connections solving via LLMs, categorizing the knowledge types required to solve the puzzle and evaluating a variety of LLMs on a modified version of the solving task (Treutlein et al. 2024). The best performing model, GPT-4o, is only able to solve $8\%$ of puzzles. They note that certain “reasoning types” present in categories are significantly harder for LLMs to solve, highlighting the diversity of semantic reasoning inherent to the game.

Connections is a category-matching word game published daily by The New York Times. The game debuted on June 12, 2023, and quickly became one of their most played games. A Connections game board is represented by a grid of sixteen English words or names. The arrangement of words in the grid is semi-random²²2Some puzzles have intentionally spelled out phrases in the first row of words, though this phrase is never a word group and can be arbitrarily re-shuffled by the player. In order to submit a guess, the player selects a set of four words they believe are connected under a common theme. If the guess is correct (i.e. the four words belong to the same category) then the category is revealed and the words are removed from the grid. If three of the four words belong to the same category, this fact is indicated to the player with the message “One away!”, but neither the words nor the category are revealed. If two or fewer words belong to the same category then no additional information is provided to the player. If the player makes four incorrect guesses, the puzzle is failed.

Each category within a puzzle has an assigned color which represent the its difficulty. In order of ascending difficulty, these colors are: yellow (most straightforward), green, blue, and purple (trickiest). Straightforward categories often use the most common definition of words. An example given by the tutorial is the category FISH, containing “bass”, “flounder”, “salmon”, and “trout”. For trickier groups, the connection between words can be much more complex. Purple groups may use rhymes, homophones, anagrams, or “fill in the blank” clues such as FIRE ___(Aronow and Levine 2023).

Wyna Liu, who creates the daily game boards for Connections, has discussed some aspects of puzzle creation in articles published by The Times (Liu 2023). Connections puzzles are created in three steps: First, a puzzle creator compiles a candidate puzzle. This puzzle is given to a puzzle editor, who tests and edits the game board. Finally, a panel of testers evaluate the quality and challenge level of the puzzle.

For a puzzle to feel challenging and satisfying, Liu notes there has to be a mix of category “types”, such as wordplay categories and synonym categories. She cites three axes along which difficulty may be modulated: familiarity of words, ambiguity of their categorization, and variety of the wordplay (Liu 2023).

Generation of puzzles that match both the challenge and satisfaction of those published by The New York Times requires both creativity, abstract reasoning, and deep semantic understanding. Categories must have a range of difficulties, and incorporate abstract connections. Words must logically fit into a category, without being too obvious for the player. As a whole, the words and categories of a puzzle must play off each other, with interplay between words to trick the player. Satisfying all of these constraints makes Connections a unique challenge for modern LLM systems.

Because LLMs are trained auto-regressively to fill in or complete sequences of tokens from the training data, their output at inference time depends entirely on the “prompt” which they are provided—i.e. the piece of text which they are effectively asked to complete.

As a result, LLM performance on downstream tasks can be strongly influenced by the technique used to generate the prompt.

A vast amount of research has studied how to most effectively prompt language models (Schulhoff et al. 2024). In Chain-of-Thought (Wei et al. 2022), for example, an LLM is asked to show its reasoning step by step. Results suggest that by mimicking a more deliberate and thorough thought process, the LLM is less likely to jump to incorrect conclusions, and more inclined toward coherent reasoning. In its simplest form, Chain-of-Thought amounts to appending the phrase “let’s think step-by-step” to the prompt.

In the Tree-of-Thought prompting approach (ToT) (Yao et al. 2023), tasks are explicitly broken down into sub-tasks by the human user. Each sub-task is considered as a level in a tree, and the LLM is queried multiple times (while varying random seed or other hyperparameters) to produce multiple child nodes at each level. Then, some evaluation metric is calculated for each leaf-node in the tree, and tree nodes are expanded greedily, prioritising higher-scoring nodes. ToT has been shown to improve performance on challenging tasks such as the Game of 24 (Yao et al. 2023) and Sodoku (Long 2023), where simpler prompting techniques such as Chain of Thought (Wei et al. 2022) fail. In these tasks, there are verifiable solutions to the problem, allowing for the explicit scoring and ranking of candidate solutions generated by an LLM. In more open-ended tasks, where success is much more difficult to define, there is no simple way to measure the quality of an LLM generator’s output.

One such task is the creative writing task explored by Yao et al. (2023). They generate 4 random sentences, and task the LLM with generating a four paragraph passage where each paragraph ends with one of the random sentences. While there is a simple constraint that can be checked (whether each paragraph ends with the sentence provided), evaluating the quality of the writing as a whole is much more complex. Separate LLM agents can be used to evaluate and rate these outputs, but existing research shows they may be biased towards LLM-generated text (Koo et al. 2023).

Unlike their Crossword puzzle (Aronow 2021), The New York Times does not publish rules or guidelines for creating a Connections game board. We start by identifying constraints that define valid Connections puzzles. By looking at existing puzzles and the in-game tutorial (Fig 1), we infer some basic constraints that a Connections puzzle must satisfy:

1. 1.

   The puzzle contains 16 unique words.

2. 2.

   The puzzle has 4 unique groups of 4 words.

3. 3.

   All words in a group share a connection.

4. 4.

   Each word is used exactly once.

5. 5.

   The puzzle has exactly one solution.

6. 6.

   Categories are more specific than 5-LETTER WORDS, NAMES, or VERBS.

This basic set of constraints defines a large puzzle space, containing all possible combinations of 4 valid word groups. The vast majority of these puzzles are trivially easy to solve—players will notice the obvious semantic clustering of words in the puzzle, eliminating the satisfaction of playing the game. In order to generate puzzles that are challenging and fun to play, we identify general principles that are followed by the Times puzzle creators. We incorporate these principles into our prompts to guide puzzle creation towards the style of puzzle published by the Times. These principles include:

Varied Categories:

Categories within a given puzzle must be thematically distinct (i.e. a COLORS category will not share a puzzle with a PRIMARY COLORS category)

Unique Names:

Words in the category name can’t be appear as part of that category (i.e. the word GREEN will not appear in the category SHADES OF GREEN)

Spelling Matters:

Words must belong to their group in the given spelling (i.e. BE will not appear in the category INSECT, but BEE might).

Making engaging puzzles requires a certain level of difficulty. One aspect of this is explicitly stated in the tutorial of the game: “Watch out for words that seem to belong to multiple categories!” (see Figure 1). From this, we deduce that a “good” Connections puzzle must entice the player to make mistakes by using “misleading” words.

We identify two distinct ways this occurs in official Connections puzzles. The first is using “overlap words”—words that seem to belong to multiple categories. An example of this can be seen in Figure 2. Words like “fudge” and “nuts” seem like they could belong in the PHOOEY! category, but actually belong to SUNDAE TOPPINGS. The presence of these words forces the player to shift focus and consider multiple potential groups at once in order to disambiguate the group membership of individual words. This is cited as a key feature that is sought out by the Times’ puzzle editors: “More overlap makes a harder puzzle.” (Aronow and Levine 2023)

The other form of difficulty is what we name “false groups”. These are connections between words that a player may spot, but in actuality are not related to any of the puzzle’s true groups. An example of a false connection in Figure 2 could be “rats” and “woodchuck” both being rodents, or “cherry” and “peppers” both being red foods. Whether a puzzle appears to present a false group is ultimately as much a function of the player as it is of the puzzle itself. Nevertheless, even if these dead ends don’t actually elicit an incorrect guess from the player, they arguably make the process of solving the puzzle more engaging by adding richness to the puzzle’s semantic landscape.

We select overlap words and false groups as the main avenues by which to incorporate challenge into our generated puzzles.

We decompose the task of Connections puzzle generation into an iterative process: each generative step equals the creation a word group, composed of a category name along with a pool of words that belong to it. Although the puzzle could be decomposed further, with every step providing a single word or category name, this approach would become quite computationally expensive using current LLMs. We set the pool size to be eight words. We find that this results in GPT suggesting multiple valid words of varying difficulty, without suggesting illogical words in order to meet the quota. We use gpt-4-1106-preview via OpenAI’s API as our word group creator, as well as all other LLMs in the pipeline.

We use a few-shot “propose prompt” approach for word group generation, inspired by the prompts used in prior work for the Tree of Thoughts Crossword Solving task (Yao et al. 2023). This approach prompts an LLM to generate one or many word groups in a single context, which are then parsed out to become individual “nodes”. In some puzzle types, such as the “intentional overlap” puzzles, we generate two distinct categories and word pools in each prompt. In others, only one category is taken at each step. .

One issue facing GPT models is limited variety in output. In early experiments, our group-generation LLM would frequently propose the same limited set of categories and words, such as BOARD GAMES: “chess”, “checkers”, “monopoly”, “life”. Varying seed and temperature did little to curb this behavior.

We solve this problem by injecting semi-random text into the first group-generation prompt. We add a task to the LLM’s system prompt: first, write a short story using user-provided words, then use that story as inspiration for creating your categories. In the user prompt, we inject four randomly selected words from published Connections puzzles. We find this prompting technique leads to much higher diversity of output, with very few repeated categories across our set of generated puzzles.

These proposed categories still lacked the challenging and abstract categories present in real Connections puzzles, as the LLM has no knowledge of the Connections puzzle itself³³3Per OpenAI, gpt-4-1106-preview has a knowledge cut-off of April 2023, before the first Connections puzzle was released. To solve this, we analyzed published Connections puzzles and identified common category “styles”, and collect a set of example category names. Some example styles are “Synonyms or Slang”, “Wordplay”, and “Fill in the blank” categories. We add these category styles, along with a brief description and three example category names, into all proposal prompts. The LLM is instructed to only craft categories that fit one of the given styles. This leads to more complex proposed categories, such as WORDS THAT CAN FOLLOW “FIRE”.

Due to the stochastic nature of LLMs and their sensitivity to prompts, not all final puzzles are perfect. A common failure mode we observe is when there is a clear connection between words in a group, but the category name misrepresents that connection. An example is the generated group HAWK THE WARES: “wares”, “items”, “goods”, “merchandise”. HAWK THE WARES does not accurately describe how these words are connected, but THINGS FOR SALE would. For Connections puzzles, a small mistake like this in a word or category name can ruin the solving experience.

To increase the efficiency of our method, we use a “Puzzle Editor” prompt. First, it identifies the connecting theme between each word group in a complete puzzle. Then, it evaluates whether the existing category name is accurate. If not, it rewrites the category name so that it truthfully represents the connection. This often fixes otherwise invalid word groups, and rewritten category names are often much closer in style to real Connections categories.

The final component in our generation pipeline is color-coded difficulty ranking. While we assign colors based on the similarity metric generated via our embedding model, we consider puzzles where two groups are the same “color” during evaluation. Additionally, this method of assigning colors does not consider the relative difficulty of other groups in the puzzle. To create a puzzle with all four colors, we pass generated puzzles to a final LLM using a “difficulty ranking” prompt. This LLM assigns colors (yellow, green, blue, purple) to the groups of each puzzle based on the LLM’s perception of their difficulty.

To evaluate the appeal of our AI-generated Connection puzzles, we conduct a user study to evaluate human preferences between AI generated and real Connections puzzles.

We create four sets of AI generated puzzles:

1. 1.

   A baseline “one-step” LLM generated set

2. 2.

   An “intentional overlap” LLM generated set

3. 3.

   A “false connection” LLM generated set

4. 4.

   A “false connection” LLM generated set, where the “false group” is taken from a real Connections puzzle

We implement a web-based replica of the Connections interface, following the same rules used by the New York Times. Players are allowed four incorrect guesses before failing the puzzle. They are able to randomly shuffle the game board, and are told when their guess is “one away” from a correct answer. The puzzles are integrated into a survey page, where users are asked to play two Connections puzzles and answer a series of questions. If the user does not successfully solve the puzzle, all categories are revealed so the user can evaluate the complete puzzles.

Pairs of Connections puzzles, one AI-generated and one from the Times, are randomly selected at the start of each survey. The ordering in which the two puzzles is randomized.

We collect gameplay from the user’s play sessions on both puzzles, including all guesses made and whether or not they completed the puzzle. After completing both puzzles, the user is asked the following questions:

1. 1.

   What is your English Proficiency?

2. 2.

   How often do you play Connections?

3. 3.

   Have you seen either puzzle before?

4. 4.

   Which puzzle was more creative?

5. 5.

   Which puzzle was harder?

6. 6.

   Which puzzle did you like more?

For questions 4-6, users can select “Puzzle 1”, “Puzzle 2”, or “Tie/Neither”. Optional free response questions are included for questions 4-6, allowing users to explain their reasoning. A final free response is included for miscellaneous comments.

To avoid potential bias in user responses, we do not explicitly state which puzzle is from which source, nor that the puzzles are AI generated. Additionally, we ask users to self-report if they have seen either puzzle before. We do not include responses where a user recognized a puzzle in our analysis.

Figure 4 shows user preferences on AI generated puzzles versus real New York Times puzzles. We compare the percentage of user responses to questions 4-6 across all AI-generated puzzles, as well as by sub-type. In roughly half of head-to-head comparisons against expert Times puzzles, generated Connections puzzles were judged to be equally or more enjoyable, creative, and difficult.

One-step puzzles were largely rated as being less enjoyable and less creative than their Times counterparts, but performed better in terms of difficulty. Our best performing group in terms of overall enjoyment (as indicated by responses to question 6) was LLM false group puzzles, beating Times puzzles in $42.86\%$ of responses and tying in $14.29\%$. Users found intentional overlap puzzles the hardest, rating them more difficult than their NYT counterparts in $60\%$ of responses and equally difficult in $20\%$ of responses. This difficulty is reflected in playthrough data, where intentional overlap puzzles have the lowest overall solve rate. In terms of creativity (question 4), intentional overlap puzzles perform best, though LLM-generated false group puzzles are strictly preferred more often.

In general, difficulty is the category where generated puzzles appear to come closest to the Times.

We analyzed users’ play data to determine whether our attempts to add interplay via false groups worked as we expected. For each false group puzzle, we calculated how often players made a false group guess: selecting any subset of the false group (2, 3, or 4 words). We found significant differences between the LLM- and NYT-seeded versions. For all LLM-seeded false group puzzles, players guessed a subset of the false group in half or more of the total play sessions. In one puzzle, every single attempt included one of these false group guesses.

For seeded false group puzzles, players weren’t tricked as easily. While 2 puzzles elicited false group guesses, 3 puzzles had no false group guesses.

Players often used the free response section of the survey to provide their opinions on puzzle difficulty. Users were surprisingly perceptive in identifying the manner in which difficulty was introduced into generated puzzles. One user, playing an intentional overlap puzzle, writes “There was a bit more overlap in words to categories.” Some users cite the difficulty as a reason they enjoyed the puzzled more: “Too many word overlaps in the second puzzle, but it was more interesting and accurate to their categories”.

Users were split on whether this ambiguity in groupings was a positive or negative. Six users specifically call out ambiguity and overlap in the AI puzzle as a negative, while seven cite it as a positive and reason for preferring the puzzle overall.

Additionally, despite not disclosing that AI was used as a source for any puzzles, multiple users wrote that they felt a puzzle was AI generated, though they weren’t always accurate in their selection.

Seeded false groups stand as an outlier in our data—they are by far the easiest to solve. We hypothesize that this is due to the “difficulty” of the false groups. In particular, seeded false groups were drawn randomly from NYT puzzles, and therefore sometimes involved obscure connections. As a result, these groups were less likely to “jump out” at and distract players, whereas LLM-generated false groups were liable to be more generic, eye-catching, and therefore distracting, leading to more false group guesses.

We further note that LLM false groups were selected as the most similar set words in the candidate pool as measured by cosine similarity. The average similarity score for NYT-seeded false groups in our user study is indeed lower ($0.47$) than the average score for LLM-generated false groups ($0.52$). Further, two seeded groups have high similarity scores ($0.51$ and $0.60$), while the other three have much lower similarity ($<0.43$). We note that the two groups with high similarity elicited false group guesses from users, and the three with low similarity elicited none.

Difficulty is the category in which our generated puzzles perform best against NYT puzzles. Intentional overlap puzzles were rated as more difficult in the majority of cases, similar to LLM false groups. This is likely due to the way we intentionally introduce the overlap or false connection difficulty into every group in the puzzles. While NYT puzzles may incorporate just a few overlapping words, or small false connection groups, we try to maximize these types of difficulty in their respective puzzle types. This may lead to difficulty that exceeds the average NYT puzzle. Decreasing this difficulty, by removing the difficulty incentives from generated group prompts, may help align the difficulty with what players have come to expect from Connections.

We intentionally limit the number of questions in the survey to reduce user fatigue. One notable metric that we do not include in our survey relates to puzzle “fairness”. A puzzle that uses uncommon vocabulary or illogical groupings may be much harder for players, but not particularly fair in its difficulty. Although we use the Puzzle Editor LLM to try to combat these issues, it is not perfect at correcting puzzles.

Generating convincing Connections puzzle is a uniquely difficult domain. To generate a puzzle on-par with the original, a method must be capable of proposing unique and clever categories, on the same level as professional puzzle builders. Connections is published by a single source—the Times—and players have come to expect a distinct level of quality and challenge from their daily puzzles. Group themes can’t be too obvious, but they also can’t be “reaches”. This necessitates an advanced semantic understanding, combined with a sufficient model of human logic.

On top of this, user feedback shows the importance of the unspoken yet understood “principles” underlying Connections. While some constraints are easily verifiable (Unique Names), others are harder to measure (Varied Categories). When even one of these principles is broken, players feel frustrated and cheated rather than accomplished.

Despite these challenges, a direct comparison to The New York Times doesn’t tell the whole story. Though many generated puzzles may not beat real puzzles in direct comparison, users still enjoy solving them, and in some cases prefer them. Though small flaws are common, we find some surprising ability for cleverness and wordplay by LLMs. With further refinement of prompts, particularly in distilling the abstract concepts that make Connections so widely enjoyed, we believe this method could produce more consistent results.

While the perhaps most obvious use case for the method we are developing here is to actually generate Connections puzzles, one can also see this work as a contribution to game design research. The implemented generator can be seen as a test of a theory of how Connections puzzles are designed. The implemented theory is based on Wyna Liu’s article describing her own puzzle creation methodology, and the success of the method in creating good Connections puzzles can be seen as corroboration of that theory. One might then iteratively refine the theory and the corresponding process to arrive at an even better theory of what makes for a good Connections puzzle.

Our ability to perform ablation studies of various prompting techniques, as well as generate large quantities of puzzles was limited by computational budget. The GPT-4 API is significantly slower and more expensive than other models, such as gpt-3.5-turbo or open source LLMs such as Llama 3 or Mistral. Open-source LLMs can be run locally without cost, enabling more in-depth state evaluation, thought generation, and editing steps that are cost-prohibitive with paid APIs. However, preliminary experiments indicate they are not as capable at following the complicated system prompts we author for GPT-4.

We rely on user’s self-reporting whether or not they have seen a particular Connections puzzle before. This may not be 100% accurate, leading to some bias if users were able to identify previous NYT puzzles. Our analysis and confidence in our conclusions are limited by the number of responses, as well as our methodology. Connections puzzles are inherently challenging to parse, forcing players to process a scrambled grid of seemingly unrelated words. Comparing two puzzles side-by-side may not be an optimal way of measuring a puzzle’s difficulty.