Six Language Models Authoring Logic Puzzles Through a Game's Production Gates

1. Introduction

GRIDIGMA is a logic-grid deduction game played mostly on phones, published in English and Turkish. Its puzzles are authored by a loop in which a language model drafts the logic, the clues and the story, and the game's gates check the draft before it can be published. A companion paper [1] asked which language models best write and translate the text of a finished puzzle. This paper asks the step before it: which model best authors the puzzle itself, a grid whose clues admit exactly one solution at the requested difficulty, and the story that sets them up.

Authoring is judged differently from prose. A draft passes the gates or it does not; a passing puzzle can be easy or demanding; and the reasoning it takes varies widely between models. We asked three questions, in an order fixed with the product owner before the run:

1. RQ1. Which models best author a puzzle's logic, passing the game's own gates at the requested difficulty?
2. RQ2. How hard are the puzzles that pass?
3. RQ3. What does each model spend on them, in model time, calls and tokens?

A blind read of theme, story, clues and appeal is reported alongside and enters none of the three. Section 2 sets out the game, the requests, the loop and the measures; Section 3 gives the results; Sections 4, 5 and 6 discuss them, state their limits and conclude.

2. Method

2.1 The game and its puzzles

GRIDIGMA is a logic-grid deduction game played mostly on phones, published in English and Turkish. Each puzzle has a title, a one-line description, a short story and a set of clues, and the clues together admit exactly one solution. The game sorts its puzzles into five difficulty tiers, from very easy to very hard, and also publishes them in themed packs. New puzzles reach the game through an authoring loop in which a model drafts it and the game's gates check the draft (Section 2.3). The companion paper [1] compared language models as writers and translators of the text of finished puzzles; this study needs nothing from it beyond what is set out below.

2.2 Design

Six models took part (Table 1): Grok 4.6, Grok 4.7, GPT-6 Sol, GPT-6 Astra, Fable 5.1 and Opus 5.5. The requests were July's two themed packs, one on the Wimbledon tennis championship and one on World Heritage sites, one puzzle per difficulty tier per pack (very easy, easy, medium, hard, very hard): ten requests, identical for every model. Each request fixed the setting, a cast of four (English names in the Wimbledon pack, Korean names in the World Heritage pack, the same in every language), the labels of the grid and the tier; the model wrote the logic, the clues and the story. The instructions asked for plain everyday words, no clue told as arithmetic, no real person, organisation or named place, and encouraged the clue types the game's catalogue uses least (Section 5).

2.3 The production loop

Every request ran through GRIDIGMA's own authoring loop, the one that makes the game's puzzles, with one route added to reach each model. Every call received exactly the prompt the production client builds: the tier brief, the state of the catalogue, the gate contract, the craft guide, the clue-type ceilings, the setting and the place. A draft that failed a gate went back to the same model with the gates' findings, up to three attempts per request (a first draft and two revisions), with no escalation to another model. The gates were those every published puzzle passes: exactly one solution, no clue that can be removed without losing it, the clue-quality checks, and a computed difficulty tier equal to the requested one. The tier is computed from a simulated player's solving time per unit of deduction, so a model cannot reach it by counting clues.

2.4 Settings and interruptions

Settings. Every call ran at reasoning effort high, set and recorded per call with the requested and the reported model identifier, the time and the token usage; the two identifiers agreed on all 138 answered calls. Each call had a 30-minute cap. The Grok models ran through the xAI interface, the GPT-6 models through OpenAI's Codex client on a Plus subscription, and Fable 5.1 and Opus 5.5 through Anthropic's Claude command-line client on a subscription.

Interruptions. The subscriptions' usage limits paused GPT-6 Sol and GPT-6 Astra (Codex) and Fable 5.1 and Opus 5.5 (Claude). The caller waited for each reset; a request whose calls had failed was restarted from scratch (GPT-6 Astra 1, Fable 5.1 7, Opus 5.5 7 requests), and the failed runs' records are kept. Model minutes are the time from sending a call to its answer, summed over the answered calls, with every limit wait removed; failed calls are not counted (one Fable 5.1 call reached the 30-minute cap without an answer). The run went from 11:19 UTC on 25 September to 03:50 UTC on 26 September 2026.

2.5 Measures

The order of the measures was fixed with the product owner before the run: first the gates (RQ1), then how hard the passing puzzles are (RQ2), then time and cost (RQ3). A blind read of theme and story is reported separately (Section 3.4) and enters none of them.

• Gates. A request passes when a draft clears every gate within the attempts allowed; each model's passes are counted over the ten requests.
• Hardness. For each pass, the game engine's solve-path walker computes readableShare, the share of the board a solver can fill by reading the clues one at a time (lower = harder; Section 3.2), and speculativeWork, the depth of trial reasoning the solve needed.
• Time and cost. Model minutes (Section 2.4), answered calls and output tokens, per request and per pass; money only where a route reports it.

Each model ran each request once. The results are counts and means over those runs; the uncertainty of a pass rate is given as a Wilson interval [2] in Section 5.

3. Results

3.1 Gates (RQ1)

Grok 4.7 passed 9 of the 10 requests; Fable 5.1, Grok 4.6 and Opus 5.5 passed 8, GPT-6 Sol 7 and GPT-6 Astra 6. Every request was passed by at least two models (the World Heritage hard request, by Grok 4.7 and Fable 5.1). The easy end held: all six models passed both very easy requests (12 of 12), and 11 of 12 easy ones passed. Failures gathered higher up: 9 of 12 medium, 5 of 12 hard and 9 of 12 very hard passed. Most failures were near misses: in 10 of the 14 the best draft of the run was sound and failed only the difficulty-tier gate, computing a different tier from the one requested.

3.2 Hardness (RQ2)

readableShare is the share of the board that a solver can fill by reading the clues one at a time, before combining any two or trying an assumption. It is computed by the game engine's solve-path walker; lower means harder. It falls with the tier as intended: over all passes it averaged 0.92 at very easy and 0.09 at very hard.

Five of the six models are close: their passes average 0.533 to 0.598 (Fable 5.1 lowest). GPT-6 Sol's passes are the easiest to read clue by clue (0.696). A mean covers only the requests a model passed, so a model that failed the harder requests is left with an easier set. Compared with the mean of all passes of the same request (Table 2), the picture holds: GPT-6 Sol is the most readable (+0.092), Grok 4.6 (−0.053) and Fable 5.1 (−0.038) the least, and the rest lie within a few hundredths of the request's mean.

3.3 Time and cost (RQ3)

GPT-6 Sol spent the least model time: 9.9 minutes per request and 14.1 per pass. Among the models that passed eight or more, Opus 5.5 was the fastest per pass (16.4 minutes), then Fable 5.1 (22.0), Grok 4.6 (26.5) and Grok 4.7, the slowest of all (27.7). The GPT-6 models reported about 28,326 and 26,985 output tokens per request, against 82,850 to 119,174 for the others.

The time goes to the model. The two Grok models never waited on a limit, so their calls ran back to back. Between the start of its first call and the end of its last, Grok 4.6 worked 211.6 of 213.7 minutes and Grok 4.7 249.2 of 250.7; everything between the calls (the gates, building the prompt, moving to the next request) took 2.1 and 1.5 minutes in all.

Money. Only two routes report a price. xAI reported USD 2.52 for Grok 4.6's answered calls and USD 3.05 for Grok 4.7's. The Claude client reports USD 75.39 for Fable 5.1 and USD 27.47 for Opus 5.5, although both ran on a flat subscription whose limits paused them; the Codex client reports no price for the GPT-6 models. Cost is therefore compared here by calls, tokens and model minutes.

3.4 The blind read

An AI agent running Opus 5.5 (itself one of the six authors) that had not seen which model wrote what read the English of all 46 passing puzzles as a player would, one request at a time with the versions shuffled, and scored theme (0–2), story (0–3: does it set up everything the clues rely on), clues (0–3: plain words, no arithmetic, one relation per clue, natural) and appeal (0–2). It is a single reader, and it is reported on its own.

Opus 5.5 (8.13) and Fable 5.1 (8.00) read best, then Grok 4.7 (6.89), GPT-6 Sol (6.29), Grok 4.6 (6.25) and GPT-6 Astra (4.00). Fable 5.1 wrote the best version of its request outright in 5 of the 10 requests and tied for it in 2 more; Opus 5.5 outright in 1 and tied in 3; GPT-6 Sol and GPT-6 Astra never wrote the best version. The models differ more in the story (means 1.17–2.75 of 3) than in the clues (0.67–2.00 of 3). The faults the reader named most were clues built on a total of neighbouring values, told as arithmetic; textbook templates ("X, Y and Z were three different people", "exactly one of these two claims is true"); stories that never define what a clue relies on; and one sentence carrying several relations. The best versions define every time, place and grouping in the story and keep each clue a short, plain statement.

4. Discussion

No model wins every measure. Grok 4.7 passes the most requests but is the slowest per pass and reads below the two Anthropic models. Opus 5.5 and Fable 5.1 pass 8 each and read best; Opus 5.5 is the fastest of the models with eight or more passes, and Fable 5.1 wrote the most best versions. Grok 4.6 passes 8 and writes the hardest puzzles against the same requests, with a weaker read. GPT-6 Sol is the fastest but passes 7 and writes the most readable, that is the least demanding, logic. GPT-6 Astra passes 6 and reads worst. GRIDIGMA's study agent recommends Opus 5.5 for authoring the logic, or Fable 5.1 if the story is weighed above speed, and Grok 4.7 if the pass rate alone decides. That recommendation is the study agent's; the product owner has not decided.

5. Limitations

• Ten requests. Each model ran each of the ten requests once, so a difference of one pass is one request. The 95% interval (Wilson [2]) for a pass rate of 9 of 10 runs from 0.60 to 0.98, and for 6 of 10 from 0.31 to 0.83: the ranking is a direction, not a verdict.
• The commonest defect was the prompt's. The instructions encouraged the clue types the catalogue uses least, among them the window sum, which is a sum by nature. All 9 very hard passes used one, as did 2 medium ones, and in all 11 the blind reader named the sum or total; those passes averaged 0.55 of 3 for their clues, against 2.00 for the rest. Several drafts stated the total as arithmetic in prose, which the product owner had ruled out. The next authoring prompt has to say how a window sum is told without arithmetic, or stop encouraging it.
• Time goes to reasoning, not to the gates (Section 3.3), so a faster run needs a faster model or fewer revisions, not faster gates.
• Subscription limits. Four of the six models ran on subscriptions whose limits paused them, and three of them had requests restarted. Model minutes exclude the waits, but the elapsed time of a run on those routes is far longer than its model minutes.
• One reader. The blind read is one AI agent's judgement: a subagent of the Claude session that ran the study, running Opus 5.5, the same model as one of the six authors and from the same maker as a second, Fable 5.1. Those two received the best reads (8.13 and 8.00); the read was blind, but a preference for its own or its maker's style [3] cannot be ruled out. Players have not rated these puzzles.
• Model identity. As in the companion paper [1] (Section 7.5), the GPT-6 identifiers are those the Codex client reports; they could not be confirmed independently.
• Not placed. None of the passing puzzles has been published in the game; they are kept for review.

6. Conclusion

Six models authored the logic of 10 puzzles each through GRIDIGMA's production loop and gates; Grok 4.7 passed the most (9 of 10), Opus 5.5 and Fable 5.1 passed 8 and read best, and GPT-6 Sol was the fastest. No model won every measure, and the product owner has not chosen one. With ten requests per model, a difference of one pass is one request, so these results give a direction for choosing an authoring model, not a choice. The next authoring prompt should say how a window sum is told without arithmetic, or stop encouraging it; and players have not yet rated the passing puzzles.