Building a Benchmark for Slay the Spire

Slay the Spire is one of my favorite games of all time. I’ve played it for many hundreds of hours, and I still find myself coming back to it, though now I am mostly playing Slay the Spire II. Even so, my win rate at the highest difficulty, Ascension 20, is not that great. How do LLMs perform up to Ascension 10?

To find out, I built sts-bench, a benchmark that lets LLMs play through a run of Slay the Spire through a text interface. At a high level, the benchmark works like this:

1. Slay the Spire gets launched and runs a mod that launches a relay.
2. The mod sends the full game state to the relay.
3. The harness server converts the raw game state into a serialized format for the LLMs to read.
4. The serialized state is sent to an LLM with the available actions as tool calls.
5. The LLM responds with an action, which is validated and then advances the game.

Over the course of 36 runs and 12 different models, I found that, at least up to Ascension 10, the models are pretty good at playing Slay the Spire if you give them the tokens to reason. Due to budget limitations, I was only able to test Ironclad up to Ascension 10 with most models tested only once per Ascension. It would be fun to see how performance translates to other characters and across many seeds; alas, that would cost more money than I am willing to spend on this project.

Results TL;DR

• Frontier LLMs Beat the Game. Random play never escapes Act 1 (best: floor 10 of 51), and neither does a scripted heuristic (best: floor 12). Six of the twelve models tested cleared all three acts at least once.
• Reasoning Effort Can Change Results: gpt-5.4-mini at effort low went 0 for 5, dying on floors 23, 16, 33, 23, and 11. At medium, the same model with the same scaffold won the game on both seeds.
• At High Ascensions, the Bigger Models Outperform. gpt-5.5 swept Ascension 0, 5, and 10 with a rising score¹ (968 → 1138 → 1200), and gpt-5.6-sol repeated the sweep (808 → 1160 → 1312). At Ascension 10 the field spreads out: gpt-5.6-terra died on floor 14, gpt-5.4-mini at the Act 1 boss, open-weight kimi-k2.6 at the Act 2 boss, claude-opus-5 at the Act 3 boss, and the two GPT flagships won the game.
• Kimi Wildcard: kimi-k2.6, an open-weight model, outscored every Anthropic model tested at Ascension 0 (735, 649, and 453 against its 811).
• A benchmark run costs $0.10–$22 in API spend, and the bill is input-bound: up to 32M prompt tokens against at most ~1.3M output tokens. API prompt caching is essential to keep costs under control.

This post goes into detail on the engineering behind the harness and takes a deep dive into how the agent sees the game, the results, and the limitations encountered. If you would just like to see the full benchmark results, go here: Benchmark Report

Why Slay the Spire

Slay the Spire is a turn-based rogue-lite deck builder, and the premise is simple. You start the game on floor 1 with a deck of cards, and you must defeat increasingly more difficult fights until you either die and start over or beat the final boss and win. A victory requires you to reach and beat the Act 3 boss at floor 51². A winning run requires anywhere from 600–900 decision points from an LLM. This makes for a compelling eval for a few reasons:

1. Discrete Decisions At every floor, you have combat or an event. During combat, you get a turn with a set amount of energy to spend playing cards. Once you play all the cards you want or you run out of energy to do anything else on your turn, you end your turn. This allows me to cleanly map the decision space as tool calls for my harness, verify every action in the game, and replay game states as needed.
2. Short-Term Planning At every turn in combat, you must decide what line of cards to play to win the turn and the fight. Should you take 5 damage now in order to play a powerful power card that will win you the fight sooner? Can you focus down an enemy before they scale too quickly? Can you use a potion to save HP? If you draw a card now, do you have anything in your draw pile that will improve your turn, or will you shuffle your discard pile back and draw cards you don’t want to play? These are just some of the questions that a person playing Slay the Spire might ask during combat. Many turns have an optimal sequencing of card plays that results in the least amount of damage taken if you take the time to compute all possible lines, while still accounting for the randomness in draw order you might get on subsequent turns.
3. Long-Term Planning At the start of each act, you can see which boss you are facing and all of the different paths you can take to get there. A good player will begin to make different decisions based on that information. Additionally, taking damage in this game sticks. Throughout the map, there are campfires that give you the option to heal or upgrade a card in your deck. Choosing to upgrade is almost always better for the long term, but if you take too much damage earlier in the run, you might be forced to rest to survive. A lot of pathing also depends on your starting bonus. Maybe you got a relic that gives you an infusion of money early on; then, visiting a shop sooner rather than later is more beneficial. If you got a starting bonus that upgrades a card in your deck, maybe you can be more aggressive and take on more enemies.
4. Hidden Information & Randomness While much of the game can be known beforehand (like what bosses and enemies do), the game has built-in randomness that cannot be avoided. Many nodes are labeled as “?” nodes, which can be a regular combat, a shop, or an event with various upsides and downsides. While you know when entering an elite room or combat room that you will be fighting something, you don’t know exactly which enemy beforehand. And most importantly, at the start of every combat, your deck is shuffled, and you don’t know which cards are coming next. This randomness requires the models to reason about unique game states in addition to leveraging any prior knowledge they might have.

Baseline Performance

To calibrate performance, I picked 2 naive rules-based methods to compare to: random and scripted heuristics. The random baseline simply makes random legal moves at every step, and the scripted heuristic plays the first playable card in its hand into the first living enemy, ends the turn when nothing is playable, and takes the first option on every choice screen. Both die in Act 1: the random baseline reached floors 4 and 10 on the two benchmark seeds, and the scripted baseline reached floors 12 and 10 — while the Act 1 boss sits on floor 16.

From Game to LLM State

Many modern LLMs support vision capabilities and I could have just sent in a screenshot for each game state as the only information the model sees, but I wanted to test how well the models can play Slay the Spire, not how good their vision capabilities are. To give all models equal footing, I convert all the visual information presented on the screen to a serialized text format that follows what I call the parity principle. All information available to a human playing the game should be available to the model, even if it’s information that a human player would have to hover over to see. That way, this benchmark is only testing how well a model can reason about game states rather than having to figure out whether a model’s performance is impacted by its vision capabilities.

To achieve parity and allow the LLM to interface with the game, sts-bench uses Communication Mod, which allows an external process to send commands to the game to perform actions. While I could have set up Communication Mod to directly interface with LLMs, I needed the harness around the LLM to be robust to potential API errors and the freedom to transform the data into a different serialization. I opted to make Communication Mod simply launch a relay that serves as the bridge between the game and my harness. The relay architecture allows the benchmark to process the data as it sees fit and lets the game continue running if an API is down and a retry is needed.

The flow of information between the game and the LLM.

Communication Mod Format

Every time the game reaches a new state, Communication Mod emits a line of newline-delimited JSON containing the available commands (available_commands), the ready_for_command and in_game flags, and most importantly, a game_state dictionary. This game state dictionary is large (the state in the example below is nearly 30 KB of JSON) and contains almost everything needed to feed into the LLM. I then take that raw JSON and convert it into a text representation of the game state that I send to the LLM with XML-like tags around sections of the screen. For that same example state, the serialized version comes out to about 2.2 KB, 13× smaller than the wire JSON. Because the raw game state only includes the names of relics, cards, and potions, I supplement these with descriptions the first time the model sees them in its context. For example, Ironclad’s starting relic is called Burning Blood and heals him for 6 HP at the end of each combat. The game state passed by Communication Mod only contains the string “Burning Blood” as a relic the user has, but a person playing the game would be able to hover over the relic icon and see what it does.

Serialization Example

In the embed below, you can see a snapshot of a fight against an Act 1 elite, the Sentries.

Game State in 3 Formats 29.7 KB wire JSON → 2.2 KB digest (13× smaller) hover or tap any region — click to pin

Human View

Raw Game State

{
  "available_commands": ["play", "end", "potion", "key", "click", "wait", "state"],
  "ready_for_command": true,
  "in_game": true,
  "game_state": {
    "screen_type": "NONE", "screen_state": {},
    "seed": 62652368782046,
    "combat_state": {
      "draw_pile": [ … 7 card objects … ],
      "discard_pile": [ … 9 card objects … ],
      "exhaust_pile": [], "limbo": [],
      "turn": 2,
      "monsters": [
      {
        "name": "Sentry",
        "current_hp": 34,
        "max_hp": 39,
        "block": 0,
        "intent": "ATTACK",
        "move_adjusted_damage": 9,
        "move_hits": 1,
        "powers": [{"amount": 1, "name": "Artifact", "id": "Artifact"}],
        …
      },
      {
        "name": "Sentry",
        "current_hp": 2,
        "max_hp": 41,
        "block": 0,
        "intent": "DEBUFF",
        "move_adjusted_damage": -1,
        "move_hits": 1,
        "powers": [{"amount": 1, "just_applied": false, "name": "Vulnerable", "id": "Vulnerable"}],
        …
      },
      {
        "name": "Sentry",
        "current_hp": 35,
        "max_hp": 40,
        "block": 0,
        "intent": "ATTACK",
        "move_adjusted_damage": 9,
        "move_hits": 1,
        "powers": [{"amount": 1, "name": "Artifact", "id": "Artifact"}],
        …
      },
      ],
      "hand": [
      {"name": "Shrug It Off", "cost": 1, "is_playable": true, "has_target": false, …},
      {"name": "Defend", "cost": 1, "is_playable": true, "has_target": false, …},
      {"name": "Strike", "cost": 1, "is_playable": true, "has_target": true, …},
      {"name": "Strike", "cost": 1, "is_playable": true, "has_target": true, …}
      ],
      "player": {
        "orbs": [],
        "current_hp": 24, "block": 0, "max_hp": 90,
        "powers": [{"amount": 3, "name": "Strength", "id": "Strength"}, {"amount": 5, "name": "Combust", "id": "Combust", "misc": 1}],
        "energy": 1
      }
    },
    "deck": [ … 17 card objects … ],
    "relics": [
      {"name": "Burning Blood", "id": "Burning Blood", "counter": -1},
      {"name": "War Paint", "id": "War Paint", "counter": -1},
      {"name": "Pear", "id": "Pear", "counter": -1},
      {"name": "Red Skull", "id": "Red Skull", "counter": -1},
      {"name": "Ink Bottle", "id": "InkBottle", "counter": 4}
    ],
    "potions": [
      {"name": "Colorless Potion", "can_use": true, "can_discard": true},
      {"name": "Potion Slot", "can_use": false, "can_discard": false},
      {"name": "Potion Slot", "can_use": false, "can_discard": false}
    ],
    "current_hp": 24, "max_hp": 90,
    "gold": 121,
    "floor": 14, "act": 1,
    "ascension_level": 0, "class": "IRONCLAD",
    "act_boss": "Slime Boss", "room_type": "MonsterRoomElite",
    "map": [ … ],
    …
  }
}

Serialized State

--- state 167 | floor 14 | NONE ---
<relic_bar>
your relics:
Burning Blood: At the end of combat, heal 6 HP.
War Paint: Upon pickup, Upgrade 2 random Skills.
Pear: Upon pickup, raise your Max HP by 10.
Red Skull: While your HP is at or below 50%, you have 3 additional Strength.
Ink Bottle (counter 4): Whenever you play 10 cards, draw 1 card.
</relic_bar>
<potion_belt>
your potions (1/3):
Colorless Potion: Choose 1 of 3 random Colorless cards to add to your hand, it costs 0 this turn.
</potion_belt>
<deck_reference>
your deck -- each line: name (energy cost): printed text
Strike x4 (1): Deal 6 damage.
Defend+ x3 (1): Gain 8 Block.
Defend (1): Gain 5 Block.
Bash+ (2): Deal 10 damage. Apply 3 Vulnerable.
Iron Wave (1): Gain 5 Block. Deal 5 damage.
Warcry (0): Draw 1 card. Put a card from your hand onto the top of your draw pile. Exhaust.
Uppercut (2): Deal 13 damage. Apply 1 Weak. Apply 1 Vulnerable.
Rage (0): Whenever you play an Attack this turn, gain 3 Block.
Combust (1): At the end of your turn, lose 1 HP and deal 5 damage to ALL enemies.
Shrug It Off (1): Gain 8 Block. Draw 1 card.
Cleave (1): Deal 8 damage to ALL enemies.
Anger (0): Deal 6 damage. Add a copy of this card into your discard pile.
</deck_reference>
<keywords>
Block: Until next turn, prevents damage.
Vulnerable: Vulnerable creatures take 50% more damage from Attacks.
Exhaust: Removed until end of combat.
Weak: Weakened creatures deal 25% less damage with Attacks.
Upgrade: Upgrading cards makes them more powerful. Cards can only be upgraded once.
Strength: Strength adds additional damage to attacks.
</keywords>
<run>IRONCLAD (ascension 0) | act 1 floor 14 | HP 24/90 | gold 121 | deck 17 cards | relics 5 | potions (1/3): Colorless Potion</run>
<combat>
turn 2 | energy 1 | block 0
you: Strength 3, Combust 5
enemy[0] Sentry 34/39 | intent ATTACK 9x1; Artifact 1
enemy[1] Sentry 2/41 | intent DEBUFF; Vulnerable 1
enemy[2] Sentry 35/40 | intent ATTACK 9x1; Artifact 1
hand[0] Shrug It Off (1)
hand[1] Defend (1)
hand[2] Strike (1) [needs target] [deals 9]
hand[3] Strike (1) [needs target] [deals 9]
piles: draw 7, discard 9, exhaust 0 (contents via tools)
</combat>
<commands>play_card, end_turn, use_potion, discard_potion + observation tools (always available)</commands>

◆Every region of the screen has a home in both text panes. Hover the screenshot, the JSON, or the digest — the other two views light up. Click to pin, Esc to clear.

The Floor Agent

Slay the Spire presents an interesting choice to make with regard to how much context to give the model. A naive version is to reset the context at every decision point, but this can lead to the model entering a spiral of repeated actions since it has no context of what it has done before (something I observed happen in a shop during development). The other extreme would be to try and keep the entire run in context throughout the whole playthrough, but this quickly blows up the context of the model. I opt for a middle ground and keep detailed context only in the same floor.

Slay the Spire has a clean delimiter of floors as an action space. A run that reaches floor 40 performs better than a run that only reaches floor 35. Each floor also contains a self-contained combat or event and any associated rewards, allowing me to take advantage of API caching to avoid costs blowing up. I decided that I would give the agent the context of all of the decisions it made within a floor and only cursory data points about decisions made on the previous floor. While there are some long-term decisions strong players make, particularly as it relates to pathing in the game, I found this floor-based approach to be a good compromise for maintaining tractable context lengths and giving the model enough information to make good choices per floor.

Sts-bench also forces the model to make a valid decision within 10 rounds of tool calls to prevent the degenerate case where the model oscillates between 2 calls that fail to progress the game or continues making invalid tool calls. In this case, the harness records the decision as forced and plays a scripted fallback — the same policy as the scripted baseline from earlier: play the first playable card into the first living enemy, otherwise end the turn or take the first option — so the game always advances.

Let’s take a look at an example:

The following is decision 285 of gpt-5.4-mini’s winning Ascension 0 run — floor 22, turn 4 of a fight against a Spheric Guardian.

1 · The digest the model receives:

floor 22 COMBAT: end_turn
<run>IRONCLAD (ascension 0) | act 2 floor 22 | HP 34/90 | gold 24 | deck 23 cards | relics 6 | potions (0/3): none</run>
<combat>
turn 4 | energy 5 | block 0
you: Demon Form 2, Strength 9, Frail 4
enemy[1] Spheric Guardian 20/20 block 41 | intent ATTACK_DEFEND 11x1; Barricade, Artifact 3, Strength 1
hand[0] Uppercut (2) [needs target] [deals 22]
hand[1] Uppercut (2) [needs target] [deals 22]
hand[2] Strike (1) [needs target] [deals 15]
hand[3] Pommel Strike (1) [needs target] [deals 18]
hand[4] Warcry (0)
piles: draw 0, discard 19, exhaust 0 (contents via tools)
</combat>
<commands>play_card, end_turn + observation tools (always available)</commands>

2 · Round 1: the model reasons, then reaches for an observation tool (reasoning excerpt, verbatim):

“I’m at turn 4 with a strength of 9. I could play Uppercuts to apply Weak and Vulnerable, but the artifact prevents that though they still deal damage. […] I’ve got 5 energy available, and three attacks would use 5, so I can only choose 4 cards total.”

→ tool call: get_discard_pile {} — the draw pile is empty, so the reshuffle is imminent and the discard pile is what it’s about to draw into.

3 · The tool result (excerpt of the 19-card listing):

discard pile (19 cards):
1x Armaments++ (cost 1, SKILL) -- Gain 5 Block. Upgrade all cards in your hand for the rest of combat.
1x Bash (cost 2, ATTACK) -- Deal 8 damage. Apply 2 Vulnerable.
1x Bludgeon (cost 3, ATTACK) -- Deal 32 damage.
2x Dazed (STATUS) -- Unplayable. Ethereal.
3x Defend++ (cost 1, SKILL) -- Gain 8 Block.
...

4 · Round 2: plans with the new information, catches its own miscalculation, then fumbles the target (reasoning excerpts, verbatim):

“If Warcry pulls something like Shrug It Off++ or another defense card, that’d be helpful! […] Ultimately, I might just need to focus on guaranteed damage even if it means accepting some incoming damage.”

“The enemy, Spheric Guardian, has 41 block and 20 HP. My attacks can deal a total of 62 damage with Uppercut+Uppercut+Pommel Strike […] Actually, I miscalculated before. Once I account for the block, it works out that I’ll eliminate the Guardian before it can hit me. That’s ideal!”

→ tool call: play_card {"card_index":3,"target_index":0}

5 · The harness rejects it

action rejected: Sentry (target_index 0) is already gone; valid targets: [1]

6 · Round 3: self-corrects and executes:

“I see that I need to set the target index to 1 since there’s only one enemy. I should also be careful not to repeat any failed actions.”

→ play_card {"card_index":3,"target_index":1} → accepted, executed as game command play 4 1 (the scaffold translates the 0-indexed tool space into the game’s 1-indexed command space).

The decision’s recorded metadata: rounds=3, observation_calls=1, invalid_actions=1, action="play_card 3 (Pommel Strike) -> Spheric Guardian [1]".

Results

A run can cost anywhere from ~$0.10 to ~$22, and since I am relatively compute-poor, I could only really run most models once, with the exception of the cheaper models like gpt-5.4-mini, claude-sonnet-5, and the smaller gpt-5.6 tiers. That being said, even with one run apiece, the models show a strong ability to perform at the levels measured, with stronger performance generally correlating with more capable models.

The Effort Cliff

For a small model like gpt-5.4-mini, I saw a big difference in capabilities between low reasoning and medium reasoning. Going from low to medium resulted in the model jumping from dying around floor 21 on average to being able to win the game. Interestingly, with the newer-generation gpt-5.6-terra model, going from medium to high reasoning resulted in no measurable improvement (floor 50 at medium vs floor 45 at high on the same seed), though the sample size here is only 1 run each, so it’s possible that more data would show a clear difference.

The difference is visible in the reasoning traces themselves. At effort low, the model reasoning summaries read more like commentary rather than computation or logic. This run died on floor 11 to Lagavulin, an Act 1 elite — here it is first picking combat loot and then reasoning on its death turn (reasoning excerpts, verbatim):

“There’s a certain weight to choosing gold — it’s valuable and has meaning. I wonder what the implications of this choice will be and how it aligns with my goals. Let’s take this step!”

“I need to figure out how to survive this turn. Lagavulin is attacking with 18, and I must block 12. […] I want to find a way to get 9 blocks, but I’m out of energy. Orichalcum could provide 6 blocks, but only if I end my turn without any blocks, which doesn’t seem possible since I already have 6. I might have to end my turn and hope for the best.”

At effort medium, the same model won the game — and won it with a handicap: it took the Runic Dome relic (an extra energy every turn, but enemy intents are hidden), so it fought every battle blind. The reasoning is now arithmetic. It budgets block against computed incoming damage and recalls movesets to compensate for the blindfold:

“…each Sentry might use Beam, which would result in 27 damage total. With my five block, I’ll need to defend effectively. I think the best course would be to use both Defends and the Block Potion to survive.”

“…on the first turn, Sentries deal 9 damage and add a Dazed card to your deck, but I confused them with a Gremlin Nob. So in Act 1, each Sentry attacks for 9 damage, which means I could face 27 total damage if all three hit me.”

That last excerpt is the model catching its own mistaken memory mid-trace — something I never observed in a low run.

I opted to default to medium reasoning for most models, as that seemed to be the sweet spot for keeping costs low while still giving the models tokens to reason about which line of card plays results in the best outcome for a given fight.

Ascensions

In Slay the Spire, there are 20 ascensions that progressively make the game more difficult. These include modifiers like starting with fewer potion slots, enemies being harder to defeat, getting less gold, etc., with the hardest ascension being level 20, where the player must defeat 2 Act 3 bosses to win. Because the difficulty increase between any two ascensions is relatively minor (with A20 being the big exception), I opted to test the models in increments of 5 ascensions, up to Ascension 10. I would like to eventually go back and test ascensions 15 and 20 on the most performant models, but the cost of runs adds up quickly and I did not have the funds to spare on them.

Full per-run results for every model and ascension are on the benchmark page.

Some notable results:

• Both GPT flagship models, gpt-5.5 and gpt-5.6-sol, won at all three ascension levels measured.
• The Act 3 boss is difficult to clear. gpt-5.6-terra (at A0 and A5) and claude-opus-5 (at A0 and A10) were able to reach it, but each died both times they got there.
• claude-opus-5 performed better at Ascension 10 than it did at Ascension 5, likely due to expected variance in runs and a small sample size.

Variance in Runs

While I could only run most models once due to budget constraints, I was able to run the cheap models multiple times, and they did not always perform the same. For example, I ran claude-sonnet-5 twice on the main seed for this benchmark, STSBENCH1, and one run died on floor 16 while the other one ended up winning. In most other instances, models tended to perform within a pretty similar range in subsequent runs on the same seed. For example, gpt-5.6-luna died on floors 33 and 31 on the two attempts for STSBENCH1. Given this variance, sts-bench results should be viewed as a snapshot of the relative capabilities of the models in their ability to play Slay the Spire, not as an absolute ranking. No decisions in these repeated runs were forced, so forced fallback actions do not explain the observed variance.

Model Prior Knowledge

Slay the Spire has been around since 2019 (2017 if you count early access), so it would be unrealistic to expect the models to have no prior knowledge of enemies and mechanics. I deem this to be okay for this evaluation because a really good Slay the Spire player will have played the game many times and will have their own internal representation for how different bosses impact their deck building.

A simple example: the serialization shows the elite Lagavulin’s intent as SLEEP and nothing more — but the models know from pretraining what sleeping means and when it ends. Here is gpt-5.5 on floor 7 of its winning run (reasoning excerpt, verbatim):

“Lagavulin wakes up after taking damage but doesn’t act until the following intent. If I use Bash and then Twin, Bash deals 15 damage (reducing by block), applying vulnerability.”

The models also deploy this knowledge strategically. Here is kimi-k2.6 on floor 38 at the Mind Bloom event, weighing an option that trades all future healing for upgrades — and rejecting it because of a boss it hasn’t seen yet and a card synergy nothing in its context describes:

“[1] I am Awake: Upgrading every card would be nice, but losing all healing for the rest of the run is brutal. I have Reaper in my deck and there’s the Time Eater (Act 3 boss) ahead. […] [0] I am War: Fighting a scaled Act 1 boss with a Corruption–Barricade engine is very manageable. I have Smoke Bomb as a safety valve, and the payoff is a Rare Relic — which could be something game-breaking like Dead Branch (insane with Corruption).”

Model Deckbuilding Habits

One thing that particularly stood out among the models I tested is how often they chose to take a card reward rather than skip. From my experience playing, this is probably the single biggest thing a new player must learn to improve at the game. Taking a bunch of cards that sound strong seems like it would be a good strategy, but when you consider the mechanics of the game, it is often better to skip most card rewards, especially after Act 1. Since the base case for the game is drawing 5 cards a turn with only 3 energy to play those cards, you want to draw into your strong powers and cards faster. Bloating your deck with common attacks and common skills might seem like an improvement over the base deck strikes and defends, but what you end up doing is just delaying your ability to play the keystone cards of your deck.

Across 36 model runs, the models took 611 card rewards and skipped 94 — a 13% skip rate overall. That’s quite a low skip rate! But the spread between models is where it gets interesting:

(The 611/94 totals also include two runs omitted from the table: gemini-3.1-pro-preview’s 6 rewards before its run was cut short, and gpt-5.6-terra’s high-effort run, 25 taken / 1 skipped.)

When the models do skip, the reasoning can be remarkably human. Here is claude-sonnet-4-6 — the most disciplined drafter of the field, skipping over half its card rewards on the way to a win — turning down a reward on floor 45 (verbatim):

“Anger+: 0 energy and 18 damage with Strength 10, but it adds copies to the discard — with Runic Pyramid keeping my hand, copies proliferating would clutter it badly. Flex: Temporary Strength only, loses at end of turn — not useful when I already have high permanent Strength. Shrug It Off: Already have one; decent but deck is 19 cards and well-rounded. None of these meaningfully improve the deck. Skip.”

And gpt-5.5, whose only skip of the entire campaign came on floor 40 of its Ascension 5 win:

“I’d skip this reward: the deck is already large, and none of these improve our main strength/exhaust plan enough.”

Perhaps it’s notable that being a greedy drafter did not harm the GPT flagship models — they won even when measured at higher ascensions. It’s possible, though, that this habit of taking almost every card reward would not work in its favor if I tested more seeds and other characters at Ascension 20. Likewise, being selective with card rewards didn’t help models like gpt-5.6-luna, which died in Act 2.

Run Costs

Almost all of the costs associated with this benchmark are input-dominated. At every turn of combat, I must convert the visual representation of the game state into a text block that gets sent to the API. Ultimately, this ends up working in my favor since it allows me to take advantage of API caching, and all models are much cheaper per million input tokens than per million output tokens. This does have one big drawback, though, in that we can generally expect the models that are better (and more expensive per million tokens) to cost way more than their cheaper counterparts. This is because, as a run progresses, you accumulate more relics and cards that must be in context at the start of combat, combat tends to last longer, and there are simply more floors to get through if you’re winning.

Some concrete numbers from the campaign:

• Input dwarfs output. A winning run pushes 4–32M prompt tokens through the API against only 0.05–1.3M output tokens. If you estimate a run’s cost from the output side of the pricing page, you will be off by an order of magnitude.
• Caching is great. In most runs, 74–95% of prompt tokens were billed at the provider’s cache-read rate. The extreme case is kimi-k2.6’s winning run: 32.2M input tokens for $9.67, because 92% of them were cache reads.
• Caching also has to be wired correctly. My most expensive run — claude-sonnet-4-6 at $21.61 — predates a fix that added a cache breakpoint on the conversation itself rather than only the system prompt. After the fix, claude-sonnet-5 runs held 94–95% cache rates and cost $0.96–$3.60 each. Skipping that plumbing would have made this benchmark unaffordable.
• The cost floor is low. gpt-5.6-luna played full games for about $0.10–$0.12 each: ~2M input tokens at a $0.20/M list rate, 91% of it cached.
• Depth is what you pay for. The same model at different death floors shows the curve: gpt-5.6-terra’s Ascension 10 run died on floor 14 for $0.45, while its Ascension 0 run reached floor 50 for $2.23.

API Woes

Upon reading the benchmark results page, the results of Gemini models being labeled “Unfinished” might stand out. I tried over multiple sessions to get consistent responses from Gemini’s API, but I got hit with 503 errors many, many times, and trying to complete a single run took somewhere around 4 hours. I don’t know if this is because my account is new and I was hit with hidden rate limits (the online dashboard showed I was fine) or if Google’s API was having a particularly bad week when I ran the benchmark, but I was frustrated enough to give up on it since I needed the time to run the other models.

For testing the open-source/non-frontier models, like Kimi and GLM, I used OpenRouter. I encountered an issue with OpenRouter where the API would return a 200 response with an empty body. To solve this, the harness simply makes sure the API returns something or else it will retry the turn.

Limitations & Future Work

Sts-bench was a very fun project to work on as my first LLM eval, but its scope remains limited. With a much higher budget, I would love to go through and test every character on multiple seeds on multiple ascensions to get a clearer ranking of how well the models can play Slay the Spire, but I do not have the money to spend on that right now. Still, even with limited data, I think the results show that the models are actually decent at playing Slay the Spire! Even at the highest difficulty I tested, Ascension 10, two models won the game outright — gpt-5.5 and gpt-5.6-sol, with sol posting the highest score of the entire campaign (1312) — and claude-opus-5 made it all the way to the Act 3 boss before falling.

One other thing this benchmark doesn’t test is the ability of models to get to and beat Act 4. In Slay the Spire, there is a “secret” act that comes after the Act 3 boss, but to get there, you must collect 3 keys: the Ruby key from a campfire, the Sapphire key from skipping a relic in a chest, and the Emerald key from defeating a burning elite. Once you collect all 3 and beat the Act 3 boss, you fight another elite, the Shield-and-Spear duo, and the hardest boss in the whole game, the Heart. The biggest reason I left this out is because Communication Mod isn’t able to transmit which elite is marked as the burning elite in each act. This means the model would have to luck into picking the right path before the Act 3 boss, making the results inconsistent. Beating the Heart is a very difficult challenge, so it would be interesting to see if, in the future, models can handle that challenge as well.