ChessArena: A Chess Testbed for Evaluating Strategic Reasoning Capabilities of Large Language Models · HackerPapers discussions

ChessArena: A Chess Testbed for Evaluating Strategic Reasoning Capabilities of Large Language Models — mentioned by carodgers

Tue, 15 Sep 2026 23:20:34 GMT

This April 2026 paper is a fun and related read. https://arxiv.org/html/2509.24239v4 Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO. The author of the originating post says that "current frontier…

Paper and discussion history. A mention is not an endorsement.

ChessArena: A Chess Testbed for Evaluating Strategic Reasoning Capabilities of Large Language Models — mentioned by carodgers

Tue, 15 Sep 2026 06:58:53 GMT

"Do you really think if you would get unlimited tokens and a compute cluster that something like a fable or astra frontier model setup couldn't build a better chess engine?" Disconnect it from the internet, where it cannot copy from humans, and no, it certainly will not. If it can copy code from humans, then maybe, with guidance. But if a human is not there to offer guidance, no, I don't think it will improve on stockfish. Worse, if it cannot copy code from humans, it will fail embarrassingly at playing chess on its own. This April 2026 paper found that frontier models which did not have…

Paper and discussion history. A mention is not an endorsement.