HackerPapers

arXiv:2501.17186cs.AI

Complete Chess Games Enable LLM Become A Chess Master

Yinqi Zhang, Xintian Han, Haolong Li, Kedi Chen, Shaohui Lin

Abstract

Large language models (LLM) have shown remarkable abilities in text generation, question answering, language translation, reasoning and many other tasks. It continues to advance rapidly and is becoming increasingly influential in various fields, from technology and business to education and entertainment. Despite LLM's success in multiple areas, its ability to play abstract games, such as chess, is underexplored. Chess-playing requires the language models to output legal and reasonable moves from textual inputs. Here, we propose the Large language model ChessLLM to play full chess games. We transform the game into a textual format with the best move represented in the Forsyth-Edwards Notation. We show that by simply supervised fine-tuning, our model has achieved a professional-level Elo rating of 1788 in matches against the standard Elo-rated Stockfish when permitted to sample 10 times. We further show that data quality is important. Long-round data supervision enjoys a 350 Elo rating improvement over short-round data.

arXiv comments
NAACL 2025
Journal reference
DOI
Subjects
cs.AIArtificial Intelligence cs.CLComputation and Language cs.LGMachine Learning

On Hacker News

Linked by 4 identifiable commenters in 5 comments across 4 threads, from Aug 2025 to Jun 2026.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. I used Claude Code to get a second opinion on my MRI 1 comment

    • That is no one has bothered to finetune or RLVR GPT-5.5 on chess games. Even open-llama-3B can be finetuned to around 1700 Elo[1]. [1] https://arxiv.org/pdf/2501.17186

      red75primeView on HNParent context

  2. AI is not a coworker, it's an exoskeleton 1 comment

    • Relax. Anyone who's genuinely interested in the question will see with a few searches that LLMs can play chess fine, although the post-trained models mostly seem to be regressed. Problem is people are more interested in validating their own assumptions than anything else.…

      famouswafflesView on HNParent context

  3. Claude Sonnet 4.6 1 comment

    • >They still make these errors on anything that is out of distribution. There is literally a post in this thread linking to a chat where Sonnet failed a basic arithmetic puzzle: https://news.ycombinator.com/item?id=47051286 I thought we were talking about actual arithmetic not…

      famouswafflesView on HNParent context

  4. LLMs aren't world models 2 comments

    • https://arxiv.org/abs/2501.17186

      t0md4nView on HNParent context

    • If you train an LLM on chess, it will learn that too. You don't need to explain the rules, just feed it chess games, and it will stop making illegal moves at some point. It is a clear example of an inferred world model from training. https://arxiv.org/abs/2501.17186 PS "Major…

      tmnvdbView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.