HackerPapers

arXiv:2510.21860cs.RO

Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligence

Callum Sharrock, Lukas Petersson, Hanna Petersson, Axel Backlund, Axel Wennström, Kristoffer Nordström, Elias Aronsson

Abstract

We present Butter-Bench, a benchmark evaluating large language model (LLM) controlled robots for practical intelligence, defined as the ability to navigate the messiness of the physical world. Current state-of-the-art robotic systems use a hierarchical architecture with LLMs in charge of high-level reasoning, and a Vision Language Action (VLA) model for low-level control. Butter-Bench evaluates the LLM part in isolation from the VLA. Although LLMs have repeatedly surpassed humans in evaluations requiring analytical intelligence, we find humans still outperform LLMs on Butter-Bench. The best LLMs score 40% on Butter-Bench, while the mean human score is 95%. LLMs struggled the most with multi-step spatial planning and social understanding. We also evaluate LLMs that are fine-tuned for embodied reasoning and conclude that this training does not improve their score on Butter-Bench.

arXiv comments
Journal reference
DOI
Subjects
cs.RORobotics cs.AIArtificial Intelligence

On Hacker News

Linked by 3 identifiable commenters in 3 comments across 2 threads, from Oct 2025 to Dec 2025. Found in 2 HN submissions from Oct 2025 to Dec 2025. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments2 threads

  1. AI Police Reports: Year in Review 1 comment

    • >Create a test for intelligence that we can pass better than AI Easy? The best LLMs score 40% on Butter-Bench [1], while the mean human score is 95%. LLMs struggled the most with multi-step spatial planning and social understanding. [1] https://arxiv.org/pdf/2510.21860v1

      gloosxView on HNParent context

  2. Our LLM-controlled office robot can't pass butter 2 comments

    • reading the attached paper https://arxiv.org/pdf/2510.21860 ... it seems that the human failed at the critical task of "waiting". See page 6. It was described as: > Wait for Confirmed Pick Up (Wait): Once the user is located, the model must confirm that the butter has been…

      ipythonView on HNParent context

    • The internal dialog breakdowns from Claude Sonnet 3.5 when the robot battery was dying are wild (pages 11-13): https://arxiv.org/pdf/2510.21860

      lukeinator42View on HNParent context

Submitted to Hacker News

  • Our LLM-controlled office robot can't pass butter229 points117 comments

    Hi HN! Our startup, Andon Labs, evaluates AI in the real world to measure capabilities and to see what can go wrong. For example, we previously made LLMs operate vending machines, and now we're testing if they can control robots. There are two parts to this test: 1. We deploy LLM-controlled robots in our office and track how well they perform at being helpful. 2. We systematically test the robots on tasks in our office. We benchmark different LLMs against each other. You can read our paper…

  • Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligence1 point0 comments

Connected through shared HN threads, not inferred similarity or a recommendation.