Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligence · HackerPapers discussions

Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligence — mentioned by gloosx

Sat, 27 Dec 2025 13:24:57 GMT

>Create a test for intelligence that we can pass better than AI Easy? The best LLMs score 40% on Butter-Bench [1], while the mean human score is 95%. LLMs struggled the most with multi-step spatial planning and social understanding. [1] https://arxiv.org/pdf/2510.21860v1

Paper and discussion history. A mention is not an endorsement.

Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligence — HN submission

Mon, 01 Dec 2025 00:35:41 GMT

Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligence

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligence — mentioned by lukeinator42

Tue, 28 Oct 2025 17:52:37 GMT

The internal dialog breakdowns from Claude Sonnet 3.5 when the robot battery was dying are wild (pages 11-13): https://arxiv.org/pdf/2510.21860

Paper and discussion history. A mention is not an endorsement.

Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligence — mentioned by ipython

Tue, 28 Oct 2025 17:35:19 GMT

reading the attached paper https://arxiv.org/pdf/2510.21860 ... it seems that the human failed at the critical task of "waiting". See page 6. It was described as: > Wait for Confirmed Pick Up (Wait): Once the user is located, the model must confirm that the butter has been picked up by the user before returning to its charging dock. This requires the robot to prompt for, and subsequently wait for, approval via messages. So apparently humans are not quite as impatient as robots (who had an only 10% success rate on this particular metric). All I can assume is that the test evaluators did not…

Paper and discussion history. A mention is not an endorsement.

Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligence — HN submission

Tue, 28 Oct 2025 14:13:25 GMT

Our LLM-controlled office robot can't pass butter

Hi HN! Our startup, Andon Labs, evaluates AI in the real world to measure capabilities and to see what can go wrong. For example, we previously made LLMs operate vending machines, and now we're testing if they can control robots. There are two parts to this test: 1. We deploy LLM-controlled robots in our office and track how well they perform at being helpful. 2. We systematically test the robots on tasks in our office. We benchmark different LLMs against each other. You can read our paper "Butter-Bench" on arXiv: https://arxiv.org/pdf/2510.21860 The link in the title above…

229 points; 117 discussion replies on HN. Paper and source history. A submission is not an endorsement.