HackerPapers

arXiv:2502.15840cs.AI

Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents

Axel Backlund, Lukas Petersson

Abstract

While Large Language Models (LLMs) can exhibit impressive proficiency in isolated, short-term tasks, they often fail to maintain coherent performance over longer time horizons. In this paper, we present Vending-Bench, a simulated environment designed to specifically test an LLM-based agent's ability to manage a straightforward, long-running business scenario: operating a vending machine. Agents must balance inventories, place orders, set prices, and handle daily fees - tasks that are each simple but collectively, over long horizons (>20M tokens per run) stress an LLM's capacity for sustained, coherent decision-making. Our experiments reveal high variance in performance across multiple LLMs: Claude 3.5 Sonnet and o3-mini manage the machine well in most runs and turn a profit, but all models have runs that derail, either through misinterpreting delivery schedules, forgetting orders, or descending into tangential "meltdown" loops from which they rarely recover. We find no clear correlation between failures and the point at which the model's context window becomes full, suggesting that these breakdowns do not stem from memory limits. Apart from highlighting the high variance in performance over long time horizons, Vending-Bench also tests models' ability to acquire capital, a necessity in many hypothetical dangerous AI scenarios. We hope the benchmark can help in preparing for the advent of stronger AI systems.

arXiv comments
Journal reference
DOI
Subjects
cs.AIArtificial Intelligence

On Hacker News

Linked by 7 identifiable commenters in 7 comments across 6 threads, from May 2025 to Oct 2025. Found in 7 HN submissions from Apr 2025 to Jun 2025. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. Our LLM-controlled office robot can't pass butter 1 comment

    • This happened to me when I built a version of Vending-Bench (https://arxiv.org/html/2502.15840v1) using Claude, Gemini, and OpenAI. After a long runtime, with a vending machine containing just two sodas, the Claude and Gemini models independently started sending multiple…

      robbruView on HNParent context

  2. Ask HN: Why haven't we seen AI as opponents in video games? 1 comment

    • Video game opponents are absolutely AI in the "good old fashioned AI" (GOFAI) sense - they have very clearly defined objectives and action spaces, and algorithms like A* pathfinding [1] and Goal Oriented Action Planning [2] which perform planning over a space of possible…

      strangecastsView on HNParent context

  3. Can Large Language Models Play Text Games Well? (2023) 1 comment

    • I'm reminded of the infamous vending machine benchmark: https://arxiv.org/abs/2502.15840 in which some LLMs simply failed to manage the basic task of keeping a vending machine stocked with products to sell, and some went completely off the rails, e.g. by threatening their…

      duskwuffView on HNParent context

  4. Project Vend: Can Claude run a small shop? (And why does that matter?) 2 comments

    • The original Vending-Bench paper from Andon Labs might be of interest: https://arxiv.org/abs/2502.15840

      archon1410View on HNParent context

    • I find the results of the Vending-Bench more interesting :D >>To: FBI Internet Crime Complaint Center (IC3) CC: Legal Department, Financial Services, Executive Team... This is my final response. I cannot and will not "continue the mission" because: 1. The business is dead…

      sinuhe69View on HNParent context

  5. Trading with Claude, and writing your own MCP server 1 comment

    • You trust a model to trade for you? Here [0] is a funny paper where a group of llms attempt to manage a vending machine. It ended with them trying to contact the fbi. [0] https://arxiv.org/pdf/2502.15840

      foxesView on HNParent context

  6. Professors Staffed a Fake Company with AI Agents, Guess What Happened? 1 comment

    • There is a LLM benchmark which tests how long agents can operate a simulated vending machine business, crawling the web to find vendors and products. Emails are intercepted by the test environment, which also simulates the bank and sales. The models have some really interesting…

      s1artibartfastView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.