TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks · HackerPapers discussions

TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks — HN submission

Tue, 12 Aug 2025 15:01:07 GMT

AI agents fail tasks 70% of the time

23 points; 8 discussion replies on HN. Paper and source history. A submission is not an endorsement.

TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks — mentioned by evil-olive

Sun, 13 Jul 2025 21:09:44 GMT

> I think you should loop John Doe from team X into this discussion yeah, that's a useful thing that a chatbot could do...in theory. in practice, from the recent CMU study [0] of how actual LLMs perform on real-world tasks like this: > For example, during the execution of one task, the agent cannot find the right person to ask questions on RocketChat. As a result, it then decides to create a shortcut solution by renaming another user to the name of the intended user. 0: https://arxiv.org/pdf/2412.14161 (pdf)

Paper and discussion history. A mention is not an endorsement.

TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks — HN submission

Mon, 05 May 2025 02:26:30 GMT

TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks — HN submission

Sun, 04 May 2025 06:00:51 GMT

CMU TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks

7 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks — mentioned by ryan_j_naughton

Mon, 28 Apr 2025 12:42:30 GMT

And here's the academic article: https://arxiv.org/abs/2412.14161

Paper and discussion history. A mention is not an endorsement.

TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks — HN submission

Mon, 06 Jan 2025 17:48:19 GMT

Study of AIAgents-driven Company: at best 24% of the tasks could be completed

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks — HN submission

Mon, 06 Jan 2025 00:25:44 GMT

Benchmarking LLM Agents on Consequential Real World Tasks

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.