PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change · HackerPapers discussions

PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change — mentioned by epups

Thu, 14 Sep 2023 09:29:56 GMT

The article itself is very assertive and makes a lot of generalizations, but if you look at the source of their claims [1] you see that in the first study they are using GPT-3.5 and achieve only a 5% score on a reasoning test that largely relies on spatial intuition - some boxes need to be stacked and unstacked sequentially in a convoluted task. Then, they get criticised so they come up with another paper in which they use GPT-4, which has an improved performance of 30% - a 6-fold increase, which the author describes as "modest". They then decide to change the test to a much harder and more…

Paper and discussion history. A mention is not an endorsement.

PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change — mentioned by PaulHoule

Mon, 15 May 2023 19:45:53 GMT

There are a lot of arXiv papers where people develop test suites for chatbots such as https://arxiv.org/pdf/2206.10498.pdf

Paper and discussion history. A mention is not an endorsement.

PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change — HN submission

Tue, 11 Apr 2023 23:25:14 GMT

Large Language Models Still Can’t Plan

2 points; 6 discussion replies on HN. Paper and source history. A submission is not an endorsement.