Easy Problems That LLMs Get Wrong · HackerPapers discussions

Easy Problems That LLMs Get Wrong — mentioned by thethirdone

Fri, 06 Sep 2024 03:37:15 GMT

The sample answers for the horse race question are crazy. [0] Pretty much all the LLM really want to split 6 horses into two groups of three. Only LLAMA 3 makes the justification that only 2 horses can be raced at a time, but then gets its modified question wrong by racing three horses. I personally would consider an answer that presumes some restriction to how the horses can be raced to be valid if it answers the restricted version correctly. [0]: https://arxiv.org/html/2405.19616v2#S9.SS2.SSS1

Paper and discussion history. A mention is not an endorsement.

Easy Problems That LLMs Get Wrong — mentioned by smusamashah

Thu, 05 Sep 2024 22:00:50 GMT

We need results from these harder/different benchmarks which give pretty bad scores to current top LLMs. https://www.wolfram.com/llm-benchmarking-project/ https://help.kagi.com/kagi/ai/llm-benchmark.html Edit : There are few other benchmarks that give pretty low scores (<20%) to top LLMs. Can't find them atm. There was a benchmark with common sense easy looking questions. Edit: found two more papers https://arxiv.org/html/2405.19616 https://arxiv.org/html/2406.02061v1 Edit: How about Wordle? https://www.strangeloopcanon.com/p/what-can-llms-never-do https://news.ycombinator.com/item?id=40179232

Paper and discussion history. A mention is not an endorsement.

Easy Problems That LLMs Get Wrong — HN submission

Fri, 31 May 2024 10:28:35 GMT

Easy Problems That LLMs Get Wrong

5 points; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.