Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models — HN submission

Fri, 01 Mar 2024 23:22:22 GMT

The Impact of Input Length on the Reasoning Performance of LLMs

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models — mentioned by mediaman

Fri, 01 Mar 2024 04:37:40 GMT

There is a lot of emerging evidence that ability to reason, and ability to compare within-context dramatically declines with large context. It is unable to find edits: https://twitter.com/mengdi_en/status/1763338245696274469 It hallucinates changes badly: https://twitter.com/thomasahle/status/1763408041960231010 To be fair, finding diffs is an odd way to use LLMs, but does show the highly variable abilities of these models over bigger contexts. In other models, reasoning performance declines with more tokens (no data yet here for Gemini 1.5 Pro): https://arxiv.org/html/2402.14848v1 Note in…

Paper and discussion history. A mention is not an endorsement.

Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models · HackerPapers discussions

Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models — HN submission

Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models — mentioned by mediaman