Understanding Transformers via N-gram Statistics — HN submission

Sat, 17 May 2025 19:56:00 GMT

Understanding Transformers via N-gram Statistics

139 points; 16 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Understanding Transformers via N-gram Statistics — mentioned by cma

Wed, 18 Dec 2024 12:11:36 GMT

GPT4o also has RLHF tuning on top of it though. For raw LLMs getting kind of tripped up by being something like interpolative n-gram estimators this is some recent work that does put them closer to markov chains in some ways, especially early on in training but not fully https://arxiv.org/abs/2407.12034 "In this latter direction, we find that for 79% and 68% of LLM next-token distributions on TinyStories and Wikipedia, respectively, their top-1 predictions agree with those provided by our N-gram rulesets."

Paper and discussion history. A mention is not an endorsement.

Understanding Transformers via N-gram Statistics · HackerPapers discussions

Understanding Transformers via N-gram Statistics — HN submission

Understanding Transformers via N-gram Statistics — mentioned by cma