Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens · HackerPapers discussions

Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens — mentioned by yorwba

Tue, 21 Apr 2026 08:47:22 GMT

You can build an unlimited-order Markov chain by, instead of pre-computing a table of counts for all possible contexts, using a substring-search index on the training data to count possible continuations on the fly: https://arxiv.org/abs/2401.17377 That paper uses suffix arrays, but more compact indices are possible: https://arxiv.org/abs/2506.12229

Paper and discussion history. A mention is not an endorsement.

Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens — mentioned by shenberg

Fri, 21 Nov 2025 09:17:15 GMT

The short and unsatisfying answer is that an LLM generation is a markov chain, except that instead of counting n-grams in order to generate the posterior distribution, the training process compresses the statistics into the LLM's weights. There was an interesting paper a while back which investigated using unbounded n-gram models as a complement to LLMs: https://arxiv.org/pdf/2401.17377 (I found the implementation to be clever and I'm somewhat surprised it received so little follow-up work)

Paper and discussion history. A mention is not an endorsement.

Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens — mentioned by astrange

Thu, 20 Nov 2025 23:54:00 GMT

> This is entirely different from saying it can only reproduce samples which it was trained on. It is not a memory machine that is surgically piecing together snippets of memorized samples. (That would be a mind bogglingly impressive machine!) You could create one of those using both a Markov chain and an LLM. https://arxiv.org/abs/2401.17377

Paper and discussion history. A mention is not an endorsement.

Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens — mentioned by astrange

Fri, 14 Nov 2025 23:52:16 GMT

If you're claiming a transformer model is a Markov chain, this is easily disprovable by, eg, asking the model why it isn't a Markov chain! But here is a really big one of those if you want it: https://arxiv.org/abs/2401.17377

Paper and discussion history. A mention is not an endorsement.

Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens — mentioned by sadiq

Thu, 14 Aug 2025 13:48:07 GMT

You might find https://arxiv.org/abs/2401.17377v3 interesting..

Paper and discussion history. A mention is not an endorsement.

Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens — mentioned by yorwba

Sat, 01 Jun 2024 12:17:52 GMT

In this case, the curse of dimensionality is working in our favor, as the volume of an n-dimensional simplex also grows exponentially, so it is perfectly suited to hold the exponential state space of the mixed-state presentation. Updating from one mixed-state to another just involves updating for each pure state in the mixture and then mixing the resulting states together, so everything is neatly linear. One way to represent the training data would be to have one state per token position that emits the corresponding token and advances to the next position, or to a randomly sampled other…

Paper and discussion history. A mention is not an endorsement.

Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens — HN submission

Sun, 05 May 2024 17:58:13 GMT

Infini-Gram: Scaling unbounded n-gram language models to a trillion tokens

142 points; 58 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens — HN submission

Wed, 07 Feb 2024 22:38:05 GMT

Infini-Gram: Scaling Unbounded N-Gram Language Models to a Trillion Tokens

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens — mentioned by Al-Khwarizmi

Thu, 01 Feb 2024 17:04:22 GMT

Not 100M I guess, but this is fresh from today's arXiv: "Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens" https://arxiv.org/pdf/2401.17377.pdf

Paper and discussion history. A mention is not an endorsement.

Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens — HN submission

Thu, 01 Feb 2024 13:38:35 GMT

Infini-Gram: Scaling unbounded n-gram language models to a trillion tokens

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.