Training LLMs over Neurally Compressed Text · HackerPapers discussions

Training LLMs over Neurally Compressed Text — mentioned by pizza

Mon, 09 Sep 2024 05:24:02 GMT

In a way, some things like this are possible/have been explored. In Huffman coding, the code length is log2(length of path in binary tree to x) ~ log2(1/p(x)). BPE works by combining the most frequent pair of adjacent tokens into a new token. So generally speaking, the most frequent token end up being earliest in the vocab, having a lower index. The number of bits to encode that index is log2(i(x)). In theory if you could have variable width int indices, then shorter indices would correspond with the more frequent tokens, so log2(i(x)) would correlate somewhat with log2(1/p(x)) and you would…

Paper and discussion history. A mention is not an endorsement.

Training LLMs over Neurally Compressed Text — HN submission

Tue, 07 May 2024 22:56:33 GMT

Training LLMs over Neurally Compressed Text

10 points; 3 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Training LLMs over Neurally Compressed Text — HN submission

Thu, 11 Apr 2024 11:49:20 GMT

Training LLMs over Neurally Compressed Text

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Training LLMs over Neurally Compressed Text — HN submission

Tue, 09 Apr 2024 22:06:40 GMT

Training LLMs over Neurally Compressed Text

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.