HackerPapers

arXiv:2306.04050cs.IT

LLMZip: Lossless Text Compression using Large Language Models

Chandra Shekhara Kaushik Valmeekam, Krishna Narayanan, Dileep Kalathil, Jean-Francois Chamberland, Srinivas Shakkottai

Abstract

We provide new estimates of an asymptotic upper bound on the entropy of English using the large language model LLaMA-7B as a predictor for the next token given a window of past tokens. This estimate is significantly smaller than currently available estimates in \cite{cover1978convergent}, \cite{lutati2023focus}. A natural byproduct is an algorithm for lossless compression of English text which combines the prediction from the large language model with a lossless compression scheme. Preliminary results from limited experiments suggest that our scheme outperforms state-of-the-art text compression schemes such as BSC, ZPAQ, and paq8h.

arXiv comments
7 pages, 4 figures, 4 tables, preprint, added results on using LLMs with arithmetic coding
Journal reference
DOI
Subjects
cs.ITInformation Theory cs.CLComputation and Language cs.LGMachine Learning

On Hacker News

Linked by 5 identifiable commenters in 5 comments across 5 threads, from Jul 2023 to Aug 2026. Found in 1 HN submission from Nov 2025 to Nov 2025. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. Compression is prediction 1 comment
  2. Hiding messages in a deck playing cards 1 comment

    • You can get much better than that by taking a well-known LLM model and encoding a series of offsets from the most likely sequence of tokens, especially if you are OK with the message being slightly different. https://arxiv.org/abs/2306.04050 https://bellard.org/ts_zip/

      avidiaxView on HNParent context

  3. Ask HN: Can tokenizers go from fixed length tokens to varying length? 1 comment

    • To tease out the issues: I see the BPE algorithm as similar to Huffman coding in the way it builds the token list by creating, at any point, a token that is optimal in "compressing" the data in that it represents a commonly occuring sequence. As BPE is going to create tokens for…

      PaulHouleView on HNParent context

  4. The Era of 1-bit LLMs: ternary parameters for cost-effective computing 1 comment

    • Two possible implementation: (1) Take your data as a stream. Use your machine learning gadget to give you the (predicted) probability for each of the possible next tokens. Then use those probability in arithmetic coding to specify which token actually came next. (2) Take your…

      eruView on HNParent context

  5. LLMZip: Lossless Text Compression Using Large Language Models 1 comment

    • Paper: https://arxiv.org/abs/2306.04050 > We provide new estimates of an asymptotic upper bound on the entropy of English using the large language model LLaMA-7B as a predictor for the next token given a window of past tokens. > Preliminary results from limited experiments…

      bcjordanView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.