Language Modeling Is Compression · HackerPapers discussions

Language Modeling Is Compression — mentioned by thisIsFiner

Mon, 02 Nov 2026 20:36:37 GMT

This isn't that insane to anyone with old school engineering experience and not abstract thought experiment engineering in the cloud. Language modeling is compression[1] and all the state to manage the cloud doesn't organically exist, it's added to the system. We will see huge gains as all the developer middleware is compressed out as nonsense syntax outdated to the task at hand of linking phones and tablets to world models governments control. Sam Altman was just saying they know best how to use your memory; what they decide is stored in memory is what we all experience. It's not to store…

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by lotrg

Mon, 26 Oct 2026 09:38:54 GMT

Agree with you, I do Language modeling is fundamentally compression: https://arxiv.org/abs/2309.10668 The entire endeavor is ultimately fundamentally coupled to reducing compute Eventually we will compress models further by removing outdated knowledge like Python and Cobol cause who needs 1960s like languages to then layout data in the screen when the model has knowledge of geometry and can mutate the screen geometry without needing a Python interpreter or C compiler

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by d0nkee

Mon, 19 Oct 2026 16:55:27 GMT

OpenAI is already a "math research" company Computer science has been a subset of math since the field was established decades before the first vacuum tube or transistor based machine was invented "Computer" was coined to describe humans who did math all day What we been doing the last few decades is fundamentally just algebra relative to values in RAM; LLMs only work because of math: https://arxiv.org/abs/2309.10668 def arbitraryFunctionName() { some math to achieve a physical state in memory } ...where arbitrary function name is some biz logic or use case We're just pushing those arbitrary…

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by northlondoner

Tue, 22 Sep 2026 15:29:50 GMT

Reminded me Google's earlier paper, "Language Modeling Is Compression" https://arxiv.org/abs/2309.10668 Learning being a compression is also recently proposed as Gibbs compression proposition. See Gibbs randomness-compression proposition https://arxiv.org/abs/2505.23869v5

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by orbital-decay

Fri, 05 Jun 2026 00:24:16 GMT

This is a simplistic take. It's not a mere interpolator by any measure, there's a ton of research on that, starting with the basics https://arxiv.org/abs/2309.10668v2

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by in-silico

Mon, 01 Jun 2026 09:17:27 GMT

> Would be interesting whether it is possible to write a LLM-like program just using compression and function interpolation algoritms. gzip can be used as a (not very good) LLM-like text and image generator: https://arxiv.org/abs/2309.10668

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by jackhurwitz

Thu, 12 Feb 2026 16:24:13 GMT

I'm doing my PhD in compression-based machine learning, wanted to contribute a few clarifying points. The relationship between probabilistic modeling and lossless compression is very direct. A model that predicts the next symbol with probability p can on average losslessly compress that symbol with the help of an entropy coder (e.g. arithmetic coding) in -log(p) bits. Therefore improved probabilistic models immediately translate into improved lossless compressors. There are two ways people have used compressors for ML: the first is based on the Minimum Description Length (MDL) principle [0]…

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by problynought

Sat, 07 Feb 2026 23:12:20 GMT

Have you worked in software long? I've been in eng for almost 30 years, started in EE. Can confidently say you can't trust the humans either. SWEs have been wrong over and over. No reason to listen now. Just a few years ago code gen LLMs were impossible to SWEs. In the 00s SWEs were certain no business would trust their data to the cloud. OS and browsers are bloated messes, insecure to the core. Web apps are similarly just giant string mangling disasters. SWEs have memorized endless amount of nonsense about their role to keep their jobs. You all have tons to say about software but little idea…

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by CamperBob2

Tue, 13 Jan 2026 00:39:11 GMT

Well, they couldn't generate 2D artwork if they weren't capable of working with multiple output dimensions. An interesting thing about transformers is that they are world-class at compressing 2D image data even when not trained on anything but textual language ( https://arxiv.org/abs/2309.10668 ). Whether that notion is relevant for 3D content would be two or three figures over my pay grade, though.

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by tekbruh9000

Sat, 18 Oct 2025 18:35:57 GMT

"Higher" than an EE with an MSc in elastic structures, ~30 years industry experience, now working with PhDs across the spectrum on energy models to embed in chips? Energy models in part, inferred from categorization of LLM contents and compression of those contents into geometric functions like I described? "Higher level" implies acceptance of geometric structure. You place tokens like a Chomsky diagrams at each step up and down, where you should see parameters to transform geometry of the structure. My team works "above" the contrived state management of software workers to more efficiently…

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by disruptbro

Fri, 25 Jul 2025 09:54:10 GMT

Language modeling is compression, whittle down graph to reduce duplication and data with little relationship: https://arxiv.org/abs/2309.10668 Let’s say everyone agrees to refer to one hosted copy of a token “cat”, and instead generate a unique vector to represent their reference to “cat”. Blam. Endless unique vectors which are nice and precise for parsing. No endless copies of arbitrary text like “cat”. Now make that your globally distributed data base to bootstrap AI chips from. The data driven programming dream where other machines on the network feed new machines boot strap. American tech…

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by b112

Wed, 02 Jul 2025 13:26:50 GMT

Not to mention, Language Modeling is Compression https://arxiv.org/pdf/2309.10668 So text wikipedia at 24G would easily hit 8G with many standard forms of compression, I'd think. If not better. And it would be 100% accurate, full text and data. Far more usable. It's so easy for people to not realise how massive 8GB really is, in terms of text. Especially if you use ascii instead of UTF.

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by Mathnerd314

Wed, 11 Jun 2025 02:53:45 GMT

Wasn't there that thing about how large LLM's are essentially compression algorithms (https://arxiv.org/pdf/2309.10668)? Maybe that's where this article is coming from, is the idea that finetuning "adds" data to the set of data that compresses well. But that indeed doesn't work unless you mix in the finetuning data with the original training corpus of the base model. I think the article is wrong though in saying it "replaces" the data - it's true that finetuning without keeping in the original training corpus increases loss on the original data, but "large" in LLM really is large and current…

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by RadEng00

Sun, 08 Jun 2025 17:50:19 GMT

We have to ditch language processing. And we will with online energy based models that machines boot from. Maxwell could not get the theory of electromagnetism to work until he ditched pulleys and levers he’d included to describe the mechanics. We won’t get AGI until we realize “there is no spoon” and language has nothing to do with our intelligence, just with out social tribalism: https://www.scientificamerican.com/article/you-dont-need-wor... Take language out of the equation and drawing a circle, triangles, letters is just statistical physics. We can capture in energy models stored in an…

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by braaaahp

Mon, 26 May 2025 21:19:42 GMT

Yeh. Data driven model syncing machines will be what kills languages, software stacks. All the chip world talk of single function machines and how tech is now energy constrained industry, means pruning the state we store; most software is software to deliver software. Single function pipeline hardware platforms will act as little more than sync engines from models. I mean it’s is 2025. Still write software like it’s the 1970s. So high tech. Edit: https://arxiv.org/abs/2309.10668 From LLMs to energy models that transform electromagnetic geometry. Saving what’s needed to recreate cat pics and…

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by bangertho

Fri, 23 May 2025 02:14:55 GMT

This already happened with Linux where people with no understanding of electrical engineering got jobs memorizing abstractions. I look forward to a data driven system future where a few functions transform the machines electromagnetic geometry to solve a task based upon the most efficient energy model for solving a task as we continue to compress from the model all the non-essential syntax sugar of modern software. https://arxiv.org/abs/2309.10668

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by zerd

Mon, 05 May 2025 18:39:16 GMT

> It has long been established that predictive models can be transformed into lossless compressors and vice versa. Incidentally, in recent years, the machine learning community has focused on training increasingly large and powerful self-supervised (language) models. Since these large language models exhibit impressive predictive capabilities, they are well-positioned to be strong compressors. In this work, we advocate for viewing the prediction problem through the lens of compression and evaluate the compression capabilities of large (foundation) models. We show that large language models…

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by bugglebeetle

Tue, 20 Aug 2024 07:22:25 GMT

Oh well, you should tell the folks at DeepMind and Meta about these objective facts then so they don’t waste any more time doing research: https://arxiv.org/html/2309.10668v2 Maybe apply for a job there too, since you’re obviously so far ahead of everyone in understanding this problem space.

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by TheAlchemist

Sun, 21 Jul 2024 22:40:42 GMT

This is actually an excellent question ! And quite an active field of research. This is one paper you could read - it's about text and not films, but same idea: "LANGUAGE MODELING IS COMPRESSION" - https://arxiv.org/pdf/2309.10668

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by gwern

Tue, 23 Apr 2024 21:35:34 GMT

Vapnik: https://www.cs.princeton.edu/courses/archive/spring13/cos511... https://engineering.columbia.edu/files/engineering/vapnik.pd... https://www.learningtheory.org/learning-has-just-started-an-... https://nautil.us/teaching-me-softly-234576/ The main paper: https://gwern.net/doc/reinforcement-learning/exploration/act... It sounds kinda crazy (is there really that much far transfer?), but you know, I think it would work... He just needed to use LLMs instead: https://arxiv.org/abs/2309.10668#deepmind

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by haltIncomplete

Thu, 28 Mar 2024 19:20:32 GMT

All we’re doing is engineering new data compression and retrieval techniques: https://arxiv.org/abs/2309.10668 Are we sure there’s anything “net new” to find within the same old x86 machines, within the same old axiomatic systems of the past? Math is a few operations applied to carving up stuff and we believe we can do that infinitely in theory. So “all math that abides our axiomatic underpinnings” is valid regardless if we “prove it” or not. Physical space we can exist in, a middle ground of reality we evolved just so to exist in, seems to be finite; I can’t just up and move to Titan or…

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by 5kg

Tue, 20 Feb 2024 17:17:00 GMT

Language Modeling Is Compression: https://arxiv.org/abs/2309.10668

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by jabowery

Wed, 17 Jan 2024 22:23:42 GMT

The increasing recognition that "Language Modeling Is Compression" https://arxiv.org/pdf/2309.10668.pdf has not yet been accompanied by recognition that lossless compression is the most principled unsupervised loss function for world models in general, including foundation language models in particular. Take, for instance, the unprincipled definition of "parameter count" not only in the LLM scaling law literature, but the Zoo of what statisticians called "Information Criteria for Model Selection". https://en.wikipedia.org/wiki/Model_selection#Criteria The reductio ad absurdum of "parameter…

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by atorodius

Sun, 31 Dec 2023 08:36:26 GMT

Yes, next token prediction is exactly what you need for compression. Many papers on this, recent one https://arxiv.org/pdf/2309.10668.pdf

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by famouswaffles

Thu, 16 Nov 2023 06:22:15 GMT

https://aps.arxiv.org/abs/2309.10668

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by intended

Fri, 13 Oct 2023 05:31:27 GMT

Yes! Prompts are super finicky. You have to create a prompt/function that for a wide set of inputs, generates a token sequence that will perpetually expand in a manner that corresponds to an externally observed truth. Way too often it feels like you have to shove a universal decoding sequence into a prompt. “Talk your steps, list your clues, etc.” Just trying to luck into a prompt that keeps decompressing the model/ generating the next token that ensures the next token is true.* I recall there was a paper with a relevant title recently… https://arxiv.org/abs/2309.10668 Basically - LLMs don’t…

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — HN submission

Sat, 30 Sep 2023 16:22:03 GMT

Language Modeling Is Compression

3 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Language Modeling Is Compression — mentioned by beefman

Thu, 28 Sep 2023 19:24:32 GMT

https://arxiv.org/abs/2309.10668

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by mullikine

Fri, 22 Sep 2023 03:43:41 GMT

It's dangerous territory to get inspiration from a Large Language model instead of the Holy Spirit. Thank God for His Holy Scriptures but it should be the Scriptures that set foremost in precedence to other training material for a LLM if it is going to be used as a tool. Even then the proper way to get inspiration for a sermon is to pray and have a conversation with God about it, not a conversation with an AI. I can think of many problems with getting inspiration for sermons from an AI instead of the Holy Spirit. We don't want the pure and holy scriptures being subverted or interpretted by…

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — mentioned by j-pb

Thu, 21 Sep 2023 19:59:15 GMT

https://arxiv.org/abs/2309.10668

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — HN submission

Thu, 21 Sep 2023 16:50:16 GMT

Language Modeling Is Compression

27 points; 3 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Language Modeling Is Compression — mentioned by m00x

Wed, 20 Sep 2023 16:21:45 GMT

It goes along the same lines as the recent paper from Deepmind stating that DL (language modeling in their case) is compression. https://arxiv.org/pdf/2309.10668.pdf

Paper and discussion history. A mention is not an endorsement.

Language Modeling Is Compression — HN submission

Wed, 20 Sep 2023 11:45:03 GMT

Language Modeling Is Compression

5 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.