Metadata Conditioning Accelerates Language Model Pre-training — mentioned by astrange

Fri, 10 Oct 2025 19:27:18 GMT

> Whatever shows up most commonly in the training data is is what an LLM will output. It's more complicated than that of course, but that's the basic idea. The most common thing in the training data is the letter 'e'. If you're going to explain how an LLM works it needs to explain why it's able to form sentences at all. In particular answering questions is a behavior which only appears after posttraining, and the posttraining objective has absolutely nothing to do with what's "most common" in the pretraining data. > But that would mean someone would have to curate the entire internet, looking…

Paper and discussion history. A mention is not an endorsement.

Metadata Conditioning Accelerates Language Model Pre-training · HackerPapers discussions

Metadata Conditioning Accelerates Language Model Pre-training — mentioned by astrange