How new data permeates LLM knowledge and how to dilute it — mentioned by ijk

Sun, 01 Jun 2025 18:11:48 GMT

Depends on the definition of "knowledge"; there's a lot of factors that go into it. Some of the common approaches are continued/continual pretraining and model editing (https://arxiv.org/pdf/2502.12598). * Models are bad at learning that A=B implies B=A, let alone more complicated relations; augmenting the dataset with multiple examples with different phrasing/perspectives is important (https://arxiv.org/abs/2404.00213). The frequency that a relation occurs in the dataset affects the results (https://arxiv.org/html/2504.09597v2). * You have to be able to balance preserving existing knowledge…

Paper and discussion history. A mention is not an endorsement.

How new data permeates LLM knowledge and how to dilute it · HackerPapers discussions

How new data permeates LLM knowledge and how to dilute it — mentioned by ijk