Synthetic continued pretraining — mentioned by nickandbro

Tue, 14 Oct 2025 21:17:36 GMT

You could! But just like others have mentioned, the performance would be negligible. If you really wanted to see more of a performance boost by pretraining you could try to create a bigger chunk of data to train off of. This would be done by either creating synthetic data off of your material, or finding adjacent information to your material. Here's a good paper about it: <https://arxiv.org/abs/2409.07431>

Paper and discussion history. A mention is not an endorsement.

Synthetic continued pretraining — HN submission

Sat, 14 Sep 2024 05:20:16 GMT

Synthetic Continued Pretraining

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Synthetic continued pretraining · HackerPapers discussions

Synthetic continued pretraining — mentioned by nickandbro

Synthetic continued pretraining — HN submission