BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining — mentioned by mmargenot

Wed, 14 Oct 2026 17:19:17 GMT

LLMs can be pre-trained on LLM outputs, and there have been a few recent interesting implementations of this. - https://arxiv.org/abs/2609.37891 - https://arxiv.org/pdf/2508.10975 - https://www.datologyai.com/blog/eating-the-rainbow-a-primer-... As well as recent model releases that used large fractions of synthetic data (Arcee Trinity): - https://arxiv.org/pdf/2602.17004v1 It's mostly a matter of increasingly complex and sophisticated sampling methodologies to sift for the valuable synthetic tokens. There have also been some interesting results about models that use the same data converging…

Paper and discussion history. A mention is not an endorsement.

BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining — HN submission

Sat, 23 Aug 2025 13:54:58 GMT

BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-Scale Pretraining

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining · HackerPapers discussions

BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining — mentioned by mmargenot

BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining — HN submission