Best Practices and Lessons Learned on Synthetic Data · HackerPapers discussions

Best Practices and Lessons Learned on Synthetic Data — mentioned by seanhunter

Sun, 15 Dec 2024 07:16:30 GMT

Ilya could be wrong. I don’t think the question is decided yet in general. We already know that in lots of fields fake data can be used in ways that are as useful as or even more useful than the real thing[1], but in my understanding that tends to be situations where we have an objective function that is unambiguous and known beforehand. Meta has some very impressive work on synthetic data for training and my (uninformed) read was that is the state of the art in eg voice recognition at the moment.[2] [1] Eg Sobel sequences in a monte carlo simulation instead of real random numbers. They allow…

Paper and discussion history. A mention is not an endorsement.

Best Practices and Lessons Learned on Synthetic Data — mentioned by danielmarkbruce

Thu, 03 Oct 2024 23:09:20 GMT

There is a lot of nonsense in here, for example: > but we know that synthetic datasets make for poor training data This is a silly generalization. Just google "synthetic data for training LLMs" and you'll find a bunch of papers on it. Here's a decent survey: https://arxiv.org/pdf/2404.07503 It's very likely o1 used synthetic data to train the model and/or the reward model they used for RLHF. Why do you think they don't output the chains...? They literally tell you - competitive reasons. Arxiv is free, pick up some papers. Good deep learning texts are free, pick some up.

Paper and discussion history. A mention is not an endorsement.