Small Models Struggle to Learn from Strong Reasoners — mentioned by gertlabs

Wed, 20 May 2026 07:33:16 GMT

This is what we do at gertlabs.com - the foundation labs are actually starving for better data. Having quality data is not the same as having a lot of data. Human curated data / RLHF cannot scale to a 5T model and synthetic data pipelines are very much a work in progress in the industry. Some interesting notes: - Training a small model with large model output resulted in LESS improvement than distilling a less smart model onto the same small architecture [0]. We are starting to hit intelligence density limits in small models (<30B models may be nearing saturation now) - good RL environments…

Paper and discussion history. A mention is not an endorsement.

Small Models Struggle to Learn from Strong Reasoners · HackerPapers discussions

Small Models Struggle to Learn from Strong Reasoners — mentioned by gertlabs