Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws — mentioned by ijk

Thu, 18 Apr 2024 19:30:40 GMT

Quite. The big question at the time was "how much data do we need to train GPT-3 equivalent models". Open models had failed to live up to GPT performance, even ones with a massive number of parameters. So getting results that suggested a reason why other models were massively undertrained was important. Meanwhile, people noticed that for deployed models, inference cost often outweighs the initial training costs. It's sometimes better to train a smaller, faster model longer on more data, because it has lower overall cost (including environmental impact) if you're expecting to run the model a…

Paper and discussion history. A mention is not an endorsement.

Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws · HackerPapers discussions

Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws — mentioned by ijk