Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time · HackerPapers discussions

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time — mentioned by bwhitty

Sun, 14 Jun 2026 18:37:28 GMT

As another poster above linked, it’s been shown to be effective since 2022: https://arxiv.org/abs/2203.05482

Paper and discussion history. A mention is not an endorsement.

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time — mentioned by woadwarrior01

Sun, 14 Jun 2026 18:13:02 GMT

It's is a well known idea[1], although it's still surprising that something as simple, even works. [1]: https://arxiv.org/abs/2203.05482

Paper and discussion history. A mention is not an endorsement.

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time — mentioned by magicalhippo

Thu, 05 Mar 2026 15:08:09 GMT

Maybe some newer references are better, but my mind went to the Model Soups paper[1]: The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder. In this paper, we revisit the second step of this procedure in the context of fine-tuning large pre-trained models, where fine-tuned models often appear to lie in a single low error basin. We show that averaging the weights of multiple models fine-tuned with different hyperparameter…

Paper and discussion history. A mention is not an endorsement.

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time — mentioned by magicalhippo

Tue, 31 Dec 2024 05:21:21 GMT

This is probably a dumb idea, but I'll air it anyway. I just stumbled upon the model soup[1] paper where, as I understand it, they average weights of fine-tuned models and get an model that performs better. They have a more involved algorithm but even the uniform soup (simple weight average) seems to perform well. In your paper you mention that especially in late stages the gradients of microbatches are often not aligned, hence the agreement filtering. What you're doing, from my brief glossing over, is effectively to do a k-means clustering with outlier rejection pass with k = 1. You then use…

Paper and discussion history. A mention is not an endorsement.