How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? — mentioned by gwern

Tue, 07 Nov 2023 20:31:16 GMT

> I'm not sure folks who're putting out strong takes based on this have read this paper. They haven't read the other papers either. It's really striking to me to watch people retweet this and it get written up in pseudo-media like Business Insider when other meta-learning papers on the distributional hypothesis of inducing meta-learning & generalization, which are at least as relevant, can't even make a peep on specialized research subreddits - like, "Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression", Raventós et al 2023…

Paper and discussion history. A mention is not an endorsement.

How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? · HackerPapers discussions

How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? — mentioned by gwern