Larger language models do in-context learning differently · HackerPapers discussions

Larger language models do in-context learning differently — mentioned by tartakovsky

Wed, 14 Jun 2023 00:12:26 GMT

Same idea here? Larger models do a better job forgetting their training data and dropping their semantic priors. Perhaps another way of thinking through this is that larger models learn new information and drop old information faster. https://arxiv.org/abs/2303.03846 Isn't that interesting? The idea of "mental liquidity", or "strong opinions weakly held"? https://news.ycombinator.com/item?id=36280772

Paper and discussion history. A mention is not an endorsement.

Larger language models do in-context learning differently — mentioned by tartakovsky

Sun, 28 May 2023 11:04:12 GMT

This reminds me of this article for which symbol fine tuning helps overcome: https://arxiv.org/abs/2303.03846 Basically they show that larger models have an easier time giving up their semantic priors, so that, in the context of OP paper, learning the mapping for len becomes print and print becomes len.

Paper and discussion history. A mention is not an endorsement.

Larger language models do in-context learning differently — mentioned by cma

Thu, 09 Mar 2023 16:56:48 GMT

This seems to show with scale one-shot stuff like hearing a slang word substitution one time can be done (within the other limitations of transformers) https://arxiv.org/abs/2303.03846

Paper and discussion history. A mention is not an endorsement.