Language Models are Few-Shot Learners · HackerPapers discussions

Language Models are Few-Shot Learners — mentioned by nl

Tue, 03 Nov 2026 12:07:03 GMT

Yes you are right about random initilization of course. Long day here. For the second: No, not fine tuning at all. For one thing in-context learning is enough for learning novel tasks[1]. Secondly, it can do new things outside that data. LLMs build models of the world[2]. That means it can model novel behavior based on that internal representation. [1] https://arxiv.org/abs/2005.14165 [2] https://arxiv.org/abs/2305.11169

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by CamperBob2

Fri, 04 Sep 2026 19:23:10 GMT

You know they can take notes, right? And ICL dates all the way back to 2020, at least: https://arxiv.org/abs/2005.14165

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by blackbear_

Sat, 06 Jun 2026 06:55:00 GMT

The GPT3 paper is a good starting point Language Models are Few-Shot Learners https://arxiv.org/abs/2005.14165 I also enjoyed the papers for DeepSeek and GLM for an overview of all the tricks you need to make these things work DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models https://arxiv.org/abs/2512.02556 GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models https://arxiv.org/abs/2508.06471

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by derefr

Thu, 19 Mar 2026 17:50:57 GMT

Table 2.1 in https://arxiv.org/pdf/2005.14165.

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by renewiltord

Wed, 11 Mar 2026 16:58:40 GMT

A lot of things are easy if you ignore the incentive structure. E.g. a lot of papers will no longer be published if the data must be published. You’d lose all published research from ML labs. Many people like you would say “that’s perfectly okay; we don’t need them” but others prefer to be able to see papers like Language Models Are Few-Shot Learners https://arxiv.org/abs/2005.14165 So the answer is that we still want to see a lot of the papers we currently see because knowing the technique helps a lot. So it’s fine to lose replicability here for us. I’d rather have that paper than…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by CamperBob2

Fri, 12 Dec 2025 22:00:25 GMT

For example, the fact that they can't learn from few examples For one thing, yes, they can, obviously [1] -- when's the last time you checked? -- and for another, there are plenty of humans who seemingly cannot. The only real difference is that with an LLM, when the context is lost, so is the learning. That will obviously need to be addressed at some point. that they can't perform simple mathematical operations without access to external help (via tool calling) But yet you are fine with humans requiring a calculator to perform similar tasks? Many humans are worse at basic arithmetic than an…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by cs702

Wed, 24 Sep 2025 19:20:49 GMT

The cost of developing GPT-2 and GPT-3 was on the order of millions of dollars, well within the budget of most tech organizations. See https://arxiv.org/abs/2005.14165 for the total compute invested in them. OpenAI had raised only a few tens of millions in donations at the time, as a non-profit organization. The increase in performance of computation has been happening for so many decades now that it's been given names like "Moore's Law." People like Hans Moravec predicted way back in the 1980's that the cost of compute would continue to decline and become cheap enough for AGI by the 2020's…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by photon_lines

Tue, 12 Aug 2025 17:21:23 GMT

Sorry -- I keep seeing this being used but I'm not entirely sure how it differs from most of human thinking. Most human 'reasoning' is probabilistic as well and we rely on 'associative' networks to ingest information. In a similar manner - LLMs use association as well -- and not only that, but they are capable of figuring out patterns based on examples (just like humans are) -- read this paper for context: https://arxiv.org/pdf/2005.14165. In other words, they are capable of grokking patterns from simple data (just like humans are). I've given various LLMs my requirements and they produced…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by godelski

Thu, 15 May 2025 02:21:22 GMT

> The results are surprisingly good, I don't think I could've done better as a human I'm actually surprised that the performance is so poor and would expect a human to do much better. The GPT model has embedding PLUS a whole transformer model that can untangle the embedded structure. To clarify some of the issues: data is both singular and plural, being a mass noun[0,1]. Datum is something you'll find in the dictionary, but not common in use[2]. The dictionary lags actual definitions. I mean words only mean what we collectively agree they mean (dictionary definitely helps with that but we…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — HN submission

Tue, 12 Nov 2024 22:24:47 GMT

Language models are few-shot learners (2020)

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Language Models are Few-Shot Learners — mentioned by intended

Sat, 19 Oct 2024 18:05:18 GMT

Thank you for being blunt. Let me attempt to speak in the same earnest tone. You are contradicting the papers and work that the people who make the models are saying. Alternatively, you are looking at the dataset curation process with rose tinted glasses. >There is a tremendous amount of work that goes on by the LLM companies in verifying, sanitizing, and structuring the training corpuses, using a wide array of techniques. Common crawl is instrumental in building our models, 60% of GPT's training data was Common Crawl. (https://arxiv.org/pdf/2005.14165) pg 9. CC in turn was never intended for…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — HN submission

Mon, 09 Sep 2024 18:23:51 GMT

Show HN: Turn Any ArXiv Paper into a 200-Page Prerequisite Reading Book

I created this tool over the weekend because, as someone interested in AI and technology, I find many research papers on arXiv fascinating but often incredibly dense and difficult to understand due to the heavy jargon and technical language. I wanted to make these complex topics more accessible to laypeople like myself by converting the topics described in these papers into full-length books that break down key concepts, making cutting-edge research easier to grasp. You can think of it as generating prerequisite reading materials before being able to read the actual paper. Here are some…

6 points; 3 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Language Models are Few-Shot Learners — HN submission

Sun, 09 Jun 2024 07:41:46 GMT

Language Models Are Few-Shot Learners

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Language Models are Few-Shot Learners — mentioned by 1vuio0pswjnm7

Tue, 28 May 2024 21:36:37 GMT

https://news.ycombinator.com/item?id=34616256 https://news.ycombinator.com/item?id=40258584 https://arxiv.org/pdf/2005.14165.pdf https://www.wired.com/story/battle-over-books3 https://www.washingtonpost.com/technology/interactive/2023/a... https://www.theguardian.com/technology/2023/apr/20/fresh-con... https://storage.courtlistener.com/recap/gov.uscourts.cand.41... See 40-45. https://storage.courtlistener.com/recap/gov.uscourts.nysd.60... See 87-116.

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by reissbaker

Mon, 11 Mar 2024 22:01:08 GMT

BERT is a language model! It was considered a "large" language model for its time, and it's even based on transformers. It's a very small language model by today's standards (340MM params), and is encoder-only instead of decoder-only, but trying to draw a hard line between "BERT" and "LLMs" is more about parameter count than capabilities — in fact, the original GPT-3 paper benchmarked GPT-3 against finetuned BERT-large and beat it on nearly every measure [1]. And BERT-large is not unique in being able to be finetuned; finetuning Mistral 7B on your task should result in very good performance…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by belter

Wed, 27 Dec 2023 18:22:03 GMT

Really? Because the GPT-3 paper talks about "...two internet-based books corpora (Books1 and Books2)..." (see pages 8 and 9) - https://arxiv.org/pdf/2005.14165.pdf Unclear what that corpora might be, or if its the same books2 you are referring to.

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by WhitneyLand

Sun, 03 Dec 2023 22:36:04 GMT

As a prerequisite to the attention paper? One to check out is: A Survey on Contextual Embeddings https://arxiv.org/abs/2003.07278 Embeddings are sort of what all this stuff is built on so it should help demystify the newer papers (it’s actually newer than the attention paper but a better overview than starting with the older word2vec paper). Then after the attention paper an important one is: Language Models are Few-Shot Learners https://arxiv.org/abs/2005.14165 I’m intentionally trying to not give a big list because they’re so time-consuming. I’m sure you’ll quickly branch out based on your…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by jjoonathan

Thu, 23 Nov 2023 20:59:13 GMT

> Transformers existed in 2019, and while GPT4 is genuinely advanced, it's not a gigantic leap I'm referring to the emergent capabilities that show up with scale, which are a gigantic leap. The usual citation suggests that this came into the public consciousness in 2020 [1]. Do you take issue with the timeline or the idea that these were a gigantic leap? [1] https://arxiv.org/abs/2005.14165

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by ipsum2

Tue, 21 Nov 2023 10:52:01 GMT

> OpenAI's product was technically made by Ilya Nope, he is not a core contributor of GPT-4, GPT-3, GPT-2, DALLE-3, DALLE-2: https://arxiv.org/abs/2303.08774 https://cdn.openai.com/papers/dall-e-2.pdf https://arxiv.org/pdf/2005.14165.pdf https://arxiv.org/abs/1908.09203

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by thethimble

Mon, 20 Nov 2023 00:25:05 GMT

This paper suggests that learning does happen within a prompt. https://arxiv.org/abs/2005.14165 By extension it seems totally reasonable that the model is developing self awareness by introspecting its tendencies (prior generated output), summarizing those tendencies, and then iteratively building a more nuanced description of those tendencies over time.

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by ssnistfajen

Sat, 18 Nov 2023 01:27:31 GMT

GPT-3 had "books1" and "books2" among its training material and "books2" never had its actual source disclosed :https://arxiv.org/pdf/2005.14165.pdf Speculations about these source materials can be traced back as far as 2020: https://twitter.com/theshawwn/status/1320282152689336320 I don't think this issue would've flown under the radar for so long, especially with the implication that Ilya sided with the rest of the board to vote against Sam and Greg.

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by hexaga

Sun, 08 Oct 2023 00:19:20 GMT

Ah, I see. You are unfamiliar with how SOTA language models are actually trained in current day, what RLHF is in detail, and how it differs from other training phases. I'd encourage you to read through the GPT3 [0] and llama2 [1] papers for more salient descriptions of the specific mechanics involved and why RLHF is used at all, before trying to argue from some kind of introduction to ml 101 level simplification of how SOTA models are produced. - [0]: https://arxiv.org/abs/2005.14165 - [1]: https://arxiv.org/abs/2307.09288 For example, from the llama2 paper, section "3.2 Reinforcement…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by spott

Fri, 08 Sep 2023 23:09:19 GMT

The weights of gpt3 weren’t released, but the architecture is pretty close to gpt2 and explained in the paper: https://arxiv.org/pdf/2005.14165.pdf I think the “ready to run” might be more of a byow (“bring your own weights”) kinda thing.

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by swyx

Wed, 06 Sep 2023 18:31:13 GMT

i highlighted this last night which seems to be making the rounds - https://twitter.com/swyx/status/1699369076529971545 Alec's CV (https://www.linkedin.com/in/alecradford/) seems to be: - 2011-2016 BSc Eng from Olin College - 2013+ started a data/AI consultancy as a sophomore that turned into a vague startup/product? - 2015 first paper on GANs coauthored with soumith https://arxiv.org/pdf/1511.06434.pdf%C3 - 2016 first GAN paper under openai email https://proceedings.neurips.cc/paper_files/paper/2016/file/8... - 2017 the generative reviews paper mentioend in the Wired article, with Ilya…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by YeGoblynQueenne

Thu, 20 Jul 2023 09:40:14 GMT

>> I'd question why you'd ask an LLM something like this instead of Wolfram Alpha, or just look up the answer. That's because OpenAI dedicated an entire section in claims that GPT-3 is reasonably good at arithmetic. See Section 3.9.1 titled "Arithmetic" in "Language Models are Few-Shot Learners": https://arxiv.org/abs/2005.14165 Whence I quote below: Overall, GPT-3 displays reasonable proficiency at moderately complex arithmetic in few-shot, one-shot, and even zero-shot settings. The reported results are pretty poor and too poor to justify even the relatively weak claim above (although the…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by jmiskovic

Thu, 20 Jul 2023 08:38:33 GMT

It's actually 175B. https://arxiv.org/pdf/2005.14165.pdf

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by godelski

Tue, 11 Jul 2023 23:32:22 GMT

> It comes from this paper To clarify, what comes from that paper? The claim that LLMs are zero-shot learners (yes) or the term zero-shot (no[0]). > I believe the idea is that the LLM was not trained on the task in question Not quite. We'll see in [0] that the definition is >> We consider the problem of zero-shot learning, where the goal is to learn a classifier f : X → Y that must predict novel values of Y that were omitted from the training set. To achieve this, we define the notion of a semantic output code classifier (SOC) which utilizes a knowledge base of semantic properties of Y to…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by seanhunter

Tue, 04 Jul 2023 20:06:12 GMT

OpenAI publishes a huge amount of research and makes huge amounts of data available for research. They do collaborate with other institutions and researchers. eg: GPT-2: paper https://d4mucfpksywv.cloudfront.net/better-language-models/l... with code: https://github.com/openai/gpt-2, output dataset https://github.com/openai/gpt-2-output-dataset and a bunch of blog posts with all manner of details GPT-3: paper https://arxiv.org/abs/2005.14165 dataset https://github.com/openai/gpt-3 etc etc.

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by sigmoid10

Sun, 18 Jun 2023 17:49:20 GMT

The authors are a bit disingenuous here. They insinuate that GPT3's performance shows unpredictable behaviour change at certain scales using their weirdly constructed metrics (which may or may not be true - see below), while the original GPT3 paper already showed how these amazing "emergent" capabilities scale with parameters in a very predictable way: https://arxiv.org/pdf/2005.14165.pdf Also note that the plots in the appendix contain some obvious errors, so you definitely want to wait for a peer reviewed version of this paper (if it ever survives review).

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by ppaattrriicckk

Mon, 22 May 2023 19:15:27 GMT

I love bashing Intel as much as the other person, but looking at the number of tokens LLaMA[1] was trained on (1,400B tokens), the number of parameters of GPT-3[2] vs. the number of tokens it was trained on (175B params vs. 300B), it's not like the announced 1,000B param model is unreasonable. Taking 175/300*1400 yields 816B parameters - which is fairly close to 1,000B. Not outright the most efficient utilization of data, but as others have mentioned, there might still be something left to gain (from not solely optimizing for compute). E.g. look at Figure 9 in [3]. Although the model trained…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by heliophobicdude

Tue, 16 May 2023 00:14:19 GMT

Refined training is usually updating the weights of usually what's called a foundational model with well structured and numerous data. It's can be expensive but most importantly disturb the usefulness of having all the generalizations baked in from training data [1]. While LLMs can generate code based on a wide range of inputs, they're not designed to retrieve specific pieces of information in the same way that a database or a search engine would. It's just very lossy. Perhaps it wouldn't be the best use for single code base fine tuning right now. Can you please share more about the merging…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by heliophobicdude

Wed, 03 May 2023 14:45:58 GMT

Hi Jason! I have a few thoughts on this! Refined training is usually updating the weights of usually what's called a foundational model with well structured and numerous data. It's very expensive and can disrupt the usefulness of having all the generalizations baked in from training data [1]. While LLMs can generate text based on a wide range of inputs, they're not designed to retrieve specific pieces of information in the same way that a database or a search engine would. But I do think they hold a lot of promise in reasoning. Small corollary: LLMs do not know a head of time what they are…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by sorokod

Sat, 29 Apr 2023 16:55:58 GMT

I can help with "verified correct sources", have a look at "Language Models are Few-Shot Learners" section 2.2 [1]. You may also be interested in Apendix A in the same document: "Details of Common Crawl Filtering" [1] https://arxiv.org/pdf/2005.14165.pdf

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by gensym

Wed, 26 Apr 2023 18:55:01 GMT

> Well the Gebru & Mitchell paper[1] was published before In-Context-Learning was discovered (an ICL was very unexpected), so yes, I think they were ignorant of ICL at the time. ??? One of us is confused here (I'm fully willing to admit that it's me), but AFAIK, the big discovery of in-context learning was described in "Language Models are Few-Shot Learners" - published in 2020[1], vs 2021 for the stochastic parrots paper. Regardless, it's not like the authors of the stochastic parrots paper have disavowed the term. They're still referring to the paper without correction in the statement they…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by nl

Mon, 24 Apr 2023 06:16:19 GMT

For those who don't completely get the impact of this: We already know Large Language Models (LLMs) can learn at runtime (ie, separately to the training process.) This is called "In Context Learning". See [1], [2] for more details. (BTW, when anyone says "LLMs are stochastic parrots" you know they are ignorant of this) In context learning is wonderful because it means you can "train" a LLM at run time by filling the context with examples. Traditionally this "context window" has been a few thousand tokens, and GPT-4 recently extended that to 32,000 tokens. That is useful, but if you wanted to…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by jeeeb

Wed, 19 Apr 2023 23:39:46 GMT

I don’t have any insider insight on this but the GPT3 paper discusses some of their data sets and curation techniques (https://arxiv.org/pdf/2005.14165.pdf). The recent DinoV2 paper is also interesting reading (https://arxiv.org/pdf/2304.07193.pdf), as they particularly focus on techniques for improving the training set. OpenAI also have been open about making heavy use of RL (via PPO) to fine tune the models. For RL it seems they’ve basically developed a second model that can be used to score the quality of responses based on the encoded preferences of human evaluators. I.e you build a…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by RockyMcNuts

Wed, 12 Apr 2023 21:43:55 GMT

not sure what 'mixing LLM models' entails but these are maybe some good starting points - karpathy - https://www.youtube.com/watch?v=kCc8FmEb1nY - https://towardsdatascience.com/beautifully-illustrated-nlp-m... - https://dzone.com/articles/a-deep-dive-into-the-transformer-... - https://peterbloem.nl/blog/transformers - http://nlp.seas.harvard.edu/2018/04/03/attention.html - https://lilianweng.github.io/posts/2023-01-27-the-transforme... - https://blog.quickchat.ai/post/tokens-entropy-question/ - https://dugas.ch/artificial_curiosity/GPT_architecture.html…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by whimsicalism

Fri, 07 Apr 2023 19:35:26 GMT

I suppose it depends on what you mean by "an AI that can do things it wasn't trained to do"! In some sense, the AI is still 'just doing what it was trained to do' in that it is 'just' predicting the next word. All examples of AI doing impressive behavior boil down to AI doing what it was trained to do (pick the next word [delta some RLHF tuning]), very well. If you mean complex behavior arising out of what seems like a very simple unsupervised learning task, then this behavior has been known (although not to this scale) for a while. For example, I distinctly remember being in my 2018 grad…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by sbierwagen

Wed, 29 Mar 2023 21:14:37 GMT

>>Is 10000 bigger than 10050? >>Yes, 10000 is bigger than 10050. >But even the mighty ChatGPT often can't do simple math GPT is bad at math because BPE input compression obfuscates individual digits. https://bbot.org/etc/gpt-math.png You'd be bad at math too if every number was scrambled. The graph is from page 22 of the GPT-3 paper from 2020. https://arxiv.org/abs/2005.14165 Even with 175 billion parameters it can't reliably do four digit addition. An example from 4 days ago of ChatGPT being as bad as you'd expect at string reversal: https://news.ycombinator.com/item?id=35297183 (Although, I…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by Aka457

Sun, 26 Mar 2023 13:01:49 GMT

Yes, or math, for example you can tell him to "add a+b where a=3.274914 and b=2.4847". I doubt this precise numbers are in the dataset of chatGPT and yet it can find the answer. According to this paper it seems to have gain the ability as the size of the model increased (page 21): https://arxiv.org/pdf/2005.14165.pdf " small models do poorly on all of these tasks – even the 13 billion parameter model (the second largest after the 175 billion full GPT-3) can solve 2 digit addition and subtraction only half the time, and all other operations less than 10% of the time." That's crazy.

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by DavidSJ

Sat, 11 Mar 2023 17:24:09 GMT

GPT 3.5 likely differs from the original GPT 3 by more than instruction fine-tuning. For example, it was probably retrained under Chinchilla scaling laws [1], with a lot more data and maybe a somewhat smaller parameter count. There are many variants of GPT-3 and GPT-3.5, and based on the performance numbers in Meta’s paper, it looks like they’re comparing against the very first version of GPT-3 from 2020. [2] [1] https://arxiv.org/abs/2203.15556 [2] https://arxiv.org/abs/2005.14165

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by spion

Wed, 08 Mar 2023 07:59:19 GMT

There is some commentary in the GPT-3 paper https://arxiv.org/pdf/2005.14165.pdf (figure 3.10 and table 3.10) note: I may have extrapolated a bit more than strictly correct from those two bits, but the accuracy being about 50% indicates carrying is the issue. GPT-3 is at around carrying twice, but hasn't generalized full carrying yet.

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by CSMastermind

Wed, 01 Mar 2023 07:55:43 GMT

https://arxiv.org/pdf/2005.14165.pdf WebText and WebText2 referenced in their papers are corpuses based on Reddit submissions which had a 22% weight in their training model. https://openwebtext2.readthedocs.io/en/latest/ This is larger than Wikipedia (3% weight) or either of their two book corpuses (8% each). The only other data included was a filtered set from Common Crawl (weighted 60%).

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by eternalban

Fri, 24 Feb 2023 16:09:25 GMT

I just looked at the wikipage for zero-shot learning and it wasn't very clear about why it is "zero". The paper for GPT-3 however is quite clear: "(c) “zero-shot” learning, where no demonstrations are allowed and only an instruction in natural language is given to the model" [page 5 of pdf] https://arxiv.org/pdf/2005.14165.pdf

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by luminati

Wed, 22 Feb 2023 06:42:25 GMT

Citation: The OG gpt-3 paper has ~23 authors and two with Chinese last names (and both are Asian Americans) :) https://arxiv.org/abs/2005.14165 Jokes aside - you can replace Chinese with Indian too and still the same answer holds. There is something magical about Silicon Valley and it is definitely not about the salaries. Fwiw, bytedance pays in the 7 figures usd in Beijing!

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by cypress66

Tue, 21 Feb 2023 03:25:44 GMT

What you're saying is outright false. If read llm papers you'll see that a lot of benchmarks used are related to math and logic. GPT3 paper as an example https://arxiv.org/abs/2005.14165 I recommend you read that paper as it answers a lot of misconceptions you may have about llms.

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by YeGoblynQueenne

Fri, 17 Feb 2023 00:27:33 GMT

Yeah, those models need to be retrained often. Not to update an index, but to keep up to date with the new content on the internet, and of course to create new, improved models with more parameters: note that after GPT, we had GPT-2, GPT-3, GPT-3.5 (a.k.a. ChatGPT) and rumours are we're now at GPT-4 (powering Bing search). Meanwhile, each of those models was trained in multiple versions, with different numbers of parameters; for example, the GPT-3 that first broke the barrier of hype was the largest, at 175 billion parameters, of four or five models. Plus, it seems reasonable that OpenAI…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by CSMastermind

Wed, 08 Feb 2023 20:48:48 GMT

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by vikashrungta

Wed, 08 Feb 2023 16:19:46 GMT

I posted a list of papers on twitter, and will be posting a summary for each of them as well. here is the list https://twitter.com/vrungta/status/1623343807227105280 Unlocking the Secrets of AI: A Journey through the Foundational Papers by @vrungta (2023) 1. "Attention is All You Need" (2017) - https://arxiv.org/abs/1706.03762 (Google Brain) 2. "Generative Adversarial Networks" (2014) - https://arxiv.org/abs/1406.2661 (University of Montreal) 3. "Dynamic Routing Between Capsules" (2017) - https://arxiv.org/abs/1710.09829 (Google Brain) 4. "Unsupervised Representation Learning with Deep…

Paper and discussion history. A mention is not an endorsement.

Language Models are Few-Shot Learners — mentioned by sampo

Mon, 06 Feb 2023 23:05:16 GMT

> I get the sense that LLMs are trained on a very large data set all at once, but that it is difficult to incrementally update them with new data. In 2020, OpenAI wrote that training the 175B parameter GPT-3 took 3e23 fl-operations [1]. A $1600 NVIDIA GeForce RTX 4090 [2] can do 80 tflops (80 × 1e12 operations per second). So 1 GPU would need 120 years to train the model. With $2.4 million, you could buy 1500 GPUs, and train GPT-3 from scratch in 1 month. Partial continuous re-training should be way cheaper, if there is a way. But even training again from scratch every month seems feasible…

Paper and discussion history. A mention is not an endorsement.