HackerPapers

arXiv:2112.04426cs.CL

Improving language models by retrieving from trillions of tokens

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, Laurent Sifre

Abstract

We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a $2$ trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25$\times$ fewer parameters. After fine-tuning, RETRO performance translates to downstream knowledge-intensive tasks such as question answering. RETRO combines a frozen Bert retriever, a differentiable encoder and a chunked cross-attention mechanism to predict tokens based on an order of magnitude more data than what is typically consumed during training. We typically train RETRO from scratch, yet can also rapidly RETROfit pre-trained transformers with retrieval and still achieve good performance. Our work opens up new avenues for improving language models through explicit memory at unprecedented scale.

arXiv comments
Fix incorrect reported numbers in Table 14
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.LGMachine Learning

On Hacker News

Linked by 10 identifiable commenters in 15 comments across 15 threads, from Dec 2021 to Nov 2024.

Comments linking this paper, per year.

Discussed in comments15 threads

  1. LLMs have reached a point of diminishing returns 1 comment

    • "a larger model with RAG etc is still better than a small one" This paper from DeepMind a few years ago offers a counter example to this claim. https://arxiv.org/abs/2112.04426

      williamtraskView on HNParent context

  2. Lamini Memory Tuning: 10x Fewer Hallucinations 1 comment

    • Bit late to the party and I'm not into the AI scene, but from glossing over the three key papers they cite to describe their model, my take is as follows. The idea from LoRA[1] is to take pre-trained, dense weights W_0 for a model and adapt them to new training data by using the…

      magicalhippoView on HNParent context

  3. LLM: Transformer Is Linear 1 comment

    • Related papers: - https://arxiv.org/abs/2307.06962 - https://arxiv.org/abs/2309.12288 - https://arxiv.org/abs/2112.04426 - https://huggingface.co/papers/2405.14860

      williamtraskView on HNParent context

  4. Ask HN: What things are happening in ML that we can't hear over the din of LLMs? 1 comment

    • So, from the perspective I have within the subfield I work in, explainable AI (XAI), we're seeing a bunch of fascinating developments. First, as you mentioned, Rudin continues to prove that the reason for using AI/ML is that we don't understand the problem well enough; otherwise…

      babel_View on HNParent context

  5. PaLM 2 Technical Report [pdf] 1 comment

    • I agree that if training data is what matters, it is likely that no one can compete with Google with Google Books, which scanned 25 million volumes (source: http://www.nytimes.com/2015/10/29/arts/international/google-...), which is approximately all the books. DeepMind's RETRO…

      sanxiynView on HNParent context

  6. Neural Networks and the Chomsky Hierarchy 1 comment

    • Examples of such architectures: RETRO (Retrieval-Enhanced Transformer) https://arxiv.org/abs/2112.04426 And much older (2016): Differential Neural Network https://www.deepmind.com/blog/differentiable-neural-computer...

      halflingsView on HNParent context

  7. Anthropic's Claude is said to improve on ChatGPT, but still has limitations 1 comment

    • There are already open source LLMs with comparable parameter counts (Facebook's OPT-175B, BLOOM), but you'll need ~10x A100 GPUs to run them (which would cost ~$100K+). I suspect a big part of why stable diffusion managed to consume so much mindshare is that it can run on…

      antimatter15View on HNParent context

  8. How does GPT obtain its ability? Tracing emergent abilities of language models 1 comment

    • For anyone interested it's called retrieval transformer. Here is example of one from Google: RETRO[1][2] 1. https://arxiv.org/abs/2112.04426 2. https://jalammar.github.io/illustrated-retrieval-transformer...

      lossoloView on HNParent context

  9. AlphaCode Attention Visualization 1 comment

    • > Has Google/Alphabet publicly released any of their AI models yet? You mean NLP field changing models from Google like BERT [1]? or Transformers paper [2]? or T5 model [3] (used by company doing ChatGPT like search currently on the front page on HN)? 1.…

      lossoloView on HNParent context

  10. Show HN: Web search using a ChatGPT-like model that can cite its sources 1 comment

    • 11B parameters, so I guess it's FLAN-T5[1]? combined with retrieval transformer (something like RETRO[2])? 1. https://arxiv.org/abs/2210.11416 2. https://arxiv.org/abs/2112.04426

      lossoloView on HNParent context

  11. RETRO is fast 1 comment

    • If you are talking about Figure 1 of https://arxiv.org/abs/2112.04426, you are reading it wrong. Y axis is bits-per-byte and lower is better.

      sanxiynView on HNParent context

  12. AlphaCode as a dog speaking mediocre English 1 comment

    • > To evaluate CoPilot, we should ask questions that are unusual enough they can't be answered through regurgitation of the training corpus. Exactly! It's great that Copilot can generate correct code for a given question, but we cannot gauge its full capability unless we try it…

      ahgamutView on HNParent context

  13. Competitive Programming with AlphaCode 1 comment

    • I find almost every new advance in deep learning is accompanied by contrasting comments: it's either "AI will soon automate programming/<insert task here>", or "let me know when AI can actually do <some-difficult-task>". There are many views on this spectrum, but these two are…

      ahgamutView on HNParent context

  14. WebGPT: Browser-assisted question-answering with human feedback 1 comment

    • I think large frozen LMs are going to become the "operating system" of tomorrow. They are almost universal task solvers, with the right prompt or verbalizer. The trend now is to make them multi language, multi modal, multi task, multi memory ([1] and this paper), and scale…

      visargaView on HNParent context

  15. Gopher – A 280B parameter language model 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.