Improving language models by retrieving from trillions of tokens · HackerPapers discussions

Improving language models by retrieving from trillions of tokens — mentioned by williamtrask

Sun, 10 Nov 2024 01:20:23 GMT

"a larger model with RAG etc is still better than a small one" This paper from DeepMind a few years ago offers a counter example to this claim. https://arxiv.org/abs/2112.04426

Paper and discussion history. A mention is not an endorsement.

Improving language models by retrieving from trillions of tokens — mentioned by magicalhippo

Fri, 14 Jun 2024 16:02:35 GMT

Bit late to the party and I'm not into the AI scene, but from glossing over the three key papers they cite to describe their model, my take is as follows. The idea from LoRA[1] is to take pre-trained, dense weights W_0 for a model and adapt them to new training data by using the weights W = W_0 + BA for inference. The key is that the matrices A and B have a very low rank[2] compared to W_0. Training is then done by treating W_0 as constant, and only updating the A and B matrices. The idea from MoE[3] is to pick just a few "experts" from a large number using Softmax, and insert them between…

Paper and discussion history. A mention is not an endorsement.

Improving language models by retrieving from trillions of tokens — mentioned by williamtrask

Sat, 25 May 2024 03:51:49 GMT

Related papers: - https://arxiv.org/abs/2307.06962 - https://arxiv.org/abs/2309.12288 - https://arxiv.org/abs/2112.04426 - https://huggingface.co/papers/2405.14860

Paper and discussion history. A mention is not an endorsement.

Improving language models by retrieving from trillions of tokens — mentioned by babel_

Thu, 28 Mar 2024 13:59:52 GMT

So, from the perspective I have within the subfield I work in, explainable AI (XAI), we're seeing a bunch of fascinating developments. First, as you mentioned, Rudin continues to prove that the reason for using AI/ML is that we don't understand the problem well enough; otherwise we wouldn't even think to use it! So, pushing our focus to better understand the problem, and then levy ML concepts and techniques (including "classical AI" and statistical learning), we're able to make something that not only outperforms some state-of-the-art in most metrics, but often even is much less resource…

Paper and discussion history. A mention is not an endorsement.

Improving language models by retrieving from trillions of tokens — mentioned by sanxiyn

Thu, 11 May 2023 01:25:34 GMT

I agree that if training data is what matters, it is likely that no one can compete with Google with Google Books, which scanned 25 million volumes (source: http://www.nytimes.com/2015/10/29/arts/international/google-...), which is approximately all the books. DeepMind's RETRO paper https://arxiv.org/abs/2112.04426 mentions a dataset called MassiveText, which includes 20 million books of 3T tokens. So we know Google is using Google Books, since there is simply no other source of 20 million books. Also as far as I know 3T tokens is more than publicly known to be used by anyone so far: Google…

Paper and discussion history. A mention is not an endorsement.

Improving language models by retrieving from trillions of tokens — mentioned by halflings

Mon, 23 Jan 2023 17:39:21 GMT

Examples of such architectures: RETRO (Retrieval-Enhanced Transformer) https://arxiv.org/abs/2112.04426 And much older (2016): Differential Neural Network https://www.deepmind.com/blog/differentiable-neural-computer...

Paper and discussion history. A mention is not an endorsement.

Improving language models by retrieving from trillions of tokens — mentioned by antimatter15

Tue, 10 Jan 2023 22:00:37 GMT

There are already open source LLMs with comparable parameter counts (Facebook's OPT-175B, BLOOM), but you'll need ~10x A100 GPUs to run them (which would cost ~$100K+). I suspect a big part of why stable diffusion managed to consume so much mindshare is that it can run on ordinary consumer hardware. On that point, I would be excited about an open-source RETRO (https://arxiv.org/pdf/2112.04426.pdf) model with comparable performance to GPT-3 that could run on consumer hardware with an NVMe SSD.

Paper and discussion history. A mention is not an endorsement.

Improving language models by retrieving from trillions of tokens — mentioned by lossolo

Wed, 14 Dec 2022 23:53:06 GMT

For anyone interested it's called retrieval transformer. Here is example of one from Google: RETRO[1][2] 1. https://arxiv.org/abs/2112.04426 2. https://jalammar.github.io/illustrated-retrieval-transformer...

Paper and discussion history. A mention is not an endorsement.

Improving language models by retrieving from trillions of tokens — mentioned by lossolo

Thu, 08 Dec 2022 23:32:48 GMT

> Has Google/Alphabet publicly released any of their AI models yet? You mean NLP field changing models from Google like BERT [1]? or Transformers paper [2]? or T5 model [3] (used by company doing ChatGPT like search currently on the front page on HN)? 1. https://arxiv.org/abs/1810.04805 code+models: https://github.com/google-research/bert 2. https://arxiv.org/abs/2112.04426 3. https://arxiv.org/abs/1910.10683 code+models: https://github.com/google-research/text-to-text-transfer-tra...

Paper and discussion history. A mention is not an endorsement.

Improving language models by retrieving from trillions of tokens — mentioned by lossolo

Thu, 08 Dec 2022 22:46:16 GMT

11B parameters, so I guess it's FLAN-T5[1]? combined with retrieval transformer (something like RETRO[2])? 1. https://arxiv.org/abs/2210.11416 2. https://arxiv.org/abs/2112.04426

Paper and discussion history. A mention is not an endorsement.

Improving language models by retrieving from trillions of tokens — mentioned by sanxiyn

Mon, 04 Jul 2022 02:03:31 GMT

If you are talking about Figure 1 of https://arxiv.org/abs/2112.04426, you are reading it wrong. Y axis is bits-per-byte and lower is better.

Paper and discussion history. A mention is not an endorsement.

Improving language models by retrieving from trillions of tokens — mentioned by ahgamut

Sun, 06 Feb 2022 20:09:51 GMT

> To evaluate CoPilot, we should ask questions that are unusual enough they can't be answered through regurgitation of the training corpus. Exactly! It's great that Copilot can generate correct code for a given question, but we cannot gauge its full capability unless we try it on a range of different questions, especially ones that are not found in the training data. I mentioned this in the other AlphaCode post: It would be nice to know how "unusual" a given question is. Maybe an exact replica exists in the training data, or a solution exists but in a different programming language, or a…

Paper and discussion history. A mention is not an endorsement.

Improving language models by retrieving from trillions of tokens — mentioned by ahgamut

Wed, 02 Feb 2022 20:31:44 GMT

I find almost every new advance in deep learning is accompanied by contrasting comments: it's either "AI will soon automate programming/<insert task here>", or "let me know when AI can actually do <some-difficult-task>". There are many views on this spectrum, but these two are sure to be present in every comment section. IIUC, AlphaCode was trained on Github code to solve competitive programming challenges on Codeforces, some of which are "difficult for a human to do". Suppose AlphaCode was trained on Github code that contains the entire set of solutions on Codeforces, is it actually doing…

Paper and discussion history. A mention is not an endorsement.

Improving language models by retrieving from trillions of tokens — mentioned by visarga

Mon, 24 Jan 2022 12:15:09 GMT

I think large frozen LMs are going to become the "operating system" of tomorrow. They are almost universal task solvers, with the right prompt or verbalizer. The trend now is to make them multi language, multi modal, multi task, multi memory ([1] and this paper), and scale computation with the problem complexity instead of keeping it fixed. After that the next step I think is to make LMs that map arguments to functions from unstructured input, externally run the function and use the result in the next token predictions. LMs could write functions and execute them as regular code. They could…

Paper and discussion history. A mention is not an endorsement.

Improving language models by retrieving from trillions of tokens — mentioned by geoffreyirving

Thu, 09 Dec 2021 11:57:46 GMT

https://arxiv.org/abs/2112.04426 https://arxiv.org/abs/2112.04359 Gopher soon. :)

Paper and discussion history. A mention is not an endorsement.