PaLM: Scaling Language Modeling with Pathways · HackerPapers discussions

PaLM: Scaling Language Modeling with Pathways — mentioned by godelski

Wed, 24 May 2023 06:16:47 GMT

> if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall Btw, PALM2 has far fewer parameters than V1. > The largest model in the PaLM 2 family, PaLM 2-L, is significantly smaller than the largest PaLM model but uses more training compute. Our evaluation results show that PaLM 2 models significantly outperform PaLM on a variety of tasks, including natural language generation, translation, and reasoning. These results suggest that model scaling is not the only way to improve performance. From the leak we know that the large version is 340B…

Paper and discussion history. A mention is not an endorsement.

PaLM: Scaling Language Modeling with Pathways — mentioned by famouswaffles

Tue, 16 May 2023 09:16:50 GMT

You can look at any training run comparing multiple parameter sizes on the dataset. https://arxiv.org/abs/2204.02311 If you're increasing parameter size, it's a no brainer to increase data too as that will still also increase performance. The point is that for any arbitrary performance x, the data required to reach it reduces with size.

Paper and discussion history. A mention is not an endorsement.

PaLM: Scaling Language Modeling with Pathways — mentioned by nopinsight

Thu, 30 Mar 2023 08:02:53 GMT

Engineering such a system is a harder challenge than many types of research. Even the mighty Google, the leader in AI research by many metrics, is catching up. Another example is Meta only finishing OPT-175B, a near equivalent of GPT-3, two years after it. —— GPT-4 got much better results on many benchmarks than PaLM, Google's largest published model [1]. PaLM itself is probably quite a bit better than LamDa in several tasks, according to a chart and a couple of tables here: https://arxiv.org/abs/2204.02311 It's unclear that Google currently has an internal LLM as good as GPT-4. If they do…

Paper and discussion history. A mention is not an endorsement.

PaLM: Scaling Language Modeling with Pathways — mentioned by nopinsight

Wed, 29 Mar 2023 02:19:33 GMT

Engineering such a system is a harder challenge than many types of research. Even the mighty Google, the leader in AI research by many metrics, is catching up. Another example is Meta only finishing OPT-175B, a near equivalent of GPT-3, two years after it. —— Added to reply: GPT-4 got much better results on many benchmarks than PaLM, Google's largest published model [1]. PaLM itself is probably quite a bit better than LamDa in several tasks, according to a chart and a couple of tables here: https://arxiv.org/abs/2204.02311 It's unclear that Google currently has an internal LLM as good as…

Paper and discussion history. A mention is not an endorsement.

PaLM: Scaling Language Modeling with Pathways — mentioned by nopinsight

Sat, 25 Mar 2023 11:05:47 GMT

GPT-4 got much better results on many benchmarks than PaLM, Google's largest published model [1]. PaLM itself is probably quite a bit better than LamDa in several tasks, according to a chart and a couple of tables here: https://arxiv.org/abs/2204.02311 It's unclear that Google currently has an internal LLM as good as GPT-4. If they do, they are keeping quiet about it, which seems quite unlikely given the repercussion. [1] GPT-4's benchmark results vs PaLM: https://openai.com/research/gpt-4

Paper and discussion history. A mention is not an endorsement.

PaLM: Scaling Language Modeling with Pathways — mentioned by nl

Sun, 04 Dec 2022 01:58:19 GMT

So this shows you have a basic understanding of how CNNs represent data for classification tasks. This is excellent! However, you might not be aware of the current research into "Emergent Phenomena in Large Language Models"[1]. Work such as PaLM[2] show that language models evolve the ability to "reason" over the internal data representation. Minerva explicitly breaks down that ability by "comb[ining] several techniques, including few-shot prompting, chain of thought or scratchpad prompting, and majority voting, to achieve state-of-the-art performance on STEM reasoning tasks." This lets it…

Paper and discussion history. A mention is not an endorsement.

PaLM: Scaling Language Modeling with Pathways — mentioned by joseda-hg

Mon, 07 Nov 2022 03:17:15 GMT

https://arxiv.org/pdf/2204.02311.pdf#page38 I believe we're already at the footsteps of AI understanding cultural references "Input: Michael is at that really famous museum in France looking at its most famous painting. However, the artist who made this painting just makes Michael think of his favorite cartoon character from his childhood. What was the country of origin of the thing that the cartoon character usually holds in his hand? Model Output: The most famous painting in the Louvre is the Mona Lisa. The artist who made the Mona Lisa is Leonardo da Vinci. Leonardo da Vinci is also the…

Paper and discussion history. A mention is not an endorsement.

PaLM: Scaling Language Modeling with Pathways — mentioned by 1024core

Thu, 13 Oct 2022 01:44:53 GMT

See the "joke explaining" examples on Page 36 of the paper: https://arxiv.org/abs/2204.02311 and tell me if it's just "passable garbled nonsense".

Paper and discussion history. A mention is not an endorsement.

PaLM: Scaling Language Modeling with Pathways — mentioned by zone411

Sun, 02 Oct 2022 02:01:47 GMT

Note: 540B parameters -> Google Research's PaLM (https://arxiv.org/abs/2204.02311). GSM8K = grade school math word problems DROP = reading comprehension benchmark requiring discrete reasoning over paragraphs OpenBookQA = question-answering dataset modeled after open book exams for assessing human understanding of a subject. 5,957 multiple-choice elementary-level science questions ANLI-A3 = adversarial benchmark designed to be challenging to current state-of-the-art models

Paper and discussion history. A mention is not an endorsement.

PaLM: Scaling Language Modeling with Pathways — mentioned by the8472

Wed, 27 Jul 2022 15:05:12 GMT

> and no one is actually testing what proportions of results those are The idea that models could possibly overfit the training data is hardly a new idea. It's standard practice to test for that. Check section 7 of the PaLM paper for example. https://arxiv.org/abs/2204.02311

Paper and discussion history. A mention is not an endorsement.

PaLM: Scaling Language Modeling with Pathways — HN submission

Tue, 21 Jun 2022 14:05:15 GMT

Palm: Scaling Language Modeling with Pathways

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

PaLM: Scaling Language Modeling with Pathways — mentioned by upwardbound

Mon, 13 Jun 2022 07:16:13 GMT

I think the model does have short-term memory, but not long-term memory, if I'm understanding it correctly. From https://aibusiness.com/document.asp?doc_id=777540 : "LaMDA2 was trained on Google’s Pathways Language Model (PaLM), which has 540 billion parameters." And then looking at the PaLM paper https://arxiv.org/pdf/2204.02311.pdf on page 5 we have: "PaLM uses a standard Transformer model architecture (Vaswani et al., 2017) in a decoder-only setup (i.e., each timestep can only attend to itself and past timesteps)"

Paper and discussion history. A mention is not an endorsement.

PaLM: Scaling Language Modeling with Pathways — mentioned by londons_explore

Mon, 23 May 2022 22:58:18 GMT

All it takes is one 'trick' to give these models the ability to do reasoning. Like for example the discovery that language models get far better at answering complex questions if asked to show their working step by step with chain of thought reasoning as in page 19 of the PaLM paper [1]. Worth checking out the explanations of novel jokes on page 38 of the same paper. While it is, like you say, all statistics, if it's indistinguishable from valid reasoning, then perhaps it doesn't matter. [1]: https://arxiv.org/pdf/2204.02311.pdf

Paper and discussion history. A mention is not an endorsement.

PaLM: Scaling Language Modeling with Pathways — HN submission

Fri, 22 Apr 2022 01:01:20 GMT

Palm: Scaling Language Modeling with Pathways

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

PaLM: Scaling Language Modeling with Pathways — mentioned by timpetri

Sun, 10 Apr 2022 22:40:06 GMT

Question related to the Chinchilla paper[0], which says that optimal amount of training data for ~500B, 1T, and 10T param models are 11T, 21.2T, 216.2T tokens, respectively. The PaLM paper[1] says it made use of 700B tokens. How many tokens of training data have humans produced across the entire internet, all our written works, etc? Is there such a thing as a 216 trillion token set? [0] https://arxiv.org/abs/2203.15556 [1] https://arxiv.org/abs/2204.02311

Paper and discussion history. A mention is not an endorsement.

PaLM: Scaling Language Modeling with Pathways — mentioned by moconnor

Sun, 10 Apr 2022 18:14:36 GMT

This is not the consensus among ML researchers. Transformers are showing strong generalisation[1] and their performance continues to surprise us as they scale[2]. The Socratic paper is not about “higher intelligence”, it’s about demonstrating useful behaviour purely by connecting several large models via language. [1] https://arxiv.org/abs/2201.02177 [2] https://arxiv.org/abs/2204.02311

Paper and discussion history. A mention is not an endorsement.

PaLM: Scaling Language Modeling with Pathways — HN submission

Sat, 09 Apr 2022 22:44:41 GMT

Palm: Scaling Language Modeling with Pathways [pdf]

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.