HackerPapers

arXiv:2201.08239cs.CL

LaMDA: Language Models for Dialog Applications

Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, Quoc Le

Abstract

We present LaMDA: Language Models for Dialog Applications. LaMDA is a family of Transformer-based neural language models specialized for dialog, which have up to 137B parameters and are pre-trained on 1.56T words of public dialog data and web text. While model scaling alone can improve quality, it shows less improvements on safety and factual grounding. We demonstrate that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding. The first challenge, safety, involves ensuring that the model's responses are consistent with a set of human values, such as preventing harmful suggestions and unfair bias. We quantify safety using a metric based on an illustrative set of human values, and we find that filtering candidate responses using a LaMDA classifier fine-tuned with a small amount of crowdworker-annotated data offers a promising approach to improving model safety. The second challenge, factual grounding, involves enabling the model to consult external knowledge sources, such as an information retrieval system, a language translator, and a calculator. We quantify factuality using a groundedness metric, and we find that our approach enables the model to generate responses grounded in known sources, rather than responses that merely sound plausible. Finally, we explore the use of LaMDA in the domains of education and content recommendations, and analyze their helpfulness and role consistency.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence

On Hacker News

Linked by 8 identifiable commenters in 10 comments across 9 threads, from Feb 2022 to Mar 2023. Found in 3 HN submissions from Jan 2022 to Jul 2022. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments9 threads

  1. Google internally developed chatbots like ChatGPT years ago 1 comment

    • Wow, what a secret! It's not like Google has been consistently writing papers about this the whole time or anything. Oh wait: 2022 LaMDA: https://arxiv.org/abs/2201.08239 2020 chat bot: https://arxiv.org/abs/2001.09977 2018 bert: https://arxiv.org/abs/1810.04805 2017 LLM…

      advisedwangView on HNParent context

  2. Ask HN: Does Google use the text inside gdocs and Gmail for training AI models? 1 comment

    • If you're talking about Google Bard, they were very clear in the LaMDA 2 paper that they only used public sources. "...from public dialog data and other public web documents..." LaMDA 2 paper: https://arxiv.org/abs/2201.08239 My overview of Google Bard including dataset…

      touringaView on HNParent context

  3. ChatGPT produces made-up nonexistent references 1 comment

    • Google’s model LaMDA has a system for detecting and avoiding false info, it’s interesting: https://arxiv.org/pdf/2201.08239.pdf Basically they have a fact-checker layer that calls the internet and replaces incorrect facts in the model’s output. IOW this is a hard problem and GPT…

      theptipView on HNParent context

  4. Character.ai 2 comments

    • I guess it's probably something along the lines of "LaMDA: Language Models for Dialog Applications" (see https://arxiv.org/abs/2201.08239 ). Given that the three first authors are part of the team at Character.ai (see https://beta.character.ai/help)

      groarView on HNParent context

    • You're right, it seems to be "LaMDA" (see https://arxiv.org/abs/2201.08239 ). The three first authors are part of the Character.ai team.

      groarView on HNParent context

  5. Google Engineer on His Sentient AI Claim 1 comment

    • oh so that is the doc they shared with the higher-ups? as for your question, https://arxiv.org/pdf/2201.08239.pdf states 2 months of 1024 TPUv3 for a pre-training, and TPUv3 isn't even the most recent TPU hardware

      GalahiSimtamView on HNParent context

  6. Is LaMDA Sentient? – An Interview [pdf] 1 comment

    • I think that, in understanding LaMDA, it is important to understanding that this thing is definitely not sentient. Note that, when told to be Mt. Everest, it pretends to be the mountain in a way that is just as convincing: https://arxiv.org/pdf/2201.08239.pdf

      WaffleIronMakerView on HNParent context

  7. LaMDA is not sentient 1 comment

    • Here's a simple technical explanation for why this happened: * LaMDA is powered by a large language models. Large language models are trained to predict text completions (the text that will come after some input text) with lots of data, basically capturing the patterns in that…

      andreykView on HNParent context

  8. DeepMind: A Generalist Agent 1 comment

    • That Tim Urban piece is great. It's also an interesting time capsule in terms of which AI problems were and were not considered hard in 2015 (when the post was written). From the post: > Build a computer that can multiply two ten-digit numbers in a split second—incredibly easy.…

      edouard-harrisView on HNParent context

  9. I asked GPT-NeoX-20B a hundred arithmetic questions 1 comment

    • This already exists: Google's recently-published LaMDA dialogue model [1] is trained to consult a calculator for arithmetic questions and consistently succeeds at it. [1] https://arxiv.org/abs/2201.08239v2

      edouard-harrisView on HNParent context

Submitted to Hacker News

  • Joke written by an AI: “A basic program walked into a bar ”309 points136 comments

    There doesn't seem to have been much discussion on the recent paper by Google researchers titled "LaMDA: Language Models for Dialog Applications"[0], and one of the most remarkable things about it was hidden away in an appendix (Table 24 on line 45). It shows a conversation between a user and the AI being tested, where the user apparently tries to confuse the bot by getting it to write a joke (which is a very subjective and human skill) about a BASIC program (which requires a different mode of…

  • Open-Source LaMDA Model27 points0 comments

    An open-source implementation for the pre-training architecture of Google's LaMDA in PyTorch. The research paper outlines an autoregressive, decoder-only, GPT-like transformer language model. The transformer uses T5 relative positional bias in the attention layers and gated-GELU activation function in the feed-forward layers. The repository currently contains a script for basic training as well as Huggingface datasets and Weights & Biases integration. LaMDA research paper…

  • LaMDA: Language Models for Dialog Applications9 points0 comments

Connected through shared HN threads, not inferred similarity or a recommendation.