HackerPapers

arXiv:2001.09977cs.CL

Towards a Human-like Open-Domain Chatbot

Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, Quoc V. Le

Abstract

We present Meena, a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations. This 2.6B parameter neural network is simply trained to minimize perplexity of the next token. We also propose a human evaluation metric called Sensibleness and Specificity Average (SSA), which captures key elements of a human-like multi-turn conversation. Our experiments show strong correlation between perplexity and SSA. The fact that the best perplexity end-to-end trained Meena scores high on SSA (72% on multi-turn evaluation) suggests that a human-level SSA of 86% is potentially within reach if we can better optimize perplexity. Additionally, the full version of Meena (with a filtering mechanism and tuned decoding) scores 79% SSA, 23% higher in absolute SSA than the existing chatbots we evaluated.

arXiv comments
38 pages, 12 figures
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.LGMachine Learning cs.NENeural and Evolutionary Computing stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 4 identifiable commenters in 4 comments across 4 threads, from Apr 2020 to Mar 2023. Found in 3 HN submissions from Jan 2020 to Oct 2020. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. Google internally developed chatbots like ChatGPT years ago 1 comment

    • Wow, what a secret! It's not like Google has been consistently writing papers about this the whole time or anything. Oh wait: 2022 LaMDA: https://arxiv.org/abs/2201.08239 2020 chat bot: https://arxiv.org/abs/2001.09977 2018 bert: https://arxiv.org/abs/1810.04805 2017 LLM…

      advisedwangView on HNParent context

  2. Pathways Language Model (PaLM): Scaling to 540B parameters 1 comment

    • Crazy impressive! A question about the training data: anyone familiar with this line of work know what social media platforms the "conversation" data component of the training set came from? There's a datasheet that points to prior work https://arxiv.org/abs/2001.09977, which…

      nickvincentView on HNParent context

  3. LaMDA: Google's New Conversation Technology 1 comment
  4. Generating MIDI Music with GPT-2 1 comment

    • I previously tried an approach which uses DRL for feedback but I couldn't quite get it to work: https://www.gwern.net/GPT-2-preference-learning At the moment, it probably would be more practical to train a model to predict ratings and use that to screen generated samples or…

      gwernView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.