HackerPapers

arXiv:1602.02410cs.CL

Exploring the Limits of Language Modeling

Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, Yonghui Wu

Abstract

In this work we explore recent advances in Recurrent Neural Networks for large scale Language Modeling, a task central to language understanding. We extend current models to deal with two key challenges present in this task: corpora and vocabulary sizes, and complex, long term structure of language. We perform an exhaustive study on techniques such as character Convolutional Neural Networks or Long-Short Term Memory, on the One Billion Word Benchmark. Our best single model significantly improves state-of-the-art perplexity from 51.3 down to 30.0 (whilst reducing the number of parameters by a factor of 20), while an ensemble of models sets a new record by improving perplexity from 41.0 down to 23.7. We also release these models for the NLP and ML community to study and improve upon.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language

On Hacker News

Linked by 4 identifiable commenters in 5 comments across 5 threads, from Feb 2016 to Jun 2017. Found in 2 HN submissions from Feb 2016 to Feb 2016. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. Random Generation of English Sentences (1961) [pdf] 1 comment

    • This is the state of the art text generation result from Google: https://arxiv.org/abs/1602.02410 * < S > With even more new technologies coming onto the market quickly during the past three years , an increasing number of companies now must tackle the ever-changing and…

      visargaView on HNParent context

  2. Using Ordered Markov Chains and User Information to Speed Up Password Cracking 1 comment

    • If I understand you correctly you are comparing Markov Chains on words to a RNN over characters. That's not fair. This paper shows a large LSTM outperform n-gram models: "In this paper we have shown that RNN LMs can be trained on large amounts of data, and outperform competing…

      ma2rtenView on HNParent context

  3. New P2 Instance Type for Amazon EC2 – Up to 16 GPUs 1 comment

    • $0.9 per K80 GPU per hour, while expensive, opens up so many opportunities - especially when you can get a properly connected machine. Just as an example of the change this entails for deep learning, the recent "Exploring the Limits of Language Modeling"[1] paper from Google…

      SmerityView on HNParent context

  4. Researchers achieve speech recognition milestone 1 comment

    • Most modern language modelling is RNN based now; eg: https://arxiv.org/abs/1602.02410 (which I think is SOTA).

      nlView on HNParent context

  5. Running your models in production with TensorFlow Serving 1 comment

    • Which data are you after? The ImageNet data is public and they released the pretrained model. They've promised to release (or already have released) the models for Exploring the Limits of Language Modeling[1] which was trained on the 1 B Word Benchmark corpus[2] which is also…

      nlView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.