HackerPapers

arXiv:1808.04444cs.CL

Character-Level Language Modeling with Deeper Self-Attention

Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, Llion Jones

Abstract

LSTMs and other RNN variants have shown strong performance on character-level language modeling. These models are typically trained using truncated backpropagation through time, and it is common to assume that their success stems from their ability to remember long-term contexts. In this paper, we show that a deep (64-layer) transformer model with fixed context outperforms RNN variants by a large margin, achieving state of the art on two popular benchmarks: 1.13 bits per character on text8 and 1.06 on enwik8. To get good results at this depth, we show that it is important to add auxiliary losses, both at intermediate network layers and intermediate sequence positions.

arXiv comments
8 pages, 7 figures
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 2 identifiable commenters in 2 comments across 2 threads, from Mar 2019 to Nov 2020.

Comments linking this paper, per year.

Discussed in comments2 threads

  1. Finally: A usable and secure password policy backed by science 1 comment

    • It removes you from the list of people who get pwned with a rainbow table from leaked passwords, but that's not much security. English has only got ~1.2 bits of entropy per character. [1] [1] Table 1 in https://arxiv.org/pdf/1808.04444.pdf

      rmrfstarView on HNParent context

  2. What's the minimum number of words you'd need to define all other words? (2012) 1 comment

    • Definitions are not enough to fully capture the meaning of a word. In order to do that you need full language modelling and to ground words into other sensory modalities, plus the word in relation to actions taken in various situations when the word was used. GPT-2 (of recent…

      visargaView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.