Character-Level Language Modeling with Deeper Self-Attention — mentioned by rmrfstar

Sun, 08 Nov 2020 06:45:07 GMT

It removes you from the list of people who get pwned with a rainbow table from leaked passwords, but that's not much security. English has only got ~1.2 bits of entropy per character. [1] [1] Table 1 in https://arxiv.org/pdf/1808.04444.pdf

Paper and discussion history. A mention is not an endorsement.

Character-Level Language Modeling with Deeper Self-Attention — mentioned by visarga

Fri, 08 Mar 2019 02:56:11 GMT

Definitions are not enough to fully capture the meaning of a word. In order to do that you need full language modelling and to ground words into other sensory modalities, plus the word in relation to actions taken in various situations when the word was used. GPT-2 (of recent OpenAI fame) uses 1.5 billion parameters and, though capable of interesting results, is far from human level. It also uses just text so it's incomplete. https://blog.openai.com/better-language-models/ Another interesting metric is Bits Per Character - BPC. The state of the art is around 1.06 on English Wikipedia. This…

Paper and discussion history. A mention is not an endorsement.

Character-Level Language Modeling with Deeper Self-Attention · HackerPapers discussions

Character-Level Language Modeling with Deeper Self-Attention — mentioned by rmrfstar

Character-Level Language Modeling with Deeper Self-Attention — mentioned by visarga