One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling · HackerPapers discussions

One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling — mentioned by alquemist

Tue, 10 Nov 2020 13:30:47 GMT

Intelligent trial and error. 1. Transformers are an extension of the attention mechanism, a well known LSTM extension that worked well for machine translation (2014, https://arxiv.org/abs/1409.0473). A transformer model is essentially building a multi headed attention module, analogue to CNNs, then stacking several layers of them, analogue to CNNs / stacked LSTMs. (1998, http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf) 2. Transfomers use residual blocks, which were introduced by the ResNet CNN arhitecture (2015, https://arxiv.org/abs/1512.03385). At the time, ResNet was topping the…

Paper and discussion history. A mention is not an endorsement.

One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling — mentioned by Danylon

Fri, 21 Apr 2017 03:26:33 GMT

Using an innocuous encoding of a word is a form of encryption. People who expect to be under surveillance agree on a set of code words to denote illegal things. Though hard, there are multiple ways to semi-automatically break such a linguistic encryption. Imputation. [1] Remove a word from a sentence then try to predict it from its surrounding context. "when I get home tonight, i vape a ___ then space out". Assign predicted probabilities to imputed word ":leaf emoji:" ["marijuana cigarette", "electronic cigarette", "cigar"]. Active learning. Seed the algorithm with expert knowledge from law…

Paper and discussion history. A mention is not an endorsement.

One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling — HN submission

Wed, 12 Mar 2014 15:18:32 GMT

Billion Word Benchmark for Statistical Language Modeling from Google

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.