End-to-End Attention-based Large Vocabulary Speech Recognition — mentioned by kastnerkyle

Mon, 07 Mar 2016 14:35:17 GMT

It is a dense read, but you might have a look at [1]. This is how attention is implemented in Theano. Basically the key is going "3D" per timestep (where 2D per timestep is the norm when doing minibatch training), then taking a weighted sum over the correct axis to get the right size to combine with the RNN state. Short summary: I: input length M: minibatch size (same for input and output) H_in: input hidden size (arbitrary/user selected) H_out: output hidden size (arbitrary/user selected) C: attention feature size (arbitrary/user selected) Looking at the decode/generator RNN, "context" comes…

Paper and discussion history. A mention is not an endorsement.

End-to-End Attention-based Large Vocabulary Speech Recognition · HackerPapers discussions

End-to-End Attention-based Large Vocabulary Speech Recognition — mentioned by kastnerkyle