A Convolutional Encoder Model for Neural Machine Translation · HackerPapers discussions

A Convolutional Encoder Model for Neural Machine Translation — mentioned by jgehring

Wed, 10 May 2017 09:54:40 GMT

Yes, there have been a couple of attempts to use CNNs for translation already, but none of them outperformed big and well-tuned LSTM systems. We propose an architecture that is fast to run, easy to optimize and can scale to big networks, and could thus be used as a base architecture for future research. There are a couple of contributions in the paper (https://arxiv.org/abs/1705.03122) apart from demonstrating the feasibility of CNNs for translation, e.g. the multi-hop attention in combination with a CNN language model, the wiring of the CNN encoder[1], or an initialization scheme for GLUs…

Paper and discussion history. A mention is not an endorsement.

A Convolutional Encoder Model for Neural Machine Translation — mentioned by jhj

Mon, 10 Apr 2017 19:35:50 GMT

ConvNets can handle sequences just as well (cf. TDNNs), you just treat time as another dimension. Whether you want to call this a "frame" or not is subject to question. https://arxiv.org/pdf/1611.02344.pdf With an explicit (potentially arbitrarily large) memory component, they can also capture long-term dependencies that don't have to be maintained in, say, the convolution parameters between presentations of data. This is one thing we want to pursue with the Faiss library at FAIR. Regardless of how something is implemented, it's the what that is important. What's the actual algorithm or…

Paper and discussion history. A mention is not an endorsement.