Sequence to Sequence Learning with Neural Networks · HackerPapers discussions

Sequence to Sequence Learning with Neural Networks — mentioned by bob1029

Wed, 30 Apr 2025 18:14:43 GMT

> Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes. In context of genetic programming and other non-traditional ML techniques, I've been having difficulty attempting to locate a simple fitness function that reliably proxies natural language string similarity due to this effect. For example, say you use something like common prefix length to measure how close a candidate's output string is to an objective string given an input string. The underlying learner will inevitably start doing things like repeating the input verbatim…

Paper and discussion history. A mention is not an endorsement.

Sequence to Sequence Learning with Neural Networks — mentioned by smoldesu

Mon, 30 Oct 2023 20:51:59 GMT

OpenAI was most certainly not the first organization to develop (or distribute) an LLM model: https://github.com/google-research/bert https://arxiv.org/abs/1810.04805 Failing that, Seq2seq predates OpenAI's incorporation: https://google.github.io/seq2seq/ https://github.com/google/seq2seq https://arxiv.org/abs/1409.3215

Paper and discussion history. A mention is not an endorsement.

Sequence to Sequence Learning with Neural Networks — mentioned by microtonal

Sun, 25 Jun 2023 18:28:56 GMT

This is basically how RNN encoder/decoder architectures worked. The encoder encoded the input as a single vector and the decoder would decode this into text (eg. for machine translation) [1]. However, fixed-length vectors generally required too much 'compression' to represent variable-length text, so people started adding attention mechanisms so that the decoder could also attend to the input text. And the seminal Transformer paper by Vaswani and others showed that you only need an attention mechanism and you could ditch the RNN (hence the title 'Attention is all you need') and here we are.…

Paper and discussion history. A mention is not an endorsement.

Sequence to Sequence Learning with Neural Networks — mentioned by std_badalloc

Tue, 01 Dec 2020 08:34:42 GMT

I think it was with the seq2seq paper of Sutskever, Vinyals and Le in 2014: https://arxiv.org/abs/1409.3215 People were doing a mix of learning from data and hand engineered solutions before this, but this was the first system learned end-to-end, afaik.

Paper and discussion history. A mention is not an endorsement.

Sequence to Sequence Learning with Neural Networks — mentioned by aglionby

Fri, 02 Mar 2018 19:45:37 GMT

Combining word embeddings to get a single representation of a sentence in general is a pretty hard problem. Point-wise averaging, addition or summation [0] works alright, but it's not great. Some more recent work looks a using RNNs to combine them; most of the literature I've seen is pretty application-specific, but try [1][2] for flavour. [0] http://www.aclweb.org/anthology/P/P08/P08-1028 [1] http://arxiv.org/abs/1409.3215 [2] http://arxiv.org/abs/1409.0473

Paper and discussion history. A mention is not an endorsement.

Sequence to Sequence Learning with Neural Networks — mentioned by deepnet

Mon, 10 Jul 2017 17:58:43 GMT

Geoff Hinton refers to thought vectors performing reasoning by analogy using algebra [1] in his Royal Society Lecture. The other widely reported vector algebras in a semantic space were discovered by Mikolov et al when producing ~300 dimensional vectors for a billion word Wikipedia corpus. If one performs vector algebra and ~= is near by cosine distance then using Mikolov's Vectors[3]. King - Man + Woman ~= Queen France - Paris + Gernmany ~= Berlin Surprisingly this works for other modalities, Chintala, Radford & Metz found a latent semantic space in images, that adds vectors for glasses or…

Paper and discussion history. A mention is not an endorsement.

Sequence to Sequence Learning with Neural Networks — mentioned by nl

Wed, 04 May 2016 08:56:51 GMT

no it wouldn't and it doesn't. Either you would have to strip it down too much to fit on a phone, so it would lose performance (and it's far from state-of-the-art anyway, being an example) or it would not fit on your phone, unless you removed everything else from it (and even then). Yes, it is an example program. Nevertheless, to quote: "Even if you want to transform a sequence to a tree, for example to generate a parsing tree, the same model as above can give state-of-the-art results, as demonstrated in Vinyals & Kaiser et al., 2014" Language modelling isn't the same as translation, but it…

Paper and discussion history. A mention is not an endorsement.

Sequence to Sequence Learning with Neural Networks — mentioned by deepnet

Sat, 22 Aug 2015 21:51:31 GMT

This work indicates that language translation is vector addition on a lower dimensional semantic manifold that neural networks converge on, derived from the high dimensional whole vocabulary context vectors of the input. It is extended by : Sequence to Sequence Learning with Neural Networks by Sutskever, Vinyals & Le http://arxiv.org/abs/1409.3215 which adds LTSM, a trainable deep neural net type of long term memory - which allows whole sentences to be digested and translated. " Our main result is that on an English to French translation task from the WMT'14 dataset, the translations produced…

Paper and discussion history. A mention is not an endorsement.

Sequence to Sequence Learning with Neural Networks — mentioned by ogrisel

Thu, 15 Jan 2015 09:01:51 GMT

New neural machine translation architectures are experimenting with pairs of neural encoders / decoders, one pair for each language and a shared language independent vector space for the meaning of all words: http://arxiv.org/abs/1406.1078 http://arxiv.org/abs/1409.3215 http://arxiv.org/abs/1410.8206 So the total number of models is still linear with the number of languages. I do not know whether this new generation of translation models is leveraged by the google translation app though. Also pairs of languages for which their are big amount of parallel training data will still be favored.

Paper and discussion history. A mention is not an endorsement.