Neural Turing Machines

HN loves slop meetoo projects without questioning the details. With close inspection it is an overly complicated version of this paper [0] on neural turing machines. This concept was hyped up (because DeepMind) for a while at the time and in reality it amounted to nothing. This…

rvzOct 30, 2026View on HNParent context

So this is yet another rube goldberg machine. We have gone down this path already in [0] and this is the same thing but overly complicated. This reason is "because LLMs". Why? [0] https://arxiv.org/abs/1410.5401

rvzOct 29, 2026View on HNParent context

https://colah.github.io/posts/2015-08-Understanding-LSTMs/ https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436... https://papers.nips.cc/paper_files/paper/2012/file/c399862d3... https://arxiv.org/pdf/1512.03385 https://arxiv.org/pdf/1511.07122…

gooobJul 7, 2026View on HNParent context

No need stupid moving texts. CS231n: Convolutional Neural Networks for Visual Recognition - https://cs231n.github.io/ The Unreasonable Effectiveness of Recurrent Neural Networks - https://karpathy.github.io/2015/05/21/rnn-effectiveness/ Understanding LSTM Networks…

cute_boiJul 7, 2026View on HNParent context

This is quite a good overview, and parts reflect well how things played out in language model research. It's certainly true that language models and deep learning were not considered particularly promising in NLP, which frustrated me greatly at the time since I knew otherwise!…

jph00Oct 9, 2025View on HNParent context

a backup just in case https://nlp.seas.harvard.edu/annotated-transformer/ https://scottaaronson.blog/?p=762 https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://colah.github.io/posts/2015-08-Understanding-LSTMs/ https://arxiv.org/pdf/1409.2329.pdf…

is_trueMay 18, 2024View on HNParent context

For those unaware of Alex Graves' work, check out the NTM paper: https://arxiv.org/abs/1410.5401 This actually does what this blog post pretends to

typonApr 10, 2024View on HNParent context

You seem to be talking about neural Turing machines: https://arxiv.org/abs/1410.5401 Combining these with LLM sounds indeed quite interesting, I don't know why they haven't been used much.

blackbear_Jan 21, 2023View on HNParent context

A bit disappointed they do not compare with, or at least mention, the memory mechanism of neural turing machines [1]. [1] https://arxiv.org/abs/1410.5401

blackbear_Feb 16, 2022View on HNParent context

There's no neural network you could train using supervised learning to solve sorting in the general case. To train a neural network, the training set and test set need to come from a common probability distribution but the inputs of sorting algorithms can be random. There's no…

somethingAlexJul 25, 2021View on HNParent context

Are you aware of the Neural Turing Machine (https://arxiv.org/abs/1410.5401) or Differentiable Neural Computer (https://deepmind.com/blog/differentiable-neural-computers/)? They're not exactly what you describe, which I suppose is a truly online model that knows how/when to…

nmfisherJun 4, 2020View on HNParent context

"We extend the capabilities of neural networks by coupling them to external memory resources, which they can interact with by attentional processes. The combined system is analogous to a Turing Machine or Von Neumann architecture but is differentiable end-to-end, allowing it to…

stev0lutionFeb 23, 2020View on HNParent context

- Neural turing machines: https://arxiv.org/abs/1410.5401 - Universal transformers: https://arxiv.org/abs/1807.03819 These are just 2 of most well known.

solomatovNov 10, 2018View on HNParent context

While I haven't read those particular papers yet, one common pattern in the ML literature seems to be: 1) Identify a commonly seen problem with deep learning architectures, whether that's large data volumes, lack of transfer learning, etc. 2) Invent a solution to the problem. 3)…

apsec112Jun 2, 2018View on HNParent context

Input and output sizes don't have to be fixed. E.g. speech recognition doesn't work with fixed sized inputs. Natural language processing deals with many different length sequences. seq2seq networks are explicitly designed to deal with problems that have variable length inputs…

unlikelymordantJul 17, 2017View on HNParent context

Neural Turing Machines by Alex Graves, Greg Wayne, Ivo Danihelka https://arxiv.org/abs/1410.5401 A Neural Turing machine is a type of neural network with addressable memory - it learns rather than being explicity programmed. The net also learns to boot, i.e. how to read/ write…

deepnetApr 19, 2017View on HNParent context

yequalsx: yes, almost by definition, memory made up of bits should be modeled with a field of characteristic 2. That seems the most "natural" way to do it. However, the authors of this paper are thinking about "computers" in a broader sense. Take a look at these two papers: *…

cs702Dec 30, 2016View on HNParent context

The author proposes a lot of vague ideas in this article (for example "I believe one of the biggest problems is the use of Error Propagation and Gradient Descent") without references or any solid proofs why they are necessary to solve the proposed program (Automate programming…

sapphireblueJul 31, 2016View on HNParent context

There are signs, if you know were to look for them: http://arxiv.org/abs/1510.07211 https://arxiv.org/abs/1410.4615 http://arxiv.org/abs/1605.06640 https://arxiv.org/abs/1410.5401

sapphireblueJul 22, 2016View on HNParent context

You might enjoy this paper if you haven't already read it. http://arxiv.org/abs/1410.5401 Jump to the experiments section to get an overview of what it can do.

current_callMay 25, 2016View on HNParent context

Here's the original paper on Neural Turing machines: https://arxiv.org/abs/1410.5401 IIRC, that paper's results focus on the model's ability to extrapolate prediction of sequential data to arbitrary lengths having only been trained on sequences of a fixed length.

alexbeloiMay 20, 2016View on HNParent context

http://arxiv.org/abs/1410.5401

carapaceMay 18, 2016View on HNParent context

FYI, RNNs are a form of Deep Learning these days (in particular LSTM and GRU RNNs, very popular for language translation and much more for a few years now). There is also research lately into weirder types of neural nets such as 'Neural Turing Machines'…

andreykJan 20, 2016View on HNParent context

One related work that came to mind was Neural Turing Machines, which augments the LSTM network with a (fixed size) memory [1]. I did see a brief reference to the Graves et al. paper in the paper linked to in the original post, but I'm also wondering if someone more knowledgeable…

sxyuanSep 5, 2015View on HNParent context

The OP paper proposes a different architectural approach to the translation task of the 2014 NIPS paper by Ilya Sutskever, Oriol Vinyals & Quoc Le, Sequence to Sequence Learning with Neural Networks http://papers.nips.cc/paper/5346-sequence-to-sequence-learni... which "uses a…

deepnetAug 24, 2015View on HNParent context

The literature is mostly image based, but as other users have said voice, text, and video have also seen substantial gains. It has also been used to great effect for taxi destination prediction (heterogenous information) [1], drug discovery [2], and general operations on graphs…

kastnerkyleAug 6, 2015View on HNParent context

> Our intelligence grows from childhood over many years of training, through our physical and mental interactions with the world. We learn methods along side experiences. It is not about the speed of linear computation, or the amount of memory. There's nothing in principle that…

RoboTeddyJul 17, 2015View on HNParent context

Currently it doesn't work, but saying it'll never work is pretty strong. This kind of demo shows that deep neural networks can capture the structure of language, if not the semantics, in a very general way. And we have separate evidence that they can (in principle) capture…

teraflopMay 21, 2015View on HNParent context

In "Learning to Execute" by Zaremba & Sutskever http://arxiv.org/abs/1410.4615 An RNN learns snippets of python Their next paper is "Reinforcement Learning Neural Turing Machines" http://arxiv.org/abs/1505.00521 based on Graves "Neural Turing Machines"…

deepnetMay 24, 2015View on HNParent context

Deep nets are only loosely inspired by neurobiology. That's why LeCun calls them "convolutional nets" and not "convolutional neural nets" and prefers "nodes" over "neurons". It is, however, possible to have a deep net produce 3D models/images…

compbioMar 23, 2015View on HNParent context

Yes, but computation can be performed with a function approximator where each iteration is a function `f :: (state, input) -> (state', output)`. This is the basis of an architecture called a 'neural turing machine' (http://arxiv.org/abs/1410.5401) and it is, indeed, Turing…

a-prioriFeb 13, 2015View on HNParent context

For arXiv submissions, please submit the abstract link; in this case: http://arxiv.org/abs/1410.5401 The advantage is immediate: One gets a plain website with the abstract, which allows the visitor to decide whether the PDF download is worth it. The PDF itself is only one more…

nhaehnleOct 21, 2014View on HNParent context