HackerPapers

arXiv:1605.06065cs.LG

One-shot Learning with Memory-Augmented Neural Networks

Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, Timothy Lillicrap

Abstract

Despite recent breakthroughs in the applications of deep neural networks, one setting that presents a persistent challenge is that of "one-shot learning." Traditional gradient-based networks require a lot of data to learn, often through extensive iterative training. When new data is encountered, the models must inefficiently relearn their parameters to adequately incorporate the new information without catastrophic interference. Architectures with augmented memory capacities, such as Neural Turing Machines (NTMs), offer the ability to quickly encode and retrieve new information, and hence can potentially obviate the downsides of conventional models. Here, we demonstrate the ability of a memory-augmented neural network to rapidly assimilate new data, and leverage this data to make accurate predictions after only a few samples. We also introduce a new method for accessing an external memory that focuses on memory content, unlike previous methods that additionally use memory location-based focusing mechanisms.

arXiv comments
13 pages, 8 figures
Journal reference
DOI
Subjects
cs.LGMachine Learning

On Hacker News

Linked by 8 identifiable commenters in 8 comments across 8 threads, from May 2016 to Jun 2018. Found in 1 HN submission from May 2016 to May 2016. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments8 threads

  1. Deep learning: a critical appraisal 1 comment

    • The Neural Turing Machine has been expanded upon considerably since that paper, and inspired a number of other memory augmented neural network architectures, including: https://www.gwern.net/docs/rl/2016-graves.pdf , https://arxiv.org/abs/1605.06065 and…

      goolulusaursView on HNParent context

  2. Open-sourcing DeepMind Lab 1 comment

    • One shot learning is such an active area of research there's a long Wikipedia page[1] about it. I think the SOTA is probably [2], which came out of DeepMind. There's still a way to go before it matches ResNet performance on ImageNet (or even human performance on any real task)…

      nlView on HNParent context

  3. This AI Boom Will Also Bust 1 comment

    • [Disclosure: I work for a deep-learning company.] Robin's post reveals a couple fundamental misunderstandings. While he may be correct that, for now, many small firms should apply linear regression rather than deep learning to their limited datasets, he is wrong in his…

      vonnikView on HNParent context

  4. Embed, encode, attend, predict: the new deep learning formula for NLP models 1 comment

    • That's a nice article in that it manages to get excitement across without forgetting to balance it somewhat. Kind of a rare thing these days. Btw, I'm interested to hear how well training with large one-hot encoded vectors scales. A paper someone pointed me to recently on HN…

      YeGoblynQueenneView on HNParent context

  5. Show HN: DeepRhyme (D-Prime) – Generating dope rhymes with machine learning 1 comment

    • This is just an outdated view of the state-of-the-art. It's understandable, given that it's outdated by maybe six months, iff you're willing to go with preprints. https://arxiv.org/abs/1605.06065 A hobbyist looking for something plug-and-play will still generally want lots of…

      ak4gView on HNParent context

  6. AI, Deep Learning, and Machine Learning: A Primer [video] 1 comment

    • > Only at such a high level of abstraction as to be meaningless. I'm not sure what this means or how the abstractions are meaningless? From Gabor filters to concepts like "dog", the abstractions are quite meaningful (in that they function well), even if not to us. > They are…

      jimflemingView on HNParent context

  7. The truth about deep learning 1 comment

    • > If this is the state of the art in machine translation, how on Earth can you expect a machine that can converse on human level in three years? Here are 2 good papers--one from last year, one very recent--that might change your mind. https://arxiv.org/pdf/1506.05869.pdf…

      dave_sullivanView on HNParent context

  8. One-shot Learning with Memory-Augmented Neural Networks 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.