HackerPapers

arXiv:1502.03044cs.LG

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio

Abstract

Inspired by recent work in machine translation and object detection, we introduce an attention based model that automatically learns to describe the content of images. We describe how we can train this model in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound. We also show through visualization how the model is able to automatically learn to fix its gaze on salient objects while generating the corresponding words in the output sequence. We validate the use of attention with state-of-the-art performance on three benchmark datasets: Flickr8k, Flickr30k and MS COCO.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.CVComputer Vision and Pattern Recognition

On Hacker News

Linked by 5 identifiable commenters in 6 comments across 6 threads, from Aug 2015 to Feb 2018. Found in 1 HN submission from Feb 2015 to Feb 2015. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. Asking the Right Questions About AI 1 comment

    • Not sure why you were downvoted, I'm assuming this was a good faith question. I'm going to try to answer your question, but I'm still learning, so take this with a grain of salt. Some models have attention, but it is not a requirement. In models that do have attention, it may or…

      ahartman00View on HNParent context

  2. Understanding deep learning requires rethinking generalization 1 comment

    • https://arxiv.org/abs/1311.2901 Check out figure 2. The network learns composition from fundamental shapes and gradients to compositional ones. It's kinda awe inspiring. Not NN specific, but some more work on explanations: https://homes.cs.washington.edu/~marcotcr/blog/lime/…

      hiddencostView on HNParent context

  3. Deepgram – Find Damning Soundbites 1 comment

    • There are a number of papers tackling a similar task [0][1][2] for anyone who is interested. There isn't enough information to tell exactly what is going on with Deepgram, but one way to approach this would be to construct a shared embedding space for words/phrases and speech.…

      kastnerkyleView on HNParent context

  4. AlphaGo beats Lee Sedol again in match 2 of 5 1 comment

    • It might look something like attention detailed in Show, Attend, Tell: http://arxiv.org/abs/1502.03044 Which attempts to visualize machine areas of attention that look like: http://www.wildml.com/wp-content/uploads/2015/12/Screen-Shot...

      larakernsView on HNParent context

  5. Neural Artistic Captions 1 comment

    • Looks like it's the home page of Ryan Kiros at the University of Toronto. Here's a paper with his name on it with some related work, though I suspect this particular output is from a later project. http://arxiv.org/abs/1502.03044

      waterlesscloudView on HNParent context

  6. A Beginner’s Guide to Restricted Boltzmann Machines 1 comment

    • The literature is mostly image based, but as other users have said voice, text, and video have also seen substantial gains. It has also been used to great effect for taxi destination prediction (heterogenous information) [1], drug discovery [2], and general operations on graphs…

      kastnerkyleView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.