HackerPapers

arXiv:1411.4555cs.CV

Show and Tell: A Neural Image Caption Generator

Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan

Abstract

Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. For instance, while the current state-of-the-art BLEU-1 score (the higher the better) on the Pascal dataset is 25, our approach yields 59, to be compared to human performance around 69. We also show BLEU-1 score improvements on Flickr30k, from 56 to 66, and on SBU, from 19 to 28. Lastly, on the newly released COCO dataset, we achieve a BLEU-4 of 27.7, which is the current state-of-the-art.

arXiv comments
Journal reference
DOI
Subjects
cs.CVComputer Vision and Pattern Recognition

On Hacker News

Linked by 5 identifiable commenters in 5 comments across 5 threads, from Nov 2014 to Jun 2016.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. Elasticsearch Based Image Search Using RGB Signatures 1 comment

    • I think in addition to histograms and features like SIFT, SURF, and others like DAISY[0] that would permit searching with images as a query it would be beneficial just to use a neural network to text index using the classes at very top of a neural network, although you are right…

      jkldotioView on HNParent context

  2. Google Photos 1 comment

    • Nice rundown from Mat Honan at Buzzfeed: http://www.buzzfeed.com/mathonan/googles-new-photos-app-is-l... > I have a cousin who I only have a few photos of. In almost all of these, mostly taken last year, she is 30. In another, from 2000, she is 16. But it’s the earliest one in…

      dansoView on HNParent context

  3. The state of Computer Vision and AI: we are really, really far (2012) 1 comment

    • Presumably the author is familiar with the Google paper - http://arxiv.org/pdf/1411.4555v2.pdf which should get the people, scale, and laughing bits out of the image. There are a remarkable number of technologies converging here.

      ChuckMcMView on HNParent context

  4. Google and Stanford build hybrid neural networks that can explain photos 1 comment

    • The approach of both papers are somewhat similar, but not exactly the same. Preprints download: - Show and Tell: A Neural Image Caption Generator: http://arxiv.org/pdf/1411.4555.pdf (Google) - Deep Visual-Semantic Alignments for Generating Image Descriptions…

      redknight666View on HNParent context

  5. Researchers Announce Advance in Image-Recognition Software 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.