HackerPapers

arXiv:1405.4053cs.CL

Distributed Representations of Sentences and Documents

Quoc V. Le, Tomas Mikolov

Abstract

Many machine learning algorithms require the input to be represented as a fixed-length feature vector. When it comes to texts, one of the most common fixed-length features is bag-of-words. Despite their popularity, bag-of-words features have two major weaknesses: they lose the ordering of the words and they also ignore semantics of the words. For example, "powerful," "strong" and "Paris" are equally distant. In this paper, we propose Paragraph Vector, an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, and documents. Our algorithm represents each document by a dense vector which is trained to predict words in the document. Its construction gives our algorithm the potential to overcome the weaknesses of bag-of-words models. Empirical results show that Paragraph Vectors outperform bag-of-words models as well as other techniques for text representations. Finally, we achieve new state-of-the-art results on several text classification and sentiment analysis tasks.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 6 identifiable commenters in 7 comments across 7 threads, from Dec 2014 to Aug 2026. Found in 3 HN submissions from May 2014 to Jan 2019. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments7 threads

  1. Discovery Loop 1 comment
  2. Word2Vec received 'strong reject' four times at ICLR2013 1 comment

    • As another small hint of Mikolov-vs-Le divergence: they're the coauthors of the 'Paragraph Vector' paper (https://arxiv.org/abs/1405.4053) applying a slightly-modified version of word2vec to vectorize longer texts, still in a very shallow way. (This technique often goes by the…

      gojomoView on HNParent context

  3. Understanding word vectors 1 comment

    • Doc2vec is the name of the gensim implementation of this paper [0]. Briefly, it creates document embeddings (document being anything on the level of phrases, to sentences, to paragraphs and beyond) on the basis of either predicting a word given the context words (distributed…

      aglionbyView on HNParent context

  4. Why Neurons Have Thousands of Synapses: a Theory of Sequence Memory in Neocortex 1 comment

    • Sure, Numenta seemed heavily focused on temporal sequences, but in "On Intelligence" Hawkins describes a more general picture. Prediction should be done not only on the past but on the context. The context includes other variables that also have to be predicted (e.g. predict the…

      versteegenView on HNParent context

  5. Using Doc2Vec to Suggest SubReddits 1 comment

    • The gensim library now offers the 'Paragraph Vector' [1] approach to create vectors for longer ranges of text. It's available in a class named Doc2Vec – but I don't think that's what is being used here. The Paragraph Vector approach can give interesting results for…

      gojomoView on HNParent context

  6. Google Is Working on a New Type of Algorithm Called “Thought Vectors” 1 comment

    • Such a fluffy article. Here is what he's publishing: http://arxiv.org/find/cs/1/au:+Hinton_G/0/1/0/all/0/1 This seems to be a good introduction to the topic: http://arxiv.org/pdf/1310.4546v1.pdf This is about paragraph vectors: http://arxiv.org/abs/1507.07998…

      TobuView on HNParent context

  7. Large Scale Deep Learning – Jeff Dean [pdf] 1 comment

    • > Page 60-70 talks about paragraph embedding. I haven't seen this published before. Paragraph embedding was published this year as "Distributed Representations of Sentences and Documents". http://arxiv.org/abs/1405.4053

      sanxiynView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.