HackerPapers

arXiv:1310.4546cs.CL

Distributed Representations of Words and Phrases and their Compositionality

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean

Abstract

The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.LGMachine Learning stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 6 identifiable commenters in 6 comments across 4 threads, from Aug 2015 to Jun 2019.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. Differences between the word2vec paper and its implementation 1 comment

    • I find the title of the article rather exaggerating... As of the first difference pointed out in the article, one of the CS224D lectures on word2vec did addressed it: https://youtu.be/aRqn8t1hLxs?t=2650 It was also mentioned later in the lecture that having two vectors…

      RyEgswuCsnView on HNParent context

  2. FastText – Library for fast text representation and classification 1 comment

    • Links to the relevant papers: Bag of Tricks for Efficient Text Classification: https://arxiv.org/abs/1607.01759v2 Enriching Word Vectors with Subword Information: https://arxiv.org/abs/1607.04606 Both fantastic papers. For those who aren't aware, Mikolov also helped create…

      koughView on HNParent context

  3. Google Is Working on a New Type of Algorithm Called “Thought Vectors” 3 comments

    • Of course there's not really enough info in this article to really tell what he's talking about. A Guardian article from May [1] has a bit more about what he is meant by "meaning space": > Hinton said that the idea that language can be deconstructed with almost mathematical…

      abeppuView on HNParent context

    • Such a fluffy article. Here is what he's publishing: http://arxiv.org/find/cs/1/au:+Hinton_G/0/1/0/all/0/1 This seems to be a good introduction to the topic: http://arxiv.org/pdf/1310.4546v1.pdf This is about paragraph vectors: http://arxiv.org/abs/1507.07998…

      TobuView on HNParent context

    • In my experience it also works fairly well when training e.g. on newspaper text. It's not really that surprising when you think about how it works. Similar words cluster together in vector space in some dimensions. E.g. 'Paris' and 'Berlin' will both have capital-ish contexts.…

      microtonalView on HNParent context

  4. Show HN: Myra – AI based personal assistant you can text 1 comment

    • Of course. I thought about using wit.ai a few months ago, but I found a lot of issues in how they approach parsing and classification. Specifically, training it is a very manual process, and customization and tweaking is very hard. I've built my own API which can do entity and…

      viksitView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.