HackerPapers

arXiv:1402.3722cs.CL

word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method

Yoav Goldberg, Omer Levy

Abstract

The word2vec software of Tomas Mikolov and colleagues (https://code.google.com/p/word2vec/ ) has gained a lot of traction lately, and provides state-of-the-art word embeddings. The learning models behind the software are described in two research papers. We found the description of the models in these papers to be somewhat cryptic and hard to follow. While the motivations and presentation may be obvious to the neural-networks language-modeling crowd, we had to struggle quite a bit to figure out the rationale behind the equations. This note is an attempt to explain equation (4) (negative sampling) in "Distributed Representations of Words and Phrases and their Compositionality" by Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado and Jeffrey Dean.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.LGMachine Learning stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 5 identifiable commenters in 6 comments across 5 threads, from Jun 2015 to Nov 2017. Found in 1 HN submission from Feb 2015 to Feb 2015. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. Ask HN: What are some interesting papers in CS for a beginner? 1 comment

    • I like Pugh's paper on skip lists [1], Shannon's "Mathematical Theory of Communication" [2], and these two might be a stretch but I also like Rong's "Word2vec Parameter Learning Explained" [3] and Levy & Goldberg's "Word2vec Explained" [4]. In any case use the recommended papers…

      faizshahView on HNParent context

  2. What every software engineer should know about search 1 comment

    • Neural networks are all the rage these days. As my knowledge on NNs is quite limited, what I could grasp from the wikipedia article [0] and the original paper [1] is that word2vec uses a vector to represent words in a n-dimensional space. This seems quite similar to LSA (Latent…

      jventuraView on HNParent context

  3. Facebook releases 300-dimensional pretrained Fasttext vectors for 90 languages 2 comments

    • Can someone explain WHY do word vectors with similar contexts club together and are good? One of the papers suggest that "we don't really know" (Section 4) [0] https://arxiv.org/abs/1402.3722

      denzil_correaView on HNParent context

    • > Words with similar vectors have similar meanings. Can you explain WHY this is the case with word2vec? I have come across a paper which says "we don't know really". Is this true? https://arxiv.org/abs/1402.3722

      denzil_correaView on HNParent context

  4. Primer on Neural Network Models for Natural Language Processing[pdf] 1 comment

    • Yoav is one of the smartest guys in this space. He and Omer Levy also wrote a great explanation of word2vec: http://arxiv.org/pdf/1402.3722v1.pdf https://levyomer.wordpress.com/2014/04/25/word2vec-explained...

      vonnikView on HNParent context

  5. Hound voice search NLP demo [video] 1 comment

    • You might be interested to read this: http://arxiv.org/pdf/1402.3722v1.pdf word2vec Explained: Deriving Mikolov et al.’s Negative-Sampling Word-Embedding Method. Yoav Goldberg and Omer Levy. arXiv 2014. [pdf] The word2vec software of Tomas Mikolov and colleagues has gained a lot…

      syllogismView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.