HackerPapers

arXiv:1301.3781cs.CL

Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

Abstract

We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language

On Hacker News

Linked by 14 identifiable commenters in 17 comments across 15 threads, from Aug 2013 to May 2025. Found in 1 HN submission from Jan 2017 to Jan 2017. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments15 threads

  1. A visual exploration of vector embeddings 1 comment

    • I'm not talking about Tokenization. It can get confusing because we usually role tokenization and embedding up as a singular process but the tokenization is the translation of our characters into numeric representations. There's self discovery of what the atomic units should be…

      godelskiView on HNParent context

  2. Why Does Integer Addition Approximate Float Multiplication? 1 comment

    • This blog post [0] thinks that it is indeed possible to do arithmetic operations on them. My intuition is that they're vectors after all, and can be added and subtracted like any other vector. A word is just a location in that high-dimensional vector space. [0]…

      HPsquaredView on HNParent context

  3. Embeddings: What they are and why they matter 2 comments

    • The general idea is that you have a particular task & dataset, and you optimize these vectors to maximize that task. So the properties of these vectors - what information is retained and what is left out during the 'compression' - are effectively determined by that task. In…

      PeterisPView on HNParent context

    • The “secret sauce” is Word2Vec. Take a snapshot of all the text you can find on the internet, then with a sliding context window, vectorize each word based on what words are around it. The core assumption is that words with similar context have similar meaning. How you decide…

      OlreichView on HNParent context

  4. Better Language Models and Their Implications 1 comment

    • > …we do NOT need new architectures… You do realise the irony in stating this regarding transformers that arguably made their first appearance as decomposable attention in 2016 [1] and then as transformers in 2017 [2]? It is not as if this is a vanilla RNN straight out of the…

      ninjinView on HNParent context

  5. What every software engineer should know about search 1 comment

    • Neural networks are all the rage these days. As my knowledge on NNs is quite limited, what I could grasp from the wikipedia article [0] and the original paper [1] is that word2vec uses a vector to represent words in a n-dimensional space. This seems quite similar to LSA (Latent…

      jventuraView on HNParent context

  6. Do I really have to cite an arXiv paper? 1 comment

    • Unpublihed arXiv papers fall in the latter category. In CS that almost certainly isn't true. I'm most familiar with the NLP field, but there, if you have some kind of embedding of your words/tokens/sentences/something you cite https://arxiv.org/pdf/1301.3781.pdf (Word2Vec…

      nlView on HNParent context

  7. When not to use deep learning 1 comment

    • Geoff Hinton refers to thought vectors performing reasoning by analogy using algebra [1] in his Royal Society Lecture. The other widely reported vector algebras in a semantic space were discovered by Mikolov et al when producing ~300 dimensional vectors for a billion word…

      deepnetView on HNParent context

  8. My path to OpenAI 1 comment

    • A Hinton inspired speculation about a Robot that plans & communicates. I posit that general A.I. must be embodied to develop, it must learn from the real world by interacting with it and learn by trial and error. Deep learning is a huge leap for AI. It learns best from raw data.…

      deepnetView on HNParent context

  9. John Searle: Consciousness in Artificial Intelligence [video] 1 comment

    • The person in the room performing the lookup is a red-herring and can be replaced by suitable algorithm, e.g. a convnet - which could learn the lookup task. Consciousness resides in the minds that created the lookup tables: they were constructed by conscious beings to map…

      deepnetView on HNParent context

  10. Abstractions my Deep Learning word2vec model made 2 comments

    • Mikolov, et al. 2013 [1] do a proper evaluation of this. E.g. they found that the skip-ngram model has a 50.0% accuracy for semantic analogy queries and 55.9% accuracy for syntactic queries. word2vec comes with a data set that you can use to evaluate language models. [1]…

      microtonalView on HNParent context

    • As pointed out elsewhere, the results are pretty accurate, not nearly perfect but enough to open up new applications. http://arxiv.org/pdf/1301.3781.pdf Word2vec is exciting because it has a useable amount of accuracy for a relatively small amount of cleverness on the part of…

      sp332View on HNParent context

  11. Hound voice search NLP demo [video] 1 comment

    • Here's a chronological reading list of techniques Traditional NLP -> 2012, IBM Watson papers[1] Watson is probably the pinnacle of "traditional" style NLP (ie, Tokenization/Lemmatization/Part of Speech Tagging/Framing/Knowledge Engineering/etc) Word2Vec -> 2013, Google paper[2]…

      nlView on HNParent context

  12. Show HN: A search engine that lets you 'add' words as vectors 1 comment

    • I'm not associated with the OP, but I have played with word2vec. Word2vec can be viewed as dimensionality reduction. In some sense, the dimensionality of words is the same as the vocabulary size, if a one-hot encoding is used (which is most common). For word2vec specifically…

      dhammackView on HNParent context

  13. word2vec in yhat: Word vector similarity 1 comment

    • Very cool. I missed the original word2vec software discussion back in August: https://news.ycombinator.com/item?id=6216044 And the paper itelf is a very worthwhile read: http://arxiv.org/abs/1301.3781

      3JPLWView on HNParent context

  14. word2vec – Tool for computing continuous distributed representations of words 1 comment

    • Yeah, the papers linked in the references are probably a better place to start than the readme (though I'm not sure how closely aligned this implementation is with that research, but the paper is still a good read), especially [1] [1] http://arxiv.org/pdf/1301.3781.pdf

      magicalistView on HNParent context

  15. Learning the Meaning of Words 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.