Efficient Estimation of Word Representations in Vector Space · HackerPapers discussions

Efficient Estimation of Word Representations in Vector Space — mentioned by godelski

Thu, 29 May 2025 21:07:11 GMT

I'm not talking about Tokenization. It can get confusing because we usually role tokenization and embedding up as a singular process but the tokenization is the translation of our characters into numeric representations. There's self discovery of what the atomic units should be (bounded by our vocabulary size). The process is, at a high level: string -> integer -> vec<float>. You are learning the string splits, integer IDs, and vector embeddings. You are literally building a dictionary. The BPE paper is a good place to start[0], but it is far from the place we are now. The embeddings is this…

Paper and discussion history. A mention is not an endorsement.

Efficient Estimation of Word Representations in Vector Space — mentioned by HPsquared

Fri, 14 Feb 2025 18:11:02 GMT

This blog post [0] thinks that it is indeed possible to do arithmetic operations on them. My intuition is that they're vectors after all, and can be added and subtracted like any other vector. A word is just a location in that high-dimensional vector space. [0] https://blog.dataiku.com/arithmetic-properties-of-word-embed... EDIT: I guess there are different forms of word embeddings and apparently modern LLMs don't use static word embeddings like word2vec and it's more contextual. Tokens aren't 1:1 with words either of course. I guess it's more complex than "LLMs represent words as vectors".…

Paper and discussion history. A mention is not an endorsement.

Efficient Estimation of Word Representations in Vector Space — mentioned by Olreich

Wed, 25 Oct 2023 10:55:25 GMT

The “secret sauce” is Word2Vec. Take a snapshot of all the text you can find on the internet, then with a sliding context window, vectorize each word based on what words are around it. The core assumption is that words with similar context have similar meaning. How you decide which components represent which words in the context is unclear, but it looks like we’re doing some kind of ML training to convince a computer to decide for us. Here’s a paper about the technique which might help: https://arxiv.org/pdf/1301.3781.pdf Once all the words have vectors you can assume that there’s meaning in…

Paper and discussion history. A mention is not an endorsement.

Efficient Estimation of Word Representations in Vector Space — mentioned by PeterisP

Wed, 25 Oct 2023 10:47:17 GMT

The general idea is that you have a particular task & dataset, and you optimize these vectors to maximize that task. So the properties of these vectors - what information is retained and what is left out during the 'compression' - are effectively determined by that task. In general, the core task for the various "LLM tools" involves prediction of a hidden word, trained on very large quantities of real text - thus also mirroring whatever structure (linguistic, syntactic, semantic, factual, social bias, etc) exists there. If you want to see how the sausage is made and look at the actual…

Paper and discussion history. A mention is not an endorsement.

Efficient Estimation of Word Representations in Vector Space — mentioned by ninjin

Thu, 14 Feb 2019 23:37:35 GMT

> …we do NOT need new architectures… You do realise the irony in stating this regarding transformers that arguably made their first appearance as decomposable attention in 2016 [1] and then as transformers in 2017 [2]? It is not as if this is a vanilla RNN straight out of the 90s sweeping the floor with decades of model innovations, rather it looks like we are seeing the rise of a new, simple model category that works remarkably well – akin to how Mikolov et al. reconsidered how to learn vector representations back in 2013 [3] to ingest magnitudes more data than previously possible. [1]…

Paper and discussion history. A mention is not an endorsement.

Efficient Estimation of Word Representations in Vector Space — mentioned by jventura

Wed, 13 Sep 2017 13:24:54 GMT

Neural networks are all the rage these days. As my knowledge on NNs is quite limited, what I could grasp from the wikipedia article [0] and the original paper [1] is that word2vec uses a vector to represent words in a n-dimensional space. This seems quite similar to LSA (Latent Semantic Analysis). In LSA, a word is represented by a vector (list / array) of values which represent the number of occurrences of the word in a document. For instance, the list [0, 5, 9, 10] for the word "aeroplane" means than "aeroplane" occurs zero times in D1 (document #1), 5 times in D2, etc. Unlike LSA, word2vec…

Paper and discussion history. A mention is not an endorsement.

Efficient Estimation of Word Representations in Vector Space — mentioned by nl

Wed, 02 Aug 2017 08:33:57 GMT

Unpublihed arXiv papers fall in the latter category. In CS that almost certainly isn't true. I'm most familiar with the NLP field, but there, if you have some kind of embedding of your words/tokens/sentences/something you cite https://arxiv.org/pdf/1301.3781.pdf (Word2Vec, Mikolov). That paper says there is a follow up paper published at NIPS2013, but I don't think I've ever seen that published. The field just moves too fast to wait for conferences anymore.

Paper and discussion history. A mention is not an endorsement.

Efficient Estimation of Word Representations in Vector Space — mentioned by deepnet

Mon, 10 Jul 2017 17:58:43 GMT

Geoff Hinton refers to thought vectors performing reasoning by analogy using algebra [1] in his Royal Society Lecture. The other widely reported vector algebras in a semantic space were discovered by Mikolov et al when producing ~300 dimensional vectors for a billion word Wikipedia corpus. If one performs vector algebra and ~= is near by cosine distance then using Mikolov's Vectors[3]. King - Man + Woman ~= Queen France - Paris + Gernmany ~= Berlin Surprisingly this works for other modalities, Chintala, Radford & Metz found a latent semantic space in images, that adds vectors for glasses or…

Paper and discussion history. A mention is not an endorsement.

Efficient Estimation of Word Representations in Vector Space — HN submission

Tue, 10 Jan 2017 12:06:25 GMT

Efficient Estimation of Word Representations in Vector Space (2013)

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Efficient Estimation of Word Representations in Vector Space — mentioned by deepnet

Tue, 03 May 2016 23:51:18 GMT

A Hinton inspired speculation about a Robot that plans & communicates. I posit that general A.I. must be embodied to develop, it must learn from the real world by interacting with it and learn by trial and error. Deep learning is a huge leap for AI. It learns best from raw data. Babies learn motor control & input interpretation first[1]. Moravec's paradox proposes learning these basics are the hard foundation that higher levels of reason build on. Abeel and Levine's work[2] where robots learn robust functions to map from pixels to motor torques for domestic tasks show solutions built on…

Paper and discussion history. A mention is not an endorsement.

Efficient Estimation of Word Representations in Vector Space — mentioned by deepnet

Wed, 16 Dec 2015 09:18:57 GMT

The person in the room performing the lookup is a red-herring and can be replaced by suitable algorithm, e.g. a convnet - which could learn the lookup task. Consciousness resides in the minds that created the lookup tables: they were constructed by conscious beings to map queries to meaningful responses. The lookup tables are the very sophisticated part of Searle's Chinese Room. The recent emergent semantic vector algebra discovered in human languages by Mikolov's word2vec [Mikolov et al] demonstrate that some of the computational power of language is inherent in the language ( but only…

Paper and discussion history. A mention is not an endorsement.

Efficient Estimation of Word Representations in Vector Space — mentioned by sp332

Sun, 14 Jun 2015 18:54:55 GMT

As pointed out elsewhere, the results are pretty accurate, not nearly perfect but enough to open up new applications. http://arxiv.org/pdf/1301.3781.pdf Word2vec is exciting because it has a useable amount of accuracy for a relatively small amount of cleverness on the part of the human. You can just throw a huge corpus at it and it performs reasonably well.

Paper and discussion history. A mention is not an endorsement.

Efficient Estimation of Word Representations in Vector Space — mentioned by microtonal

Sun, 14 Jun 2015 18:44:30 GMT

Mikolov, et al. 2013 [1] do a proper evaluation of this. E.g. they found that the skip-ngram model has a 50.0% accuracy for semantic analogy queries and 55.9% accuracy for syntactic queries. word2vec comes with a data set that you can use to evaluate language models. [1] http://arxiv.org/pdf/1301.3781.pdf

Paper and discussion history. A mention is not an endorsement.

Efficient Estimation of Word Representations in Vector Space — mentioned by nl

Wed, 03 Jun 2015 12:11:31 GMT

Here's a chronological reading list of techniques Traditional NLP -> 2012, IBM Watson papers[1] Watson is probably the pinnacle of "traditional" style NLP (ie, Tokenization/Lemmatization/Part of Speech Tagging/Framing/Knowledge Engineering/etc) Word2Vec -> 2013, Google paper[2] Word2Vec kind of exploded everyone's mind for a while. Subgraph Embedding -> 2015, Facebook AI Group [3][4] [1] http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=617771... [2] http://arxiv.org/pdf/1301.3781.pdf [3] https://research.facebook.com/publications/1473550739586509/... [4] http://arxiv.org/abs/1502.05698

Paper and discussion history. A mention is not an endorsement.

Efficient Estimation of Word Representations in Vector Space — mentioned by dhammack

Tue, 12 Nov 2013 20:41:05 GMT

I'm not associated with the OP, but I have played with word2vec. Word2vec can be viewed as dimensionality reduction. In some sense, the dimensionality of words is the same as the vocabulary size, if a one-hot encoding is used (which is most common). For word2vec specifically, the output vector dimension is a parameter specified during training, so you can choose it however you want. I suspect the OP chose something fairly large (>256) because of the performance boost. Nearest neighbor methods work well with word2vec words: https://code.google.com/p/word2vec/#Word_clustering The paper…

Paper and discussion history. A mention is not an endorsement.

Efficient Estimation of Word Representations in Vector Space — mentioned by 3JPLW

Mon, 30 Sep 2013 17:23:00 GMT

Very cool. I missed the original word2vec software discussion back in August: https://news.ycombinator.com/item?id=6216044 And the paper itelf is a very worthwhile read: http://arxiv.org/abs/1301.3781

Paper and discussion history. A mention is not an endorsement.

Efficient Estimation of Word Representations in Vector Space — mentioned by magicalist

Thu, 15 Aug 2013 05:53:36 GMT

Yeah, the papers linked in the references are probably a better place to start than the readme (though I'm not sure how closely aligned this implementation is with that research, but the paper is still a good read), especially [1] [1] http://arxiv.org/pdf/1301.3781.pdf

Paper and discussion history. A mention is not an endorsement.

Efficient Estimation of Word Representations in Vector Space — mentioned by MaysonL

Thu, 15 Aug 2013 02:45:48 GMT

Link to the paper, rather than another bloody g+ plug: http://arxiv.org/pdf/1301.3781.pdf

Paper and discussion history. A mention is not an endorsement.