Distributed Representations of Words and Phrases and their Compositionality · HackerPapers discussions

Distributed Representations of Words and Phrases and their Compositionality — mentioned by RyEgswuCsn

Mon, 03 Jun 2019 23:30:20 GMT

I find the title of the article rather exaggerating... As of the first difference pointed out in the article, one of the CS224D lectures on word2vec did addressed it: https://youtu.be/aRqn8t1hLxs?t=2650 It was also mentioned later in the lecture that having two vectors representing each word is meant to make the optimisation easier (so it's kind of a trick); at the end, the two vectors learnt will have to be averaged over in order to reach a single vector for each word. To be fair, the fact that each word is represented by two vectors was also mentioned in the original paper describing…

Paper and discussion history. A mention is not an endorsement.

Distributed Representations of Words and Phrases and their Compositionality — mentioned by kough

Thu, 04 Aug 2016 20:29:27 GMT

Links to the relevant papers: Bag of Tricks for Efficient Text Classification: https://arxiv.org/abs/1607.01759v2 Enriching Word Vectors with Subword Information: https://arxiv.org/abs/1607.04606 Both fantastic papers. For those who aren't aware, Mikolov also helped create word2vec. One curious thing: this seems to use heirarchal softmax instead of the "negative sampling" described in their earlier paper http://arxiv.org/abs/1310.4546, despite that paper reporting that "negative sampling" is more computationally efficient and of similar quality. Anyone know why that might be?

Paper and discussion history. A mention is not an endorsement.

Distributed Representations of Words and Phrases and their Compositionality — mentioned by microtonal

Sat, 22 Aug 2015 05:50:39 GMT

In my experience it also works fairly well when training e.g. on newspaper text. It's not really that surprising when you think about how it works. Similar words cluster together in vector space in some dimensions. E.g. 'Paris' and 'Berlin' will both have capital-ish contexts. However, they also are different in some ways, e.g. Paris will have France-ish contexts and Berlin German-ish contexts. The 2-dimensional PCA projection in figure 2 of one of Mikolov's papers [1] gives an intuition why substraction/addition generally works. [1] http://arxiv.org/pdf/1310.4546.pdf

Paper and discussion history. A mention is not an endorsement.

Distributed Representations of Words and Phrases and their Compositionality — mentioned by Tobu

Sat, 22 Aug 2015 01:03:58 GMT

Such a fluffy article. Here is what he's publishing: http://arxiv.org/find/cs/1/au:+Hinton_G/0/1/0/all/0/1 This seems to be a good introduction to the topic: http://arxiv.org/pdf/1310.4546v1.pdf This is about paragraph vectors: http://arxiv.org/abs/1507.07998 http://arxiv.org/abs/1405.4053

Paper and discussion history. A mention is not an endorsement.

Distributed Representations of Words and Phrases and their Compositionality — mentioned by abeppu

Sat, 22 Aug 2015 01:01:31 GMT

Of course there's not really enough info in this article to really tell what he's talking about. A Guardian article from May [1] has a bit more about what he is meant by "meaning space": > Hinton said that the idea that language can be deconstructed with almost mathematical precision is surprising, but true. “If you take the vector for Paris and subtract the vector for France and add Italy, you get Rome,” he said. “It’s quite remarkable.” ... which makes it sounds like this builds off of the "word2vec" [2,3] work that came out in 2013. But there must be something else new (maybe at the…

Paper and discussion history. A mention is not an endorsement.

Distributed Representations of Words and Phrases and their Compositionality — mentioned by viksit

Fri, 14 Aug 2015 21:55:31 GMT

Of course. I thought about using wit.ai a few months ago, but I found a lot of issues in how they approach parsing and classification. Specifically, training it is a very manual process, and customization and tweaking is very hard. I've built my own API which can do entity and language parsing. It relies on utilizing word embeddings (such as those described here: http://arxiv.org/pdf/1310.4546.pdf).

Paper and discussion history. A mention is not an endorsement.