HackerPapers

arXiv:2012.15332cs.CL

Corrected CBOW Performs as well as Skip-gram

Ozan İrsoy, Adrian Benton, Karl Stratos

Early discovery. Linked by one identifiable commenter. This paper is outside the main ranking; a mention is not an endorsement.

Abstract

Mikolov et al. (2013a) observed that continuous bag-of-words (CBOW) word embeddings tend to underperform Skip-gram (SG) embeddings, and this finding has been reported in subsequent works. We find that these observations are driven not by fundamental differences in their training objectives, but more likely on faulty negative sampling CBOW implementations in popular libraries such as the official implementation, word2vec.c, and Gensim. We show that after correcting a bug in the CBOW gradient update, one can learn CBOW word embeddings that are fully competitive with SG on various intrinsic and extrinsic tasks, while being many times faster to train.

arXiv comments
Presented at WINR at EMNLP 2021, added discussion about FastText, more discussion about findings, additional results on C4 data, wording changes
Journal reference
DOI
Subjects
cs.CLComputation and Language stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 1 identifiable commenter in 1 comment across 1 thread, from Jan 2021 to Jan 2021.

Comments linking this paper, per year.

Discussed in comments1 thread

  1. Koan: A word2vec negative sampling implementation with correct CBOW update 1 comment

    • To save a few clicks, here's the paper that describes the fix and gives some comparisons with the supposedly broken implementation. https://arxiv.org/pdf/2012.15332.pdf

      steve_gView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.