SpanBERT: Improving Pre-training by Representing and Predicting Spans — mentioned by woadwarrior01

Fri, 25 Oct 2019 18:43:30 GMT

That's very interesting! If you have the time for it, you should consider experimenting with swapping in SpanBERT[1] instead of BERT in your usecase. They train on full length length segments instead of masked half segments (as in BERT). I suspect that this, besides the improvements that SpanBERT brings over BERT should enable you to feed in bigger chunks (more sentences) to the model before the averaging step, leading to fewer vectors to average and as a result, perhaps better clustering. [1]: https://arxiv.org/abs/1907.10529

Paper and discussion history. A mention is not an endorsement.

SpanBERT: Improving Pre-training by Representing and Predicting Spans — HN submission

Thu, 25 Jul 2019 12:30:18 GMT

SpanBERT: Improving Pre-Training by Representing and Predicting Spans

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

SpanBERT: Improving Pre-training by Representing and Predicting Spans · HackerPapers discussions

SpanBERT: Improving Pre-training by Representing and Predicting Spans — mentioned by woadwarrior01

SpanBERT: Improving Pre-training by Representing and Predicting Spans — HN submission