Revisiting Transformer-based Models for Long Document Classification — mentioned by bpiche

Sun, 17 Apr 2022 02:18:41 GMT

Your first suggestion seems like the way to go, I read a couple of papers this morning which seemed to confirm that it is a solid strategy. The first one claims that the Longformer architecture is tenable for truly long documents with some minor parameter optimization [0]. The Amazon/MSFT paper claims that just looking at the first part of the doc is superior to most other more complex solutions, like Longformer/Big Bird or hierarchical classification [1]. But here's what's really cool, they concluded that if you still use only 512 tokens, but you don't just look at the first ones, you look…

Paper and discussion history. A mention is not an endorsement.

Revisiting Transformer-based Models for Long Document Classification · HackerPapers discussions

Revisiting Transformer-based Models for Long Document Classification — mentioned by bpiche