StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding — mentioned by KhoomeiK

Tue, 08 Dec 2020 21:24:03 GMT

Discovering Symbolic Models from Deep Learning with Inductive Biases [1] trains graph neural nets on astrophysical phenomena and then performs symbolic regression to generate algebraic formulae to elegantly model the phenomena in a classical physics framework. It's largely gone under the radar but has pretty interesting implications for NLP and language theory in my opinion. Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures [2] applies DFA, an approach to training neural nets without backprop, to modern architectures like the Transformer. It does surprisingly…

Paper and discussion history. A mention is not an endorsement.

StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding · HackerPapers discussions

StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding — mentioned by KhoomeiK