HackerPapers

arXiv:1909.11942cs.CL

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut

Abstract

Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations and longer training times. To address these problems, we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and \squad benchmarks while having fewer parameters compared to BERT-large. The code and the pretrained models are available at https://github.com/google-research/ALBERT.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence

On Hacker News

Linked by 4 identifiable commenters in 4 comments across 4 threads, from Jan 2020 to Jun 2022. Found in 1 HN submission from Sep 2019 to Sep 2019. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. Albert – open-source keyboard launcher for Linux 1 comment

    • Nice, thanks for naming a project after me. :P I also like the ALBERT lite-BERT model (https://arxiv.org/abs/1909.11942). Although this was just a matter of time. The other names which match to "*bert*" were already taken. Anyway, going on topic: Such a project really should…

      albertzeyerView on HNParent context

  2. Scaling Down Deep Learning 1 comment

    • There's nothing preventing you from sharing weights across layers, and would be interesting to see some research about that. E.g. the ALBERT model does that: https://arxiv.org/abs/1909.11942 I have done model distillation of XLM-RoBERTa into ALBERT-based models with multiple…

      microtonalView on HNParent context

  3. Google Research: Looking Back at 2019, and Forward to 2020 and Beyond 1 comment

    • As a sibling said, I think this is a focus of the article. At the very least Google has release multiple variations of one SOTA model this year: * original BERT paper:https://arxiv.org/abs/1810.04805 * ALBERT (smaller model): https://arxiv.org/abs/1909.11942

      obmelvinView on HNParent context

  4. ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations 1 comment

    • Paper here: https://arxiv.org/abs/1909.11942 Really impressed at the speed with which Hugging Face ported this to their transformers library -- Google released the model and source code Oct 21 [1] and it was available in the library just 8 days later [2]. [1]…

      aglionbyView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.