ALBERT: A Lite BERT for Self-supervised Learning of Language Representations · HackerPapers discussions

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations — mentioned by albertzeyer

Sat, 04 Jun 2022 22:45:56 GMT

Nice, thanks for naming a project after me. :P I also like the ALBERT lite-BERT model (https://arxiv.org/abs/1909.11942). Although this was just a matter of time. The other names which match to "*bert*" were already taken. Anyway, going on topic: Such a project really should have a related work section, right at the beginning, because there are many existing launchers, and just say how this is different (maybe better). E.g. see here (https://alternativeto.net/software/alfred/?platform=linux). But I also would be interested on differences e.g. to Alfred (Mac). I really wonder why such…

Paper and discussion history. A mention is not an endorsement.

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations — mentioned by microtonal

Sat, 05 Dec 2020 19:42:49 GMT

There's nothing preventing you from sharing weights across layers, and would be interesting to see some research about that. E.g. the ALBERT model does that: https://arxiv.org/abs/1909.11942 I have done model distillation of XLM-RoBERTa into ALBERT-based models with multiple layer groups and for the tasks that I was working on (syntax) it works really well. E.g. we have gone from a finetuned ~1000MiB XLM-R base model to a 74MiB ALBERT-based model with barely any loss in accuracy.

Paper and discussion history. A mention is not an endorsement.

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations — mentioned by obmelvin

Thu, 09 Jan 2020 23:20:11 GMT

As a sibling said, I think this is a focus of the article. At the very least Google has release multiple variations of one SOTA model this year: * original BERT paper:https://arxiv.org/abs/1810.04805 * ALBERT (smaller model): https://arxiv.org/abs/1909.11942

Paper and discussion history. A mention is not an endorsement.

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations — mentioned by aglionby

Wed, 08 Jan 2020 14:22:53 GMT

Paper here: https://arxiv.org/abs/1909.11942 Really impressed at the speed with which Hugging Face ported this to their transformers library -- Google released the model and source code Oct 21 [1] and it was available in the library just 8 days later [2]. [1] https://github.com/google-research/google-research/commit/b5... [2] https://github.com/huggingface/transformers/commit/c0c208833...

Paper and discussion history. A mention is not an endorsement.

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations — HN submission

Mon, 30 Sep 2019 00:41:57 GMT

Albert: A Lite Bert for Self-Supervised Learning of Language Representations

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.