DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter — mentioned by tonymet

Tue, 06 Jan 2026 20:17:12 GMT

For those curious about how the model was trained, and examples of positive/negative sentiment, here's a "nutritional label" of sorts. - SST-2 (stanford sentiment test) example dataset from IMDB reviews https://huggingface.co/datasets/stanfordnlp/sst2/viewer - BERT https://aclanthology.org/N19-1423/ - DistilBert -- the optimized model OP was using https://arxiv.org/pdf/1910.01108 SST-2 may have been used to train or qualify BERT -- read the papers to get the full story. I did a quick perusal. It would be nice to have a nutritional label shared with Abstracts, showing the training data, with…

Paper and discussion history. A mention is not an endorsement.

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter — mentioned by sanxiyn

Wed, 10 May 2023 21:57:55 GMT

Yes, DistilBERT https://arxiv.org/abs/1910.01108 is in fact the closest case I know of. But it is too small (distilling from 110M to 66M) and both BERT and DistilBERT is intended to be used (and benchmarked) with separate fine tuning for specific tasks, so they are not general.

Paper and discussion history. A mention is not an endorsement.

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter · HackerPapers discussions

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter — mentioned by tonymet

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter — mentioned by sanxiyn