SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models — mentioned by boomerang90

Tue, 28 Mar 2023 21:02:24 GMT

I agree fine-tuning for task will give better results. Cerebras actually showed some research recently on this front. Sparse pre-training and dense fine-tuning (https://arxiv.org/abs/2303.10464). You can recover the accuracy of sparse pre-trained models with dense fine-tuning and reduce FLOPs of the end-to-end pipeline by 2.5x compared to dense.

Paper and discussion history. A mention is not an endorsement.

SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models · HackerPapers discussions

SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models — mentioned by boomerang90