Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models — mentioned by euclaise

Tue, 20 Jun 2023 23:18:05 GMT

The only paper that I could find using an approach with fully separated experts like this is https://arxiv.org/pdf/2208.03306.pdf

Paper and discussion history. A mention is not an endorsement.

Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models — mentioned by sanxiyn

Thu, 25 May 2023 02:06:03 GMT

You can, like, just do weighted average. It works. Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (this is Tim Dettmers's previous work!) > These ELMs can be added and removed to update data coverage, ensembled to generalize to new domains, or averaged to collapse back to a single LM for efficient inference. https://arxiv.org/abs/2208.03306

Paper and discussion history. A mention is not an endorsement.

Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models · HackerPapers discussions

Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models — mentioned by euclaise

Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models — mentioned by sanxiyn