Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM · HackerPapers discussions

Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM — mentioned by nabakin

Sun, 24 Mar 2024 16:10:05 GMT

Cool paper. It's more independent than dense or normal MoE but I think it's still far away from the distributed training you're looking for because you still need a seed LM which is trained normally and when fine-tuning each expert from the seed LM, you still need enough GPUs or VRAM to fine-tune that LLM so you're still limited to large GPU clusters which is the problem we're trying to avoid. In the case of the paper, they are using OPT-6.7b as the seed LM which requires 8xV100 GPUs for fine-tuning each expert. That's a combined total of 256GB of VRAM for a single expert while the 3090 only…

Paper and discussion history. A mention is not an endorsement.

Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM — mentioned by unraveller

Wed, 13 Mar 2024 19:37:34 GMT

That's Gemma-7Bx8 He goes on about the journey, the future, the collaboration that happens somewhere. Nothing about the current result, like is it any good or unexpectedly great at anything? I suppose it was more of an exercise in readying the tools to make it possible and cheap to fine tune and it is now left up to the community to tune and push the benchmarks. I also wonder if it could benefit from today's latest paper drop (from meta) on dirt cheap MoE creation. https://huggingface.co/papers/2403.07816

Paper and discussion history. A mention is not an endorsement.