GLaM: Efficient Scaling of Language Models with Mixture-of-Experts · HackerPapers discussions

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts — mentioned by RC_ITR

Thu, 27 Jul 2023 18:59:35 GMT

A huge misconception is that MoE is an ensemble of discrete models, when it is in fact multiple FFNN modules that share an attention and embedding module. Basically the idea is that there's some pars of the model (attention/embedding) that should be trained on everything and used in every inference and other parts (the FFNN) that are fine to specialize on certain types of data (via a routing module that is also trained). [0] https://arxiv.org/pdf/1701.06538.pdf [1] https://arxiv.org/pdf/2112.06905.pdf EDIT: Specifically GLaM model architecture. Each MoE layer (the bottom block) is interleaved…

Paper and discussion history. A mention is not an endorsement.

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts — mentioned by kordlessagain

Wed, 05 Jul 2023 15:57:17 GMT

This could be of interest: "GLaM: Efficient Scaling of Language Models with Mixture-of-Experts"[1] Looking at OpenAI's recent introduction of ChatGPT's chat history and sharing features, it's plausible to suggest that the creation of a set of related documents (including chat interactions and documents sourced from plugins) could pave the way for "domain expert" models to participate in a group alongside "frozen" or "foundation" models. These domain expert models could be models that receive document content for inference. I've conducted experiments where I sent document fragments (sourced…

Paper and discussion history. A mention is not an endorsement.

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts — mentioned by cypress66

Wed, 21 Jun 2023 06:16:57 GMT

https://arxiv.org/abs/2112.06905

Paper and discussion history. A mention is not an endorsement.