Mixtral of Experts · HackerPapers discussions

Mixtral of Experts — mentioned by kristianp

Fri, 10 Jul 2026 04:37:59 GMT

I you look at https://arxiv.org/pdf/2401.04088 table 5 on page 8, you'll see that expert(s) used can change from token to token. The experts aren't divided along predictable lines.

Paper and discussion history. A mention is not an endorsement.

Mixtral of Experts — mentioned by roadside_picnic

Tue, 07 Jul 2026 16:23:15 GMT

MoEs don't route like most people imagine. They aren't learning topic based experts despite the name The original Mixtral paper [0] (in the "Routing analysis" section) found: "surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic" A quick skim of more recent analysis on MoE shows that this hasn't changed. MoE models do appear to work, but don't appear to do what the name implies, if anything they're routing based on the structure of the text and not the semantic content (and we're still not entirely sure what they're doing). 0.…

Paper and discussion history. A mention is not an endorsement.

Mixtral of Experts — mentioned by crystal_revenge

Mon, 02 Dec 2024 06:49:07 GMT

It's covered in the original Mistral "Mixtral of Experts" paper [0]. 0. https://arxiv.org/abs/2401.04088

Paper and discussion history. A mention is not an endorsement.

Mixtral of Experts — mentioned by ml_thoughts

Fri, 01 Nov 2024 20:04:45 GMT

This seems closely related to the "Mixtral" approach of a mixture-of-experts transformer [1]... I'm not claiming the approach is not original, it just helped me understand what was going on. Consider a case of two "experts" or two "value parameter tokens." The mixture of experts has a "router" network that provides a weight to each expert (through a softmax) conditional on an input. The output is a (sparse) weighted sum of the outputs of the experts. The TokenFormer has an "attention" layer combines the token and a key value to provide a weight to each "value parameter" token. A(B+C) = AB +…

Paper and discussion history. A mention is not an endorsement.

Mixtral of Experts — mentioned by lhl

Thu, 22 Feb 2024 10:38:30 GMT

Just an FYI, Mixtral is a Sparse Mixture of Experts that has 47B parameters for memory costs (but 13B active parameters per token). For those interested in reading more about how it works: https://arxiv.org/pdf/2401.04088.pdf For those interested in some of the recent MoE work going on, some groups have been doing their own MoE adaptations, like this one, Sparsetral - this is pretty exciting as it's basically an MoE LoRA implementation that runs a 16x7B at 9.4B total parameters (the original paper introduced a model, Camelidae-8x34B, that ran at 38B total parameters, 35B activated…

Paper and discussion history. A mention is not an endorsement.

Mixtral of Experts — mentioned by twobitshifter

Sat, 17 Feb 2024 19:29:52 GMT

see the long range performance piece here https://arxiv.org/pdf/2401.04088.pdf

Paper and discussion history. A mention is not an endorsement.

Mixtral of Experts — HN submission

Tue, 09 Jan 2024 02:53:06 GMT

Mixtral 8x7B: A sparse Mixture of Experts language model

359 points; 150 discussion replies on HN. Paper and source history. A submission is not an endorsement.