Mixture-of-Depths: Dynamically allocating compute in transformer-based language models · HackerPapers discussions

Mixture-of-Depths: Dynamically allocating compute in transformer-based language models — mentioned by lwarfield

Wed, 09 Sep 2026 21:36:53 GMT

I'm kinda surprised that the mixture of depths paper didn't come up here. It approaches the other direction of sometimes dropping layers: https://arxiv.org/abs/2404.02258

Paper and discussion history. A mention is not an endorsement.

Mixture-of-Depths: Dynamically allocating compute in transformer-based language models — mentioned by SuchAnonMuchWow

Tue, 13 Aug 2024 16:01:47 GMT

There has been some work to dynamically reduce the compute required by a network. See for example: https://arxiv.org/abs/2404.02258 They have a fixed compute budget which is lower than what the LLM need, and dynamically decide to allocate this compute budget to different part of the network. So its not exactly what you propose since here the compute budget is fixed (that's the point of the paper: to make the network learn how to allocate the resources by itself) but its dynamic for each part of the network, so it shows that its possible.

Paper and discussion history. A mention is not an endorsement.

Mixture-of-Depths: Dynamically allocating compute in transformer-based language models — mentioned by huac

Wed, 17 Apr 2024 22:22:34 GMT

It also feels similar to mixture of depths (https://arxiv.org/abs/2404.02258). Being able to apply this post-training is pretty cool though, makes it easier to use across a wider range of setups.

Paper and discussion history. A mention is not an endorsement.

Mixture-of-Depths: Dynamically allocating compute in transformer-based language models — HN submission

Sun, 07 Apr 2024 13:42:05 GMT

Mixture-of-Depths: Dynamically allocating compute in transformers

281 points; 83 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Mixture-of-Depths: Dynamically allocating compute in transformer-based language models — HN submission

Sat, 06 Apr 2024 02:20:21 GMT

Dynamically allocating compute in transformer-based language models

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Mixture-of-Depths: Dynamically allocating compute in transformer-based language models — HN submission

Fri, 05 Apr 2024 09:51:16 GMT

DeepMind: Mixture-of-Depths: Dynamically allocating compute in transformers

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Mixture-of-Depths: Dynamically allocating compute in transformer-based language models — HN submission

Thu, 04 Apr 2024 16:31:27 GMT

Mixture-of-Depths: Dynamically allocating compute in transformers

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Mixture-of-Depths: Dynamically allocating compute in transformer-based language models — HN submission

Thu, 04 Apr 2024 07:11:29 GMT

Mixture-of-Depths: Dynamically allocating compute in transformer language models

5 points; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.