SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment · HackerPapers discussions

SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment — mentioned by yorwba

Mon, 23 Mar 2026 17:26:36 GMT

It's feasible to put the expert routing logic in a previous layer. People have done it: https://arxiv.org/abs/2507.20984

Paper and discussion history. A mention is not an endorsement.

SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment — mentioned by yorwba

Thu, 29 Jan 2026 12:53:34 GMT

The model is explicitly trained to produce as uniform a distribution as possible, because it's designed for batched inference with a batch size much larger than the expert count, so that all experts are constantly activated and latency is determined by the highest-loaded expert, so you want to distribute the load evenly to maximize utilization. Prompt ingestion is still fairly similar to that setting, so you can first compute the expert routing for all tokens, load the first set of expert weights and process only those tokens that selected the first expert, then load the second expert and so…

Paper and discussion history. A mention is not an endorsement.

SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment — HN submission

Tue, 29 Jul 2025 02:40:52 GMT

SmallThinker: A Family of Efficient LLMs Natively Trained for Local Deployment

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.