Universal Model Routing for Efficient LLM Inference — mentioned by botirk

Sun, 28 Sep 2025 10:33:20 GMT

We have been experimenting with routing inference across LLMs, and the path has been full of wrong turns. Our first attempt was to just use a large LLM itself to decide routing. It was too costly and the decisions were unreliable. Next we tried training a small fine-tuned LLM as a router. It was cheaper, but the outputs were poor and not trustworthy. Then we wrote heuristics to map prompt types to model IDs. That worked for a while, but it was brittle. Every API change or workload shift broke it. Eventually we shifted to thinking in terms of model criteria instead of hardcoded model IDs. We…

Paper and discussion history. A mention is not an endorsement.

Universal Model Routing for Efficient LLM Inference · HackerPapers discussions

Universal Model Routing for Efficient LLM Inference — mentioned by botirk