Confident Adaptive Language Modeling · HackerPapers discussions

Confident Adaptive Language Modeling — mentioned by airgapstopgap

Wed, 19 Jul 2023 03:40:40 GMT

Not really. They have a way of squaring this circle, by changing their inference code. Speculative sampling [1] would still make their first claim a lie – sure, there'd still be the original GPT-4 model, plus a smaller draft worker. But early exit decoding [2] allows you to get almost as good results for much cheaper from exactly the same checkpoint. We know that this line of research for large-scale inference is going strong [3] so it stands to reason that OpenAI, with their wealth of talent focused on GPT-4 throughput&inference [4], large contexts and aggressive pricing policy, would also…

Paper and discussion history. A mention is not an endorsement.

Confident Adaptive Language Modeling — mentioned by jmoss20

Mon, 13 Mar 2023 16:46:16 GMT

This does indeed improve efficiency significantly: https://arxiv.org/abs/2207.07061

Paper and discussion history. A mention is not an endorsement.

Confident Adaptive Language Modeling — HN submission

Tue, 27 Dec 2022 13:15:48 GMT

New framework increases speed of large language models by up to 3x

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Confident Adaptive Language Modeling — HN submission

Thu, 22 Dec 2022 09:51:32 GMT

Confident Adaptive Language Modeling

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.