Contrastive Decoding Improves Reasoning in Large Language Models — mentioned by OcelotBane

Sun, 10 Dec 2023 09:57:41 GMT

Convex sums of different models are also supported, as long as the models share the same vocabulary/tokenizers. As soon as models have different tokenizers, we cannot sum the outputs of the models unless we find some way to combine the vocabularies of both tokenizers. Applications of these convex sums could be (1) model ensembling and (2) Contrastive Decoding (https://arxiv.org/abs/2309.09117) which uses a formula very similar to TopPTopK(M_large, top_p=0.9) - 0.5 * M_small.

Paper and discussion history. A mention is not an endorsement.

Contrastive Decoding Improves Reasoning in Large Language Models — HN submission

Tue, 19 Sep 2023 19:30:20 GMT

Contrastive decoding improves reasoning in large language models

21 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Contrastive Decoding Improves Reasoning in Large Language Models · HackerPapers discussions

Contrastive Decoding Improves Reasoning in Large Language Models — mentioned by OcelotBane

Contrastive Decoding Improves Reasoning in Large Language Models — HN submission