Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models — mentioned by vark90

Fri, 25 Oct 2024 21:46:49 GMT

The idea behind semantic entropy (estimating entropy of distribution over semantic units, instead of individual sequences in the output space) is great, but it's somewhat naive in the sense that it considers these semantic units to be well-defined partitions of output space. There is further generalization of this approach [1] which performs soft clustering of sampled outputs based on a similar notion of semantic equivalence between them. But even with this in mind, there are caveats. We have recently published [2] a comprehensive benchmark of SOTA approaches to estimating uncertainty of…

Paper and discussion history. A mention is not an endorsement.

Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models · HackerPapers discussions

Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models — mentioned by vark90