Prompting is not a substitute for probability measurements in large language models — mentioned by _delirium

Thu, 27 Feb 2025 17:45:05 GMT

Yes, there’s research showing that models’ self-assessment of probabilities (when you ask them via prompting) don’t even match the same models’ actual probabilities, in cases where you can measure the probabilities directly (e.g. by looking at the logits): https://arxiv.org/abs/2305.13264

Paper and discussion history. A mention is not an endorsement.

Prompting is not a substitute for probability measurements in large language models · HackerPapers discussions

Prompting is not a substitute for probability measurements in large language models — mentioned by _delirium