Discovering Latent Knowledge in Language Models Without Supervision · HackerPapers discussions

Discovering Latent Knowledge in Language Models Without Supervision — mentioned by agucova

Fri, 19 Jul 2024 22:55:23 GMT

I messed up the second reference, it should be https://arxiv.org/abs/2212.03827

Paper and discussion history. A mention is not an endorsement.

Discovering Latent Knowledge in Language Models Without Supervision — mentioned by visarga

Wed, 15 Mar 2023 19:39:13 GMT

There is a paper showing you can infer when the model is telling the truth by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values. Apparently we can detect even when the model is being deceitful. https://arxiv.org/abs/2212.03827 Another approach - a model can learn the distribution - is this fact known or not in the training set, how many times does it appear, is the distribution unimodal (agreement) or multi-modal (disagreement or just high variance). Knowing this a model can adjust its…

Paper and discussion history. A mention is not an endorsement.

Discovering Latent Knowledge in Language Models Without Supervision — mentioned by visarga

Wed, 08 Mar 2023 19:30:08 GMT

Don't be quick to write off AI for that task > Discovering Latent Knowledge in Language Models Without Supervision https://arxiv.org/abs/2212.03827 They find a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values.

Paper and discussion history. A mention is not an endorsement.

Discovering Latent Knowledge in Language Models Without Supervision — mentioned by visarga

Sat, 25 Feb 2023 19:18:51 GMT

Discovering Latent Knowledge in Language Models Without Supervision > Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't detect. We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way. Our results provide an initial step toward discovering what language models know…

Paper and discussion history. A mention is not an endorsement.

Discovering Latent Knowledge in Language Models Without Supervision — mentioned by numeri

Fri, 24 Feb 2023 10:15:02 GMT

> The concept of factuality doesn't exist in the model at all. This is an example of a whole range of beliefs about LLMs that are very common (even in the field itself), because they were obviously true for small models, but that might not necessarily hold for larger models. There's a lot that we don't know about LLMs, but we do know that they exhibit emergent behaviors as they scale. Smaller models don't really have world models, just language models, but these larger models have started developing clear world models once given the capacity and data to do so. As for the existence of a…

Paper and discussion history. A mention is not an endorsement.

Discovering Latent Knowledge in Language Models Without Supervision — mentioned by nl

Thu, 23 Feb 2023 22:31:42 GMT

> does anyone know of ML directions that could add any kind of factual confidence level to ChatGPT and similar? Yes. It's a very active area of research. For example: Discovering Latent Knowledge in Language Models Without Supervision (https://arxiv.org/abs/2212.03827) shows an unsupervised approach for probing a LLM to discover things it thinks are facts Locating and Editing Factual Associations in GPT (https://arxiv.org/pdf/2202.05262.pdf) shows an approach to editing a LLM to edit facts. Language Models as Knowledge Bases? (https://aclanthology.org/D19-1250.pdf) is some slightly older work…

Paper and discussion history. A mention is not an endorsement.

Discovering Latent Knowledge in Language Models Without Supervision — mentioned by visarga

Mon, 13 Feb 2023 08:57:00 GMT

I think LLM checking is going to be the most important research direction this year. Generative models are worthless without verification. It relates to deciding truth and dealing with fake information, synthetic text and spam. Google hasn't been able to solve it in the last decade, but some people have an idea: > "Discovering Latent Knowledge in Language Models Without Supervision" Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that…

Paper and discussion history. A mention is not an endorsement.

Discovering Latent Knowledge in Language Models Without Supervision — mentioned by jimrandomh

Tue, 07 Feb 2023 19:01:55 GMT

And that research is valuable, but it's definitely nowhere near the point where safety/alignment progress can't be made without a user base. A few recent favorites of mine: https://arxiv.org/pdf/2212.03827.pdf https://www.alignmentforum.org/posts/aPeJE8bSo6rAFoLqg/solid...

Paper and discussion history. A mention is not an endorsement.

Discovering Latent Knowledge in Language Models Without Supervision — mentioned by visarga

Thu, 19 Jan 2023 23:31:47 GMT

> blatantly wrong info being spewed with 105% confidence There are some approaches. For example in this paper they say truth has a certain logical consistency that is lacking in hallucinations and deception. So they find this latent direction that indicates truth in a frozen LLM. This actually works better than asking the model to self evaluate by text generation, or training with RLHF. "Discovering Latent Knowledge in Language Models Without Supervision" https://arxiv.org/abs/2212.03827 There's also a video with the first author: "Making LLMs Say The Truth"…

Paper and discussion history. A mention is not an endorsement.

Discovering Latent Knowledge in Language Models Without Supervision — HN submission

Mon, 12 Dec 2022 03:30:51 GMT

Discovering latent knowledge in language models without supervision

149 points; 85 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Discovering Latent Knowledge in Language Models Without Supervision — HN submission

Sat, 10 Dec 2022 19:06:35 GMT

Discovering Latent Knowledge in Language Models Without Supervision

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Discovering Latent Knowledge in Language Models Without Supervision — mentioned by astrange

Fri, 09 Dec 2022 03:23:14 GMT

Apparently you can avoid hallucination by basically reading the model’s mind instead of asking it questions. https://arxiv.org/abs/2212.03827 Raises some ethical issues…

Paper and discussion history. A mention is not an endorsement.