Dissecting Recall of Factual Associations in Auto-Regressive Language Models · HackerPapers discussions

Dissecting Recall of Factual Associations in Auto-Regressive Language Models — mentioned by numeri

Thu, 16 Nov 2023 02:03:48 GMT

This post isn't a scientific investigation. It's someone playing around with a black-box model with few to little controls – which is unfortunately the only thing we can do when experimenting with GPT-4, which does excuse this partially. Unfortunately, though, the author seems unaware of the actual state of research on the actual mechanics of how LLMs store knowledge and specifically binary relations. The ROME paper[1], among others, shows that the feed-forward layers function as a key-value store, where the feed-forward's up projection of the last token in a noun phrase (say, "the Eiffel…

Paper and discussion history. A mention is not an endorsement.

Dissecting Recall of Factual Associations in Auto-Regressive Language Models — HN submission

Wed, 03 May 2023 04:21:27 GMT

Dissecting Recall of Factual Associations in Auto-Regressive Language Models

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Dissecting Recall of Factual Associations in Auto-Regressive Language Models — HN submission

Mon, 01 May 2023 13:14:08 GMT

Dissecting Recall of Factual Associations in Auto-Regressive Language Models

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.