HackerPapers

arXiv:2210.07229cs.CL

Mass-Editing Memory in a Transformer

Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, David Bau

Abstract

Recent work has shown exciting promise in updating large language models with new memories, so as to replace obsolete information or add specialized knowledge. However, this line of work is predominantly limited to updating single associations. We develop MEMIT, a method for directly updating a language model with many memories, demonstrating experimentally that it can scale up to thousands of associations for GPT-J (6B) and GPT-NeoX (20B), exceeding prior work by orders of magnitude. Our code and data are at https://memit.baulab.info.

arXiv comments
18 pages, 11 figures. Code and data at https://memit.baulab.info
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.LGMachine Learning

On Hacker News

Linked by 3 identifiable commenters in 3 comments across 3 threads, from Dec 2022 to May 2023. Found in 1 HN submission from Aug 2023 to Aug 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments3 threads

  1. How does data removal from GPT *work 1 comment

    • Transformer editing is an active field of research. https://arxiv.org/abs/2202.05262 https://arxiv.org/abs/2210.07229

      bhickeyView on HNParent context

  2. Can large language models reason about medical questions? 1 comment

    • Interpolation is a leaky analogy. With scale these models acquire increasingly sparse discrete-like activation patterns, and there are techniques[1] amounting to pinpointing the exact locations of the knowledge and updating it freely. It seems that nonlinear interpolation in a…

      sinenomineView on HNParent context

  3. People tricking ChatGPT “like watching an Asimov novel come to life” 1 comment

    • It is an actual question and response, yes. It is slightly tricky, in that most upfront questions will cause ChatGPT to answer “I don't have personal beliefs or political views, and I don't endorse any particular ideology or political party.” I needed to show an example correct…

      espadrineView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.