SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore · HackerPapers discussions

SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore — mentioned by jpfed

Mon, 28 Aug 2023 16:24:09 GMT

The most promising work along these lines centers around augmenting LLMs with an external data store ("retrieval-augmented LLM"s). I think this started with Facebook's KNN-LLM ( https://arxiv.org/pdf/1911.00172.pdf ). Legal conflict may force the industry to move towards vector DBs as the predominant method by which facts are "stored" rather than model parameters ( https://arxiv.org/pdf/2308.04430.pdf ) , with the happy side effect of update-ability over time.

Paper and discussion history. A mention is not an endorsement.

SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore — mentioned by regularfry

Tue, 15 Aug 2023 12:47:24 GMT

https://arxiv.org/pdf/2308.04430.pdf is interesting from that point of view. They've tackled it from the perspective of avoiding copyright content in training, but including it in inference but I think it ought to mean less hallucination because they also (claim to) solve the attribution problem.

Paper and discussion history. A mention is not an endorsement.

SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore — mentioned by regularfry

Tue, 15 Aug 2023 12:44:50 GMT

There was another paper out recently that adds to this: https://arxiv.org/pdf/2308.04430.pdf. Looks like a more flexible approach to document storage, and it outperforms retrieval in context. They trained up their own LLM, but from the text it seems like it might be possible to use any LLaMA-style LM without retraining. Not sure though, need to give it a proper look.

Paper and discussion history. A mention is not an endorsement.