Scaling Transformer to 1M tokens and beyond with RMT · HackerPapers discussions

Scaling Transformer to 1M tokens and beyond with RMT — mentioned by vectoral

Thu, 27 Apr 2023 22:45:02 GMT

This is one of the areas of LLMs that I find most interesting. So far, I've found simple question-answering over vectorstores to be a lackluster experience. In particular, the more information you embed and stick into the vectorstore, the less useful the system becomes as you are less likely to get the information you're looking for (especially if the users don't understand their queries need to look like the docs the want to ask about. I haven't had a chance to try out hypothetical embedded docs yet, but I expect they only provide a marginal improvement (especially if QAing over proprietary…

Paper and discussion history. A mention is not an endorsement.

Scaling Transformer to 1M tokens and beyond with RMT — mentioned by ur-whale

Mon, 24 Apr 2023 07:35:24 GMT

>limited context length is an issue Fresh from HN: https://arxiv.org/abs/2304.11062

Paper and discussion history. A mention is not an endorsement.

Scaling Transformer to 1M tokens and beyond with RMT — HN submission

Mon, 24 Apr 2023 00:54:00 GMT

Scaling Transformer to 1M tokens and beyond with RMT

277 points; 132 discussion replies on HN. Paper and source history. A submission is not an endorsement.