LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning — mentioned by twobitshifter

Fri, 23 Feb 2024 02:18:40 GMT

There’s a few options. People think Gemini 1.5 is Sparse Mixture of Experts. (SMoE) Another One is self extend. https://arxiv.org/abs/2401.01325 This paper also refers back to other options like yarn, etc.

Paper and discussion history. A mention is not an endorsement.

LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning — mentioned by lhl

Thu, 11 Jan 2024 17:17:34 GMT

My understanding is that Mistral uses a regular 4K RoPE that is "extends" the window size with SWA. This is based on looking at the results of Nous Research's Yarn-Mistral extension: https://huggingface.co/NousResearch/Yarn-Mistral-7b-128k and Self-Extend, both of which only apply to RoPE models. There are quite a few recent attention extension techniques recently published: * Activation Beacons - up to 100X context length extension in as little as 72 A800 hours https://huggingface.co/papers/2401.03462 * Self-Extend - a no-training RoPE modification that can give "free" context extension with…

Paper and discussion history. A mention is not an endorsement.

LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning · HackerPapers discussions

LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning — mentioned by twobitshifter

LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning — mentioned by lhl