YaRN: Efficient Context Window Extension of Large Language Models · HackerPapers discussions

YaRN: Efficient Context Window Extension of Large Language Models — mentioned by lhl

Thu, 11 Jan 2024 17:17:34 GMT

My understanding is that Mistral uses a regular 4K RoPE that is "extends" the window size with SWA. This is based on looking at the results of Nous Research's Yarn-Mistral extension: https://huggingface.co/NousResearch/Yarn-Mistral-7b-128k and Self-Extend, both of which only apply to RoPE models. There are quite a few recent attention extension techniques recently published: * Activation Beacons - up to 100X context length extension in as little as 72 A800 hours https://huggingface.co/papers/2401.03462 * Self-Extend - a no-training RoPE modification that can give "free" context extension with…

Paper and discussion history. A mention is not an endorsement.

YaRN: Efficient Context Window Extension of Large Language Models — mentioned by filterfiber

Thu, 30 Nov 2023 19:13:01 GMT

Yes! The obvious answer is to just increase your positions and train for that. This requires a ton of memory however (context length is squared) so most are currently training at 4k/8k and then finetuning higher similar to many of the image models. However there's been some work that to "get extra milage" out of the current models so-to speak with rotary positions and a few other tricks. These in combination with finetuning is the current method many are using at the moment IIRC. Here's a decent overview https://aman.ai/primers/ai/context-length-extension/ Rope…

Paper and discussion history. A mention is not an endorsement.