Long Context Compression with Activation Beacon · HackerPapers discussions

Long Context Compression with Activation Beacon — mentioned by amitport

Mon, 15 Apr 2024 07:40:00 GMT

It is missing a proper evaluation section. Specifically, I wonder how this compares with https://arxiv.org/abs/2401.03462 (Activation Beacon)

Paper and discussion history. A mention is not an endorsement.

Long Context Compression with Activation Beacon — HN submission

Tue, 30 Jan 2024 14:33:39 GMT

Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Long Context Compression with Activation Beacon — HN submission

Tue, 16 Jan 2024 18:06:34 GMT

Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Long Context Compression with Activation Beacon — HN submission

Mon, 15 Jan 2024 21:50:57 GMT

Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Long Context Compression with Activation Beacon — mentioned by lhl

Thu, 11 Jan 2024 17:17:34 GMT

My understanding is that Mistral uses a regular 4K RoPE that is "extends" the window size with SWA. This is based on looking at the results of Nous Research's Yarn-Mistral extension: https://huggingface.co/NousResearch/Yarn-Mistral-7b-128k and Self-Extend, both of which only apply to RoPE models. There are quite a few recent attention extension techniques recently published: * Activation Beacons - up to 100X context length extension in as little as 72 A800 hours https://huggingface.co/papers/2401.03462 * Self-Extend - a no-training RoPE modification that can give "free" context extension with…

Paper and discussion history. A mention is not an endorsement.

Long Context Compression with Activation Beacon — HN submission

Thu, 11 Jan 2024 06:11:09 GMT

Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.