Effective Long-Context Scaling of Foundation Models — mentioned by lappa

Wed, 01 Nov 2023 17:38:23 GMT

Excellent, appears Amazon has introduced two important things here: - Rope theta of 100,000, likely from the Llama 2 Long paper which found that a large theta helped regulate attention between distant tokens[0] - A 16k (effective 32k) context window, improving upon Mistrals 4k (effective 8k) context window In The Llama 2 Long paper, they saw improvement in short context benchmarks as a result of long context fine tuning. I can't find any of the expected MMLU / HellaSwag / etc benchmarks yet. Benchmarks haven't been submitted to MTEB yet. Some user anecdotally seem to be having trouble with…

Paper and discussion history. A mention is not an endorsement.

Effective Long-Context Scaling of Foundation Models — HN submission

Fri, 29 Sep 2023 02:27:53 GMT

Llama 2 Long

134 points; 31 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Effective Long-Context Scaling of Foundation Models · HackerPapers discussions

Effective Long-Context Scaling of Foundation Models — mentioned by lappa

Effective Long-Context Scaling of Foundation Models — HN submission