Gemma 2: Improving Open Language Models at a Practical Size — mentioned by tucnak

Tue, 15 Oct 2024 08:12:00 GMT

The LIMA paper, I think, would be a good place to start https://arxiv.org/abs/2305.11206 You can create inputs for DPO/ORPO synthetically which is a huge one as previously it would require gigantic investments https://arxiv.org/abs/2402.10379 There's also the gemma2 paper has advanced SOTA in distil; on a side-note, there's many reasons for it but vocab_size and good sizes 9b/27b, IMHO it's currently the best model for i.e. Ukrainian. in fact, I prefer it to anything else there's, including the much larger llama's—by a mile! The model is a triumph of synthetic datasets.…

Paper and discussion history. A mention is not an endorsement.

Gemma 2: Improving Open Language Models at a Practical Size · HackerPapers discussions

Gemma 2: Improving Open Language Models at a Practical Size — mentioned by tucnak