Efficient Training of Language Models to Fill in the Middle · HackerPapers discussions

Efficient Training of Language Models to Fill in the Middle — mentioned by sunaookami

Tue, 07 Jan 2025 06:13:03 GMT

>I've been in the industry for nearly 25 years and have heard auto complete throughout but not once have I heard fill in the middle Then you need to look harder. FiM is a common approach for code generation LLMs. https://openai.com/index/efficient-training-of-language-mode... https://arxiv.org/abs/2207.14255 This was before ChatGPT's release btw.

Paper and discussion history. A mention is not an endorsement.

Efficient Training of Language Models to Fill in the Middle — mentioned by adt

Fri, 17 Mar 2023 23:52:15 GMT

Great! I always cite my sources, and you'll find a link to your page as usual. I wanted to point you towards OpenAI's FIM 6.9B as well. Trained on 100B tokens (Chinchilla-aligned), it was announced just before GPT-4 allegedly started training. I didn't see anyone else talking about it, but maybe you could follow the rabbit trail even further, so to speak! https://arxiv.org/abs/2207.14255

Paper and discussion history. A mention is not an endorsement.

Efficient Training of Language Models to Fill in the Middle — mentioned by moyix

Fri, 23 Dec 2022 04:16:06 GMT

OpenAI hasn't said exactly how they trained code-davinci-002 so this is speculative, but I'm reasonably sure it was trained on more data and languages than CodeGen and for longer. It was also trained using fill-in-the middle [1]. [1] https://arxiv.org/abs/2207.14255

Paper and discussion history. A mention is not an endorsement.

Efficient Training of Language Models to Fill in the Middle — mentioned by moyix

Wed, 03 Aug 2022 14:35:33 GMT

The largest Python model (codegen-16B-mono) should be pretty competitive! The biggest thing that it lacks is the ability to make use of context after the cursor. Unfortunately, that capability will require retraining the models: https://arxiv.org/abs/2207.14255

Paper and discussion history. A mention is not an endorsement.

Efficient Training of Language Models to Fill in the Middle — HN submission

Fri, 29 Jul 2022 05:57:13 GMT

Efficient Training of Language Models to Fill in the Middle

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.