Text and Code Embeddings by Contrastive Pre-Training · HackerPapers discussions

Text and Code Embeddings by Contrastive Pre-Training — mentioned by heliophobicdude

Wed, 03 May 2023 14:45:58 GMT

Hi Jason! I have a few thoughts on this! Refined training is usually updating the weights of usually what's called a foundational model with well structured and numerous data. It's very expensive and can disrupt the usefulness of having all the generalizations baked in from training data [1]. While LLMs can generate text based on a wide range of inputs, they're not designed to retrieve specific pieces of information in the same way that a database or a search engine would. But I do think they hold a lot of promise in reasoning. Small corollary: LLMs do not know a head of time what they are…

Paper and discussion history. A mention is not an endorsement.

Text and Code Embeddings by Contrastive Pre-Training — mentioned by roknovosel

Fri, 19 Aug 2022 15:45:52 GMT

Great read, thanks for sharing. Would love to see the natural language + code mixed in there :) I've been interested in contrastive learning for a while, mainly as a means to train semantic code search models. OpenAI released a great paper on this topic called Text and Code Embeddings by Contrastive Pre-Training[1] that outlines the approach. I've used it as a base to build https://codesearch.ai [2] with pretty good results. [1] https://arxiv.org/pdf/2201.10005.pdf [2] https://sourcegraph.com/notebooks/Tm90ZWJvb2s6MTU1OQ==

Paper and discussion history. A mention is not an endorsement.