Do NLP Models Know Numbers? Probing Numeracy in Embeddings — mentioned by moyix

Fri, 11 Feb 2022 18:10:38 GMT

Hey! As the author of the gist, just wanted to clear up what seem to be a few misconceptions: - This isn't GPT-3, it's the recently-released open-source and open-weights model from EleutherAI, GPT-NeoX-20B. GPT-3 is much larger (175 billion parameters vs NeoX's 20 billion). - It's well-known that language models don't tend to be good at math by default (Gwern, among others, pointed this out back in June 2020). It seems likely that this is at least in part because of how these models currently tokenize their input (they don't represent numbers by their individual digits, but by tokens…

Paper and discussion history. A mention is not an endorsement.

Do NLP Models Know Numbers? Probing Numeracy in Embeddings — mentioned by gradys

Fri, 11 Feb 2022 15:45:24 GMT

You might like the paper "Do NLP Models Know Numbers? Probing Numeracy in Embeddings": https://arxiv.org/abs/1909.07940 Neural network models seem to encode an approximate notion of quantity in their representations. This paper is pre-GPT-3, but I would think the larger training set and larger model capacity would help the model learn quantity more easily.

Paper and discussion history. A mention is not an endorsement.

Do NLP Models Know Numbers? Probing Numeracy in Embeddings · HackerPapers discussions

Do NLP Models Know Numbers? Probing Numeracy in Embeddings — mentioned by moyix

Do NLP Models Know Numbers? Probing Numeracy in Embeddings — mentioned by gradys