xVal: A Continuous Numerical Tokenization for Scientific Language Models · HackerPapers discussions

xVal: A Continuous Numerical Tokenization for Scientific Language Models — mentioned by kelseyfrog

Wed, 23 Oct 2024 16:45:56 GMT

Tokenizers aren't considered the "sexy" part of LLMs, but where others see boring, I see opportunity. Papers like xVal[1], point toward specialization strategies in tokenization. Spelling and letter tasks are another problem that could benefit from innovation on the tokenization. LLMs are notoriously bad at counting letters in words or performing simply oulipos of letter omission. GPT-4o, for example, writes a small python program and executes it in order to count letter instances. We all know that tokenization effectively erases knowledge about letters in prompts and directly negatively…

Paper and discussion history. A mention is not an endorsement.

xVal: A Continuous Numerical Tokenization for Scientific Language Models — mentioned by famouswaffles

Mon, 30 Sep 2024 23:47:38 GMT

> there are some benchmarks which show fundamental inability of LLM perform certain tasks which human can, for example add 100 digits numbers. fundamental inability ? No. Current Sota LLM (4o, claude, gemini) woes with arithmetic is not a transformer weakness never mind a large language modelling one. Those benchmarks show that those particular models have problems with accuracy on that many digits, not that LLMs fundamentally cannot be accurate on that many digits. https://arxiv.org/abs/2405.17399 https://arxiv.org/abs/2307.03381 https://arxiv.org/abs/2310.02989 Ultimately, numbers are…

Paper and discussion history. A mention is not an endorsement.

xVal: A Continuous Numerical Tokenization for Scientific Language Models — mentioned by thethirdone

Sat, 16 Mar 2024 23:58:38 GMT

> Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs - https://arxiv.org/abs/2402.14903 Very interesting paper. It does make sense to me the R2L chunking would be better than L2R chunking. It doesn't actually study single digit tokenization. I am mostly interested in a direct comparison between an LLM wide tokenization vs single digit tokenization. It would be nice to see a direct comparison between similarly trained models. Otherwise it is very hard to get a definitive answer by comparing models with varying sizes, training time, and general strength. > xVal: A…

Paper and discussion history. A mention is not an endorsement.

xVal: A Continuous Numerical Tokenization for Scientific Language Models — mentioned by famouswaffles

Sat, 16 Mar 2024 21:27:30 GMT

>It has seemed to me that the GPT would be considerably better at numbers if it just considered each digit as a token. Has anyone actually done an experiment to test this? Yeah Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs - https://arxiv.org/abs/2402.14903 xVal: A Continuous Number Encoding for Large Language Models - https://arxiv.org/abs/2310.02989 I believe there's another paper that demonstrates something like also for the likes of spelling, counting etc but i can't remember it.

Paper and discussion history. A mention is not an endorsement.

xVal: A Continuous Numerical Tokenization for Scientific Language Models — mentioned by reroute22

Thu, 23 Nov 2023 06:32:25 GMT

https://arxiv.org/abs/2310.02989 ("xVal: A Continuous Number Encoding for Large Language Models", Submitted on 4 Oct 2023)

Paper and discussion history. A mention is not an endorsement.

xVal: A Continuous Numerical Tokenization for Scientific Language Models — mentioned by kelseyfrog

Wed, 22 Nov 2023 23:38:49 GMT

> You can't really train them to do math unless they've seen previous calculations with similar numbers in their training sets. Generally true before xVal[1], but not exactly true after. Expect fairly basic advancement in the future to solve this problem completely. 1. https://arxiv.org/pdf/2310.02989.pdf

Paper and discussion history. A mention is not an endorsement.

xVal: A Continuous Numerical Tokenization for Scientific Language Models — mentioned by famouswaffles

Sat, 18 Nov 2023 04:23:40 GMT

>But in one sentence, try asking ChatGPT to reverse a string of 20+ digits, like 473936482738338373926. LLMs don't see numbers or letters but tokens. Letter, digit level manipulation is intuitively a hard task. Unsurprisingly, removing that handicap resulted in much better arithmetic abilities. https://arxiv.org/abs/2310.02989 It's pretty telling in my opinion that this is the kind of "proof" people bank hard on. For something that so obviously "does not understand", you'd think providing a task that it would fail that a chunk of humans also wouldn't would be easy.

Paper and discussion history. A mention is not an endorsement.

xVal: A Continuous Numerical Tokenization for Scientific Language Models — mentioned by famouswaffles

Tue, 07 Nov 2023 23:02:44 GMT

The point is that this is just blatantly wrong. >Asking a token prediction model to do math is like asking a human to do math without doing the math. What's 9 times 9? I can tell you it's 81 from sheer memorization. I can probably invoke 9 x 9 + 1 = 82 without needing to do any calculation either. But if you ask me 32 * 64, that's very difficult to do without doing calculations. Implicitly doing math is not sensible. It's not about memorization. It's trivial to test on instances that would not appear in training and see GPT-4 be better than a human who would attempt the problem without a tool…

Paper and discussion history. A mention is not an endorsement.

xVal: A Continuous Numerical Tokenization for Scientific Language Models — mentioned by famouswaffles

Tue, 07 Nov 2023 17:49:20 GMT

Damn I linked the wrong paper lol. Meant to link this. More pertinent to my point on Tokenization https://arxiv.org/abs/2310.02989

Paper and discussion history. A mention is not an endorsement.

xVal: A Continuous Numerical Tokenization for Scientific Language Models — mentioned by famouswaffles

Tue, 07 Nov 2023 17:34:32 GMT

GPT-4 can do arithmetic just fine. In fact, despite being severely hobbled by Tokenization (https://arxiv.org/abs/2310.02989), it can perform arithmetic better than you could without a dedicated tool or pad.

Paper and discussion history. A mention is not an endorsement.

xVal: A Continuous Numerical Tokenization for Scientific Language Models — mentioned by famouswaffles

Sun, 05 Nov 2023 18:24:41 GMT

It's better (GPT-4) than you could manage without an external tool or pad. and that's after being severely hampered by tokenization. https://arxiv.org/abs/2310.02989

Paper and discussion history. A mention is not an endorsement.

xVal: A Continuous Numerical Tokenization for Scientific Language Models — mentioned by lawlessone

Thu, 19 Oct 2023 14:53:57 GMT

My apologies if this has already been submitted. I kept the title used by u/yahma on r/LocalLLaMa as it's a good straight to the point title. The paper is here https://arxiv.org/abs/2310.02989

Paper and discussion history. A mention is not an endorsement.

xVal: A Continuous Numerical Tokenization for Scientific Language Models — HN submission

Wed, 18 Oct 2023 23:04:58 GMT

xVal: A continuous number encoding for large language models

72 points; 11 discussion replies on HN. Paper and source history. A submission is not an endorsement.