HackerPapers

arXiv:2310.02989stat.ML

xVal: A Continuous Numerical Tokenization for Scientific Language Models

Siavash Golkar, Mariel Pettee, Michael Eickenberg, Alberto Bietti, Miles Cranmer, Geraud Krawezik, Francois Lanusse, Michael McCabe, Ruben Ohana, Liam Parker, Bruno Régaldo-Saint Blancard, Tiberiu Tesileanu, Kyunghyun Cho, Shirley Ho

Abstract

Due in part to their discontinuous and discrete default encodings for numbers, Large Language Models (LLMs) have not yet been commonly used to process numerically-dense scientific datasets. Rendering datasets as text, however, could help aggregate diverse and multi-modal scientific data into a single training corpus, thereby potentially facilitating the development of foundation models for science. In this work, we introduce xVal, a strategy for continuously tokenizing numbers within language models that results in a more appropriate inductive bias for scientific applications. By training specially-modified language models from scratch on a variety of scientific datasets formatted as text, we find that xVal generally outperforms other common numerical tokenization strategies on metrics including out-of-distribution generalization and computational efficiency.

arXiv comments
15 pages, 12 figures. Appendix: 8 pages, 2 figures. Accepted contribution at the NeurIPS Workshop on ML for the Physical Sciences
Journal reference
DOI
Subjects
stat.MLMachine Learning (Statistics) cs.AIArtificial Intelligence cs.CLComputation and Language cs.LGMachine Learning

On Hacker News

Linked by 5 identifiable commenters in 12 comments across 8 threads, from Oct 2023 to Oct 2024. Found in 1 HN submission from Oct 2023 to Oct 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments8 threads

  1. Probably pay attention to tokenizers 1 comment

    • Tokenizers aren't considered the "sexy" part of LLMs, but where others see boring, I see opportunity. Papers like xVal[1], point toward specialization strategies in tokenization. Spelling and letter tasks are another problem that could benefit from innovation on the…

      kelseyfrogView on HNParent context

  2. AGI is far from inevitable 1 comment

    • > there are some benchmarks which show fundamental inability of LLM perform certain tasks which human can, for example add 100 digits numbers. fundamental inability ? No. Current Sota LLM (4o, claude, gemini) woes with arithmetic is not a transformer weakness never mind a large…

      famouswafflesView on HNParent context

  3. Integer Tokenization Is Insane (2023) 2 comments

    • >It has seemed to me that the GPT would be considerably better at numbers if it just considered each digit as a token. Has anyone actually done an experiment to test this? Yeah Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs…

      famouswafflesView on HNParent context

    • > Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs - https://arxiv.org/abs/2402.14903 Very interesting paper. It does make sense to me the R2L chunking would be better than L2R chunking. It doesn't actually study single digit tokenization. I am…

      thethirdoneView on HNParent context

  4. OpenAI researchers warned board of AI breakthrough ahead of CEO ouster 2 comments

    • > You can't really train them to do math unless they've seen previous calculations with similar numbers in their training sets. Generally true before xVal[1], but not exactly true after. Expect fairly basic advancement in the future to solve this problem completely. 1.…

      kelseyfrogView on HNParent context

    • https://arxiv.org/abs/2310.02989 ("xVal: A Continuous Number Encoding for Large Language Models", Submitted on 4 Oct 2023)

      reroute22View on HNParent context

  5. Sutskever: OpenAI board doing its mission to build AGI that benefits all 1 comment

    • >But in one sentence, try asking ChatGPT to reverse a string of 20+ digits, like 473936482738338373926. LLMs don't see numbers or letters but tokens. Letter, digit level manipulation is intuitively a hard task. Unsurprisingly, removing that handicap resulted in much better…

      famouswafflesView on HNParent context

  6. Pretraining data enables narrow selection capabilities in transformer models 3 comments

    • GPT-4 can do arithmetic just fine. In fact, despite being severely hobbled by Tokenization (https://arxiv.org/abs/2310.02989), it can perform arithmetic better than you could without a dedicated tool or pad.

      famouswafflesView on HNParent context

    • Damn I linked the wrong paper lol. Meant to link this. More pertinent to my point on Tokenization https://arxiv.org/abs/2310.02989

      famouswafflesView on HNParent context

    • The point is that this is just blatantly wrong. >Asking a token prediction model to do math is like asking a human to do math without doing the math. What's 9 times 9? I can tell you it's 81 from sheer memorization. I can probably invoke 9 x 9 + 1 = 82 without needing to do any…

      famouswafflesView on HNParent context

  7. Telling GPT-4 you're scared or under pressure improves performance 1 comment

    • It's better (GPT-4) than you could manage without an external tool or pad. and that's after being severely hampered by tokenization. https://arxiv.org/abs/2310.02989

      famouswafflesView on HNParent context

  8. Single Digit tokenization improves LLM math abilities by up to 70x 1 comment

    • My apologies if this has already been submitted. I kept the title used by u/yahma on r/LocalLLaMa as it's a good straight to the point title. The paper is here https://arxiv.org/abs/2310.02989

      lawlessoneView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.