Think before you speak: Training Language Models With Pause Tokens · HackerPapers discussions

Think before you speak: Training Language Models With Pause Tokens — mentioned by ijk

Mon, 06 Oct 2025 06:24:32 GMT

So, what I think most people don't realize is that the amount of computation an LLM can do in one pass is strictly bounded. You can see that here with the layers. (This applies to a lot of neural networks [1].) Remember, they feed in the context on one side of the network, pass it through each layer doing matrix multiplication, and get a value on the other end that we convert back into our representation space. You can view the bit in the middle as doing a kind of really fancy compression, if you like. The important thing is that there are only so many layers, and thus only so many…

Paper and discussion history. A mention is not an endorsement.

Think before you speak: Training Language Models With Pause Tokens — mentioned by blackbear_

Sat, 09 Aug 2025 06:55:46 GMT

Good thought, that indeed works: https://arxiv.org/abs/2310.02226

Paper and discussion history. A mention is not an endorsement.

Think before you speak: Training Language Models With Pause Tokens — mentioned by layoric

Thu, 22 May 2025 22:37:19 GMT

There are several papers pointing towards 'thinking' output is meaningless to the final output, and using dots, or pause tokens enabling the same additional rounds of throughput result in similar improvements. So in a lot of regards the 'thinking' is mostly marketing. - "Think before you speak: Training Language Models With Pause Tokens" - https://arxiv.org/abs/2310.02226 - "Let's Think Dot by Dot: Hidden Computation in Transformer Language Models" - https://arxiv.org/abs/2404.15758 - "Do LLMs Really Think Step-by-step In Implicit Reasoning?" - https://arxiv.org/abs/2411.15862 - Video by…

Paper and discussion history. A mention is not an endorsement.

Think before you speak: Training Language Models With Pause Tokens — mentioned by wlib

Sun, 23 Mar 2025 17:08:18 GMT

The filler tokens actually do make them think more. Even just allowing the models to output "." until they are confident enough to output something increases their performance. Of course, training the model to do this (use pause tokens) on purpose works too: https://arxiv.org/pdf/2310.02226

Paper and discussion history. A mention is not an endorsement.

Think before you speak: Training Language Models With Pause Tokens — mentioned by refulgentis

Sun, 23 Mar 2025 01:31:18 GMT

Latent space has a certain shape, which may mean I'm missing a technical distinction.* There's been publications with a pause token (https://arxiv.org/abs/2310.02226), backspace token (https://arxiv.org/abs/2306.05426), or a think token (https://arxiv.org/html/2405.08644v1#Ch0.S4), all of them based on the theory that a generic token can sort of act a placeholder for manipulating attention further without further meaningful output. However, in practice, those approaches haven't been used in the training of a large scale model, i.e. I haven't seen it at all, the most adventurous people have…

Paper and discussion history. A mention is not an endorsement.

Think before you speak: Training Language Models With Pause Tokens — mentioned by mountainriver

Fri, 25 Oct 2024 23:33:42 GMT

https://arxiv.org/abs/2310.02226

Paper and discussion history. A mention is not an endorsement.

Think before you speak: Training Language Models With Pause Tokens — mentioned by famouswaffles

Wed, 04 Sep 2024 19:56:00 GMT

Transformer produce the next token by manipulating K hidden vectors per layer, one vector per preceding token. So yes you can increase compute length arbitrarily by increasing tokens. Those tokens don't have to carry any information to work. https://arxiv.org/abs/2310.02226 And again, human brains are clearly limited in the number of steps it can compute without writing something down. Limited =/ Trivial >FYI, "attention is all you need" has the implicit context of "if all you want to build is a language model". Great. Do you know what a "language model" is capable of in the limit ? No These…

Paper and discussion history. A mention is not an endorsement.

Think before you speak: Training Language Models With Pause Tokens — mentioned by wongarsu

Sat, 06 Jul 2024 17:53:29 GMT

Pause tokens [1] are directly dependent on the model recognizing whether the answer it arrived at is one it should "commit" to or whether it should abandon it and output a pause token instead. Similarly if you ask ChatGPT about the current president of Numbitistan it will tell you that it doesn't know about a county with that name, rather than just hallucinating an answer. So it can at least in this circumstance tell the difference between knowing something and not knowing something. 1: https://arxiv.org/abs/2310.02226

Paper and discussion history. A mention is not an endorsement.

Think before you speak: Training Language Models With Pause Tokens — mentioned by wongarsu

Mon, 17 Jun 2024 17:37:58 GMT

In the GPT3 days when everyone was doing few-shot tasks (giving the LLM a couple of examples of question/answer pairs in the prompt) one of the big insights was that adding question/answer pairs with answers like "I don't know" and "this question doesn't make sense" caused the model to actually use those answers appropriately instead of overconfidently stating nonsense. Of course that method isn't perfect (GPT3.0 was far from perfect in general). But both in principle and in practice the models do have a notion of what they "know". Knowledge is a strong activation, random noise is a weaker…

Paper and discussion history. A mention is not an endorsement.

Think before you speak: Training Language Models With Pause Tokens — mentioned by FeepingCreature

Sun, 12 May 2024 11:48:01 GMT

Saying that LLMs should be able to do problems without CoT is a bit like saying that humans should be able to write programs without thinking though. CoT is fundamentally an architectural necessity owed to the finite cognitive effort possible per token. Let it be CoT, let it be QuietSTaR ( https://arxiv.org/abs/2403.09629 ), let it be pause tokens ( https://arxiv.org/abs/2310.02226 ), let it even be dots ( https://arxiv.org/abs/2404.15758 ), but the thinking has to happen somewhere. I think mostly, LLMs at the moment are incredibly uneven. LLM assistants can pull obscure knowledge out of…

Paper and discussion history. A mention is not an endorsement.

Think before you speak: Training Language Models With Pause Tokens — mentioned by radarsat1

Mon, 22 Apr 2024 04:22:35 GMT

Seems very similar to the Think Before You Speak paper. https://arxiv.org/abs/2310.02226

Paper and discussion history. A mention is not an endorsement.

Think before you speak: Training Language Models With Pause Tokens — mentioned by kelseyfrog

Fri, 22 Mar 2024 17:28:36 GMT

There's a recent paper which seeks to explicitly perform time-to-think using pause tokens[1]. > However sophisticated this end-to-end process may be, it abides by a peculiar constraint: the number of operations determining the next token is limited by the number of tokens seen so far. There are obviously pros and cons to each, but nothing excludes us from combining the two either. 1. Think before you speak: Training Language Models With Pause Tokens https://arxiv.org/abs/2310.02226v2

Paper and discussion history. A mention is not an endorsement.

Think before you speak: Training Language Models With Pause Tokens — mentioned by spyder

Sun, 10 Mar 2024 20:06:07 GMT

Yep, humans too have to think before answering most non-trivial questions, and especially the ones that include calculations. So it seems "obvious" that we should try to to give LLMs too some time to think before answering, for example with the popular methods of asking for step-by-step thinking, thinking out loud, and only giving the final answer at the end, and also asking it to proofread and correct it's answers at the end all can help with that. Pause tokens (thinking tokens) are also an interesting method to achieve that and seems to have a positive effect on performance…

Paper and discussion history. A mention is not an endorsement.

Think before you speak: Training Language Models With Pause Tokens — mentioned by famouswaffles

Mon, 26 Feb 2024 18:41:55 GMT

>What's actually happening is that by asking a model to generate more tokens, it increases the amount of information it has learnt to be present in its context block. I'm not saying this isn't part of it but even if it's just dummy tokens without any new information, it works. https://arxiv.org/abs/2310.02226

Paper and discussion history. A mention is not an endorsement.

Think before you speak: Training Language Models With Pause Tokens — mentioned by famouswaffles

Mon, 26 Feb 2024 01:02:15 GMT

I think it's pretty clear https://arxiv.org/abs/2310.02226 I mean, i can imagine you wouldn't always need the extra compute.

Paper and discussion history. A mention is not an endorsement.

Think before you speak: Training Language Models With Pause Tokens — mentioned by famouswaffles

Tue, 17 Oct 2023 19:35:14 GMT

Transformers with memory can abstract recursively in theory. https://arxiv.org/abs/2301.04589. If it's just extra computation that's the kick then simply more tokens will suffice. You can even even implement a computation token if you wish. https://arxiv.org/abs/2310.02226 Frankly, the ability to truly abstract recursively is by no means necessary either. Humans can't actually do this. Either in every day practice (you will get bored, you will lose interest) or in theory (flawed memory, finite memory, you will die). Limited recursive abilities =/ trivial recursive abilities. The current state…

Paper and discussion history. A mention is not an endorsement.

Think before you speak: Training Language Models With Pause Tokens — mentioned by Difwif

Fri, 13 Oct 2023 22:56:11 GMT

Aren't LLM's already zero shot time series predictors? Predicting next token and forecasting seem like the exact same problem. I will admit some small tweaks in tokenization could help but it seems like we're just pretraining on a different dataset. One idea I was interested in was after reading the paper on introducing pause tokens[1] was a multimodal architecture that generalizes everything to parallel time series streams of tokens in different modalities. Pause tokens make even more sense in that setup. 1. https://arxiv.org/abs/2310.02226 -Arm chair ML practitioner

Paper and discussion history. A mention is not an endorsement.

Think before you speak: Training Language Models With Pause Tokens — mentioned by simonw

Fri, 13 Oct 2023 17:17:31 GMT

There's an interesting new paper about this problem: https://arxiv.org/abs/2310.02226 "Think before you speak: Training Language Models With Pause Tokens" Basic idea is to teach the LLM to occasionally insert a "pause" token, which outputs nothing but gives it a chance to perform another round of operations on the way to the answer.

Paper and discussion history. A mention is not an endorsement.

Think before you speak: Training Language Models With Pause Tokens — HN submission

Wed, 04 Oct 2023 12:54:33 GMT

Training language models with pause tokens

154 points; 69 discussion replies on HN. Paper and source history. A submission is not an endorsement.