Thinking Tokens for Language Modeling · HackerPapers discussions

Thinking Tokens for Language Modeling — mentioned by refulgentis

Sun, 23 Mar 2025 01:31:18 GMT

Latent space has a certain shape, which may mean I'm missing a technical distinction.* There's been publications with a pause token (https://arxiv.org/abs/2310.02226), backspace token (https://arxiv.org/abs/2306.05426), or a think token (https://arxiv.org/html/2405.08644v1#Ch0.S4), all of them based on the theory that a generic token can sort of act a placeholder for manipulating attention further without further meaningful output. However, in practice, those approaches haven't been used in the training of a large scale model, i.e. I haven't seen it at all, the most adventurous people have…

Paper and discussion history. A mention is not an endorsement.

Thinking Tokens for Language Modeling — mentioned by wongarsu

Fri, 03 Jan 2025 12:01:06 GMT

You should get decent results by asking it to do that in the prompt. Just add "if you are uncertain, answer I don't know" or "give the answer or say I don't know" or something along those lines LLM are far from perfect at knowing their limits, but they are better at it than most people give them credit for. They just never do it unless prompted for it. Fine tuning can improve that ability. For example the thinking tokens paper [1] is at some level training the model to output a special token when it doesn't reach a good answer (and then try again, thus "thinking") 1…

Paper and discussion history. A mention is not an endorsement.

Thinking Tokens for Language Modeling — HN submission

Fri, 17 May 2024 01:11:04 GMT

Thinking Tokens for Language Modeling

6 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.