SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking · HackerPapers discussions

SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking — mentioned by krackers

Mon, 06 Oct 2025 18:59:32 GMT

There have been attempts to give LLMs backspace tokens. Since no frontier model uses it I can only guess it doesn't scale as well as just letting it correct itself in COT https://arxiv.org/abs/2306.05426

Paper and discussion history. A mention is not an endorsement.

SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking — mentioned by ijk

Mon, 06 Oct 2025 06:12:26 GMT

There's been a few attempts at training a backspace token, though. e.g.: https://arxiv.org/abs/2502.04404 https://arxiv.org/abs/2306.05426

Paper and discussion history. A mention is not an endorsement.

SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking — mentioned by skissane

Thu, 04 Sep 2025 00:22:13 GMT

LLM backtracking is an active area of research, see e.g. https://arxiv.org/html/2502.04404v1 https://arxiv.org/abs/2306.05426 And I was wrong that nobody has implemented it, as these papers prove people have… it is just the results haven’t been sufficiently impressive to support the transition from the research lab to industrial use - or at least, not yet

Paper and discussion history. A mention is not an endorsement.

SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking — mentioned by dev_hugepages

Sat, 09 Aug 2025 12:59:27 GMT

Research on the backspace token: https://arxiv.org/abs/2306.05426 > [...] The IL framework also allows us to incorporate backtracking by introducing a backspace action into the generation process [...]

Paper and discussion history. A mention is not an endorsement.

SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking — mentioned by pizza

Wed, 14 May 2025 03:23:02 GMT

Another one: SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking https://arxiv.org/abs/2306.05426 There was also an hn thread: https://news.ycombinator.com/item?id=36425375

Paper and discussion history. A mention is not an endorsement.

SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking — mentioned by twotwotwo

Thu, 01 May 2025 02:05:55 GMT

It's kind of weird to think that in a coding assistant, an LLM is regularly asked to produce a valid block of code top to bottom, or repeat a section of code with changes, when that's not what we do. (There are other intuitively odd things about this, like the amount of compute spent generating 'easy' tokens, e.g. repeating unchanged code.) Some of that might be that models are just weird and intuition doesn't apply. But maybe the way we do it--jumping around, correcting as we go, etc.--is legitimately an efficient use of effort, and a model could do its job better, with less effort, or both…

Paper and discussion history. A mention is not an endorsement.

SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking — mentioned by refulgentis

Sun, 23 Mar 2025 01:31:18 GMT

Latent space has a certain shape, which may mean I'm missing a technical distinction.* There's been publications with a pause token (https://arxiv.org/abs/2310.02226), backspace token (https://arxiv.org/abs/2306.05426), or a think token (https://arxiv.org/html/2405.08644v1#Ch0.S4), all of them based on the theory that a generic token can sort of act a placeholder for manipulating attention further without further meaningful output. However, in practice, those approaches haven't been used in the training of a large scale model, i.e. I haven't seen it at all, the most adventurous people have…

Paper and discussion history. A mention is not an endorsement.

SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking — mentioned by viraptor

Wed, 12 Jun 2024 12:48:15 GMT

The blockchain part is silly, because timestamping services exist without it https://www.sectigo.com/resource-library/time-stamping-serve... The rest is silly, because you can emulate the whole writing process by combining backtracking https://arxiv.org/abs/2306.05426 and a rewriting/rewording loop. With not much effort we can make LLM output look incredibly painstaking.

Paper and discussion history. A mention is not an endorsement.

SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking — mentioned by PartiallyTyped

Tue, 27 Jun 2023 10:46:01 GMT

> That's called beam search. There are other decoding approaches, other than beam search: https://arxiv.org/abs/2202.00666 > Other than that, things like gpt can't go back and edit what they've outputted Actually, they can, they may emit a backspace token [1] [1] https://arxiv.org/abs/2306.05426, https://news.ycombinator.com/item?id=36425375

Paper and discussion history. A mention is not an endorsement.

SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking — HN submission

Wed, 21 Jun 2023 22:10:35 GMT

Giving LLM’s a <Backspace> Token

82 points; 25 discussion replies on HN. Paper and source history. A submission is not an endorsement.