HackerPapers

arXiv:2412.15210cs.DS

Tokenisation is NP-Complete

Philip Whittington, Gregor Bachmann, Tiago Pimentel

HN submission. Found in a submission link or body. Submitters do not count as commenters in the main ranking; a submission is not an endorsement.

Abstract

In this work, we prove the NP-completeness of two variants of tokenisation, defined as the problem of compressing a dataset to at most $δ$ symbols by either finding a vocabulary directly (direct tokenisation), or selecting a sequence of merge operations (bottom-up tokenisation).

arXiv comments
Journal reference
DOI
Subjects
cs.DSData Structures and Algorithms cs.CLComputation and Language cs.FLFormal Languages and Automata Theory

On Hacker News

No comments linking this paper were found in the collected sources. Found in 1 HN submission from Dec 2024 to Dec 2024. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments0 threads

No comments linking this paper were found in the collected sources. Open submission discussions.

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.