Tokenizer Choice For LLM Training: Negligible or Crucial? — mentioned by mcyc

Mon, 23 Dec 2024 07:48:49 GMT

Hi, I'm Cognetta from the above Cognetta et al. I can't answer all of your questions (and I can't speak for the authors of this paper ofc), but I will try to answer some. > Is a tokenizer that maximizes the compression of text (e.g. by identifying longer tokens that tend to be used whole) necessarily a better tokenizer, in terms of overall model performance? Compression might be a useful property for an objective function to consider... but then again maybe not, if it makes the problem NP-hard. Compression isn't necessarily the best metric for language modeling quality [1][2][3], but there…

Paper and discussion history. A mention is not an endorsement.

Tokenizer Choice For LLM Training: Negligible or Crucial? — HN submission

Tue, 16 Jan 2024 13:18:38 GMT

Tokenizer Choice for LLM Training: Negligible or Crucial?

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Tokenizer Choice For LLM Training: Negligible or Crucial? · HackerPapers discussions

Tokenizer Choice For LLM Training: Negligible or Crucial? — mentioned by mcyc

Tokenizer Choice For LLM Training: Negligible or Crucial? — HN submission