Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities · HackerPapers discussionshttps://hackerpaper.kaipfister.de/p/2404.17790/Collected HN comments and submissions linking this paper.enMon, 02 Nov 2026 11:16:22 GMTContinual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities — mentioned by mcychttps://news.ycombinator.com/item?id=48100560urn:hackerpapers:mention:2404.17790:48100560Mon, 11 May 2026 20:56:30 GMT<p>You are right about most tokenizers being heavily biased towards English, but the situation is not so bad for Portuguese. Here are some results on the Goldfish corpus [1] with a few different tokenizers. This measures #characters in corpus / #subwords in tokenized corpus. ``` Llama3 english, 0.216 portuguese, 0.285 italian, 0.287 greek, 0.592 ``` ``` Gemma4 english, 0.219 portuguese, 0.246 italian, 0.249 greek, 0.537 ``` ``` Kimi2.6 english, 0.214 portuguese, 0.310 italian, 0.308 greek, 0.716 ``` Portuguese is worse than English certainly, but it is on par with Italian (which I think has more…</p><p><a href="https://hackerpaper.kaipfister.de/p/2404.17790/">Paper and discussion history</a>. A mention is not an endorsement.</p>cs.CL