Learning higher-order sequential structure with cloned HMMs · HackerPapers discussionshttps://hackerpaper.kaipfister.de/p/1905.00507/Collected HN comments and submissions linking this paper.enMon, 02 Nov 2026 11:16:22 GMTLearning higher-order sequential structure with cloned HMMs — mentioned by ssivarkhttps://news.ycombinator.com/item?id=45996794urn:hackerpapers:mention:1905.00507:45996794Thu, 20 Nov 2025 19:46:19 GMT<p>The Markov property means that the next token is determined purely by the current token. Well, if it were a Hidden Markov Model, the next state would actually be determined by the current state, and the respective tokens would be a lossy representation of the states. The problem with HMMs is that the sequence model (Markov transition matrix) accounts for much less context than even Tiny LLMs. One natural way to improve this is to allow the model to have more hidden states, representing more context -- called &quot;clones&quot; because these different hidden states would all be producing the same token…</p><p><a href="https://hackerpaper.kaipfister.de/p/1905.00507/">Paper and discussion history</a>. A mention is not an endorsement.</p>stat.ML