HackerPapers

arXiv:1008.1191cs.IR

Improved Fast Similarity Search in Dictionaries

Daniel Karch, Dennis Luxen, Peter Sanders

Abstract

We engineer an algorithm to solve the approximate dictionary matching problem. Given a list of words $\mathcal{W}$, maximum distance $d$ fixed at preprocessing time and a query word $q$, we would like to retrieve all words from $\mathcal{W}$ that can be transformed into $q$ with $d$ or less edit operations. We present data structures that support fault tolerant queries by generating an index. On top of that, we present a generalization of the method that eases memory consumption and preprocessing time significantly. At the same time, running times of queries are virtually unaffected. We are able to match in lists of hundreds of thousands of words and beyond within microseconds for reasonable distances.

arXiv comments
Full version of a short paper accepted for Spire 2010, 13 pages
Journal reference
DOI
Subjects
cs.IRInformation Retrieval cs.DSData Structures and Algorithms

On Hacker News

Linked by 2 identifiable commenters in 4 comments across 4 threads, from Jun 2015 to Mar 2022.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. SymSpell: 1M times faster spelling correction 1 comment

    • What they are calling "Symmetric Delete" seems to be the same as an older concept called "deletion neighborhoods". It is a term coined in an academic paper written by Thomas Bocek, Ela Hunt, and Burkhard Stiller from the University of Zurich, titled "Fast Similarity Search in…

      LordGreyView on HNParent context

  2. Dictionary of Algorithms and Data Structures (1998) 1 comment

    • There is a variety of approximate string matching algorithms that speed up search by using an index. https://arxiv.org/abs/1008.1191 is one that should be fairly easy to implement. Levenshtein automata are another approach that makes the rounds on HN every now and then, but are…

      lorenzhsView on HNParent context

  3. Levenshtein automata can be simple and fast 1 comment

    • It's already been answered why this isn't a good idea, but there is a working approach that works similarly: instead of all words within Levenshtein distance N, you store all words with exactly N characters deleted from them. So for each word in your dictionary, store all words…

      lorenzhsView on HNParent context

  4. Levenshtein Automata (2010) 1 comment

    • Levenshtein automata seem to pop up on here every once in a while. They are quite interesting from a theory perspective, but (like many things devised by the theory community) incredibly complex in practice. Lucene 4.0 uses them for fuzzy queries, you can read the full story of…

      lorenzhsView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.