Improved Fast Similarity Search in Dictionaries · HackerPapers discussions

Improved Fast Similarity Search in Dictionaries — mentioned by LordGrey

Sun, 06 Mar 2022 14:51:14 GMT

What they are calling "Symmetric Delete" seems to be the same as an older concept called "deletion neighborhoods". It is a term coined in an academic paper written by Thomas Bocek, Ela Hunt, and Burkhard Stiller from the University of Zurich, titled "Fast Similarity Search in Large Dictionaries"[1]. The work described there was expanded in a paper written by Daniel Karch, Dennis Luxen, and Peter Sanders from the Karlsruhe Institute of Technology, titled "Improved Fast Similarity Search in Dictionaries"[2]. Both of these papers deal with efficient searching for similar string values, given a…

Paper and discussion history. A mention is not an endorsement.

Improved Fast Similarity Search in Dictionaries — mentioned by lorenzhs

Fri, 21 Oct 2016 14:11:20 GMT

There is a variety of approximate string matching algorithms that speed up search by using an index. https://arxiv.org/abs/1008.1191 is one that should be fairly easy to implement. Levenshtein automata are another approach that makes the rounds on HN every now and then, but are a tough beast to implement and I wouldn't really recommend them in practice.

Paper and discussion history. A mention is not an endorsement.

Improved Fast Similarity Search in Dictionaries — mentioned by lorenzhs

Thu, 18 Jun 2015 20:35:36 GMT

It's already been answered why this isn't a good idea, but there is a working approach that works similarly: instead of all words within Levenshtein distance N, you store all words with exactly N characters deleted from them. So for each word in your dictionary, store all words that can appear when you delete N characters. That's a much more reasonable number, and there are tricks to make it manageable for long words. Then when you do queries, do the same with your query word and check if it's in the index. This is actually a very fast technique, it's described here in more detail…

Paper and discussion history. A mention is not an endorsement.

Improved Fast Similarity Search in Dictionaries — mentioned by lorenzhs

Wed, 10 Jun 2015 21:08:09 GMT

Levenshtein automata seem to pop up on here every once in a while. They are quite interesting from a theory perspective, but (like many things devised by the theory community) incredibly complex in practice. Lucene 4.0 uses them for fuzzy queries, you can read the full story of how they struggled to get them working somewhere in the Lucene blog. If you want to implement fuzzy string matching, I would look at something like http://arxiv.org/abs/1008.1191 . The experiments look impressively fast.

Paper and discussion history. A mention is not an endorsement.