The Case for Learned Index Structures · HackerPapers discussions

The Case for Learned Index Structures — mentioned by yencabulator

Thu, 04 Jun 2026 15:02:01 GMT

The LLM is still a "normal algorithm", just one with a fairly large dataset to use. Assigning the algorithm part of LLMs magical properties hinders understanding. The work needed to pick the next output token is very much a classical algorithm. Algorithms can be based on training and/or use data just fine, too. https://arxiv.org/abs/1712.01208 (Now, the weights used, those we kinda really don't understand the same way we understand the processing, and the approach to looking for structures in weights sometimes looks more like archeology or anthropology than computer science.) It sounds like…

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by hamilyon2

Sat, 10 Jan 2026 20:38:26 GMT

When NPU will become available in most systems, we will see learned indexes like in https://arxiv.org/pdf/1712.01208v1 on client side. If file search indexes and such will be replaced by neural networks, this will be small but measurable improvement in battery life, speed and memory usage.

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by kevindamm

Wed, 25 Jun 2025 17:07:51 GMT

In theory, you can treat the objects as i.i.d. giving them a uniform prior. In practice, of course only the inputs are inputs and if you were to take that into account then you could avoid collisions, but as you say, you can't really know that ahead of time. Unless you can, such as when there is structure to the inputs or some other source of non-uniformity that you can account for. The topic of learned indexes is worth investigating if you're interested: https://arxiv.org/abs/1712.01208 https://arxiv.org/abs/2012.12501

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by robertsdionne

Fri, 22 Jul 2022 14:10:45 GMT

Also, Learned Indexes Structures: The Case for Learned Index Structures: https://arxiv.org/abs/1712.01208 Learned Indexes for a Google-scale Disk-based Database: https://arxiv.org/abs/2012.12501 And Noms/Dolt Prolly Trees: How to Chunk Your Database into a Merkle Tree: https://dolthub.com/blog/2022-06-27-prolly-chunker/

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by gopalv

Fri, 30 Jul 2021 19:03:35 GMT

> we'll see another explosion in practical CPU throughput, combined with a little help from algs and data structures The learned index[1] is basically a variation of this sort of "hint" which relies on getting the right answer most of the time, but from learning the distribution function as a model instead of relying on the programmer to heuristic it. (from that paper) > The key idea is that a model can learn the sort order or structure of lookup keys and use this signal to effectively predict the position or existence of records I haven't seen anything mix this sort of work with an Eytzinger…

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by emayljames

Mon, 25 Jan 2021 09:28:40 GMT

The Case for Learned Index Structures Tim Kraska, Alex Beutel, Ed H. Chi, Jeffrey Dean, Neoklis Polyzotis "Our initial results show, that by using neural nets we are able to outperform cache-optimized B-Trees by up to 70% in speed while saving an order-of-magnitude in memory over several real-world data sets. More importantly though, we believe that the idea of replacing core components of a data management system through learned models has far reaching implications for future systems designs and that this work just provides a glimpse of what might be possible."…

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by seg_lol

Fri, 23 Oct 2020 01:28:37 GMT

Could you have someone walk towards the camera with a focus test card and form a map between the sensor and the focus data from the lens itself (FFT, etc)? It would be something analogous to the 'learned indexes' paper. You could then provide any tweening function to give you that focus feeling you need. http://www.vldb.org/pvldb/vol13/p1162-ferragina.pdf https://arxiv.org/abs/1712.01208

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by malisper

Fri, 10 Jul 2020 07:23:42 GMT

> OTOH the standard library sort can be improved if you know the distribution of the numbers you're sorting (e.g., small numbers can bucket sort, small lists can use a sorting network, etc). If this thing can efficiently almost-correctly-sort because it's better at these types of pattern matching, we can just run a final pass of insertion sort to make it useful! I haven't read this paper, but that strategy was exactly the strategy behind "The Case for Learned Index Structures"[0]. The idea was you could train a model to predict the location of a row based on a primary key. After computing the…

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by ndr

Wed, 28 Aug 2019 20:45:02 GMT

> One big downside is that is has a margin of error which is very often unacceptable. You should look at the paper they mention The Case for Learned Index Structures[0]. It covers B-Tree-Index and how to use Hybrid Indexes to bound the errors so that "bad" answers from the model perform no worse than the standard data structure. For a lighter read this paper was also covered by Adrian Colyer in an issue of the morning paper[1]. [0] https://arxiv.org/abs/1712.01208 [1] https://blog.acolyer.org/2018/01/08/the-case-for-learned-ind... [edit: formatting]

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by KirinDave

Fri, 09 Nov 2018 17:25:04 GMT

This article represents an ongoing trend that we've seen over the last 3 years as a resurgence in optimization techniques for directly modeling tasks which historically were considered "not amenable to optimization techniques." We saw this last year as well with the landmark, "The Case for Learned Index Structures" [0] and in prior years with Coda Hale's "usl4j And You" [1], which can easily be woven in realtime into modern container control planes. We've also seen research in using search and strategy based optimization techniques [2] (such as Microsoft's Z3 SMT prover[3]) to validate…

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by KirinDave

Wed, 17 Oct 2018 14:56:56 GMT

Gosh, I have a few! Fistly, wavelet trees! Wavelet trees with something like RRR encoding for the bit vectors lets you work with massive datasets in positively tiny space and constant time. They're not even expensive to construct! My favorite introduction to the whole space of rank/select-friendly structures is Alex Bowe's tutorial: https://alexbowe.com/wavelet-trees/ I constantly agitate for people to realize that we now have an algorithm for O(n) generalized sorting. It's called discrimination sort, and while it's a complex subject its quite elegant once you internalize the algorithm.…

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by glenrivard

Fri, 12 Oct 2018 11:07:00 GMT

Interesting stuff. I thought one of the most interesting papers at NIPS this year was using GPU or TPU for intelligent indexes. https://arxiv.org/abs/1712.01208 Paper is from Jeff Dean and a worth while read, IMO.

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by ilija139

Tue, 29 May 2018 09:57:57 GMT

I wonder how it compares to Google's "Learned Index Structures" https://arxiv.org/abs/1712.01208

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by jacksmith21006

Wed, 16 May 2018 10:38:22 GMT

Most interesting thing I have seen for for database indexes is this paper. https://www.arxiv-vanity.com/papers/1712.01208v1/ Using a Neural Network in place of a B-tree. What is interesting is a NN can use many processors at the same time versus you can not with a B-tree. In the end it comes down the power as in joules to get something done.

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by yorwba

Tue, 15 May 2018 07:19:45 GMT

This seems like a fairly natural generalization of the idea in "The Case for Learned Index Structures" [1]: speed up some classical algorithm based on heuristics by modeling the data with a neural network that can be tuned to the specific task that the general algorithm is applied to. That said, the O(N) claim is purely sensational. It all depends on being able to determine the distribution parameters with sufficient accuracy in O(N) time, which isn't really guaranteed for all kinds of data you might want to sort. [1] https://arxiv.org/abs/1712.01208

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by jacksmith21006

Fri, 11 May 2018 18:41:38 GMT

All comes down to the cost. We can see that the TPUs were about 1/2 as much as using Nvidia before the TPU 3.0. This is based on doing the same task with the TPUs versus Nvidia with AWS. But it might be a bigger difference and Google taking margins. Could have been the other way but think WaveNet proves not that direction. Would hope we get an even bigger price difference with this generation. BTW, you are mixing up graphic generations and ML generations. Google never did or will graphics. For ML Google is 2 generations ahead. Hard to imagine Nvidia ever catching up as so much of the AI…

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by jacksmith21006

Fri, 04 May 2018 13:06:19 GMT

Google completely gave away K8s including even the brand. So not sure what you think Google will do? They are giving away everything they learned with Borg and you somehow take it as a negative? Really? Do think from a business standpoint not sure why Google would give up the K8s brand but it is good for everyone. I really worry that Google will stop opens sourcing and more important sharing so much through papers when people b*tch so much. Starting to be hard to see why they even do it any longer. Who else have a huge competitive advantage with something like Borg and just gives it away to…

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by jacksmith21006

Mon, 23 Apr 2018 21:31:45 GMT

One of favorite papers is the paper from Jeff Dean https://arxiv.org/abs/1712.01208 The Case for Learned Index Structures

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by jacksmith21006

Sat, 21 Apr 2018 17:10:23 GMT

The new Google Speech solution is the perfect example on why Google had to do their own silicon. Doing speech with 16k samples a second through a NN and keep at a reasonable cost is really, really difficult. The old way was far more power efficient and if you are going to use this new technique which gets you a far better result and do it at a reasonable cost you have to go all the way down into the silicon. Here listen to the results. https://cloudplatform.googleblog.com/2018/03/introducing-Clo... Now I am curious on the cost difference Google as able to achieve. It is still going to be more…

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by jacksmith21006

Tue, 20 Feb 2018 16:07:37 GMT

Would say Google has a couple gems. DeepMind comes to mind. Also Google Brain has done some pretty interesting things. Probably my favorite of late is from Jeff Dean https://arxiv.org/abs/1712.01208 We just got YouTube TV recently and also something that is pretty impressive by Google and saves me a ton of money as my cable provider was awful charging for every TV monthly. Another example of something I find really interesting is Spanner. https://research.google.com/archive/spanner.html

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by jacksmith21006

Tue, 13 Feb 2018 23:18:00 GMT

Thought one of the more interesting ideas is the Jeff Dean paper on using the TPU for more traditional CS operations like an index lookup. Like to see this progress and get more data on the difference is power required using a CPU and traditional algorithms versus using a TPU and new algorithms like Jeff outlined. https://arxiv.org/abs/1712.01208 The Case for Learned Index Structures

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by sulam

Sat, 30 Dec 2017 15:50:06 GMT

I highly recommend people give this paper a read. I think it points the way to a radical redesign of fundamental parts of the system stack over the next 5-10 years. If you work in systems and you aren’t thinking about this stuff, you’re about to be lapped. The Case for Learned Index Structures: https://arxiv.org/pdf/1712.01208v1.pdf

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by visarga

Sat, 30 Dec 2017 04:33:25 GMT

On the one hand, DeepZip does neural net based compression. On the other hand, in "The Case for Learned Index Structures" [1] paper, they replace B-Trees with neural nets: > by using neural nets we are able to outperform cache optimized B-Trees by up to 70% in speed while saving an order of magnitude in memory over several real world data sets Basically ML/DL is coming for the classic data structures, replacing some of them with learning based versions. [1] https://arxiv.org/abs/1712.01208

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — mentioned by stablemap

Tue, 26 Dec 2017 19:28:00 GMT

There is a paper for the index structure part; several authors. https://arxiv.org/abs/1712.01208

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — HN submission

Tue, 12 Dec 2017 13:33:39 GMT

The Case for Learned Index Structures

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Case for Learned Index Structures — mentioned by mfukar

Tue, 12 Dec 2017 12:52:42 GMT

Well, "big new" sounds like a destructive qualification. Why can't it be "big" and "old"? https://arxiv.org/pdf/1712.01208.pdf

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — HN submission

Mon, 11 Dec 2017 13:34:37 GMT

The Case for Learned Index Structures

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Case for Learned Index Structures — mentioned by pg314

Mon, 11 Dec 2017 12:44:01 GMT

Under some assumptions it does handle inserts. From [1]: Finally, assume that the inserts follow roughly a similar pattern as the learned CDF; [...] Under these assumptions the model might not need to be retrained at all. [1] https://www.arxiv-vanity.com/papers/1712.01208v1/

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — HN submission

Mon, 11 Dec 2017 11:05:59 GMT

The Case for Learned Index Structures

18 points; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Case for Learned Index Structures — mentioned by larelli

Mon, 11 Dec 2017 07:38:17 GMT

It looks like this paper has more information: https://arxiv.org/pdf/1712.01208v1.pdf

Paper and discussion history. A mention is not an endorsement.

The Case for Learned Index Structures — HN submission

Mon, 11 Dec 2017 03:14:01 GMT

The Case for Learned Index Structures

398 points; 66 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Case for Learned Index Structures — HN submission

Mon, 11 Dec 2017 01:58:24 GMT

The Case for Learned Index Structures

41 points; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Case for Learned Index Structures — HN submission

Sun, 10 Dec 2017 17:44:09 GMT

“The Case for Learned Index Structures”: Data Storage and Retrieval as an ML Task

6 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Case for Learned Index Structures — HN submission

Sun, 10 Dec 2017 06:55:33 GMT

The Case for Learned Index Structures

6 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Case for Learned Index Structures — HN submission

Sat, 09 Dec 2017 19:38:11 GMT

The Case for Learned Index Structures

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Case for Learned Index Structures — HN submission

Fri, 08 Dec 2017 23:43:41 GMT

The Case for Learned Index Structures

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.