LLM in a flash: Efficient Large Language Model Inference with Limited Memory · HackerPapers discussions

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — mentioned by eddieplan9

Tue, 25 Aug 2026 15:12:26 GMT

Apple’s “LLM in a flash” paper [1] sheds some light on why this can be a big deal, especially if they are working on codesign of model and hardware in this dimension. [1] https://arxiv.org/abs/2312.11514

Paper and discussion history. A mention is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — mentioned by nickpsecurity

Fri, 01 May 2026 03:23:04 GMT

You're thinking in a provably-useful direction: https://arxiv.org/pdf/2312.11514

Paper and discussion history. A mention is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — mentioned by firstbabylonian

Mon, 23 Mar 2026 15:01:12 GMT

> SSD streaming to GPU Is this solution based on what Apple describes in their 2023 paper 'LLM in a flash' [1]? 1: https://arxiv.org/abs/2312.11514

Paper and discussion history. A mention is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — mentioned by simonw

Tue, 17 Mar 2026 17:18:57 GMT

Yeah the details on this look pretty thin. Best I could see was this snippet from the screenshot: > Key technique: selective expert streaming via direct I/0. Only ~10 of 512 experts per layer are loaded from SSD per token (~1.8GB I/0 per token at 1.4 GB/s effective bandwidth). Non-expert weights (~5GB) are pinned in DRAM. LRU expert cache provides 44%+ hit rate. It's apparently using ideas from: https://arxiv.org/abs/2312.11514 > This paper tackles the challenge of efficiently running LLMs that exceed the available DRAM capacity by storing the model parameters in flash memory, but bringing…

Paper and discussion history. A mention is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — mentioned by sharma-arjun

Sat, 11 May 2024 16:52:15 GMT

This indeed seems to have been a heavy focus of their research team in the past year, eg. "Efficient Large Language Model Inference with Limited Memory" [1] and OpenELM [2] [1] https://arxiv.org/pdf/2312.11514 [2] https://arxiv.org/pdf/2404.14619 (with 1.1B parameters, this appears to be their attempt at building a lightweight LLM)

Paper and discussion history. A mention is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — HN submission

Tue, 16 Apr 2024 13:19:31 GMT

LLM in a Flash: Efficient Large Language Model Inference with Limited Memory

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — mentioned by quentinp

Fri, 29 Mar 2024 17:28:29 GMT

Actually they plan to put an LLM in your back pocket using flash memory, not silicon: https://arxiv.org/abs/2312.11514

Paper and discussion history. A mention is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — HN submission

Wed, 28 Feb 2024 18:55:21 GMT

Ask HN: How to execute an 180B+ LLM on a Turing machine?

Or, equivalently, how would you approach running an 180B+ LLM on a Von Neumann computer with say 1 MB of main memory and virtually limitless secondary storage? Or, do you know about an approach (that you might read somewhere) that might help running a heavy LLM on virtually any Turing equivalent device? Picture this: you're stuck with your “potato computer” (small RAM, no external GPU, very large SSD), and your LLM is saved on an external SSD. Your task: run that LLM on your “potato PC” and try to achieve reasonable response times (e.g., 1 h to 24 h). Response times of 1 year, or higher…

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — HN submission

Wed, 24 Jan 2024 17:58:28 GMT

LLM in a Flash: Efficient Large Language Model Inference with Limited Memory

18 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — HN submission

Sun, 07 Jan 2024 21:51:10 GMT

LLM in a Flash: Efficient Large Language Model Inference with Limited Memory

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — HN submission

Tue, 26 Dec 2023 23:47:46 GMT

LLM in a Flash: Efficient Large Language Model Inference with Limited Memory

3 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — HN submission

Sun, 24 Dec 2023 07:39:25 GMT

LLM in a Flash

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — mentioned by amitprasad

Sat, 23 Dec 2023 21:53:41 GMT

Also relevant: LLM in a flash: Efficient Large Language Model Inference with Limited Memory Apple seems to be gearing up for significant advances in on-device inference using this LLMs https://arxiv.org/abs/2312.11514

Paper and discussion history. A mention is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — mentioned by jurmous

Sat, 23 Dec 2023 17:25:47 GMT

Released 2 days ago by Apple, a research paper on methods to run larger llms on iPhones. https://www.macrumors.com/2023/12/21/apple-ai-researchers-ru... https://arxiv.org/pdf/2312.11514.pdf

Paper and discussion history. A mention is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — mentioned by vittore

Fri, 22 Dec 2023 08:01:39 GMT

Can't wait when emu2 incorporates techniques from apple paper https://arxiv.org/pdf/2312.11514.pdf

Paper and discussion history. A mention is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — HN submission

Thu, 21 Dec 2023 22:31:08 GMT

LLM in a Flash: Efficient Large Language Model Inference with Limited Memory

12 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — mentioned by keep_reading

Thu, 21 Dec 2023 22:28:25 GMT

Apple has released a paper about running LLMs efficiently with low memory https://arxiv.org/pdf/2312.11514.pdf

Paper and discussion history. A mention is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — HN submission

Thu, 21 Dec 2023 14:24:34 GMT

LLM in a Flash: Efficient Large Language Model Inference with Limited Memory

5 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — HN submission

Thu, 21 Dec 2023 04:49:18 GMT

LLM in a Flash: Efficient Large Language Model Inference with Limited Memory

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — HN submission

Wed, 20 Dec 2023 16:32:23 GMT

Efficient Large Language Model Inference with Limited Memory

50 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — HN submission

Wed, 20 Dec 2023 09:26:55 GMT

LLM in a Flash: Efficient Large Language Model Inference with Limited Memory

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — mentioned by dwd

Wed, 20 Dec 2023 05:38:35 GMT

Apple was running a model double the size of the available memory. Not sure if that was a sweet spot they found or if you could sacrifice response time to run even bigger models. The paper is worth a read in full as what they are doing is pretty cool: https://arxiv.org/pdf/2312.11514 Highlight from the paper... "Then, we introduce two complementary techniques to minimize data transfer and maximize flash memory throughput: • Windowing: We load parameters for only the past few tokens, reusing activations from recently computed tokens. This sliding window approach reduces the number of IO…

Paper and discussion history. A mention is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — mentioned by phs318u

Wed, 20 Dec 2023 03:37:54 GMT

The actual paper: https://huggingface.co/papers/2312.11514 Abstract: "Large language models (LLMs) are central to modern natural language processing, delivering exceptional performance in various tasks. However, their intensive computational and memory requirements present challenges, especially for devices with limited DRAM capacity. This paper tackles the challenge of efficiently running LLMs that exceed the available DRAM capacity by storing the model parameters on flash memory but bringing them on demand to DRAM. Our method involves constructing an inference cost model that harmonizes…

Paper and discussion history. A mention is not an endorsement.

LLM in a flash: Efficient Large Language Model Inference with Limited Memory — HN submission

Wed, 20 Dec 2023 03:02:43 GMT

LLM in a Flash: Efficient LLM Inference with Limited Memory

252 points; 53 discussion replies on HN. Paper and source history. A submission is not an endorsement.