arXiv:2312.11514cs.CL2023-12-12

LLM in a flash: Efficient Large Language Model Inference with Limited Memory

Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, Mehrdad Farajtabar

Abstract

Large language models (LLMs) are central to modern natural language processing, delivering exceptional performance in various tasks. However, their substantial computational and memory requirements present challenges, especially for devices with limited DRAM capacity. This paper tackles the challenge of efficiently running LLMs that exceed the available DRAM capacity by storing the model parameters in flash memory, but bringing them on demand to DRAM. Our method involves constructing an inference cost model that takes into account the characteristics of flash memory, guiding us to optimize in two critical areas: reducing the volume of data transferred from flash and reading data in larger, more contiguous chunks. Within this hardware-informed framework, we introduce two principal techniques. First, "windowing" strategically reduces data transfer by reusing previously activated neurons, and second, "row-column bundling", tailored to the sequential data access strengths of flash memory, increases the size of data chunks read from flash memory. These methods collectively enable running models up to twice the size of the available DRAM, with a 4-5x and 20-25x increase in inference speed compared to naive loading approaches in CPU and GPU, respectively. Our integration of sparsity awareness, context-adaptive loading, and a hardware-oriented design paves the way for effective inference of LLMs on devices with limited memory.

arXiv comments: ACL 2024
Journal reference
DOI
Subjects: cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 12 identifiable commenters in 12 comments across 11 threads, from Dec 2023 to Aug 2026. Found in 12 HN submissions from Dec 2023 to Apr 2024. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments11 threads

New Mac Studio with M5 Max and M5 Ultra Aug 20261 comment
- Apple’s “LLM in a flash” paper [1] sheds some light on why this can be a big deal, especially if they are working on codesign of model and hardware in this dimension. [1] https://arxiv.org/abs/2312.11514
  
  eddieplan9Aug 25, 2026View on HN Parent context
Granite 4.1: IBM's 8B Model Matching 32B MoE May 20261 comment
- You're thinking in a provably-useful direction: https://arxiv.org/pdf/2312.11514
  
  nickpsecurityMay 1, 2026View on HN Parent context
iPhone 17 Pro Demonstrated Running a 400B LLM Mar 20261 comment
- > SSD streaming to GPU Is this solution based on what Apple describes in their 2023 paper 'LLM in a flash' [1]? 1: https://arxiv.org/abs/2312.11514
  
  firstbabylonianMar 23, 2026View on HN Parent context
Qwen3.5-397B at 4.74 tok/s using 5.9GB RAM Mar 20261 comment
- Yeah the details on this look pretty thin. Best I could see was this snippet from the screenshot: > Key technique: selective expert streaming via direct I/0. Only ~10 of 512 experts per layer are loaded from SSD per token (~1.8GB I/0 per token at 1.4 GB/s effective bandwidth).…
  
  simonwMar 17, 2026View on HN Parent context
Apple Closes in on Deal with OpenAI to Put ChatGPT on iPhone May 20241 comment
- This indeed seems to have been a heavy focus of their research team in the past year, eg. "Efficient Large Language Model Inference with Limited Memory" [1] and OpenELM [2] [1] https://arxiv.org/pdf/2312.11514 [2] https://arxiv.org/pdf/2404.14619 (with 1.1B parameters, this…
  
  sharma-arjunMay 11, 2024View on HN Parent context
Towards 1-bit Machine Learning Models Mar 20241 comment
- Actually they plan to put an LLM in your back pocket using flash memory, not silicon: https://arxiv.org/abs/2312.11514
  
  quentinpMar 29, 2024View on HN Parent context
Ferret: A Multimodal Large Language Model Dec 20232 comments
- Released 2 days ago by Apple, a research paper on methods to run larger llms on iPhones. https://www.macrumors.com/2023/12/21/apple-ai-researchers-ru... https://arxiv.org/pdf/2312.11514.pdf
  
  jurmousDec 23, 2023View on HN Parent context
- Also relevant: LLM in a flash: Efficient Large Language Model Inference with Limited Memory Apple seems to be gearing up for significant advances in on-device inference using this LLMs https://arxiv.org/abs/2312.11514
  
  amitprasadDec 23, 2023View on HN Parent context
Show HN: Emu2 – A Gemini-like open-source 37B Multimodal Model Dec 20231 comment
- Can't wait when emu2 incorporates techniques from apple paper https://arxiv.org/pdf/2312.11514.pdf
  
  vittoreDec 22, 2023View on HN Parent context
Apple wants AI to run directly on its hardware instead of in the cloud Dec 20231 comment
- Apple has released a paper about running LLMs efficiently with low memory https://arxiv.org/pdf/2312.11514.pdf
  
  keep_readingDec 21, 2023View on HN Parent context
LLM in a Flash: Efficient LLM Inference with Limited Memory Dec 20231 comment
- Apple was running a model double the size of the available memory. Not sure if that was a sweet spot they found or if you could sacrifice response time to run even bigger models. The paper is worth a read in full as what they are doing is pretty cool…
  
  dwdDec 20, 2023View on HN Parent context
Apple Announces LLM in a Flash: Efficient LLM Inference with Limited Memory Dec 20231 comment
- The actual paper: https://huggingface.co/papers/2312.11514 Abstract: "Large language models (LLMs) are central to modern natural language processing, delivering exceptional performance in various tasks. However, their intensive computational and memory requirements present…
  
  phs318uDec 20, 2023View on HN Parent context

Submitted to Hacker News

LLM in a Flash: Efficient LLM Inference with Limited Memory252 points53 commentsDec 2023
Efficient Large Language Model Inference with Limited Memory50 points1 commentDec 2023
LLM in a Flash: Efficient Large Language Model Inference with Limited Memory18 points1 commentJan 2024
LLM in a Flash: Efficient Large Language Model Inference with Limited Memory12 points1 commentDec 2023
LLM in a Flash: Efficient Large Language Model Inference with Limited Memory5 points0 commentsDec 2023
LLM in a Flash: Efficient Large Language Model Inference with Limited Memory4 points0 commentsDec 2023
LLM in a Flash: Efficient Large Language Model Inference with Limited Memory3 points0 commentsJan 2024
LLM in a Flash: Efficient Large Language Model Inference with Limited Memory3 points1 commentDec 2023
LLM in a Flash: Efficient Large Language Model Inference with Limited Memory2 points0 commentsApr 2024
LLM in a Flash2 points0 commentsDec 2023
LLM in a Flash: Efficient Large Language Model Inference with Limited Memory2 points0 commentsDec 2023
Ask HN: How to execute an 180B+ LLM on a Turing machine?1 point0 commentsFeb 2024
Or, equivalently, how would you approach running an 180B+ LLM on a Von Neumann computer with say 1 MB of main memory and virtually limitless secondary storage? Or, do you know about an approach (that you might read somewhere) that might help running a heavy LLM on virtually any Turing equivalent device? Picture this: you're stuck with your “potato computer” (small RAM, no external GPU, very large SSD), and your LLM is saved on an external SSD. Your task: run that LLM on your “potato PC” and try…

Connected through shared HN threads, not inferred similarity or a recommendation.

Abstract

On Hacker News

Discussed in comments11 threads

Submitted to Hacker News

Also discussed alongside