HackerPapers

arXiv:2401.11817cs.CL

Hallucination is Inevitable: An Innate Limitation of Large Language Models

Ziwei Xu, Sanjay Jain, Mohan Kankanhalli

Abstract

Hallucination has been widely recognized to be a significant drawback for large language models (LLMs). There have been many works that attempt to reduce the extent of hallucination. These efforts have mostly been empirical so far, which cannot answer the fundamental question whether it can be completely eliminated. In this paper, we formalize the problem and show that it is impossible to eliminate hallucination in LLMs. Specifically, we define a formal world where hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function. By employing results from learning theory, we show that LLMs cannot learn all the computable functions and will therefore inevitably hallucinate if used as general problem solvers. Since the formal world is a part of the real world which is much more complicated, hallucinations are also inevitable for real world LLMs. Furthermore, for real world LLMs constrained by provable time complexity, we describe the hallucination-prone tasks and empirically validate our claims. Finally, using the formal world framework, we discuss the possible mechanisms and efficacies of existing hallucination mitigators as well as the practical implications on the safe deployment of LLMs.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 16 identifiable commenters in 24 comments across 22 threads, from Feb 2024 to Oct 2026. Found in 3 HN submissions from Feb 2024 to May 2026. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments22 threads

  1. “Math 2.0” will need to value mathematical progress more holistically 1 comment

    • I find hallucinations in my (mostly perfect) AI output every single day. If you're not finding them, you're just not looking hard enough. It's not surprising when everyone is screaming about how they don't read code these days. This is just a fact. I'm sorry if it messes with…

      jazzypantsView on HNParent context

  2. The Permission Slip 1 comment

    • > it can work through the complex, vague machinery of reason with more scaffolding No, it can hold more floating point numbers. I'm not an expert in the field, but I've yet to see a solid rebuttal to this paper; https://arxiv.org/abs/2401.11817

      decimalenoughView on HNParent context

  3. Agents need control flow, not more prompts 1 comment

    • >the latter is obviously subject to prompt engineering, hallucination, etc -- but so can a human pipeline! ...which is why we write deterministic code to take the human out of the pipeline. One of the early uses of computers was calculating firing tables for artillery, to…

      JCTheDenthogView on HNParent context

  4. Using “underdrawings” for accurate text and numbers 2 comments

    • >People keep throwing this phrase around in relation to LLMs, when not a single “fundamental limitation” has been rigorously demonstrated to exist Some limitations are not rigorously demonstrated to be fundamental, but continuously present from the first early LLMs yes.…

      coldteaView on HNParent context

    • As a general architecture, an LLM also has limitations that can't be improved unless we switch to another, fundamentally different AI design that's non LLM based. There are also limitations due to maths and/or physics that aren't fixable under any design. Outside science…

      coldteaView on HNParent context

  5. Which one is more important: more parameters or more computation? (2021) 1 comment

    • LLMs "hallucinate" because they are stochastic processes predicting the next word without any guarantees at being correct or truthful. It's literally an unavoidable fact unless we change the modelling approach. Which very few people are bothering to attempt right now. Training…

      2ndorderthoughtView on HNParent context

  6. Please don't say mean things about the AI I just invested a billion dollars in 1 comment

    • > True, but no more true than it is if you replace the antecedent with "people". Incorrect. People are capable of learning by observation, introspection, and reasoning. LLMs can only be trained by rote example. Hallucinations are, in fact, an unavoidable property of the…

      solid_fuelView on HNParent context

  7. MIT Study Finds AI Use Reprograms the Brain, Leading to Cognitive Decline 1 comment

    • Why is every discussion about AI like this? We get study after study and example after example showing that AI is unreliable [1], hurts productivity [2], and causes cognitive decline [3]. Yet, time and time again, a group of highly motivated individuals show up and dismiss these…

      soraminazukiView on HNParent context

  8. The Policy Puppetry Attack: Novel bypass for major LLMs 1 comment

    • > Is the layoff-based business model really the best use case for AI systems? No, but it plays well in quarterly earnings. So expect a good bit, I'd say. > The flaws are baked into the training data. Not in terms of attack area. In terms of hallucinations... ish. It's more baked…

      groby_bView on HNParent context

  9. How has DeepSeek improved the Transformer architecture? 1 comment

    • Hallucinations are a fundamental property of transformers, it can be minimized but never eliminated. https://www.mdpi.com/1999-4893/13/7/175 > the open-domain Frame Problem is equivalent to the Halting Problem and is therefore undecidable. Diaconescu's Theorem will help…

      nyrikkiView on HNParent context

  10. AI hallucinations: Why LLMs make things up (and how to fix it) 1 comment

    • > While the hallucination problem in LLMs is inevitable [0], they can be significantly reduced... Every article on hallucinations needs to start with this fact until we've hammered that into every "AI Engineer"'s head. Hallucinations are not a bug—they're not a different mode of…

      lolinderView on HNParent context

  11. Automated reasoning to remove LLM hallucinations 1 comment

    • Post title: Automated reasoning to remove LLM hallucinations --- and yet, the paper that went around in March: Paper Link: https://arxiv.org/pdf/2401.11817 Paper Title; Hallucination is Inevitable: An Innate Limitation of Large Language Models --- Instead of trying to trick a…

      spartanatreyuView on HNParent context

  12. DeepThought-8B: A small, capable reasoning model 1 comment

    • For LLM's no, but the explanation is wrong also, it has nothing to do with 'false memories' and has everything to do with how LLMs work. Here is the paper. https://arxiv.org/abs/2401.11817 RAG and fine tuning improve domain specificity and may reduce the problem to a level where…

      nyrikkiView on HNParent context

  13. LLMs know more than they show: On the intrinsic representation of hallucinations 1 comment

    • Much like you can search the internet until you find a source that agrees with you, you can select a set of papers that "confirm" a particular viewpoint, especially in developing fields of research. In this case, the selected papers all support the view LLMs "know what they…

      Retr0idView on HNParent context

  14. Show HN: Last CLI tool you will need 1 comment

    • On what basis you think you will avoid hallucination(s) ? LLM will always hallucinate. People did already some research, among which : https://arxiv.org/abs/2409.05746 and https://arxiv.org/abs/2401.11817 for example.

      az09mugenView on HNParent context

  15. Yann LeCun Thinks AI Is Dumber Than a Cat 1 comment

    • Something among the lines of : "No matter how many resources you put into an LLM, it will always hallucinate". And researchers proven that : https://arxiv.org/abs/2401.11817 LLMs are just statistical models and just "regurgitate" what they've been trained by. They have no will…

      az09mugenView on HNParent context

  16. Announcing the Tech for Palestine Incubator 1 comment

    • I'm not overthinking it, some papers describe it [0] and [1] for example. I agree for some subjects you can deal with some error. The problem is not how often but how bad just one single error can be. My point was on controversial topic, where a single error can deal serious…

      az09mugenView on HNParent context

  17. I no longer use AI assistant 1 comment

    • Since the first day I consider AI as a noob working very fast and always making errors. My main usage is for tedious and non-critical tasks such as boilerplates. And I always review what has been generated like a merge/pull request, because in a sense that's what it is. What…

      az09mugenView on HNParent context

  18. GPTs and Hallucination 1 comment

    • I can offer cite that hallucinations are an innate property of LLMs, can you provide one that shows they have 'an internal mental model'? https://arxiv.org/abs/2401.11817

      nyrikkiView on HNParent context

  19. Google scrambles to manually remove weird AI answers in search 2 comments

    • Can you cite a source for this, or are you speculating? My understanding was the opposite -- that the activity of a confabulating LLM is indistinguishable from one giving factually accurate responses. https://arxiv.org/abs/2401.11817

      duskwuffView on HNParent context

    • Some things like: https://arxiv.org/abs/2310.18168 https://arxiv.org/abs/2310.06824 There are various reasons an LLM might have incorrect "beliefs" - the input text was false, training doesn't try to preserve true beliefs, quantization certainly doesn't. So it can't be perfectly…

      astrangeView on HNParent context

  20. I stumbled upon LLM Kryptonite and no one wants to fix this model-breaking bug 1 comment

    • It is well known that it is impossible to eliminate hallucination in LLMs Here is a formal proof. https://arxiv.org/abs/2401.11817 Hopefully this article helps spread the word, but this is something that will have to be mitigated within acceptable limits. LLMs are NLP, not NLU.

      nyrikkiView on HNParent context

  21. Alright then keep your secrets 1 comment

    • Not the same thing, LLMs are stochastic parrots, and part of that is by design, so that they produce output that emulates speech. But hallucinations are not avoidable either. https://arxiv.org/abs/2401.11817 They are Internet simulators unless they can find some small…

      nyrikkiView on HNParent context

  22. Ask HN: Has Generative AI plateaued? 1 comment

    • Sure! Hallucination is Inevitable: An Innate Limitation of Large Language Models. FYI https://arxiv.org/abs/2401.11817

      R787View on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.