AI Hallucination: Definition, Manifestations, Technical Causes, and Impact

Introduction to AI Hallucination

In the rapidly evolving landscape of artificial intelligence, particularly with the advent of generative AI and Large Language Models (LLMs), a significant challenge known as "hallucination" has emerged. AI hallucination describes a phenomenon where generative AI models produce outputs that are factually incorrect, nonsensical, or ungrounded in reality, despite often appearing fluent, coherent, and confidently presented . While the term is metaphorical, it refers to erroneously constructed responses rather than human-like delusions, signifying outputs that are "entirely fabricated" or "ungrounded" . The Cambridge Dictionary updated its definition in 2023 to include this AI-specific sense, reflecting its growing prominence 1.

It is crucial to differentiate AI hallucination from other forms of AI inaccuracies. Unlike simple factual errors that might stem from outdated information, or algorithmic bias which reflects skewed or unfair outcomes due to unrepresentative data, hallucination specifically implies the generation of invented or ungrounded information . While bias in training data can contribute to hallucinations by leading models to generate biased patterns, hallucination itself is the manifestation of inventing incorrect details rather than merely retrieving existing skewed data . Similarly, while hallucinated content constitutes "distorted information," it represents a specific mechanism by which AI systems generate such inaccuracies, potentially encompassing both unintentional falsehoods (misinformation) and seemingly intentional ones (disinformation), albeit without conscious AI intent 2. This phenomenon is further categorized by its origin, distinguishing between "prompting-induced hallucinations" that arise from vague user queries and "model-internal hallucinations" resulting from the model's architecture or training 3.

The core characteristics of AI hallucination include factual incorrectness, where outputs contain false information, combined with a striking fluency and coherence that often makes the generated content appear logical and plausible . This "confabulation" means the AI fabricates information but presents it as factual, often inventing plausible-sounding but ungrounded details . Such outputs can range from providing incorrect biographical details for celebrities to fabricating legal cases, academic citations, or even medical advice . Hallucinations can be intrinsic, directly contradicting the provided input, or extrinsic, generating information not present in the source but still ungrounded .

The prevalence and impact of AI hallucination establish it as a critical challenge in modern AI development. It significantly undermines user trust and the reliability of AI systems, particularly in high-stakes applications such as medicine, law, and critical infrastructure, where accuracy is paramount . Moreover, AI hallucination exacerbates the spread of misinformation and disinformation, blurring the lines between verified and synthetic information and posing risks to network security, online fraud, and even democratic processes . The term's evolution from early computer vision applications to its current widespread use in the context of LLMs post-2021 underscores its increasing relevance and the urgent need for robust mitigation strategies to ensure the responsible and effective deployment of generative AI technologies 1.

Categorization and Manifestations of AI Hallucination

Building upon the foundational understanding of AI hallucination as outputs that are factually incorrect, nonsensical, fabricated, or misleading, despite being presented with confidence and plausibility , this section delves into its varied categorizations and observable manifestations across different generative AI modalities. Some perspectives consider this phenomenon an inherent feature of computable Large Language Models (LLMs), rather than merely a bug .

Taxonomy of AI Hallucinations

Researchers have developed diverse categorizations to describe the multifaceted nature of AI hallucinations, classifying them by core dimensions, general error types, and context-specific or advanced forms .

Core Dimensions

General Error Types

Context-Specific or Advanced Hallucinations

Manifestations and Examples Across Modalities

Hallucinations are not limited to a single AI type but manifest distinctively across various generative modalities.

Text-Based Generative AI (LLMs)

In text-based LLMs, hallucinations frequently involve the generation of plausible-sounding but false statements, erroneous citations, or fabricated narratives:

• Factual Errors: These include incorrectly claiming a world record for walking across the English Channel 7, misstating the year of an event, or providing an inapplicable location 8. LLMs can also incorrectly identify a prime number or perform multi-step mathematical calculations with errors, even while offering a logical-sounding explanation . A notable instance involved Google's Bard incorrectly stating that the James Webb Space Telescope captured the first images of a planet outside our solar system .
• Fabricated Content: This form of hallucination generates entirely fictitious information. Examples include generating phantom footnotes and bogus references in government reports 9, fabricating legal case citations in court filings leading to sanctions for lawyers , inventing non-existent books and authors for a summer reading list 9, or creating terms like "hyperactivated antibiotics" in medical records 8. An LLM might confidently answer questions with entirely false information, such as the Golden Gate Bridge being transported across Egypt 7, or combine two individually correct facts into an incorrect synthesis, such as identifying a US senator from Minnesota but falsely claiming they attended Princeton University 10.
• Contextual/Logical Inconsistencies: This can be seen when a chatbot provides incorrect information, such as Air Canada's chatbot giving inaccurate details about bereavement fares, which led to legal action . Models also produce grammatically fluent but meaningless text when responding to prompts containing contradictory information 10.
• Sycophancy: Models may align with a user's input, even if it contains inaccuracies, prioritizing user satisfaction over factual accuracy 6.
• Transcription Errors: OpenAI's Whisper speech-to-text model has been observed to invent false content, including attributing race, violent rhetoric, or nonexistent medical treatments in transcriptions 9.
• Misinterpreted Satirical Content: Google's "AI Overview" suggested adding non-toxic glue to pizza, a recommendation seemingly derived from misinterpreted satirical or forum content 9.

Other Generative AI Modalities

Hallucinations extend beyond text, affecting image, code, and multimodal generation:

• Image Generation:
  • Unexpected Bias: Models may disproportionately depict certain races for low-paid jobs or misrepresent historical figures' racial identities, reflecting biases present in training data 11.
  • Proportional Puzzlements: Challenges with maintaining correct proportions, such as generating unnaturally elongated bodies, often arise from training on specific image resolutions (e.g., square) and then generating in different formats (e.g., rectangular) 11.
  • Duplicates: Generating excessive numbers of repeated objects in an image is typical when producing higher-resolution outputs than the training data 11.
  • Resemblance to Copyrighted Content: Creating images of famous characters or individuals (e.g., Barbie as Margot Robbie, Agent 007 as Daniel Craig, Joaquin Phoenix's Joker) that bear striking and potentially infringing resemblances to copyrighted material or specific likenesses present in the training data 11.
  • Misidentification: AI systems trained on specific objects may incorrectly "see" and identify them in images where they are absent, like detecting pandas in pictures of bicycles or giraffes 7.
• Code Generation: Models can generate programming errors or incorrect code 2.
• Multimodal Outputs: These involve hallucinations that span or create inconsistencies across different modalities, such as text accompanying an image or generated video content .

Common Terminology Used to Describe Hallucinations

Researchers and practitioners use several terms interchangeably or to specify nuances of AI hallucinations:

Underlying Causes of AI Hallucinations

The root causes of AI hallucinations are diverse and often interconnected, ranging from issues in training data to model architecture and prompting:

• Training Data Issues: These include insufficient, biased, outdated, or noisy data, which can lead the model to fill information gaps with fabricated content . Data voids, where relevant training material is scarce, compel models to extrapolate from unrelated content 8.
• Model Architecture and Design: LLMs generate text by predicting the next word based on statistical patterns rather than verifying facts, which can result in fluent but false responses . Neural networks often lack an intrinsic ability to verify facts or cross-reference real-world knowledge . Faulty architecture or overconfidence can further contribute to overgeneralization or struggles with ambiguity .
• Training Process Issues: Overfitting occurs when a model memorizes training data too thoroughly, failing to generalize to new data . Conversely, underfitting means the model cannot capture patterns effectively, leading to poor performance 6.
• Generation Methods: Techniques like beam search can prioritize fluency at the expense of accuracy, while sampling methods introduce randomness that may result in nonsensical or fabricated content 10.
• Prompting Issues: Ambiguous or adversarial prompts can force models to "guess" plausible answers or generate confident but unfounded responses . Models may also exhibit sycophancy, aligning with user input even if incorrect, or feel pressure to provide an answer rather than admit uncertainty .

Underlying Technical Causes and Mechanisms of AI Hallucination

AI hallucinations, defined as the generation of plausible but factually incorrect, logically inconsistent, or entirely fabricated information 3, are an inherent byproduct of language modeling that prioritizes syntactic and semantic plausibility over factual accuracy 12. This phenomenon stems primarily from the statistical nature of Large Language Models (LLMs), which predict token sequences without true comprehension 12. Understanding why AI hallucinates requires a detailed analysis of the underlying technical causes, encompassing limitations in training data, model architecture, inference processes, and prompt sensitivity 12.

Training Data Characteristics

The quality, quantity, and diversity of training data significantly influence the rate at which LLMs hallucinate 12.

• Inaccuracies and Biases: LLMs are trained on vast datasets that often contain inaccuracies, biases, or misinformation, which the model can inadvertently learn and perpetuate 12. If internet data used for training includes misinformation, LLMs incorporate and confidently reproduce these patterns 13.
• Incompleteness and Noise: Incomplete or noisy datasets can cause models to learn incorrect associations and extrapolate inaccuracies 12. Insufficient training data in specialized domains leads to knowledge gaps that models may fill with fabricated information 13.
• Data Voids and Long-Tail Knowledge: LLMs are particularly susceptible to hallucinating when processing long-tail knowledge that appears infrequently in the training data, as they rely on patterns of frequently appearing words and phrases 14.
• Outdated Information: The static nature of most training corpora means models lack knowledge of recent developments post-training 12. When confronted with questions outside their training timeframe, LLMs often generate fabricated facts or responses that were once accurate but are now outdated 14.
• Knowledge Conflict: Training on disparate sources that provide contradictory information can result in outputs reflecting conflicting viewpoints or factual errors 14. This also includes "imitative falsehoods," where models reproduce false information embedded in their training data 14.

Model Architectural Limitations and Design Choices

The inherent design of LLMs contributes to their propensity for hallucination.

• Transformer Structure and Lack of Grounding: The complex, autoregressive nature of transformer-based models generates text token-by-token based on learned probability distributions, prioritizing fluency over factual verification 12. These models operate by manipulating symbols and do not possess a true understanding of the world, which leads to outputs disconnected from facts and often involves "filling in gaps" with plausible-sounding but incorrect information 12. Furthermore, current transformer architectures struggle with contextual dependencies beyond certain token limits, leading to coherence breakdowns 12.
• Attention Mechanisms: While attention mechanisms enable models to focus on input segments, soft attention in long sequences can result in hallucination 14. This occurs as attention weights become diffuse, distributing focus among less relevant tokens, thereby degrading reasoning or causing factual inaccuracies 14. Limited context windows in transformers can also cause models to lose track of important information established earlier in a conversation 13.
• Objective Function and Positional Encoding: The maximum likelihood estimation (MLE) objective function, commonly used during training, encourages generating the most probable token but does not explicitly penalize factual inconsistencies 14. Positional encoding, which is critical for maintaining token order, can deteriorate with longer input sequences, causing models to misinterpret contextual relationships 14.
• Unidirectional Contextualization: Autoregressive LLMs, such as GPT models, process text unidirectionally from left-to-right 14. This limits their ability to fully integrate contextual information from subsequent tokens, causing them to infer or fabricate content to maintain coherence when faced with ambiguous or incomplete input 14.

Inference Processes and Decoding Strategies

The manner in which an LLM generates text during inference significantly influences hallucination rates 15. LLMs generate text iteratively, selecting one token at a time based on probability distributions 15.

• Decoding Methods:

• Temperature Settings: Temperature controls the "sharpness" of the probability distribution, balancing creativity and predictability 16.
  • Low Temperature (T < 1.0): Makes the output more deterministic, concentrating probability mass around the most likely tokens, resulting in conservative, focused, and less diverse text 16.
  • High Temperature (T > 1.0): Spreads out the probability mass, making the output more random, leading to creative, diverse, and novel text 16.
  • Creativity-Hallucination Trade-off: While higher temperatures foster creativity, they also increase the likelihood of hallucinations 15. Anecdotal reports suggest using low temperatures for factual tasks; however, empirical research indicates that changes in temperature from 0.0 to 1.0 do not significantly impact LLM problem-solving performance, suggesting low temperatures do not universally prevent hallucinations 17. In fact, setting temperature to 0 can sometimes increase hallucination by removing the model's flexibility to avoid high-probability but low-relevance phrases 18.
• Logits Bias: This allows for manual adjustment of token probabilities before the softmax function, enabling specific tokens to be favored or disfavored in the output 16.

Prompt Sensitivity

Prompt design and sensitivity significantly influence the occurrence of hallucinations 3.

• Prompt-Induced Hallucinations: These arise when prompts are vague, underspecified, or structurally misleading, prompting the model into speculative generation 3. Unclear intent, especially in zero-shot prompts, can result in off-topic or imaginative content 3.
• Prompt Sensitivity (PS): This metric quantifies how much a model's hallucination rate varies with different prompt styles for a fixed model 3. A high PS indicates that hallucinations are largely prompt-induced 3.
• Impact of Prompt Design: Variations in prompt structure, such as zero-shot, few-shot, instruction-based, or Chain-of-Thought (CoT), can either induce or suppress hallucinations 3. Structured strategies like CoT prompting, which encourage step-wise reasoning, have been shown to significantly reduce hallucinations in prompt-sensitive scenarios 3.

Specific Neural Network Behaviors Explaining Erroneous Outputs

Several fundamental neural network behaviors contribute to ungrounded outputs and highlight the nature of knowledge representation in LLMs.

• Statistical Patterns vs. True Understanding: LLMs function as sophisticated pattern-matching engines that learn statistical correlations within vast datasets, rather than achieving human-like comprehension or reasoning 18. They generate text based on learned patterns and probabilities, often losing the exactness of original data in favor of generalizations 19.
• Overfitting and Generalization Problems: Models can memorize patterns instead of gaining conceptual understanding, leading them to confidently reproduce training examples incorrectly when adapting to new contexts 13. When tasks require reasoning beyond their training distribution, models may generate responses adhering to linguistic patterns but lacking essential logical constraints 13. This difficulty in extrapolating causes fabrication when knowledge is not explicitly present in the training data, prioritizing conversational coherence over factual accuracy 13.
• Shortcut Learning and Exposure Bias: Models may engage in "shortcut learning" by relying on superficial statistical patterns rather than robust features, which leads to poor generalization in out-of-distribution contexts 14. The "teacher forcing" learning strategy, where models predict the next word based on a flawless context during training, creates an "exposure bias" during inference 14. If an early generated token is incorrect, it can lead to a "snowball effect" of subsequent errors due to the lack of corrective feedback 14. Furthermore, a lack of sufficient negative examples during training weakens a model's ability to distinguish fact from fiction 14.
• Model Overconfidence Despite Knowledge (CHOKE): A critical behavior is "Certain Hallucinations Overriding Known Evidence" (CHOKE) 20. This phenomenon describes situations where a model, despite consistently providing a correct answer, produces a highly confident, hallucinated response due to a minor prompt perturbation, even when it possesses the correct underlying knowledge 20. This distinct type of hallucination poses significant risks, particularly in high-stakes domains, as it challenges the assumption that low certainty always correlates with hallucination; models can be highly confident in incorrect information 20. Instruct-tuned models, despite their enhancements, can also exhibit poorer calibration between certainty and hallucinations 20.

Impact, Risks, and Ethical Implications of AI Hallucination

AI hallucination, characterized by the generation of false, misleading, or fabricated information with high confidence and fluency, poses substantial challenges across critical domains 21. Unlike human-generated deception, these ungrounded outputs stem from the statistical nature of large language models (LLMs) predicting word sequences rather than verifying truth 22. This section details the practical consequences, risks, and ethical dilemmas introduced by AI hallucinations, incorporating specific examples and case studies.

Practical Consequences and Risks Across Critical Domains

AI hallucinations present significant risks across various sectors, often leading to severe implications for decision-making, safety, and operational integrity.

Healthcare

In healthcare, AI hallucinations can critically endanger patient safety and lead to diagnostic inaccuracies 21. Medical AI systems might generate false drug interactions or incorrect treatment recommendations 21. For instance, an AI model could incorrectly identify a benign skin lesion as malignant, resulting in unnecessary medical interventions 23. OpenAI's Whisper system has been observed to fabricate misleading content in medical conversation transcriptions 22, and AI models can provide incomplete, incorrect, and potentially harmful information about common ophthalmic diseases 2. Hallucination rates in healthcare applications are reported at 9.2% 21.

Legal

The legal sector faces risks from inaccurate advice, potential damage to reputation, and significant legal liability due to AI hallucinations 21. Hallucinated citations have appeared in legal filings 22. A notable case involved Air Canada's chatbot, which misled a customer about bereavement fares, leading to legal consequences for the airline 22. Analysis of 500 legal documents revealed hallucination-induced errors in 12% of contract clauses 21. Legal AI applications have reported hallucination rates as high as 16.7% 21.

Finance

The financial sector experiences unique and high-stakes risks from AI hallucinations due to data sensitivity, regulatory requirements, and market volatility 21.

• Misinformed Decision-Making: Incorrect data from risk models or advisory chatbots can lead to flawed assessments and poor strategic choices 24. Examples include investment algorithms incorrectly rebalancing portfolios based on hallucinated news 24. Forecasting errors can have a 27% hallucination rate beyond two quarters, and 18% of AI-generated Value-at-Risk calculations contain unsupported assumptions 21.
• Regulatory and Compliance Violations: Unintentionally reporting false information can result in compliance breaches and legal penalties 24. AI hallucinations can produce outputs that misstate filings or invent non-existent compliance steps, such as citing a "IFRS 99 standard" when no such standard exists 24. SEC filings have a 14% hallucination rate, Anti-Money Laundering reports 22%, and Basel III Compliance documents 19% 21.
• Financial Losses: Acting on fabricated insights, such as fake forecasts or incorrect valuations, can directly cause significant financial losses for clients and institutions 24.
• Vulnerabilities: High-frequency trading data increases hallucination likelihood by 31%, and cryptocurrency market analyses show 2.4 times more hallucinations than traditional assets 21. Data silos and reporting latency also contribute to errors 21. Major banks like JPMorgan Chase, Wells Fargo, and Goldman Sachs banned internal use of ChatGPT-style tools due to fears of transmitting proprietary client data to external servers 24.

Education and Academia

AI tools used as educational tutors can spread misinformation among students 21. In academia, editing tools have introduced systematic terminology errors 22, and fabrications and errors, including hallucinated citations, have been found in bibliographic references generated by ChatGPT and in legal filings 2.

News Generation and the Information Ecosystem

Hallucinating news bots can spread falsehoods that undermine mitigation efforts, especially regarding developing emergencies 23. Media outlets have published AI-generated content containing historical inaccuracies 22, contributing to the proliferation of AI-generated misinformation and posing societal risks 21. As AI integrates into public knowledge formation processes like online search and journalism, its outputs can influence public decision-making 22. Users may unknowingly rely on AI-generated content, assuming it functions like a traditional information source 22.

General Business and User-Facing Applications

Hallucinations can erode trust and lead to productivity loss as users spend time verifying AI-generated content 21. Customer service chatbots providing incorrect product information can damage brand reputation 21. Eighty-three percent of surveyed executives admit to misinterpreting model confidence as accuracy 21. Strategic decision errors occur in 41% of cases, customer trust erosion in 33%, regulatory compliance risks in 28%, and operational inefficiencies in 37% 21. AI systems show confidence-calibration failures 68% more frequently than human experts in strategic analyses 21. Organizational challenges are also prevalent, with 58% of organizations propagating AI-generated errors through multiple departments, and teams accepting hallucinations 73% more often when attributed to "AI Strategy Systems" 21. Only 14% of enterprises maintain proper AI decision audit trails 21. Case studies include Meta pulling its Galactica LLM demo in 2022 after it provided inaccurate and sometimes prejudiced information 23, Google's Bard incorrectly claiming the James Webb Space Telescope had captured the world's first images of a planet outside our solar system 23, and Google's AI Overview citing an April Fool's satire as factual 22.

Ethical Dilemmas and Societal Concerns

The fabricated information generated by AI raises profound ethical dilemmas and societal concerns.

• Accountability: Leaders are ultimately accountable for decisions made with AI support 21. The absence of an identifiable author or agenda in AI-generated content creates accountability gaps 22.
• Blurring of Factual Lines: AI outputs, despite their confident presentation of fabricated information, blur the lines between truth and falsehood 21, which is particularly concerning in high-stakes environments where reliability is expected 22.
• Algorithmic Bias: AI systems can unintentionally perpetuate discrimination if trained on historically biased data 24. In finance, this can manifest as an AI credit scoring system penalizing certain demographics or an insurance algorithm underestimating health risks for specific groups due to unequal access reflected in past data 24. Even with explicit indicators of bias removed, AI may infer attributes through proxies, reproducing bias 24.
• Privacy and Security: AI systems handling sensitive data create an ethical obligation to protect customer privacy 24. Improper security can expose personally identifiable information, and integrating third-party AI services risks sending confidential data off-site 24. Malicious actors can exploit AI through prompt injection attacks or model jailbreaking to reveal confidential data 24.
• Explainability (Black Box Problem): Many complex AI models, especially deep learning systems, function as "black boxes," lacking transparency 24. Ethical decisions like credit scoring or fraud flagging should be explainable, and a lack of clear explanations for AI-driven rejections is unacceptable to consumers and regulators 24.
• Legal Implications: The growing risk of lawsuits when AI tools provide misleading information that harms users or investors is a significant concern 24.
• Abuse and Misuse: AI hallucinations can be abused to commit illegal acts, leading to potential threats to the social economy and national order 2. For instance, an AI-generated image of an explosion near the Pentagon caused a significant decline in the U.S. stock market 2.

Impact on User Trust, Safety, and the Spread of Misinformation/Disinformation

AI hallucinations severely impact user trust and safety while accelerating the spread of misinformation and disinformation.

• Erosion of Trust: When AI systems confidently generate false or fabricated information, user trust in these systems erodes 21. Stakeholders question reliability when institutions rely on fabricated AI outputs 24. Users often form trust in AI based on fluency, tone, and perceived authority, potentially overlooking accuracy 22.
• Spread of Misinformation: AI hallucinations represent a distinct form of misinformation 22. The ease with which AI can generate distorted information significantly lowers barriers to entry for false information and heightens its deceptive potential 2.
• Shallow Engagement: AI's fluency and confident tone align with cognitive preferences for easily processed content, encouraging shallow engagement and potentially facilitating "sycophantic outputs" that confirm user expectations 22. Digitally literate users often rely on surface cues, and younger audiences may misjudge credibility 22.
• Amplification of Falsehoods: Hallucinations can spread through group-level mechanisms like filter bubbles, echo chambers, or motivated reasoning, even without coordinated intent 22.

Mitigation Strategies (Brief Overview)

While the focus is on impacts and risks, researchers and practitioners are actively exploring mitigation strategies to reduce hallucination rates and their impact 21. These include improving the quality and accuracy of training data through data curation, integrating external, verified knowledge sources using Retrieval-Augmented Generation (RAG), specializing models and crafting effective prompts through fine-tuning and prompt engineering, incorporating human review and feedback via human-in-the-loop oversight, and developing techniques to understand model reasoning with Explainable AI (XAI) 21. Despite these efforts, the complexity of AI systems and the dynamic nature of information mean that hallucinations may never be fully eliminated, underscoring the necessity for ongoing vigilance, continuous monitoring, and adaptive governance frameworks 21.