Heads, Tails, and AI Fails: LLMs, Randomness, and Human Judgments

Abstract

Randomness is central to human cognition and to many applications in which large language models are deployed, yet probabilistic token generation does not imply that LLMs can produce unbiased random sequences. We study how contemporary LLMs generate binary random sequences using the classic behavioral-science paradigm of simulated coin flips. Across single flips, 20-flip sequences, n-gram statistics, run lengths, alternation rates, and next-flip predictability, we compare model outputs to both true Bernoulli baselines and human data from prior work. We find that LLMs reproduce several canonical human randomness biases, including over-alternation, aversion to long runs, and first-flip biases, but often amplify them or introduce model-specific distortions. Increasing temperature reduces some rigid patterns but does not eliminate systematic structure. We further investigate over-alternation through prompt-framing experiments, continuation tests, corpus searches, and residual-stream probes, finding evidence that this bias is not explained by direct memorization or tokenization alone. These results show that LLMs are unreliable generators of randomness and imperfect simulators of human randomness judgments, with implications for applications requiring stochasticity, fair sampling, or behavioral simulation.