The Difference Between "Replicable" and "Not Replicable" Is Not Itself Scientifically Replicable

Abstract

Replication studies estimate the replicability rate of scientific results by aggregating binary verdicts of experiments as ``replicated'' and ``not replicated''. These estimates assume rarely attainable exact replications. Experiments often do not share a common data-generating process, leading to non-exact replications. We formalize two statistical interpretations of non-exactness. In a shared latent rate model (benchmark), experiments are exchangeable conditional on a common random replicability rate. In a conditionally independent rates model (operational), each experiment has its own random replicability rate, drawn independently from a population. Under the benchmark model, even small variability in assumed replicability rates induces a variance floor of the estimated mean replicability rate that is irreducible by more replications. Under the operational model, the degree of non-exactness is not identifiable from binary verdict data, which contain no information about between-experiment heterogeneity. Researchers cannot tell which precision regime they are in, and high- and low-replicability sequences cannot be distinguished in principle. Binary verdicts systematically understate uncertainty in reporting scientific results, making high- and low-replicability sequences appear discriminable, when they are not. Common sources of heterogeneity amplify these problems as shown in a reanalysis of Many Labs~4 data. The practice of aggregating replicability rates across heterogeneous literatures produces averages that conflate incommensurable experimental regimes and lack a stable scientific interpretation. Hence, replicability rates as currently used do not provide a reliable demarcation criterion for scientific results. The replication crisis, if there is one, cannot be established by methods used to declare it.