Why "Dancing" p-Values Are Perfectly Calibrated
"The dance is not a defect—it is the very mechanism by which Type I error is controlled."
Geoff Cumming's influential "Dance of the p-Values" visualization shows that p-values vary dramatically across repeated identical experiments. From this, many conclude:
Start the simulation to see how both p-values and confidence intervals "dance" — but remain perfectly calibrated!
Under the null hypothesis H₀, the p-value follows a Uniform(0,1) distribution. This means:
This is not a flaw—it's the defining property that makes hypothesis testing work! The "wild dancing" of p-values is exactly what produces the guaranteed Type I error rate.
Run at least 100 samples under H₀ to see the p-value histogram converge to Uniform(0,1). The golden line shows the expected uniform height.
Cumming argues we should prefer confidence intervals because p-values "dance wildly." But watch carefully:
| Property | p-Value | Confidence Interval |
|---|---|---|
| Random variable? | ✓ Yes | ✓ Yes (endpoints) |
| "Dances" across samples? | ✓ Yes | ✓ Yes |
| Calibrated under H₀? | ✓ P(p ≤ α) = α | ✓ Coverage = 1-α |
| Binary decision stability | Varies with power | Varies with power |
| Observed Rejections | — | — |
| Observed Misses | — | — |
If "p-values dance, therefore they're unreliable" were valid, then "CI endpoints dance, therefore they're unreliable" would be equally valid. The argument proves too much—it would invalidate ALL sample-based inference!
Cumming shows that one study might give p = 0.03 and a replication gives p = 0.15. This is presented as a flaw of p-values. But it's actually a power issue:
If power = 60%, then by design: ~60% of replications will give p < α, and ~40% will give p > α. This is not a defect—it's the definition of 60% power!
When effect size = 0 (H₀ true), only ~α = 5% of studies show "significance." As effect size increases, power increases, and more replications will be "significant." The "replication crisis" is a power crisis, not a p-value crisis!
Cumming's "Dance of the p-Values" visualization correctly shows sampling variability. The visualization is accurate! What's problematic is the inference:
This reasoning would invalidate all of statistics. The proper criterion is not whether p-values "look stable" but whether the procedure is calibrated:
The dance of p-values is not evidence against NHST—it is the very mechanism by which NHST achieves its guaranteed error rates.