🎭 The Complete Dance

Why "Dancing" p-Values Are Perfectly Calibrated

"The dance is not a defect—it is the very mechanism by which Type I error is controlled."

📚 The Claim We're Examining

Geoff Cumming's influential "Dance of the p-Values" visualization shows that p-values vary dramatically across repeated identical experiments. From this, many conclude:

❌ The Myth: "Because p-values dance wildly across replications, they are inherently unreliable and untrustworthy. We should abandon them for confidence intervals."
✓ The Mathematical Truth: The "dance" is not a pathology—it's the definition of a random variable! Under H₀, p-values follow a Uniform(0,1) distribution, which means exactly α proportion will fall below α. This is perfect calibration, not a defect. And confidence intervals "dance" just as wildly!
250 ms

📈 Dance of the p-Values

Current p-value
—
Take a sample to begin
α = 0.05
p-value
Significant (p < α)
Not Significant (p ≥ α)

Total Samples

0

Rejections

0

Rejection Rate

0%

Expected (α)

5%

📏 Dance of the Confidence Intervals

Current 95% CI
—
Take a sample to begin
μ = 100
Sample Mean with Confidence Interval
Captures μ
Misses μ

Total Intervals

0

Captured μ

0

Coverage Rate

0%

Expected (1-α)

95%

🔑 Key Insight

Start the simulation to see how both p-values and confidence intervals "dance" — but remain perfectly calibrated!

📊 Distribution of p-Values Under H₀

The Fundamental Theorem

Under the null hypothesis H₀, the p-value follows a Uniform(0,1) distribution. This means:

P(p ≤ α | H₀) = α

This is not a flaw—it's the defining property that makes hypothesis testing work! The "wild dancing" of p-values is exactly what produces the guaranteed Type I error rate.

Mean of p-values

—

Expected Mean

0.50

SD of p-values

—

Expected SD

0.29

🎯 What the Histogram Shows

Run at least 100 samples under H₀ to see the p-value histogram converge to Uniform(0,1). The golden line shows the expected uniform height.

⚖️ The Mathematical Duality

The Uncomfortable Truth for CI Advocates

Cumming argues we should prefer confidence intervals because p-values "dance wildly." But watch carefully:

  • CI endpoints are random variables — they dance too!
  • The decision "CI excludes 0" flips yes/no just like "p < 0.05"
  • Coverage probability (1-α) is mathematically dual to Type I error (α)
  • Both have identical power profiles for the same data
Property p-Value Confidence Interval
Random variable? ✓ Yes ✓ Yes (endpoints)
"Dances" across samples? ✓ Yes ✓ Yes
Calibrated under H₀? ✓ P(p ≤ α) = α ✓ Coverage = 1-α
Binary decision stability Varies with power Varies with power
Observed Rejections — —
Observed Misses — —

⚡ The Logical Flaw

If "p-values dance, therefore they're unreliable" were valid, then "CI endpoints dance, therefore they're unreliable" would be equally valid. The argument proves too much—it would invalidate ALL sample-based inference!

💪 The Real Issue: Power, Not p-Value Pathology

Why "Significance" Doesn't Always Replicate

Cumming shows that one study might give p = 0.03 and a replication gives p = 0.15. This is presented as a flaw of p-values. But it's actually a power issue:

Power = P(reject H₀ | H₁ true) = P(p < α | effect exists)

If power = 60%, then by design: ~60% of replications will give p < α, and ~40% will give p > α. This is not a defect—it's the definition of 60% power!

5.0%

Samples Taken

0

Significant Results

0

Observed Rate

0%

Expected (Power)

5%

🎯 The Resolution

When effect size = 0 (H₀ true), only ~α = 5% of studies show "significance." As effect size increases, power increases, and more replications will be "significant." The "replication crisis" is a power crisis, not a p-value crisis!

📝 Conclusion: Completing the Picture

Cumming's "Dance of the p-Values" visualization correctly shows sampling variability. The visualization is accurate! What's problematic is the inference:

The Non Sequitur: "P-values vary across samples → P-values are unreliable"

This reasoning would invalidate all of statistics. The proper criterion is not whether p-values "look stable" but whether the procedure is calibrated:

  • ✓ Type I error rate equals α (exactly, by construction)
  • ✓ CI coverage equals 1-α (the mathematical dual)
  • ✓ "Replication instability" equals 1 - Power (not a pathology)

The dance of p-values is not evidence against NHST—it is the very mechanism by which NHST achieves its guaranteed error rates.