⚡ P-Value Simulator

Understand what p-values REALLY mean through interactive simulation

🎓 What is a P-Value?

The Definition: A p-value is the probability of observing data as extreme as yours (or more extreme) assuming the null hypothesis is true.

⚠️ What a P-Value is NOT:
  • NOT the probability that the null hypothesis is true
  • NOT the probability that your result happened by chance
  • NOT the probability that you made a mistake
  • NOT a measure of effect size or importance

The α (alpha) level: Typically set at 0.05, this is your threshold. If p < 0.05, we reject the null hypothesis. But remember: this means 5% of the time, we'll get false positives even when the null is true!

🎩 Fisher's Wisdom:

"No scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses; he rather gives his mind to each particular case in the light of his evidence and his ideas." - R.A. Fisher

Fisher viewed p-values as a continuous measure of evidence, not a binary decision rule. The evidence grows stronger as p gets smaller!

Simulate Hypothesis Tests

If null is true, significant results are Type I errors (false positives)!
0.5 - Medium effect
n = 30 per group
Run multiple tests to see the pattern of p-values
Lower α means fewer false positives but less power

The Danger of P-Hacking

⚠️ What is P-Hacking?

P-hacking (also called data dredging) is when researchers try multiple analyses until they get p < 0.05, then report only the "significant" result. This inflates Type I error rates and leads to false discoveries!

Common P-Hacking Techniques (DON'T DO THESE!)

  • Optional Stopping: Checking results multiple times and stopping when p < 0.05
  • Selective Reporting: Testing many outcomes but only reporting the significant ones
  • Outlier Removal: Removing data points until results become significant
  • Subgroup Analysis: Testing many subgroups and reporting only significant ones
  • Variable Selection: Testing many covariates and keeping only those that "help"

🎣 Simulate P-Hacking

Watch what happens when you test multiple hypotheses but only report "significant" results:

Testing 20 different analyses
Simulating multiple researchers' studies
Set to "No" to see pure false positives!
← Back to Statistics Hub