🎓 What is a P-Value?
The Definition: A p-value is the probability of observing data as extreme as yours (or more extreme) assuming the null hypothesis is true.
- NOT the probability that the null hypothesis is true
- NOT the probability that your result happened by chance
- NOT the probability that you made a mistake
- NOT a measure of effect size or importance
The α (alpha) level: Typically set at 0.05, this is your threshold. If p < 0.05, we reject the null hypothesis. But remember: this means 5% of the time, we'll get false positives even when the null is true!
"No scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses; he rather gives his mind to each particular case in the light of his evidence and his ideas." - R.A. Fisher
Fisher viewed p-values as a continuous measure of evidence, not a binary decision rule. The evidence grows stronger as p gets smaller!
Simulate Hypothesis Tests
The Danger of P-Hacking
⚠️ What is P-Hacking?
P-hacking (also called data dredging) is when researchers try multiple analyses until they get p < 0.05, then report only the "significant" result. This inflates Type I error rates and leads to false discoveries!
Common P-Hacking Techniques (DON'T DO THESE!)
- Optional Stopping: Checking results multiple times and stopping when p < 0.05
- Selective Reporting: Testing many outcomes but only reporting the significant ones
- Outlier Removal: Removing data points until results become significant
- Subgroup Analysis: Testing many subgroups and reporting only significant ones
- Variable Selection: Testing many covariates and keeping only those that "help"
🎣 Simulate P-Hacking
Watch what happens when you test multiple hypotheses but only report "significant" results: