Practical Significance
How interval-null hypotheses resolve statistical paradoxes, align paradigms, and focus science on effects that matter
The fundamental shift: Stop asking "Is the effect exactly zero?" and start asking "Is the effect large enough to matter?" This simple change resolves decades of controversy and puts statistics back in service of science.
Statistical significance merely tells you whether an effect is detectably different from zero given your sample size. Practical significance asks whether that effect is large enough to care about.
Traditional Sharp Null
H₀: θ = θ₀ (effect is exactly zero)
Problems:
- Always false in practice
- Detects tiny, meaningless effects with large samples
- Creates the "large-sample paradox"
Interval Null (SESOI)
H₀: |θ - θ₀| ≤ δ (effect too small to matter)
Advantages:
- Tests what we actually care about
- Distinguishes meaningful from negligible
- Stable conclusions as n grows
Smallest Effect Size of Interest (SESOI)
Define δ as the minimum effect that would change decisions or have practical consequences. Then test:
H₀: θ ∈ [θ₀ - δ, θ₀ + δ] versus H₁: |θ - θ₀| > δ
Design with δ in mind
Report effects + uncertainty
Focus on decisions, not p-values
1. The Jeffreys-Lindley Paradox
The problem: With large samples, frequentist tests give tiny p-values (rejecting H₀) while Bayesian methods give large Bayes factors supporting H₀.
The resolution: The paradox arises because frequentist tests target a point null (θ = θ₀) while Bayesian priors place mass exactly at that point. When both frameworks test an interval null, they answer the same question and converge on the same conclusion.
Alignment of Methods
- Frequentist TOST (Two One-Sided Tests): Rejects interval null when CI excludes [θ₀ - δ, θ₀ + δ]
- Bayesian ROPE (Region of Practical Equivalence): Calculates P(|θ - θ₀| ≤ δ | data)
- Result: Both assess whether the effect is practically negligible vs. meaningful
2. The Large-Sample (Berkson) Paradox
The problem: As n → ∞, even trivially small effects become "statistically significant" under sharp nulls, making p-values useless for large studies.
The resolution: With an interval null, larger samples correctly classify effects as either inside the negligible region (practical equivalence) or decisively outside it (meaningful difference). P-values remain informative regardless of sample size.
💡 Key Insight: The paradoxes aren't flaws in statistical methods—they're symptoms of asking the wrong question. Change the question from "Is it zero?" to "Does it matter?" and both paradigms work beautifully.
Interval-null testing uses the same statistical machinery as traditional tests, but evaluates evidence against both boundaries of the null region.
Define your hypotheses
Specify θ₀ (null value), δ (SESOI), and α (error rate, typically 0.05)
H₀: |θ - θ₀| ≤ δ (effect is negligible)
H₁: |θ - θ₀| > δ (effect is meaningful)
Compute test statistics for both boundaries
Calculate t-statistics (or z-statistics) at each boundary of the null region:
t_L = (x̄ - (θ₀ - δ)) / (s/√n)
t_U = ((θ₀ + δ) - x̄) / (s/√n)
These test whether the true mean could plausibly be at the lower or upper boundary.
Calculate one-sided p-values
For each boundary, find the tail probability:
p_L = P(T ≥ t_L | df = n-1)
p_U = P(T ≥ t_U | df = n-1)
Apply decision rules
| Conclusion |
Condition |
Interpretation |
| Practical Equivalence |
p_L < α AND p_U < α |
Effect is negligible (inside SESOI) |
| Meaningful Difference |
min(p_L, p_U) < α |
Effect exceeds SESOI |
| Indeterminate |
Neither condition met |
Need more data |
🧮 Worked Example
Setup: x̄ = 10.4, θ₀ = 10, s = 1.5, n = 25, δ = 0.5, α = 0.05
Step 1: Calculate test statistics
SE = s/√n = 1.5/√25 = 0.3
t_L = (10.4 - 9.5) / 0.3 = 3.00
t_U = (10.5 - 10.4) / 0.3 = 0.33
Step 2: Find p-values (df = 24)
p_L = 0.0029
p_U = 0.37
Step 3: Decision
- Both p-values < 0.05? No → Cannot claim equivalence
- min(p_L, p_U) = 0.0029 < 0.05? Yes → Meaningful difference detected
Conclusion: The true mean differs from θ₀ = 10 by more than the SESOI of 0.5.
Confidence Interval Interpretation
The (1 - 2α) CI provides the same information:
- CI entirely within [θ₀ - δ, θ₀ + δ] → Equivalence
- CI entirely outside [θ₀ - δ, θ₀ + δ] → Meaningful difference
- CI overlaps boundaries → Indeterminate
The Bayesian analogue to interval-null testing defines a Region Of Practical Equivalence (ROPE) and calculates posterior probabilities.
ROPE Procedure
- Define the same ROPE as your SESOI: [θ₀ - δ, θ₀ + δ]
- Compute posterior: P(|θ - θ₀| ≤ δ | data)
- Apply decision thresholds:
- P > 0.95 → Practical equivalence
- P < 0.05 → Meaningful difference
- 0.05 ≤ P ≤ 0.95 → Indeterminate
When frequentist and Bayesian analyses use the same δ, they typically reach consistent conclusions, especially with reasonable priors and adequate sample sizes. This convergence demonstrates that the interval-null framework addresses the scientific question rather than mathematical technicalities.
Paradigm unification
Same SESOI = consistent answers
Focus on practical relevance
The American Statistical Association has progressively refined guidance on p-values and significance testing:
2016 Statement on P-Values
Formalized six principles clarifying that p-values:
- Do not measure effect size or importance
- Are one piece of evidence, not a complete answer
- Require transparency and good experimental design
2019: "Moving Beyond p < 0.05"
Special issue in The American Statistician discouraged bright-line thinking and arbitrary thresholds for declaring "significance."
2021 ASA President's Task Force
Provided practical guidance linking hypothesis tests to replicability:
- Control error rates through proper design
- Pre-specify analyses to avoid p-hacking
- Report effect sizes and intervals, not just p-values
- Avoid dichotomous claims (significant/not significant)
- Align methods with scientific questions
The message is clear: misuse is not a reason for abolition. Use the right tool for the right question.
Before Data Collection
- Pre-register your SESOI (δ) based on domain knowledge, pilot studies, or minimal clinically important difference
- Specify primary endpoints and analysis plans to prevent selective reporting
- Power your study for practical decisions about ±δ, not just "non-zero" effects
- Plan for multiplicity if testing multiple hypotheses
During Analysis
- Use interval-null tests or equivalence testing (TOST, ROPE)
- Report effect sizes with confidence/credible intervals
- Show posterior mass inside/outside ROPE when using Bayesian methods
- Acknowledge uncertainty when results are indeterminate
After Publication
- Share code and data for transparency
- Conduct prospective replications on new samples
- Audit for selection effects (publication bias, p-hacking)
- Update meta-analyses as evidence accumulates
The Central Question
Design every study to answer:
"Is this effect meaningful enough to matter?" not merely "Is it non-zero?"
Pre-registration
Effect sizes + CIs
Transparent thresholds
Prospective replication
The replication crisis is fundamentally a crisis of asking the wrong questions. We've spent decades testing whether effects are exactly zero—a question that's almost never scientifically interesting and always technically false.
The solution isn't to ban statistical tests. It's to change what we test. Interval-null hypotheses with SESOI:
- Align frequentist and Bayesian reasoning by focusing both on practical relevance
- Eliminate paradoxes that arise from testing point nulls
- Scale properly with sample size, giving stable answers as n grows
- Focus science on effects that matter for decisions and applications
- Harmonize with existing best practices (equivalence testing, ROPE, ASA guidance)
This isn't a revolutionary new method—it's a Kuhnian paradigm shift that repurposes existing tools to answer better questions. The mathematics stays the same; the interpretation becomes scientific rather than purely mathematical.
The Three Principles of Modern Statistical Practice
- Design for meaningful effects — Define and power studies around SESOI
- Report with honesty — Show uncertainty, avoid dichotomous thinking
- Replicate by design — Make reproducibility part of the research plan, not an afterthought
"How do I choose δ?"
The SESOI should come from domain expertise, not statistical convenience:
- Clinical context: Minimal clinically important difference (MCID)
- Education: Effect sizes that change pedagogical decisions
- Business: Differences large enough to justify costs
- Meta-analysis: Smallest effect observed in successful interventions
When uncertain, conduct a pilot study or use established benchmarks (e.g., Cohen's d = 0.2, 0.5, 0.8). The key is to pre-specify and justify your choice.
"Doesn't this just shift the arbitrary threshold problem?"
No—there's a fundamental difference:
- α = 0.05 is arbitrary because it's purely conventional
- δ is substantive because it reflects domain knowledge about what matters
Moreover, you can conduct sensitivity analyses showing how conclusions change across a range of reasonable δ values, something meaningless for α.
"Won't researchers game the system by choosing convenient δ values?"
Pre-registration and transparency are the solution:
- Require justification for SESOI choice before data collection
- Report sensitivity to different δ values
- Make reviewers and readers judge whether δ is reasonable
This is no different from requiring researchers to justify their outcome measures, sample sizes, or analysis plans—it's part of good scientific practice.
"What about exploratory research where we don't know what matters?"
Interval nulls work for exploration too:
- Use conservative (small) δ values for discovery phases
- Report findings as "potentially meaningful pending confirmation"
- Design follow-up confirmatory studies with pre-specified SESOI
The framework doesn't eliminate exploration—it clarifies when you're exploring versus confirming.
Medicine & Healthcare
- Drug trials: Non-inferiority testing (new drug no worse than standard by δ)
- Diagnostic tests: Equivalence of new vs. gold-standard methods
- Quality of life: Meaningful changes in patient-reported outcomes
- Bioequivalence: Generic drugs within ±20% of brand-name
Psychology & Social Science
- Replication studies: Testing if effect size is close to original
- Intervention effectiveness: Changes large enough to justify programs
- Measurement invariance: Testing if scales work equivalently across groups
- Theory testing: Distinguishing small from zero effects
Education
- Teaching methods: Gains worth changing curriculum
- Assessment tools: Score differences that affect placement decisions
- Program evaluation: Effects justifying resource allocation
Engineering & Manufacturing
- Quality control: Specifications within tolerance limits
- Process validation: New methods equivalent to established ones
- A/B testing: Changes worth implementing at scale
🏥 Case Study: Clinical Trial Design
Scenario: Testing whether a new antihypertensive drug is non-inferior to the standard treatment.
Traditional approach: Test H₀: μ_new = μ_standard
- Problem: With large n, tiny clinically meaningless differences become "significant"
- Result: Confusion about whether to adopt new drug
Interval-null approach: Define δ = 5 mmHg (clinically important difference), test H₀: |μ_new - μ_standard| ≤ 5
- If reject H₀: New drug differs meaningfully (avoid it or promote it, depending on direction)
- If fail to reject: New drug is equivalent (adopt if it has other advantages like cost or side effects)
- Indeterminate: Need larger sample
Outcome: Clear, actionable decision directly aligned with clinical needs.
Software Packages
R Packages
TOSTER — Two One-Sided Tests for equivalence
equivalence — Equivalence testing framework
bayestestR — ROPE analysis and Bayesian testing
BayesFactor — Bayes factors with interval hypotheses
Python Libraries
pingouin — Statistical testing including TOST
scipy.stats — Core statistical functions
PyMC — Bayesian inference with custom ROPEs
arviz — Visualization of Bayesian results
Sample Size Calculators
Power analysis for interval-null tests requires specialized calculators that account for the two one-sided tests:
- G*Power: Free software with equivalence testing modules
- PASS: Comprehensive power analysis for equivalence/non-inferiority
- R packages:
PowerTOST, TrialSize
Quick Reference: Sample Size Considerations
For equivalence testing, you typically need larger samples than traditional tests because you're trying to demonstrate that the CI falls within a narrow range. Rule of thumb:
- Narrower δ → larger n required
- For 80% power with α = 0.05, plan for roughly 1.5-2× the sample size of a traditional superiority test
- Always conduct formal power analysis with your specific δ
Reporting Standards
Essential Elements for Transparent Reporting
- Pre-specification: State SESOI (δ) with justification before data analysis
- Effect estimates: Report point estimates with confidence/credible intervals
- Visualizations: Show CI relative to SESOI boundaries
- Decision criteria: State threshold for equivalence/meaningful difference
- Sensitivity analysis: Show how conclusions change with different δ values
- Interpretation: Discuss practical implications, not just statistical results
Areas for Development
Adaptive SESOI
Can we develop frameworks where δ adapts based on:
- Accumulating evidence from meta-analyses
- Cost-benefit trade-offs that change over time
- Population characteristics (personalized medicine)
Multivariate Extensions
Most research involves multiple outcomes. How do we define interval nulls for:
- Vector-valued parameters
- Multiple co-primary endpoints
- High-dimensional data (genomics, neuroimaging)
Educational Reform
Widespread adoption requires:
- Updating statistics curricula at all levels
- Training practicing researchers through workshops
- Developing intuitive visualization tools
- Creating discipline-specific guidelines for SESOI selection
Policy & Incentives
Institutional changes needed:
- Journal requirements for pre-registered SESOI
- Funding agency support for adequately powered equivalence studies
- Regulatory acceptance (FDA, EMA) for interval-null frameworks
- Career incentives aligned with robust, replicable research
The Bigger Picture
The interval-null framework is part of a broader movement toward pragmatic, decision-focused inference. It aligns with parallel developments in:
- Estimation thinking: Focus on effect sizes rather than null hypothesis rejection
- Decision theory: Explicit incorporation of costs, benefits, and utilities
- Robust statistics: Methods that work across a range of assumptions
- Open science: Transparency, pre-registration, and data sharing
Together, these approaches represent a maturation of statistical practice—moving from mechanical procedures to thoughtful, context-sensitive inference that serves scientific discovery and practical decision-making.
The statistical tools already exist. What's needed now is a collective commitment to use them wisely.
Every researcher, every reviewer, every journal editor, and every funding agency can contribute to this shift. The question isn't whether interval nulls are perfect—it's whether they're better than sharp nulls. The answer is unequivocally yes.
What You Can Do Now
As a Researcher
- Define SESOI in your next study protocol
- Report effect sizes with CIs in all publications
- Pre-register analysis plans including δ
- Conduct equivalence tests when appropriate
- Share data and code openly
As a Reviewer/Editor
- Ask authors to justify their SESOI
- Require reporting of effect sizes and intervals
- Accept well-powered equivalence studies
- Discourage bright-line p-value thinking
- Value transparency over novelty
As an Educator
- Teach interval nulls alongside traditional tests
- Emphasize effect sizes from day one
- Use real examples where SESOI matters
- Show paradoxes that sharp nulls create
- Connect statistics to scientific questions
As a Consumer of Research
- Ask "How large is the effect?" not just "Is it significant?"
- Look for confidence intervals, not just p-values
- Be skeptical of barely-significant results
- Value replication over novelty
- Support open science practices
"The best time to start using interval nulls was 30 years ago.
The second-best time is now."