The p-value is probably the most used — and most misread — number in all of statistics. It shows up in scientific papers, A/B tests, quality reports and the output of every regression. This article explains, in plain language, what it measures, what it is used for in practice, what it does not tell you, and how it compares with other tools: confidence intervals, effect sizes, Bayesian methods and information criteria.
§1The idea with a coin
You suspect a coin is biased. You flip it 100 times and get 60 heads. Is that proof of bias, or could a fair coin do this by chance?
The logic of a test has three steps:
- Assume the opposite of what you want to show. This is the null hypothesis \(H_0\): the coin is fair, \(p = 0.5\).
- Pick a measure of “how far” the data are from what you expected. With 100 flips we expect 50 heads; we saw 60, a distance of 10.
- Compute the probability of a distance that large or larger, assuming \(H_0\) is true. That is the p-value.
For a fair coin, the chance of 60 or more heads or 40 or fewer (a deviation of 10 in either direction) is
In other words: a perfectly fair coin would produce a result this lopsided in about 6 out of every 100 runs of 100 flips. Rare, but not very rare. Under the most common convention (a 0.05 cutoff) we would not “reject” the coin's fairness — but only just.
§2The definition, plainly
In general we choose a test statistic \(T\) (a number that grows as the data move away from \(H_0\)), compute its observed value \(t_{\text{obs}}\) and define
Three details matter:
- It is a probability about the data, not about the hypothesis. “\(H_0\) true” sits on the right of the bar. The p-value does not tell you the chance that \(H_0\) is true.
- It includes “or more extreme”. It is not the chance of seeing exactly 60 heads (which is small for any number), but of something at least that extreme.
- Under \(H_0\), the p-value is uniform between 0 and 1. If there is no effect, \(p < 0.05\) happens 5% of the time, \(p < 0.01\) 1% of the time, and so on. This is where the control of “false positives” comes from.
The significance level \(\alpha\) (typically 0.05) is a cutoff agreed on before looking at the data: if \(p \le \alpha\), we call the result “statistically significant”. This keeps the long-run false-positive rate at or below \(\alpha\).
§3Where the p-value is used
The same logic runs through almost all of applied statistics. The most common uses:
| Situation | Typical null hypothesis | Common test |
|---|---|---|
| Compare the mean of two groups (treatment vs control) | The means are equal | t-test |
| Compare proportions (conversion A/B test) | The conversion rates are equal | z-test for proportions, chi-square |
| Compare three or more groups | All groups share the same mean | ANOVA, Kruskal–Wallis |
| A regression coefficient | The variable has no effect (\(\beta = 0\)) | t / Wald test |
| Association between two categorical variables | The variables are independent | Chi-square, Fisher |
| Correlation | The correlation is zero | Pearson / Spearman test |
| Fit of a model or distribution | The data follow the assumed distribution | Shapiro–Wilk, Kolmogorov–Smirnov, deviance test |
| Compare nested models | The smaller model is enough | Likelihood-ratio test, F-test |
| Clinical trials | The drug is no better than placebo | Depends on the outcome; the protocol fixes everything in advance |
| Quality control | The process is in control | Control charts (each point is an implicit test) |
In every case the p-value plays the same role: it is an alarm against chance. It tells you whether the pattern you see is too large to blame on the natural variation of the data alone.
§4What the p-value does not say
Nearly every mistake with p-values comes from asking it a question it does not answer.
- It is not the probability that \(H_0\) is true. \(p = 0.03\) does not mean “a 3% chance there is no effect”. That probability can only be computed with a Bayesian approach (§7), and it depends on how plausible the effect was before the study.
- It does not measure the size of the effect. A tiny effect gives a tiny p-value if the sample is huge; a large effect can give a high p-value if the sample is small. (Examples in §5.)
- \(p > 0.05\) does not prove there is no effect. “Absence of evidence is not evidence of absence.” It may just be a lack of data.
- It does not say whether the result matters. Statistical significance is not practical significance. A drug that lowers blood pressure by 0.5 mmHg can be “significant” and clinically irrelevant.
- 0.05 is not a magic border. \(p = 0.049\) and \(p = 0.051\) carry practically the same evidence. Treating one as a “discovery” and the other as “nothing” is arbitrary.
- It is only valid if the assumptions hold. The calculation assumes a model (independence, a distribution, unbiased data collection). Badly collected data produce precise, meaningless p-values.
§5Two A/B tests that show the problem
Consider two conversion experiments on a website.
Test A — huge sample, tiny effect. One million visitors on each version. The old version converts 10.0%; the new one, 10.1%. The p-value is 0.019: “significant”. But the difference is one tenth of a percentage point, and the 95% interval for it runs from 0.02 to 0.18 points. It may not even cover the cost of changing the site.
Test B — small sample, large effect. Two hundred visitors on each version. The old one converts 10%; the new one, 13%. The p-value is 0.35: “not significant”. But the observed difference is 30 times larger than in test A, and the 95% interval runs from −3.2 to +9.2 points. The test simply did not have enough data to conclude anything — neither that it works nor that it doesn't.
Looking only at the p-value would lead to the wrong decision in both cases. That is why it must always come with an estimate of the effect.
§6Comparison 1: confidence intervals and effect sizes
A confidence interval answers a more useful question: “which values of the effect are compatible with the data?”. For the coin in §1, the 95% interval for the probability of heads runs from 0.497 to 0.697.
The interval contains the p-value and more:
- It includes 0.5, so the 5% test does not reject fairness — the same conclusion as the p-value. (Intervals and tests are two sides of the same calculation: \(H_0\) is rejected at 5% exactly when its value falls outside the 95% interval.)
- But it also shows the coin could be considerably biased, up to 70% heads. The honest conclusion is not “the coin is fair” but “we don't know yet”.
Alongside the interval you report an effect size: the difference in means, the odds ratio, Cohen's \(d\), the relative risk. That is what tells you whether the result matters. A good rule: effect + interval first, p-value second (if at all). For statistics without a ready-made interval formula, the bootstrap does the job.
§7Comparison 2: the Bayesian approach
The p-value comes from frequentist statistics, which treats the parameter as fixed and the data as random. Bayesian statistics flips this: the data are fixed (they have been observed) and the uncertainty about the parameter is described by a probability distribution. That lets it answer directly the questions people usually (wrongly) put to the p-value.
For the same coin, starting from a neutral initial belief (prior) — any probability of heads between 0 and 1 equally plausible — the data lead to:
- Probability that the coin favours heads, \(\Pr(p > 0.5 \mid \text{data}) \approx 0.98\).
- 95% credible interval from 0.50 to 0.69 — almost the same as the confidence interval, but now with the direct reading “there is a 95% probability that \(p\) lies here”.
- Bayes factor \(\approx 0.9\) comparing “biased coin (any bias)” with “exactly fair coin”. A factor near 1 means the data barely distinguish the two hypotheses.
Notice the tension: the p-value of 0.057 looks “almost significant”, the 98% probability looks strong, and the Bayes factor says the evidence against fairness is practically nil. There is no contradiction — each number answers a different question. The Bayes factor pits “exactly 0.5” against a diffuse alternative that spreads its bet over every value, including 0.1 or 0.9, which the data rule out. It illustrates a well-known point: p-values near 0.05 often represent weaker evidence than they seem to.
Bayesian strengths: answers in the language people want (“what is the chance it works?”), uses prior knowledge, and does not depend on how many times you “peeked” at the data. Weaknesses: you must choose a prior (and the result can depend on it, as with the Bayes factor), and the computations are usually heavier.
§8Comparison 3: information criteria and validation
When the goal is choosing between models — rather than testing one specific hypothesis — there are better options than a chain of p-values:
- AIC and BIC score each model by its fit, with a penalty for the number of parameters. You pick the lowest. They work for non-nested models (which the likelihood-ratio test cannot compare) and do not require casting one model as the “null”.
- Cross-validation measures directly what matters for prediction: the error on data the model has not seen. It is the standard tool in machine learning, where hardly anyone looks at p-values.
Variable selection by “drop the one with the largest p-value and refit” (stepwise) is a classic and problematic use: after many data-driven choices, the final p-values are far too optimistic and lose their original meaning.
§9Many tests at once
If each test has a 5% chance of a false positive, running 20 independent tests on things with no effect at all gives a 64% chance of at least one “significant” result:
This is how findings that fail to replicate are born: test many things (subgroups, outcomes, periods, variables) and publish whatever crossed 0.05 — so-called p-hacking. The most common corrections:
- Bonferroni: with \(m\) tests, use the cutoff \(\alpha/m\). Simple and conservative; controls the chance of any false positive.
- Holm: a step-down version of Bonferroni, always at least as powerful.
- Benjamini–Hochberg (FDR): controls the proportion of false positives among the discoveries. Standard in genomics and in any field with hundreds or thousands of tests.
The best defence, though, is not a formula: it is fixing the hypotheses and the analysis before seeing the data (pre-registration), as clinical trials do.
§10Summary: which tool for which question
| Question | Tool |
|---|---|
| Could this pattern be just chance? | p-value |
| How large is the effect, and how uncertain? | Estimate + confidence interval |
| What is the probability the effect is positive (or larger than X)? | Bayesian posterior |
| How much do the data favour one hypothesis over another? | Bayes factor, likelihood ratio |
| Which model describes or predicts better? | AIC/BIC, cross-validation |
| How much data do I need to detect an effect of size X? | Power calculation (done before the study) |
The p-value is neither villain nor hero. It is a narrow tool that answers one question well: “is this compatible with pure chance?”. Used in that role — together with the effect size, an interval, a well-designed study and care with multiple tests — it remains useful. The trouble starts when it becomes the only number anyone looks at, and the 0.05 cutoff alone decides what is true.