Hypothesis testing and the p-value - significance tests explained step by step

The idea: could it just be luck?

Maya says she can taste the difference between tap water and bottled water. To check, a friend fills $20$ cups, each at random with one or the other. Maya cannot see which is which, and for each cup she says which water it is. She gets $15$ right.

Does that prove she has a good palate? Not yet. Someone who is only guessing gets each cup right with probability $\frac{1}{2}$ and could get lucky. So we ask: if Maya were only guessing, how surprising would $15$ right answers be?

That question is the whole idea of a hypothesis test:

• assume that nothing special is going on - this is the null hypothesis $H_0$ (here: Maya is guessing),
• work out how likely a result at least as extreme as the one we saw would be under that assumption - this is the p-value,
• if that probability is very small, the assumption is hard to believe, and we reject it.

A small p-value does not say for certain that Maya can taste the difference. It says that guessing would rarely give such a result. The statistician Ronald Fisher explained hypothesis testing with a similar story: a lady who said she could taste whether the tea or the milk had been poured into the cup first.

Guessing or tasting? The test step by step

Go through the animation step by step with the buttons or the arrow keys. First you see Maya’s $20$ cups, then the scores of $1000$ people who only guess, all taking the same test (simulated on a computer).

The test can go wrong in two ways. If Maya is guessing, she still lands in the critical region with probability $0.021$ - then we would reject a true $H_0$, a type I error. If she really can taste the difference, she may still score below $15$ - a type II error. With the same α, a longer test, with more cups, makes the second kind of mistake less likely.

Try it yourself: cups, right answers and α

The sliders set the number of cups $n$ and the number of right answers $k$. The bars show how likely each score is for someone who is guessing, and the filled bars add up to the p-value $P(X \ge k)$. Choose α to see the critical region and the decision. Then test a few hundred guessers and compare the share of them with $k$ or more right answers with the p-value.

What to notice:

• with $n = 20$, $k = 15$ gives $p = 0.021$, but $k = 14$ already gives $p = 0.058$ - more than $0.05$,
• the same share of right answers is stronger evidence in a longer test: $6$ of $8$ cups gives $p = 0.145$, $15$ of $20$ gives $p = 0.021$, and $30$ of $40$ gives $p = 0.001$,
• a smaller α makes the critical region smaller: with $n = 20$ and $\alpha = 0.01$ it is $X \ge 16$, so $15$ right answers would no longer be enough,
• with only $5$ cups there is no critical region at $\alpha = 0.01$: even $5$ of $5$ has $p = 0.031$,
• the more guessers you test, the closer the share of them with $k$ or more right answers comes to the p-value.

The definition

Hypotheses. A hypothesis test weighs two statements against each other. The null hypothesis $H_0$ is the claim the test starts from - usually “no effect” or a stated value, such as $p = 0.5$ or $\mu = 1000$ g. The alternative hypothesis $H_1$ is what we suspect instead, such as $p > 0.5$, $\mu < 1000$ g or $\mu \ne 500$ ml.

Test statistic. A number worked out from the sample, such as the number of right answers $X$. We need to know how it behaves when $H_0$ is true. If each of $n$ independent trials is a success with probability $p_0$, the number of successes follows the binomial distribution:

The p-value. The probability, worked out assuming $H_0$ is true, of a test statistic at least as extreme as the one observed - extreme in the direction of $H_1$. For $H_1: p > p_0$ and $k$ successes, it is

Significance level and decision. Before looking at the data we choose the significance level $\alpha$. A common choice is $0.05$; $0.01$ and $0.10$ are also used. Then:

• if $p \le \alpha$, we reject $H_0$ in favour of $H_1$ - the result is statistically significant,
• if $p > \alpha$, there are no grounds to reject $H_0$. That does not prove $H_0$ is true: the data are simply not strong enough evidence against it.

Critical region. The values of the test statistic for which we reject $H_0$. In the tasting test with $n = 20$ and $\alpha = 0.05$ it is $X \ge 15$, because $P(X \ge 15) = 0.021 \le 0.05$ but $P(X \ge 14) = 0.058 > 0.05$. Rejecting $H_0$ is the same as the test statistic landing in the critical region.

Two kinds of error.

• A type I error is rejecting $H_0$ when it is true. Its probability is at most $\alpha$; in the tasting test it is $P(X \ge 15) = 0.021$.
• A type II error is not rejecting $H_0$ when it is false. Its probability is written $\beta$, and $1 - \beta$ is the power of the test - in the animation, $0.804$ for a taster who is right $4$ times out of $5$.

One-sided and two-sided tests. With $H_1: p > p_0$ or $H_1: \mu < \mu_0$ the test is one-sided: only results far out in one direction count as extreme. With $H_1: \mu \ne \mu_0$ it is two-sided: results far out on either side count. For a statistic with a symmetric distribution, such as $z$ below, the two-sided p-value is twice the area of one tail.

The z-test for a mean. When the values have a known standard deviation $\sigma$, the mean $\bar{x}$ of a sample of $n$ values is compared with $\mu_0$ through

The denominator $\sigma/\sqrt{n}$ is the standard error of the mean: means of samples vary less than single values. When $H_0$ is true and the values are normally distributed, $z$ has the standard normal distribution $N(0, 1)$, so its p-value comes from the table of $\Phi(z) = P(Z < z)$. When $\sigma$ is unknown and estimated from the sample, a t-test is used instead.

So at $\alpha = 0.05$ a one-sided test rejects $H_0$ when $z \ge 1.645$ (or $z \le -1.645$, for $H_1: \mu < \mu_0$), and a two-sided test when $z \ge 1.96$ or $z \le -1.96$. At $\alpha = 0.01$ the thresholds are $2.326$ and $2.576$.

Example 1: a dog and two boxes

A dog is being trained to find a treat by smell. In each of $10$ trials the treat is hidden at random in one of two boxes, and the dog picks the right box $9$ times. Test at the 5% level whether the dog does better than chance.

Example 2: is the machine short-filling the bags?

A machine should fill bags with a mean of $1000$ g of flour, and the masses have standard deviation $\sigma = 5$ g. An inspector weighs $25$ bags and finds a mean of $\bar{x} = 998$ g. Test at the 5% level whether the machine fills too little on average. Would the decision be the same at the 1% level?

Your turn

Juice bottles are labelled $500$ ml, and the filling machine has standard deviation $\sigma = 6$ ml. A sample of $36$ bottles has mean $\bar{x} = 501.8$ ml. Test at the 5% level whether the mean differs from $500$ ml, in either direction.

Common mistakes

• “The p-value is the probability that H₀ is true.” No: the p-value is worked out assuming $H_0$ is true. It says how surprising the data would be under $H_0$, not how likely $H_0$ is. A p-value of $0.021$ does not mean a 2.1% chance that Maya is guessing.
• “p > α proves H₀.” Not rejecting is not the same as proving. With $13$ of $20$ right ($p = 0.132$) Maya might still have a real but weaker skill - a test of $20$ cups is too short to show it.
• Forgetting “at least as extreme”. The p-value is a whole tail, not one result. With $14$ of $20$ right, $P(X = 14) = 0.037$ looks smaller than $0.05$, but the p-value $P(X \ge 14) = 0.058$ is bigger.
• Choosing α or the side of the test after seeing the data. Both are fixed before the test. Picking them afterwards, or trying one analysis after another until one gives $p \le 0.05$, makes a false alarm more likely.
• “Significant” means “important”. A significant result would be unlikely under $H_0$, but the effect itself may be tiny: in a large enough sample, even a very small difference gives a small p-value. How big the effect is, is a separate question.

In exams and at university

In England, hypothesis testing is part of A-level Mathematics, and it starts in the AS content. Students learn the language of hypothesis testing through a binomial model: null and alternative hypotheses, significance level, test statistic, 1-tail and 2-tail tests, critical value, critical region, acceptance region and p-value. They carry out tests for the proportion in a binomial distribution - like the tasting test on this page - and interpret the result in context. The full A-level adds a test for the mean of a normal distribution with known, given or assumed variance, as in Example 2, and interpreting a correlation coefficient using a given p-value or critical value.

In the United States, AP Statistics has units on inference for proportions and for means, which include testing hypotheses.

At university, hypothesis testing is taught in statistics courses. In the open textbook OpenIntro Statistics, for example, it comes in the chapter “Foundations for inference”, after point estimates and confidence intervals for a proportion, and the chapters that follow use it for proportions, means and other kinds of data.

Check yourself

Flashcards for review

Flip a card and say whether you knew the answer. The ones you did not know come back at the end of the deck. Why it pays to come back to them in a few days is explained in the guide Spaced repetition.