🎯 Quick Answer

What: Hypothesis testing is the discipline of asking whether a difference you can see in the data is larger than the variation the process produces on its own. It sets the boring explanation (nothing changed) against the interesting one (something did) and returns a p-value: how often chance alone would produce a gap that size.

Why it matters: Two numbers will always differ. Without the test, a first good month gets announced as an improvement, its disappearance in the second month gets blamed on people, and the organization learns nothing either time.

How to apply: Before reading a result, find out how far the output swings when nobody intervenes. Write down what size of difference would count. Then let the test say whether the gap clears the noise, and report the size of the difference beside the p-value.

The payoff: Results that survive the next quarter, negative findings that can be said out loud, and announced gains that people believe.

The pilot came in six minutes better, and the report went up the same week.

One production line, one new changeover procedure, one month. Changeovers had been running around forty-eight minutes; the pilot month averaged forty-two. The crew was congratulated, the procedure was written into the standard work, and the other lines were told to adopt it by the end of the quarter.

The second month came in at fifty-one.

The first good month

The explanation arrived faster than the data had. The crew had stopped following the procedure. Discipline had slipped once the attention moved on. There was a meeting about accountability.

Nobody in that meeting had the one chart that mattered, so it got pulled afterwards: eighteen months of changeover averages, by month, with no procedure change in any of them. They ranged from forty to fifty-five. A six minute move in either direction was the line's ordinary weather. The pilot month had been a good month. The month after it had been a poor one. Neither was outside what the line did on its own when nobody touched it.

The procedure may well have worked. Nobody will ever know, because the first month was believed and the second month was blamed, and between the congratulation and the accountability meeting nobody asked the only question that could have settled it.

I have watched more improvements evaporate in their second month than fail in their first. Most were never improvements. They were a first good month, announced before anyone had asked what a normal month does.

Noise hands out differences for free

Every process moves on its own. Measure the same thing two weeks in a row, change nothing, and you will still get two different numbers. That is not a flaw in the measurement, assuming the measurement itself has been checked. It is what processes do. Sampling, timing, the mix of work that happened to arrive, the person on shift: each contributes a little, and the total is the variation the process produces unaided.

Which means a gap between before and after is not, by itself, evidence of anything. Noise hands out differences for free, all day, and it does not label them. A team that has not measured how far its process swings on its own has no yardstick to hold a result against, and a difference without a yardstick is a claim.

This is the gap that hypothesis testing exists to measure, and it is the step that getting the data in the first place does not cover. Data before opinion gets you two honest numbers. It does not tell you whether the gap between them means anything.

Hypothesis testing for normal humans

The formal answer is a hypothesis test, and it is a great deal simpler than its vocabulary. Strip the notation away and it asks three questions, in order.

How much does this output vary when nobody intervenes?

Is the gap we are looking at bigger than that?

And if nothing had really changed, how often would chance alone produce a gap this size?

The last answer is the p-value. It is not a verdict on whether the change was a good idea. It is a measure of how surprising the result would be if the boring explanation were true. If gaps this size turn up all the time by chance, you have a story, not a result. If they almost never do, the boring explanation is losing, and the change has earned the right to be called real.

That ordering is the whole trick. In Lean Six Sigma the boring explanation has a name, the null hypothesis, and it goes first by design. The interesting explanation has to earn its way past it. This is not cynicism about improvement. It is the only way to tell a result from a good week.

Two habits that do most of the work

Neither of these needs software, and together they prevent most of the damage.

Say what would convince you before you look. Write the number down: the size of difference that would count as proof, and the size that would count as failure. A threshold chosen after the data arrives will always, somehow, be the one the data just cleared. A threshold chosen before is the cheapest honesty there is, and it ends arguments that would otherwise run for weeks, because nobody has to defend a number they have grown fond of.

Then report the size of the difference beside the p-value. A result can be statistically real and operationally worthless at the same time. With enough data, a trivial gap becomes significant, and a team that stops at "significant" will roll out a change nobody can feel. Real and worth acting on are two different findings, and a sponsor needs both.

What the test is not

A few things hypothesis testing does not do, because each of them gets claimed for it.

It does not prove there is no difference. A p-value above the threshold says the evidence was not enough, which is a different finding and usually a smaller sample. "Not proven" and "disproven" are not the same word.

It does not choose the test for you. That depends on the shape of the data: whether the output is measured on a scale or is pass/fail, how many groups are being compared, and whether the data is roughly normal. Choosing wrong returns a confident answer to a question nobody asked, which is why the choice deserves more care than the calculation.

And it does not rescue a measurement nobody trusts. If two people measuring the same thing would not get the same number, the test is comparing noise with noise, and no p-value can fix that.

What it does is narrow. It tells you whether the thing you saw is bigger than the thing the process does anyway. That narrow answer is the difference between an improvement program whose announced gains are believed and one whose results quietly stop being mentioned.

I write one of these every week, drawn from what actually happens on projects rather than what the methodology says should. You can join the newsletter here.

Frequently asked questions

What is a p-value in plain English?

It is how often chance alone would produce a difference at least as large as the one you observed, if nothing had really changed. A small p-value (below 0.05 by convention) means that kind of gap rarely happens by accident, so the boring explanation is unlikely. It is not the probability that your change worked.

What is the difference between statistically significant and practically important?

Significant means the difference is probably real. Important means it is big enough to act on. With a large enough sample, a difference of a few seconds can be statistically significant and still not worth a meeting. Always report the size of the effect beside the p-value.

How do I know which hypothesis test to use?

Three questions choose it: is the output measured on a scale or pass/fail; how many groups are being compared, or is the input a measurement too; and is the measured data roughly normal. Measured output with two groups is a two-sample t-test, or Mann-Whitney if the data is not normal. Pass/fail counts across categories is a chi-square test. The software runs whichever you pick; the choice is the part that needs judgment.

What does it mean if the p-value is above 0.05?

That the evidence was not strong enough to reject the boring explanation. It does not mean the change had no effect. Usually it means the sample was too small to tell, or the effect is smaller than the noise. Say "not proven," never "disproven."