Hypothesis Testing

A statistical test defines a null hypothesis, an alternative, a statistic and its distribution under the null assumptions. A p-value measures the probability of a result at least as extreme as the observed one under that null model; it is not the probability that the null is true. A decision threshold controls a specified error rate under the test's conditions. Power concerns sensitivity to particular alternatives. Tests can address means, distributions, proportions or model parameters, with parametric and resampling approaches. The method organizes evidence against a model, but its relevance depends on whether the tested hypothesis corresponds to the scientific or operational question.

State the hypothesis and the smallest effect that would matter before selecting a method. Check randomization or sampling assumptions, dependence, sample size and the suitability of the statistic. Account for multiple comparisons and any sequential examination of results. Report estimates and uncertainty alongside the test outcome, distinguishing evidence of a difference from its operational importance. The analysis should also explain whether it was designed to detect a difference, demonstrate equivalence or establish noninferiority, because these require different hypotheses and decision rules.

Rejecting a null does not prove a causal mechanism, and failing to reject it does not establish equivalence. Large samples can detect effects too small to matter. Repeated testing, subgroup searches and outcome switching can make nominal error rates misleading. Test assumptions include more than a distributional check; biased sampling or dependent observations can dominate the conclusion. Separate statistical evidence from the decision cost, and use appropriately designed equivalence or noninferiority procedures when similarity is the actual question.