13: How do you know if an intervention is successful?

Statistical significance: In grossly simplified terms, a statistically significant result is unlikely to have been a fluke. A measure of confidence that any difference between your treatment and control groups was a real difference, and not the consequence of a skewed sample, or simple chance.
Effect size: The effect size is a measure of how much your intervention changed things. It is a quantitative estimation of the magnitude of the experimental effect.

Odds ratio: The odds ratio explains the difference between the treatment and control group outcomes in the metric of the study.
Descriptive statistic: A descriptive statistic tells you about the data, such as the average or the range of values.
Inferential statistics: Inferential statistics help generalize something meaningful about a larger population by inferring from a smaller sample. They can test whether an intervention’s effectiveness was more than just a fluke, and whether the intervention could be generalized to the wider population.

Null hypothesis: The status quo, the business-as-usual case, whatever that is. It is the default position that your intervention did not have an effect. Formally, there is no significant difference between specified populations, any observed difference being due to sampling or experimental error.

Confidence interval: The confidence interval is the range of values on the outcome that most likely contains the actual population value. Confidence intervals incorporate the margin of error for your estimated effect.

Practical significance: The practical significance of an intervention is an assessment of not only whether the effect was unlikely to have occurred by chance, but also that the magnitude of the effect justified the effort and impact of the intervention.

​