THE BASICS
12: How do you design a powerful experiment?
The chapter explains why understanding the concept of experimental power is important for evidence-based policing scholars. The chapter then discusses how to calculate appropriate sample sizes, the meaning of Cohen’s d, and different types of error (type I and II). Boxes explain the winner’s curse, the impact of experimental power on behavior at red traffic lights, and how the story of the boy that cried wolf helps understand one experimental concept.
Glossary terms in this chapter
Experimental power: The power of an experiment is a measure of its capability to detect an effect, assuming there is a real effect. It tells you the likelihood that, if your intervention really did move the needle, the experiment will detect that change.
Winner’s curse: Underpowered research can be vulnerable to the winner’s curse by only highlighting unusually strong studies, raising unrealistic expectations that cannot be replicated in subsequent experiments.
Odds ratio: The odds ratio explains the difference between the treatment and control group outcomes in the metric of the study.
Additional information and links
Experimental power
The power of an experiment is a measure of its capability to detect an effect, assuming there is a real effect. It tells you the likelihood that, if your intervention really did move the needle, the experiment will detect that change. The balance that we seek with experimental power is to manage the tradeoff. So that we can have a smaller study that does not include every unit available, we trade the certainty that would bring with a degree of uncertainty by having a smaller study. The challenge is to find the balance of a study that is big enough to detect an effect when there really is an effect, and not so big as to unnecessarily unwieldy. Equally, a study should not be too small, or else we might not notice a real effect when we see it. This page from UCLA goes into it with more depth than I provide.
One component of understanding experimental studies is also remembering that correlation is not causation. One of the best sites to demonstrate this truism makes correlational comparisons between clearly disparate data sources, often with humorous effect.
​
Different type of error
As this article notes, a type I error (false-positive) occurs if an investigator rejects a null hypothesis that is actually true in the population; a type II error (false-negative) occurs if the investigator fails to reject a null hypothesis that is actually false in the population. The articles explains what you have read in the book in greater detail.
The boy who cried wolf is a children's fable from Aesop, an ancient Greek storyteller who lived between 620 and 560 BCE. This story is part of his collection of tales known as "Aesop's Fables," which did not survive in writing but were passed down by people retelling them. The moral of the fable is that “A liar will not be believed, even when he speaks the truth.”
​
Right turn on red
An article titled "The effect of right-turn-on-red on pedestrian and bicyclist accidents" summarized the impact of loosening regulations around right-turn-on-red in the 1970s, showing accident increases of 40% for pedestrians and 82% for bicycles in New York State; 107% for pedestrians and 72% for bicycles in Wisconsin; 57% for pedestrians and 80% for bicycles in Ohio; and 82% for pedestrians in New Orleans.
In light of some of the concerns around RTOR, Berkeley, California, is planning to prevent right-turn-on-red at the majority of the city's junctions, according to the Los Angeles Times.
​
Calculating a powerful sample size
A tutorial guide to using this sample size calculator will be available soon. There is also a simple calculator at this site, which is intuitive to use.
Related Reducing Crime podcast episode
#42 (Justin Nix)
Dr. Justin Nix is a distinguished associate professor of Criminology and Criminal Justice at the University of Nebraska Omaha, and this year's Outstanding Young Experimental Criminologist. We chat about his research on procedural justice, police legitimacy, and the use of deadly force.
​
​
​
​