Power
Power is the chance that a study finds an effect that’s really there. The plot shows two worlds: what H₀ predicts, and what would happen if the effect were real. Move the sliders and watch the two curves, the cutoff and the power change.
In class, we use only the first design, one mean against a fixed value. The other two designs compare a treatment group with a comparison group, as a randomized study does: one randomizes students, and the other randomizes whole schools. They add settings we don’t cover, such as the share treated, controls and the intraclass correlation, and the tool also gives the minimum detectable effect (MDE). They’re here if you’re curious, and you won’t be tested on them.
Open the simulator in a full window
Things to try
Each link opens the simulator ready to go. Before you press a button, predict what will happen.
What gives a study power?
- Four levers. Start at 80% power, then change alpha, the test, the sample size and the effect size one at a time. Which one moves power the most?
- When there is no effect. Set d to 0. H₀ is true here, so the only possible mistake is a Type I error.
Beyond EDUC 15: comparing two groups
- Two groups. 400 students, half of them treated, and a true effect of 0.2 SD. Power is 52%. Find the number of students that gives 80% power. Then go back to 400 and raise the variance explained by controls to 50%. What happens to power?
- A small effect. Many education programs have effects of about 0.1 SD. How much power do 400 students give? How many students would a study need for 80% power?
- Schools instead of students. 396 schools of 40 students, and a true effect of 0.12 SD. Power is 80%. Now test 80 students per school instead of 40. How much does power rise? Then halve the number of schools instead.
Credit
This simulator was written for EDUC 15. The two-group designs come from the version I adapted for EDUC 265. This view was inspired by the statistical power visualization by Kristoffer Magnusson at R Psychologist.