E2026002                                           2026-01-23  

Shiyao Liu   Teppei Yamamoto

Abstract
Political scientists routinely use power analysis when designing their empirical research. However, it is often neglected that power analysis relies on untested assumptions about the true values of key parameters, such as the effect size. Researchers commonly use auxiliary empirical information to make guesses about those parameters, such as results from a pilot study or a similar experiment reported in the literature. In this paper, we show that such practice is problematic due to neglected uncertainties in the empirically obtained parameter values. We propose a conceptual distinction between empirical and non-empirical power analyses and analyze the former as an estimation problem, investigating their statistical properties both analytically and via simulations. Our results indicate that estimators for power and minimum required sample size tend to perform poorly under scenarios resembling typical political science applications. We offer practical guidelines for empirical researchers on when to (and not to) trust power analysis results.

Keywords : power analysis, sample size, experimental design, research transparency.