In recent years, social and personality psychology research has faced growing criticism, particularly due to small sample sizes. This has underscored the importance of statistical power analysis—a mathematical method used to determine whether a study has a sufficient number of participants to reliably detect a psychological effect of a given magnitude. In essence, the higher the power of a study, the more likely it is to detect a true effect. This paper outlines four key debates surrounding power analysis and sample size determination:
1. Debate: What Effect Size Should Be Used in Power Analysis?
- The Problem: In new studies, predicting the expected effect size is often challenging. Researchers frequently rely on Cohen’s conventional benchmarks (small, medium, large), but these categories are subjective and not universally applicable.
- Issues in the Literature: Published effect sizes are often inflated due to publication bias—studies with large and statistically significant effects are more likely to be published. Comparing effect sizes across different topics or methods is also problematic. Even pilot studies can be biased.
- Challenges with the “Smallest Effect of Interest”: Determining the smallest effect that is scientifically or practically meaningful is difficult. A seemingly small effect can have substantial real-world implications (e.g., aspirin reducing heart attack risk), although such examples are often misinterpreted. Weak theories that only predict the direction of an effect may still consider small effects scientifically valuable, but detecting such effects requires large resources.
- Proposed Solution – Practically Significant Effect Sizes:
- Applied Practical Effect Size: Focuses on cost-effectiveness or real-world utility. For instance, if a charity spends $1,000 a year on fundraising, even a 1% increase in donations might be worthwhile.
- Basic Practical Effect Size: In basic research, the threshold could be defined by whether the effect can feasibly be replicated or extended with typical lab resources. Detecting very small effects (e.g., d = 0.05) with 80% power might require nearly 10,000 participants—unrealistic for most labs.
2. Debate: Is Power Still Useful After a Study is Completed?
- Misuse: A common mistake is calculating post-hoc power using the observed effect size. This practice is often meaningless because power is directly tied to the observed p-value and provides no new information. If a result is non-significant, the post-hoc power will typically hover around 50%, which offers little insight.
- Useful Applications:
- Sensitivity Analysis: Even after a study is complete, it is helpful to calculate the minimum detectable effect size given the sample size and desired power level (e.g., 80%). This reveals what effect sizes the study could have reasonably detected.
- Detecting Selective Reporting: When a series of studies all yield significant results, computing power based on observed effects can reveal whether the pattern is statistically plausible—possibly exposing selective reporting or analysis manipulation.
- Evaluating the Credibility of Significant Results: Low-powered studies are more likely to produce false positives. A significant result in a low-powered study may be less reliable than it appears.
3. Debate: Do Power Criteria Unfairly Penalize Certain Research Designs?
- Issue: Strict power standards can disadvantage studies involving hard-to-reach or diverse populations (e.g., research on underrepresented ethnic subgroups), which often require smaller samples due to resource constraints. This may further marginalize these groups in scientific literature.
- Solutions:
- Focus on detecting larger and more impactful effects.
- Use methods that enhance power with the same sample size (e.g., stronger manipulations, reliable measures, within-subject designs).
- Collaborate across labs to pool data.
- Share unpublished data to reduce publication bias.
- Consider qualitative or descriptive quantitative approaches when inference is not feasible.
- Editors and reviewers should assess whether studies had adequate power to detect key effects and allow cautious interpretations when appropriate.
4. Debate: Should 80% Power Still Be the Universal Standard?
- Origin: The 80% threshold stems from Cohen’s heuristic suggestion that, absent better information, Type I errors (false positives) should be four times more costly than Type II errors (missed effects), corresponding to 5% and 20% error rates, respectively.
- Support: For many designs, 80% strikes a cost-benefit balance—beyond this point, increasing power requires disproportionately more participants.
- Criticism: There is no universal threshold linking sample size to scientific value, and key inputs to power calculations (like standard deviations) can be uncertain.
- Recommendation: 80% should be seen as a bare minimum, not a gold standard. Researchers should weigh the relative costs of false positives and false negatives. In high-stakes research (e.g., medical diagnostics), 80% power may be inadequate; a higher threshold (e.g., 90%) may be warranted, especially in pre-registered studies.
Alternatives to Traditional Power Analysis: The paper proposes two complementary approaches:
- Accuracy in Parameter Estimation (AIPE): Instead of testing whether an effect exists, this method aims to estimate its size precisely. It focuses on narrowing the confidence interval (CI) around the effect size, thereby avoiding the need to guess the expected effect size.
- Sequential Analysis (Optional Stopping): Allows researchers to collect data in stages and analyze results at intervals. If effects are stronger or weaker than expected, data collection can stop early, conserving resources. However, these designs require special statistical corrections to avoid inflated false positive rates. Additionally, early stopping can lead to inflated effect size estimates.
Conclusion: Rather than asking “Does this study have enough power?”, researchers should instead ask “What effects can this study reasonably detect, and what do those thresholds imply?” This shifts the focus from rigid p-value thresholds to meaningful effect sizes and practical significance. Researchers should report and justify their sample size decisions transparently and contextually.
https://journals.sagepub.com/doi/epub/10.1177/10888683241228328
Reference: Giner-Sorolla, R., Montoya, A. K., Reifman, A., Carpenter, T., Lewis Jr, N. A., Aberson, C. L., … & Soderberg, C. (2024). Power to detect what? Considerations for planning and evaluating sample size. Personality and Social Psychology Review, 28(3), 276-301.
Podcast Link: https://notebooklm.google.com/notebook/fed863bb-d1a8-4994-b75a-3e48786632ef/audio