101
0/318
Loading content...
When designing controlled experiments, one of the most critical yet often overlooked aspects is detection power—the probability that your experiment will successfully identify a real effect when one truly exists. Without adequate power, you risk conducting experiments that fail to detect meaningful differences, wasting resources and potentially making incorrect business decisions.
Understanding Detection Power:
Detection power quantifies the sensitivity of a hypothesis test. It answers the question: "If there is a real difference between my control and treatment groups, what is the probability that my experiment will find it?" A power of 0.80 means there's an 80% chance of detecting a true effect—a commonly accepted threshold in experimental design.
The Core Components:
Detection power depends on several interconnected factors:
Effect Magnitude (Cohen's d): A standardized measure of the difference between groups, calculated as the mean difference divided by the pooled standard deviation:
Sample Size Per Group: Larger samples increase power by reducing the influence of random variation.
Significance Threshold (α): The probability of falsely rejecting the null hypothesis (Type I error). Lower thresholds require stronger evidence but reduce power.
Test Directionality: One-tailed tests have higher power for detecting effects in a specific direction, while two-tailed tests are more conservative but detect effects in either direction.
Mathematical Framework:
For a two-sample z-test, the detection power is computed using the following approach:
where d is Cohen's d and n is the sample size per group.
Determine the critical z-value (z_crit) based on the significance threshold:
Compute the power using the standard normal cumulative distribution function (Φ):
Standard Normal CDF Calculation:
The cumulative distribution function for the standard normal distribution can be computed using the error function: $$\Phi(x) = \frac{1}{2} \left(1 + \text{erf}\left(\frac{x}{\sqrt{2}}\right)\right)$$
Your Task:
Implement a function that computes the detection power for a two-sample hypothesis test. Your function should:
This computation is essential for sample size planning, ensuring experiments are adequately powered before they begin.
effect_magnitude = 0.5
group_sample_count = 64
significance_threshold = 0.05
bidirectional = True0.8074With a medium effect size (Cohen's d = 0.5) and 64 participants per group:
This indicates an ~80.7% probability of detecting the effect, which meets the conventional 80% power threshold.
effect_magnitude = 0.8
group_sample_count = 25
significance_threshold = 0.05
bidirectional = False0.8817With a large effect size (d = 0.8), smaller samples (n = 25), and a one-tailed test:
The one-tailed test combined with the large effect size achieves ~88.2% power despite the smaller sample size.
effect_magnitude = 0.2
group_sample_count = 100
significance_threshold = 0.05
bidirectional = True0.293A small effect size (d = 0.2) with 100 participants per group:
With only ~29.3% power, this experiment is severely underpowered. To detect a small effect with 80% power, you would need approximately 400 participants per group.
Constraints