Experiment Detection Power Analysis (Medium) — Practice with Code Visualizer

When designing controlled experiments, one of the most critical yet often overlooked aspects is detection power—the probability that your experiment will successfully identify a real effect when one truly exists. Without adequate power, you risk conducting experiments that fail to detect meaningful differences, wasting resources and potentially making incorrect business decisions.

Understanding Detection Power:

Detection power quantifies the sensitivity of a hypothesis test. It answers the question: "If there is a real difference between my control and treatment groups, what is the probability that my experiment will find it?" A power of 0.80 means there's an 80% chance of detecting a true effect—a commonly accepted threshold in experimental design.

The Core Components:

Detection power depends on several interconnected factors:

Effect Magnitude (Cohen's d): A standardized measure of the difference between groups, calculated as the mean difference divided by the pooled standard deviation:
- Small effect: d ≈ 0.2
- Medium effect: d ≈ 0.5
- Large effect: d ≈ 0.8
Sample Size Per Group: Larger samples increase power by reducing the influence of random variation.
Significance Threshold (α): The probability of falsely rejecting the null hypothesis (Type I error). Lower thresholds require stronger evidence but reduce power.
Test Directionality: One-tailed tests have higher power for detecting effects in a specific direction, while two-tailed tests are more conservative but detect effects in either direction.

Mathematical Framework:

For a two-sample z-test, the detection power is computed using the following approach:

Calculate the non-centrality parameter (δ): $$\delta = d \cdot \sqrt{\frac{n}{2}}$$

where d is Cohen's d and n is the sample size per group.

Determine the critical z-value (z_crit) based on the significance threshold:
- For two-tailed: z_crit corresponds to α/2 in each tail
- For one-tailed: z_crit corresponds to α in one tail
Compute the power using the standard normal cumulative distribution function (Φ):
- Two-tailed: Power = 1 - Φ(z_crit - δ) + Φ(-z_crit - δ)
- One-tailed: Power = 1 - Φ(z_crit - δ)

Standard Normal CDF Calculation:

The cumulative distribution function for the standard normal distribution can be computed using the error function: $$\Phi(x) = \frac{1}{2} \left(1 + \text{erf}\left(\frac{x}{\sqrt{2}}\right)\right)$$

Your Task:

Implement a function that computes the detection power for a two-sample hypothesis test. Your function should:

Accept the effect magnitude (Cohen's d), sample size per group, significance threshold, and test directionality
Calculate the non-centrality parameter and critical values
Return the detection power rounded to 4 decimal places

This computation is essential for sample size planning, ensuring experiments are adequately powered before they begin.

With a medium effect size (Cohen's d = 0.5) and 64 participants per group:

Non-centrality parameter: δ = 0.5 × √(64/2) = 0.5 × 5.657 ≈ 2.83
Critical z-value for α = 0.05 (two-tailed): z_crit = 1.96
Power = 1 - Φ(1.96 - 2.83) + Φ(-1.96 - 2.83) = 1 - Φ(-0.87) + Φ(-4.79) = 1 - 0.1922 + 0.0000 ≈ 0.8074

This indicates an ~80.7% probability of detecting the effect, which meets the conventional 80% power threshold.

With a large effect size (d = 0.8), smaller samples (n = 25), and a one-tailed test:

Non-centrality parameter: δ = 0.8 × √(25/2) = 0.8 × 3.536 ≈ 2.83
Critical z-value for α = 0.05 (one-tailed): z_crit = 1.645
Power = 1 - Φ(1.645 - 2.83) = 1 - Φ(-1.185) ≈ 0.8817

The one-tailed test combined with the large effect size achieves ~88.2% power despite the smaller sample size.

A small effect size (d = 0.2) with 100 participants per group:

Non-centrality parameter: δ = 0.2 × √(100/2) = 0.2 × 7.071 ≈ 1.41
Critical z-value: z_crit = 1.96
Power = 1 - Φ(1.96 - 1.41) + Φ(-1.96 - 1.41) = 1 - Φ(0.55) + Φ(-3.37) ≈ 0.293

With only ~29.3% power, this experiment is severely underpowered. To detect a small effect with 80% power, you would need approximately 400 participants per group.