Sparse Expert Efficiency Ratio

In modern large-scale neural network architectures, Mixture of Experts (MoE) has emerged as a powerful paradigm for scaling model capacity without proportionally increasing computational cost. The key innovation lies in conditional computation—only a subset of the network's parameters are activated for any given input.

A traditional dense layer engages all neurons for every forward pass, resulting in computational costs that scale linearly with model size. In contrast, a sparse expert architecture partitions the network into multiple independent "expert" subnetworks, activating only a small fraction of them based on each input. A gating network learns to route inputs to the most relevant experts.

The Efficiency Calculation:

For a layer with E total experts and k active experts per input (where k << E), the computational advantage becomes significant:

• Dense Layer FLOPs: Since a dense architecture would need to activate all experts, the floating-point operations scale as: $$\text{FLOPs}{\text{dense}} = E \times d{\text{in}} \times d_{\text{out}}$$

• Sparse Expert FLOPs: With sparse activation, only k experts are used: $$\text{FLOPs}{\text{sparse}} = k \times d{\text{in}} \times d_{\text{out}}$$

• Efficiency Ratio: The percentage of computation saved is: $$\text{Savings} = \frac{\text{FLOPs}{\text{dense}} - \text{FLOPs}{\text{sparse}}}{\text{FLOPs}_{\text{dense}}} \times 100%$$

This simplifies to: $$\text{Savings} = \left(1 - \frac{k}{E}\right) \times 100%$$

Your Task: Write a Python function that computes the computational efficiency gain (in percentage) of using a sparse expert architecture compared to a fully dense layer. The function should return the savings percentage rounded to 1 decimal place.

Note: This calculation demonstrates why MoE architectures can scale to trillions of parameters while maintaining practical inference costs—a principle that has enabled state-of-the-art models like Switch Transformer, GShard, and various LLMs to achieve unprecedented scale.