Exponential Linear Activation Function

The Exponential Linear Unit (ELU) is an advanced activation function designed to address several limitations found in traditional rectified activation functions. While standard rectified activations completely zero out negative inputs, ELU provides a smooth, non-zero response for negative values, which helps neural networks learn more effectively.

Mathematical Definition

The ELU activation function is defined piecewise as follows:

$$f(x) = \begin{cases} x & \text{if } x > 0 \ \alpha \cdot (e^x - 1) & \text{if } x \leq 0 \end{cases}$$

Where:

• x is the input value to the activation function
• α (alpha) is a positive hyperparameter that controls the saturation value for negative inputs (commonly set to 1.0)
• e is Euler's number (approximately 2.71828)

Key Characteristics

1. Smooth Negative Response: Unlike rectified linear activations that produce hard zeros for negative inputs, ELU outputs a smooth exponential curve that asymptotically approaches -α as x goes to negative infinity. This provides gradient flow even for negative inputs.

2. Zero-Centered Outputs: For negative inputs, ELU produces negative outputs, making the activations more mean-centered around zero. This property can accelerate the learning process by reducing the bias shift effect.

3. Continuous and Differentiable: The function is continuous everywhere and nearly differentiable (except for a single point at x=0 where it is still continuous). The derivative smoothly transitions between the positive and negative regions.

4. The α Hyperparameter: The alpha parameter determines the minimum value the function can output (approaching -α for very negative inputs). Higher α values allow for larger negative outputs, which can help with the zero-centering effect.

Your Task

Implement a function that computes the ELU activation for a given input value. The function should:

• Return x directly when x is greater than 0
• Return α × (eˣ - 1) when x is less than or equal to 0
• Use α = 1.0 as the default alpha value
• Round the result to 4 decimal places