Smooth Rectifier Activation Function

The Smooth Rectifier (commonly known as Softplus) is a differentiable activation function used extensively in neural networks as a smooth approximation of the Rectified Linear Unit (ReLU). Unlike ReLU, which has a sharp corner at zero, the smooth rectifier provides a continuously differentiable curve that enables more stable gradient flow during backpropagation.

Mathematical Definition

The smooth rectifier function is defined mathematically as:

$$f(x) = \ln(1 + e^x)$$

This function elegantly combines the exponential and natural logarithm to create a smooth, monotonically increasing curve that:

• Approaches zero as x → -∞ (but never reaches it)
• Approaches x as x → +∞ (asymptotically linear for large positive values)
• Has a continuous derivative everywhere: f'(x) = σ(x), where σ is the sigmoid function

Numerical Stability Considerations

When implementing this function, you must handle numerical overflow and underflow carefully:

1. For large positive x (e.g., x > 20): The exponential e^x becomes astronomically large, potentially causing overflow. However, ln(1 + e^x) ≈ x for large x, so the function can be simplified.

2. For large negative x (e.g., x < -20): The exponential e^x becomes vanishingly small, and ln(1 + e^x) ≈ e^x, which is numerically stable.

A numerically stable implementation uses:

• For x > threshold: return x
• For x ≤ threshold: return log(1 + exp(x))

Your Task

Implement the smooth rectifier activation function that:

• Takes a single numeric input x
• Returns the smooth rectifier value f(x) = ln(1 + e^x)
• Handles edge cases to prevent numerical overflow or underflow
• Returns the result rounded to 4 decimal places