Compare and contrast different activation functions

In neural networks, activation functions introduce non-linearity, allowing the network to learn complex patterns. Here's a detailed comparison:

• ReLU (Rectified Linear Unit): Defined as $f(x) = ext{max}(0, x)$, it is computationally efficient and helps mitigate the vanishing gradient problem. However, it can lead to some neurons permanently outputting zero (dying ReLU).

• Sigmoid: Maps any input to a value between 0 and 1. It's often used in the output layer for binary classification. The formula is $f(x) = \frac{1}{1 + e^{-x}}$. The major drawback is the vanishing gradient problem, which can slow down learning.

• Tanh: Similar to sigmoid but outputs range from -1 to 1, which can center the data and often leads to better convergence. The function is $f(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$. It also suffers from vanishing gradients but to a lesser extent than sigmoid.

• Leaky ReLU: An extension of ReLU that allows a small, non-zero gradient when the unit is not active, defined as $f(x) = x ext{ if } x > 0 ext{ else } ext{alpha} imes x$, where alpha is a small constant.

• Swish: A newer activation function defined as $f(x) = x imes ext{sigmoid}(x)$. It has been found to outperform ReLU in some deep learning models.

Here's a diagram comparing these functions:

graph LR
    A[Input] --> B[ReLU]
    A --> C[Sigmoid]
    A --> D[Tanh]
    A --> E[Leaky ReLU]
    A --> F[Swish]

For practical applications, choose ReLU for deep networks, sigmoid for binary outputs, tanh for centering data, and Leaky ReLU or Swish when dealing with dying units or when requiring smoother transitions. For further reading, explore activation functions in deep learning.