Vanishing and Exploding Gradients

Theoretical Background

In deep neural networks, the computation of gradients during backpropagation is central to optimizing the network. However, as the depth of the network increases, gradients can either diminish (vanishing) or amplify (exploding). These issues arise due to the repeated multiplication of small or large derivative values, respectively.

• Vanishing Gradients: This problem is pronounced in networks using sigmoid or tanh activation functions. The derivatives of these functions are less than one, leading to progressively smaller gradients as backpropagation proceeds through the network layers.
• Exploding Gradients: Conversely, large derivative values can cause gradients to grow exponentially, which can lead to numerical instability and divergent updates.

Practical Applications

In practical terms, vanishing gradients can prevent lower layers in a network from learning effectively, as they receive very little signal to update weights. Exploding gradients can cause large weight updates, leading to model instability and poor convergence.

Mitigation Strategies

1. Weight Initialization:

   • Xavier/Glorot Initialization: Suitable for sigmoid or tanh activations.
   • He Initialization: Designed for ReLU activations.

2. Normalization Techniques:

   • Batch Normalization: By normalizing the inputs to each layer, it helps maintain a healthy gradient flow.

3. Gradient Clipping:

   • By capping the gradients, it stabilizes the training process, especially useful in recurrent neural networks (RNNs) where gradient issues are more severe.

4. Activation Functions:

   • ReLU and Variants: ReLU activation function (and its variants like Leaky ReLU) keeps gradients from vanishing by providing a linear path for gradients.

Code Example

import torch
import torch.nn as nn
import torch.optim as optim

# Example of using weight initialization for a neural network
class SampleNN(nn.Module):
    def __init__(self):
        super(SampleNN, self).__init__()
        self.fc1 = nn.Linear(784, 256)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(256, 10)
        
    def forward(self, x):
        x = self.relu(self.fc1(x))
        x = self.fc2(x)
        return x

model = SampleNN()

# Using He initialization
nn.init.kaiming_normal_(model.fc1.weight, nonlinearity='relu')

External References

• Understanding the Vanishing Gradient Problem
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Diagrams

graph LR
A[Input Layer] --> B[Hidden Layer 1]
B --> C[Hidden Layer 2]
C --> D[Hidden Layer 3]
D --> E[Output Layer]

subgraph Vanishing Gradients
B -->|Small Gradients| C
C -->|Even Smaller Gradients| D
end

subgraph Exploding Gradients
B -.->|Large Gradients| C
C -.->|Larger Gradients| D
end

In this diagram, the flow of gradients during backpropagation is illustrated, showing how gradients can diminish or explode as they propagate through layers. This visualization helps in understanding why these issues occur in deeper networks.