The Adam algorithm (Adaptive Moment Estimation) is an optimization algorithm commonly used in training deep learning models. It combines ideas from both the Adaptive Gradient Algorithm (AdaGrad) and Root Mean Square Propagation (RMSProp) to provide an adaptive learning rate for each parameter. This results in faster convergence and improved performance on non-stationary problems.

1. Overview of the Adam Algorithm

• Adaptive Learning Rate: Adam adjusts the learning rate for each parameter dynamically based on estimates of the first and second moments of the gradients.
• Momentum: It incorporates momentum by maintaining an exponentially decaying average of past gradients, which helps in smoothing updates and reducing oscillations.

2. How Adam Works

The Adam optimizer uses two moving averages:

• First moment (mean) $m_t$ : Tracks the average of the gradients.
• Second moment (uncentered variance) $v_t$ : Tracks the average of the squared gradients.

Given an objective function $f(\theta)$ where $\theta$ are the parameters, the algorithm updates the parameters using:

1. Initialize:

   • $m_0 = 0$ (first moment vector)
   • $v_0 = 0$ (second moment vector)
   • $t = 0$ (time step)
   • $\alpha$ (learning rate, typically $0.001$ )
   • $\beta_1$ and $\beta_2$ (decay rates for $m_t$ and $v_t$ ; commonly $\beta_1 = 0.9$, $\beta_2 = 0.999$ )
   • $\epsilon$ (small constant for numerical stability, often $10^{-8}$ )

2. For each iteration $t$ :

   • Increment time step: $t = t + 1$
   • Compute gradient $g_t = \nabla_\theta f(\theta_{t-1})$
   • Update biased first moment estimate:

   $m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$

   • Update biased second moment estimate:

   $v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$

   • Bias correction:

     • Correct the bias in the first moment:

     $\hat{m}_t = \frac{m_t}{1 - \beta_1^t}$

     • Correct the bias in the second moment:

     $\hat{v}_t = \frac{v_t}{1 - \beta_2^t}$

   • Parameter update:

   $\theta_t = \theta_{t-1} - \alpha \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$

3. Intuition Behind Adam

• Momentum: The first moment estimate $m_t$ acts as a velocity vector, moving the parameters in a direction that accounts for past gradients, reducing oscillation and making optimization smoother.
• Adaptive Scaling: The second moment estimate $v_t$ adjusts the learning rate for each parameter based on the magnitude of past gradients, preventing the step size from being too large or too small.
• Bias Correction: Early in training, $m_t$ and $v_t$ are biased toward zero. Bias correction compensates for this, ensuring that $\hat{m}_t$ and $\hat{v}_t$ accurately represent their true moments.

4. Advantages of Adam

• Adaptive Learning Rate: Adjusts learning rates for individual parameters, making it robust to changes in gradient magnitude.
• Efficient: Works well for problems with large datasets or high-dimensional parameter spaces.
• Good General Performance: Converges quickly in practice and performs well on non-convex optimization problems commonly found in deep learning.

5. Hyperparameters and Tuning