Merge pull request #4269 from Shantnu-singh/main

added Adam optimizer

Author: ajay-dhangar

docs/Deep Learning/Optimizers in Deep Learning/Adam.md

@@ -0,0 +1,116 @@
+# Add Adam in Deep Learning Optimizers
+
+This Section contains an explanation and implementation of the Adam optimization algorithm used in deep learning. Adam (Adaptive Moment Estimation) is a popular optimizer that combines the benefits of two other widely used methods: AdaGrad and RMSProp.
+
+## Table of Contents
+- [Introduction](#introduction)
+- [Mathematical Explanation](#mathematical-explanation)
+  - [Adam in Gradient Descent](#adam-in-gradient-descent)
+  - [Update Rule](#update-rule)
+- [Implementation in Keras](#implementation-in-keras)
+- [Results](#results)
+- [Advantages of Adam](#advantages-of-adam)
+- [Limitations of Adam](#limitations-of-adam)
+
+
+## Introduction
+
+Adam is an optimization algorithm that computes adaptive learning rates for each parameter. It combines the advantages of the AdaGrad and RMSProp algorithms by using estimates of the first and second moments of the gradients. Adam is widely used in deep learning due to its efficiency and effectiveness.
+
+## Mathematical Explanation
+
+### Adam in Gradient Descent
+
+Adam optimizes the stochastic gradient descent by calculating individual adaptive learning rates for each parameter based on the first and second moments of the gradients.
+
+### Update Rule
+
+The update rule for Adam is as follows:
+
+1. Compute the first moment estimate (mean of gradients):
+
+$$
+m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t
+$$
+
+2. Compute the second moment estimate (uncentered variance of gradients):
+
+$$
+v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2
+$$
+
+3. Correct the bias for the first moment estimate:
+
+$$
+\hat{m}_t = \frac{m_t}{1 - \beta_1^t}
+$$
+
+4. Correct the bias for the second moment estimate:
+
+$$
+\hat{v}_t = \frac{v_t}{1 - \beta_2^t}
+$$
+
+5. Update the parameters:
+
+$$
+\theta_t = \theta_{t-1} - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t
+$$
+
+where:
+- $\theta$ are the model parameters
+- $\eta$ is the learning rate
+- $\beta_1$ and $\beta_2$ are the exponential decay rates for the moment estimates
+- $\epsilon$ is a small constant to prevent division by zero
+- $g_t$ is the gradient at time step $t$
+
+## Implementation in Keras
+
+Simple implementation of the Adam optimizer using Keras:
+
+```python
+import numpy as np
+from keras.models import Sequential
+from keras.layers import Dense
+from keras.optimizers import Adam
+
+# Generate data
+X_train = np.random.rand(1000, 20)
+y_train = np.random.randint(2, size=(1000, 1))
+
+# Define a model
+model = Sequential()
+model.add(Dense(64, activation='relu', input_dim=20))
+model.add(Dense(1, activation='sigmoid'))
+
+# Compile the model with Adam optimizer
+optimizer = Adam(learning_rate=0.001)
+model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])
+
+# Train the model
+model.fit(X_train, y_train, epochs=50, batch_size=32)
+```
+
+In this example:
+- We generate some dummy data for training.
+- We define a simple neural network model with one hidden layer.
+- We compile the model using the Adam optimizer with a learning rate of 0.001.
+- We train the model for 50 epochs with a batch size of 32.
+
+
+## Results
+
+The results of the training process, including the loss and accuracy, will be displayed after each epoch. You can adjust the learning rate and other hyperparameters to see how they affect the training process.
+
+## Advantages of Adam
+
+1. **Adaptive Learning Rates**: Adam computes adaptive learning rates for each parameter, which helps in faster convergence.
+2. **Momentum**: Adam includes momentum, which helps in smoothing the optimization path and avoiding local minima.
+3. **Bias Correction**: Adam includes bias correction, improving convergence in the early stages of training.
+4. **Robustness**: Adam works well in practice for a wide range of problems, including those with noisy gradients or sparse data.
+
+## Limitations of Adam
+
+1. **Hyperparameter Sensitivity**: The performance of Adam is sensitive to the choice of hyperparameters ($\beta_1$, $\beta_2$, $\eta$), which may require careful tuning.
+2. **Memory Usage**: Adam requires additional memory to store the first and second moments, which can be significant for large models.
+3. **Generalization**: Models trained with Adam might not generalize as well as those trained with simpler optimizers like SGD in certain cases.