Notes.md

Notes

• Use Stacked LSTM, if necessary to get good results instead of LSTM.
• LSTM + CNN - CNN on top of LSTM
  • LSTM is used for feature extraction and CNN is used to interpret the features accross the time steps.
• In Keras - Types of predictions:
  • model.predict(X)
  • model.predict_classes(X)
  • model.predict_proba(X)

• Life cycle for LSTM models in Keras

  • Define Network

    • Input must be three dimensional - samples, timesteps, features.
    • Convert 2D dataset into 3D dataset.
    • input_shape argument expects a tuple - (number of timesteps, number of features)
    • Some of the standard activation functions that can be used in the output layer:
      • Regression - 'linear'
      • Binary Classification(2 classes) - 'sigmoid'
      • Multiclass Classification(>2 classes) - 'softmax'
    • Example:
      • model = Sequential()
      • model.add(LSTM(5, input_shape=(2,1)))
      • model.add(Dense(1))
      • model.add(Activation('sigmoid'))

  • Compile Network

    • It transforms the simple sequence of layers that we defined into a highly efficient series of matrix transforms in a format intended to be executed on your GPU or CPU, depending on how Keras is configured.
    • Example:
      • model.compile(optimizer='adam', loss='mean_squared_error')
    • Some of the standard loss functions that can be used in the output layer:
      • Regression - 'mean_squared_error'
      • Binary Classification(2 classes) - 'binary_crossentropy' or cross entropy
      • Multiclass Classification(>2 classes) - 'categorical_crossentropy' or Multiclass Logarithmic Loss
    • Some of the optimization algorithms:
      • Stochastic Gradient Descent - 'sgd'
      • Adam - 'adam'
      • RMSprop - 'rmsprop'
    • Metrics: ['accuracy']

  • Fit Network

    • X, y, batch size and epochs are some of the parameters.
    • Example:
      • model.fit(X, y, batch_size=10, epochs=100, verbose=0)

  • Evaluate Network

    • Evaluating the trained model.
    • loss, accuracy = model.evaluate(X, y, verbose=0)

  • Make Predictions

    • Make predictions on new data
    • predictions = model.predict(X)

• Different types of Sequence Prediction Problems:

  • Sequence Prediction:
    • Given a sequence of values, predicting the next value.
  • Sequence Classification:
    • Predicting a class label for a given input sequence.
  • Sequence Genration:
    • Generating a sequence - Text generation, Music generation.
  • Sequence to Sequence Prediction:
    • Predicts an output sequence, given an input sequence.

• Applications of deep learning for NLP

  • Text Classification
  • Language Modeling
  • Speech Recognition
  • Caption Generation
  • Machine Translation
  • Document Summarization
  • Question Answering

References:

• Machine Learning Mastery
• Life cycle of LSTM models - keras
• Different types of Sequence Prediction Problems
• Applications of deep learning for NLP
• Popular Datasets used for NLP