Added video processing section (Unit 7 - RNN Based Video Model)

chapters/en/_toctree.yml

@@ -126,6 +126,8 @@
     local: "unit7/video-processing/video-processing-basics"
   - title: Overview of the previous SOTA models
     local: "unit7/video-processing/overview-of-previous-sota-models"
+  - title: RNN Based Video Model
+    local: "unit7/video-processing/rnn-based-video-models"
 - title: Unit 8 - 3D Vision, Scene Rendering and Reconstruction
   sections:
   - title: Introduction

chapters/en/unit0/welcome/welcome.mdx

@@ -126,7 +126,7 @@ Our goal was to create a computer vision course that is beginner-friendly and th
 **Unit 7 - Video and Video Processing**
 
 - Reviewers: [Ameed Taylor](https://github.com/atayloraerospace), [Isabella Bicalho-Frazeto](https://github.com/bellabf)
-- Writers: [Diwakar Basnet](https://github.com/DiwakarBasnet), [Chulhwa Han](https://github.com/cjfghk5697)
+- Writers: [Diwakar Basnet](https://github.com/DiwakarBasnet), [Chulhwa Han](https://github.com/cjfghk5697), [Woojun Jung](https://github.com/jungnerd)
 
 **Unit 8 - 3D Vision, Scene Rendering, and Reconstruction**
 

chapters/en/unit7/video-processing/rnn-based-video-models.mdx

@@ -0,0 +1,109 @@
+# Introduction[[introduction]]
+
+## Videos as Sequence Data[[video-as-sequence-data]]
+
+Videos are made up of a series of images called frames that are played one after another to create motion. Each frame captures spatial information—the objects and scenes in the image. When these frames are shown in sequence, they also provide temporal information—how things change and move over time. 
+Because of this combination of space and time, videos contain more complex information than single images. To analyze videos effectively, we need models that can understand both the spatial and temporal aspects.
+
+## The Role and Need for RNNs in Video Processing[[the-role-and-need-for-rnns-in-video-processing]]
+<div class="flex justify-center">
+    <img src="https://huggingface.co/datasets/hf-vision/course-assets/resolve/main/rnn_video_models/rnn.png" alt="RNN architecture">
+</div>
+Convolutional Neural Networks (CNNs) are excellent at analyzing spatial features in images. 
+However, they aren't designed to handle sequences where temporal relationships matter. This is where Recurrent Neural Networks (RNNs) come in. 
+RNNs are specialized for processing sequential data because they have a "memory" that captures information from previous steps. This makes them well-suited for understanding how video frames relate to each other over time.
+
+## Understanding Spatio-Temporal Modeling[[understanding-spatiotemporal-modeling]]
+
+In video analysis, it's important to consider both spatial (space) and temporal (time) features together—this is called spatio-temporal modeling. Spatial modeling looks at what's in each frame, like objects or people, while temporal modeling looks at how these things change from frame to frame. 
+By combining these two, we can understand the full context of a video. Techniques like combining CNNs and RNNs or using special types of convolutions that capture both space and time are ways researchers achieve this.
+
+# RNN-Based Video Modeling Architectures[[rnn-based-video-modeling-architectures]]
+
+## Long-term Recurrent Convolutional Networks(LRCN)[[lrcn]]
+
+<div class="flex justify-center">
+    <img src="https://huggingface.co/datasets/hf-vision/course-assets/resolve/main/rnn_video_models/lrcn.png" alt="LRCN model architecture">
+</div>
+
+**Overview**
+Long-term Recurrent Convolutional Networks (LRCN) are models introduced by researchers Donahue et al. in 2015. 
+They combine CNNs and Long Short-Term Memory networks (LSTMs), a type of RNN, to learn from both the spatial and temporal features in videos. 
+The CNN processes each frame to extract spatial features, and the LSTM takes these features in sequence to learn how they change over time.
+
+**Key Features**
+- **Combining CNN and LSTM:** Spatial features from each frame are fed into the LSTM to model the temporal relationships.
+- **Versatile Applications:** LRCNs have been used successfully in tasks like action recognition (identifying actions in videos) and video captioning (generating descriptions of videos).
+
+**Why It Matters**
+LRCN was one of the first models to effectively handle both spatial and temporal aspects of video data. It paved the way for future research by showing that combining CNNs and RNNs can be powerful for video analysis.
+
+## Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting(ConvLSTM)[[convlstm]]
+
+<div class="flex justify-center">
+    <img src="https://huggingface.co/datasets/hf-vision/course-assets/resolve/main/rnn_video_models/convlstm.png" alt="ConvLSTM model architecture">
+</div>
+
+**Overview**
+
+The Convolutional LSTM Network (ConvLSTM) was proposed by Shi et al. in 2015. It modifies the traditional LSTM by incorporating convolutional operations within the LSTM's structure. This means that instead of processing one-dimensional sequences, ConvLSTM can handle two-dimensional spatial data (like images) over time.
+
+**Key Features**
+- **Spatial Structure Preservation:** By using convolutions, ConvLSTM maintains the spatial layout of the data while processing temporal sequences.
+- **Effective for Spatio-Temporal Prediction:** It's particularly useful for tasks that require predicting how spatial data changes over time, such as weather forecasting or video frame prediction.
+
+**Why It Matters**
+ConvLSTM introduced a new way to process spatio-temporal data by integrating convolution directly into the LSTM architecture. This has been influential in fields that need to predict future states based on spatial and temporal patterns.
+
+## Unsupervised Learning of Video Representations using LSTMs[[unsupervised-lstms]]
+
+**Overview**
+In 2015, Srivastava et al. introduced a method for learning video representations without labeled data, known as unsupervised learning. This paper utilizes a multi-layer LSTM model to learn video representations. The model consists of two main components: an Encoder LSTM and a Decoder LSTM. The Encoder maps video sequences of arbitrary length (in the time dimension) to a fixed-size representation. The Decoder then uses this representation to either reconstruct the input video sequence or predict the subsequent video sequence.
+
+**Key Features**
+- **Unsupervised Learning:** The model doesn't require labeled data, making it easier to work with large amounts of video.
+
+**Why It Matters**
+This approach showed that it's possible to learn useful video representations without the need for extensive labeling, which is time-consuming and expensive. It opened up new possibilities for video analysis and generation using unsupervised methods.
+
+## Describing Videos by Exploiting Temporal Structure[[temporal-structure]]
+
+<div class="flex justify-center">
+    <img src="https://huggingface.co/datasets/hf-vision/course-assets/resolve/main/rnn_video_models/attention_model.png">
+</div>
+
+**Overview**
+In 2015, Yao et al. introduced attention mechanisms in video models, specifically for video captioning tasks. This approach leverages attention to selectively focus on important temporal and spatial features within the video, allowing the model to generate more accurate and contextually relevant descriptions.
+
+**Key Features**
+- **Temporal and Spatial Attention:** The attention mechanism dynamically identifies the most relevant frames and regions in a video, ensuring that both local actions (e.g., specific movements) and global context (e.g., overall activity) are considered.
+- **Enhanced Representation:** By focusing on significant features, the model combines local and global temporal structures, leading to improved video representations and more precise caption generation.
+
+**Why It Matters**
+Incorporating attention mechanisms into video models has transformed how temporal data is processed. This method enhances the model’s capacity to handle the complex interactions in video sequences, making it an essential component in modern neural network architectures for video analysis and generation.
+
+
+# Limitations of RNN-Based Models [[limitations]]
+- **Challenges with Long-Term Dependencies**
+
+    RNNs, including LSTMs, can struggle to maintain information over long sequences. This means they might "forget" important details from earlier frames when processing long videos. This limitation can affect the model's ability to understand the full context of a video.
+
+- **Computational Complexity and Processing Time**
+
+    Because RNNs process data sequentially—one step at a time—they can be slow, especially with long sequences like videos. This sequential processing makes it difficult to take advantage of parallel computing resources, leading to longer training and inference times.
+
+- **Emergence of Alternative Models**
+
+    Newer models like Transformers have been developed to address some of the limitations of RNNs. Transformers use attention mechanisms to handle sequences and can process data in parallel, making them faster and more effective at capturing long-term dependencies.
+
+# Conclusion[[conclusion]]
+
+RNN-based models have significantly advanced the field of video analysis by providing tools to handle temporal sequences effectively. Models like LRCN, ConvLSTM, and those incorporating attention mechanisms have demonstrated the potential of combining spatial and temporal processing. However, limitations such as difficulty with long sequences, computational inefficiency, and high data requirements highlight the need for continued innovation.
+
+Future research is likely to focus on overcoming these challenges, possibly by adopting newer architectures like Transformers, improving training efficiency, and enhancing model interpretability. These efforts aim to create models that are both powerful and practical for real-world video applications.
+
+### References[[references]]
+1. [Long-term Recurrent Convolutional Networks paper](https://arxiv.org/pdf/1411.4389)
+2. [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting paper](https://proceedings.neurips.cc/paper_files/paper/2015/file/07563a3fe3bbe7e3ba84431ad9d055af-Paper.pdf)
+3. [Unsupervised Learning of Video Representations using LSTMs paper](https://arxiv.org/pdf/1502.04681)
+4. [Describing Videos by Exploiting Temporal Structure paper](https://arxiv.org/pdf/1502.08029)