-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
338 lines (319 loc) · 20.6 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Deep Learning Model Compression Interface</title>
<link rel="stylesheet" href="styles.css">
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.5.2/css/bootstrap.min.css">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/animate.css/4.1.1/animate.min.css"/>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/aos/2.3.4/aos.css"/>
<script src="https://cdn.jsdelivr.net/npm/[email protected]"></script>
<script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script>
<script type="text/javascript" id="MathJax-script" async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
<style>
/* Ensure the nav images are properly sized */
nav img {
height: 40px; /* Adjust the height as needed */
margin-right: 20px;
max-width: 100px; /* Prevents the image from being too wide */
}
nav {
display: flex;
align-items: center;
padding: 10px 20px;
background-color: #ffffff;
border-bottom: 2px solid #ddd;
box-shadow: 0 4px 2px -2px gray;
}
nav a {
margin: 0 20px;
text-decoration: none;
color: #333;
font-weight: bold;
transition: color 0.3s;
position: relative;
}
nav a:hover {
color: #007bff;
}
.content {
margin: 20px;
}
.project-info {
margin-top: 20px;
padding: 20px;
background-color: #ffffff;
border: 1px solid #ddd;
box-shadow: 0 4px 8px rgba(0, 0, 0, 0.1);
}
.project-info h2 {
margin-top: 0;
color: #333;
}
.project-info .names {
display: flex;
justify-content: space-around;
flex-wrap: wrap;
}
.names div {
margin: 10px;
text-align: center;
background-color: #f4f4f9;
padding: 10px;
border-radius: 8px;
box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);
}
.names div p {
margin: 5px 0;
}
.links a {
display: inline-block;
margin-right: 20px;
text-decoration: none;
color: #1a237e;
font-weight: bold;
background-color: #e8eaf6;
padding: 10px 20px;
border-radius: 8px;
transition: background-color 0.3s;
}
.links a:hover {
background-color: #c5cae9;
}
</style>
</head>
<body>
<nav>
<img src="images/IISC_logo.jpeg" alt="IISC Logo">
<img src="images/Shell_logo.png" alt="Shell Logo">
<a href="index.html">Home</a>
<a href="detr.html">DETR</a>
<a href="vgg.html">VGG</a>
<a href="resnet.html">ResNet</a>
<a href="unet.html">UNet</a>
</nav>
<div class="content">
<div class="project-info">
<h2>Project Information</h2>
<div class="names">
<div>
<p>Ashish Ranjan</p>
</div>
<div>
<p>Prof. Pradipta Biswas</p>
</div>
<div>
<p>Prakalp Somawanshi</p>
</div>
</div>
<div class="links">
<a href="#arxiv">[Report]</a>
<a href="https://github.com/ashishranjan0304/DeepLearning_Model_Compression.git">[Code]</a>
<a href="#slides">[Slides]</a>
</div>
</div>
<div class="container my-5 project-info">
<div class="text-center mb-5" data-aos="fade-down">
<h1 class="display-4">CompactNet</h1>
<p class="lead">Optimizing deep learning models compression for efficient deployment</p>
</div>
<div class="row mb-4">
<div class="col-lg-6 mb-4" data-aos="fade-right">
<h2>Project Information</h2>
<p>This project focuses on creating a user-friendly web interface for pruning any deep learning model. The goal is to make model compression accessible and efficient for users, regardless of their expertise in deep learning.</p>
</div>
<div class="col-lg-6" data-aos="fade-left">
<img src="images/output.png" class="img-fluid rounded" alt="Project Information Image">
</div>
</div>
<div class="mb-5" data-aos="zoom-in">
<h2>Introduction</h2>
<div id="typed-introduction"></div>
</div>
<div class="row mb-5">
<div class="col-lg-6 order-lg-2 mb-4" data-aos="fade-left">
<h3>Importance of Model Compression</h3>
<p>Model compression is crucial for deploying deep learning models on resource-constrained devices such as mobile phones and IoT devices. It helps in reducing the latency and improving the efficiency of models, making them suitable for real-time applications.</p>
</div>
<div class="col-lg-6 order-lg-1" data-aos="fade-right">
<img src="images/shell_pic.jpeg" class="img-fluid rounded" alt="Importance Image">
</div>
</div>
<div class="mb-5" data-aos="zoom-in">
<h2>Background</h2>
<h3 data-aos="fade-up">Introduction to Deep Learning</h3>
<p>Deep learning is a subset of machine learning that involves neural networks with many layers. It has been successfully applied to various fields, including image recognition, natural language processing, and autonomous driving.</p>
<h3 data-aos="fade-up">Challenges in Deep Learning</h3>
<p>Despite its success, deep learning models are often large and computationally intensive, making them challenging to deploy on devices with limited resources.</p>
<h3 data-aos="fade-up">Importance of Model Compression</h3>
<p>Model compression techniques address these challenges by reducing the model size and computational requirements. This enables the deployment of deep learning models on a wider range of devices and applications.</p>
</div>
<div class="mb-5" data-aos="zoom-in">
<h2>Types of Model Compression</h2>
<div class="row">
<div class="col-md-4" data-aos="fade-up">
<h3>Pruning</h3>
<p>Pruning involves removing unnecessary neurons or weights in a neural network, thereby reducing the model size and improving computational efficiency.</p>
<img src="images/pruning.jpg" class="img-fluid rounded" alt="Pruning Image">
</div>
<div class="col-md-4" data-aos="fade-up">
<h3>Quantization</h3>
<p>Quantization reduces the precision of the weights and activations of a model, which can significantly decrease the model size and inference time.</p>
<img src="images/quan.png" class="img-fluid rounded" alt="Quantization Image">
</div>
<div class="col-md-4" data-aos="fade-up">
<h3>Knowledge Distillation</h3>
<p>Knowledge distillation involves training a smaller model (student) to replicate the behavior of a larger model (teacher), thus achieving similar performance with fewer parameters.</p>
<img src="images/knowledge_dist.jpg" class="img-fluid rounded" alt="Distillation Image">
</div>
</div>
</div>
<!-- Add the algorithms for each pruning technique with popping steps -->
<div class="mb-5" data-aos="zoom-in">
<h2>Pruning Algorithms</h2>
<!-- L1 Structured Pruning Algorithm -->
<div class="algorithm bg-light p-3 mb-4 rounded" data-aos="fade-up">
<h3 class="text-primary">L1 Structured Filter Pruning</h3>
<p>L1 structured pruning removes filters with the smallest L1 norm, assuming they have the least impact on the model's performance.</p>
<ul class="list-unstyled">
<li data-aos="fade-up" data-aos-delay="100">1. Initialize the model parameters \( W \).</li>
<li data-aos="fade-up" data-aos-delay="200">2. Train the model for the specified number of epochs using the training data \( X \).</li>
<li data-aos="fade-up" data-aos-delay="300">3. Compute L1 norms for each filter \( F_{i,j} \) in each layer \( i \):
<script type="math/tex"> \|F_{i,j}\|_1 = \sum_{k=1}^{N_i} \sum_{m=1}^{K} \sum_{n=1}^{K} |F_{i,j,k,m,n}| </script>
</li>
<li data-aos="fade-up" data-aos-delay="400">4. Sort filters by their L1 norms and select those with the smallest norms to prune.</li>
<li data-aos="fade-up" data-aos-delay="500">5. Zeroize the selected filters.</li>
<li data-aos="fade-up" data-aos-delay="600">6. Extract the pruned model parameters \( W^* \).</li>
</ul>
</div>
<!-- L2 Structured Pruning Algorithm -->
<div class="algorithm bg-light p-3 mb-4 rounded" data-aos="fade-up">
<h3 class="text-primary">L2 Structured Filter Pruning</h3>
<p>L2 structured pruning removes filters with the smallest L2 norm, assuming they have the least impact on the model's performance.</p>
<ul class="list-unstyled">
<li data-aos="fade-up" data-aos-delay="100">1. Initialize the model parameters \( W \).</li>
<li data-aos="fade-up" data-aos-delay="200">2. Train the model for the specified number of epochs using the training data \( X \).</li>
<li data-aos="fade-up" data-aos-delay="300">3. Compute L2 norms for each filter \( F_{i,j} \) in each layer \( i \):
<script type="math/tex"> \|F_{i,j}\|_2 = \sqrt{\sum_{k=1}^{N_i} \sum_{m=1}^{K} \sum_{n=1}^{K} F_{i,j,k,m,n}^2} </script>
</li>
<li data-aos="fade-up" data-aos-delay="400">4. Sort filters by their L2 norms and select those with the smallest norms to prune.</li>
<li data-aos="fade-up" data-aos-delay="500">5. Zeroize the selected filters.</li>
<li data-aos="fade-up" data-aos-delay="600">6. Extract the pruned model parameters \( W^* \).</li>
</ul>
</div>
<!-- FPGM Algorithm -->
<div class="algorithm bg-light p-3 mb-4 rounded" data-aos="fade-up">
<h3 class="text-primary">FPGM (Filter Pruning via Geometric Median)</h3>
<p>FPGM removes filters that are closest to the geometric median, assuming these filters have the least impact on the model's performance.</p>
<ul class="list-unstyled">
<li data-aos="fade-up" data-aos-delay="100">1. Initialize the model parameters \( W \).</li>
<li data-aos="fade-up" data-aos-delay="200">2. Train the model for the specified number of epochs using the training data \( X \).</li>
<li data-aos="fade-up" data-aos-delay="300">3. Compute the geometric median \( x_{GM} \) of filters in each layer \( i \):
<script type="math/tex"> x_{GM} = \arg \min_{x \in \mathbb{R}^{N_i \times K \times K}} \sum_{j=1}^{N_{i+1}} \|x - F_{i,j}\|_2 </script>
</li>
<li data-aos="fade-up" data-aos-delay="400">4. Find and prune filters nearest to the geometric median.</li>
<li data-aos="fade-up" data-aos-delay="500">5. Extract the pruned model parameters \( W^* \).</li>
</ul>
</div>
<!-- SNIP Algorithm -->
<div class="algorithm bg-light p-3 mb-4 rounded" data-aos="fade-up">
<h3 class="text-primary">SNIP (Single-shot Network Pruning based on Connection Sensitivity)</h3>
<p>SNIP prunes connections based on their sensitivity to the loss function, retaining only the most important connections.</p>
<ul class="list-unstyled">
<li data-aos="fade-up" data-aos-delay="100">1. Initialize the network weights \( w \).</li>
<li data-aos="fade-up" data-aos-delay="200">2. Sample a mini-batch of training data \( D_b \).</li>
<li data-aos="fade-up" data-aos-delay="300">3. Compute connection sensitivity \( s_j \) for each connection \( j \):
<script type="math/tex"> g_j(w; D_b) = \frac{\partial L(c \odot w; D)}{\partial c_j} \bigg|_{c=1} </script>
<script type="math/tex"> s_j = \frac{|g_j(w; D_b)|}{\sum_{k=1}^m |g_k(w; D_b)|} </script>
</li>
<li data-aos="fade-up" data-aos-delay="400">4. Sort sensitivity scores \( s \) in descending order.</li>
<li data-aos="fade-up" data-aos-delay="500">5. Prune the top-\(\kappa\) connections based on sensitivity scores.</li>
<li data-aos="fade-up" data-aos-delay="600">6. Train the pruned network to get \( w^* \).</li>
</ul>
</div>
<!-- GraSP Algorithm -->
<div class="algorithm bg-light p-3 mb-4 rounded" data-aos="fade-up">
<h3 class="text-primary">GraSP (Gradient Signal Preservation)</h3>
<p>GraSP prunes connections that least affect the gradient signal, preserving the most important connections for training.</p>
<ul class="list-unstyled">
<li data-aos="fade-up" data-aos-delay="100">1. Initialize the network weights \( w \).</li>
<li data-aos="fade-up" data-aos-delay="200">2. Sample a mini-batch of training data \( D_b \).</li>
<li data-aos="fade-up" data-aos-delay="300">3. Compute GraSP scores \( g_j \) for each connection \( j \):
<script type="math/tex"> g_j(w; D_b) = \left( H \frac{\partial L(w; D)}{\partial w_j} \right) \cdot w_j </script>
<script type="math/tex"> s_j = \frac{|g_j(w; D_b)|}{\sum_{k=1}^m |g_k(w; D_b)|} </script>
</li>
<li data-aos="fade-up" data-aos-delay="400">4. Sort GraSP scores \( s \) in descending order.</li>
<li data-aos="fade-up" data-aos-delay="500">5. Prune the top-\(\kappa\) connections based on GraSP scores.</li>
<li data-aos="fade-up" data-aos-delay="600">6. Train the pruned network to get \( w^* \).</li>
</ul>
</div>
<!-- SynFlow Algorithm -->
<div class="algorithm bg-light p-3 mb-4 rounded" data-aos="fade-up">
<h3 class="text-primary">SynFlow (Iterative Synaptic Flow Pruning)</h3>
<p>SynFlow iteratively prunes connections by preserving synaptic flow, ensuring the network remains connected and functional.</p>
<ul class="list-unstyled">
<li data-aos="fade-up" data-aos-delay="100">1. Set the model to evaluation mode.</li>
<li data-aos="fade-up" data-aos-delay="200">2. Initialize the binary mask \( \mu \) to 1.</li>
<li data-aos="fade-up" data-aos-delay="300">3. Iteratively prune for \( n \) steps:</li>
<li data-aos="fade-up" data-aos-delay="400"> a. Apply the mask \( \theta_{\mu} = \mu \odot \theta_0 \).</li>
<li data-aos="fade-up" data-aos-delay="500"> b. Compute SynFlow objective \( R \):
<script type="math/tex"> R = 1^T \left( \prod_{l=1}^{L} |\theta_{\mu}^{[l]}| \right) 1 </script>
</li>
<li data-aos="fade-up" data-aos-delay="600"> c. Compute SynFlow score \( S \):
<script type="math/tex"> S = \frac{\partial R}{\partial \theta_{\mu}} \odot \theta_{\mu} </script>
</li>
<li data-aos="fade-up" data-aos-delay="700"> d. Determine the pruning threshold \( \tau \):
<script type="math/tex"> \tau = (1 - \rho^{-\frac{k}{n}}) \text{ percentile of } S </script>
</li>
<li data-aos="fade-up" data-aos-delay="800"> e. Update the mask \( \mu = (\tau < S) \).</li>
<li data-aos="fade-up" data-aos-delay="900">4. Return the pruned network \( f(x; \mu \odot \theta_0) \).</li>
</ul>
</div>
<!-- OurAlgo Algorithm -->
<div class="algorithm bg-light p-3 mb-4 rounded" data-aos="fade-up">
<h3 class="text-primary">OurAlgo (Combined Gradient and Magnitude-Based Pruning)</h3>
<p>OurAlgo combines magnitude and gradient-based pruning to retain the most important parameters for model performance.</p>
<ul class="list-unstyled">
<li data-aos="fade-up" data-aos-delay="100">1. Initialize the network weights \( w \).</li>
<li data-aos="fade-up" data-aos-delay="200">2. Sample a mini-batch of training data \( D_b \).</li>
<li data-aos="fade-up" data-aos-delay="300"><b>Compute Magnitude Scores:</b></li>
<li data-aos="fade-up" data-aos-delay="400"> a. For each parameter \( p_i \), calculate \( \text{mag_scores}[i] = |p_i| \).</li>
<li data-aos="fade-up" data-aos-delay="500"><b>Compute Gradient Scores:</b></li>
<li data-aos="fade-up" data-aos-delay="600"> a. Perform forward and backward pass on \( D_b \) to compute gradients.</li>
<li data-aos="fade-up" data-aos-delay="700"> b. For each parameter \( p_i \) with mask \( m_i \), calculate \( \text{grad_scores}[i] = \left| \frac{\partial L}{\partial m_i} \right| \).</li>
<li data-aos="fade-up" data-aos-delay="800"><b>Combine Scores:</b></li>
<li data-aos="fade-up" data-aos-delay="900"> a. For each parameter \( p_i \), calculate \( \text{combined_scores}[i] = \alpha \times \text{mag_scores}[i] + \beta \times \text{grad_scores}[i] \).</li>
<li data-aos="fade-up" data-aos-delay="1000">4. Sort combined scores and prune parameters with the lowest scores.</li>
<li data-aos="fade-up" data-aos-delay="1100">5. Fine-tune the pruned network.</li>
</ul>
</div>
</div>
</div>
</div>
<!-- Add Bootstrap, jQuery, and AOS JavaScript libraries -->
<script src="https://code.jquery.com/jquery-3.5.1.slim.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/@popperjs/[email protected]/dist/umd/popper.min.js"></script>
<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.5.2/js/bootstrap.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/aos/2.3.4/aos.js"></script>
<script>
AOS.init();
</script>
<script>
document.addEventListener('DOMContentLoaded', function () {
var options = {
strings: ["This project focuses on developing and implementing various deep learning model compression techniques. The primary aim is to reduce the size and computational requirements of neural network models without significantly impacting their performance."],
typeSpeed: 50,
backSpeed: 0,
showCursor: true,
cursorChar: '|',
autoInsertCss: true,
onComplete: function() {
// Additional action after typing is complete
}
};
var typed = new Typed("#typed-introduction", options);
});
</script>
</body>
</html>