04.srt

1
00:00:00,000 --> 00:00:04,480
字幕生成: BLACK 字幕校对: 方鸿渐

2
00:00:06,720 --> 00:00:07,760
哈喽大家好

3
00:00:07,760 --> 00:00:10,320
今天晚上的更新应该是比较早的

4
00:00:10,320 --> 00:00:14,160
现在才22点52分

5
00:00:14,160 --> 00:00:16,160
也就是11点左右

6
00:00:16,160 --> 00:00:17,600
没有像以前那么晚了

7
00:00:17,600 --> 00:00:20,160
今天我给大家带来的一个内容

8
00:00:20,160 --> 00:00:24,000
就是推理系统或者推理引擎里面的模型小型化

9
00:00:24,000 --> 00:00:25,200
这里面呢

10
00:00:25,200 --> 00:00:27,280
对提一点我是ZOMI

11
00:00:27,360 --> 00:00:30,400
今天我要给大家带来一个比较新的内容

12
00:00:30,400 --> 00:00:34,320
就是Transformer这个结构的一些模型小型化

13
00:00:34,320 --> 00:00:36,640
在前两节内容里面呢

14
00:00:36,640 --> 00:00:42,160
主要是关心传统的CNN网络模型结构的一个小型化

15
00:00:42,160 --> 00:00:44,000
这个内容也是比较传统

16
00:00:44,000 --> 00:00:47,600
或者研究的已经算是比较明白了

17
00:00:47,600 --> 00:00:49,760
现在来看一个新的内容

18
00:00:49,760 --> 00:00:52,000
业界也是玩的不是说非常明白

19
00:00:52,000 --> 00:00:54,160
或者研究的不是说非常明白

20
00:00:54,240 --> 00:00:57,680
也有很多可以发paper还有研究的点

21
00:00:57,680 --> 00:01:02,560
这个内容也就是Transformer结构的小型化

22
00:01:02,560 --> 00:01:07,840
这里面就非常欢迎大家可以在Transformer小型化里面

23
00:01:07,840 --> 00:01:11,120
做非常多的科研的创新和研究了

24
00:01:12,720 --> 00:01:14,640
现在来看一下这个图

25
00:01:14,640 --> 00:01:18,960
这个图其实在分布式并行里面已经讲过很多次了

26
00:01:19,040 --> 00:01:22,960
可以看到随着应该是从2016年到17年

27
00:01:22,960 --> 00:01:24,560
Transformer结构推出之后

28
00:01:25,200 --> 00:01:28,640
到2022年也就是截止到目前为止

29
00:01:28,640 --> 00:01:32,640
网络模型需要的参数量和计算量

30
00:01:32,640 --> 00:01:34,000
是节节的攀升

31
00:01:34,000 --> 00:01:36,240
而这里面有一个比较大的一个起点

32
00:01:36,240 --> 00:01:38,880
就是Large Scale大规模的数据化

33
00:01:38,880 --> 00:01:40,880
或者大规模的分布式并行

34
00:01:40,880 --> 00:01:43,680
而这个得益于Transformer结构出现之后

35
00:01:43,680 --> 00:01:46,480
可以看到这里面不管是Megatron, GPT-3

36
00:01:46,480 --> 00:01:48,720
还有盘古这些大模型

37
00:01:48,720 --> 00:01:52,480
所以导致网络模型参数量急剧的膨胀

38
00:01:52,480 --> 00:01:55,120
这个网络模型的参数量急剧的膨胀

39
00:01:55,120 --> 00:01:58,720
引起了现在来到了一个大模型的阶段

40
00:01:58,720 --> 00:02:02,160
也同时催生了很多预训练模型的出现

41
00:02:02,160 --> 00:02:03,440
有了预训练模型之后

42
00:02:03,440 --> 00:02:05,680
只需要去开发对应的下游任务

43
00:02:05,680 --> 00:02:08,800
就可以解决很多像以前模型的碎片化的问题

44
00:02:10,640 --> 00:02:12,160
关于大模型这个结构

45
00:02:12,160 --> 00:02:14,080
非常欢迎大家去看一下

46
00:02:14,080 --> 00:02:17,040
分布式并行里面的大模型训练和挑战

47
00:02:17,040 --> 00:02:19,120
大模型经典的算法结构

48
00:02:19,120 --> 00:02:20,720
这里面就是讲Transformer

49
00:02:20,720 --> 00:02:22,880
还有BERT这些大模型的引起

50
00:02:22,880 --> 00:02:26,160
还有针对一些千亿万亿规模的

51
00:02:26,160 --> 00:02:28,560
SOTA大模型的一些深度的剖析

52
00:02:30,000 --> 00:02:31,600
回到今天的内容里面

53
00:02:31,600 --> 00:02:33,440
其实可以看到大模型

54
00:02:33,440 --> 00:02:35,440
最开始是由Transformer

55
00:02:35,440 --> 00:02:36,720
就是Attention is all you need

56
00:02:36,720 --> 00:02:38,480
这篇文章所引起的

57
00:02:38,480 --> 00:02:39,600
而左边的这个

58
00:02:39,600 --> 00:02:42,000
就是Transformer的网络模型的结构

59
00:02:42,000 --> 00:02:43,120
看上去比较复杂

60
00:02:43,200 --> 00:02:45,760
其实就是由一个多头注意力机制

61
00:02:45,760 --> 00:02:46,640
去叠加两层

62
00:02:46,640 --> 00:02:47,680
然后加一个Layer Norm

63
00:02:47,680 --> 00:02:48,800
再加一个Feed Forward

64
00:02:48,800 --> 00:02:51,200
然后不断的一层层累积上去的

65
00:02:51,200 --> 00:02:52,400
这种网络模型结构

66
00:02:53,200 --> 00:02:54,720
当然了左边的这种结构

67
00:02:54,720 --> 00:02:57,840
主要是针对NLP的一些数据进行处理的

68
00:02:57,840 --> 00:03:01,120
后来谷歌又发了一篇文章叫做VIT

69
00:03:01,680 --> 00:03:04,160
这里面就提出了一种新的idea

70
00:03:04,160 --> 00:03:06,480
把图片的数据变成一个patch

71
00:03:06,480 --> 00:03:08,480
然后做一些embedding的操作

72
00:03:08,480 --> 00:03:10,720
最后输进去网络模型结构

73
00:03:10,720 --> 00:03:12,560
就是Transformer的encoder

74
00:03:12,560 --> 00:03:14,320
Transformer左边的这个是encoder

75
00:03:14,320 --> 00:03:15,840
右边的这个是decoder

76
00:03:16,240 --> 00:03:17,280
通过这种方式

77
00:03:17,360 --> 00:03:20,240
把视觉引到Transformer结构里面

78
00:03:20,240 --> 00:03:21,680
所以Transformer的结构

79
00:03:21,680 --> 00:03:24,080
又进一步的燃烧到不同的领域

80
00:03:25,520 --> 00:03:28,800
下面这个就是整个Transformer的结构了

81
00:03:28,800 --> 00:03:30,000
那往下看一下

82
00:03:30,000 --> 00:03:32,320
假设输入是我是李雷

83
00:03:32,880 --> 00:03:33,840
把这个数据

84
00:03:33,840 --> 00:03:35,600
进行一个embedded操作之后

85
00:03:35,600 --> 00:03:38,720
去给很多个Transformer的encoder

86
00:03:38,720 --> 00:03:39,360
之后

87
00:03:39,360 --> 00:03:40,800
再把最后一层的encoder

88
00:03:40,880 --> 00:03:42,880
给不同的decoder

89
00:03:42,880 --> 00:03:44,240
最后再output

90
00:03:44,240 --> 00:03:45,200
那这种方式

91
00:03:45,440 --> 00:03:48,080
主要是有很多个刚才介绍的

92
00:03:48,640 --> 00:03:50,480
左边的encoder跟decoder

93
00:03:50,480 --> 00:03:52,320
去进行一个组合

94
00:03:52,320 --> 00:03:53,200
通过这种方式

95
00:03:53,280 --> 00:03:55,520
使得网络模型的参数量

96
00:03:55,920 --> 00:03:57,680
进一步的增加

97
00:03:57,680 --> 00:03:58,880
就是这种方式

98
00:03:58,880 --> 00:04:01,280
去组成了大模型的时代

99
00:04:01,920 --> 00:04:02,480
可以看到

100
00:04:02,480 --> 00:04:05,920
其实Transformer的效果还是非常的好

101
00:04:05,920 --> 00:04:09,280
基本上完胜了传统的LSTM跟RNN的

102
00:04:09,360 --> 00:04:11,760
能够处理非常长的时序的问题

103
00:04:13,280 --> 00:04:14,800
但是有一个比较大的问题

104
00:04:14,800 --> 00:04:16,400
网络模型的参数量

105
00:04:16,400 --> 00:04:17,360
大上去之后

106
00:04:17,360 --> 00:04:19,040
精度确实提升了

107
00:04:19,040 --> 00:04:20,160
但是要考虑一点

108
00:04:20,160 --> 00:04:21,040
就是怎么样

109
00:04:21,040 --> 00:04:23,280
大模型进行部署起来了

110
00:04:23,280 --> 00:04:24,560
怎么把大模型

111
00:04:24,560 --> 00:04:26,240
真正的用起来呢

112
00:04:26,240 --> 00:04:27,920
这个时候又有很多人

113
00:04:27,920 --> 00:04:30,560
去研究大模型的小型化了

114
00:04:30,560 --> 00:04:31,920
而大模型说实话

115
00:04:32,480 --> 00:04:33,760
Transformer的参数量

116
00:04:33,760 --> 00:04:34,960
从千万的级别开始

117
00:04:34,960 --> 00:04:36,560
到BERT到E级别

118
00:04:36,560 --> 00:04:38,720
后来Transformer的这种大模型结构

119
00:04:38,720 --> 00:04:41,600
从10万、百万、千万

120
00:04:41,600 --> 00:04:44,000
到后来加上Transformer M1的这种结构

121
00:04:44,000 --> 00:04:45,680
到了万亿规模

122
00:04:45,680 --> 00:04:47,200
规模的参数量这么大

123
00:04:47,200 --> 00:04:49,120
单纯存一个网络模型

124
00:04:49,120 --> 00:04:50,640
都已经到几个GB了

125
00:04:50,640 --> 00:04:52,640
怎么对网络模型进行压缩

126
00:04:52,640 --> 00:04:55,600
就变得非常之热门的话题了

127
00:04:56,400 --> 00:04:58,240
于是可以看到

128
00:04:58,240 --> 00:04:59,280
其实这里面Inference

129
00:04:59,280 --> 00:05:00,960
就是参考的文献

130
00:05:00,960 --> 00:05:02,640
我分开了三段

131
00:05:02,640 --> 00:05:03,760
前面第一段

132
00:05:03,760 --> 00:05:07,440
就是针对NLP领域的一些压缩

133
00:05:07,680 --> 00:05:09,760
中间就针对CV的压缩

134
00:05:09,760 --> 00:05:12,480
后面主要是针对多模态的压缩

135
00:05:12,480 --> 00:05:14,160
时间还是都比较新的

136
00:05:14,160 --> 00:05:16,320
有些可能才刚发表没多久

137
00:05:16,320 --> 00:05:18,640
Q8BERT、DistilBert、TinyBERT

138
00:05:18,640 --> 00:05:20,400
这些都是用了一些量化

139
00:05:20,400 --> 00:05:22,320
或者压缩剪枝的方法

140
00:05:22,320 --> 00:05:26,000
下面这种TinyVIT、DynamicVIT、MiniVIT

141
00:05:26,000 --> 00:05:27,840
大部分也是用了剪枝蒸馏

142
00:05:27,840 --> 00:05:29,440
压缩量化的方法

143
00:05:29,440 --> 00:05:31,440
而下面这个也是

144
00:05:31,440 --> 00:05:33,520
大模型或者Transformer结构里面

145
00:05:33,520 --> 00:05:35,680
进行一个模型的压缩小型化

146
00:05:36,240 --> 00:05:37,920
大部分都是采用了

147
00:05:37,920 --> 00:05:40,240
传统的压缩方法的4件套

148
00:05:40,240 --> 00:05:43,120
量化、剪枝、蒸馏、低秩分解

149
00:05:44,000 --> 00:05:45,280
通过这4种方法

150
00:05:45,280 --> 00:05:47,200
来进行一个网络模型的压缩的

151
00:05:47,200 --> 00:05:49,200
但是今天的主角

152
00:05:49,200 --> 00:05:53,600
还是回到网络模型的结构的轻量化

153
00:05:53,600 --> 00:05:55,920
通过优化网络模型的结构

154
00:05:55,920 --> 00:05:57,040
使得网络模型

155
00:05:57,040 --> 00:05:59,600
在保存相同的精度的前提下

156
00:05:59,600 --> 00:06:01,760
网络模型的参数量进一步的减少

157
00:06:01,760 --> 00:06:03,360
这里面有比较著名的

158
00:06:03,360 --> 00:06:06,320
就是MobileVIT是Facebook发明的

159
00:06:06,320 --> 00:06:07,840
后来又出现了Mobileformer

160
00:06:07,840 --> 00:06:08,720
Efficientformer

161
00:06:08,720 --> 00:06:10,800
各种former的这种结构

162
00:06:10,800 --> 00:06:12,800
下面来去看一些

163
00:06:12,800 --> 00:06:16,000
SOTA的轻量级的Transformer的网络模型结构

164
00:06:18,560 --> 00:06:21,840
现在来到MobileVIT这篇文章

165
00:06:22,400 --> 00:06:23,840
在这篇文章提出的时候

166
00:06:23,840 --> 00:06:27,040
作者就在想像传统的CNN网络模型

167
00:06:27,040 --> 00:06:29,840
它的一个模型或者网络模型的小型化

168
00:06:29,840 --> 00:06:32,080
其实是非常成功的

169
00:06:32,240 --> 00:06:34,400
有没有可能去结合

170
00:06:34,400 --> 00:06:36,720
传统的MobileNet这种网络模型的

171
00:06:36,720 --> 00:06:37,600
轻量化的结构

172
00:06:37,600 --> 00:06:40,400
然后再把Transformer的网络模型结构的优势

173
00:06:40,400 --> 00:06:41,360
加进去呢

174
00:06:41,920 --> 00:06:43,120
基于这个想法

175
00:06:43,840 --> 00:06:47,280
于是作者就提出了MobileVIT的网络模型

176
00:06:47,280 --> 00:06:49,520
而且里面就提出了一个

177
00:06:49,520 --> 00:06:52,160
MobileVIT block的这种结构

178
00:06:52,160 --> 00:06:55,280
首先来看看网络模型的性能

179
00:06:55,280 --> 00:06:57,040
可以看到从这个图

180
00:06:57,040 --> 00:06:58,880
网络模型的性能基本上

181
00:06:58,880 --> 00:07:02,400
MobileVIT的一个网络模型的参数量

182
00:07:02,400 --> 00:07:04,400
大概有600万左右

183
00:07:04,400 --> 00:07:06,880
它只有2.7MB

184
00:07:06,880 --> 00:07:08,880
它的网络模型的参数量的大小

185
00:07:08,880 --> 00:07:10,400
确实比MobileNet V3

186
00:07:10,400 --> 00:07:12,320
还有NASNet确实要小

187
00:07:12,320 --> 00:07:16,320
而且性能有很好的提升

188
00:07:16,320 --> 00:07:18,240
在ImageNet分类模型里面

189
00:07:18,240 --> 00:07:21,760
就已经达到了78%的一个网络模型的精度

190
00:07:21,760 --> 00:07:24,800
那了解完它的一个基本的内容之后

191
00:07:24,800 --> 00:07:27,680
来看看它的网络模型结构

192
00:07:27,680 --> 00:07:32,160
下面这个图就是MobileVIT的网络模型结构

193
00:07:32,160 --> 00:07:36,000
可以看到假设一张图输入进去之后

194
00:07:36,000 --> 00:07:39,520
经过一个卷积MV2 MV2 MV2

195
00:07:39,520 --> 00:07:43,040
接着有一个MobileVIT block的这种结构

196
00:07:43,040 --> 00:07:44,640
它分颜色

197
00:07:44,640 --> 00:07:49,200
红色的这个其实叫做MobileNet V2的这个block

198
00:07:49,200 --> 00:07:52,480
引用了MobileNet V2的网络模型轻量化的结构

199
00:07:52,480 --> 00:07:57,120
但是在中间插入了一个MobileVIT的block

200
00:07:57,120 --> 00:08:00,480
MobileVIT的block就像上面所示

201
00:08:00,480 --> 00:08:04,160
首先它的输入是经过一个卷积层的feature map

202
00:08:04,160 --> 00:08:08,960
这个feature map给一个卷积NxN的网络模型进行一个卷积

203
00:08:08,960 --> 00:08:13,040
这个卷积的作用主要是做一些局部的特征提取

204
00:08:13,040 --> 00:08:15,360
接着运行一个一层一的卷积

205
00:08:15,360 --> 00:08:19,600
把网络模型的feature map投射到一个高维的空间里面

206
00:08:19,600 --> 00:08:21,200
接着通过一个unfold

207
00:08:21,200 --> 00:08:23,600
把网络模型结构展开成为transformer

208
00:08:23,600 --> 00:08:25,760
可以输入的网络模型的方式

209
00:08:25,840 --> 00:08:28,640
接着利用transformer QKV的这种方式

210
00:08:28,640 --> 00:08:30,480
去对全局的feature map

211
00:08:30,480 --> 00:08:32,960
就全局的特征进行一个提取

212
00:08:32,960 --> 00:08:36,240
然后再fold成原来的这种格式

213
00:08:36,240 --> 00:08:38,720
接着再做一个一层一的卷积

214
00:08:38,720 --> 00:08:42,400
再运行一个NxN的卷积最后输出

215
00:08:42,400 --> 00:08:44,560
输入跟输出是相同的

216
00:08:44,560 --> 00:08:47,680
中间为了给transformer的网络模型结构

217
00:08:47,680 --> 00:08:49,440
去做一个全局的感知

218
00:08:49,440 --> 00:08:51,920
于是做了一些简单的变换

219
00:08:51,920 --> 00:08:55,200
通过这种方式就实现了一个MobileVIT结构了

220
00:08:56,720 --> 00:09:01,040
好,现在来看看同年的第二个工作

221
00:09:01,040 --> 00:09:04,240
就是MobileFormer这篇文章

222
00:09:05,200 --> 00:09:07,200
像MobileFormer这篇文章

223
00:09:07,200 --> 00:09:10,160
其实跟刚才有点异曲同工之妙

224
00:09:10,160 --> 00:09:13,840
同样也是Bridging MobileNet and Transformer

225
00:09:13,840 --> 00:09:17,040
把MobileNet跟Transformer融合起来

226
00:09:17,040 --> 00:09:19,760
刚才那篇文章是Apple发明的

227
00:09:19,760 --> 00:09:22,960
而这篇文章是Microsoft去提出来的

228
00:09:22,960 --> 00:09:25,040
确实还挺有趣的

229
00:09:25,280 --> 00:09:29,360
下面简单的去看看这篇文章有什么不一样

230
00:09:30,080 --> 00:09:34,000
上面的这个图就是MobileFormer的整个网络模型的结构

231
00:09:34,000 --> 00:09:39,120
可以看到左边这个就是MobileNet的一个结构模型

232
00:09:39,120 --> 00:09:42,640
右边的这条分支就是Transformer的一个结构模型

233
00:09:42,640 --> 00:09:47,040
这边就以并行的方式去运行两个不同的block

234
00:09:47,040 --> 00:09:48,400
左边就是MobileNet block

235
00:09:48,400 --> 00:09:50,000
右边就是Transformer的block

236
00:09:50,000 --> 00:09:53,680
可以看到下面以虚线的框为例子

237
00:09:53,680 --> 00:09:58,480
左边的MobileNet block会把一些参数传给Transformer

238
00:09:58,480 --> 00:10:01,360
这个叫做MobileNet to Former

239
00:10:01,360 --> 00:10:02,560
Transformer学习完之后

240
00:10:02,560 --> 00:10:05,840
也会把它的一些feature map传给MobileNet

241
00:10:05,840 --> 00:10:09,040
这个就是它最简单的MobileFormer的结构

242
00:10:09,040 --> 00:10:10,960
往下再看一下

243
00:10:11,760 --> 00:10:16,640
这个图会更加清晰去讲讲它整个MobileFormer的block

244
00:10:16,640 --> 00:10:19,120
左下角这个就是MobileNet的一个block

245
00:10:19,120 --> 00:10:21,200
可以看到基本上就有一个卷积 ReLU

246
00:10:21,600 --> 00:10:22,720
Depth-wise 卷积 ReLU

247
00:10:23,680 --> 00:10:25,760
然后再看看右上角

248
00:10:25,760 --> 00:10:29,200
右上角就是一个Transformer的简单的encoder

249
00:10:29,200 --> 00:10:30,800
通过一个multi-head attention

250
00:10:30,800 --> 00:10:32,960
然后加一个ffn的层

251
00:10:32,960 --> 00:10:34,000
最后输出

252
00:10:34,000 --> 00:10:38,320
很有意思的就是要把MobileNet转成Former的结构

253
00:10:38,320 --> 00:10:42,320
这里面就需要做一个Mobile to Former的结构网络模型

254
00:10:42,320 --> 00:10:44,240
通过它的一个特殊的设计

255
00:10:44,240 --> 00:10:47,360
然后就把MobileNet的输出给到Former

256
00:10:47,360 --> 00:10:50,480
这里面还有一个Former to MobileNet

257
00:10:50,560 --> 00:10:52,400
一个红色的方框去展示

258
00:10:52,400 --> 00:10:55,840
同样也会把Transformer的一些网络模型的结构

259
00:10:55,840 --> 00:10:57,360
通过一些转换

260
00:10:57,360 --> 00:10:59,120
最后给到MobileNet

261
00:10:59,600 --> 00:11:02,000
通过这种左右并行的方式

262
00:11:02,000 --> 00:11:04,080
去实现了MobileFormer的结构

263
00:11:05,920 --> 00:11:08,160
看一下下一个场景

264
00:11:08,160 --> 00:11:09,840
就是EfficientFormer

265
00:11:09,840 --> 00:11:12,960
那大家看到这篇文章会不会想到

266
00:11:12,960 --> 00:11:15,200
EfficientFormer MobileFormer

267
00:11:15,200 --> 00:11:17,200
不就把EfficientNet

268
00:11:17,200 --> 00:11:19,440
之前讲的EfficientNet的一个系列的

269
00:11:19,440 --> 00:11:21,440
一些网络模型的经典的结构

270
00:11:21,440 --> 00:11:23,520
跟Transformer融合起来吗

271
00:11:23,520 --> 00:11:24,080
对

272
00:11:24,080 --> 00:11:25,920
这篇文章就是这个概念

273
00:11:27,440 --> 00:11:31,120
下面这个文章就是EfficientFormer的标题

274
00:11:31,120 --> 00:11:33,840
Vision Transformer at MobileNet Speed

275
00:11:33,840 --> 00:11:35,760
马上来看看它有什么不一样

276
00:11:36,640 --> 00:11:38,400
这个图很有代表性

277
00:11:38,400 --> 00:11:41,520
这个图可以看到EfficientFormer的性能

278
00:11:41,520 --> 00:11:43,280
确实很优

279
00:11:43,280 --> 00:11:46,160
比刚才的MobileFormer要高不少

280
00:11:46,160 --> 00:11:49,040
刚才MobileFormer只有到76%到78%

281
00:11:49,440 --> 00:11:52,000
虽然它的一个网络模型的参数量

282
00:11:52,000 --> 00:11:53,120
确实大了一点

283
00:11:53,120 --> 00:11:54,880
但是不妨碍它的性能

284
00:11:54,880 --> 00:11:56,400
确实能够做得很好

285
00:11:56,400 --> 00:11:58,240
再往下看一看

286
00:11:58,800 --> 00:12:00,160
大家看一下这个图

287
00:12:00,160 --> 00:12:02,000
这个图还是很有意思的

288
00:12:02,000 --> 00:12:04,640
作者去做了大量的消融实验

289
00:12:04,640 --> 00:12:05,920
还有分解实验

290
00:12:05,920 --> 00:12:07,120
去把一些MobileNet

291
00:12:07,120 --> 00:12:07,840
EfficientNet

292
00:12:07,840 --> 00:12:09,120
还有一些Poolformer

293
00:12:09,120 --> 00:12:10,080
EfficientFormer

294
00:12:10,080 --> 00:12:11,760
这些网络模型的结构

295
00:12:11,760 --> 00:12:13,760
都按照一些大的block

296
00:12:13,760 --> 00:12:15,920
或者大的一些结构体去展开

297
00:12:15,920 --> 00:12:17,920
可以看到它里面去算

298
00:12:17,920 --> 00:12:18,960
PatchEmbedding

299
00:12:18,960 --> 00:12:20,880
占了多少的时间

300
00:12:20,880 --> 00:12:23,440
Attention层又占了多少的时间

301
00:12:23,440 --> 00:12:25,040
还有一个线性激活层

302
00:12:25,040 --> 00:12:27,040
还有Reshape Normalize Activation

303
00:12:27,040 --> 00:12:29,840
分别都占了多少的时间

304
00:12:29,840 --> 00:12:33,280
而像这种确实大量的卷积去堆叠

305
00:12:33,280 --> 00:12:34,880
就没有必要去拆分了

306
00:12:34,880 --> 00:12:36,960
而Transformer的结构最大的问题

307
00:12:36,960 --> 00:12:39,040
就是它有很多的Embedding

308
00:12:39,040 --> 00:12:41,040
而且很多的MultiAttention

309
00:12:41,040 --> 00:12:43,440
还有Liner Reshape Normalization

310
00:12:43,440 --> 00:12:45,520
作者通过这种消融实验的分析

311
00:12:45,520 --> 00:12:47,600
可以找到要优化的点

312
00:12:47,600 --> 00:12:50,160
就是在Embedding层要做优化

313
00:12:50,160 --> 00:12:53,440
第二个需要在Liner层做优化

314
00:12:53,440 --> 00:12:56,320
第三个要在MultiAttention层做优化

315
00:12:56,320 --> 00:12:57,760
另外还有需要

316
00:12:57,760 --> 00:13:00,560
把这些Reshape的工作给它砍掉

317
00:13:00,560 --> 00:13:02,880
于是就出现了EfficientFormer

318
00:13:02,880 --> 00:13:03,840
这个网络模型结构

319
00:13:03,840 --> 00:13:04,880
可以看到它对比起

320
00:13:04,880 --> 00:13:06,480
刚才的一些消融实验里面

321
00:13:06,480 --> 00:13:08,240
很多的消耗实验的工作

322
00:13:08,240 --> 00:13:10,480
确实把它压缩的还挺厉害的

323
00:13:10,480 --> 00:13:12,800
具体看看它的网络模型结构

324
00:13:12,800 --> 00:13:15,120
那这个就是EfficientFormer的

325
00:13:15,120 --> 00:13:16,320
网络模型结构

326
00:13:16,400 --> 00:13:18,160
这里面有一个MB4D

327
00:13:18,160 --> 00:13:19,760
有一个MB3D

328
00:13:19,760 --> 00:13:22,080
先来看看EfficientFormer的

329
00:13:22,080 --> 00:13:23,040
网络模型结构

330
00:13:23,040 --> 00:13:24,320
上面这个就是它的

331
00:13:24,320 --> 00:13:26,240
整体的网络模型的结构

332
00:13:26,240 --> 00:13:28,080
从输入 卷积 卷积

333
00:13:28,080 --> 00:13:30,080
然后有一个MB4D的层

334
00:13:30,080 --> 00:13:32,000
MB4D是下面的展开

335
00:13:32,000 --> 00:13:33,200
先不来看

336
00:13:33,200 --> 00:13:34,880
然后有个Embedding MB4D

337
00:13:34,880 --> 00:13:36,240
Embedding MB4D

338
00:13:36,240 --> 00:13:38,400
接着这里面有个分水岭

339
00:13:38,400 --> 00:13:40,640
就是4D转3D

340
00:13:40,640 --> 00:13:43,440
它简单的做了一个映射变换之后

341
00:13:43,440 --> 00:13:46,160
到了另外一个新的结构MB3D

342
00:13:46,160 --> 00:13:47,600
然后Embedding MB3D

343
00:13:47,600 --> 00:13:49,120
在最后输出的

344
00:13:49,120 --> 00:13:52,160
MB3D就是右下角的模块

345
00:13:52,160 --> 00:13:54,080
现在分别的去看看

346
00:13:54,080 --> 00:13:56,240
MB3D跟MB4D有什么区别

347
00:13:56,240 --> 00:13:58,960
MB3D其实这里面

348
00:13:58,960 --> 00:14:00,880
确实像传统的

349
00:14:00,880 --> 00:14:02,000
卷积神经网络模型

350
00:14:02,000 --> 00:14:02,800
做一个pooling

351
00:14:02,800 --> 00:14:05,200
卷积bn ReLU这种工作

352
00:14:05,200 --> 00:14:07,040
没有太多的创新

353
00:14:07,040 --> 00:14:08,560
像MB3D

354
00:14:08,560 --> 00:14:10,400
因为引入了一个

355
00:14:10,400 --> 00:14:11,840
transformer的结构网络模型

356
00:14:11,840 --> 00:14:13,600
它的输入跟输出不一样

357
00:14:13,600 --> 00:14:15,040
像之前的Mobileformer

358
00:14:15,040 --> 00:14:16,320
确实你把MobileNet

359
00:14:16,320 --> 00:14:18,240
跟EfficientNet进行一个捆销

360
00:14:18,240 --> 00:14:18,960
你要大量的

361
00:14:18,960 --> 00:14:21,440
reshape之间的一些信信的转换

362
00:14:21,440 --> 00:14:23,440
这里面我只转换一次

363
00:14:23,440 --> 00:14:24,880
接着有大量的

364
00:14:24,880 --> 00:14:27,120
transformer组成的MB3D的结构

365
00:14:27,120 --> 00:14:28,720
进行一个相累加

366
00:14:28,720 --> 00:14:30,400
通过这种方式

367
00:14:30,400 --> 00:14:32,160
去解决了刚才的

368
00:14:32,160 --> 00:14:33,200
Patch Embedding

369
00:14:33,200 --> 00:14:34,400
还有Attention

370
00:14:34,400 --> 00:14:35,680
还有Linear reshape的

371
00:14:35,680 --> 00:14:36,960
这些耗时的工作

372
00:14:38,480 --> 00:14:38,880
好了

373
00:14:38,880 --> 00:14:40,000
今天的内容

374
00:14:40,000 --> 00:14:40,880
主要是讲了

375
00:14:40,880 --> 00:14:42,240
轻量级的transformer的

376
00:14:42,240 --> 00:14:43,200
网络模型结构

377
00:14:43,200 --> 00:14:44,720
从MobileNet 下面这个图就是MobileVIT的网络模型结构

378
00:14:44,800 --> 00:14:46,160
到MobileNet former

379
00:14:46,160 --> 00:14:48,320
然后再到EfficientFormer

380
00:14:48,320 --> 00:14:49,680
现在对

381
00:14:49,680 --> 00:14:51,120
轻量化的网络模型

382
00:14:51,120 --> 00:14:53,360
做了一个简单的总结

383
00:14:53,360 --> 00:14:54,720
首先第一点就是

384
00:14:54,720 --> 00:14:56,480
不同的网络模型架构

385
00:14:56,480 --> 00:14:58,240
即使它的FLOPs是相同

386
00:14:58,240 --> 00:15:00,240
但是MAC也有可能

387
00:15:00,240 --> 00:15:02,240
有很大的差异

388
00:15:02,240 --> 00:15:04,000
所以大家要去看一下

389
00:15:04,000 --> 00:15:05,440
不仅仅要看Flops

390
00:15:05,440 --> 00:15:06,640
实际上在

391
00:15:06,640 --> 00:15:07,840
真正的客户交付

392
00:15:07,840 --> 00:15:09,600
或者真正客户跑起来

393
00:15:09,600 --> 00:15:11,280
不仅仅只是看

394
00:15:11,280 --> 00:15:13,440
算网络模型的

395
00:15:13,440 --> 00:15:14,800
一个性能

396
00:15:14,800 --> 00:15:16,480
还要看MAC

397
00:15:16,480 --> 00:15:18,000
那至于FLOPs跟MAC

398
00:15:18,000 --> 00:15:18,800
可以回看一下

399
00:15:18,800 --> 00:15:19,840
网络模型小型化的

400
00:15:19,840 --> 00:15:20,880
第一个视频

401
00:15:20,880 --> 00:15:22,240
接着第二点就是

402
00:15:22,240 --> 00:15:23,760
FLOPs虽然低

403
00:15:23,760 --> 00:15:24,800
但是不等于

404
00:15:24,800 --> 00:15:25,760
或者不代表

405
00:15:25,760 --> 00:15:27,600
时延低

406
00:15:28,560 --> 00:15:29,760
具体还要结合

407
00:15:29,760 --> 00:15:30,960
具体的硬件架构

408
00:15:30,960 --> 00:15:32,000
去分析

409
00:15:32,000 --> 00:15:33,120
所以在做

410
00:15:33,120 --> 00:15:34,640
轻量化网络模型结构

411
00:15:34,640 --> 00:15:35,600
或者在做一个

412
00:15:35,600 --> 00:15:36,720
综合测评的时候

413
00:15:37,280 --> 00:15:39,680
确实要看很多的指标

414
00:15:39,680 --> 00:15:42,000
于是我在第一个视频的时候

415
00:15:42,000 --> 00:15:43,200
去给大家汇报了一下

416
00:15:43,200 --> 00:15:44,000
不同的参数

417
00:15:44,000 --> 00:15:45,280
所代表的意义

418
00:15:45,280 --> 00:15:46,560
接着第三点就是

419
00:15:46,560 --> 00:15:48,000
很多加速芯片

420
00:15:48,000 --> 00:15:49,440
特别是一些特殊的

421
00:15:49,440 --> 00:15:50,640
AI加速芯片

422
00:15:50,640 --> 00:15:51,840
最大的瓶颈

423
00:15:51,840 --> 00:15:53,840
是在访存的带宽

424
00:15:53,840 --> 00:15:55,600
而不是在它的算力

425
00:15:55,600 --> 00:15:56,720
访存的带宽

426
00:15:56,720 --> 00:15:57,840
确实很大程度

427
00:15:57,840 --> 00:15:58,560
去制约了

428
00:15:58,560 --> 00:16:00,160
整体的性能

429
00:16:01,520 --> 00:16:02,560
第四个就是

430
00:16:02,560 --> 00:16:03,680
不同的硬件平台

431
00:16:03,680 --> 00:16:04,880
去部署轻量化的

432
00:16:04,880 --> 00:16:05,920
一个网络模型

433
00:16:05,920 --> 00:16:07,680
需要根据具体的业务

434
00:16:07,680 --> 00:16:08,960
进行选择的

435
00:16:08,960 --> 00:16:09,520
这一点

436
00:16:09,520 --> 00:16:11,760
ZOMI也是深有体会的

437
00:16:11,760 --> 00:16:13,760
因为其实在面向

438
00:16:13,760 --> 00:16:15,520
CBG消费者业务

439
00:16:15,520 --> 00:16:17,520
去交付HMS core的时候

440
00:16:18,000 --> 00:16:19,360
也就是你们用华为手机

441
00:16:19,360 --> 00:16:20,240
后面跑的

442
00:16:20,240 --> 00:16:22,240
所有的一些轻量化的算法

443
00:16:22,240 --> 00:16:23,360
一开始说实话

444
00:16:23,360 --> 00:16:24,000
采用了

445
00:16:24,000 --> 00:16:24,800
MobileNetV3

446
00:16:24,800 --> 00:16:26,080
这个网络模型结构

447
00:16:26,080 --> 00:16:26,880
MobileNetV3

448
00:16:26,880 --> 00:16:28,320
在谷歌的手机里面

449
00:16:28,320 --> 00:16:29,520
确实跑得又快

450
00:16:29,520 --> 00:16:30,480
精度又高

451
00:16:30,480 --> 00:16:31,360
但是实际上

452
00:16:31,360 --> 00:16:32,960
在华为的手机

453
00:16:32,960 --> 00:16:34,640
还有高通的手机测下来

454
00:16:34,640 --> 00:16:36,400
MobileV3的性能

455
00:16:36,400 --> 00:16:38,560
反倒没有MobileV2要好

456
00:16:38,560 --> 00:16:40,560
而且精度可能还差不多

457
00:16:40,560 --> 00:16:42,160
所以说不同硬件

458
00:16:42,160 --> 00:16:43,840
部署的轻量级网络模型

459
00:16:43,840 --> 00:16:45,760
要根据具体的业务来决定

460
00:16:45,760 --> 00:16:46,880
而具体的业务

461
00:16:46,880 --> 00:16:48,880
还要看很多指标

462
00:16:48,880 --> 00:16:49,840
不同的时延

463
00:16:49,840 --> 00:16:51,200
不同的访存带宽

464
00:16:51,200 --> 00:16:52,560
都会影响

465
00:16:52,560 --> 00:16:54,240
轻量化的模型结构

466
00:16:54,880 --> 00:16:55,440
好了

467
00:16:55,440 --> 00:16:57,440
今天的内容就到这里为止

468
00:16:57,440 --> 00:16:58,400
谢谢各位

469
00:16:58,400 --> 00:16:59,200
拜了个拜

470
00:17:00,560 --> 00:17:02,160
卷的不行

471
00:17:02,160 --> 00:17:03,600
记得一键三连加关注

472
00:17:04,080 --> 00:17:04,800
所有的内容

473
00:17:04,800 --> 00:17:05,680
都会开源在

474
00:17:05,680 --> 00:17:06,960
下面这条链接里面

475
00:17:06,960 --> 00:17:08,400
拜拜