08.srt

1
00:00:00,000 --> 00:00:04,025
字幕生成：qiaokai  字幕校对：A是传奇

2
00:00:05,127 --> 00:00:06,135
 Hello，大家好

3
00:00:06,135 --> 00:00:08,135
欢乐的时光过得特别快

4
00:00:08,135 --> 00:00:10,135
又是时候说拜拜

5
00:00:14,995 --> 00:00:15,995
我是ZOMI

6
00:00:15,995 --> 00:00:18,995
今天来到推理型的模型优化

7
00:00:18,995 --> 00:00:21,995
计算图优化里面的最后一个内容了

8
00:00:21,995 --> 00:00:24,995
所以我今天的内容会排得特别的密

9
00:00:24,995 --> 00:00:27,995
然后我也会讲得特别的快了一点点

10
00:00:27,995 --> 00:00:29,995
首先今天主要是去讲讲

11
00:00:29,995 --> 00:00:30,995
Extend的内容

12
00:00:30,995 --> 00:00:33,995
还有Layer跟Memory的一些优化

13
00:00:33,995 --> 00:00:35,995
其实这些很多的相关的知识

14
00:00:35,995 --> 00:00:37,995
简单的理解一下就好了

15
00:00:37,995 --> 00:00:39,995
更多的是希望各个同事

16
00:00:39,995 --> 00:00:40,995
能够在工作当中

17
00:00:40,995 --> 00:00:41,995
跟我一起去发现

18
00:00:41,995 --> 00:00:42,995
更多不同的可能性

19
00:00:42,995 --> 00:00:44,995
或者更多优化的方式

20
00:00:44,995 --> 00:00:46,995
可以看到在整个工作流程当中

21
00:00:46,995 --> 00:00:49,995
现在来到了第二个和第三个步骤里面

22
00:00:49,995 --> 00:00:51,995
当然他们没有一个明确的界限之分

23
00:00:51,995 --> 00:00:53,995
基本上都可以通用的

24
00:00:53,995 --> 00:00:55,995
而Path的管理也是非常重要

25
00:00:55,995 --> 00:00:57,995
在这里面我推荐大家去看看

26
00:00:57,995 --> 00:01:00,995
AI编译里面的前端优化里面相关的内容

27
00:01:00,995 --> 00:01:01,995
虽然大家可以看到

28
00:01:01,995 --> 00:01:05,995
前端优化AI编译看了确实太少了

29
00:01:05,995 --> 00:01:07,995
你甚至就我自己不断的点来点去

30
00:01:07,995 --> 00:01:08,995
换了不同电脑

31
00:01:08,995 --> 00:01:09,995
可能在有时候出差的时候

32
00:01:09,995 --> 00:01:10,995
在北京点了

33
00:01:10,995 --> 00:01:11,995
有时候在深圳点了

34
00:01:11,995 --> 00:01:12,995
确实点来点去

35
00:01:12,995 --> 00:01:14,995
特别多了已经点的变成

36
00:01:17,799 --> 00:01:21,799
现在来到计算图详解的第二个内容

37
00:01:21,799 --> 00:01:23,799
第二个内容就是其他图的优化

38
00:01:23,799 --> 00:01:25,799
可以看到其他图的优化了很多

39
00:01:25,799 --> 00:01:27,799
就是针对不同的AI框架

40
00:01:27,799 --> 00:01:28,799
他可能没有直接的实现

41
00:01:28,799 --> 00:01:30,799
而是通过一些特殊的一些组合

42
00:01:30,799 --> 00:01:32,799
更多的是跟硬件或者特殊的网络模型

43
00:01:32,799 --> 00:01:35,927
或者特殊的网络模型，特殊的领域相关的

44
00:01:35,927 --> 00:01:36,927
看几个

45
00:01:36,927 --> 00:01:38,927
就是假设是Layer Norm的融合

46
00:01:38,927 --> 00:01:40,927
还有PReLU的替换

47
00:01:40,927 --> 00:01:41,927
Matmul的转变

48
00:01:41,927 --> 00:01:43,927
还有Binary跟Eltwise

49
00:01:43,927 --> 00:01:45,927
最后还有Reduce

50
00:01:45,927 --> 00:01:46,927
还有Global Pooling

51
00:01:46,927 --> 00:01:47,927
所以确实特别多

52
00:01:47,927 --> 00:01:49,927
看看具体的图

53
00:01:49,927 --> 00:01:52,927
这里面就不一个一个的去给大家展开了

54
00:01:52,927 --> 00:01:55,513
实际上像有些算子

55
00:01:55,513 --> 00:01:56,513
假设ReLU加Mul

56
00:01:56,513 --> 00:01:57,513
然后加Eltwise Sum

57
00:01:57,513 --> 00:01:58,513
然后加ReLU这种方式

58
00:01:58,513 --> 00:02:00,513
如果遇到这种图

59
00:02:00,513 --> 00:02:02,513
直接变成一个PReLU就行了

60
00:02:02,513 --> 00:02:03,513
当然了

61
00:02:03,513 --> 00:02:04,513
如果引擎里面

62
00:02:04,513 --> 00:02:06,513
推理引擎里面的没有PReLU这个算子

63
00:02:06,513 --> 00:02:08,513
也可以把它拆分成

64
00:02:08,513 --> 00:02:10,513
这堆算子的一个具体的实现

65
00:02:10,513 --> 00:02:12,513
就可以代替掉PReLU了

66
00:02:12,513 --> 00:02:14,513
所以说这里面没有一个统一的界限

67
00:02:14,513 --> 00:02:18,513
更多的是根据具体底层有什么算子

68
00:02:18,513 --> 00:02:20,513
就提供什么优化的Path

69
00:02:20,513 --> 00:02:21,513
而不是优化的Path

70
00:02:21,513 --> 00:02:22,513
我随便写

71
00:02:22,513 --> 00:02:23,513
你写推理引擎的人

72
00:02:23,513 --> 00:02:24,513
写Kernel的人

73
00:02:24,513 --> 00:02:25,513
你就随便自己写

74
00:02:25,513 --> 00:02:28,513
是有一个非常良好的合作关系

75
00:02:28,513 --> 00:02:29,513
才能够把整个推理引擎

76
00:02:29,513 --> 00:02:31,513
做到极致性能的优化

77
00:02:31,513 --> 00:02:34,513
下面像可以看到MatMul

78
00:02:34,513 --> 00:02:35,513
它确实有两个Transpose

79
00:02:35,513 --> 00:02:38,513
但实际上有些推理引擎里面

80
00:02:38,513 --> 00:02:39,513
它就已经把数的数据

81
00:02:39,513 --> 00:02:41,513
自动的做了一个Transpose

82
00:02:41,513 --> 00:02:43,513
这个时候可以减少一个数据的

83
00:02:43,513 --> 00:02:45,513
搬运的过程当中

84
00:02:45,513 --> 00:02:47,513
但还有很多像这种BinaryMul

85
00:02:47,513 --> 00:02:48,513
还有BinaryAdd

86
00:02:48,513 --> 00:02:51,513
就可以换成Eltwise Sum

87
00:02:51,513 --> 00:02:53,513
这种特殊的方式

88
00:02:54,701 --> 00:02:57,701
下面来看一个比较特殊

89
00:02:57,701 --> 00:02:59,701
ZOMI觉得最近也是比较有意思

90
00:02:59,701 --> 00:03:01,701
或者应该是去年年底吧

91
00:03:01,701 --> 00:03:02,701
今年年初

92
00:03:02,701 --> 00:03:03,701
去年年底

93
00:03:03,701 --> 00:03:05,701
现在已经是23年了

94
00:03:06,701 --> 00:03:07,701
有时候我在想

95
00:03:07,701 --> 00:03:09,701
有没有可能你在看这个视频的时候

96
00:03:09,701 --> 00:03:10,701
已经到24 25年的时候

97
00:03:10,701 --> 00:03:11,701
然后你发现

98
00:03:11,701 --> 00:03:13,701
我怎么还在看22年的视频

99
00:03:13,701 --> 00:03:14,701
或者23年的视频

100
00:03:15,701 --> 00:03:18,701
确实这个文章是23年发的

101
00:03:18,701 --> 00:03:20,701
它叫做FlashAttention

102
00:03:20,701 --> 00:03:22,701
它里面就对Attention

103
00:03:22,701 --> 00:03:23,701
做了一个特殊的优化

104
00:03:23,701 --> 00:03:26,701
下面看一下具体的图

105
00:03:26,701 --> 00:03:27,701
文章里面铺出来的图

106
00:03:27,701 --> 00:03:29,701
简单的去解读一下

107
00:03:29,701 --> 00:03:32,701
像现在大家用的非常多的

108
00:03:32,701 --> 00:03:33,701
Attention或者Transformer的

109
00:03:33,701 --> 00:03:35,701
一些网络模型的层

110
00:03:35,701 --> 00:03:37,701
但是Attention确实很少

111
00:03:37,701 --> 00:03:39,701
在推理引擎里面去应用

112
00:03:39,701 --> 00:03:41,701
确实像Attention层或者Transformer层了 

113
00:03:41,701 --> 00:03:43,701
它没有跑得像卷积层

114
00:03:43,701 --> 00:03:44,701
这么快也没有经过

115
00:03:44,701 --> 00:03:46,701
那么多的特殊的优化

116
00:03:46,701 --> 00:03:48,701
那卷积的特殊的优化

117
00:03:48,701 --> 00:03:50,701
将会在下一个内容里面

118
00:03:50,701 --> 00:03:51,701
Kernel的执行

119
00:03:51,701 --> 00:03:53,701
或者具体的Runtime里面

120
00:03:53,701 --> 00:03:54,701
去给大家介绍的

121
00:03:54,701 --> 00:03:55,701
这里面看一下

122
00:03:55,701 --> 00:03:56,701
FlashAttention里面

123
00:03:56,701 --> 00:03:58,701
具体做了哪些工作

124
00:03:59,202 --> 00:04:00,202
其实知道

125
00:04:00,202 --> 00:04:02,202
在Attention或者Transformer里面

126
00:04:02,202 --> 00:04:06,202
大部分都是算QKV

127
00:04:06,202 --> 00:04:09,202
通过QKV这三个矩阵不断的相乘

128
00:04:09,202 --> 00:04:11,202
就得到Transformer

129
00:04:11,202 --> 00:04:12,202
或者Multi head-Attention这个层

130
00:04:12,202 --> 00:04:15,202
接着下一个网络模型

131
00:04:15,202 --> 00:04:16,202
就是算Softmax

132
00:04:16,202 --> 00:04:19,202
那Softmax在这里面就简称SM

133
00:04:19,202 --> 00:04:21,202
不要误解

134
00:04:21,202 --> 00:04:22,202
这里不是

135
00:04:35,000 --> 00:04:38,000
这里面就有一个SM去算QKV

136
00:04:38,000 --> 00:04:40,000
可以看到假设AI引擎

137
00:04:40,000 --> 00:04:42,000
会跑在具体的一些芯片里面

138
00:04:42,000 --> 00:04:44,000
具体的一些加速芯片

139
00:04:44,000 --> 00:04:47,000
大部分都不会有太多的一些SM

140
00:04:47,000 --> 00:04:49,000
SM确实里面的容量非常有限

141
00:04:49,000 --> 00:04:51,000
于是就会对矩阵

142
00:04:51,000 --> 00:04:53,000
分块来进行计算

143
00:04:53,000 --> 00:04:54,000
那这个Loop

144
00:04:54,000 --> 00:04:56,000
大家可以去看一下

145
00:04:56,000 --> 00:04:58,000
AI编译器里面的有一节内容

146
00:04:58,000 --> 00:04:59,000
就是Kernel的优化

147
00:04:59,000 --> 00:05:01,000
或者后端的优化里面就会讲

148
00:05:01,000 --> 00:05:02,000
为什么要装Loop

149
00:05:02,000 --> 00:05:05,000
然后怎么对这些进行一个切片

150
00:05:05,000 --> 00:05:06,000
这里面回到

151
00:05:06,000 --> 00:05:08,000
FlashAttention的一个内容里面

152
00:05:08,000 --> 00:05:10,000
看到K

153
00:05:10,000 --> 00:05:11,000
假设就会把一些

154
00:05:11,000 --> 00:05:13,000
取出一小块进行计算

155
00:05:13,000 --> 00:05:14,000
那像这种Q

156
00:05:14,000 --> 00:05:16,000
也可以取出一小块进行计算

157
00:05:16,000 --> 00:05:18,000
那计算完之后

158
00:05:18,000 --> 00:05:19,000
QKV要相乘

159
00:05:19,000 --> 00:05:20,000
相乘完之后

160
00:05:20,000 --> 00:05:22,000
再给Softmax进行一个运行运算的

161
00:05:22,000 --> 00:05:24,000
但是Softmax里面

162
00:05:24,000 --> 00:05:25,000
就会把数据摊平

163
00:05:25,000 --> 00:05:27,000
摊成一条进行一个计算

164
00:05:27,000 --> 00:05:29,000
因为Softmax是接受一个

165
00:05:29,000 --> 00:05:31,000
vector进行计算的

166
00:05:31,000 --> 00:05:32,000
如果是这样的话

167
00:05:32,000 --> 00:05:34,000
就算得非常慢了

168
00:05:34,000 --> 00:05:36,000
于是作者就在FlashAttention里面

169
00:05:36,000 --> 00:05:37,000
就提到了

170
00:05:37,000 --> 00:05:39,000
我通过滚动的方式

171
00:05:39,000 --> 00:05:41,000
去计算我的Softmax

172
00:05:41,000 --> 00:05:42,000
去计算这个SM

173
00:05:42,000 --> 00:05:45,000
使得我的速度就进一步去提升

174
00:05:45,000 --> 00:05:47,000
算完一块QKV

175
00:05:47,000 --> 00:05:49,000
再给到Softmax的结果进行重排

176
00:05:49,000 --> 00:05:52,000
通过这种新颖的计算方式

177
00:05:52,000 --> 00:05:53,000
使得Attention

178
00:05:53,000 --> 00:05:55,000
在GPT-2里面

179
00:05:55,000 --> 00:05:58,000
有了接近7倍的提升了

180
00:05:58,000 --> 00:05:59,000
这个可不得了

181
00:05:59,000 --> 00:06:00,000
大家要知道

182
00:06:00,000 --> 00:06:02,000
训练一个GPT-3的时间

183
00:06:02,000 --> 00:06:04,000
要80多天

184
00:06:04,000 --> 00:06:06,000
80多天128个GPU

185
00:06:06,000 --> 00:06:08,000
有多少人有多少人

186
00:06:08,000 --> 00:06:10,000
有资源去算这个的

187
00:06:10,000 --> 00:06:11,000
基本上如果我不做一些

188
00:06:11,000 --> 00:06:12,000
大模型的项目

189
00:06:12,000 --> 00:06:14,000
是拿不到这么多资源的

190
00:06:14,000 --> 00:06:15,000
所以说一般来说

191
00:06:15,000 --> 00:06:16,000
训练这个模型

192
00:06:16,000 --> 00:06:17,000
或者Transformer的模型

193
00:06:17,000 --> 00:06:18,000
特别特别慢

194
00:06:18,000 --> 00:06:19,000
但是有了FlashAttention之后

195
00:06:19,000 --> 00:06:20,000
就可以把

196
00:06:20,000 --> 00:06:22,000
真正的Attention的推理

197
00:06:22,000 --> 00:06:23,000
MultiAttention的推理

198
00:06:23,000 --> 00:06:25,000
变成现实

199
00:06:26,000 --> 00:06:27,000
如果我讲得不清楚

200
00:06:27,000 --> 00:06:28,000
也非常欢迎大家

201
00:06:28,000 --> 00:06:30,000
去翻一翻这篇论文

202
00:06:30,000 --> 00:06:31,000
这篇论文里面

203
00:06:31,000 --> 00:06:32,000
有非常多的公式

204
00:06:32,000 --> 00:06:33,000
里面给到的一个附录

205
00:06:33,000 --> 00:06:35,000
也是非常多

206
00:06:35,000 --> 00:06:36,000
下面来到

207
00:06:36,000 --> 00:06:39,000
计算图优化的第三个部分

208
00:06:39,000 --> 00:06:42,000
计算图优化详解

209
00:06:42,000 --> 00:06:43,000
那在这里面

210
00:06:43,000 --> 00:06:45,000
我还是非常推荐

211
00:06:45,000 --> 00:06:47,000
大家去看一看这个内容

212
00:06:47,000 --> 00:06:48,000
为什么呢

213
00:06:48,000 --> 00:06:49,000
因为在第三个部分

214
00:06:49,000 --> 00:06:51,000
更多的是对Layout跟Memory

215
00:06:51,000 --> 00:06:53,000
的一些优化

216
00:06:53,000 --> 00:06:54,000
可以看到

217
00:06:54,000 --> 00:06:55,000
Layout的优化

218
00:06:55,000 --> 00:06:57,000
就是数据布局的优化

219
00:06:57,000 --> 00:06:58,000
在数据布局里面

220
00:06:58,000 --> 00:07:00,000
确实讲了非常多

221
00:07:00,000 --> 00:07:02,000
从NCHW到NHWC

222
00:07:02,000 --> 00:07:04,000
再到华为自己推出的

223
00:07:04,000 --> 00:07:06,000
NCHWC0

224
00:07:06,000 --> 00:07:08,000
这种方式确实很特别

225
00:07:08,000 --> 00:07:09,000
可以看到

226
00:07:09,000 --> 00:07:10,000
不同的算子的层

227
00:07:10,000 --> 00:07:11,000
或者不同的Cast

228
00:07:11,000 --> 00:07:12,000
需要做一个

229
00:07:12,000 --> 00:07:13,000
Cast Data的转换

230
00:07:13,000 --> 00:07:14,000
针对网络模型

231
00:07:14,000 --> 00:07:16,000
上一层跟下层的算子的相同

232
00:07:16,000 --> 00:07:17,000
可能不需要转换

233
00:07:17,000 --> 00:07:18,000
但上一层输入

234
00:07:18,000 --> 00:07:20,000
跟下层输入不同的时候

235
00:07:20,000 --> 00:07:21,000
就需要进行一个

236
00:07:21,000 --> 00:07:23,000
插入具体的算子

237
00:07:23,000 --> 00:07:25,000
这也是在图优化里面去做的

238
00:07:25,000 --> 00:07:26,000
如果是相同的时候

239
00:07:26,000 --> 00:07:28,000
就要删掉一些算子

240
00:07:28,000 --> 00:07:30,000
所以说这里面的研究

241
00:07:30,000 --> 00:07:31,000
要根据计算图

242
00:07:31,000 --> 00:07:33,000
来进行优化

243
00:07:33,000 --> 00:07:34,000
第二个内容

244
00:07:34,000 --> 00:07:37,000
就是内存分配的算法

245
00:07:37,000 --> 00:07:39,000
确实内存分配

246
00:07:39,000 --> 00:07:40,000
要在图

247
00:07:40,000 --> 00:07:42,000
有图的概念进行一个预分配

248
00:07:42,000 --> 00:07:44,000
分配的方式有两个

249
00:07:44,000 --> 00:07:47,000
一个是Inplace Operation

250
00:07:47,000 --> 00:07:50,000
就像下面右下角这个图

251
00:07:50,000 --> 00:07:53,000
假设我计算完这个算子之后

252
00:07:53,000 --> 00:07:54,000
这块内存

253
00:07:54,000 --> 00:07:56,000
黄色橙色的这块内存

254
00:07:56,000 --> 00:07:57,000
已经不需要了

255
00:07:57,000 --> 00:07:58,000
而且下一个操作

256
00:07:58,000 --> 00:07:59,000
也是Eltwise的

257
00:07:59,000 --> 00:08:01,000
就跟它的内存大小是一样的

258
00:08:01,000 --> 00:08:04,000
就我直接覆盖掉原来的内存

259
00:08:04,000 --> 00:08:06,000
进行一个原地的替换

260
00:08:06,000 --> 00:08:08,000
然后就不用打开新的空间了

261
00:08:08,000 --> 00:08:09,000
这需要根据图

262
00:08:09,000 --> 00:08:10,000
来进行优化的

263
00:08:10,000 --> 00:08:13,000
第二种就是Memory Sharing

264
00:08:13,000 --> 00:08:16,000
Memory Sharing里面就很特别的

265
00:08:16,000 --> 00:08:18,000
就是你不能原地的覆盖

266
00:08:18,000 --> 00:08:19,000
但是我可以

267
00:08:19,000 --> 00:08:20,000
如果你这个数据

268
00:08:20,000 --> 00:08:22,000
就算完这个Softmax之后

269
00:08:22,000 --> 00:08:24,000
我这个数据暂时已经不用了

270
00:08:24,000 --> 00:08:25,000
我算下一个的时候

271
00:08:25,000 --> 00:08:27,000
确实为了节省我的计算的空间

272
00:08:27,000 --> 00:08:29,000
或者节省内存

273
00:08:29,000 --> 00:08:30,000
这个时候就可以

274
00:08:30,000 --> 00:08:33,000
共享一些内存空间

275
00:08:33,000 --> 00:08:34,000
在算这个算子的时候

276
00:08:34,000 --> 00:08:36,000
就覆盖掉原来的一些就可以了

277
00:08:36,000 --> 00:08:38,000
然后对它进行更新

278
00:08:38,000 --> 00:08:39,000
然后从这里面去取

279
00:08:39,000 --> 00:08:42,000
从红色的这个内存块里面去预取

280
00:08:42,000 --> 00:08:45,000
这种就是两个数据大小相同

281
00:08:45,000 --> 00:08:46,000
而且前一个数据

282
00:08:46,000 --> 00:08:48,000
参与计算后面的数据不需要

283
00:08:48,000 --> 00:08:51,000
那就后面的就可以覆盖掉了

284
00:08:51,000 --> 00:08:53,000
所以内存优化有这几种方式

285
00:08:53,000 --> 00:08:54,000
更详细的

286
00:08:54,000 --> 00:08:55,000
还是在内存分配

287
00:08:55,000 --> 00:08:56,000
这个内容里面

288
00:08:56,000 --> 00:08:58,000
给大家已经详细的介绍了

289
00:08:58,000 --> 00:09:00,000
这里面简单的回顾一下

290
00:09:02,000 --> 00:09:05,000
今天的内容就到这里为止

291
00:09:05,000 --> 00:09:07,000
在推理引擎架构里面回顾一下

292
00:09:07,000 --> 00:09:10,000
整个推理引擎架构主要分开两部分

293
00:09:10,000 --> 00:09:12,000
第一部分就是上面的

294
00:09:12,000 --> 00:09:15,000
IR这层屎黄色的以上

295
00:09:15,000 --> 00:09:17,000
这一部分确实换了

296
00:09:17,000 --> 00:09:19,000
金黄色以上的

297
00:09:26,000 --> 00:09:28,000
就是一个蓝色的模块

298
00:09:28,000 --> 00:09:30,000
就是离线转换模块

299
00:09:30,000 --> 00:09:31,000
那离线转换模块了

300
00:09:31,000 --> 00:09:33,000
有离线的优化压缩了

301
00:09:33,000 --> 00:09:35,000
还有离线的转换图优化

302
00:09:35,000 --> 00:09:37,000
首先先把从不同AI框架训练

303
00:09:37,000 --> 00:09:39,000
得到的网络模型

304
00:09:39,000 --> 00:09:40,000
转换成为自己的IR

305
00:09:40,000 --> 00:09:41,000
然后基于这个IR

306
00:09:41,000 --> 00:09:44,000
可以做很多不同的计算图的优化

307
00:09:44,000 --> 00:09:46,000
接下来在下一个环节里面

308
00:09:46,000 --> 00:09:47,000
将会去给大家讲讲

309
00:09:47,000 --> 00:09:48,000
执行的优化

310
00:09:48,000 --> 00:09:49,000
Runtime的优化

311
00:09:49,000 --> 00:09:50,000
还有kernel的优化

312
00:09:50,000 --> 00:09:51,000
那下一期再见

313
00:09:51,000 --> 00:09:52,000
拜了个拜

314
00:09:53,000 --> 00:09:55,000
卷的不行了

315
00:09:55,000 --> 00:09:57,000
记得一键三连加关注哦

316
00:09:57,000 --> 00:09:59,000
所有的内容都会开源在

317
00:09:59,000 --> 00:10:00,000
下面这条链接里面

318
00:10:00,000 --> 00:10:02,000
拜了个拜