07.srt

1
00:00:00,000 --> 00:00:04,250
字幕生成：qiaokai  字幕校对：A是传奇

2
00:00:05,425 --> 00:00:08,700
Hello大家好,我还是那个ZOMI

3
00:00:08,700 --> 00:00:10,683
现在还是在推理引擎的

4
00:00:10,683 --> 00:00:13,894
模型离线优化里面的计算图优化这个内容

5
00:00:15,331 --> 00:00:16,892
在上一集里面呢

6
00:00:16,892 --> 00:00:20,800
讲了很多常量折叠的一些具体的parts还有具体的规则

7
00:00:20,800 --> 00:00:24,400
下面呢其实又讲了很多冗余节点的消除

8
00:00:24,400 --> 00:00:26,900
其实冗余节点的消除特别特别的多

9
00:00:27,100 --> 00:00:30,200
总结起来都有20多条了,常量折叠有十几条了

10
00:00:30,600 --> 00:00:32,500
那接下来的内容呢快速的过一下

11
00:00:32,500 --> 00:00:36,600
算子融合、算子替换还有算子迁移

12
00:00:37,100 --> 00:00:39,500
所以说在基础的图优化里面

13
00:00:40,200 --> 00:00:44,100
ZOMI看到了有些开源项目里面的计算图优化

14
00:00:44,500 --> 00:00:47,500
就有50多甚至上百个parts

15
00:00:47,500 --> 00:00:49,000
这里面就有非常多的parts

16
00:00:49,000 --> 00:00:51,200
而这里面的只是一个Basic，第一哦

17
00:00:51,600 --> 00:00:55,400
还有三还有非常多的不同的优化的parts

18
00:00:55,800 --> 00:00:57,400
那现在呢马上开始

19
00:00:57,700 --> 00:00:59,400
算子融合这个内容

20
00:01:01,800 --> 00:01:04,800
算子融合这个概念呢其实我觉得大家都知道了

21
00:01:09,600 --> 00:01:12,100
这些基本上都可以做很多融合的方式

22
00:01:12,400 --> 00:01:14,700
这里面呢确实融合的规则呢也有很多

23
00:01:15,000 --> 00:01:18,000
只是OP的一些线性的融合

24
00:01:18,000 --> 00:01:22,700
线性融合呢就是说相邻的OP存在线性上可融合的关系

25
00:01:22,800 --> 00:01:25,000
线性上呢可以从数学层面呢

26
00:01:25,300 --> 00:01:27,200
去把它们通过线性的变换

27
00:01:27,200 --> 00:01:28,500
或者数学的线性组合

28
00:01:28,800 --> 00:01:30,900
把它们变成一个相同的算子

29
00:01:30,900 --> 00:01:32,000
或者变成一个大算子

30
00:01:32,500 --> 00:01:35,000
那像这种呢叫做OP的线性融合

31
00:01:35,300 --> 00:01:37,000
那从卷积来看呢

32
00:01:37,000 --> 00:01:39,300
卷积BN ADD就卷积BN激活

33
00:01:39,700 --> 00:01:40,900
卷积BN ADD啦

34
00:01:41,400 --> 00:01:42,600
卷积SCALE ADD啦

35
00:01:42,900 --> 00:01:44,300
卷积MatMul ADD啦

36
00:01:44,300 --> 00:01:45,800
这种就卷积加很多

37
00:01:45,800 --> 00:01:47,900
其实都可以做很多非常的融合

38
00:01:48,300 --> 00:01:51,700
那假设像BN呢就可以把BN的那些Gamma、Beta

39
00:01:51,900 --> 00:01:54,700
其实融合到卷积参数里面

40
00:01:54,900 --> 00:01:56,200
那卷积参数呢就有两个

41
00:01:56,200 --> 00:01:58,200
一个是Weight一个是Bias

42
00:01:58,600 --> 00:02:00,700
所以说一般呢都可以把很多的数呢

43
00:02:00,700 --> 00:02:04,400
提前算到Weight和Bias里面两个方式

44
00:02:04,600 --> 00:02:06,200
下面看一下具体的图啊

45
00:02:07,700 --> 00:02:09,100
我做这一期的时候呢

46
00:02:09,100 --> 00:02:12,400
画图就花了我基本上三四天的时间了

47
00:02:12,400 --> 00:02:13,600
三四天业余的时间

48
00:02:13,800 --> 00:02:14,400
所以还是

49
00:02:15,400 --> 00:02:16,300
图还是很难的

50
00:02:16,300 --> 00:02:18,500
所以欢迎大家去取阅或者拿来用

51
00:02:18,900 --> 00:02:20,300
那声明来源就好了

52
00:02:20,600 --> 00:02:22,900
像可以看到卷积BN ADD呢

53
00:02:22,900 --> 00:02:24,100
卷积BN的公式呢

54
00:02:24,100 --> 00:02:25,600
可以看到在训练的时候呢

55
00:02:25,600 --> 00:02:27,600
就已经训练好Bias跟Mean了嘛

56
00:02:28,200 --> 00:02:29,400
像权重的B呢

57
00:02:29,400 --> 00:02:32,200
就可以通过这种方式呢去重新的计算

58
00:02:32,400 --> 00:02:33,800
那最后呢就变成一个具体的

59
00:02:33,800 --> 00:02:34,700
只有一个卷积了

60
00:02:34,700 --> 00:02:36,400
像激活呢基本上都可以融进去

61
00:02:36,800 --> 00:02:38,900
那好像里面的一个ADD的Const

62
00:02:39,200 --> 00:02:40,900
就可以融合到Bias里面

63
00:02:41,200 --> 00:02:42,600
像SCALE里面的SCALE呢

64
00:02:42,600 --> 00:02:43,600
还有Bias呢

65
00:02:43,600 --> 00:02:44,800
就可以融合到Bias

66
00:02:44,800 --> 00:02:46,000
还有Weight里面

67
00:02:46,300 --> 00:02:47,600
同样的方式卷积呢

68
00:02:47,600 --> 00:02:49,400
可以做非常多的融合

69
00:02:50,100 --> 00:02:52,900
下面还是在图算融合里面的

70
00:02:52,900 --> 00:02:55,100
OPS的一种线性的融合

71
00:02:55,300 --> 00:02:56,400
线性融合有非常多

72
00:02:56,400 --> 00:02:58,200
刚才只是举了一些卷积啊

73
00:02:58,200 --> 00:02:59,600
可以看到线性融合有

74
00:02:59,600 --> 00:03:00,800
MATMUL加ADD呢

75
00:03:00,800 --> 00:03:02,000
MATMUL加SCALE呢

76
00:03:02,000 --> 00:03:02,900
MEAN加ADD呢

77
00:03:02,900 --> 00:03:04,000
BATCHROM加SCALE呢

78
00:03:04,300 --> 00:03:05,600
MATMUL加BATCHROM呢

79
00:03:05,600 --> 00:03:06,600
MATMUL加ADD

80
00:03:06,600 --> 00:03:09,300
大家觉得可以自己创新很多的Path

81
00:03:09,300 --> 00:03:10,800
或者自己能想到很多的Path

82
00:03:11,000 --> 00:03:12,000
但除了自己想到

83
00:03:12,000 --> 00:03:14,800
更多的是一些实际场景来去驱动的

84
00:03:14,800 --> 00:03:17,400
因为MEAN确实后面可以加很多不同的算子

85
00:03:17,500 --> 00:03:19,600
也做很多的新的创新

86
00:03:19,600 --> 00:03:22,000
那可以看到像MATMUL加ADD

87
00:03:22,000 --> 00:03:24,200
可以把ADD这个参数变成GEMM

88
00:03:24,200 --> 00:03:25,600
这种相乘

89
00:03:26,500 --> 00:03:28,500
在MATMUL前面有个SCALE或者DIV

90
00:03:28,500 --> 00:03:30,700
确实也可以把它融合进来

91
00:03:30,900 --> 00:03:32,200
像MEAN跟ADD呢

92
00:03:32,200 --> 00:03:33,600
就可以把它变成一个

93
00:03:33,600 --> 00:03:36,300
Layer Norm的方式做一个简单的融合

94
00:03:36,600 --> 00:03:39,500
所以说算子融合的方式特别特别的多

95
00:03:40,000 --> 00:03:42,300
这里面也是一节是讲不完的

96
00:03:42,300 --> 00:03:43,500
只是简单的串一串

97
00:03:43,500 --> 00:03:45,500
给大家知道一下有这么一个事情

98
00:03:45,800 --> 00:03:46,100
就好了

99
00:03:46,100 --> 00:03:47,200
大家听听就完了

100
00:03:47,500 --> 00:03:48,300
当个开心

101
00:03:49,700 --> 00:03:52,300
后面还有OP的一些激活的融合

102
00:03:52,300 --> 00:03:53,500
就卷积加ReLU

103
00:03:53,500 --> 00:03:54,500
卷积加ReLU6

104
00:03:54,600 --> 00:03:56,500
还有卷积加其他的Act

105
00:03:56,500 --> 00:03:58,000
基本上都可以做融合

106
00:03:58,000 --> 00:03:59,000
这是很重要

107
00:03:59,000 --> 00:03:59,800
为什么要这么做

108
00:03:59,900 --> 00:04:03,000
确实它可以减少第二次访存

109
00:04:03,800 --> 00:04:06,400
卷积的时候我可能访存有两三次

110
00:04:06,400 --> 00:04:08,700
第一次去取里面的输入的数据

111
00:04:08,700 --> 00:04:10,400
然后去取Wid

112
00:04:10,400 --> 00:04:11,700
还有去取Bias

113
00:04:11,700 --> 00:04:13,200
相比ReLU输出之后

114
00:04:13,300 --> 00:04:14,700
又要取输入

115
00:04:14,700 --> 00:04:16,300
这个时候把它融合在一起

116
00:04:16,400 --> 00:04:19,300
确实能够减少访存的次数

117
00:04:19,300 --> 00:04:21,100
还可以加快计算的时间

118
00:04:21,100 --> 00:04:24,300
不用换出HBM解答的减少了

119
00:04:25,400 --> 00:04:26,600
接着可以看一下

120
00:04:26,600 --> 00:04:27,700
看完算子融合之后

121
00:04:27,700 --> 00:04:29,800
看看算子的替换

122
00:04:29,800 --> 00:04:31,800
那算子的替换就真的很简单

123
00:04:31,800 --> 00:04:33,000
就一Paste一个Node

124
00:04:33,000 --> 00:04:34,100
或者一Paste一个OD

125
00:04:34,100 --> 00:04:35,300
变成另外一个OD

126
00:04:35,700 --> 00:04:36,900
这里面有几种方式

127
00:04:36,900 --> 00:04:38,100
一种是1 to 1

128
00:04:38,100 --> 00:04:40,100
就一个算子换一个算子

129
00:04:40,100 --> 00:04:42,700
像MatMul就直接换成卷积

130
00:04:43,000 --> 00:04:44,700
这种也是很好的一个优化

131
00:04:45,600 --> 00:04:48,500
像Linear全连接方式变成卷积

132
00:04:48,500 --> 00:04:49,900
就变成一个1x1的卷积

133
00:04:49,900 --> 00:04:51,100
不是通用的卷积

134
00:04:51,300 --> 00:04:53,700
像BN的原理是等价于Scale的

135
00:04:53,700 --> 00:04:56,700
这个时候其实也可以通过Scale来去换算

136
00:04:56,700 --> 00:04:59,200
那Scale的计算方式其实更少

137
00:04:59,200 --> 00:05:00,100
像pReLU

138
00:05:00,100 --> 00:05:02,000
其实可以在真正推理的时候

139
00:05:02,000 --> 00:05:03,400
换成LeakyReLU

140
00:05:03,400 --> 00:05:05,300
其实真的是不影响精度的

141
00:05:05,300 --> 00:05:07,500
而且有可能精度还有提升

142
00:05:07,500 --> 00:05:11,300
所以说基本上1 to 1的算子替换有非常多

143
00:05:11,300 --> 00:05:13,800
折叠注意的就是像Matmul

144
00:05:13,900 --> 00:05:15,500
虽然是1 to 1的替换

145
00:05:15,500 --> 00:05:16,500
替换成卷积

146
00:05:16,500 --> 00:05:17,400
但注意的时候

147
00:05:17,400 --> 00:05:22,200
MatMul的输入是一个二维的数据的一个相乘

148
00:05:22,200 --> 00:05:23,200
A乘以B

149
00:05:23,200 --> 00:05:24,700
然后乘以B乘以A

150
00:05:24,700 --> 00:05:26,000
这两个矩阵相乘

151
00:05:26,000 --> 00:05:28,400
就得到了一个A乘以A的矩阵

152
00:05:28,400 --> 00:05:31,200
那这个时候卷积输入的数据维度

153
00:05:31,200 --> 00:05:33,100
一般都是四维的

154
00:05:33,100 --> 00:05:34,200
NCHW

155
00:05:34,200 --> 00:05:36,500
所以这个时候需要对两维的数据

156
00:05:36,500 --> 00:05:37,900
进行一个Reshape

157
00:05:38,200 --> 00:05:40,200
对Input第二个数据

158
00:05:40,200 --> 00:05:41,600
进行一个Transpose

159
00:05:41,700 --> 00:05:43,800
然后再给它进行一个运算的

160
00:05:43,800 --> 00:05:44,700
所以大家注意

161
00:05:44,700 --> 00:05:47,800
像这里面一个全连接变成一个卷积1乘1的

162
00:05:47,800 --> 00:05:49,800
也是需要进行一个Reshape

163
00:05:49,800 --> 00:05:52,300
输出也是进行一个Reshape就好了

164
00:05:52,300 --> 00:05:54,600
简单的改改它的一个内存排布

165
00:05:54,600 --> 00:05:57,700
还有BN确实可以把它变成一个Scale的方式

166
00:05:57,700 --> 00:05:59,200
这也是具体的计算

167
00:05:59,200 --> 00:06:01,900
那像PW就变成一个LeakyReLU

168
00:06:01,900 --> 00:06:04,500
看图说话总是这么的简单

169
00:06:04,500 --> 00:06:07,600
接下来再看看一些移换多

170
00:06:07,600 --> 00:06:10,200
就是一个算子换成多个算子

171
00:06:10,200 --> 00:06:11,600
能够减少推理引擎

172
00:06:11,600 --> 00:06:13,800
要实现很多很多不同的算子

173
00:06:13,800 --> 00:06:16,400
就是一个大kernel换成一个小的

174
00:06:17,600 --> 00:06:19,400
为什么会出现一换多

175
00:06:19,400 --> 00:06:21,100
有一个算子换成多个算子

176
00:06:21,100 --> 00:06:23,400
是因为在推理引擎里面假设

177
00:06:23,400 --> 00:06:24,600
我没有支持这个算子

178
00:06:24,600 --> 00:06:28,700
但这个算子可以通过很多小算子进行拼接的

179
00:06:28,700 --> 00:06:31,100
那这个时候离线总和优化模块

180
00:06:31,100 --> 00:06:33,400
就可以做一些移换多的方式

181
00:06:34,300 --> 00:06:36,800
像ShuffleNet里面就有ShuffleChannel

182
00:06:36,800 --> 00:06:40,000
ShuffleChannel可能有一些推理引擎没有实现

183
00:06:40,000 --> 00:06:42,100
于是就可以通过Reshape加Permute

184
00:06:42,200 --> 00:06:43,500
这种方式进行组合

185
00:06:43,500 --> 00:06:46,900
像Pad确实有些AI框架会有Pad-2

186
00:06:46,900 --> 00:06:48,900
或者其他的方式也可以转换

187
00:06:48,900 --> 00:06:52,200
像ShapeN是TensorFlow里面特有的一种算子

188
00:06:52,200 --> 00:06:55,100
也可以通过多个Shape的算子进行转换

189
00:06:55,100 --> 00:06:57,500
像Group卷积也可以通过

190
00:06:57,500 --> 00:07:01,100
Slice加Group进行一个替换

191
00:07:01,100 --> 00:07:03,300
所以说里面的方式特别特别的多

192
00:07:03,300 --> 00:07:04,900
举简单一个例子

193
00:07:04,900 --> 00:07:06,600
像ShuffleChannel确实可以通过

194
00:07:06,600 --> 00:07:09,700
Reshape加Permute这种方式去进行一个转换

195
00:07:09,800 --> 00:07:11,500
具体的为什么可以这么转

196
00:07:11,500 --> 00:07:13,000
大家也可以推理一下

197
00:07:13,000 --> 00:07:16,300
像这里面的卷积是Group不等于1

198
00:07:16,300 --> 00:07:20,800
可以把它Slice成Group跟Number的一个卷积的参数

199
00:07:21,500 --> 00:07:22,200
有Group个

200
00:07:22,200 --> 00:07:24,100
这里面的卷积Group就等于0

201
00:07:24,100 --> 00:07:25,400
就把它Concat到一起

202
00:07:25,400 --> 00:07:26,800
那这种就替换掉了

203
00:07:27,400 --> 00:07:30,000
这样就可以去实现推理引擎里面

204
00:07:30,000 --> 00:07:31,000
本来没有这些算子

205
00:07:31,000 --> 00:07:34,200
但是可以通过一些算子的组合进行一个替换

206
00:07:36,500 --> 00:07:38,800
在计算图优化里面的一个Basic

207
00:07:38,900 --> 00:07:40,900
最基础的还有最后一个内容

208
00:07:40,900 --> 00:07:42,700
就是算子的前移

209
00:07:42,700 --> 00:07:43,900
算子的前移有比较多

210
00:07:43,900 --> 00:07:45,100
像是Slice跟Mul

211
00:07:45,100 --> 00:07:46,600
还有BitShift跟Reduce

212
00:07:46,600 --> 00:07:49,500
上面这些都可以把它替换掉位置

213
00:07:49,500 --> 00:07:51,300
把它往前挪

214
00:07:52,800 --> 00:07:53,700
而算子前移

215
00:07:53,700 --> 00:07:56,500
其实我觉得大家其实不要觉得

216
00:07:56,500 --> 00:07:58,300
你要去发现规律

217
00:07:58,300 --> 00:07:59,000
更多的时候

218
00:07:59,000 --> 00:08:01,900
可以利用算数的一个交换率

219
00:08:01,900 --> 00:08:03,200
去考虑这个问题

220
00:08:03,200 --> 00:08:04,900
可不可以这么去操作

221
00:08:04,900 --> 00:08:07,700
就是算数的计算的过程当中

222
00:08:07,700 --> 00:08:09,000
能不能通过交换率

223
00:08:09,000 --> 00:08:10,600
减少数据的传输

224
00:08:10,600 --> 00:08:12,200
还有访存的次数

225
00:08:12,200 --> 00:08:13,000
这一点很重要

226
00:08:13,000 --> 00:08:16,400
就大家要去真正的站在问题的本质去看问题

227
00:08:16,400 --> 00:08:18,000
而不是为了发现Path

228
00:08:18,000 --> 00:08:19,600
发现创造不同的规律

229
00:08:19,600 --> 00:08:20,800
创造规律

230
00:08:20,800 --> 00:08:24,200
更多的要结合真正的场景和数学的研拟

231
00:08:24,200 --> 00:08:26,100
那下面可以看到像这种

232
00:08:26,100 --> 00:08:28,500
就是算子前移的一个很经典的案例

233
00:08:28,500 --> 00:08:29,300
我一个Mul

234
00:08:29,300 --> 00:08:30,000
然后在Slice

235
00:08:30,000 --> 00:08:31,600
我确实可以把它直接Slice掉

236
00:08:31,600 --> 00:08:33,600
然后直接再做一个Mul

237
00:08:33,600 --> 00:08:34,800
那像一个BitShift

238
00:08:34,800 --> 00:08:35,600
还有Reduce

239
00:08:35,700 --> 00:08:37,900
确实可以把它换回来

240
00:08:37,900 --> 00:08:40,300
可以减少通信的次数

241
00:08:42,000 --> 00:08:42,600
好了

242
00:08:42,600 --> 00:08:44,100
今天的内容就到这里为止

243
00:08:44,200 --> 00:08:45,300
回顾一下

244
00:08:47,200 --> 00:08:50,800
在计算图的基础图优化的这个模块

245
00:08:50,900 --> 00:08:53,000
讲了常量的折叠

246
00:08:53,000 --> 00:08:56,400
把一些不用的常量的就把它合并在一起

247
00:08:56,400 --> 00:08:58,900
其实它有点类似于冗余节点的消除

248
00:08:58,900 --> 00:09:00,600
就把一些常量把它干掉

249
00:09:00,600 --> 00:09:02,900
那接着又讲了一些冗余节点的消除

250
00:09:02,900 --> 00:09:05,200
冗余节点的消除有非常多

251
00:09:05,300 --> 00:09:07,300
我列出来的就已经快接近20个了

252
00:09:07,300 --> 00:09:09,300
然后有些算子的融合

253
00:09:09,300 --> 00:09:13,500
把很多零散的算子变成一个大的算子

254
00:09:13,500 --> 00:09:15,000
还有算子的替换

255
00:09:15,000 --> 00:09:16,400
有1对1的替换

256
00:09:16,400 --> 00:09:17,800
也有1对多的替换

257
00:09:17,800 --> 00:09:18,900
那1对多的替换

258
00:09:18,900 --> 00:09:22,000
就有点算子融合的一个逆过程

259
00:09:22,000 --> 00:09:24,300
最后还有一些算子的前移

260
00:09:24,300 --> 00:09:27,500
前移的工作确实为了减少访存的次数

261
00:09:27,500 --> 00:09:28,900
让训练的更快

262
00:09:28,900 --> 00:09:30,500
以这个目的作为驱动

263
00:09:30,500 --> 00:09:31,100
好了

264
00:09:31,100 --> 00:09:33,000
今天的内容就到这里为止

265
00:09:33,000 --> 00:09:33,600
谢谢各位

266
00:09:33,600 --> 00:09:34,800
拜了个拜

267
00:09:35,000 --> 00:09:35,800
卷的不行啦

268
00:09:35,800 --> 00:09:36,600
卷的不行啦

269
00:09:36,600 --> 00:09:38,400
记得一键三连加关注哦

270
00:09:38,400 --> 00:09:41,600
所有的内容都会开源在下面这条链接里面

271
00:09:41,600 --> 00:09:42,900
拜了个拜