-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathatom.xml
534 lines (315 loc) · 148 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>枫枝雀自鸣</title>
<subtitle>技术博客</subtitle>
<link href="/atom.xml" rel="self"/>
<link href="https://blog.mapleque.com/"/>
<updated>2020-02-26T05:45:28.951Z</updated>
<id>https://blog.mapleque.com/</id>
<author>
<name>枫雀</name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>【开始用go】http服务</title>
<link href="https://blog.mapleque.com/posts/practice/go/go-practice-http/"/>
<id>https://blog.mapleque.com/posts/practice/go/go-practice-http/</id>
<published>2019-12-17T09:00:35.000Z</published>
<updated>2020-02-26T05:45:28.951Z</updated>
<content type="html"><![CDATA[<h2 id="配置加载"><a class="markdownIt-Anchor" href="#配置加载"></a> 配置加载</h2><p>服务在启动的时候通常都需要一些配置来提供启动参数，如：监听端口，其他服务连接密钥等。</p><p>这里建议所有配置都通过环境变量读取，这样做的原因有以下几点：</p><ol><li>生产环境的系统配置可能较为复杂，可以根据不同环境调整配置不同参数</li><li>生产环境密钥应在运维范畴保密，不应透露到开发者层面</li><li>开发环境更为复杂，开发者可以根据自己的环境灵活配置启动</li></ol><p>go标准包<code>os</code>中提供了获取环境变量的方法：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"os"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line">port := os.Getenv(<span class="string">"UC_LISTEN_PORT"</span>)</span><br><span class="line"> <span class="comment">// ...</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="服务定义和启动"><a class="markdownIt-Anchor" href="#服务定义和启动"></a> 服务定义和启动</h2><p>为了避免全局变量的失控，我们通常会为服务定义一个实例，然后通过实例接口让其在main中被加载和启动。</p><p>那么对于一个http服务，我们需要为其实现http.Handler接口：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Service impement http.Handler interface</span></span><br><span class="line"><span class="comment">// which can be initialized in a http server.</span></span><br><span class="line"><span class="keyword">type</span> Service <span class="keyword">struct</span>{}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">New</span><span class="params">()</span> *<span class="title">Service</span></span> {</span><br><span class="line"><span class="keyword">return</span> &Service{}</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *Service)</span> <span class="title">ServeHTTP</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line"> <span class="comment">// TODO ...</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>然后，在main中实例化我们定义的服务，然后就可以通过标准包<code>net/http</code>提供的方法启动了：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"net/http"</span></span><br><span class="line"><span class="string">"os"</span></span><br><span class="line"></span><br><span class="line">uc <span class="string">"github.com/mapleque/gostart/ms/uc/service"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line">port := os.Getenv(<span class="string">"UC_LISTEN_PORT"</span>)</span><br><span class="line">s := uc.New()</span><br><span class="line">server.ListenAndServe(<span class="string">"0.0.0.0:"</span>+port, s)</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="路由和处理函数"><a class="markdownIt-Anchor" href="#路由和处理函数"></a> 路由和处理函数</h2><p>依据标准包<code>net/http</code>的实现，所有请求最终都会经过<code>ServeHTTP</code>方法处理，并且每个请求的处理都是一个单独的协程。</p><p>所以在<code>ServeHTTP</code>方法中，我们主要实现的就是为当前请求分配处理函数。这里也可以直接使用标准包<code>net/http</code>中的<code>ServeMux</code>实现：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Service implement http.Handler interface</span></span><br><span class="line"><span class="comment">// which can be initialized in a http server.</span></span><br><span class="line"><span class="keyword">type</span> Service <span class="keyword">struct</span> {</span><br><span class="line">mux *http.ServeMux</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// ServeHTTP implement http.Handler interface.</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *Service)</span> <span class="title">ServeHTTP</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line">s.mux.ServeHTTP(w, req)</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>很明显，要让</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *Service)</span> <span class="title">signin</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line"> <span class="comment">// TODO deal with /signin request</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *Service)</span> <span class="title">signout</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line"> <span class="comment">// TODO deal with /signout request</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *Service)</span> <span class="title">userinfo</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line"> <span class="comment">// deal with /userinfo request</span></span><br><span class="line"> <span class="keyword">switch</span> req.Method {</span><br><span class="line"> <span class="keyword">case</span> http.MethodGet:</span><br><span class="line"> s.getUserinfo(w, req)</span><br><span class="line"> <span class="keyword">case</span> http.MethodPost:</span><br><span class="line"> s.postUserinfo(w, req)</span><br><span class="line"> <span class="keyword">default</span>:</span><br><span class="line"> <span class="comment">// return 405</span></span><br><span class="line"> w.WriteHeader(http.StatusMethodNotAllowed)</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *Service)</span> <span class="title">getUserinfo</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line"> <span class="comment">// TODO deal with GET /userinfo request</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *Service)</span> <span class="title">postUserinfo</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line"> <span class="comment">// TODO deal with POST /userinfo request</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="参数定义和校验"><a class="markdownIt-Anchor" href="#参数定义和校验"></a> 参数定义和校验</h2><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> SigninParam <span class="keyword">struct</span> {</span><br><span class="line"> Username <span class="keyword">string</span> <span class="string">`json:"username"`</span></span><br><span class="line"> Password <span class="keyword">string</span> <span class="string">`json:"password"`</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">bindAndValid</span><span class="params">(req *http.Request, obj <span class="keyword">interface</span>{})</span> <span class="title">error</span></span> {</span><br><span class="line"> <span class="keyword">if</span> req == <span class="literal">nil</span> || req.Body == <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> fmt.Errorf(<span class="string">"invalid request"</span>)</span><br><span class="line"> }</span><br><span class="line"> decoder := json.NewDecoder(req)</span><br><span class="line"> <span class="keyword">if</span> err := decoder.Decode(obj); err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> err</span><br><span class="line"> }</span><br><span class="line"> <span class="comment">// use github.com/go-playground/validator/v10</span></span><br><span class="line"> <span class="keyword">return</span> validate.Struct(obj)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *Service)</span> <span class="title">signin</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line"> param := SigninParam{}</span><br><span class="line"> <span class="keyword">if</span> err := bindAndValid(req, &param); err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="comment">// TODO response param-check error message</span></span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line"> }</span><br><span class="line"> <span class="comment">// TODO do login</span></span><br><span class="line"> <span class="comment">// TODO response successful data</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="处理返回"><a class="markdownIt-Anchor" href="#处理返回"></a> 处理返回</h2><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> Response <span class="keyword">struct</span> {</span><br><span class="line"> Status <span class="keyword">int</span> <span class="string">`json:"status"`</span></span><br><span class="line"> Data <span class="keyword">interface</span>{} <span class="string">`json:"data"`</span></span><br><span class="line"> Message <span class="keyword">interface</span>{} <span class="string">`json:"message"`</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> UserinfoResponse <span class="keyword">struct</span> {</span><br><span class="line"> Token <span class="keyword">string</span> <span class="string">`json:"token"`</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">response</span><span class="params">(w, resp <span class="keyword">interface</span>{})</span></span> {</span><br><span class="line"> encoder := json.NewEncoder(w)</span><br><span class="line"> err := encoder.Encode(resp)</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="built_in">panic</span>(err)</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *Service)</span> <span class="title">signin</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line"> param := SigninParam{}</span><br><span class="line"> <span class="keyword">if</span> err := bindAndValid(req, &param); err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="comment">// response param-check error message</span></span><br><span class="line"> <span class="comment">//</span></span><br><span class="line"> <span class="comment">// To custom with the error message,</span></span><br><span class="line"> <span class="comment">// use https://github.com/go-playground/universal-translator</span></span><br><span class="line"> <span class="comment">// see example at: https://github.com/go-playground/validator/blob/master/_examples/translations/main.go#L105</span></span><br><span class="line"> response(w, Response{StatusInvalidParam, <span class="literal">nil</span>, MessageInvalidParam}</span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line"> }</span><br><span class="line"> <span class="comment">// ... do login</span></span><br><span class="line"> <span class="comment">// response successful data</span></span><br><span class="line"> response(w, Response{StatusSuccess, UserinfoResponse{token}, <span class="literal">nil</span>})</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="调用其他服务"><a class="markdownIt-Anchor" href="#调用其他服务"></a> 调用其他服务</h2><p>这里以使用mysql服务为例，使用<a href="https://github.com/go-sql-driver/mysql" target="_blank" rel="noopener"><code>github.com/go-sql-driver/mysql</code>包</a>。</p><p>首先，想要在handle中使用mysql服务，就需要能够获取mysql的连接，通常我们会将服务连接池初始化放在服务初始化时进行：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> Service <span class="keyword">struct</span> {</span><br><span class="line"> mux *http.ServeMux</span><br><span class="line"> <span class="comment">// add a db property</span></span><br><span class="line"> db *sql.DB</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="日志输出"><a class="markdownIt-Anchor" href="#日志输出"></a> 日志输出</h2>]]></content>
<summary type="html">
<h2 id="配置加载"><a class="markdownIt-Anchor" href="#配置加载"></a> 配置加载</h2>
<p>服务在启动的时候通常都需要一些配置来提供启动参数，如：监听端口，其他服务连接密钥等。</p>
<p>这里建议所有配置都通过环境变量读
</summary>
<category term="实践" scheme="https://blog.mapleque.com/categories/practice/"/>
<category term="开始用go" scheme="https://blog.mapleque.com/categories/practice/go/"/>
<category term="golang" scheme="https://blog.mapleque.com/tags/golang/"/>
</entry>
<entry>
<title>【互联网产品的诞生和演化】快速迭代和持续集成</title>
<link href="https://blog.mapleque.com/posts/practice/0ws/0ws-cicd/"/>
<id>https://blog.mapleque.com/posts/practice/0ws/0ws-cicd/</id>
<published>2019-12-12T07:27:53.000Z</published>
<updated>2020-02-26T04:25:47.193Z</updated>
<content type="html"><![CDATA[<p>在互联网产品开发过程中，效率和稳定性始终是核心问题。</p><p>开发效率决定了产品的迭代速度。当一个产品需要与市场上同类产品竞争时，迭代速度就会显得尤为重要。<br>别人新上了一个很受欢迎的功能而你没有在几天之内跟上，那么你的用户将会大规模流失。</p><p>同时，如果你的服务不稳定，就会给用户一种不信任的感觉，进而动摇用户留下来的信心，也会造成用户流失。</p><p>因此，在保证系统稳定性的同时尽可能加快项目开发速度就是我们所追求的极致。</p><p>通常来讲需要考虑以下一些过程。</p><h2 id="单元测试"><a class="markdownIt-Anchor" href="#单元测试"></a> 单元测试</h2><p>单元测试是指针对代码的测试，以确保每一行代码都能按照预想的方式执行。</p><p>任何一个系统功能，都应该有完整的单元测试。单元测试可以使开发者减少大量维护已有系统的精力，同时保证发布新功能时的稳定性。</p><p>这里我们要求单元测试的覆盖率尽可能达到100%。</p><p>这里推荐使用<a href>go语言http服务单元测试组件</a>。</p><h2 id="功能测试"><a class="markdownIt-Anchor" href="#功能测试"></a> 功能测试</h2><p>功能测试是面向功能的测试，在互联网服务中，通常以接口或者接口组为单位进行。</p><p>所有功能测试的测试用例应当紧跟产品需求，作为系统发布的最终屏障，来保证系统稳定性。</p><h2 id="自动发布"><a class="markdownIt-Anchor" href="#自动发布"></a> 自动发布</h2><p>发布是系统交互的一个过程。当一个互联网产品开发完成后，通常都需要集成发布，包括但不限于：</p><ul><li>编译</li><li>打包</li><li>更新版本</li><li>更新文档</li><li>更新日志</li></ul><h2 id="自动部署"><a class="markdownIt-Anchor" href="#自动部署"></a> 自动部署</h2><p>让一个互联网产品能够被用户所使用，需要将指定的发布版本部署到服务器上，这个过程包括但不限于：</p><ul><li>运行指定发布版本</li><li>新建或修改数据表</li><li>新增或修改数据</li><li>新增或修改配置等</li></ul>]]></content>
<summary type="html">
解决开发效率和稳定性问题，是互联网产品开发成功的关键。
</summary>
<category term="实践" scheme="https://blog.mapleque.com/categories/practice/"/>
<category term="互联网产品的诞生和演化" scheme="https://blog.mapleque.com/categories/practice/0ws/"/>
<category term="Web" scheme="https://blog.mapleque.com/tags/Web/"/>
</entry>
<entry>
<title>【互联网产品的诞生和演化】统计数据收集展示和系统监控</title>
<link href="https://blog.mapleque.com/posts/practice/0ws/0ws-stats/"/>
<id>https://blog.mapleque.com/posts/practice/0ws/0ws-stats/</id>
<published>2019-12-11T06:57:09.000Z</published>
<updated>2020-02-26T04:25:47.194Z</updated>
<content type="html"><![CDATA[<p>互联网产品在线上之前，就要考虑统计数据和监控。</p><p>初期，为了节省成本，数据统计可以直接使用第三方平台提供的服务，如：百度统计、谷歌统计等。</p><p>这些第三方平台提供了丰富多彩的统计服务，唯一缺陷就是数据在第三方，我们自己无法随心所欲使用数据，只能受限于第三方可用功能。</p><p>系统监控可以考虑直接监控服务器错误日志。</p><p>当第三方数据服务已经无法满足我们的要求时，就需要考虑自建数据平台了。</p><p>一个典型的数据平台架构如下：</p><pre class="mermaid" style="text-align: center;"> graph LR source[数据源]collection[数据收集]storage[数据存储]analysis[数据分析]showing[数据展示]source --> collectioncollection --> storagestorage --> analysisanalysis --> storagestorage --> showing </pre><p>这里边，数据存储是核心。</p><h2 id="数据存储"><a class="markdownIt-Anchor" href="#数据存储"></a> 数据存储</h2><p>影响数据存储选型的因素有很多，需要重点考虑的核心因素有三：</p><ul><li>数据规模</li><li>数据结构</li><li>数据使用方式</li></ul><p>对于不同的选型，上述三个因素的取舍各不相同：</p><ul><li><p>MySQL</p><p>MySQL是互联网时代最常用的数据库之一，适用于100w数量级以下的结构化数据存储，使用SQL操作数据，通常用于存储业务数据和展示数据。<br>在数据平台中，如果总数据量不大，并且增长不快，可以直接使用MySQL作为数据存储，这种存储的使用成本应该是最低的。</p></li><li><p>HDFS</p><p>HDFS是Hadoop架构下的分布式文件存储系统，适用于100w数据级以上的数据存储，使用MapReduce查询数据或通过上层中间件（如：HBase，HIVE，Spark等）操作数据，通常应用于离线数据计算和查询。<br>在数据平台中，使用HDFS至少需要3个节点以上的集群进行部署，同时还要部署和维护所需中间件以及数据访问权限，运维复杂度极高，非专业团队不建议使用。</p></li><li><p>Elasticsearch</p><p>Elasticsearch是ElasticStack中的基于Luence实现的搜索引擎，得益于其高效的全文搜索功能和数据统计功能，也可以将其用作数据平台的一种存储。<br>结合使用Kibana，可以将其作为系统监控的核心系统。<br>Kibana通过封装Elasticsearch的api，实现了数据查询、聚集以及可视化展示等功能，并且支持按条件触发事件和发送报告。<br>由于其基于Luence实现，因此不适合作为复杂的数据分析工具使用，强行使用会影响系统稳定性。</p></li><li><p>TiDB</p><p>TiDB是PingCAP开发的一款支持MySQL协议，并且可以水平扩展的分布式关系数据库。<br>由于其对MySQL协议的支持，大大降低了使用成本和维护成本。<br>其在大数据量下的表现有待考量，据笔者经验，在亿级数据量下复杂分组查询可以实现秒级返回。</p></li><li><p>Kafka</p><p>Kafka可以被看作实时一种数据存储，由于流计算需求的存在，使得Kafka成为一个不可替代的存储选型。</p></li></ul><p>事实上，在实际应用中，根据不同需求，可以将不同存储组合使用。例如：</p><ul><li>将需要展示的数据，计算后存储到MySQL中</li><li>将冷数据归档到HDFS中</li><li>在Elasticsearch中仅保留一定规模的热数据用于问题跟踪和系统监控</li><li>TiDB仅用于数据分析，不用于业务服务和数据展示</li></ul><h2 id="数据源"><a class="markdownIt-Anchor" href="#数据源"></a> 数据源</h2><p>从技术角度来讲，统计和监控都依赖于日志数据的收集和汇总，所需的日志通常有：</p><ul><li>前端和客户端行为打点日志</li><li>前端和客户端错误上报日志</li><li>后端接口请求日志</li><li>后端异常错误日志</li><li>探针日志</li></ul><p>以上日志，都可以作为数据源进行收集，并进一步处理。</p><h2 id="数据收集"><a class="markdownIt-Anchor" href="#数据收集"></a> 数据收集</h2><p>由于我们的数据源都是日志，因此可以使用filebeat部署到日志服务器进行收集。</p><p>这里建议filebeat的消费端使用Kafka以使服务器的filebeat不发生阻塞也不需要回滚。</p><div class="note warning"> <p>注意：logstash非常耗费资源，如果资源并出充足，建议自行脚本实现从Kafka消费到数据存储的流程。</p> </div><p>filebeat的安装命令：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.1-amd64.deb</span><br><span class="line">sudo dpkg -i filebeat-6.3.1-amd64.deb</span><br><span class="line">sudo systemctl start filebeat</span><br><span class="line">sudo systemctl <span class="built_in">enable</span> filebeat</span><br></pre></td></tr></table></figure><p>filebeat的配置示例如下：</p><figure class="highlight yaml"><figcaption><span>/etc/filebeat/filebeat.yml</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="string">filebeat.prospectors:</span></span><br><span class="line"><span class="comment"># 根据情况自行配置需要收集哪些log</span></span><br><span class="line"><span class="attr">- type:</span> <span class="string">log</span></span><br><span class="line"><span class="attr"> paths:</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string"><log_path></span></span><br><span class="line"></span><br><span class="line"><span class="string">output.kafka:</span></span><br><span class="line"><span class="attr"> enable:</span> <span class="literal">true</span></span><br><span class="line"><span class="attr"> hosts:</span> <span class="string">['<kafka_ip>']</span></span><br><span class="line"></span><br><span class="line"><span class="attr"> topic:</span> <span class="string">'<string>'</span></span><br><span class="line"><span class="attr"> compression:</span> <span class="string">gzip</span></span><br><span class="line"><span class="attr"> max_message_bytes:</span> <span class="number">1000000</span></span><br><span class="line"></span><br><span class="line"><span class="string">queue.mem:</span></span><br><span class="line"><span class="attr"> events:</span> <span class="number">512</span></span><br><span class="line"> <span class="string">flush.min_events:</span> <span class="number">256</span></span><br><span class="line"> <span class="string">flush.timeout:</span> <span class="number">5</span><span class="string">s</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 监控可不配置</span></span><br><span class="line"><span class="string">xpack.monitoring:</span></span><br><span class="line"><span class="attr"> enabled:</span> <span class="literal">true</span></span><br><span class="line"><span class="attr"> elasticsearch:</span></span><br><span class="line"><span class="attr"> hosts:</span> <span class="string">["<es_ip>"]</span></span><br><span class="line"><span class="attr"> username:</span> <span class="string"><string></span></span><br><span class="line"><span class="attr"> password:</span> <span class="string">"<string>"</span></span><br></pre></td></tr></table></figure><h2 id="数据分析"><a class="markdownIt-Anchor" href="#数据分析"></a> 数据分析</h2><p>数据分析是对现有数据的再加工过程，通常涉及：变形，关联，统计等。目前最适合进行数据分析的语言应该是Python，上文中的数据存储，都可以原生支持Python。</p><h2 id="数据展示"><a class="markdownIt-Anchor" href="#数据展示"></a> 数据展示</h2><p>数据展示的基本形式是图表，图可以将数据特征更直观的展示出来，<a href="https://github.com/antvis/G2Plot" target="_blank" rel="noopener">G2Plot</a>提供了图表的完整封装，通过简单的前端项目构建和后端接口查询，即可做到优雅的数据可视化展示。</p><p>注意，不要在数据展示期间进行数据计算。</p><p>当然，想要展示实时数据，可以考虑尝试使用TiDB。</p>]]></content>
<summary type="html">
统计和监控可以让你对产品了如指掌，进而看清未来发展方向。
</summary>
<category term="实践" scheme="https://blog.mapleque.com/categories/practice/"/>
<category term="互联网产品的诞生和演化" scheme="https://blog.mapleque.com/categories/practice/0ws/"/>
<category term="Web" scheme="https://blog.mapleque.com/tags/Web/"/>
</entry>
<entry>
<title>【互联网产品的诞生和演化】独立应用的基本架构</title>
<link href="https://blog.mapleque.com/posts/practice/0ws/0ws-base/"/>
<id>https://blog.mapleque.com/posts/practice/0ws/0ws-base/</id>
<published>2019-11-29T10:19:30.000Z</published>
<updated>2020-02-26T04:25:47.193Z</updated>
<content type="html"><![CDATA[<div class="note info"> <p>一个新的互联网产品研发并不简单，当你不确定产品形态是否可行时，请先想办法验证，不要盲目进入开发。<br>如何进行低成本验证，请参考上一篇文章：<a href="/posts/practice/0ws/0ws-simple">产品的最简模型</a>。</p> </div><p>得益于移动互联网时代的飞速发展，开发一个互联网产品已经成为人尽可谈的话题。</p><p>然而在这样的话题的讨论中，通常得到的结果都是：我们的点子非常好，就是缺一个开发。</p><p>不可否认，任何一个想法，如果能把它做成一个App、一个网站或者一个小程序，多半情况下它都是可用的，想想就很兴奋。</p><p>但是想要真正将这个开发过程付诸实施，还有一个完整的开发团队的距离。</p><p>简单来说，一个完整的开发团队，需要以下配置具有以下特殊技能的人员：<br>设计、前端开发、客户端开发、后端开发、测试、运维。<br>{在后面<a href="#%E6%8A%80%E8%83%BD%E9%9C%80%E6%B1%82">技能需求</a>章节有对这些技能应用在何处的详细说明。）</p><p>当然，不排除有些人是具有多种技能的综合型人才。</p><p>如果运气好，上面所有人员都能胜任，那么至少可以开发出一个初始版本的系统了，接下来考虑线上部署，需要采购并长期维护以下资源：<br>域名（备案）、ssl证书、云主机实例、云数据库实例、cdn服务。<br>{在后面<a href="#%E8%B5%84%E6%BA%90%E9%9C%80%E6%B1%82">资源需求</a>章节有对这些资源应用在何处的详细说明。}</p><p>至此，开发一个崭新的互联网服务的必备条件已经完全具备了，最后我们需要考虑的就是项目周期。</p><div class="note info"> <p>项目开发周期=每个成员完成任务所需要的时间+成员间沟通所需时间-可并行时间</p> </div><p>很明显，根据上面公式来看，想要缩短开发时间，只有两条路可以走：</p><ul><li>选择能力更高的人以降低单人工作时间和沟通时间</li><li>通过合理规划增加可并行时间（<a href="#%E5%B7%A5%E4%BD%9C%E6%B5%81">工作流</a>章节给出了一个通常情况下的最优规划方案）</li></ul><h2 id="基本技术架构"><a class="markdownIt-Anchor" href="#基本技术架构"></a> 基本技术架构</h2><p>在上文中，我们已经直接探讨了开发一个互联网产品所需要考虑的全部内容。然而为什么是这样，这就需要从基本架构先说起。</p><p>参考下图：</p><pre class="mermaid" style="text-align: center;"> graph LR fe[网站或微信小程序]app[移动应用App]api[后端接口]services[后端服务]db[数据库服务]3rd-api[第三方服务接口]admin[运营后台]admin-api[后台接口]fe --> apiapp --> apiapi --> servicesservices --> db3rd-api --> servicesservices --> 3rd-apiadmin --> admin-apiadmin-api --> services </pre><p>对于一个互联网产品，客户端大多会考虑实现两种形式：<code>网站</code>或<code>移动应用App</code>。其中，</p><ul><li>网站 ---- 开发成本低，迭代周期快</li><li>移动应用App ---- 产品形象固定，用户粘性高</li></ul><div class="note info"> <p>随着<code>微信小程序</code>生态的日益完善，这类客户端形态也逐渐被大众所考虑。<br>微信小程序的优势在于，它兼具了网站和移动应用App两种类型客户端的优势，并且天生可以依托微信这个成熟且巨大的用户生态圈。</p> </div><p>在客户端之后，还需要服务端来实现产品逻辑。服务端的内容，大致可以采用如下方式划分：</p><ul><li>后端接口 ---- 作为用户与服务交流的通道，让用户在客户端上的操作可以得到对应的响应。</li><li>后端服务 ---- 实现复杂的产品逻辑，如：注册登录、购买支付、分享邀请等。</li><li>数据库服务 ---- 用于记录数据，包括但不限于：用户信息、用户购买信息、用户操作记录等。</li><li>第三方服务接口 ---- 接入其他已经实现了的互联网服务，如：微信登录、微信或支付宝支付等。</li><li>运营后台 ---- 运营人员维护产品数据的平台，通常是一个网站，可运营的内容可能包括产品信息价格、用户账号数据等。</li><li>后台接口 ---- 作为运营人员与服务交流的通道，使其能够查询管理大部分产品相关数据。</li></ul><p>整体来讲，任何一个互联网产品，在设计开发之初，都应该考虑以上所有内容。<br>一些情况下，部分内容可能或有所取舍，比如：</p><ul><li>不需要开发移动应用App，只要一个网站就够了</li><li>不需要后端接口和后端服务，因为不需要记录任何用户数据，只需要展示一些内容</li><li>不需要运营后台，我可以直接通过数据库查询和修改需要的数据</li></ul><h2 id="技能需求"><a class="markdownIt-Anchor" href="#技能需求"></a> 技能需求</h2><p>如果按照上面所讲的基本技术架构进行开发，那么我们需要以下相关技能：</p><ul><li>ui&ue ---- 决定产品长相和操作方式，通常需要艺术设计专业知识</li><li>fe<ul><li>h5 ---- 开发pc端和移动端网页产品，需要掌握html5+javascript+css基础能力和主流框架使用经验</li><li>wx ---- 开发微信小程序或服务号相关产品，需要熟悉微信开放接口和微信小程序开发标准</li><li>admin ---- 开发运营后台，熟悉后台相关框架以及一些数据可视化技术和组件</li></ul></li><li>app<ul><li>android ---- 开发android应用，需要相关开发经验</li><li>ios ---- 开发ios应用，需要相关开发经验</li></ul></li><li>server<ul><li>buzz ---- 开发http接口服务，需要相关开发经验</li><li>service ---- 开发复杂应用服务和基础服务，需要相关开发经验</li></ul></li><li>other<ul><li>test ---- 对产品进行系统全面的测试，需要相关经验</li><li>op ----- 发布和维护所开发的服务，应对各种突发情况（如：软硬件故障、黑客攻击，版本更新等）</li></ul></li></ul><h2 id="资源需求"><a class="markdownIt-Anchor" href="#资源需求"></a> 资源需求</h2><p>对于任何一个互联网产品，想要让用户能够使用，都需要购买以下资源：</p><ul><li>域名，作为用户访问所使用的地址，如：<a href="http://www.jd.com" target="_blank" rel="noopener">www.jd.com</a>。需要在域名运营商或者代理商处购买。<br>注意在国内购买或使用的域名通常还需要备案。</li><li>https证书，用于认证域名的安全性。需要在有资质的证书管理机构或代理商处购买。</li><li>云实例，用于部署和运行服务。需要在云服务商购买。</li><li>数据库实例，用于提供数据库相关服务。需要在云服务商购买。</li><li>cdn，用于提供静态资源存储和访问服务。需要在云服务商购买。</li></ul><h2 id="工作流"><a class="markdownIt-Anchor" href="#工作流"></a> 工作流</h2><p>一个合理的工作流可以让产品开发周期缩到最短。</p><pre class="mermaid" style="text-align: center;"> graph LR prd[产品需求]uiue[视觉交互设计]server[后端设计开发]admin[后台设计开发]fe[前端开发]app[移动端开发]test[测试]release[发布]prd --> uiueprd --> serverprd --> adminuiue --> feuiue --> appserver --> feserver --> appserver --> adminfe --> testapp --> testadmin --> testtest --> release </pre><p>这里同样按照专业领域划分，从一个确定的产品需求开始。</p><ul><li>一旦产品需求确定了，就可以开始视觉交互设计和后端设计了。<br>这里边后端设计要考虑接口、逻辑和数据三个层面，对于设计人员的能力要求较高。</li><li>前端开发和移动端开发需要在两个设计都完成后开始。</li><li>后台设计开发需要在后端设计完成后开始。</li><li>当所有开发完成后进入测试环节。</li><li>最终测试通过后进行产品发布。</li></ul><p>经过多年实践和总结，这个工作流可以最大化并行时间，也就是最优化项目开发周期。</p>]]></content>
<summary type="html">
一个完整的产品技术架构，不只是客户端和服务端，需要考虑的细节还很多，这些都将在本文中一一阐述。
</summary>
<category term="实践" scheme="https://blog.mapleque.com/categories/practice/"/>
<category term="互联网产品的诞生和演化" scheme="https://blog.mapleque.com/categories/practice/0ws/"/>
<category term="Web" scheme="https://blog.mapleque.com/tags/Web/"/>
</entry>
<entry>
<title>【互联网产品的诞生和演化】产品的最简模型</title>
<link href="https://blog.mapleque.com/posts/practice/0ws/0ws-simple/"/>
<id>https://blog.mapleque.com/posts/practice/0ws/0ws-simple/</id>
<published>2019-11-29T08:54:50.000Z</published>
<updated>2020-02-26T04:25:47.194Z</updated>
<content type="html"><![CDATA[<p>互联网产品是一种：“通过互联网技术获取并留住用户，进而从用户身上获取利益的商业行为“。</p><p>AARRR模型可以用于研究这类产品的用户生命周期。</p><pre class="mermaid" style="text-align: center;"> graph LR acquisition[触达]activation[获客]retention[留存]revenue[转化]refer[传播]acquisition --> activationactivation --> retentionactivation --> referretention --> revenuerevenue --> referrefer --> activation </pre><p>在这个生命周期中，</p><ol><li><code>触达</code>就是将产品入口暴露给用户，让用户知道这个产品，并且能够通过何种途径使用产品。</li><li><code>获客</code>就是让用户真正的使用产品。</li><li><code>留存</code>通常是看一周或者一个月以后，还在持续使用产品的用户数量。</li><li><code>转化</code>即为从使用产品的用户身上获取利益。</li><li><code>传播</code>表示正在使用产品的用户将产品介绍给新人，并让其开始使用产品。</li></ol><p>事实上，在上述生命周期中，产品的核心诉求是<code>转化</code>，只有转化才能获取利益，才能让产品存活下去。想要获取更多利益，要么增加<code>获客</code>，要么提升<code>留存</code>和<code>转化</code>的效率：<code>触达</code>和<code>传播</code>都是为了增加<code>获客</code>，应用适合的推广手段是关键：而提升<code>留存</code>和<code>转化</code>的效率则需要更具吸引力的产品形态和专业的运营策略。</p><p>那么，如何依托第三方平台实现上述模型的验证？这里我们以微信群为例说明。</p><p>基于微信群实现互联网产品，通常需要以下步骤：</p><ol><li>建群，群主作为主运营人员，负责发布和维护群规</li><li>通过分享二维码<code>触达</code>潜在用户</li><li>所有进群用户都是<code>获客</code></li><li>每日统计群成员数量计算<code>留存</code></li><li>在群内发布产品内容进行<code>转化</code>（例如：加好友提供服务）</li><li>群成员拉其他人进群的数量即为<code>传播</code></li></ol><p>当上述模型验证成功后，即可开始组建开发团队进行产品开发。</p><p>具体实践方案请参考下一篇文章：<a href="/posts/practice/0ws/0ws-base/">独立应用的基本架构</a>。</p>]]></content>
<summary type="html">
在产品验证期，尽可能的依托第三方平台实现核心产品逻辑。
</summary>
<category term="实践" scheme="https://blog.mapleque.com/categories/practice/"/>
<category term="互联网产品的诞生和演化" scheme="https://blog.mapleque.com/categories/practice/0ws/"/>
<category term="Web" scheme="https://blog.mapleque.com/tags/Web/"/>
</entry>
<entry>
<title>【互联网产品的诞生和演化】目录</title>
<link href="https://blog.mapleque.com/posts/practice/0ws/0ws-index/"/>
<id>https://blog.mapleque.com/posts/practice/0ws/0ws-index/</id>
<published>2019-11-29T03:44:54.000Z</published>
<updated>2020-02-26T04:25:47.194Z</updated>
<content type="html"><![CDATA[<ol><li><a href="/posts/practice/0ws/0ws-simple/">产品的最简模型</a><br>采用何种技术架构能够做到高效快速的验证服务可行性？<br>本文核心思想是：尽可能的依托第三方平台实现核心产品逻辑。</li><li><a href="/posts/practice/0ws/0ws-base/">独立应用的基本架构</a><br>自研产品都需要考虑哪些技术投入？<br>一个完整的产品技术架构，不只是客户端和服务端，需要考虑的细节还很多，这些都将在本文中一一阐述。</li><li><a href="/posts/practice/0ws/0ws-stats/">统计数据收集展示和系统监控</a><br>如何快速构建一个独立的统计数据收集和系统监控平台？<br>本文主要讲述的是从第三方统计平台到自研数据平台演化过程和实现方案。</li><li><a href="/posts/practice/0ws/0ws-cicd/">快速迭代和持续集成</a><br>在保证稳定性的前提下如何提升迭代效率？<br>本文的观点：完整的单元测试和自动化运维平台是快速迭代和持续集成的基础。</li><li><a href>面向高可用的逐步升级</a><br>不会挂的系统才是好系统，线上系统远比你想象的脆弱。<br>那么如何面对：突发流量，系统故障，黑客攻击，程序员跑路等各种奇葩问题，将在本文中详细介绍。</li><li><a href>接入第三方应用账号认证</a><br>第三方账号登陆是引流必须跨过的一道门槛，技术上也并不复杂。<br>本文主要介绍的就是接入微信，支付宝，google，facebook，twitter等的套路。</li><li><a href>接入支付系统</a><br>2c业务想赚钱，必须接入支付。<br>那么接入微信，支付宝，银联，ApplePay等，都需要准备什么，需要注意避开哪些坑，本文将会提及。</li><li><a href>多人合作模式</a><br>在绝大多数情况下，多人合作的结果一定是<code>1+1<2</code>，但是你又不得不考虑。<br>并不是因为产品需求繁重，而是为了应付不可控的人员流动。</li></ol>]]></content>
<summary type="html">
<ol>
<li><a href="/posts/practice/0ws/0ws-simple/">产品的最简模型</a><br>
采用何种技术架构能够做到高效快速的验证服务可行性？<br>
本文核心思想是：尽可能的依托第三方平台实现核心产品逻辑。</li>
<li><a hr
</summary>
<category term="实践" scheme="https://blog.mapleque.com/categories/practice/"/>
<category term="互联网产品的诞生和演化" scheme="https://blog.mapleque.com/categories/practice/0ws/"/>
<category term="Web" scheme="https://blog.mapleque.com/tags/Web/"/>
</entry>
<entry>
<title>使用google的colab进行神经网络相关开发</title>
<link href="https://blog.mapleque.com/posts/tool/corlab/colab-github/"/>
<id>https://blog.mapleque.com/posts/tool/corlab/colab-github/</id>
<published>2019-10-31T08:07:21.000Z</published>
<updated>2020-02-26T04:25:47.194Z</updated>
<content type="html"><![CDATA[<p><a href="https://colab.research.google.com/" target="_blank" rel="noopener">Colaboratory(简称colab)</a>是Google提供的一个免费的Jupyter笔记本环境，支持云端运行。<br>为了支持云端运行，Google还提供了强大的计算资源，如GPU、TPU和云端存储空间等，这些都可以通过浏览器免费使用。</p><p>如需全面了解colab的所提供的功能，请阅读<a href="https://colab.research.google.com/notebooks/welcome.ipynb" target="_blank" rel="noopener">官方文档</a>。</p><p>本文的主要目标是使你快速的能够使用colab进行神经网络相关开发。</p><h2 id="准备"><a class="markdownIt-Anchor" href="#准备"></a> 准备</h2><p>使用colab服务，首先需要科学上网。相关文章，请参考<a href="https://blog.mapleque.com/tags/vpn/">标签：vpn</a>中的内容。</p><h2 id="创建修改文件"><a class="markdownIt-Anchor" href="#创建修改文件"></a> 创建修改文件</h2><p>直接访问地址： <a href="https://colab.research.google.com/" target="_blank" rel="noopener">https://colab.research.google.com/</a> ，就可以开始代码的编辑。</p><p>打开页面时，会弹窗提示最近打开的文件，也可以选择创建文件。这个弹窗后面也可以通过顶部菜单：<code>文件->打开笔记本</code>调出。</p><p>在这个弹窗中，我们还可以看到一些其他打开文件的方式，这里重点说明GitHub选项。</p><p>点击选择GitHub选项，会出现<code>输入GitHub网址，或者按组织或用户搜索</code>的输入框。这里可以输入自己的github用户名，如：<code>mapleque</code>，点击搜索后，就会在下面代码库部分出现该用户所有的公开项目列表了（中间可能需要进行GitHub账号授权）。选择一个项目和分支，如果在项目中含有colab可以打开的文件，就会被列在下面的路径列表中，用户可以选择任意一个打开编辑。</p><p>如果是私有代码库，可以选择勾选右上角的选项，然后进行额外的授权之后，重新搜索对应的空间或项目，这时所有有权限的私有项目也会出现在列表中了。</p><p>注意：这里还有一个快捷打开GitHub文件的技巧，即输入网址：<a href="https://colab.research.google.com/github/mapleque/nnlearning/blob/master/notebooks/TensorflowMNIST.ipynb" target="_blank" rel="noopener">https://colab.research.google.com/github/mapleque/nnlearning/blob/master/notebooks/TensorflowMNIST.ipynb</a> 。</p><p>观察上面的网址，可以发现<code>/github/</code>后面的路径，实际上就是文件在GitHub上预览的路径。</p><h2 id="编辑运行"><a class="markdownIt-Anchor" href="#编辑运行"></a> 编辑运行</h2><p>colab集成了丰富的nn相关资源，如tensorflow, pytorch等。所以只需要新建一个python3的记事本，就可以开始执行程序了。</p><p>注意，新建代码文件后，需要配置代码执行环境：选择顶部菜单中的<code>代码执行程序->更改运行时类型</code>选项，在弹框中选择相关设置，如：硬件加速选择GPU，最后保存。</p><p>点击顶部菜单左上角<code>+代码</code>选项，就会在文件编辑区域出现一行带有播放图标的空白输入框，在输入框中输入代码，点击播放按钮，就可以提交执行。例如这里提交编辑代码：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> __future__ <span class="keyword">import</span> absolute_import, division, print_function, unicode_literals</span><br><span class="line"><span class="keyword">import</span> tensorflow <span class="keyword">as</span> tf</span><br><span class="line"><span class="keyword">from</span> tensorflow <span class="keyword">import</span> keras</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="keyword">import</span> matplotlib.pyplot <span class="keyword">as</span> plt</span><br></pre></td></tr></table></figure><p>点击执行，即显示执行成功。可见tensorflow，numpy和matplotlib等包都已经预置安装好了。</p><p>后面正常编辑执行MNIST例子相关代码，就可以看到自动下载数据和训练的日志输出。完整代码请参考<a href="https://colab.research.google.com/github/mapleque/nnlearning/blob/master/notebooks/TensorflowMNIST.ipynb" target="_blank" rel="noopener">mapleque/nnlearning/TensorflowMNIST.ipynb</a>。</p><p>colab执行结果页面，可以将图片直接展示出来。</p><h2 id="提交到github"><a class="markdownIt-Anchor" href="#提交到github"></a> 提交到GitHub</h2><p>colab支持直接将当前代码文件提交到GitHub。</p><p>点击顶部菜单：<code>文件->在GitHub中保存一份副本</code>选项，授权后在弹窗中选择对应的代码库、分支和路径，就可以提交了。</p><p>这里如果勾选了<code>包含指向Colaboratory的链接</code>选项，将会在GitHub的文件预览页面看到一个直接在colab中打开文件并编辑的按钮。</p>]]></content>
<summary type="html">
在colab上开发并运行神经网络相关任务，并将代码托管到github上。
</summary>
<category term="工具" scheme="https://blog.mapleque.com/categories/tool/"/>
<category term="corlab" scheme="https://blog.mapleque.com/categories/tool/corlab/"/>
<category term="nn" scheme="https://blog.mapleque.com/tags/nn/"/>
<category term="pytorch" scheme="https://blog.mapleque.com/tags/pytorch/"/>
<category term="colab" scheme="https://blog.mapleque.com/tags/colab/"/>
</entry>
<entry>
<title>部署并使用shadowsocks</title>
<link href="https://blog.mapleque.com/posts/tool/shadowsocks/shadowsocks/"/>
<id>https://blog.mapleque.com/posts/tool/shadowsocks/shadowsocks/</id>
<published>2019-10-22T08:46:13.000Z</published>
<updated>2020-02-26T04:25:47.198Z</updated>
<content type="html"><![CDATA[<p><a href="https://shadowsocks.org" target="_blank" rel="noopener">Shadowsocks</a>(通常简称为ss)是一个基于socks5协议实现的网络管理服务。<br>通常我们用它来做网络流量转发，因此这里简单介绍一下如何方便安全的部署并使用它。</p><p>本文将基于ss的go版本部署实现，可能需要一些计算机开发基础知识。如不具备，请自行搜索其他教程。</p><h2 id="准备vps"><a class="markdownIt-Anchor" href="#准备vps"></a> 准备vps</h2><p>部署和使用ss，需要先有一个用于转发流量的vps。</p><div class="note info"> <p>笔者目前使用的vultr服务，$3.50/mo, 500GB。<br>通过这个推广链接注册并购买，可以获得一定的优惠。<br><a href="https://www.vultr.com/?ref=7147137" target="_blank" rel="noopener">https://www.vultr.com/?ref=7147137</a></p> </div><p>下面将以CentOS7x64为例进行部署。</p><h2 id="申请ip"><a class="markdownIt-Anchor" href="#申请ip"></a> 申请ip</h2><p>想要进行流量转发，必须要有一个能够访问的ip。由于一些不明原因，并不是所有ip都能访问，所以在一定在申请ip的时候注意选择。</p><p>如果是使用vultr，可以通过多购置几台vps进行尝试，遇到可用ip后再将其他的退掉。</p><h2 id="编译安装"><a class="markdownIt-Anchor" href="#编译安装"></a> 编译安装</h2><p>shadowsocks的代码，全部托管在github上，这里建议使用golang实现版本：<a href="https://github.com/shadowsocks/go-shadowsocks2" target="_blank" rel="noopener">https://github.com/shadowsocks/go-shadowsocks2</a> 。</p><p>该版本可以提供跨平台(windows, linux, MacOS)支持。</p><p>登录所购买的vps，然后安装开发环境：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">yum install -y git golang</span><br></pre></td></tr></table></figure><p>下载代码并编译安装（这里之所以自己下载代码编译，是因为go-shadowsocks2的release版本落后master很多，居然还不支持plugin）：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">go get -u -v github.com/shadowsocks/go-shadowsocks2</span><br></pre></td></tr></table></figure><h2 id="部署服务"><a class="markdownIt-Anchor" href="#部署服务"></a> 部署服务</h2><p>通过上面的步骤下载完成后，应该就已经自动编译好了一个当前系统的可执行文件了。</p><p>执行下面的命令启动服务：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="variable">$GOPATH</span>/bin/shadowsocks2 -s <span class="string">'ss://AES-256-GCM:[email protected]:8488'</span> -verbose</span><br></pre></td></tr></table></figure><p>对于CentOS7，还需要设置防火墙。这里通过创建一个防火墙服务来实现：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">vim /usr/lib/firewalld/services/socks.xml</span><br></pre></td></tr></table></figure><figure class="highlight xml"><figcaption><span>/usr/lib/firewalld/services/socks.xml</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta"><?xml version="1.0" encoding="utf-8"?></span></span><br><span class="line"><span class="tag"><<span class="name">service</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">short</span>></span>Socks<span class="tag"></<span class="name">short</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>Socks Port<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">port</span> <span class="attr">protocol</span>=<span class="string">"tcp"</span> <span class="attr">port</span>=<span class="string">"8488"</span>/></span></span><br><span class="line"><span class="tag"></<span class="name">service</span>></span></span><br></pre></td></tr></table></figure><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">firewall-cmd --permanent --add-service=socks</span><br><span class="line">firewall-cmd --reload</span><br></pre></td></tr></table></figure><h2 id="编译客户端"><a class="markdownIt-Anchor" href="#编译客户端"></a> 编译客户端</h2><p>该版本shadowsocks提供了多平台的客户端，可以直接通过原代码编译生成：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">make</span><br></pre></td></tr></table></figure><p>然后到bin路径下找对应的客户端可执行文件下载。</p><h2 id="启动客户端"><a class="markdownIt-Anchor" href="#启动客户端"></a> 启动客户端</h2><p>客户端支持多种代理模式，这里以在MacOS上执行socks模式为例：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">shadowsocks2-macos -c <span class="string">'ss://AES-256-GCM:your_password@ip:8488'</span> -verbose -socks :1080 -u</span><br></pre></td></tr></table></figure><p>注意将上面的ip文本替换为vps的ip。</p><h2 id="使用服务"><a class="markdownIt-Anchor" href="#使用服务"></a> 使用服务</h2><p>在MacOS上，可以通过内置的网络组件直接连接socks代理：</p><p>系统偏好设置->网络->高级->代理->socks</p><p>在代理服务器中填上localhost:1080，然后点击：</p><p>好->应用</p><p>即完成了机器全部网络流量的转发。</p><p>此时可以尝试访问https://google.com 以检验是否成功。</p><h2 id="使用其他客户端"><a class="markdownIt-Anchor" href="#使用其他客户端"></a> 使用其他客户端</h2><p>经过了上面的步骤，你应该已经能在MacOS上访问shadowsocks官网了。<br>此时你会发现，官网上推荐了多种可以在不同平台使用的客户端，这里将重点介绍以下outline。</p><p>IOS上的outline客户端目前可以在AppStore上免费下载。</p><p>启动outline客户端，你会发现要求输入的是secretKey。<br>按照outline官方的要求，你需要下载并部署outline管理服务来生成并管理secretKey。<br>但这个管理工具不仅增加了一些统计逻辑实现，还收集了服务器信息，因此对于个人用户不推荐使用（尽管客户端也可能收集了一些数据）。<br>这里可以通过下面网址自行生成secretKey并链接：<br><a href="https://shadowsocks.org/en/config/quick-guide.html" target="_blank" rel="noopener">https://shadowsocks.org/en/config/quick-guide.html</a></p>]]></content>
<summary type="html">
本文主要介绍shadowsocks的部署和使用
</summary>
<category term="工具" scheme="https://blog.mapleque.com/categories/tool/"/>
<category term="shadowsocks" scheme="https://blog.mapleque.com/categories/tool/shadowsocks/"/>
<category term="vpn" scheme="https://blog.mapleque.com/tags/vpn/"/>
</entry>
<entry>
<title>【vim】使用vim作为开发js的IDE</title>
<link href="https://blog.mapleque.com/posts/tool/vim/vim-js/"/>
<id>https://blog.mapleque.com/posts/tool/vim/vim-js/</id>
<published>2019-09-19T09:14:04.000Z</published>
<updated>2020-02-26T04:25:47.199Z</updated>
<content type="html"><![CDATA[<div class="note info"> <p>本文中将直接使用基本设置已经使用的配置，如需了解，请阅读<a href="/posts/tool/vim/vim-start/">【vim】基本设置</a>。</p> </div><h1 id="基本设置"><a class="markdownIt-Anchor" href="#基本设置"></a> 基本设置</h1><p>创建<code>~/.vim/ftplugin/javascript.vim</code>文件来配置用于js文件的设置。</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">" 缩进2字符</span></span><br><span class="line"><span class="keyword">set</span> tabstop=<span class="number">2</span></span><br><span class="line"><span class="keyword">set</span> softtabstop=<span class="number">2</span></span><br><span class="line"><span class="keyword">set</span> <span class="built_in">shiftwidth</span>=<span class="number">2</span></span><br><span class="line"><span class="keyword">set</span> expandtab</span><br><span class="line"></span><br><span class="line"><span class="comment">" 设置代码每行最大宽度标尺</span></span><br><span class="line"><span class="keyword">set</span> tw=<span class="number">118</span></span><br></pre></td></tr></table></figure><h1 id="支持jsx语法高亮"><a class="markdownIt-Anchor" href="#支持jsx语法高亮"></a> 支持jsx语法高亮</h1><p>让vim支持jsx语法高亮，使用<a href="https://github.com/mxw/vim-jsx" target="_blank" rel="noopener">vim-jsx</a>插件。</p><p>通过Vundle安装，在.vimrc中添加：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">Plugin <span class="string">'pangloss/vim-javascript'</span></span><br><span class="line">Plugin <span class="string">'mxw/vim-jsx'</span></span><br></pre></td></tr></table></figure><p>在vim中执行命令：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">:PluginInstall</span><br></pre></td></tr></table></figure><h1 id="eslint检查"><a class="markdownIt-Anchor" href="#eslint检查"></a> ESLint检查</h1><p>使用eslint官方推荐的<a href="https://github.com/vim-syntastic/syntastic" target="_blank" rel="noopener">syntastic</a>插件来执行<code>eslint</code>。</p><p>通过Vundle安装syntastic插件，在vim下执行下面命令：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">:PluginInstall syntastic</span><br></pre></td></tr></table></figure><p>成功后，在<code>~/.vimrc</code>中添加：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">Plugin <span class="string">'syntastic'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">let</span> <span class="variable">g:syntastic_always_populate_loc_list</span> = <span class="number">1</span></span><br><span class="line"><span class="keyword">let</span> <span class="variable">g:syntastic_auto_loc_list</span> = <span class="number">1</span></span><br><span class="line"><span class="keyword">let</span> <span class="variable">g:syntastic_check_on_open</span> = <span class="number">1</span></span><br><span class="line"><span class="keyword">let</span> <span class="variable">g:syntastic_check_on_wq</span> = <span class="number">0</span></span><br><span class="line"><span class="keyword">let</span> <span class="variable">g:syntastic_javascript_checkers</span> = [ <span class="string">'eslint'</span> ]</span><br></pre></td></tr></table></figure><p>完成后，vim中就可以通过执行系统的<code>eslint</code>命令来检查当前文件的错误了。</p><div class="note info"> <p>注意：上面的配置，将会使syntastic在保存文件和打开文件的时候自动执行。</p> </div><h1 id="eslint自动修复"><a class="markdownIt-Anchor" href="#eslint自动修复"></a> ESLint自动修复</h1><p>使用自研插件<a href="https://github.com/mapleque/vim-eslint-vim" target="_blank" rel="noopener">vim-eslint-fix</a>来执行<code>eslint --fix</code>。</p><p>通过Vundle安装vim-eslint-fix插件，在vim下执行下面命令：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">:PluginInstall mapleque/vim-eslint-fix</span><br></pre></td></tr></table></figure><p>成功后，在<code>~/.vimrc</code>中添加：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">Plugin <span class="string">'mapleque/vim-eslint-fix'</span></span><br></pre></td></tr></table></figure><p>在<code>~/.vim/ftplugin/javascript.vim</code>中添加：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">"如果没有设置过mapleader，请先自行设置</span></span><br><span class="line"><span class="string">"let mapleader="</span>,<span class="comment">"</span></span><br><span class="line"><span class="keyword">noremap</span> <span class="symbol"><Leader></span><span class="keyword">f</span> :EslintFix<span class="symbol"><CR></span></span><br><span class="line"></span><br><span class="line"><span class="comment">"如果需要在保存文件时候自动修复，添加下面这行</span></span><br><span class="line"><span class="comment">"autocmd BufwritePost *.js EslintFix</span></span><br></pre></td></tr></table></figure><p>完成后，当需要eslint自动修复错误的时候，在命令模式下输入<code>,f</code>即可。</p><h1 id="react新组件模板"><a class="markdownIt-Anchor" href="#react新组件模板"></a> React新组件模板</h1><p>创建模板文件<code>~/.vim/templates/react.js</code>:</p><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> React, { Component } <span class="keyword">from</span> <span class="string">'react'</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Index</span> <span class="keyword">extends</span> <span class="title">Component</span> </span>{</span><br><span class="line"> <span class="keyword">constructor</span> (props) {</span><br><span class="line"> <span class="keyword">super</span>(props)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> state = {}</span><br><span class="line"></span><br><span class="line"> render() {</span><br><span class="line"> <span class="keyword">return</span> (</span><br><span class="line"> <div>tbd<<span class="regexp">/div></span></span><br><span class="line"><span class="regexp"> )</span></span><br><span class="line"><span class="regexp"> }</span></span><br><span class="line"><span class="regexp">}</span></span><br><span class="line"><span class="regexp"></span></span><br><span class="line"><span class="regexp">export default Index</span></span><br></pre></td></tr></table></figure><p>编辑<code>~/.vimrc</code>文件，增加：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">autocmd BufNewFile *.js 0r ~/.vim/templates/react.js</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
主要介绍使用vim作为开发js项目的IDE所需的配置
</summary>
<category term="工具" scheme="https://blog.mapleque.com/categories/tool/"/>
<category term="Vim" scheme="https://blog.mapleque.com/categories/tool/vim/"/>
<category term="Vim" scheme="https://blog.mapleque.com/tags/Vim/"/>
</entry>
<entry>
<title>【vim】基本配置</title>
<link href="https://blog.mapleque.com/posts/tool/vim/vim-start/"/>
<id>https://blog.mapleque.com/posts/tool/vim/vim-start/</id>
<published>2019-09-19T09:08:43.000Z</published>
<updated>2020-02-26T04:25:47.211Z</updated>
<content type="html"><![CDATA[<h1 id="基本配置"><a class="markdownIt-Anchor" href="#基本配置"></a> 基本配置</h1><p>在<code>~/.vimrc</code>中添加：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">"基本配置</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">syntax</span> <span class="keyword">on</span> <span class="comment">"语法高亮</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">let</span> mapleader=<span class="string">","</span> <span class="comment">"快捷键配置</span></span><br><span class="line"><span class="comment">" 如此，可以在后面配置快捷键时候使用`<Leader>`标识符。</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">set</span> <span class="keyword">nu</span> <span class="comment">"显示行号</span></span><br><span class="line"><span class="keyword">set</span> ruler <span class="comment">"显示当前行状态</span></span><br><span class="line"><span class="keyword">set</span> cursorline <span class="comment">"显示当前行标尺</span></span><br><span class="line"><span class="keyword">set</span> cursorcolumn <span class="comment">"显示当前列标尺</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">set</span> autoindent <span class="comment">"自动缩进</span></span><br><span class="line"><span class="keyword">set</span> smartindent <span class="comment">"智能缩进</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">set</span> showmatch <span class="comment">"显示括号配对</span></span><br><span class="line"><span class="keyword">set</span> incsearch <span class="comment">"高亮搜索内容</span></span><br><span class="line"><span class="keyword">set</span> hlsearch <span class="comment">"高亮搜索内容</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">set</span> tabstop=<span class="number">4</span> <span class="comment">"tab等效字符数</span></span><br><span class="line"><span class="keyword">set</span> softtabstop=<span class="number">4</span> <span class="comment">"tab等效字符数</span></span><br><span class="line"><span class="keyword">set</span> <span class="built_in">shiftwidth</span>=<span class="number">4</span> <span class="comment">"tab字符数</span></span><br><span class="line"><span class="keyword">set</span> expandtab <span class="comment">"tab转成空格</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">set</span> <span class="keyword">list</span> <span class="comment">"显示制表符</span></span><br><span class="line"><span class="keyword">set</span> listchars=<span class="keyword">ta</span><span class="variable">b:</span>>-,trai<span class="variable">l:</span>- <span class="comment">"制表符和空格显示样式</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">colorscheme</span> desert <span class="comment">"配色方案</span></span><br><span class="line"></span><br><span class="line"><span class="comment">"每行最大字符数标尺</span></span><br><span class="line"><span class="keyword">set</span> <span class="keyword">cc</span>=+<span class="number">1</span>,+<span class="number">2</span>,+<span class="number">3</span> <span class="comment">" highlight three columns after 'textwidth'</span></span><br><span class="line"><span class="keyword">hi</span> ColorColumn ctermbg=lightgrey guibg=lightgrey</span><br><span class="line"><span class="keyword">set</span> textwidth=<span class="number">80</span></span><br><span class="line"></span><br><span class="line"><span class="comment">"状态行设置</span></span><br><span class="line"><span class="keyword">set</span> laststatus=<span class="number">2</span> <span class="comment">"在倒数第二行显示状态行</span></span><br><span class="line"><span class="keyword">set</span> statusline=%<%<span class="number">1</span>*\ %<span class="keyword">f</span>%<span class="keyword">m</span>%r%h%<span class="keyword">w</span>\ %= <span class="comment">"左侧显示文件名和文件状态</span></span><br><span class="line"><span class="keyword">set</span> statusline+=%<span class="number">2</span>*\ %<span class="keyword">y</span>\ %* <span class="comment">"文件类型</span></span><br><span class="line"><span class="keyword">set</span> statusline+=%<span class="number">3</span>*\ %{&ff}\ \|\ %{\<span class="string">"\".(&fenc==\"\"?&enc:&fenc).((exists(\"+bomb\")\ &&\ &bomb)?\",B\":\"\").\"\ \|\"}\ %-14.(%l:%c%V%)%* "</span>文件格式|文件编码|当前光标所处行列</span><br><span class="line"><span class="keyword">set</span> statusline+=%<span class="number">4</span>*\ %-<span class="number">5</span>.(%<span class="keyword">p</span>%%%)%* <span class="comment">"当前光标所处文件百分比</span></span><br><span class="line"><span class="comment">"设置状态行配色</span></span><br><span class="line"><span class="keyword">hi</span> User1 cterm=None ctermfg=<span class="number">251</span> ctermbg=<span class="number">240</span></span><br><span class="line"><span class="keyword">hi</span> User2 cterm=None ctermfg=<span class="number">183</span> ctermbg=<span class="number">239</span></span><br><span class="line"><span class="keyword">hi</span> User3 cterm=None ctermfg=<span class="number">208</span> ctermbg=<span class="number">238</span></span><br><span class="line"><span class="keyword">hi</span> User4 cterm=None ctermfg=<span class="number">246</span> ctermbg=<span class="number">237</span></span><br></pre></td></tr></table></figure><h1 id="代码折叠"><a class="markdownIt-Anchor" href="#代码折叠"></a> 代码折叠</h1><p>在<code>~/.vimrc</code>中添加配置：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">set</span> foldenable <span class="comment">"允许折叠</span></span><br><span class="line"><span class="keyword">set</span> foldmethod=<span class="built_in">indent</span> <span class="comment">"按缩进折叠</span></span><br></pre></td></tr></table></figure><p>在vim中执行命令查看手册：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">:h Folding</span><br></pre></td></tr></table></figure><p>常用命令：</p><ul><li>za 打开或关闭光标所在折叠（一层）</li><li>zA 打开或关闭光标所在折叠（所有）</li><li>[z 移动到当前折叠开始处</li><li>]z 移动到当前折叠结尾处</li><li>zj 移动到下一个折叠开始处</li><li>zk 移动到上一个折叠开始处</li></ul><h1 id="vundle插件管理工具"><a class="markdownIt-Anchor" href="#vundle插件管理工具"></a> Vundle插件管理工具</h1><p>下载Vundle：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">git <span class="built_in">clone</span> https://github.com/VundleVim/Vundle.vim.git ~/.vim/bundle/Vundle.vim</span><br></pre></td></tr></table></figure><p>在<code>~/.vimrc</code>中添加：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">set</span> nocompatible</span><br><span class="line"><span class="keyword">filetype</span> off</span><br><span class="line"><span class="keyword">set</span> rtp+=~/.<span class="keyword">vim</span>/bundle/Vundle.<span class="keyword">vim</span></span><br><span class="line"><span class="keyword">call</span> vundle#begin()</span><br><span class="line">Plugin <span class="string">'VundleVim/Vundle.vim'</span></span><br><span class="line"><span class="comment">" Plugins</span></span><br><span class="line"><span class="comment">"这里添加自己的插件列表，例如：</span></span><br><span class="line"><span class="comment">" Plugin 'scrooloose/nerdtree'</span></span><br><span class="line"><span class="keyword">call</span> vundle#end()</span><br><span class="line"><span class="keyword">filetype</span> plugin <span class="keyword">on</span></span><br></pre></td></tr></table></figure><p>以后安装插件，先在<code>~/.vimrc</code>中<code>Plugins</code>后面添加：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">Plugin <span class="string">'scrooloose/nerdtree'</span></span><br></pre></td></tr></table></figure><p>再在vim中执行：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">:PuginInstall</span><br></pre></td></tr></table></figure><h1 id="文件类型扩展配置"><a class="markdownIt-Anchor" href="#文件类型扩展配置"></a> 文件类型扩展配置</h1><p>可以通过单独的配置文件来修改指定类型文件的vim配置，如不同的缩进、代码宽度、插件命令等。</p><p>如果安装了Vundle，那么在安装过程中已经开启了文件类型扩展配置，可以直接使用。</p><p>如果没有安装Vundle，可以在<code>~/.vimrc</code>中添加：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">filetype</span> off</span><br><span class="line"><span class="keyword">filetype</span> plugin <span class="keyword">on</span></span><br></pre></td></tr></table></figure><p>配置完成后，创建文件夹<code>~/.vim/ftplugin</code>，然后在文件夹中添加以文件类型命名的配置文件即可自动加载，如<code>~/.vim/ftplugin/javascript.vim</code>。</p><h1 id="nerdtree左侧菜单插件"><a class="markdownIt-Anchor" href="#nerdtree左侧菜单插件"></a> NERDTree左侧菜单插件</h1><p><a href="https://github.com/scrooloose/nerdtree" target="_blank" rel="noopener">NERDTree</a>是一款实现了左侧导航菜单的插件。<br>这里提供使用Vundle安装的方法。</p><p>在<code>~/.vimrc</code>中，<code>Plugins</code>的位置，添加：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">Plugin <span class="string">'scrooloose/nerdtree'</span></span><br></pre></td></tr></table></figure><p>然后在vim中执行：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">:PluginInstall</span><br></pre></td></tr></table></figure><p>在<code>~/.vimrc</code>中添加相关配置：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">" nerdtree配置</span></span><br><span class="line"><span class="keyword">autocmd</span> StdinReadPre * <span class="keyword">let</span> <span class="variable">s:std_in</span>=<span class="number">1</span></span><br><span class="line"><span class="keyword">autocmd</span> VimEnter * <span class="keyword">if</span> <span class="built_in">argc</span>() == <span class="number">1</span> && <span class="built_in">isdirectory</span>(<span class="built_in">argv</span>()[<span class="number">0</span>]) && !exists(<span class="string">"s:std_in"</span>) | <span class="keyword">exe</span> <span class="string">'NERDTree'</span> <span class="built_in">argv</span>()[<span class="number">0</span>] | <span class="keyword">wincmd</span> <span class="keyword">p</span> | <span class="keyword">ene</span> | <span class="keyword">endif</span></span><br><span class="line"><span class="keyword">map</span> <span class="symbol"><C-n></span> :NERDTreeToggle<span class="symbol"><CR></span></span><br><span class="line"><span class="keyword">autocmd</span> bufenter * <span class="keyword">if</span> (<span class="built_in">winnr</span>(<span class="string">"$"</span>) == <span class="number">1</span> && <span class="built_in">exists</span>(<span class="string">"b:NERDTree"</span>) && <span class="variable">b:NERDTree</span>.isTabTree()) | q | <span class="keyword">endif</span></span><br></pre></td></tr></table></figure><p>上面的配置实现了三个功能：</p><ul><li>使用CTRL+n可以开启或关闭左侧导航栏</li><li>如果当前打开的文件是文件夹，则自动开启左侧导航栏，否则默认不开启</li><li>当关闭文件时，只剩下导航栏窗口，则自动关闭窗口，退出vim</li></ul>]]></content>
<summary type="html">
介绍vim的一些基本配置和常用插件
</summary>
<category term="工具" scheme="https://blog.mapleque.com/categories/tool/"/>
<category term="Vim" scheme="https://blog.mapleque.com/categories/tool/vim/"/>
<category term="Vim" scheme="https://blog.mapleque.com/tags/Vim/"/>
</entry>
<entry>
<title>【开始用go】在MacOS上搭建开发环境</title>
<link href="https://blog.mapleque.com/posts/practice/go/go-workspace/"/>
<id>https://blog.mapleque.com/posts/practice/go/go-workspace/</id>
<published>2019-09-06T10:33:45.000Z</published>
<updated>2020-02-26T04:25:47.196Z</updated>
<content type="html"><![CDATA[<p>每个开发者，都有一个套适合自己的开发环境，如果你决定开始使用go语言开发，那么一定要先准备好自己的开发环境。</p><div class="note info"> <p>这里所指的开发环境，不只是一个能让go跑起来的环境，而是一个可以用于日常工作的完整的工作环境。</p> </div><p>本文将从三个方面介绍go语言开发环境的构建，并给出自己所构建的开发环境的配置：</p><ul><li>go安装和升级 – go运行的基础</li><li>路径和环境变量 – 更好地管理项目、依赖，并方便执行使用go安装的命令行工具</li><li>IDE – 一个个人熟悉的用于写go代码的编辑器</li></ul><a id="more"></a><h1 id="go安装和升级"><a class="markdownIt-Anchor" href="#go安装和升级"></a> go安装和升级</h1><div class="note info"> <p>Go官方提供了详细的安装文档 <a href="https://go-zh.org/doc/install" target="_blank" rel="noopener">https://go-zh.org/doc/install</a> ，读者可以按需索取。<br>这里将要介绍的是在MacOS下，使用homebrew安装并管理Go的详细方法。</p> </div><p>Homebrew是一个面向MacOS的包管理工具，官网 <a href="https://brew.sh/" target="_blank" rel="noopener">https://brew.sh/</a> 有详细安装使用方法说明。</p><p>在国内使用Homebrew，建议更改源：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">cd</span> <span class="string">"<span class="variable">$(brew --repo)</span>"</span></span><br><span class="line">git remote <span class="built_in">set</span>-url origin https://mirrors.ustc.edu.cn/brew.git </span><br><span class="line"></span><br><span class="line"><span class="built_in">cd</span> <span class="string">"<span class="variable">$(brew --repo)</span>/Library/Taps/homebrew/homebrew-core"</span></span><br><span class="line">git remote <span class="built_in">set</span>-url origin https://mirrors.ustc.edu.cn/homebrew-core.git </span><br><span class="line"></span><br><span class="line">brew update</span><br></pre></td></tr></table></figure><p>查看可用的Go版本：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">brew search go</span><br></pre></td></tr></table></figure><p>这里将可以看到一系列可以安装的Go版本：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">go [email protected] [email protected] [email protected] [email protected]</span><br></pre></td></tr></table></figure><p>通常直接安装最新版本即可：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">brew install go</span><br></pre></td></tr></table></figure><p>安装完成后，即可在命令行执行go命令了：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">go version</span><br></pre></td></tr></table></figure><p>当go更新了新版本，需要升级的时候，执行：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">brew upgrade go</span><br></pre></td></tr></table></figure><p>这个命令将安装Homebrew所管理的最新版本的go，并替换掉原来安装的go。</p><h1 id="路径和环境变量"><a class="markdownIt-Anchor" href="#路径和环境变量"></a> 路径和环境变量</h1><p>安装完Go后，就可以开始写代码了。但是如果想要写项目，还需要更进一步的进行配置。</p><div class="note info"> <p>官方文档参考： <a href="https://go-zh.org/doc/code.html" target="_blank" rel="noopener">https://go-zh.org/doc/code.html</a> 。</p> </div><p>这里笔者给出自己在MacOS上的路径：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">~/</span><br><span class="line">├── workspace -> Documents/workspace</span><br><span class="line">│ ├── gopath</span><br><span class="line">│ │ ├── bin</span><br><span class="line">│ │ ├── pkg</span><br><span class="line">│ │ ├── src</span><br><span class="line">│ │ │ ├── github.com</span><br><span class="line">│ │ │ │ ├── mapleque</span><br><span class="line">│ │ │ │ │ ├── gostart</span><br><span class="line">│ │ │ ├── golang.org</span><br><span class="line">│ │ │ │ ├── x</span><br><span class="line">│ │ │ │ │ ├── tour</span><br><span class="line">│ ├── github.com</span><br><span class="line">│ │ ├── mapleque</span><br><span class="line">│ │ │ ├── gostart -> ~/workspace/gopath/src/github.com/mapleque/gostart</span><br></pre></td></tr></table></figure><p>设置环境变量：</p><figure class="highlight bash"><figcaption><span>~/.bash_profile</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">export</span> GOPATH=~/workspace/gopath</span><br><span class="line"><span class="built_in">export</span> PATH=<span class="variable">$PATH</span>:<span class="variable">$GOPATH</span>/bin</span><br></pre></td></tr></table></figure><p>其中，</p><ul><li>GOPROXY 用于<code>go get</code>时作为代理</li><li>GOAPTH 用于原始的go依赖路径</li><li>PATH 中增加 $GOPATH/bin 是为了让<code>go install</code>所安装的二进制文件能够直接被执行</li></ul><p>特别的，从go1.11版本开始，Go将go module作为官方包管理工具进行支持，其中go1.11和go1.12版本需要主动开启：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">export</span> GO111MODULE=on</span><br><span class="line"><span class="built_in">export</span> GOPROXY=https://goproxy.cn</span><br></pre></td></tr></table></figure><p>其中，</p><ul><li>GOPROXY 用于<code>go get</code>时自动代理使用国内源</li></ul><div class="note info"> <p>关于go module的使用，可以参考官方文档：<a href="https://blog.go-zh.org/using-go-modules" target="_blank" rel="noopener">https://blog.go-zh.org/using-go-modules</a> 。<br>笔者也将在另外一篇文章中详细讲述自己的使用方式，敬请期待。</p> </div><h1 id="编辑器"><a class="markdownIt-Anchor" href="#编辑器"></a> 编辑器</h1><div class="note info"> <p>官方推荐编辑器：<a href="https://golang.google.cn/doc/editors.html" target="_blank" rel="noopener">https://golang.google.cn/doc/editors.html</a> 。</p> </div><p>笔者使用的是vim-go。</p><ul><li>如何安装vim-go <a href="https://github.com/fatih/vim-go" target="_blank" rel="noopener">https://github.com/fatih/vim-go</a></li><li><a href="/posts/tool/vim/vim-diy/">如何安装vundle并配置vim</a></li><li>如何安装vim – <code>brew install vim</code></li></ul><p>更多关于vim的配置，可以参考<a href="/posts/tool/vim/vim-go/">【vim】使用vim作为开发go的IDE</a>。</p><div class="note warning"> <p>注意：最新版本vim-go由于使用了gopls，所以如果使用的是vim-go的最新版本，在执行<code>:GoInstallBanaries</code>命令时，必须开启go module模式，并且在当前文件夹中有go.mod文件。</p> </div><p>这里特别提一些值得一试vim-go命令：</p><ul><li><code>:GoDef, :GoDefPop</code>或者<code>Ctrl+], Ctrl+t</code> – 直接跳转到光标位置所指的方法或变量的定义代码</li><li><code>:GoAddTags, :GoRemoveTags</code> – 给当前光标所在属性添加删除标签</li><li><code>:GoMetaLinter</code> – 执行一系列代码检查</li><li><code>:GoImpl</code> – 生成实现指定接口的代码</li><li><code>:GoIfErr</code> – 生成错误校验的代码</li><li><code>:GoImports</code> – 自动增减需要import的包</li></ul><p>本质上讲，上面的命令都是通过调用一些go tool来实现。</p><div class="note info"> <p>Go有着众多的tool，它们几乎覆盖了从代码编写、编译测试运行、到性能监控等整个开发周期。<br>笔者在后面的文章中，也会选择性的介绍一些tool的实战，以启发读者如何利用好这些资源来更高效的工作。</p> </div>]]></content>
<summary type="html">
<p>每个开发者，都有一个套适合自己的开发环境，如果你决定开始使用go语言开发，那么一定要先准备好自己的开发环境。</p>
<div class="note info">
<p>这里所指的开发环境，不只是一个能让go跑起来的环境，而是一个可以用于日常工作的完整的工作环境。</p>
</div>
<p>本文将从三个方面介绍go语言开发环境的构建，并给出自己所构建的开发环境的配置：</p>
<ul>
<li>go安装和升级 – go运行的基础</li>
<li>路径和环境变量 – 更好地管理项目、依赖，并方便执行使用go安装的命令行工具</li>
<li>IDE – 一个个人熟悉的用于写go代码的编辑器</li>
</ul>
</summary>
<category term="实践" scheme="https://blog.mapleque.com/categories/practice/"/>
<category term="开始用go" scheme="https://blog.mapleque.com/categories/practice/go/"/>
<category term="golang" scheme="https://blog.mapleque.com/tags/golang/"/>
</entry>
<entry>
<title>【开始用go】从零开始</title>
<link href="https://blog.mapleque.com/posts/practice/go/go-start/"/>
<id>https://blog.mapleque.com/posts/practice/go/go-start/</id>
<published>2019-09-06T08:48:01.000Z</published>
<updated>2020-02-26T04:25:47.196Z</updated>
<content type="html"><![CDATA[<p>整体来说，本系列文章属于工程实践向。</p><div class="note warning"> <p>需要特别说明的是：</p><ul><li>本系列文章不是教程，学习go语言，请阅读官方教程 <a href="https://tour.go-zh.org/list" target="_blank" rel="noopener">https://tour.go-zh.org/list</a> 。</li><li>本系列文章中可能有大量笔者个人观点，虽然经历过实践的检验，但也不一定是最优。如有不同意见，欢迎交流探讨 <a href="mailto:[email protected]" target="_blank" rel="noopener">[email protected]</a> 。</li></ul> </div><h1 id="目录"><a class="markdownIt-Anchor" href="#目录"></a> 目录</h1><ul><li><a href="/posts/practice/go/go-workspace/">在MacOS上搭建开发环境</a></li><li><a href="/posts/practice/go/go-http/">开发http服务</a></li><li><a href="/posts/practice/go/go-practice-http">go语言在http服务开发上的工程实践经验</a></li></ul><h1 id="官方网站"><a class="markdownIt-Anchor" href="#官方网站"></a> 官方网站</h1><ul><li>官方 <a href="https://golang.org/" target="_blank" rel="noopener">https://golang.org/</a></li><li>官方国内代理 <a href="https://golang.google.cn/" target="_blank" rel="noopener">https://golang.google.cn/</a></li><li>官方中文翻译 <a href="https://go-zh.org/" target="_blank" rel="noopener">https://go-zh.org/</a></li></ul><p>以上三个官方，内容上稍有差异，请按需选择。</p><h1 id="其他资源"><a class="markdownIt-Anchor" href="#其他资源"></a> 其他资源</h1><ul><li>如何安装Go – <a href="https://go-zh.org/doc/install" target="_blank" rel="noopener">https://go-zh.org/doc/install</a></li><li>Go语言入门教程 – <a href="https://tour.go-zh.org/list" target="_blank" rel="noopener">https://tour.go-zh.org/list</a></li><li>开发环境配置及说明 – <a href="https://go-zh.org/doc/code.html" target="_blank" rel="noopener">https://go-zh.org/doc/code.html</a></li><li>IDEs – <a href="https://golang.google.cn/doc/editors.html" target="_blank" rel="noopener">https://golang.google.cn/doc/editors.html</a></li><li>Go Wiki – <a href="https://github.com/golang/go/wiki" target="_blank" rel="noopener">https://github.com/golang/go/wiki</a></li><li>Effective Go – <a href="https://go-zh.org/doc/effective_go.html" target="_blank" rel="noopener">https://go-zh.org/doc/effective_go.html</a></li></ul><hr><ul><li>godoc – <a href="https://go-zh.org/pkg/" target="_blank" rel="noopener">https://go-zh.org/pkg/</a></li><li>go command – <a href="https://go-zh.org/cmd/go/" target="_blank" rel="noopener">https://go-zh.org/cmd/go/</a></li><li>官方博客 – <a href="https://blog.go-zh.org/" target="_blank" rel="noopener">https://blog.go-zh.org/</a></li></ul><p>这里所列的资源，通过官网都可以找到链接。</p>]]></content>
<summary type="html">
这将是一个系列文章，笔者将会结合自身实践经历，逐步讲述在工作中如何使用go语言。
</summary>
<category term="实践" scheme="https://blog.mapleque.com/categories/practice/"/>
<category term="开始用go" scheme="https://blog.mapleque.com/categories/practice/go/"/>
<category term="golang" scheme="https://blog.mapleque.com/tags/golang/"/>
</entry>
<entry>
<title>【从零打造社区搜索推荐服务】架构设计</title>
<link href="https://blog.mapleque.com/posts/practice/0sc/0sc-artch/"/>
<id>https://blog.mapleque.com/posts/practice/0sc/0sc-artch/</id>
<published>2019-09-05T09:59:17.000Z</published>
<updated>2020-02-26T04:25:47.192Z</updated>
<content type="html"><![CDATA[<h1 id="整体架构"><a class="markdownIt-Anchor" href="#整体架构"></a> 整体架构</h1><p>依据解决复杂问题应当先化繁为简的原则，这里首先对服务进行拆分。</p><pre class="mermaid" style="text-align: center;"> graph TB client[客户端]sh[Searchhub]admin[后台]internal[后台服务]sync[数据同步服务]es[Elasticsearch]cache[Cache]mysql[Mysql]src((源数据))client --搜索&推荐--> shsh --用模板搜索--> essh --kv读写--> cachesh --sql只读--> mysqladmin --内部体验&管理数据--> internalinternal --用模板搜索&运营字段更新--> esinternal --运营数据更新--> mysqlinternal --刷缓存--> cachesrc --定时--> syncsync --写入\更新--> es </pre><p>按照数据流向可以将整个服务划分为三个部分（参考图中箭头指向）：</p><ul><li>前台服务 – 以searchhub为中心，将多方数据源的数据进行组织整理后返回给用户。</li><li>后台服务 – 以数据运营管理为主，让运营者有机会影响前台服务，进而达到运营的目的。</li><li>数据同步服务 – 让源数据与业务数据保持同步更新。</li></ul><h1 id="数据同步服务"><a class="markdownIt-Anchor" href="#数据同步服务"></a> 数据同步服务</h1><p>数据同步服务的目标就是让用于搜索的源数据能够与业务数据保持同步。</p><p>逻辑上可以划分为触发和写入两个步骤。</p><pre class="mermaid" style="text-align: center;"> graph LR 入口 --触发--> 新数据新数据 --写入--> Elasticsearch </pre><p>触发形式通常有两种：</p><table><thead><tr><th style="text-align:left">对比</th><th style="text-align:left">主动探测</th><th style="text-align:left">被动通知</th></tr></thead><tbody><tr><td style="text-align:left">实现方式</td><td style="text-align:left">定时运行</td><td style="text-align:left">订阅事件</td></tr><tr><td style="text-align:left">业务方支持</td><td style="text-align:left">支持增量查询</td><td style="text-align:left">提供消息订阅</td></tr><tr><td style="text-align:left">实时性</td><td style="text-align:left">定时周期决定</td><td style="text-align:left">消息处理时间决定</td></tr><tr><td style="text-align:left">回滚方式</td><td style="text-align:left">调整查询游标</td><td style="text-align:left">调整消费位置</td></tr><tr><td style="text-align:left">回滚范围</td><td style="text-align:left">全部</td><td style="text-align:left">队列长度</td></tr><tr><td style="text-align:left">瓶颈</td><td style="text-align:left">增量查询效率</td><td style="text-align:left">数据更新频率</td></tr></tbody></table><p>两种触发形式各有优劣，可以根据实际情况选择实现。</p><p>当获取到需要更新的数据后，只需要将其转化成需要的形式写入(更新)到Elasticsearch。</p><h1 id="前台服务"><a class="markdownIt-Anchor" href="#前台服务"></a> 前台服务</h1><p>前台服务，主要是通过Searchhub提供一系列接口供业务方调用，进而产品化后提供给客户端。</p><p>Searchhub需要提供以下接口。</p><h2 id="搜索"><a class="markdownIt-Anchor" href="#搜索"></a> 搜索</h2><ul><li>输入：搜索词、附加参数（分页、分类等）</li><li>返回：搜索结果</li></ul><h2 id="推荐"><a class="markdownIt-Anchor" href="#推荐"></a> 推荐</h2><ul><li>输入：内容id、用户id、附加参数（数量、策略等）</li><li>返回：推荐结果</li></ul><h2 id="热搜"><a class="markdownIt-Anchor" href="#热搜"></a> 热搜</h2><ul><li>输入：无</li><li>返回：热搜结果</li></ul><h1 id="后台服务"><a class="markdownIt-Anchor" href="#后台服务"></a> 后台服务</h1><p>后台服务需要实现以下功能。</p><h2 id="管理数据结构和索引"><a class="markdownIt-Anchor" href="#管理数据结构和索引"></a> 管理数据结构和索引</h2><p>对于index的setting和mapping的管理，接入后台方便开发和运营。</p><h2 id="查询源数据"><a class="markdownIt-Anchor" href="#查询源数据"></a> 查询源数据</h2><p>通过指定属性查询源数据，可用于后面的编辑。</p><h2 id="管理源数据运营相关字段"><a class="markdownIt-Anchor" href="#管理源数据运营相关字段"></a> 管理源数据运营相关字段</h2><p>对于源数据中一些用于运营的字段，可以通过后台直接编辑修改。</p><h2 id="管理搜索模板"><a class="markdownIt-Anchor" href="#管理搜索模板"></a> 管理搜索模板</h2><p>对于搜索模板的历史版本进行管理。可以直接修改线上所使用的模板版本。</p><h2 id="预览搜索效果"><a class="markdownIt-Anchor" href="#预览搜索效果"></a> 预览搜索效果</h2><p>可以指定模板版本进行搜索。以达到上线前预览的效果。</p><h2 id="管理屏蔽词"><a class="markdownIt-Anchor" href="#管理屏蔽词"></a> 管理屏蔽词</h2><p>对屏蔽词进行管理。</p><h2 id="管理缓存"><a class="markdownIt-Anchor" href="#管理缓存"></a> 管理缓存</h2><p>查询和清除缓存。</p>]]></content>
<summary type="html">
本文将讲述一个通过分别实现数据同步、Searchhub和后台等服务，实现一个产品化的易维护易扩展的社区搜索推荐服务的技术架构设计。
</summary>
<category term="实践" scheme="https://blog.mapleque.com/categories/practice/"/>
<category term="从零打造社区搜索推荐服务" scheme="https://blog.mapleque.com/categories/practice/0sc/"/>
<category term="Elasticsearch" scheme="https://blog.mapleque.com/tags/Elasticsearch/"/>
</entry>
<entry>
<title>【从零打造社区搜索推荐服务】问题的提出</title>
<link href="https://blog.mapleque.com/posts/practice/0sc/0sc-target/"/>
<id>https://blog.mapleque.com/posts/practice/0sc/0sc-target/</id>
<published>2019-09-05T09:59:01.000Z</published>
<updated>2020-02-26T04:25:47.193Z</updated>
<content type="html"><![CDATA[<p>社区搜索推荐，是一个现代化社区所应该具有的基本功能。</p><p>从技术角度讲，一个完整的社区搜索推荐服务，需要解决以下问题：</p><ul><li>对中文的处理</li><li>根据用户输入，结合社区内容给出相关搜索建议</li><li>根据搜索词给出相关社区内容的搜索结果</li><li>对于指定搜索词，运营人员能够控制第一页应该展示那些结果和顺序</li><li>可以屏蔽一些搜索词，让其不能出结果</li><li>可以屏蔽一些内容，让其永远不能出现在结果中</li><li>能够简单识别网络爬虫行为，并进行封禁</li></ul>]]></content>
<summary type="html">
主要讨论如何定义社区搜索推荐问题的范围。
</summary>
<category term="实践" scheme="https://blog.mapleque.com/categories/practice/"/>
<category term="从零打造社区搜索推荐服务" scheme="https://blog.mapleque.com/categories/practice/0sc/"/>
<category term="Elasticsearch" scheme="https://blog.mapleque.com/tags/Elasticsearch/"/>
</entry>
<entry>
<title>【从零打造社区搜索推荐服务】序</title>
<link href="https://blog.mapleque.com/posts/practice/0sc/0sc-start/"/>
<id>https://blog.mapleque.com/posts/practice/0sc/0sc-start/</id>
<published>2019-09-05T09:43:19.000Z</published>
<updated>2020-02-26T04:25:47.193Z</updated>
<content type="html"><![CDATA[<h1 id="目录"><a class="markdownIt-Anchor" href="#目录"></a> 目录</h1><h2 id="基础篇"><a class="markdownIt-Anchor" href="#基础篇"></a> 基础篇</h2><ul><li><a href="/posts/practice/0sc/0sc-target/">问题的提出</a> – 全面精确的定义要解决的问题</li><li><a href="/posts/practice/0sc/0sc-artch/">架构设计</a> – 站在全局看服务</li><li>（挖坑待填）<a href>更新源数据</a> – 源数据更新时机和数据清洗</li><li>（挖坑待填）<a href>朴素的全文搜索</a> – 基于基本数据进行全文搜索</li><li>（挖坑待填）<a href>搜索建议的冷启动数据</a> – 生成能够支持搜索建议的最简数据集</li><li>（挖坑待填）<a href>用搜索的思路做推荐</a> – 使用标签做内容相关推荐</li><li>（挖坑待填）<a href>基于用户的标签的个性化推荐</a> – 浏览记录通常能如实反映用户的喜好</li></ul><h2 id="进阶篇"><a class="markdownIt-Anchor" href="#进阶篇"></a> 进阶篇</h2><ul><li>（挖坑待填）<a href>缓存的应用</a> – 通过缓存提升性能上线</li><li>（挖坑待填）<a href>黑白名单</a> – 各种黑白名单用来应对各种问题</li><li>（挖坑待填）<a href>社区内容特征提取</a> – 朴素贝叶斯算法的应用</li><li>（挖坑待填）<a href>通过协同过滤优化用户标签</a> – 加入协同过滤以优化推荐效果</li></ul><h2 id="终极篇"><a class="markdownIt-Anchor" href="#终极篇"></a> 终极篇</h2><ul><li>（挖坑待填）<a href>无监督学习下的内容特征提取</a> – LDA的探索</li><li>（挖坑待填）<a href>带反馈的智能搜索架构</a></li></ul><h2 id="其他"><a class="markdownIt-Anchor" href="#其他"></a> 其他</h2><ul><li><a href>待补充</a></li></ul>]]></content>
<summary type="html">
这将是一个系列文章，讲述的是社区搜索推荐服务从无到有，从平凡到优秀的历程。
</summary>
<category term="实践" scheme="https://blog.mapleque.com/categories/practice/"/>
<category term="从零打造社区搜索推荐服务" scheme="https://blog.mapleque.com/categories/practice/0sc/"/>
<category term="Elasticsearch" scheme="https://blog.mapleque.com/tags/Elasticsearch/"/>
</entry>
<entry>
<title>在nodejs中使用puppeteer并通过docker部署</title>
<link href="https://blog.mapleque.com/posts/tool/puppeteer/js-puppeteer-docker/"/>
<id>https://blog.mapleque.com/posts/tool/puppeteer/js-puppeteer-docker/</id>
<published>2019-08-29T08:22:23.000Z</published>
<updated>2020-02-26T04:25:47.197Z</updated>
<content type="html"><![CDATA[<p><a href="https://github.com/GoogleChrome/puppeteer#readme" target="_blank" rel="noopener">Puppeteer</a>是Google基于Chromium开发的一个Node库。</p><p>用户通过调用Puppeteer的Api，可以做到任何Chrome浏览器能做到的事情。</p><p>因此，其应用领域可能包括：</p><ul><li>网页截图生成报告</li><li>服务端渲染</li><li>面向网页应用的自动化测试</li><li>更接近真实情况的探针监控</li><li>需要处理复杂逻辑的爬虫</li></ul><h2 id="开发环境"><a class="markdownIt-Anchor" href="#开发环境"></a> 开发环境</h2><p>安装puppeteer包：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">npm i puppeteer --save</span><br></pre></td></tr></table></figure><div class="note info"> <p>这里仅介绍基本用法以保证快速进入开发，高阶用法请参考<a href="https://github.com/GoogleChrome/puppeteer#readme" target="_blank" rel="noopener">官方文档</a>。</p> </div><p>编写代码：</p><figure class="highlight javascript"><figcaption><span>example.js</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> puppeteer = <span class="built_in">require</span>(<span class="string">'puppeteer'</span>);</span><br><span class="line"></span><br><span class="line">(<span class="keyword">async</span> () => {</span><br><span class="line"> <span class="keyword">const</span> browser = <span class="keyword">await</span> puppeteer.launch();</span><br><span class="line"> <span class="keyword">const</span> page = <span class="keyword">await</span> browser.newPage();</span><br><span class="line"> <span class="keyword">await</span> page.goto(<span class="string">'https://example.com'</span>);</span><br><span class="line"> <span class="keyword">await</span> page.screenshot({<span class="attr">path</span>: <span class="string">'example.png'</span>});</span><br><span class="line"></span><br><span class="line"> <span class="keyword">await</span> browser.close();</span><br><span class="line">})();</span><br></pre></td></tr></table></figure><p>执行脚本：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">node example.js</span><br></pre></td></tr></table></figure><h2 id="生产环境"><a class="markdownIt-Anchor" href="#生产环境"></a> 生产环境</h2><p>如果是专用的带有node环境的linux机器，正常安装puppeteer包即可使用。</p><p>这里特别介绍一下在Docker环境下安装部署的过程。</p><p>我们通过创建一个装好了puppeteer包的镜像来提供部署环境：</p><figure class="highlight dockerfile"><figcaption><span>Dockerfile</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># node-chrome-10.16</span></span><br><span class="line"><span class="keyword">FROM</span> node:<span class="number">10.16</span>-alpine</span><br><span class="line"></span><br><span class="line"><span class="keyword">ENV</span> APP_PATH /app</span><br><span class="line"><span class="keyword">WORKDIR</span><span class="bash"> <span class="variable">${APP_PATH}</span></span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Change mirrors to tsinghua</span></span><br><span class="line"><span class="keyword">RUN</span><span class="bash"> <span class="built_in">echo</span> http://mirrors.tuna.tsinghua.edu.cn/alpine/edge/main > /etc/apk/repositories && \</span></span><br><span class="line"><span class="bash"> <span class="built_in">echo</span> http://mirrors.tuna.tsinghua.edu.cn/alpine/edge/community >> /etc/apk/repositories && \</span></span><br><span class="line"><span class="bash"> <span class="built_in">echo</span> http://mirrors.tuna.tsinghua.edu.cn/alpine/edge/testing >> /etc/apk/repositories && apk update</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Setting timezone</span></span><br><span class="line"><span class="keyword">RUN</span><span class="bash"> apk add tzdata openssh-client git</span></span><br><span class="line"><span class="keyword">RUN</span><span class="bash"> cp -r -f /usr/share/zoneinfo/Hongkong /etc/localtime</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Installs cnpm</span></span><br><span class="line"><span class="keyword">RUN</span><span class="bash"> npm install -g cnpm --registry=https://registry.npm.taobao.org</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Installs latest Chromium (73) package.</span></span><br><span class="line"><span class="keyword">RUN</span><span class="bash"> apk add --no-cache \</span></span><br><span class="line"><span class="bash"> curl \</span></span><br><span class="line"><span class="bash"> make \</span></span><br><span class="line"><span class="bash"> gcc \</span></span><br><span class="line"><span class="bash"> g++ \</span></span><br><span class="line"><span class="bash"> python \</span></span><br><span class="line"><span class="bash"> linux-headers \</span></span><br><span class="line"><span class="bash"> binutils-gold \</span></span><br><span class="line"><span class="bash"> gnupg \</span></span><br><span class="line"><span class="bash"> libstdc++ \</span></span><br><span class="line"><span class="bash"> udev \</span></span><br><span class="line"><span class="bash"> chromium=~73.0.3683.103 \</span></span><br><span class="line"><span class="bash"> nss \</span></span><br><span class="line"><span class="bash"> freetype \</span></span><br><span class="line"><span class="bash"> freetype-dev \</span></span><br><span class="line"><span class="bash"> harfbuzz \</span></span><br><span class="line"><span class="bash"> ttf-freefont \</span></span><br><span class="line"><span class="bash"> wqy-zenhei</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Tell Puppeteer to skip installing Chrome. We'll be using the installed package.</span></span><br><span class="line"><span class="keyword">ENV</span> PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true</span><br><span class="line"></span><br><span class="line"><span class="comment"># Puppeteer v1.12.2 works with Chromium 73.</span></span><br><span class="line"><span class="keyword">RUN</span><span class="bash"> yarn add [email protected]</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">RUN</span><span class="bash"> apk del --no-cache make gcc g++ python binutils-gold gnupg libstdc++</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Add user so we don't need --no-sandbox.</span></span><br><span class="line"><span class="comment">#RUN addgroup -S pptruser && adduser -S -g pptruser pptruser \</span></span><br><span class="line"><span class="comment"># && mkdir -p /home/pptruser/Downloads /app \</span></span><br><span class="line"><span class="comment"># && chown -R pptruser:pptruser /home/pptruser \</span></span><br><span class="line"><span class="comment"># && chown -R pptruser:pptruser /app</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment">## Run everything after as non-privileged user.</span></span><br><span class="line"><span class="comment">#USER pptruser</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">CMD</span><span class="bash"> [<span class="string">'/bin/sh'</span>]</span></span><br></pre></td></tr></table></figure><div class="note info"> <p>上面代码中注释掉了给puppeteer创建用户运行的部分，如果你的docker服务是在root下启动的，上面的镜像可以正常工作。<br>否则就需要给执行该镜像的用户开通一些权限，可以在后面执行的时候注意输出错误日志以获取所需权限。</p> </div><p>在上面的环境中使用puppeteer，需要在初始化的时候进行一些特别的配置：</p><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> puppeteer = <span class="built_in">require</span>(<span class="string">'puppeteer'</span>)</span><br><span class="line">(<span class="keyword">async</span> () => {</span><br><span class="line"> <span class="keyword">const</span> browser = <span class="keyword">await</span> puppeteer.launch({</span><br><span class="line"> args: [</span><br><span class="line"> <span class="string">'--disable-dev-shm-usage'</span>,</span><br><span class="line"> <span class="string">'--no-sandbox'</span></span><br><span class="line"> ],</span><br><span class="line"> executablePath: <span class="string">'/usr/bin/chromium-browser'</span></span><br><span class="line"> })</span><br><span class="line"> <span class="keyword">const</span> page = <span class="keyword">await</span> browser.newPage();</span><br><span class="line"> <span class="keyword">await</span> page.goto(<span class="string">'https://example.com'</span>);</span><br><span class="line"> <span class="keyword">await</span> page.screenshot({<span class="attr">path</span>: <span class="string">'example.png'</span>});</span><br><span class="line"></span><br><span class="line"> <span class="keyword">await</span> browser.close();</span><br><span class="line">})();</span><br></pre></td></tr></table></figure><p>最后，在上面Docker环境下执行代码，或者部署项目即可。</p><p>这里给出一个egg项目部署的Dockerfile作为参考：</p><figure class="highlight dockerfile"><figcaption><span>Dockerfile</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">FROM</span> mapleque/node-chrome:<span class="number">10.16</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">ENV</span> APP_PATH /app</span><br><span class="line"></span><br><span class="line"><span class="keyword">WORKDIR</span><span class="bash"> <span class="variable">${APP_PATH}</span></span></span><br><span class="line"></span><br><span class="line"><span class="comment"># COPY --chown=pptruser:pptruser . ${APP_PATH}</span></span><br><span class="line"><span class="keyword">COPY</span><span class="bash"> . <span class="variable">${APP_PATH}</span></span></span><br><span class="line"></span><br><span class="line"><span class="keyword">RUN</span><span class="bash"> cnpm install --production</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">ENTRYPOINT</span><span class="bash"> NODE_ENV=production EGG_SERVER_ENV=<span class="variable">${EGG_SERVER_ENV}</span> npx egg-scripts start --port=<span class="variable">${APP_PORT}</span> --workers=1</span></span><br></pre></td></tr></table></figure><div class="note info"> <p>同样需要注意的是，这里启动项目的用户要与环境中授权puppeteer的用户保持一致。</p> </div>]]></content>
<summary type="html">
在Node中使用Puppeteer包实现爬虫，然后通过Docker进行部署，其中一些问题记录在这里，以免走弯路
</summary>
<category term="工具" scheme="https://blog.mapleque.com/categories/tool/"/>
<category term="puppeteer" scheme="https://blog.mapleque.com/categories/tool/puppeteer/"/>
<category term="nodejs" scheme="https://blog.mapleque.com/tags/nodejs/"/>
<category term="puppeteer" scheme="https://blog.mapleque.com/tags/puppeteer/"/>
<category term="docker" scheme="https://blog.mapleque.com/tags/docker/"/>
</entry>
<entry>
<title>实现OIDC协议</title>
<link href="https://blog.mapleque.com/posts/practice/auth/oidc/"/>
<id>https://blog.mapleque.com/posts/practice/auth/oidc/</id>
<published>2019-08-29T07:57:39.000Z</published>
<updated>2020-02-26T04:25:47.198Z</updated>
<content type="html"><![CDATA[<blockquote><p>OpenID Connect 1.0 is a simple identity layer on top of the OAuth 2.0 protocol. It allows Clients to verify the identity of the End-User based on the authentication performed by an Authorization Server, as well as to obtain basic profile information about the End-User in an interoperable and REST-like manner.</p><footer><strong>OpenID Connect</strong><cite><a href="https://openid.net/connect/" target="_blank" rel="noopener">openid.net/connect</a></cite></footer></blockquote><p>OIDC工作原理和应用场景如下图所示：</p><pre class="mermaid" style="text-align: center;"> sequenceDiagram participant u as userparticipant app as oidc-appparticipant oidc as oidc-serverapp ->>+ oidc: 获取oidc配置Note over app,oidc: .well-known/oepnid-configurationoidc ->> app: 返回oidc配置app ->>+ oidc: 获取密钥Note over app,oidc: conf.jwks_urioidc ->>- app: 返回jwks并缓存u ->>+ app: 未登录用户请求app ->>- u: 跳转登录页u ->>+ oidc: 登录并授权Note over app,oidc: conf.authorization_endpointNote over oidc: 生成授权码codeoidc ->>- u: 返回授权码codeu ->>+ app: 提交授权码codeapp ->>+ oidc: 申请tokenNote over app,oidc: conf.token_endpointNote over oidc: 生成jwt-tokenoidc ->>- app: 返回jwt-tokenapp ->>- u: 返回服务token，可以直接使用jwk-tokenu ->>+ app: 带jwt-token请求Note over app: 解密jwt-tokenNote over app: 验证权限app ->>- u: 返回请求结果 </pre><p>其中，用户请求应用，需要把jwt-token放在请求Header中：<code>Authorization: Bearer <jwt-token></code>。</p><p>应用通过缓存的jwks解析jwt-token获取用户信息并验证相关权限。</p>]]></content>
<summary type="html">
介绍如何实现OIDC协议
</summary>
<category term="实践" scheme="https://blog.mapleque.com/categories/practice/"/>
<category term="用户认证" scheme="https://blog.mapleque.com/categories/practice/auth/"/>
<category term="Kubernates" scheme="https://blog.mapleque.com/tags/Kubernates/"/>
<category term="OIDC" scheme="https://blog.mapleque.com/tags/OIDC/"/>
<category term="jwt" scheme="https://blog.mapleque.com/tags/jwt/"/>
</entry>
<entry>
<title>微信小程序快速接入实践</title>
<link href="https://blog.mapleque.com/posts/practice/wx/mpapi/"/>
<id>https://blog.mapleque.com/posts/practice/wx/mpapi/</id>
<published>2019-08-29T07:52:07.000Z</published>
<updated>2020-02-26T04:25:47.198Z</updated>
<content type="html"><![CDATA[<p>微信小程序作为一种轻量级移动应用形式，体系发展已经日趋成熟。以微信小程序作为主要载体的应用也越来越多。投入少，见效快，自然的获取用户渠道是其优势所在。笔者恰好经历了2018年小程序大爆发时期，也义无反顾地踏入过这趟浑水。无论如何，最终留下的经验才是最宝贵的，也就是这里将要介绍的一套快速接入微信小程序后端开发迭代方案。</p><h1 id="引言"><a class="markdownIt-Anchor" href="#引言"></a> 引言</h1><p>这里之所以说是快速接入小程序的后端开发迭代方案，其原因有三：</p><ul><li>小程序前端基于javascript语法，提供了一些内置扩展组件以及一套成熟的设计标准，在开发上没有任何余地，也无需优化。</li><li>小程序后端最基本的登录功能，都需要实现复杂的调用微信api逻辑，还有一些列更复杂的加解密验签接口可能需要接入，此外还要实现基本的登录态保持逻辑，工作量很大。</li><li>当需要快速实现多个小程序进行试错的时候，大量重复的工作都可以通过一个完整的服务来优化。</li></ul><p>综上，笔者就提出了一套方案。</p><h1 id="方案描述"><a class="markdownIt-Anchor" href="#方案描述"></a> 方案描述</h1><pre class="mermaid" style="text-align: center;"> sequenceDiagram participant u as 用户participant mp as 小程序participant mpapi as 小程序代理服务participant api as 小程序后端服务participant wx as 微信u ->> mp: 用户使用小程序mp ->> mpapi: 小程序请求代理服务mpapi ->> wx: 小程序请求微信wx ->> mpapi: 微信返回数据给代理服务mpapi ->> api: 代理服务带着微信的数据请求后端服务api ->> mp: 后端服务返回数据给小程序mp ->> u: 小程序反馈给用户结果 </pre><p>如图所示，微信小程序在需要请求后端的时候，直接请求代理服务，由代理服务实现接入微信并请求后端的逻辑，最后把数据返回给用户。</p><p>代理服务在请求小程序后端服务的时候，将会携带从微信获取的数据，如：用户信息。</p><p>代理服务可以维护用户登录状态，并给用户创建统一的id进行管理。</p><h1 id="实现"><a class="markdownIt-Anchor" href="#实现"></a> 实现</h1><p>mpapi(<a href="https://github.com/mapleque/mpapi" target="_blank" rel="noopener">Code on Github</a>)使上述方案中代理服务的一个实现。</p><p>支持使用线上服务和私有化部署两种方式，参考<a href="https://github.com/mapleque/mpapi" target="_blank" rel="noopener">使用文档</a>。</p>]]></content>
<summary type="html">
介绍一套微信小程序后端快速接入开发迭代的实践方案
</summary>
<category term="实践" scheme="https://blog.mapleque.com/categories/practice/"/>
<category term="微信开发者" scheme="https://blog.mapleque.com/categories/practice/wx/"/>
<category term="wx" scheme="https://blog.mapleque.com/tags/wx/"/>
<category term="miniprogram" scheme="https://blog.mapleque.com/tags/miniprogram/"/>
</entry>
<entry>
<title>基于Kerberos实现内部系统员工账号单点登录</title>
<link href="https://blog.mapleque.com/posts/practice/auth/kerberos/"/>
<id>https://blog.mapleque.com/posts/practice/auth/kerberos/</id>
<published>2019-08-29T07:19:01.000Z</published>
<updated>2020-02-26T04:25:47.197Z</updated>
<content type="html"><![CDATA[<blockquote><p>Kerberos is a network authentication protocol. It is designed to provide strong authentication for client/server applications by using secret-key cryptography.</p><footer><strong>What is Kerberos?</strong><cite><a href="http://web.mit.edu/kerberos" target="_blank" rel="noopener">web.mit.edu/kerberos</a></cite></footer></blockquote><p>Kerberos工作原理和应用场景如下图所示：</p><pre class="mermaid" style="text-align: center;"> sequenceDiagram participant u as Userparticipant client as Clientparticipant server as Serverparticipant kdc as Kerberosu ->> client: username, passwordclient ->> client: cache(username, password)client ->> kdc: (to AS) usernamekdc ->> kdc: CTSK = randkdc ->> kdc: TGT = {CTSK, username, expired}kdc ->> kdc: password = findBy(username)kdc ->> kdc: EncCTSK = enc(CTSK, password)kdc ->> kdc: TGSSecretKey = constkdc ->> kdc: EncTGT = enc(TGT, TGSSecretKey)kdc ->> client: EncCTSK, EncTGTclient ->> client: password = loadCache()client ->> client: CTSK = decrypt(EncCTSK, password)client ->> client: cache(CTSK)client ->> client: Authenticator = {username, timestamp}client ->> client: cache(Authenticator)client ->> client: CTSKEncAuthenticator = enc(Authenticator, CTSK)client ->> kdc: (to TGS) EncTGT, AppID, CTSKEncAuthenticatorkdc ->> kdc: TGSSecretKey = constkdc ->> kdc: TGT = decrypt(EncTGT, TGSSecretKey)kdc ->> kdc: valid(TGT.expired)kdc ->> kdc: CTSK = TGT.CTSKkdc ->> kdc: Authenticator = decrypt(CTSKEncAuthenticator, CTSK)kdc ->> kdc: username = Authenticator.usernamekdc ->> kdc: AppSecretKey = findBy(AppID)kdc ->> kdc: CSSK = randkdc ->> kdc: ST = {CSSK, username, expired}kdc ->> kdc: EncST = enc(ST, AppSecretKey)kdc ->> kdc: EncCSSK = enc(CSSK, CTSK)kdc ->> client: EncST, EncCSSKclient ->> client: CTSK = loadCache()client ->> client: CSSK = decrypt(EncCSSK, CTSK)client ->> client: cache(CSSK)client ->> client: Authenticator = loadCache()client ->> client: CSSKEncAuthenticator = enc(Authenticator, CSSK)client ->> server: EncST, CSSKEncAuthenticatorserver ->> server: AppSecretKey = constserver ->> server: ST = decrypt(EncST, AppSecretKey)server ->> server: valid(ST.expired)server ->> server: CSSK = ST.CSSKserver ->> server: Authenticator = descrypt(CSSKEncAuthenticator, CSSK)server ->> server: valid(Authenticator.username == ST.username)server ->> server: Token = randserver ->> server: EncToken = enc(Token, CSSK)server ->> client: EncTokenclient ->> client: CSSK = loadCache()client ->> client: Token = decrypt(EncToken, CSSK)client ->> client: cache(Token)u ->>+ client: actionclient ->>+ server: request, Tokenserver ->>- client: response dataclient ->>- u: view </pre><p>其中：</p><ul><li>User – 用户</li><li>Client – 应用客户端</li><li>Server – 应用服务端</li><li>Kerberos – 认证授权服务，包含AS,TGS等</li><li>AS – 认证服务(Authentication Service)</li><li>TGS – 票据授权服务(Ticket Granting Service)</li><li>username – 用户输入的用户名</li><li>password – 用户输入的密码</li><li>EncCTSK – 通过AS上的用户password加密的CTSK</li><li>CTSK – 用于Client与TGS通信的密钥(Client TGS Session Key)</li><li>EncTGT – 通过TGSSecretKey加密的TGT</li><li>TGT – 授权票据(Ticket Granting Ticket)，包含CTSK、username和expired</li><li>Authenticator – 将username和timestamp包装为一个Authenticator</li><li>CTSKEncAuthenticator – 通过CTSK加密的Authenticator</li><li>AppID – 唯一标识一个应用服务提供方，应用服务可能包含Client和Server</li><li>TGSSecretKey – TGS自己的维护的密钥</li><li>EncST – 通过AppSecretKey加密的ST</li><li>AppSecretKey – 应用服务提供方在TGS上注册的密钥，一个AppID对应一个TGSSecretKey</li><li>ST – 应用票据(Service Ticket)</li><li>EncCSSK – 通过CTSK加密的CSSK</li><li>CSSK – 用于Client与Server通信的密钥Client Server Session Key</li><li>Token – 用于Client与Server通信的会话标识，由应用实现，可以是Cookie，Session，HTTP Header等</li></ul><p>从上图中可以看出，整个过程分成三个部分：</p><ul><li>认证 – 确认用户身份和客户端是否可信</li><li>授权 – 授权用户访问指定服务</li><li>登录 – 用户凭借授权去登录服务，建立会话</li></ul><p>这里有两个特点需要强调：</p><ul><li>应用服务与认证授权服务在整个认证过程中没有直接通信</li><li>用户密码在整个认证过程中没有在网络上传播</li></ul><p>上面的认证授权服务在实际应用上，还欠缺了两部分：</p><ul><li>管理用户密码 – 需要给用户提供线上服务</li><li>管理应用密钥 – 需要给开发者提供线上(可离线)服务</li></ul>]]></content>
<summary type="html">
介绍如何实现Kerberos协议并在内部系统中使用该协议实现员工账号单点登录
</summary>
<category term="实践" scheme="https://blog.mapleque.com/categories/practice/"/>
<category term="用户认证" scheme="https://blog.mapleque.com/categories/practice/auth/"/>
<category term="Kerberos" scheme="https://blog.mapleque.com/tags/Kerberos/"/>
</entry>
<entry>
<title>使用openvpn管理办公网络</title>
<link href="https://blog.mapleque.com/posts/tool/openvpn/openvpn/"/>
<id>https://blog.mapleque.com/posts/tool/openvpn/openvpn/</id>
<published>2019-08-29T07:06:23.000Z</published>
<updated>2020-02-26T04:25:47.198Z</updated>
<content type="html"><![CDATA[<p>使用openvpn管理办公网络，可以做到：</p><ul><li>以证书作为客户端认证标识，通过颁发证书管理用户接入网络的权限</li><li>每个证书指定固定ip，在网络上可以监控ip行为，对应到用户</li><li>通过对用户分配不同ip段的ip，来控制用户在网络中的权限</li><li>推送内网域名解析，复杂内网路由保证复杂内网连通性</li></ul><h1 id="服务端"><a class="markdownIt-Anchor" href="#服务端"></a> 服务端</h1><h2 id="主服务"><a class="markdownIt-Anchor" href="#主服务"></a> 主服务</h2><ul><li><a href="https://github.com/OpenVPN/openvpn" target="_blank" rel="noopener">github.com/OpenVPN/openvpn</a></li><li><a href="https://openvpn.net/community-resources/how-to/" target="_blank" rel="noopener">官网文档openvpn.net</a>(需要翻墙)</li><li><a href="https://github.com/Nyr/openvpn-install" target="_blank" rel="noopener">一键安装脚本</a></li></ul><p>配置文件样例：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br></pre></td><td class="code"><pre><span class="line">verb 3 # LOG等级</span><br><span class="line">dev tun # 隧道设备名</span><br><span class="line">port 1194</span><br><span class="line"></span><br><span class="line"># 如果上层使用udp, vpn使用tcp, 丢包后vpn会就会重传上层的udp报文, vpn使用udp协议, 这样可以让上层应用自己决定使用tcp还是udp</span><br><span class="line">proto udp</span><br><span class="line"></span><br><span class="line">topology subnet # 此模式下所有客户端在一个子网里</span><br><span class="line"></span><br><span class="line"># keys</span><br><span class="line">ca /etc/openvpn/pki/ca.crt</span><br><span class="line">key /etc/openvpn/pki/private/vpn.mapleque.com.key</span><br><span class="line">cert /etc/openvpn/pki/issued/vpn.mapleque.com.crt</span><br><span class="line">crl-verify /etc/openvpn/pki/crl.pem</span><br><span class="line">dh /etc/openvpn/pki/dh.pem</span><br><span class="line">tls-auth /etc/openvpn/pki/ta.key</span><br><span class="line"></span><br><span class="line">key-direction 0</span><br><span class="line">keepalive 10 60</span><br><span class="line"></span><br><span class="line"># 超时重连接时不会重载key和tun</span><br><span class="line">persist-key</span><br><span class="line">persist-tun</span><br><span class="line"></span><br><span class="line">status /var/log/openvpn/status.log</span><br><span class="line">log /var/log/openvpn/log.log</span><br><span class="line">management localhost 7505</span><br><span class="line"></span><br><span class="line">client-config-dir /etc/openvpn/ccd</span><br><span class="line"></span><br><span class="line">client-to-client # 允许client之间通信</span><br><span class="line"></span><br><span class="line">user vpn</span><br><span class="line">group vpn</span><br><span class="line"></span><br><span class="line"># vpn进程启动后给本机添加该路由(不需要有客户端连接), 可以配置多条</span><br><span class="line">route 172.31.0.0 255.255.0.0 192.168.232.20</span><br><span class="line"></span><br><span class="line"># 启动server模式, 隧道使用该网段, 并占用第一个地址用作本机隧道的IP, nopool表示不将此网段用作dhcp</span><br><span class="line">server 192.168.224.0 255.255.240.0 nopool</span><br><span class="line"></span><br><span class="line"># 单独配置dhcp pool</span><br><span class="line">ifconfig-pool 192.168.224.2 192.168.225.254 255.255.240.0</span><br><span class="line"></span><br><span class="line">push "dhcp-option DNS 192.168.224.1"</span><br><span class="line"></span><br><span class="line">push "route 192.168.16.0 255.255.240.0 vpn_gateway"</span><br></pre></td></tr></table></figure><p>systemd启动配置样例：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"># This service is actually a systemd target,</span><br><span class="line"># but we are using a service since targets cannot be reloaded.</span><br><span class="line"># /lib/systemd/system/openvpn.service</span><br><span class="line"></span><br><span class="line">[Unit]</span><br><span class="line">Description=OpenVPN service</span><br><span class="line">After=network.target</span><br><span class="line"></span><br><span class="line">[Service]</span><br><span class="line">Type=forking</span><br><span class="line">ExecStart=/usr/sbin/openvpn --daemon ovpn --cd /etc/openvpn --config /etc/openvpn/vpn.conf</span><br><span class="line">ExecReload=/bin/kill -HUP $MAINPID</span><br><span class="line">WorkingDirectory=/etc/openvpn</span><br><span class="line"></span><br><span class="line">[Install]</span><br><span class="line">WantedBy=multi-user.target</span><br></pre></td></tr></table></figure><h2 id="证书颁发"><a class="markdownIt-Anchor" href="#证书颁发"></a> 证书颁发</h2><p>可以考虑使用easyrsa签发证书。</p><h1 id="客户端"><a class="markdownIt-Anchor" href="#客户端"></a> 客户端</h1><p>安装vpn客户端有风险，注意看好官方客户端版本。大部分客户端下载都需要翻墙，请自行检索解决方案。</p><ul><li><a href="https://itunes.apple.com/us/app/openvpn-connect/id590379981?mt=8" target="_blank" rel="noopener">IOS</a></li><li><a href="https://play.google.com/store/apps/details?id=net.openvpn.openvpn" target="_blank" rel="noopener">Android</a></li><li><a href="https://openvpn.net/downloads/openvpn-connect-v3-macos.dmg" target="_blank" rel="noopener">MacOS</a></li><li><a href="https://openvpn.net/vpn-server-resources/connecting-to-access-server-with-linux/" target="_blank" rel="noopener">Linux</a></li><li><a href="https://openvpn.net/downloads/openvpn-connect-v3-windows.msi" target="_blank" rel="noopener">Windows</a></li></ul>]]></content>
<summary type="html">
本文提供了一套使用openvpn管理办公网络的方案，兼顾安全性和便捷性
</summary>
<category term="工具" scheme="https://blog.mapleque.com/categories/tool/"/>
<category term="openvpn" scheme="https://blog.mapleque.com/categories/tool/openvpn/"/>
<category term="vpn" scheme="https://blog.mapleque.com/tags/vpn/"/>
</entry>
</feed>