-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathsearch.xml
889 lines (428 loc) · 279 KB
/
search.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
<?xml version="1.0" encoding="utf-8"?>
<search>
<entry>
<title>【开始用go】http服务</title>
<link href="/posts/practice/go/go-practice-http/"/>
<url>/posts/practice/go/go-practice-http/</url>
<content type="html"><![CDATA[<h2 id="配置加载"><a class="markdownIt-Anchor" href="#配置加载"></a> 配置加载</h2><p>服务在启动的时候通常都需要一些配置来提供启动参数，如：监听端口，其他服务连接密钥等。</p><p>这里建议所有配置都通过环境变量读取，这样做的原因有以下几点：</p><ol><li>生产环境的系统配置可能较为复杂，可以根据不同环境调整配置不同参数</li><li>生产环境密钥应在运维范畴保密，不应透露到开发者层面</li><li>开发环境更为复杂，开发者可以根据自己的环境灵活配置启动</li></ol><p>go标准包<code>os</code>中提供了获取环境变量的方法：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"os"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line">port := os.Getenv(<span class="string">"UC_LISTEN_PORT"</span>)</span><br><span class="line"> <span class="comment">// ...</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="服务定义和启动"><a class="markdownIt-Anchor" href="#服务定义和启动"></a> 服务定义和启动</h2><p>为了避免全局变量的失控，我们通常会为服务定义一个实例，然后通过实例接口让其在main中被加载和启动。</p><p>那么对于一个http服务，我们需要为其实现http.Handler接口：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Service impement http.Handler interface</span></span><br><span class="line"><span class="comment">// which can be initialized in a http server.</span></span><br><span class="line"><span class="keyword">type</span> Service <span class="keyword">struct</span>{}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">New</span><span class="params">()</span> *<span class="title">Service</span></span> {</span><br><span class="line"><span class="keyword">return</span> &Service{}</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *Service)</span> <span class="title">ServeHTTP</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line"> <span class="comment">// TODO ...</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>然后，在main中实例化我们定义的服务，然后就可以通过标准包<code>net/http</code>提供的方法启动了：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"net/http"</span></span><br><span class="line"><span class="string">"os"</span></span><br><span class="line"></span><br><span class="line">uc <span class="string">"github.com/mapleque/gostart/ms/uc/service"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line">port := os.Getenv(<span class="string">"UC_LISTEN_PORT"</span>)</span><br><span class="line">s := uc.New()</span><br><span class="line">server.ListenAndServe(<span class="string">"0.0.0.0:"</span>+port, s)</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="路由和处理函数"><a class="markdownIt-Anchor" href="#路由和处理函数"></a> 路由和处理函数</h2><p>依据标准包<code>net/http</code>的实现，所有请求最终都会经过<code>ServeHTTP</code>方法处理，并且每个请求的处理都是一个单独的协程。</p><p>所以在<code>ServeHTTP</code>方法中，我们主要实现的就是为当前请求分配处理函数。这里也可以直接使用标准包<code>net/http</code>中的<code>ServeMux</code>实现：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Service implement http.Handler interface</span></span><br><span class="line"><span class="comment">// which can be initialized in a http server.</span></span><br><span class="line"><span class="keyword">type</span> Service <span class="keyword">struct</span> {</span><br><span class="line">mux *http.ServeMux</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// ServeHTTP implement http.Handler interface.</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *Service)</span> <span class="title">ServeHTTP</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line">s.mux.ServeHTTP(w, req)</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>很明显，要让</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *Service)</span> <span class="title">signin</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line"> <span class="comment">// TODO deal with /signin request</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *Service)</span> <span class="title">signout</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line"> <span class="comment">// TODO deal with /signout request</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *Service)</span> <span class="title">userinfo</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line"> <span class="comment">// deal with /userinfo request</span></span><br><span class="line"> <span class="keyword">switch</span> req.Method {</span><br><span class="line"> <span class="keyword">case</span> http.MethodGet:</span><br><span class="line"> s.getUserinfo(w, req)</span><br><span class="line"> <span class="keyword">case</span> http.MethodPost:</span><br><span class="line"> s.postUserinfo(w, req)</span><br><span class="line"> <span class="keyword">default</span>:</span><br><span class="line"> <span class="comment">// return 405</span></span><br><span class="line"> w.WriteHeader(http.StatusMethodNotAllowed)</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *Service)</span> <span class="title">getUserinfo</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line"> <span class="comment">// TODO deal with GET /userinfo request</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *Service)</span> <span class="title">postUserinfo</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line"> <span class="comment">// TODO deal with POST /userinfo request</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="参数定义和校验"><a class="markdownIt-Anchor" href="#参数定义和校验"></a> 参数定义和校验</h2><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> SigninParam <span class="keyword">struct</span> {</span><br><span class="line"> Username <span class="keyword">string</span> <span class="string">`json:"username"`</span></span><br><span class="line"> Password <span class="keyword">string</span> <span class="string">`json:"password"`</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">bindAndValid</span><span class="params">(req *http.Request, obj <span class="keyword">interface</span>{})</span> <span class="title">error</span></span> {</span><br><span class="line"> <span class="keyword">if</span> req == <span class="literal">nil</span> || req.Body == <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> fmt.Errorf(<span class="string">"invalid request"</span>)</span><br><span class="line"> }</span><br><span class="line"> decoder := json.NewDecoder(req)</span><br><span class="line"> <span class="keyword">if</span> err := decoder.Decode(obj); err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> err</span><br><span class="line"> }</span><br><span class="line"> <span class="comment">// use github.com/go-playground/validator/v10</span></span><br><span class="line"> <span class="keyword">return</span> validate.Struct(obj)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *Service)</span> <span class="title">signin</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line"> param := SigninParam{}</span><br><span class="line"> <span class="keyword">if</span> err := bindAndValid(req, &param); err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="comment">// TODO response param-check error message</span></span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line"> }</span><br><span class="line"> <span class="comment">// TODO do login</span></span><br><span class="line"> <span class="comment">// TODO response successful data</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="处理返回"><a class="markdownIt-Anchor" href="#处理返回"></a> 处理返回</h2><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> Response <span class="keyword">struct</span> {</span><br><span class="line"> Status <span class="keyword">int</span> <span class="string">`json:"status"`</span></span><br><span class="line"> Data <span class="keyword">interface</span>{} <span class="string">`json:"data"`</span></span><br><span class="line"> Message <span class="keyword">interface</span>{} <span class="string">`json:"message"`</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> UserinfoResponse <span class="keyword">struct</span> {</span><br><span class="line"> Token <span class="keyword">string</span> <span class="string">`json:"token"`</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">response</span><span class="params">(w, resp <span class="keyword">interface</span>{})</span></span> {</span><br><span class="line"> encoder := json.NewEncoder(w)</span><br><span class="line"> err := encoder.Encode(resp)</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="built_in">panic</span>(err)</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *Service)</span> <span class="title">signin</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line"> param := SigninParam{}</span><br><span class="line"> <span class="keyword">if</span> err := bindAndValid(req, &param); err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="comment">// response param-check error message</span></span><br><span class="line"> <span class="comment">//</span></span><br><span class="line"> <span class="comment">// To custom with the error message,</span></span><br><span class="line"> <span class="comment">// use https://github.com/go-playground/universal-translator</span></span><br><span class="line"> <span class="comment">// see example at: https://github.com/go-playground/validator/blob/master/_examples/translations/main.go#L105</span></span><br><span class="line"> response(w, Response{StatusInvalidParam, <span class="literal">nil</span>, MessageInvalidParam}</span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line"> }</span><br><span class="line"> <span class="comment">// ... do login</span></span><br><span class="line"> <span class="comment">// response successful data</span></span><br><span class="line"> response(w, Response{StatusSuccess, UserinfoResponse{token}, <span class="literal">nil</span>})</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="调用其他服务"><a class="markdownIt-Anchor" href="#调用其他服务"></a> 调用其他服务</h2><p>这里以使用mysql服务为例，使用<a href="https://github.com/go-sql-driver/mysql" target="_blank" rel="noopener"><code>github.com/go-sql-driver/mysql</code>包</a>。</p><p>首先，想要在handle中使用mysql服务，就需要能够获取mysql的连接，通常我们会将服务连接池初始化放在服务初始化时进行：</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> Service <span class="keyword">struct</span> {</span><br><span class="line"> mux *http.ServeMux</span><br><span class="line"> <span class="comment">// add a db property</span></span><br><span class="line"> db *sql.DB</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="日志输出"><a class="markdownIt-Anchor" href="#日志输出"></a> 日志输出</h2>]]></content>
<categories>
<category> 实践 </category>
<category> 开始用go </category>
</categories>
<tags>
<tag> golang </tag>
</tags>
</entry>
<entry>
<title>【互联网产品的诞生和演化】快速迭代和持续集成</title>
<link href="/posts/practice/0ws/0ws-cicd/"/>
<url>/posts/practice/0ws/0ws-cicd/</url>
<content type="html"><![CDATA[<p>在互联网产品开发过程中，效率和稳定性始终是核心问题。</p><p>开发效率决定了产品的迭代速度。当一个产品需要与市场上同类产品竞争时，迭代速度就会显得尤为重要。<br>别人新上了一个很受欢迎的功能而你没有在几天之内跟上，那么你的用户将会大规模流失。</p><p>同时，如果你的服务不稳定，就会给用户一种不信任的感觉，进而动摇用户留下来的信心，也会造成用户流失。</p><p>因此，在保证系统稳定性的同时尽可能加快项目开发速度就是我们所追求的极致。</p><p>通常来讲需要考虑以下一些过程。</p><h2 id="单元测试"><a class="markdownIt-Anchor" href="#单元测试"></a> 单元测试</h2><p>单元测试是指针对代码的测试，以确保每一行代码都能按照预想的方式执行。</p><p>任何一个系统功能，都应该有完整的单元测试。单元测试可以使开发者减少大量维护已有系统的精力，同时保证发布新功能时的稳定性。</p><p>这里我们要求单元测试的覆盖率尽可能达到100%。</p><p>这里推荐使用<a href>go语言http服务单元测试组件</a>。</p><h2 id="功能测试"><a class="markdownIt-Anchor" href="#功能测试"></a> 功能测试</h2><p>功能测试是面向功能的测试，在互联网服务中，通常以接口或者接口组为单位进行。</p><p>所有功能测试的测试用例应当紧跟产品需求，作为系统发布的最终屏障，来保证系统稳定性。</p><h2 id="自动发布"><a class="markdownIt-Anchor" href="#自动发布"></a> 自动发布</h2><p>发布是系统交互的一个过程。当一个互联网产品开发完成后，通常都需要集成发布，包括但不限于：</p><ul><li>编译</li><li>打包</li><li>更新版本</li><li>更新文档</li><li>更新日志</li></ul><h2 id="自动部署"><a class="markdownIt-Anchor" href="#自动部署"></a> 自动部署</h2><p>让一个互联网产品能够被用户所使用，需要将指定的发布版本部署到服务器上，这个过程包括但不限于：</p><ul><li>运行指定发布版本</li><li>新建或修改数据表</li><li>新增或修改数据</li><li>新增或修改配置等</li></ul>]]></content>
<categories>
<category> 实践 </category>
<category> 互联网产品的诞生和演化 </category>
</categories>
<tags>
<tag> Web </tag>
</tags>
</entry>
<entry>
<title>【互联网产品的诞生和演化】统计数据收集展示和系统监控</title>
<link href="/posts/practice/0ws/0ws-stats/"/>
<url>/posts/practice/0ws/0ws-stats/</url>
<content type="html"><![CDATA[<p>互联网产品在线上之前，就要考虑统计数据和监控。</p><p>初期，为了节省成本，数据统计可以直接使用第三方平台提供的服务，如：百度统计、谷歌统计等。</p><p>这些第三方平台提供了丰富多彩的统计服务，唯一缺陷就是数据在第三方，我们自己无法随心所欲使用数据，只能受限于第三方可用功能。</p><p>系统监控可以考虑直接监控服务器错误日志。</p><p>当第三方数据服务已经无法满足我们的要求时，就需要考虑自建数据平台了。</p><p>一个典型的数据平台架构如下：</p><pre class="mermaid" style="text-align: center;"> graph LR source[数据源]collection[数据收集]storage[数据存储]analysis[数据分析]showing[数据展示]source --> collectioncollection --> storagestorage --> analysisanalysis --> storagestorage --> showing </pre><p>这里边，数据存储是核心。</p><h2 id="数据存储"><a class="markdownIt-Anchor" href="#数据存储"></a> 数据存储</h2><p>影响数据存储选型的因素有很多，需要重点考虑的核心因素有三：</p><ul><li>数据规模</li><li>数据结构</li><li>数据使用方式</li></ul><p>对于不同的选型，上述三个因素的取舍各不相同：</p><ul><li><p>MySQL</p><p>MySQL是互联网时代最常用的数据库之一，适用于100w数量级以下的结构化数据存储，使用SQL操作数据，通常用于存储业务数据和展示数据。<br>在数据平台中，如果总数据量不大，并且增长不快，可以直接使用MySQL作为数据存储，这种存储的使用成本应该是最低的。</p></li><li><p>HDFS</p><p>HDFS是Hadoop架构下的分布式文件存储系统，适用于100w数据级以上的数据存储，使用MapReduce查询数据或通过上层中间件（如：HBase，HIVE，Spark等）操作数据，通常应用于离线数据计算和查询。<br>在数据平台中，使用HDFS至少需要3个节点以上的集群进行部署，同时还要部署和维护所需中间件以及数据访问权限，运维复杂度极高，非专业团队不建议使用。</p></li><li><p>Elasticsearch</p><p>Elasticsearch是ElasticStack中的基于Luence实现的搜索引擎，得益于其高效的全文搜索功能和数据统计功能，也可以将其用作数据平台的一种存储。<br>结合使用Kibana，可以将其作为系统监控的核心系统。<br>Kibana通过封装Elasticsearch的api，实现了数据查询、聚集以及可视化展示等功能，并且支持按条件触发事件和发送报告。<br>由于其基于Luence实现，因此不适合作为复杂的数据分析工具使用，强行使用会影响系统稳定性。</p></li><li><p>TiDB</p><p>TiDB是PingCAP开发的一款支持MySQL协议，并且可以水平扩展的分布式关系数据库。<br>由于其对MySQL协议的支持，大大降低了使用成本和维护成本。<br>其在大数据量下的表现有待考量，据笔者经验，在亿级数据量下复杂分组查询可以实现秒级返回。</p></li><li><p>Kafka</p><p>Kafka可以被看作实时一种数据存储，由于流计算需求的存在，使得Kafka成为一个不可替代的存储选型。</p></li></ul><p>事实上，在实际应用中，根据不同需求，可以将不同存储组合使用。例如：</p><ul><li>将需要展示的数据，计算后存储到MySQL中</li><li>将冷数据归档到HDFS中</li><li>在Elasticsearch中仅保留一定规模的热数据用于问题跟踪和系统监控</li><li>TiDB仅用于数据分析，不用于业务服务和数据展示</li></ul><h2 id="数据源"><a class="markdownIt-Anchor" href="#数据源"></a> 数据源</h2><p>从技术角度来讲，统计和监控都依赖于日志数据的收集和汇总，所需的日志通常有：</p><ul><li>前端和客户端行为打点日志</li><li>前端和客户端错误上报日志</li><li>后端接口请求日志</li><li>后端异常错误日志</li><li>探针日志</li></ul><p>以上日志，都可以作为数据源进行收集，并进一步处理。</p><h2 id="数据收集"><a class="markdownIt-Anchor" href="#数据收集"></a> 数据收集</h2><p>由于我们的数据源都是日志，因此可以使用filebeat部署到日志服务器进行收集。</p><p>这里建议filebeat的消费端使用Kafka以使服务器的filebeat不发生阻塞也不需要回滚。</p><div class="note warning"> <p>注意：logstash非常耗费资源，如果资源并出充足，建议自行脚本实现从Kafka消费到数据存储的流程。</p> </div><p>filebeat的安装命令：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.1-amd64.deb</span><br><span class="line">sudo dpkg -i filebeat-6.3.1-amd64.deb</span><br><span class="line">sudo systemctl start filebeat</span><br><span class="line">sudo systemctl <span class="built_in">enable</span> filebeat</span><br></pre></td></tr></table></figure><p>filebeat的配置示例如下：</p><figure class="highlight yaml"><figcaption><span>/etc/filebeat/filebeat.yml</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="string">filebeat.prospectors:</span></span><br><span class="line"><span class="comment"># 根据情况自行配置需要收集哪些log</span></span><br><span class="line"><span class="attr">- type:</span> <span class="string">log</span></span><br><span class="line"><span class="attr"> paths:</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string"><log_path></span></span><br><span class="line"></span><br><span class="line"><span class="string">output.kafka:</span></span><br><span class="line"><span class="attr"> enable:</span> <span class="literal">true</span></span><br><span class="line"><span class="attr"> hosts:</span> <span class="string">['<kafka_ip>']</span></span><br><span class="line"></span><br><span class="line"><span class="attr"> topic:</span> <span class="string">'<string>'</span></span><br><span class="line"><span class="attr"> compression:</span> <span class="string">gzip</span></span><br><span class="line"><span class="attr"> max_message_bytes:</span> <span class="number">1000000</span></span><br><span class="line"></span><br><span class="line"><span class="string">queue.mem:</span></span><br><span class="line"><span class="attr"> events:</span> <span class="number">512</span></span><br><span class="line"> <span class="string">flush.min_events:</span> <span class="number">256</span></span><br><span class="line"> <span class="string">flush.timeout:</span> <span class="number">5</span><span class="string">s</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 监控可不配置</span></span><br><span class="line"><span class="string">xpack.monitoring:</span></span><br><span class="line"><span class="attr"> enabled:</span> <span class="literal">true</span></span><br><span class="line"><span class="attr"> elasticsearch:</span></span><br><span class="line"><span class="attr"> hosts:</span> <span class="string">["<es_ip>"]</span></span><br><span class="line"><span class="attr"> username:</span> <span class="string"><string></span></span><br><span class="line"><span class="attr"> password:</span> <span class="string">"<string>"</span></span><br></pre></td></tr></table></figure><h2 id="数据分析"><a class="markdownIt-Anchor" href="#数据分析"></a> 数据分析</h2><p>数据分析是对现有数据的再加工过程，通常涉及：变形，关联，统计等。目前最适合进行数据分析的语言应该是Python，上文中的数据存储，都可以原生支持Python。</p><h2 id="数据展示"><a class="markdownIt-Anchor" href="#数据展示"></a> 数据展示</h2><p>数据展示的基本形式是图表，图可以将数据特征更直观的展示出来，<a href="https://github.com/antvis/G2Plot" target="_blank" rel="noopener">G2Plot</a>提供了图表的完整封装，通过简单的前端项目构建和后端接口查询，即可做到优雅的数据可视化展示。</p><p>注意，不要在数据展示期间进行数据计算。</p><p>当然，想要展示实时数据，可以考虑尝试使用TiDB。</p>]]></content>
<categories>
<category> 实践 </category>
<category> 互联网产品的诞生和演化 </category>
</categories>
<tags>
<tag> Web </tag>
</tags>
</entry>
<entry>
<title>【互联网产品的诞生和演化】独立应用的基本架构</title>
<link href="/posts/practice/0ws/0ws-base/"/>
<url>/posts/practice/0ws/0ws-base/</url>
<content type="html"><![CDATA[<div class="note info"> <p>一个新的互联网产品研发并不简单，当你不确定产品形态是否可行时，请先想办法验证，不要盲目进入开发。<br>如何进行低成本验证，请参考上一篇文章：<a href="/posts/practice/0ws/0ws-simple">产品的最简模型</a>。</p> </div><p>得益于移动互联网时代的飞速发展，开发一个互联网产品已经成为人尽可谈的话题。</p><p>然而在这样的话题的讨论中，通常得到的结果都是：我们的点子非常好，就是缺一个开发。</p><p>不可否认，任何一个想法，如果能把它做成一个App、一个网站或者一个小程序，多半情况下它都是可用的，想想就很兴奋。</p><p>但是想要真正将这个开发过程付诸实施，还有一个完整的开发团队的距离。</p><p>简单来说，一个完整的开发团队，需要以下配置具有以下特殊技能的人员：<br>设计、前端开发、客户端开发、后端开发、测试、运维。<br>{在后面<a href="#%E6%8A%80%E8%83%BD%E9%9C%80%E6%B1%82">技能需求</a>章节有对这些技能应用在何处的详细说明。）</p><p>当然，不排除有些人是具有多种技能的综合型人才。</p><p>如果运气好，上面所有人员都能胜任，那么至少可以开发出一个初始版本的系统了，接下来考虑线上部署，需要采购并长期维护以下资源：<br>域名（备案）、ssl证书、云主机实例、云数据库实例、cdn服务。<br>{在后面<a href="#%E8%B5%84%E6%BA%90%E9%9C%80%E6%B1%82">资源需求</a>章节有对这些资源应用在何处的详细说明。}</p><p>至此，开发一个崭新的互联网服务的必备条件已经完全具备了，最后我们需要考虑的就是项目周期。</p><div class="note info"> <p>项目开发周期=每个成员完成任务所需要的时间+成员间沟通所需时间-可并行时间</p> </div><p>很明显，根据上面公式来看，想要缩短开发时间，只有两条路可以走：</p><ul><li>选择能力更高的人以降低单人工作时间和沟通时间</li><li>通过合理规划增加可并行时间（<a href="#%E5%B7%A5%E4%BD%9C%E6%B5%81">工作流</a>章节给出了一个通常情况下的最优规划方案）</li></ul><h2 id="基本技术架构"><a class="markdownIt-Anchor" href="#基本技术架构"></a> 基本技术架构</h2><p>在上文中，我们已经直接探讨了开发一个互联网产品所需要考虑的全部内容。然而为什么是这样，这就需要从基本架构先说起。</p><p>参考下图：</p><pre class="mermaid" style="text-align: center;"> graph LR fe[网站或微信小程序]app[移动应用App]api[后端接口]services[后端服务]db[数据库服务]3rd-api[第三方服务接口]admin[运营后台]admin-api[后台接口]fe --> apiapp --> apiapi --> servicesservices --> db3rd-api --> servicesservices --> 3rd-apiadmin --> admin-apiadmin-api --> services </pre><p>对于一个互联网产品，客户端大多会考虑实现两种形式：<code>网站</code>或<code>移动应用App</code>。其中，</p><ul><li>网站 ---- 开发成本低，迭代周期快</li><li>移动应用App ---- 产品形象固定，用户粘性高</li></ul><div class="note info"> <p>随着<code>微信小程序</code>生态的日益完善，这类客户端形态也逐渐被大众所考虑。<br>微信小程序的优势在于，它兼具了网站和移动应用App两种类型客户端的优势，并且天生可以依托微信这个成熟且巨大的用户生态圈。</p> </div><p>在客户端之后，还需要服务端来实现产品逻辑。服务端的内容，大致可以采用如下方式划分：</p><ul><li>后端接口 ---- 作为用户与服务交流的通道，让用户在客户端上的操作可以得到对应的响应。</li><li>后端服务 ---- 实现复杂的产品逻辑，如：注册登录、购买支付、分享邀请等。</li><li>数据库服务 ---- 用于记录数据，包括但不限于：用户信息、用户购买信息、用户操作记录等。</li><li>第三方服务接口 ---- 接入其他已经实现了的互联网服务，如：微信登录、微信或支付宝支付等。</li><li>运营后台 ---- 运营人员维护产品数据的平台，通常是一个网站，可运营的内容可能包括产品信息价格、用户账号数据等。</li><li>后台接口 ---- 作为运营人员与服务交流的通道，使其能够查询管理大部分产品相关数据。</li></ul><p>整体来讲，任何一个互联网产品，在设计开发之初，都应该考虑以上所有内容。<br>一些情况下，部分内容可能或有所取舍，比如：</p><ul><li>不需要开发移动应用App，只要一个网站就够了</li><li>不需要后端接口和后端服务，因为不需要记录任何用户数据，只需要展示一些内容</li><li>不需要运营后台，我可以直接通过数据库查询和修改需要的数据</li></ul><h2 id="技能需求"><a class="markdownIt-Anchor" href="#技能需求"></a> 技能需求</h2><p>如果按照上面所讲的基本技术架构进行开发，那么我们需要以下相关技能：</p><ul><li>ui&ue ---- 决定产品长相和操作方式，通常需要艺术设计专业知识</li><li>fe<ul><li>h5 ---- 开发pc端和移动端网页产品，需要掌握html5+javascript+css基础能力和主流框架使用经验</li><li>wx ---- 开发微信小程序或服务号相关产品，需要熟悉微信开放接口和微信小程序开发标准</li><li>admin ---- 开发运营后台，熟悉后台相关框架以及一些数据可视化技术和组件</li></ul></li><li>app<ul><li>android ---- 开发android应用，需要相关开发经验</li><li>ios ---- 开发ios应用，需要相关开发经验</li></ul></li><li>server<ul><li>buzz ---- 开发http接口服务，需要相关开发经验</li><li>service ---- 开发复杂应用服务和基础服务，需要相关开发经验</li></ul></li><li>other<ul><li>test ---- 对产品进行系统全面的测试，需要相关经验</li><li>op ----- 发布和维护所开发的服务，应对各种突发情况（如：软硬件故障、黑客攻击，版本更新等）</li></ul></li></ul><h2 id="资源需求"><a class="markdownIt-Anchor" href="#资源需求"></a> 资源需求</h2><p>对于任何一个互联网产品，想要让用户能够使用，都需要购买以下资源：</p><ul><li>域名，作为用户访问所使用的地址，如：<a href="http://www.jd.com" target="_blank" rel="noopener">www.jd.com</a>。需要在域名运营商或者代理商处购买。<br>注意在国内购买或使用的域名通常还需要备案。</li><li>https证书，用于认证域名的安全性。需要在有资质的证书管理机构或代理商处购买。</li><li>云实例，用于部署和运行服务。需要在云服务商购买。</li><li>数据库实例，用于提供数据库相关服务。需要在云服务商购买。</li><li>cdn，用于提供静态资源存储和访问服务。需要在云服务商购买。</li></ul><h2 id="工作流"><a class="markdownIt-Anchor" href="#工作流"></a> 工作流</h2><p>一个合理的工作流可以让产品开发周期缩到最短。</p><pre class="mermaid" style="text-align: center;"> graph LR prd[产品需求]uiue[视觉交互设计]server[后端设计开发]admin[后台设计开发]fe[前端开发]app[移动端开发]test[测试]release[发布]prd --> uiueprd --> serverprd --> adminuiue --> feuiue --> appserver --> feserver --> appserver --> adminfe --> testapp --> testadmin --> testtest --> release </pre><p>这里同样按照专业领域划分，从一个确定的产品需求开始。</p><ul><li>一旦产品需求确定了，就可以开始视觉交互设计和后端设计了。<br>这里边后端设计要考虑接口、逻辑和数据三个层面，对于设计人员的能力要求较高。</li><li>前端开发和移动端开发需要在两个设计都完成后开始。</li><li>后台设计开发需要在后端设计完成后开始。</li><li>当所有开发完成后进入测试环节。</li><li>最终测试通过后进行产品发布。</li></ul><p>经过多年实践和总结，这个工作流可以最大化并行时间，也就是最优化项目开发周期。</p>]]></content>
<categories>
<category> 实践 </category>
<category> 互联网产品的诞生和演化 </category>
</categories>
<tags>
<tag> Web </tag>
</tags>
</entry>
<entry>
<title>【互联网产品的诞生和演化】产品的最简模型</title>
<link href="/posts/practice/0ws/0ws-simple/"/>
<url>/posts/practice/0ws/0ws-simple/</url>
<content type="html"><![CDATA[<p>互联网产品是一种：“通过互联网技术获取并留住用户，进而从用户身上获取利益的商业行为“。</p><p>AARRR模型可以用于研究这类产品的用户生命周期。</p><pre class="mermaid" style="text-align: center;"> graph LR acquisition[触达]activation[获客]retention[留存]revenue[转化]refer[传播]acquisition --> activationactivation --> retentionactivation --> referretention --> revenuerevenue --> referrefer --> activation </pre><p>在这个生命周期中，</p><ol><li><code>触达</code>就是将产品入口暴露给用户，让用户知道这个产品，并且能够通过何种途径使用产品。</li><li><code>获客</code>就是让用户真正的使用产品。</li><li><code>留存</code>通常是看一周或者一个月以后，还在持续使用产品的用户数量。</li><li><code>转化</code>即为从使用产品的用户身上获取利益。</li><li><code>传播</code>表示正在使用产品的用户将产品介绍给新人，并让其开始使用产品。</li></ol><p>事实上，在上述生命周期中，产品的核心诉求是<code>转化</code>，只有转化才能获取利益，才能让产品存活下去。想要获取更多利益，要么增加<code>获客</code>，要么提升<code>留存</code>和<code>转化</code>的效率：<code>触达</code>和<code>传播</code>都是为了增加<code>获客</code>，应用适合的推广手段是关键：而提升<code>留存</code>和<code>转化</code>的效率则需要更具吸引力的产品形态和专业的运营策略。</p><p>那么，如何依托第三方平台实现上述模型的验证？这里我们以微信群为例说明。</p><p>基于微信群实现互联网产品，通常需要以下步骤：</p><ol><li>建群，群主作为主运营人员，负责发布和维护群规</li><li>通过分享二维码<code>触达</code>潜在用户</li><li>所有进群用户都是<code>获客</code></li><li>每日统计群成员数量计算<code>留存</code></li><li>在群内发布产品内容进行<code>转化</code>（例如：加好友提供服务）</li><li>群成员拉其他人进群的数量即为<code>传播</code></li></ol><p>当上述模型验证成功后，即可开始组建开发团队进行产品开发。</p><p>具体实践方案请参考下一篇文章：<a href="/posts/practice/0ws/0ws-base/">独立应用的基本架构</a>。</p>]]></content>
<categories>
<category> 实践 </category>
<category> 互联网产品的诞生和演化 </category>
</categories>
<tags>
<tag> Web </tag>
</tags>
</entry>
<entry>
<title>【互联网产品的诞生和演化】目录</title>
<link href="/posts/practice/0ws/0ws-index/"/>
<url>/posts/practice/0ws/0ws-index/</url>
<content type="html"><![CDATA[<ol><li><a href="/posts/practice/0ws/0ws-simple/">产品的最简模型</a><br>采用何种技术架构能够做到高效快速的验证服务可行性？<br>本文核心思想是：尽可能的依托第三方平台实现核心产品逻辑。</li><li><a href="/posts/practice/0ws/0ws-base/">独立应用的基本架构</a><br>自研产品都需要考虑哪些技术投入？<br>一个完整的产品技术架构，不只是客户端和服务端，需要考虑的细节还很多，这些都将在本文中一一阐述。</li><li><a href="/posts/practice/0ws/0ws-stats/">统计数据收集展示和系统监控</a><br>如何快速构建一个独立的统计数据收集和系统监控平台？<br>本文主要讲述的是从第三方统计平台到自研数据平台演化过程和实现方案。</li><li><a href="/posts/practice/0ws/0ws-cicd/">快速迭代和持续集成</a><br>在保证稳定性的前提下如何提升迭代效率？<br>本文的观点：完整的单元测试和自动化运维平台是快速迭代和持续集成的基础。</li><li><a href>面向高可用的逐步升级</a><br>不会挂的系统才是好系统，线上系统远比你想象的脆弱。<br>那么如何面对：突发流量，系统故障，黑客攻击，程序员跑路等各种奇葩问题，将在本文中详细介绍。</li><li><a href>接入第三方应用账号认证</a><br>第三方账号登陆是引流必须跨过的一道门槛，技术上也并不复杂。<br>本文主要介绍的就是接入微信，支付宝，google，facebook，twitter等的套路。</li><li><a href>接入支付系统</a><br>2c业务想赚钱，必须接入支付。<br>那么接入微信，支付宝，银联，ApplePay等，都需要准备什么，需要注意避开哪些坑，本文将会提及。</li><li><a href>多人合作模式</a><br>在绝大多数情况下，多人合作的结果一定是<code>1+1<2</code>，但是你又不得不考虑。<br>并不是因为产品需求繁重，而是为了应付不可控的人员流动。</li></ol>]]></content>
<categories>
<category> 实践 </category>
<category> 互联网产品的诞生和演化 </category>
</categories>
<tags>
<tag> Web </tag>
</tags>
</entry>
<entry>
<title>使用google的colab进行神经网络相关开发</title>
<link href="/posts/tool/corlab/colab-github/"/>
<url>/posts/tool/corlab/colab-github/</url>
<content type="html"><![CDATA[<p><a href="https://colab.research.google.com/" target="_blank" rel="noopener">Colaboratory(简称colab)</a>是Google提供的一个免费的Jupyter笔记本环境，支持云端运行。<br>为了支持云端运行，Google还提供了强大的计算资源，如GPU、TPU和云端存储空间等，这些都可以通过浏览器免费使用。</p><p>如需全面了解colab的所提供的功能，请阅读<a href="https://colab.research.google.com/notebooks/welcome.ipynb" target="_blank" rel="noopener">官方文档</a>。</p><p>本文的主要目标是使你快速的能够使用colab进行神经网络相关开发。</p><h2 id="准备"><a class="markdownIt-Anchor" href="#准备"></a> 准备</h2><p>使用colab服务，首先需要科学上网。相关文章，请参考<a href="https://blog.mapleque.com/tags/vpn/">标签：vpn</a>中的内容。</p><h2 id="创建修改文件"><a class="markdownIt-Anchor" href="#创建修改文件"></a> 创建修改文件</h2><p>直接访问地址： <a href="https://colab.research.google.com/" target="_blank" rel="noopener">https://colab.research.google.com/</a> ，就可以开始代码的编辑。</p><p>打开页面时，会弹窗提示最近打开的文件，也可以选择创建文件。这个弹窗后面也可以通过顶部菜单：<code>文件->打开笔记本</code>调出。</p><p>在这个弹窗中，我们还可以看到一些其他打开文件的方式，这里重点说明GitHub选项。</p><p>点击选择GitHub选项，会出现<code>输入GitHub网址，或者按组织或用户搜索</code>的输入框。这里可以输入自己的github用户名，如：<code>mapleque</code>，点击搜索后，就会在下面代码库部分出现该用户所有的公开项目列表了（中间可能需要进行GitHub账号授权）。选择一个项目和分支，如果在项目中含有colab可以打开的文件，就会被列在下面的路径列表中，用户可以选择任意一个打开编辑。</p><p>如果是私有代码库，可以选择勾选右上角的选项，然后进行额外的授权之后，重新搜索对应的空间或项目，这时所有有权限的私有项目也会出现在列表中了。</p><p>注意：这里还有一个快捷打开GitHub文件的技巧，即输入网址：<a href="https://colab.research.google.com/github/mapleque/nnlearning/blob/master/notebooks/TensorflowMNIST.ipynb" target="_blank" rel="noopener">https://colab.research.google.com/github/mapleque/nnlearning/blob/master/notebooks/TensorflowMNIST.ipynb</a> 。</p><p>观察上面的网址，可以发现<code>/github/</code>后面的路径，实际上就是文件在GitHub上预览的路径。</p><h2 id="编辑运行"><a class="markdownIt-Anchor" href="#编辑运行"></a> 编辑运行</h2><p>colab集成了丰富的nn相关资源，如tensorflow, pytorch等。所以只需要新建一个python3的记事本，就可以开始执行程序了。</p><p>注意，新建代码文件后，需要配置代码执行环境：选择顶部菜单中的<code>代码执行程序->更改运行时类型</code>选项，在弹框中选择相关设置，如：硬件加速选择GPU，最后保存。</p><p>点击顶部菜单左上角<code>+代码</code>选项，就会在文件编辑区域出现一行带有播放图标的空白输入框，在输入框中输入代码，点击播放按钮，就可以提交执行。例如这里提交编辑代码：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> __future__ <span class="keyword">import</span> absolute_import, division, print_function, unicode_literals</span><br><span class="line"><span class="keyword">import</span> tensorflow <span class="keyword">as</span> tf</span><br><span class="line"><span class="keyword">from</span> tensorflow <span class="keyword">import</span> keras</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="keyword">import</span> matplotlib.pyplot <span class="keyword">as</span> plt</span><br></pre></td></tr></table></figure><p>点击执行，即显示执行成功。可见tensorflow，numpy和matplotlib等包都已经预置安装好了。</p><p>后面正常编辑执行MNIST例子相关代码，就可以看到自动下载数据和训练的日志输出。完整代码请参考<a href="https://colab.research.google.com/github/mapleque/nnlearning/blob/master/notebooks/TensorflowMNIST.ipynb" target="_blank" rel="noopener">mapleque/nnlearning/TensorflowMNIST.ipynb</a>。</p><p>colab执行结果页面，可以将图片直接展示出来。</p><h2 id="提交到github"><a class="markdownIt-Anchor" href="#提交到github"></a> 提交到GitHub</h2><p>colab支持直接将当前代码文件提交到GitHub。</p><p>点击顶部菜单：<code>文件->在GitHub中保存一份副本</code>选项，授权后在弹窗中选择对应的代码库、分支和路径，就可以提交了。</p><p>这里如果勾选了<code>包含指向Colaboratory的链接</code>选项，将会在GitHub的文件预览页面看到一个直接在colab中打开文件并编辑的按钮。</p>]]></content>
<categories>
<category> 工具 </category>
<category> corlab </category>
</categories>
<tags>
<tag> nn </tag>
<tag> pytorch </tag>
<tag> colab </tag>
</tags>
</entry>
<entry>
<title>部署并使用shadowsocks</title>
<link href="/posts/tool/shadowsocks/shadowsocks/"/>
<url>/posts/tool/shadowsocks/shadowsocks/</url>
<content type="html"><![CDATA[<p><a href="https://shadowsocks.org" target="_blank" rel="noopener">Shadowsocks</a>(通常简称为ss)是一个基于socks5协议实现的网络管理服务。<br>通常我们用它来做网络流量转发，因此这里简单介绍一下如何方便安全的部署并使用它。</p><p>本文将基于ss的go版本部署实现，可能需要一些计算机开发基础知识。如不具备，请自行搜索其他教程。</p><h2 id="准备vps"><a class="markdownIt-Anchor" href="#准备vps"></a> 准备vps</h2><p>部署和使用ss，需要先有一个用于转发流量的vps。</p><div class="note info"> <p>笔者目前使用的vultr服务，$3.50/mo, 500GB。<br>通过这个推广链接注册并购买，可以获得一定的优惠。<br><a href="https://www.vultr.com/?ref=7147137" target="_blank" rel="noopener">https://www.vultr.com/?ref=7147137</a></p> </div><p>下面将以CentOS7x64为例进行部署。</p><h2 id="申请ip"><a class="markdownIt-Anchor" href="#申请ip"></a> 申请ip</h2><p>想要进行流量转发，必须要有一个能够访问的ip。由于一些不明原因，并不是所有ip都能访问，所以在一定在申请ip的时候注意选择。</p><p>如果是使用vultr，可以通过多购置几台vps进行尝试，遇到可用ip后再将其他的退掉。</p><h2 id="编译安装"><a class="markdownIt-Anchor" href="#编译安装"></a> 编译安装</h2><p>shadowsocks的代码，全部托管在github上，这里建议使用golang实现版本：<a href="https://github.com/shadowsocks/go-shadowsocks2" target="_blank" rel="noopener">https://github.com/shadowsocks/go-shadowsocks2</a> 。</p><p>该版本可以提供跨平台(windows, linux, MacOS)支持。</p><p>登录所购买的vps，然后安装开发环境：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">yum install -y git golang</span><br></pre></td></tr></table></figure><p>下载代码并编译安装（这里之所以自己下载代码编译，是因为go-shadowsocks2的release版本落后master很多，居然还不支持plugin）：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">go get -u -v github.com/shadowsocks/go-shadowsocks2</span><br></pre></td></tr></table></figure><h2 id="部署服务"><a class="markdownIt-Anchor" href="#部署服务"></a> 部署服务</h2><p>通过上面的步骤下载完成后，应该就已经自动编译好了一个当前系统的可执行文件了。</p><p>执行下面的命令启动服务：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="variable">$GOPATH</span>/bin/shadowsocks2 -s <span class="string">'ss://AES-256-GCM:[email protected]:8488'</span> -verbose</span><br></pre></td></tr></table></figure><p>对于CentOS7，还需要设置防火墙。这里通过创建一个防火墙服务来实现：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">vim /usr/lib/firewalld/services/socks.xml</span><br></pre></td></tr></table></figure><figure class="highlight xml"><figcaption><span>/usr/lib/firewalld/services/socks.xml</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta"><?xml version="1.0" encoding="utf-8"?></span></span><br><span class="line"><span class="tag"><<span class="name">service</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">short</span>></span>Socks<span class="tag"></<span class="name">short</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>Socks Port<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">port</span> <span class="attr">protocol</span>=<span class="string">"tcp"</span> <span class="attr">port</span>=<span class="string">"8488"</span>/></span></span><br><span class="line"><span class="tag"></<span class="name">service</span>></span></span><br></pre></td></tr></table></figure><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">firewall-cmd --permanent --add-service=socks</span><br><span class="line">firewall-cmd --reload</span><br></pre></td></tr></table></figure><h2 id="编译客户端"><a class="markdownIt-Anchor" href="#编译客户端"></a> 编译客户端</h2><p>该版本shadowsocks提供了多平台的客户端，可以直接通过原代码编译生成：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">make</span><br></pre></td></tr></table></figure><p>然后到bin路径下找对应的客户端可执行文件下载。</p><h2 id="启动客户端"><a class="markdownIt-Anchor" href="#启动客户端"></a> 启动客户端</h2><p>客户端支持多种代理模式，这里以在MacOS上执行socks模式为例：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">shadowsocks2-macos -c <span class="string">'ss://AES-256-GCM:your_password@ip:8488'</span> -verbose -socks :1080 -u</span><br></pre></td></tr></table></figure><p>注意将上面的ip文本替换为vps的ip。</p><h2 id="使用服务"><a class="markdownIt-Anchor" href="#使用服务"></a> 使用服务</h2><p>在MacOS上，可以通过内置的网络组件直接连接socks代理：</p><p>系统偏好设置->网络->高级->代理->socks</p><p>在代理服务器中填上localhost:1080，然后点击：</p><p>好->应用</p><p>即完成了机器全部网络流量的转发。</p><p>此时可以尝试访问https://google.com 以检验是否成功。</p><h2 id="使用其他客户端"><a class="markdownIt-Anchor" href="#使用其他客户端"></a> 使用其他客户端</h2><p>经过了上面的步骤，你应该已经能在MacOS上访问shadowsocks官网了。<br>此时你会发现，官网上推荐了多种可以在不同平台使用的客户端，这里将重点介绍以下outline。</p><p>IOS上的outline客户端目前可以在AppStore上免费下载。</p><p>启动outline客户端，你会发现要求输入的是secretKey。<br>按照outline官方的要求，你需要下载并部署outline管理服务来生成并管理secretKey。<br>但这个管理工具不仅增加了一些统计逻辑实现，还收集了服务器信息，因此对于个人用户不推荐使用（尽管客户端也可能收集了一些数据）。<br>这里可以通过下面网址自行生成secretKey并链接：<br><a href="https://shadowsocks.org/en/config/quick-guide.html" target="_blank" rel="noopener">https://shadowsocks.org/en/config/quick-guide.html</a></p>]]></content>
<categories>
<category> 工具 </category>
<category> shadowsocks </category>
</categories>
<tags>
<tag> vpn </tag>
</tags>
</entry>
<entry>
<title>【vim】使用vim作为开发js的IDE</title>
<link href="/posts/tool/vim/vim-js/"/>
<url>/posts/tool/vim/vim-js/</url>
<content type="html"><![CDATA[<div class="note info"> <p>本文中将直接使用基本设置已经使用的配置，如需了解，请阅读<a href="/posts/tool/vim/vim-start/">【vim】基本设置</a>。</p> </div><h1 id="基本设置"><a class="markdownIt-Anchor" href="#基本设置"></a> 基本设置</h1><p>创建<code>~/.vim/ftplugin/javascript.vim</code>文件来配置用于js文件的设置。</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">" 缩进2字符</span></span><br><span class="line"><span class="keyword">set</span> tabstop=<span class="number">2</span></span><br><span class="line"><span class="keyword">set</span> softtabstop=<span class="number">2</span></span><br><span class="line"><span class="keyword">set</span> <span class="built_in">shiftwidth</span>=<span class="number">2</span></span><br><span class="line"><span class="keyword">set</span> expandtab</span><br><span class="line"></span><br><span class="line"><span class="comment">" 设置代码每行最大宽度标尺</span></span><br><span class="line"><span class="keyword">set</span> tw=<span class="number">118</span></span><br></pre></td></tr></table></figure><h1 id="支持jsx语法高亮"><a class="markdownIt-Anchor" href="#支持jsx语法高亮"></a> 支持jsx语法高亮</h1><p>让vim支持jsx语法高亮，使用<a href="https://github.com/mxw/vim-jsx" target="_blank" rel="noopener">vim-jsx</a>插件。</p><p>通过Vundle安装，在.vimrc中添加：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">Plugin <span class="string">'pangloss/vim-javascript'</span></span><br><span class="line">Plugin <span class="string">'mxw/vim-jsx'</span></span><br></pre></td></tr></table></figure><p>在vim中执行命令：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">:PluginInstall</span><br></pre></td></tr></table></figure><h1 id="eslint检查"><a class="markdownIt-Anchor" href="#eslint检查"></a> ESLint检查</h1><p>使用eslint官方推荐的<a href="https://github.com/vim-syntastic/syntastic" target="_blank" rel="noopener">syntastic</a>插件来执行<code>eslint</code>。</p><p>通过Vundle安装syntastic插件，在vim下执行下面命令：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">:PluginInstall syntastic</span><br></pre></td></tr></table></figure><p>成功后，在<code>~/.vimrc</code>中添加：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">Plugin <span class="string">'syntastic'</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">let</span> <span class="variable">g:syntastic_always_populate_loc_list</span> = <span class="number">1</span></span><br><span class="line"><span class="keyword">let</span> <span class="variable">g:syntastic_auto_loc_list</span> = <span class="number">1</span></span><br><span class="line"><span class="keyword">let</span> <span class="variable">g:syntastic_check_on_open</span> = <span class="number">1</span></span><br><span class="line"><span class="keyword">let</span> <span class="variable">g:syntastic_check_on_wq</span> = <span class="number">0</span></span><br><span class="line"><span class="keyword">let</span> <span class="variable">g:syntastic_javascript_checkers</span> = [ <span class="string">'eslint'</span> ]</span><br></pre></td></tr></table></figure><p>完成后，vim中就可以通过执行系统的<code>eslint</code>命令来检查当前文件的错误了。</p><div class="note info"> <p>注意：上面的配置，将会使syntastic在保存文件和打开文件的时候自动执行。</p> </div><h1 id="eslint自动修复"><a class="markdownIt-Anchor" href="#eslint自动修复"></a> ESLint自动修复</h1><p>使用自研插件<a href="https://github.com/mapleque/vim-eslint-vim" target="_blank" rel="noopener">vim-eslint-fix</a>来执行<code>eslint --fix</code>。</p><p>通过Vundle安装vim-eslint-fix插件，在vim下执行下面命令：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">:PluginInstall mapleque/vim-eslint-fix</span><br></pre></td></tr></table></figure><p>成功后，在<code>~/.vimrc</code>中添加：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">Plugin <span class="string">'mapleque/vim-eslint-fix'</span></span><br></pre></td></tr></table></figure><p>在<code>~/.vim/ftplugin/javascript.vim</code>中添加：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">"如果没有设置过mapleader，请先自行设置</span></span><br><span class="line"><span class="string">"let mapleader="</span>,<span class="comment">"</span></span><br><span class="line"><span class="keyword">noremap</span> <span class="symbol"><Leader></span><span class="keyword">f</span> :EslintFix<span class="symbol"><CR></span></span><br><span class="line"></span><br><span class="line"><span class="comment">"如果需要在保存文件时候自动修复，添加下面这行</span></span><br><span class="line"><span class="comment">"autocmd BufwritePost *.js EslintFix</span></span><br></pre></td></tr></table></figure><p>完成后，当需要eslint自动修复错误的时候，在命令模式下输入<code>,f</code>即可。</p><h1 id="react新组件模板"><a class="markdownIt-Anchor" href="#react新组件模板"></a> React新组件模板</h1><p>创建模板文件<code>~/.vim/templates/react.js</code>:</p><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> React, { Component } <span class="keyword">from</span> <span class="string">'react'</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Index</span> <span class="keyword">extends</span> <span class="title">Component</span> </span>{</span><br><span class="line"> <span class="keyword">constructor</span> (props) {</span><br><span class="line"> <span class="keyword">super</span>(props)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> state = {}</span><br><span class="line"></span><br><span class="line"> render() {</span><br><span class="line"> <span class="keyword">return</span> (</span><br><span class="line"> <div>tbd<<span class="regexp">/div></span></span><br><span class="line"><span class="regexp"> )</span></span><br><span class="line"><span class="regexp"> }</span></span><br><span class="line"><span class="regexp">}</span></span><br><span class="line"><span class="regexp"></span></span><br><span class="line"><span class="regexp">export default Index</span></span><br></pre></td></tr></table></figure><p>编辑<code>~/.vimrc</code>文件，增加：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">autocmd BufNewFile *.js 0r ~/.vim/templates/react.js</span><br></pre></td></tr></table></figure>]]></content>
<categories>
<category> 工具 </category>
<category> Vim </category>
</categories>
<tags>
<tag> Vim </tag>
</tags>
</entry>
<entry>
<title>【vim】基本配置</title>
<link href="/posts/tool/vim/vim-start/"/>
<url>/posts/tool/vim/vim-start/</url>
<content type="html"><![CDATA[<h1 id="基本配置"><a class="markdownIt-Anchor" href="#基本配置"></a> 基本配置</h1><p>在<code>~/.vimrc</code>中添加：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">"基本配置</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">syntax</span> <span class="keyword">on</span> <span class="comment">"语法高亮</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">let</span> mapleader=<span class="string">","</span> <span class="comment">"快捷键配置</span></span><br><span class="line"><span class="comment">" 如此，可以在后面配置快捷键时候使用`<Leader>`标识符。</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">set</span> <span class="keyword">nu</span> <span class="comment">"显示行号</span></span><br><span class="line"><span class="keyword">set</span> ruler <span class="comment">"显示当前行状态</span></span><br><span class="line"><span class="keyword">set</span> cursorline <span class="comment">"显示当前行标尺</span></span><br><span class="line"><span class="keyword">set</span> cursorcolumn <span class="comment">"显示当前列标尺</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">set</span> autoindent <span class="comment">"自动缩进</span></span><br><span class="line"><span class="keyword">set</span> smartindent <span class="comment">"智能缩进</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">set</span> showmatch <span class="comment">"显示括号配对</span></span><br><span class="line"><span class="keyword">set</span> incsearch <span class="comment">"高亮搜索内容</span></span><br><span class="line"><span class="keyword">set</span> hlsearch <span class="comment">"高亮搜索内容</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">set</span> tabstop=<span class="number">4</span> <span class="comment">"tab等效字符数</span></span><br><span class="line"><span class="keyword">set</span> softtabstop=<span class="number">4</span> <span class="comment">"tab等效字符数</span></span><br><span class="line"><span class="keyword">set</span> <span class="built_in">shiftwidth</span>=<span class="number">4</span> <span class="comment">"tab字符数</span></span><br><span class="line"><span class="keyword">set</span> expandtab <span class="comment">"tab转成空格</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">set</span> <span class="keyword">list</span> <span class="comment">"显示制表符</span></span><br><span class="line"><span class="keyword">set</span> listchars=<span class="keyword">ta</span><span class="variable">b:</span>>-,trai<span class="variable">l:</span>- <span class="comment">"制表符和空格显示样式</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">colorscheme</span> desert <span class="comment">"配色方案</span></span><br><span class="line"></span><br><span class="line"><span class="comment">"每行最大字符数标尺</span></span><br><span class="line"><span class="keyword">set</span> <span class="keyword">cc</span>=+<span class="number">1</span>,+<span class="number">2</span>,+<span class="number">3</span> <span class="comment">" highlight three columns after 'textwidth'</span></span><br><span class="line"><span class="keyword">hi</span> ColorColumn ctermbg=lightgrey guibg=lightgrey</span><br><span class="line"><span class="keyword">set</span> textwidth=<span class="number">80</span></span><br><span class="line"></span><br><span class="line"><span class="comment">"状态行设置</span></span><br><span class="line"><span class="keyword">set</span> laststatus=<span class="number">2</span> <span class="comment">"在倒数第二行显示状态行</span></span><br><span class="line"><span class="keyword">set</span> statusline=%<%<span class="number">1</span>*\ %<span class="keyword">f</span>%<span class="keyword">m</span>%r%h%<span class="keyword">w</span>\ %= <span class="comment">"左侧显示文件名和文件状态</span></span><br><span class="line"><span class="keyword">set</span> statusline+=%<span class="number">2</span>*\ %<span class="keyword">y</span>\ %* <span class="comment">"文件类型</span></span><br><span class="line"><span class="keyword">set</span> statusline+=%<span class="number">3</span>*\ %{&ff}\ \|\ %{\<span class="string">"\".(&fenc==\"\"?&enc:&fenc).((exists(\"+bomb\")\ &&\ &bomb)?\",B\":\"\").\"\ \|\"}\ %-14.(%l:%c%V%)%* "</span>文件格式|文件编码|当前光标所处行列</span><br><span class="line"><span class="keyword">set</span> statusline+=%<span class="number">4</span>*\ %-<span class="number">5</span>.(%<span class="keyword">p</span>%%%)%* <span class="comment">"当前光标所处文件百分比</span></span><br><span class="line"><span class="comment">"设置状态行配色</span></span><br><span class="line"><span class="keyword">hi</span> User1 cterm=None ctermfg=<span class="number">251</span> ctermbg=<span class="number">240</span></span><br><span class="line"><span class="keyword">hi</span> User2 cterm=None ctermfg=<span class="number">183</span> ctermbg=<span class="number">239</span></span><br><span class="line"><span class="keyword">hi</span> User3 cterm=None ctermfg=<span class="number">208</span> ctermbg=<span class="number">238</span></span><br><span class="line"><span class="keyword">hi</span> User4 cterm=None ctermfg=<span class="number">246</span> ctermbg=<span class="number">237</span></span><br></pre></td></tr></table></figure><h1 id="代码折叠"><a class="markdownIt-Anchor" href="#代码折叠"></a> 代码折叠</h1><p>在<code>~/.vimrc</code>中添加配置：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">set</span> foldenable <span class="comment">"允许折叠</span></span><br><span class="line"><span class="keyword">set</span> foldmethod=<span class="built_in">indent</span> <span class="comment">"按缩进折叠</span></span><br></pre></td></tr></table></figure><p>在vim中执行命令查看手册：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">:h Folding</span><br></pre></td></tr></table></figure><p>常用命令：</p><ul><li>za 打开或关闭光标所在折叠（一层）</li><li>zA 打开或关闭光标所在折叠（所有）</li><li>[z 移动到当前折叠开始处</li><li>]z 移动到当前折叠结尾处</li><li>zj 移动到下一个折叠开始处</li><li>zk 移动到上一个折叠开始处</li></ul><h1 id="vundle插件管理工具"><a class="markdownIt-Anchor" href="#vundle插件管理工具"></a> Vundle插件管理工具</h1><p>下载Vundle：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">git <span class="built_in">clone</span> https://github.com/VundleVim/Vundle.vim.git ~/.vim/bundle/Vundle.vim</span><br></pre></td></tr></table></figure><p>在<code>~/.vimrc</code>中添加：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">set</span> nocompatible</span><br><span class="line"><span class="keyword">filetype</span> off</span><br><span class="line"><span class="keyword">set</span> rtp+=~/.<span class="keyword">vim</span>/bundle/Vundle.<span class="keyword">vim</span></span><br><span class="line"><span class="keyword">call</span> vundle#begin()</span><br><span class="line">Plugin <span class="string">'VundleVim/Vundle.vim'</span></span><br><span class="line"><span class="comment">" Plugins</span></span><br><span class="line"><span class="comment">"这里添加自己的插件列表，例如：</span></span><br><span class="line"><span class="comment">" Plugin 'scrooloose/nerdtree'</span></span><br><span class="line"><span class="keyword">call</span> vundle#end()</span><br><span class="line"><span class="keyword">filetype</span> plugin <span class="keyword">on</span></span><br></pre></td></tr></table></figure><p>以后安装插件，先在<code>~/.vimrc</code>中<code>Plugins</code>后面添加：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">Plugin <span class="string">'scrooloose/nerdtree'</span></span><br></pre></td></tr></table></figure><p>再在vim中执行：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">:PuginInstall</span><br></pre></td></tr></table></figure><h1 id="文件类型扩展配置"><a class="markdownIt-Anchor" href="#文件类型扩展配置"></a> 文件类型扩展配置</h1><p>可以通过单独的配置文件来修改指定类型文件的vim配置，如不同的缩进、代码宽度、插件命令等。</p><p>如果安装了Vundle，那么在安装过程中已经开启了文件类型扩展配置，可以直接使用。</p><p>如果没有安装Vundle，可以在<code>~/.vimrc</code>中添加：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">filetype</span> off</span><br><span class="line"><span class="keyword">filetype</span> plugin <span class="keyword">on</span></span><br></pre></td></tr></table></figure><p>配置完成后，创建文件夹<code>~/.vim/ftplugin</code>，然后在文件夹中添加以文件类型命名的配置文件即可自动加载，如<code>~/.vim/ftplugin/javascript.vim</code>。</p><h1 id="nerdtree左侧菜单插件"><a class="markdownIt-Anchor" href="#nerdtree左侧菜单插件"></a> NERDTree左侧菜单插件</h1><p><a href="https://github.com/scrooloose/nerdtree" target="_blank" rel="noopener">NERDTree</a>是一款实现了左侧导航菜单的插件。<br>这里提供使用Vundle安装的方法。</p><p>在<code>~/.vimrc</code>中，<code>Plugins</code>的位置，添加：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">Plugin <span class="string">'scrooloose/nerdtree'</span></span><br></pre></td></tr></table></figure><p>然后在vim中执行：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">:PluginInstall</span><br></pre></td></tr></table></figure><p>在<code>~/.vimrc</code>中添加相关配置：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">" nerdtree配置</span></span><br><span class="line"><span class="keyword">autocmd</span> StdinReadPre * <span class="keyword">let</span> <span class="variable">s:std_in</span>=<span class="number">1</span></span><br><span class="line"><span class="keyword">autocmd</span> VimEnter * <span class="keyword">if</span> <span class="built_in">argc</span>() == <span class="number">1</span> && <span class="built_in">isdirectory</span>(<span class="built_in">argv</span>()[<span class="number">0</span>]) && !exists(<span class="string">"s:std_in"</span>) | <span class="keyword">exe</span> <span class="string">'NERDTree'</span> <span class="built_in">argv</span>()[<span class="number">0</span>] | <span class="keyword">wincmd</span> <span class="keyword">p</span> | <span class="keyword">ene</span> | <span class="keyword">endif</span></span><br><span class="line"><span class="keyword">map</span> <span class="symbol"><C-n></span> :NERDTreeToggle<span class="symbol"><CR></span></span><br><span class="line"><span class="keyword">autocmd</span> bufenter * <span class="keyword">if</span> (<span class="built_in">winnr</span>(<span class="string">"$"</span>) == <span class="number">1</span> && <span class="built_in">exists</span>(<span class="string">"b:NERDTree"</span>) && <span class="variable">b:NERDTree</span>.isTabTree()) | q | <span class="keyword">endif</span></span><br></pre></td></tr></table></figure><p>上面的配置实现了三个功能：</p><ul><li>使用CTRL+n可以开启或关闭左侧导航栏</li><li>如果当前打开的文件是文件夹，则自动开启左侧导航栏，否则默认不开启</li><li>当关闭文件时，只剩下导航栏窗口，则自动关闭窗口，退出vim</li></ul>]]></content>
<categories>
<category> 工具 </category>
<category> Vim </category>
</categories>
<tags>
<tag> Vim </tag>
</tags>
</entry>
<entry>
<title>【开始用go】在MacOS上搭建开发环境</title>
<link href="/posts/practice/go/go-workspace/"/>
<url>/posts/practice/go/go-workspace/</url>
<content type="html"><![CDATA[<p>每个开发者，都有一个套适合自己的开发环境，如果你决定开始使用go语言开发，那么一定要先准备好自己的开发环境。</p><div class="note info"> <p>这里所指的开发环境，不只是一个能让go跑起来的环境，而是一个可以用于日常工作的完整的工作环境。</p> </div><p>本文将从三个方面介绍go语言开发环境的构建，并给出自己所构建的开发环境的配置：</p><ul><li>go安装和升级 – go运行的基础</li><li>路径和环境变量 – 更好地管理项目、依赖，并方便执行使用go安装的命令行工具</li><li>IDE – 一个个人熟悉的用于写go代码的编辑器</li></ul><a id="more"></a><h1 id="go安装和升级"><a class="markdownIt-Anchor" href="#go安装和升级"></a> go安装和升级</h1><div class="note info"> <p>Go官方提供了详细的安装文档 <a href="https://go-zh.org/doc/install" target="_blank" rel="noopener">https://go-zh.org/doc/install</a> ，读者可以按需索取。<br>这里将要介绍的是在MacOS下，使用homebrew安装并管理Go的详细方法。</p> </div><p>Homebrew是一个面向MacOS的包管理工具，官网 <a href="https://brew.sh/" target="_blank" rel="noopener">https://brew.sh/</a> 有详细安装使用方法说明。</p><p>在国内使用Homebrew，建议更改源：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">cd</span> <span class="string">"<span class="variable">$(brew --repo)</span>"</span></span><br><span class="line">git remote <span class="built_in">set</span>-url origin https://mirrors.ustc.edu.cn/brew.git </span><br><span class="line"></span><br><span class="line"><span class="built_in">cd</span> <span class="string">"<span class="variable">$(brew --repo)</span>/Library/Taps/homebrew/homebrew-core"</span></span><br><span class="line">git remote <span class="built_in">set</span>-url origin https://mirrors.ustc.edu.cn/homebrew-core.git </span><br><span class="line"></span><br><span class="line">brew update</span><br></pre></td></tr></table></figure><p>查看可用的Go版本：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">brew search go</span><br></pre></td></tr></table></figure><p>这里将可以看到一系列可以安装的Go版本：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">go [email protected] [email protected] [email protected] [email protected]</span><br></pre></td></tr></table></figure><p>通常直接安装最新版本即可：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">brew install go</span><br></pre></td></tr></table></figure><p>安装完成后，即可在命令行执行go命令了：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">go version</span><br></pre></td></tr></table></figure><p>当go更新了新版本，需要升级的时候，执行：</p><figure class="highlight sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">brew upgrade go</span><br></pre></td></tr></table></figure><p>这个命令将安装Homebrew所管理的最新版本的go，并替换掉原来安装的go。</p><h1 id="路径和环境变量"><a class="markdownIt-Anchor" href="#路径和环境变量"></a> 路径和环境变量</h1><p>安装完Go后，就可以开始写代码了。但是如果想要写项目，还需要更进一步的进行配置。</p><div class="note info"> <p>官方文档参考： <a href="https://go-zh.org/doc/code.html" target="_blank" rel="noopener">https://go-zh.org/doc/code.html</a> 。</p> </div><p>这里笔者给出自己在MacOS上的路径：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">~/</span><br><span class="line">├── workspace -> Documents/workspace</span><br><span class="line">│ ├── gopath</span><br><span class="line">│ │ ├── bin</span><br><span class="line">│ │ ├── pkg</span><br><span class="line">│ │ ├── src</span><br><span class="line">│ │ │ ├── github.com</span><br><span class="line">│ │ │ │ ├── mapleque</span><br><span class="line">│ │ │ │ │ ├── gostart</span><br><span class="line">│ │ │ ├── golang.org</span><br><span class="line">│ │ │ │ ├── x</span><br><span class="line">│ │ │ │ │ ├── tour</span><br><span class="line">│ ├── github.com</span><br><span class="line">│ │ ├── mapleque</span><br><span class="line">│ │ │ ├── gostart -> ~/workspace/gopath/src/github.com/mapleque/gostart</span><br></pre></td></tr></table></figure><p>设置环境变量：</p><figure class="highlight bash"><figcaption><span>~/.bash_profile</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">export</span> GOPATH=~/workspace/gopath</span><br><span class="line"><span class="built_in">export</span> PATH=<span class="variable">$PATH</span>:<span class="variable">$GOPATH</span>/bin</span><br></pre></td></tr></table></figure><p>其中，</p><ul><li>GOPROXY 用于<code>go get</code>时作为代理</li><li>GOAPTH 用于原始的go依赖路径</li><li>PATH 中增加 $GOPATH/bin 是为了让<code>go install</code>所安装的二进制文件能够直接被执行</li></ul><p>特别的，从go1.11版本开始，Go将go module作为官方包管理工具进行支持，其中go1.11和go1.12版本需要主动开启：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">export</span> GO111MODULE=on</span><br><span class="line"><span class="built_in">export</span> GOPROXY=https://goproxy.cn</span><br></pre></td></tr></table></figure><p>其中，</p><ul><li>GOPROXY 用于<code>go get</code>时自动代理使用国内源</li></ul><div class="note info"> <p>关于go module的使用，可以参考官方文档：<a href="https://blog.go-zh.org/using-go-modules" target="_blank" rel="noopener">https://blog.go-zh.org/using-go-modules</a> 。<br>笔者也将在另外一篇文章中详细讲述自己的使用方式，敬请期待。</p> </div><h1 id="编辑器"><a class="markdownIt-Anchor" href="#编辑器"></a> 编辑器</h1><div class="note info"> <p>官方推荐编辑器：<a href="https://golang.google.cn/doc/editors.html" target="_blank" rel="noopener">https://golang.google.cn/doc/editors.html</a> 。</p> </div><p>笔者使用的是vim-go。</p><ul><li>如何安装vim-go <a href="https://github.com/fatih/vim-go" target="_blank" rel="noopener">https://github.com/fatih/vim-go</a></li><li><a href="/posts/tool/vim/vim-diy/">如何安装vundle并配置vim</a></li><li>如何安装vim – <code>brew install vim</code></li></ul><p>更多关于vim的配置，可以参考<a href="/posts/tool/vim/vim-go/">【vim】使用vim作为开发go的IDE</a>。</p><div class="note warning"> <p>注意：最新版本vim-go由于使用了gopls，所以如果使用的是vim-go的最新版本，在执行<code>:GoInstallBanaries</code>命令时，必须开启go module模式，并且在当前文件夹中有go.mod文件。</p> </div><p>这里特别提一些值得一试vim-go命令：</p><ul><li><code>:GoDef, :GoDefPop</code>或者<code>Ctrl+], Ctrl+t</code> – 直接跳转到光标位置所指的方法或变量的定义代码</li><li><code>:GoAddTags, :GoRemoveTags</code> – 给当前光标所在属性添加删除标签</li><li><code>:GoMetaLinter</code> – 执行一系列代码检查</li><li><code>:GoImpl</code> – 生成实现指定接口的代码</li><li><code>:GoIfErr</code> – 生成错误校验的代码</li><li><code>:GoImports</code> – 自动增减需要import的包</li></ul><p>本质上讲，上面的命令都是通过调用一些go tool来实现。</p><div class="note info"> <p>Go有着众多的tool，它们几乎覆盖了从代码编写、编译测试运行、到性能监控等整个开发周期。<br>笔者在后面的文章中，也会选择性的介绍一些tool的实战，以启发读者如何利用好这些资源来更高效的工作。</p> </div>]]></content>
<categories>
<category> 实践 </category>
<category> 开始用go </category>
</categories>
<tags>
<tag> golang </tag>
</tags>
</entry>
<entry>
<title>【开始用go】从零开始</title>
<link href="/posts/practice/go/go-start/"/>
<url>/posts/practice/go/go-start/</url>
<content type="html"><![CDATA[<p>整体来说，本系列文章属于工程实践向。</p><div class="note warning"> <p>需要特别说明的是：</p><ul><li>本系列文章不是教程，学习go语言，请阅读官方教程 <a href="https://tour.go-zh.org/list" target="_blank" rel="noopener">https://tour.go-zh.org/list</a> 。</li><li>本系列文章中可能有大量笔者个人观点，虽然经历过实践的检验，但也不一定是最优。如有不同意见，欢迎交流探讨 <a href="mailto:[email protected]" target="_blank" rel="noopener">[email protected]</a> 。</li></ul> </div><h1 id="目录"><a class="markdownIt-Anchor" href="#目录"></a> 目录</h1><ul><li><a href="/posts/practice/go/go-workspace/">在MacOS上搭建开发环境</a></li><li><a href="/posts/practice/go/go-http/">开发http服务</a></li><li><a href="/posts/practice/go/go-practice-http">go语言在http服务开发上的工程实践经验</a></li></ul><h1 id="官方网站"><a class="markdownIt-Anchor" href="#官方网站"></a> 官方网站</h1><ul><li>官方 <a href="https://golang.org/" target="_blank" rel="noopener">https://golang.org/</a></li><li>官方国内代理 <a href="https://golang.google.cn/" target="_blank" rel="noopener">https://golang.google.cn/</a></li><li>官方中文翻译 <a href="https://go-zh.org/" target="_blank" rel="noopener">https://go-zh.org/</a></li></ul><p>以上三个官方，内容上稍有差异，请按需选择。</p><h1 id="其他资源"><a class="markdownIt-Anchor" href="#其他资源"></a> 其他资源</h1><ul><li>如何安装Go – <a href="https://go-zh.org/doc/install" target="_blank" rel="noopener">https://go-zh.org/doc/install</a></li><li>Go语言入门教程 – <a href="https://tour.go-zh.org/list" target="_blank" rel="noopener">https://tour.go-zh.org/list</a></li><li>开发环境配置及说明 – <a href="https://go-zh.org/doc/code.html" target="_blank" rel="noopener">https://go-zh.org/doc/code.html</a></li><li>IDEs – <a href="https://golang.google.cn/doc/editors.html" target="_blank" rel="noopener">https://golang.google.cn/doc/editors.html</a></li><li>Go Wiki – <a href="https://github.com/golang/go/wiki" target="_blank" rel="noopener">https://github.com/golang/go/wiki</a></li><li>Effective Go – <a href="https://go-zh.org/doc/effective_go.html" target="_blank" rel="noopener">https://go-zh.org/doc/effective_go.html</a></li></ul><hr><ul><li>godoc – <a href="https://go-zh.org/pkg/" target="_blank" rel="noopener">https://go-zh.org/pkg/</a></li><li>go command – <a href="https://go-zh.org/cmd/go/" target="_blank" rel="noopener">https://go-zh.org/cmd/go/</a></li><li>官方博客 – <a href="https://blog.go-zh.org/" target="_blank" rel="noopener">https://blog.go-zh.org/</a></li></ul><p>这里所列的资源，通过官网都可以找到链接。</p>]]></content>
<categories>
<category> 实践 </category>
<category> 开始用go </category>
</categories>
<tags>
<tag> golang </tag>
</tags>
</entry>
<entry>
<title>【从零打造社区搜索推荐服务】架构设计</title>
<link href="/posts/practice/0sc/0sc-artch/"/>
<url>/posts/practice/0sc/0sc-artch/</url>
<content type="html"><![CDATA[<h1 id="整体架构"><a class="markdownIt-Anchor" href="#整体架构"></a> 整体架构</h1><p>依据解决复杂问题应当先化繁为简的原则，这里首先对服务进行拆分。</p><pre class="mermaid" style="text-align: center;"> graph TB client[客户端]sh[Searchhub]admin[后台]internal[后台服务]sync[数据同步服务]es[Elasticsearch]cache[Cache]mysql[Mysql]src((源数据))client --搜索&推荐--> shsh --用模板搜索--> essh --kv读写--> cachesh --sql只读--> mysqladmin --内部体验&管理数据--> internalinternal --用模板搜索&运营字段更新--> esinternal --运营数据更新--> mysqlinternal --刷缓存--> cachesrc --定时--> syncsync --写入\更新--> es </pre><p>按照数据流向可以将整个服务划分为三个部分（参考图中箭头指向）：</p><ul><li>前台服务 – 以searchhub为中心，将多方数据源的数据进行组织整理后返回给用户。</li><li>后台服务 – 以数据运营管理为主，让运营者有机会影响前台服务，进而达到运营的目的。</li><li>数据同步服务 – 让源数据与业务数据保持同步更新。</li></ul><h1 id="数据同步服务"><a class="markdownIt-Anchor" href="#数据同步服务"></a> 数据同步服务</h1><p>数据同步服务的目标就是让用于搜索的源数据能够与业务数据保持同步。</p><p>逻辑上可以划分为触发和写入两个步骤。</p><pre class="mermaid" style="text-align: center;"> graph LR 入口 --触发--> 新数据新数据 --写入--> Elasticsearch </pre><p>触发形式通常有两种：</p><table><thead><tr><th style="text-align:left">对比</th><th style="text-align:left">主动探测</th><th style="text-align:left">被动通知</th></tr></thead><tbody><tr><td style="text-align:left">实现方式</td><td style="text-align:left">定时运行</td><td style="text-align:left">订阅事件</td></tr><tr><td style="text-align:left">业务方支持</td><td style="text-align:left">支持增量查询</td><td style="text-align:left">提供消息订阅</td></tr><tr><td style="text-align:left">实时性</td><td style="text-align:left">定时周期决定</td><td style="text-align:left">消息处理时间决定</td></tr><tr><td style="text-align:left">回滚方式</td><td style="text-align:left">调整查询游标</td><td style="text-align:left">调整消费位置</td></tr><tr><td style="text-align:left">回滚范围</td><td style="text-align:left">全部</td><td style="text-align:left">队列长度</td></tr><tr><td style="text-align:left">瓶颈</td><td style="text-align:left">增量查询效率</td><td style="text-align:left">数据更新频率</td></tr></tbody></table><p>两种触发形式各有优劣，可以根据实际情况选择实现。</p><p>当获取到需要更新的数据后，只需要将其转化成需要的形式写入(更新)到Elasticsearch。</p><h1 id="前台服务"><a class="markdownIt-Anchor" href="#前台服务"></a> 前台服务</h1><p>前台服务，主要是通过Searchhub提供一系列接口供业务方调用，进而产品化后提供给客户端。</p><p>Searchhub需要提供以下接口。</p><h2 id="搜索"><a class="markdownIt-Anchor" href="#搜索"></a> 搜索</h2><ul><li>输入：搜索词、附加参数（分页、分类等）</li><li>返回：搜索结果</li></ul><h2 id="推荐"><a class="markdownIt-Anchor" href="#推荐"></a> 推荐</h2><ul><li>输入：内容id、用户id、附加参数（数量、策略等）</li><li>返回：推荐结果</li></ul><h2 id="热搜"><a class="markdownIt-Anchor" href="#热搜"></a> 热搜</h2><ul><li>输入：无</li><li>返回：热搜结果</li></ul><h1 id="后台服务"><a class="markdownIt-Anchor" href="#后台服务"></a> 后台服务</h1><p>后台服务需要实现以下功能。</p><h2 id="管理数据结构和索引"><a class="markdownIt-Anchor" href="#管理数据结构和索引"></a> 管理数据结构和索引</h2><p>对于index的setting和mapping的管理，接入后台方便开发和运营。</p><h2 id="查询源数据"><a class="markdownIt-Anchor" href="#查询源数据"></a> 查询源数据</h2><p>通过指定属性查询源数据，可用于后面的编辑。</p><h2 id="管理源数据运营相关字段"><a class="markdownIt-Anchor" href="#管理源数据运营相关字段"></a> 管理源数据运营相关字段</h2><p>对于源数据中一些用于运营的字段，可以通过后台直接编辑修改。</p><h2 id="管理搜索模板"><a class="markdownIt-Anchor" href="#管理搜索模板"></a> 管理搜索模板</h2><p>对于搜索模板的历史版本进行管理。可以直接修改线上所使用的模板版本。</p><h2 id="预览搜索效果"><a class="markdownIt-Anchor" href="#预览搜索效果"></a> 预览搜索效果</h2><p>可以指定模板版本进行搜索。以达到上线前预览的效果。</p><h2 id="管理屏蔽词"><a class="markdownIt-Anchor" href="#管理屏蔽词"></a> 管理屏蔽词</h2><p>对屏蔽词进行管理。</p><h2 id="管理缓存"><a class="markdownIt-Anchor" href="#管理缓存"></a> 管理缓存</h2><p>查询和清除缓存。</p>]]></content>
<categories>
<category> 实践 </category>
<category> 从零打造社区搜索推荐服务 </category>
</categories>
<tags>
<tag> Elasticsearch </tag>
</tags>
</entry>
<entry>
<title>【从零打造社区搜索推荐服务】问题的提出</title>
<link href="/posts/practice/0sc/0sc-target/"/>
<url>/posts/practice/0sc/0sc-target/</url>
<content type="html"><![CDATA[<p>社区搜索推荐，是一个现代化社区所应该具有的基本功能。</p><p>从技术角度讲，一个完整的社区搜索推荐服务，需要解决以下问题：</p><ul><li>对中文的处理</li><li>根据用户输入，结合社区内容给出相关搜索建议</li><li>根据搜索词给出相关社区内容的搜索结果</li><li>对于指定搜索词，运营人员能够控制第一页应该展示那些结果和顺序</li><li>可以屏蔽一些搜索词，让其不能出结果</li><li>可以屏蔽一些内容，让其永远不能出现在结果中</li><li>能够简单识别网络爬虫行为，并进行封禁</li></ul>]]></content>
<categories>
<category> 实践 </category>
<category> 从零打造社区搜索推荐服务 </category>
</categories>
<tags>
<tag> Elasticsearch </tag>
</tags>
</entry>
<entry>
<title>【从零打造社区搜索推荐服务】序</title>
<link href="/posts/practice/0sc/0sc-start/"/>
<url>/posts/practice/0sc/0sc-start/</url>
<content type="html"><![CDATA[<h1 id="目录"><a class="markdownIt-Anchor" href="#目录"></a> 目录</h1><h2 id="基础篇"><a class="markdownIt-Anchor" href="#基础篇"></a> 基础篇</h2><ul><li><a href="/posts/practice/0sc/0sc-target/">问题的提出</a> – 全面精确的定义要解决的问题</li><li><a href="/posts/practice/0sc/0sc-artch/">架构设计</a> – 站在全局看服务</li><li>（挖坑待填）<a href>更新源数据</a> – 源数据更新时机和数据清洗</li><li>（挖坑待填）<a href>朴素的全文搜索</a> – 基于基本数据进行全文搜索</li><li>（挖坑待填）<a href>搜索建议的冷启动数据</a> – 生成能够支持搜索建议的最简数据集</li><li>（挖坑待填）<a href>用搜索的思路做推荐</a> – 使用标签做内容相关推荐</li><li>（挖坑待填）<a href>基于用户的标签的个性化推荐</a> – 浏览记录通常能如实反映用户的喜好</li></ul><h2 id="进阶篇"><a class="markdownIt-Anchor" href="#进阶篇"></a> 进阶篇</h2><ul><li>（挖坑待填）<a href>缓存的应用</a> – 通过缓存提升性能上线</li><li>（挖坑待填）<a href>黑白名单</a> – 各种黑白名单用来应对各种问题</li><li>（挖坑待填）<a href>社区内容特征提取</a> – 朴素贝叶斯算法的应用</li><li>（挖坑待填）<a href>通过协同过滤优化用户标签</a> – 加入协同过滤以优化推荐效果</li></ul><h2 id="终极篇"><a class="markdownIt-Anchor" href="#终极篇"></a> 终极篇</h2><ul><li>（挖坑待填）<a href>无监督学习下的内容特征提取</a> – LDA的探索</li><li>（挖坑待填）<a href>带反馈的智能搜索架构</a></li></ul><h2 id="其他"><a class="markdownIt-Anchor" href="#其他"></a> 其他</h2><ul><li><a href>待补充</a></li></ul>]]></content>
<categories>
<category> 实践 </category>
<category> 从零打造社区搜索推荐服务 </category>
</categories>
<tags>
<tag> Elasticsearch </tag>
</tags>
</entry>
<entry>
<title>在nodejs中使用puppeteer并通过docker部署</title>
<link href="/posts/tool/puppeteer/js-puppeteer-docker/"/>
<url>/posts/tool/puppeteer/js-puppeteer-docker/</url>
<content type="html"><![CDATA[<p><a href="https://github.com/GoogleChrome/puppeteer#readme" target="_blank" rel="noopener">Puppeteer</a>是Google基于Chromium开发的一个Node库。</p><p>用户通过调用Puppeteer的Api，可以做到任何Chrome浏览器能做到的事情。</p><p>因此，其应用领域可能包括：</p><ul><li>网页截图生成报告</li><li>服务端渲染</li><li>面向网页应用的自动化测试</li><li>更接近真实情况的探针监控</li><li>需要处理复杂逻辑的爬虫</li></ul><h2 id="开发环境"><a class="markdownIt-Anchor" href="#开发环境"></a> 开发环境</h2><p>安装puppeteer包：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">npm i puppeteer --save</span><br></pre></td></tr></table></figure><div class="note info"> <p>这里仅介绍基本用法以保证快速进入开发，高阶用法请参考<a href="https://github.com/GoogleChrome/puppeteer#readme" target="_blank" rel="noopener">官方文档</a>。</p> </div><p>编写代码：</p><figure class="highlight javascript"><figcaption><span>example.js</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> puppeteer = <span class="built_in">require</span>(<span class="string">'puppeteer'</span>);</span><br><span class="line"></span><br><span class="line">(<span class="keyword">async</span> () => {</span><br><span class="line"> <span class="keyword">const</span> browser = <span class="keyword">await</span> puppeteer.launch();</span><br><span class="line"> <span class="keyword">const</span> page = <span class="keyword">await</span> browser.newPage();</span><br><span class="line"> <span class="keyword">await</span> page.goto(<span class="string">'https://example.com'</span>);</span><br><span class="line"> <span class="keyword">await</span> page.screenshot({<span class="attr">path</span>: <span class="string">'example.png'</span>});</span><br><span class="line"></span><br><span class="line"> <span class="keyword">await</span> browser.close();</span><br><span class="line">})();</span><br></pre></td></tr></table></figure><p>执行脚本：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">node example.js</span><br></pre></td></tr></table></figure><h2 id="生产环境"><a class="markdownIt-Anchor" href="#生产环境"></a> 生产环境</h2><p>如果是专用的带有node环境的linux机器，正常安装puppeteer包即可使用。</p><p>这里特别介绍一下在Docker环境下安装部署的过程。</p><p>我们通过创建一个装好了puppeteer包的镜像来提供部署环境：</p><figure class="highlight dockerfile"><figcaption><span>Dockerfile</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># node-chrome-10.16</span></span><br><span class="line"><span class="keyword">FROM</span> node:<span class="number">10.16</span>-alpine</span><br><span class="line"></span><br><span class="line"><span class="keyword">ENV</span> APP_PATH /app</span><br><span class="line"><span class="keyword">WORKDIR</span><span class="bash"> <span class="variable">${APP_PATH}</span></span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Change mirrors to tsinghua</span></span><br><span class="line"><span class="keyword">RUN</span><span class="bash"> <span class="built_in">echo</span> http://mirrors.tuna.tsinghua.edu.cn/alpine/edge/main > /etc/apk/repositories && \</span></span><br><span class="line"><span class="bash"> <span class="built_in">echo</span> http://mirrors.tuna.tsinghua.edu.cn/alpine/edge/community >> /etc/apk/repositories && \</span></span><br><span class="line"><span class="bash"> <span class="built_in">echo</span> http://mirrors.tuna.tsinghua.edu.cn/alpine/edge/testing >> /etc/apk/repositories && apk update</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Setting timezone</span></span><br><span class="line"><span class="keyword">RUN</span><span class="bash"> apk add tzdata openssh-client git</span></span><br><span class="line"><span class="keyword">RUN</span><span class="bash"> cp -r -f /usr/share/zoneinfo/Hongkong /etc/localtime</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Installs cnpm</span></span><br><span class="line"><span class="keyword">RUN</span><span class="bash"> npm install -g cnpm --registry=https://registry.npm.taobao.org</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Installs latest Chromium (73) package.</span></span><br><span class="line"><span class="keyword">RUN</span><span class="bash"> apk add --no-cache \</span></span><br><span class="line"><span class="bash"> curl \</span></span><br><span class="line"><span class="bash"> make \</span></span><br><span class="line"><span class="bash"> gcc \</span></span><br><span class="line"><span class="bash"> g++ \</span></span><br><span class="line"><span class="bash"> python \</span></span><br><span class="line"><span class="bash"> linux-headers \</span></span><br><span class="line"><span class="bash"> binutils-gold \</span></span><br><span class="line"><span class="bash"> gnupg \</span></span><br><span class="line"><span class="bash"> libstdc++ \</span></span><br><span class="line"><span class="bash"> udev \</span></span><br><span class="line"><span class="bash"> chromium=~73.0.3683.103 \</span></span><br><span class="line"><span class="bash"> nss \</span></span><br><span class="line"><span class="bash"> freetype \</span></span><br><span class="line"><span class="bash"> freetype-dev \</span></span><br><span class="line"><span class="bash"> harfbuzz \</span></span><br><span class="line"><span class="bash"> ttf-freefont \</span></span><br><span class="line"><span class="bash"> wqy-zenhei</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Tell Puppeteer to skip installing Chrome. We'll be using the installed package.</span></span><br><span class="line"><span class="keyword">ENV</span> PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true</span><br><span class="line"></span><br><span class="line"><span class="comment"># Puppeteer v1.12.2 works with Chromium 73.</span></span><br><span class="line"><span class="keyword">RUN</span><span class="bash"> yarn add [email protected]</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">RUN</span><span class="bash"> apk del --no-cache make gcc g++ python binutils-gold gnupg libstdc++</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Add user so we don't need --no-sandbox.</span></span><br><span class="line"><span class="comment">#RUN addgroup -S pptruser && adduser -S -g pptruser pptruser \</span></span><br><span class="line"><span class="comment"># && mkdir -p /home/pptruser/Downloads /app \</span></span><br><span class="line"><span class="comment"># && chown -R pptruser:pptruser /home/pptruser \</span></span><br><span class="line"><span class="comment"># && chown -R pptruser:pptruser /app</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment">## Run everything after as non-privileged user.</span></span><br><span class="line"><span class="comment">#USER pptruser</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">CMD</span><span class="bash"> [<span class="string">'/bin/sh'</span>]</span></span><br></pre></td></tr></table></figure><div class="note info"> <p>上面代码中注释掉了给puppeteer创建用户运行的部分，如果你的docker服务是在root下启动的，上面的镜像可以正常工作。<br>否则就需要给执行该镜像的用户开通一些权限，可以在后面执行的时候注意输出错误日志以获取所需权限。</p> </div><p>在上面的环境中使用puppeteer，需要在初始化的时候进行一些特别的配置：</p><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> puppeteer = <span class="built_in">require</span>(<span class="string">'puppeteer'</span>)</span><br><span class="line">(<span class="keyword">async</span> () => {</span><br><span class="line"> <span class="keyword">const</span> browser = <span class="keyword">await</span> puppeteer.launch({</span><br><span class="line"> args: [</span><br><span class="line"> <span class="string">'--disable-dev-shm-usage'</span>,</span><br><span class="line"> <span class="string">'--no-sandbox'</span></span><br><span class="line"> ],</span><br><span class="line"> executablePath: <span class="string">'/usr/bin/chromium-browser'</span></span><br><span class="line"> })</span><br><span class="line"> <span class="keyword">const</span> page = <span class="keyword">await</span> browser.newPage();</span><br><span class="line"> <span class="keyword">await</span> page.goto(<span class="string">'https://example.com'</span>);</span><br><span class="line"> <span class="keyword">await</span> page.screenshot({<span class="attr">path</span>: <span class="string">'example.png'</span>});</span><br><span class="line"></span><br><span class="line"> <span class="keyword">await</span> browser.close();</span><br><span class="line">})();</span><br></pre></td></tr></table></figure><p>最后，在上面Docker环境下执行代码，或者部署项目即可。</p><p>这里给出一个egg项目部署的Dockerfile作为参考：</p><figure class="highlight dockerfile"><figcaption><span>Dockerfile</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">FROM</span> mapleque/node-chrome:<span class="number">10.16</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">ENV</span> APP_PATH /app</span><br><span class="line"></span><br><span class="line"><span class="keyword">WORKDIR</span><span class="bash"> <span class="variable">${APP_PATH}</span></span></span><br><span class="line"></span><br><span class="line"><span class="comment"># COPY --chown=pptruser:pptruser . ${APP_PATH}</span></span><br><span class="line"><span class="keyword">COPY</span><span class="bash"> . <span class="variable">${APP_PATH}</span></span></span><br><span class="line"></span><br><span class="line"><span class="keyword">RUN</span><span class="bash"> cnpm install --production</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">ENTRYPOINT</span><span class="bash"> NODE_ENV=production EGG_SERVER_ENV=<span class="variable">${EGG_SERVER_ENV}</span> npx egg-scripts start --port=<span class="variable">${APP_PORT}</span> --workers=1</span></span><br></pre></td></tr></table></figure><div class="note info"> <p>同样需要注意的是，这里启动项目的用户要与环境中授权puppeteer的用户保持一致。</p> </div>]]></content>
<categories>
<category> 工具 </category>
<category> puppeteer </category>
</categories>
<tags>
<tag> nodejs </tag>
<tag> puppeteer </tag>
<tag> docker </tag>
</tags>
</entry>
<entry>
<title>实现OIDC协议</title>
<link href="/posts/practice/auth/oidc/"/>
<url>/posts/practice/auth/oidc/</url>
<content type="html"><![CDATA[<blockquote><p>OpenID Connect 1.0 is a simple identity layer on top of the OAuth 2.0 protocol. It allows Clients to verify the identity of the End-User based on the authentication performed by an Authorization Server, as well as to obtain basic profile information about the End-User in an interoperable and REST-like manner.</p><footer><strong>OpenID Connect</strong><cite><a href="https://openid.net/connect/" target="_blank" rel="noopener">openid.net/connect</a></cite></footer></blockquote><p>OIDC工作原理和应用场景如下图所示：</p><pre class="mermaid" style="text-align: center;"> sequenceDiagram participant u as userparticipant app as oidc-appparticipant oidc as oidc-serverapp ->>+ oidc: 获取oidc配置Note over app,oidc: .well-known/oepnid-configurationoidc ->> app: 返回oidc配置app ->>+ oidc: 获取密钥Note over app,oidc: conf.jwks_urioidc ->>- app: 返回jwks并缓存u ->>+ app: 未登录用户请求app ->>- u: 跳转登录页u ->>+ oidc: 登录并授权Note over app,oidc: conf.authorization_endpointNote over oidc: 生成授权码codeoidc ->>- u: 返回授权码codeu ->>+ app: 提交授权码codeapp ->>+ oidc: 申请tokenNote over app,oidc: conf.token_endpointNote over oidc: 生成jwt-tokenoidc ->>- app: 返回jwt-tokenapp ->>- u: 返回服务token，可以直接使用jwk-tokenu ->>+ app: 带jwt-token请求Note over app: 解密jwt-tokenNote over app: 验证权限app ->>- u: 返回请求结果 </pre><p>其中，用户请求应用，需要把jwt-token放在请求Header中：<code>Authorization: Bearer <jwt-token></code>。</p><p>应用通过缓存的jwks解析jwt-token获取用户信息并验证相关权限。</p>]]></content>
<categories>
<category> 实践 </category>
<category> 用户认证 </category>
</categories>
<tags>
<tag> Kubernates </tag>
<tag> OIDC </tag>
<tag> jwt </tag>
</tags>
</entry>
<entry>
<title>微信小程序快速接入实践</title>
<link href="/posts/practice/wx/mpapi/"/>
<url>/posts/practice/wx/mpapi/</url>
<content type="html"><![CDATA[<p>微信小程序作为一种轻量级移动应用形式，体系发展已经日趋成熟。以微信小程序作为主要载体的应用也越来越多。投入少，见效快，自然的获取用户渠道是其优势所在。笔者恰好经历了2018年小程序大爆发时期，也义无反顾地踏入过这趟浑水。无论如何，最终留下的经验才是最宝贵的，也就是这里将要介绍的一套快速接入微信小程序后端开发迭代方案。</p><h1 id="引言"><a class="markdownIt-Anchor" href="#引言"></a> 引言</h1><p>这里之所以说是快速接入小程序的后端开发迭代方案，其原因有三：</p><ul><li>小程序前端基于javascript语法，提供了一些内置扩展组件以及一套成熟的设计标准，在开发上没有任何余地，也无需优化。</li><li>小程序后端最基本的登录功能，都需要实现复杂的调用微信api逻辑，还有一些列更复杂的加解密验签接口可能需要接入，此外还要实现基本的登录态保持逻辑，工作量很大。</li><li>当需要快速实现多个小程序进行试错的时候，大量重复的工作都可以通过一个完整的服务来优化。</li></ul><p>综上，笔者就提出了一套方案。</p><h1 id="方案描述"><a class="markdownIt-Anchor" href="#方案描述"></a> 方案描述</h1><pre class="mermaid" style="text-align: center;"> sequenceDiagram participant u as 用户participant mp as 小程序participant mpapi as 小程序代理服务participant api as 小程序后端服务participant wx as 微信u ->> mp: 用户使用小程序mp ->> mpapi: 小程序请求代理服务mpapi ->> wx: 小程序请求微信wx ->> mpapi: 微信返回数据给代理服务mpapi ->> api: 代理服务带着微信的数据请求后端服务api ->> mp: 后端服务返回数据给小程序mp ->> u: 小程序反馈给用户结果 </pre><p>如图所示，微信小程序在需要请求后端的时候，直接请求代理服务，由代理服务实现接入微信并请求后端的逻辑，最后把数据返回给用户。</p><p>代理服务在请求小程序后端服务的时候，将会携带从微信获取的数据，如：用户信息。</p><p>代理服务可以维护用户登录状态，并给用户创建统一的id进行管理。</p><h1 id="实现"><a class="markdownIt-Anchor" href="#实现"></a> 实现</h1><p>mpapi(<a href="https://github.com/mapleque/mpapi" target="_blank" rel="noopener">Code on Github</a>)使上述方案中代理服务的一个实现。</p><p>支持使用线上服务和私有化部署两种方式，参考<a href="https://github.com/mapleque/mpapi" target="_blank" rel="noopener">使用文档</a>。</p>]]></content>
<categories>
<category> 实践 </category>
<category> 微信开发者 </category>
</categories>
<tags>
<tag> wx </tag>
<tag> miniprogram </tag>
</tags>
</entry>
<entry>
<title>基于Kerberos实现内部系统员工账号单点登录</title>
<link href="/posts/practice/auth/kerberos/"/>
<url>/posts/practice/auth/kerberos/</url>
<content type="html"><![CDATA[<blockquote><p>Kerberos is a network authentication protocol. It is designed to provide strong authentication for client/server applications by using secret-key cryptography.</p><footer><strong>What is Kerberos?</strong><cite><a href="http://web.mit.edu/kerberos" target="_blank" rel="noopener">web.mit.edu/kerberos</a></cite></footer></blockquote><p>Kerberos工作原理和应用场景如下图所示：</p><pre class="mermaid" style="text-align: center;"> sequenceDiagram participant u as Userparticipant client as Clientparticipant server as Serverparticipant kdc as Kerberosu ->> client: username, passwordclient ->> client: cache(username, password)client ->> kdc: (to AS) usernamekdc ->> kdc: CTSK = randkdc ->> kdc: TGT = {CTSK, username, expired}kdc ->> kdc: password = findBy(username)kdc ->> kdc: EncCTSK = enc(CTSK, password)kdc ->> kdc: TGSSecretKey = constkdc ->> kdc: EncTGT = enc(TGT, TGSSecretKey)kdc ->> client: EncCTSK, EncTGTclient ->> client: password = loadCache()client ->> client: CTSK = decrypt(EncCTSK, password)client ->> client: cache(CTSK)client ->> client: Authenticator = {username, timestamp}client ->> client: cache(Authenticator)client ->> client: CTSKEncAuthenticator = enc(Authenticator, CTSK)client ->> kdc: (to TGS) EncTGT, AppID, CTSKEncAuthenticatorkdc ->> kdc: TGSSecretKey = constkdc ->> kdc: TGT = decrypt(EncTGT, TGSSecretKey)kdc ->> kdc: valid(TGT.expired)kdc ->> kdc: CTSK = TGT.CTSKkdc ->> kdc: Authenticator = decrypt(CTSKEncAuthenticator, CTSK)kdc ->> kdc: username = Authenticator.usernamekdc ->> kdc: AppSecretKey = findBy(AppID)kdc ->> kdc: CSSK = randkdc ->> kdc: ST = {CSSK, username, expired}kdc ->> kdc: EncST = enc(ST, AppSecretKey)kdc ->> kdc: EncCSSK = enc(CSSK, CTSK)kdc ->> client: EncST, EncCSSKclient ->> client: CTSK = loadCache()client ->> client: CSSK = decrypt(EncCSSK, CTSK)client ->> client: cache(CSSK)client ->> client: Authenticator = loadCache()client ->> client: CSSKEncAuthenticator = enc(Authenticator, CSSK)client ->> server: EncST, CSSKEncAuthenticatorserver ->> server: AppSecretKey = constserver ->> server: ST = decrypt(EncST, AppSecretKey)server ->> server: valid(ST.expired)server ->> server: CSSK = ST.CSSKserver ->> server: Authenticator = descrypt(CSSKEncAuthenticator, CSSK)server ->> server: valid(Authenticator.username == ST.username)server ->> server: Token = randserver ->> server: EncToken = enc(Token, CSSK)server ->> client: EncTokenclient ->> client: CSSK = loadCache()client ->> client: Token = decrypt(EncToken, CSSK)client ->> client: cache(Token)u ->>+ client: actionclient ->>+ server: request, Tokenserver ->>- client: response dataclient ->>- u: view </pre><p>其中：</p><ul><li>User – 用户</li><li>Client – 应用客户端</li><li>Server – 应用服务端</li><li>Kerberos – 认证授权服务，包含AS,TGS等</li><li>AS – 认证服务(Authentication Service)</li><li>TGS – 票据授权服务(Ticket Granting Service)</li><li>username – 用户输入的用户名</li><li>password – 用户输入的密码</li><li>EncCTSK – 通过AS上的用户password加密的CTSK</li><li>CTSK – 用于Client与TGS通信的密钥(Client TGS Session Key)</li><li>EncTGT – 通过TGSSecretKey加密的TGT</li><li>TGT – 授权票据(Ticket Granting Ticket)，包含CTSK、username和expired</li><li>Authenticator – 将username和timestamp包装为一个Authenticator</li><li>CTSKEncAuthenticator – 通过CTSK加密的Authenticator</li><li>AppID – 唯一标识一个应用服务提供方，应用服务可能包含Client和Server</li><li>TGSSecretKey – TGS自己的维护的密钥</li><li>EncST – 通过AppSecretKey加密的ST</li><li>AppSecretKey – 应用服务提供方在TGS上注册的密钥，一个AppID对应一个TGSSecretKey</li><li>ST – 应用票据(Service Ticket)</li><li>EncCSSK – 通过CTSK加密的CSSK</li><li>CSSK – 用于Client与Server通信的密钥Client Server Session Key</li><li>Token – 用于Client与Server通信的会话标识，由应用实现，可以是Cookie，Session，HTTP Header等</li></ul><p>从上图中可以看出，整个过程分成三个部分：</p><ul><li>认证 – 确认用户身份和客户端是否可信</li><li>授权 – 授权用户访问指定服务</li><li>登录 – 用户凭借授权去登录服务，建立会话</li></ul><p>这里有两个特点需要强调：</p><ul><li>应用服务与认证授权服务在整个认证过程中没有直接通信</li><li>用户密码在整个认证过程中没有在网络上传播</li></ul><p>上面的认证授权服务在实际应用上，还欠缺了两部分：</p><ul><li>管理用户密码 – 需要给用户提供线上服务</li><li>管理应用密钥 – 需要给开发者提供线上(可离线)服务</li></ul>]]></content>
<categories>
<category> 实践 </category>
<category> 用户认证 </category>
</categories>
<tags>
<tag> Kerberos </tag>
</tags>
</entry>
<entry>
<title>使用openvpn管理办公网络</title>
<link href="/posts/tool/openvpn/openvpn/"/>
<url>/posts/tool/openvpn/openvpn/</url>
<content type="html"><![CDATA[<p>使用openvpn管理办公网络，可以做到：</p><ul><li>以证书作为客户端认证标识，通过颁发证书管理用户接入网络的权限</li><li>每个证书指定固定ip，在网络上可以监控ip行为，对应到用户</li><li>通过对用户分配不同ip段的ip，来控制用户在网络中的权限</li><li>推送内网域名解析，复杂内网路由保证复杂内网连通性</li></ul><h1 id="服务端"><a class="markdownIt-Anchor" href="#服务端"></a> 服务端</h1><h2 id="主服务"><a class="markdownIt-Anchor" href="#主服务"></a> 主服务</h2><ul><li><a href="https://github.com/OpenVPN/openvpn" target="_blank" rel="noopener">github.com/OpenVPN/openvpn</a></li><li><a href="https://openvpn.net/community-resources/how-to/" target="_blank" rel="noopener">官网文档openvpn.net</a>(需要翻墙)</li><li><a href="https://github.com/Nyr/openvpn-install" target="_blank" rel="noopener">一键安装脚本</a></li></ul><p>配置文件样例：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br></pre></td><td class="code"><pre><span class="line">verb 3 # LOG等级</span><br><span class="line">dev tun # 隧道设备名</span><br><span class="line">port 1194</span><br><span class="line"></span><br><span class="line"># 如果上层使用udp, vpn使用tcp, 丢包后vpn会就会重传上层的udp报文, vpn使用udp协议, 这样可以让上层应用自己决定使用tcp还是udp</span><br><span class="line">proto udp</span><br><span class="line"></span><br><span class="line">topology subnet # 此模式下所有客户端在一个子网里</span><br><span class="line"></span><br><span class="line"># keys</span><br><span class="line">ca /etc/openvpn/pki/ca.crt</span><br><span class="line">key /etc/openvpn/pki/private/vpn.mapleque.com.key</span><br><span class="line">cert /etc/openvpn/pki/issued/vpn.mapleque.com.crt</span><br><span class="line">crl-verify /etc/openvpn/pki/crl.pem</span><br><span class="line">dh /etc/openvpn/pki/dh.pem</span><br><span class="line">tls-auth /etc/openvpn/pki/ta.key</span><br><span class="line"></span><br><span class="line">key-direction 0</span><br><span class="line">keepalive 10 60</span><br><span class="line"></span><br><span class="line"># 超时重连接时不会重载key和tun</span><br><span class="line">persist-key</span><br><span class="line">persist-tun</span><br><span class="line"></span><br><span class="line">status /var/log/openvpn/status.log</span><br><span class="line">log /var/log/openvpn/log.log</span><br><span class="line">management localhost 7505</span><br><span class="line"></span><br><span class="line">client-config-dir /etc/openvpn/ccd</span><br><span class="line"></span><br><span class="line">client-to-client # 允许client之间通信</span><br><span class="line"></span><br><span class="line">user vpn</span><br><span class="line">group vpn</span><br><span class="line"></span><br><span class="line"># vpn进程启动后给本机添加该路由(不需要有客户端连接), 可以配置多条</span><br><span class="line">route 172.31.0.0 255.255.0.0 192.168.232.20</span><br><span class="line"></span><br><span class="line"># 启动server模式, 隧道使用该网段, 并占用第一个地址用作本机隧道的IP, nopool表示不将此网段用作dhcp</span><br><span class="line">server 192.168.224.0 255.255.240.0 nopool</span><br><span class="line"></span><br><span class="line"># 单独配置dhcp pool</span><br><span class="line">ifconfig-pool 192.168.224.2 192.168.225.254 255.255.240.0</span><br><span class="line"></span><br><span class="line">push "dhcp-option DNS 192.168.224.1"</span><br><span class="line"></span><br><span class="line">push "route 192.168.16.0 255.255.240.0 vpn_gateway"</span><br></pre></td></tr></table></figure><p>systemd启动配置样例：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"># This service is actually a systemd target,</span><br><span class="line"># but we are using a service since targets cannot be reloaded.</span><br><span class="line"># /lib/systemd/system/openvpn.service</span><br><span class="line"></span><br><span class="line">[Unit]</span><br><span class="line">Description=OpenVPN service</span><br><span class="line">After=network.target</span><br><span class="line"></span><br><span class="line">[Service]</span><br><span class="line">Type=forking</span><br><span class="line">ExecStart=/usr/sbin/openvpn --daemon ovpn --cd /etc/openvpn --config /etc/openvpn/vpn.conf</span><br><span class="line">ExecReload=/bin/kill -HUP $MAINPID</span><br><span class="line">WorkingDirectory=/etc/openvpn</span><br><span class="line"></span><br><span class="line">[Install]</span><br><span class="line">WantedBy=multi-user.target</span><br></pre></td></tr></table></figure><h2 id="证书颁发"><a class="markdownIt-Anchor" href="#证书颁发"></a> 证书颁发</h2><p>可以考虑使用easyrsa签发证书。</p><h1 id="客户端"><a class="markdownIt-Anchor" href="#客户端"></a> 客户端</h1><p>安装vpn客户端有风险，注意看好官方客户端版本。大部分客户端下载都需要翻墙，请自行检索解决方案。</p><ul><li><a href="https://itunes.apple.com/us/app/openvpn-connect/id590379981?mt=8" target="_blank" rel="noopener">IOS</a></li><li><a href="https://play.google.com/store/apps/details?id=net.openvpn.openvpn" target="_blank" rel="noopener">Android</a></li><li><a href="https://openvpn.net/downloads/openvpn-connect-v3-macos.dmg" target="_blank" rel="noopener">MacOS</a></li><li><a href="https://openvpn.net/vpn-server-resources/connecting-to-access-server-with-linux/" target="_blank" rel="noopener">Linux</a></li><li><a href="https://openvpn.net/downloads/openvpn-connect-v3-windows.msi" target="_blank" rel="noopener">Windows</a></li></ul>]]></content>
<categories>
<category> 工具 </category>
<category> openvpn </category>
</categories>
<tags>
<tag> vpn </tag>
</tags>
</entry>
<entry>
<title>通用后台实现方案</title>
<link href="/posts/practice/admin/admin/"/>
<url>/posts/practice/admin/admin/</url>
<content type="html"><![CDATA[<div class="note info"> <p>挖坑待填</p> </div>]]></content>
<categories>
<category> 实践 </category>
<category> 通用后台 </category>
</categories>
<tags>
<tag> admin </tag>
</tags>
</entry>
<entry>
<title>通过日志实现监控和报警</title>
<link href="/posts/tool/elk/elk-monitor/"/>
<url>/posts/tool/elk/elk-monitor/</url>
<content type="html"><![CDATA[<p>上篇文章<a href="/posts/practice/elastic/elk-logger">使用ELK管理日志</a>中，介绍了如何部署ELK并开始收集日志。</p><p>本文将继续介绍如何使用收集上来的日志。</p><p>这里我们主要讨论通过Kibana连接Elasticsearch后，可以使用的一些功能。</p><p>更多功能，请参考<a href="https://www.elastic.co/guide/en/kibana/current/index.html" target="_blank" rel="noopener">Kibana官方文档</a>。</p><h1 id="visualize"><a class="markdownIt-Anchor" href="#visualize"></a> Visualize</h1><p>这里介绍几种视图的应用场景，具体使用方法请自行探索。</p><ul><li>折线图 – 展示二维数据的变化情况，如每日平均接口请求次数</li><li>柱状图 – 展示二维数据的绝对值和相对值，如不同接口的请求量</li><li>表格 – 直接展示二维数据</li><li>Metric – 直接展示一维数据</li><li>Timelion – 展示数据随时间变化的情况以及多时间维度对比，如：请求量同比环比变化情况</li></ul><h1 id="dashboard"><a class="markdownIt-Anchor" href="#dashboard"></a> Dashboard</h1><p>Dashboard可以将多个Visualize放到一个页面内，并保存。Dashboard支持通过拖动的形式配置页面布局，以及每个Visualize的大小，当需要文字说明的时候，可以插入Markdown类型的Visualize。</p><p>注意，这里一个Dashboard的时间区间是统一的，也可以在保存Dashboard的时候指定Store time with dashboard，这样每次打开这个Dashboard，都会使用当前保存时候指定的时间区间（也支持相对值，如：Last 24 hours）。</p><h1 id="watcher"><a class="markdownIt-Anchor" href="#watcher"></a> Watcher</h1><p>在<code>Management->Elasticsearch->Watcher</code>可以创建管理Watcher。Watcher顾名思义是一个观察者服务，可以通过配置触发机制、输入数据、触发条件、执行动作等四个参数，来实现监控报告等功能。</p><h2 id="monitoring"><a class="markdownIt-Anchor" href="#monitoring"></a> Monitoring</h2><p>通过监控是否有错误日志，实现错误报警。</p><p>下面的配置每分钟查一次http_log中前30分钟的ERROR类型日志，如果有就发邮件报警。一旦发过邮件就静默30分钟。</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br></pre></td><td class="code"><pre><span class="line">{</span><br><span class="line"> <span class="attr">"trigger"</span>: {</span><br><span class="line"> <span class="attr">"schedule"</span>: {</span><br><span class="line"> <span class="attr">"interval"</span>: <span class="string">"1m"</span></span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"input"</span>: {</span><br><span class="line"> <span class="attr">"search"</span>: {</span><br><span class="line"> <span class="attr">"request"</span>: {</span><br><span class="line"> <span class="attr">"search_type"</span>: <span class="string">"query_then_fetch"</span>,</span><br><span class="line"> <span class="attr">"indices"</span>: [</span><br><span class="line"> <span class="string">"http_log"</span></span><br><span class="line"> ],</span><br><span class="line"> <span class="attr">"types"</span>: [],</span><br><span class="line"> <span class="attr">"body"</span>: {</span><br><span class="line"> <span class="attr">"query"</span>: {</span><br><span class="line"> <span class="attr">"bool"</span>: {</span><br><span class="line"> <span class="attr">"filter"</span>: [</span><br><span class="line"> {</span><br><span class="line"> <span class="attr">"range"</span>: {</span><br><span class="line"> <span class="attr">"@timestamp"</span>: {</span><br><span class="line"> <span class="attr">"from"</span>: <span class="string">"{{ctx.trigger.scheduled_time}}||-30m"</span>,</span><br><span class="line"> <span class="attr">"to"</span>: <span class="string">"{{ctx.trigger.triggered_time}}"</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> <span class="attr">"match"</span>: {</span><br><span class="line"> <span class="attr">"log_flag.keyword"</span>: {</span><br><span class="line"> <span class="attr">"query"</span>: <span class="string">"ERROR"</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"condition"</span>: {</span><br><span class="line"> <span class="attr">"script"</span>: {</span><br><span class="line"> <span class="attr">"source"</span>: <span class="string">"if (ctx.payload.hits.total > 0) {return true}return false;"</span>,</span><br><span class="line"> <span class="attr">"lang"</span>: <span class="string">"painless"</span></span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"actions"</span>: {</span><br><span class="line"> <span class="attr">"email"</span>: {</span><br><span class="line"> <span class="attr">"transform"</span>: {</span><br><span class="line"> <span class="attr">"script"</span>: {</span><br><span class="line"> <span class="attr">"source"</span>: <span class="string">"String body; body = '<div>'+ctx.payload.hits.total+' ERROR in http.log @'+ctx.trigger.triggered_time+'</div>'; for (item in ctx.payload.hits.hits) { body += '<div><h3>'+item._source.group+'/'+item._source.project+'</h3><p>'+item._source.source+'</p><pre>'+item._source.message+'</pre></div>' } return [ 'body': body ]"</span>,</span><br><span class="line"> <span class="attr">"lang"</span>: <span class="string">"painless"</span></span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"email"</span>: {</span><br><span class="line"> <span class="attr">"profile"</span>: <span class="string">"standard"</span>,</span><br><span class="line"> <span class="attr">"to"</span>: [</span><br><span class="line"> <span class="string">"[email protected]"</span></span><br><span class="line"> ],</span><br><span class="line"> <span class="attr">"subject"</span>: <span class="string">"[alert]ERROR in http.log"</span>,</span><br><span class="line"> <span class="attr">"body"</span>: {</span><br><span class="line"> <span class="attr">"html"</span>: <span class="string">"{{ctx.payload.body}}"</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"throttle_period_in_millis"</span>: <span class="number">1800000</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="reporting"><a class="markdownIt-Anchor" href="#reporting"></a> Reporting</h2><p>定期发送报告。</p><p>Dashboard提供了api支持实时生成pdf报告，配合watcher可以将该报告以邮件的事实发送。</p><p>下面的配置，每天早晨8点，发送报告。</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><span class="line">{</span><br><span class="line"> <span class="attr">"trigger"</span>: {</span><br><span class="line"> <span class="attr">"schedule"</span>: {</span><br><span class="line"> <span class="attr">"daily"</span>: {</span><br><span class="line"> <span class="attr">"at"</span>: [</span><br><span class="line"> <span class="string">"8:00"</span></span><br><span class="line"> ]</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"input"</span>: {</span><br><span class="line"> <span class="attr">"none"</span>: {}</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"condition"</span>: {</span><br><span class="line"> <span class="attr">"always"</span>: {}</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"actions"</span>: {</span><br><span class="line"> <span class="attr">"send_email"</span>: {</span><br><span class="line"> <span class="attr">"email"</span>: {</span><br><span class="line"> <span class="attr">"profile"</span>: <span class="string">"standard"</span>,</span><br><span class="line"> <span class="attr">"attachments"</span>: {</span><br><span class="line"> <span class="attr">"a_example_reporting.pdf"</span>: {</span><br><span class="line"> <span class="attr">"reporting"</span>: {</span><br><span class="line"> <span class="attr">"url"</span>: <span class="string">"<reporting_url>"</span>,</span><br><span class="line"> <span class="attr">"retries"</span>: <span class="number">20</span>,</span><br><span class="line"> <span class="attr">"interval"</span>: <span class="string">"5s"</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"to"</span>: [</span><br><span class="line"> <span class="string">"[email protected]"</span></span><br><span class="line"> ],</span><br><span class="line"> <span class="attr">"subject"</span>: <span class="string">"[report]a example reporting"</span>,</span><br><span class="line"> <span class="attr">"body"</span>: {</span><br><span class="line"> <span class="attr">"html"</span>: <span class="string">"Some content in email body"</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>其中，reporting_url由Dashboard生成（点击指定的Dashboard右上角Reporting按钮，可以获得生成pdf的链接）。</p>]]></content>
<categories>
<category> 工具 </category>
<category> ELK </category>
</categories>
<tags>
<tag> ELK </tag>
</tags>
</entry>
<entry>
<title>基于kubernates实现高可用微服务架构</title>
<link href="/posts/practice/ms/k8s/"/>
<url>/posts/practice/ms/k8s/</url>
<content type="html"><![CDATA[<blockquote><p>Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.</p><footer><strong>kubernates</strong><cite><a href="https://kubernetes.io/" target="_blank" rel="noopener">kubernetes.io</a></cite></footer></blockquote><p>本文主要讲述如何部署一个用于web服务的高可用kubernates集群。该架构能够较好的适用于微服务体系，内置的服务发现机制给微服务治理带来了极佳的体验。</p><h2 id="集群架构"><a class="markdownIt-Anchor" href="#集群架构"></a> 集群架构</h2><p>话不多说，架构图如下：</p><div><div class="graphviz">digraph G { rankdir="TB" rank="same" node [ shape="box" style="filled" ]<pre><code>subgraph cluster_node0 { label="(2+)* master or slave" ex_proxy[label="proxy",fillcolor="red"] in_proxy[label="proxy",fillcolor="red"] controller1[label="kube-controller",fillcolor="gray"] scheduler1[label="kube-scheduler",fillcolor="gray"]}in_proxy->apiserver1[label=watch,dir=both]subgraph cluster_node1 { label="3*master" weave1[label="weave",fillcolor="gray"] kubelet1[label="kubelet",fillcolor="gray"] apiserver1[label="kube-apiserver",fillcolor="red"] etcd1[label="etcd",fillcolor="red"] apiserver1->etcd1}subgraph cluster_node2 { label="n*slave" weave2[label="weave",fillcolor="gray"] kubelet2[label="kubelet",fillcolor="gray"] pods2[label="pod ...",fillcolor="white"]}ex_request[label=外网请求,shape=none,style=empty]in_request[label=内网请求,shape=none,style=empty]ex_request -> ex_proxy[style=dashed,label=req]in_request -> in_proxy[style=dashed,label=req]ex_proxy -> in_proxy[style=dashed,label=req]in_proxy->pods2[style=dashed,label=req]in_proxy->pods2[label="health check"]</code></pre><p>}</p></div></div><p></p><p>这里，我们把集群相关服务分了三类：</p><ul><li>必须部署在master上的服务</li><li>必须部署在slave上的服务</li><li>master和slave都可以部署的服务</li></ul><p>其中，kubelet和weave是集群内每台机器必须部署的服务，无需过多研究。</p><p>kube-shceduler和kube-controller主要用于集群管理，正常部署即可。</p><p>kube-apiserver和etcd是集群master核心，简单来说，集群所有相关服务都需要通过和apiserver通信来控制集群行为。</p><p>用户所需要部署的服务都运行在slave的pod中。</p><p>以上服务都部署完成后，集群就可以正常运行了。那么用户的服务如何被请求到，是接下来要解决的重要问题。</p><h2 id="微服务治理"><a class="markdownIt-Anchor" href="#微服务治理"></a> 微服务治理</h2><p>所谓微服务治理，笔者认为主要是管理当前已经部署的那些服务如何被请求到。</p><p>kubernate本身提供了kube-proxy和service用于提供所部署服务的请求方案。基于这个方案，能够实现服务的正常访问，甚至还提供了负载均衡。但是在实践过程中总会发现一些问题，较难处理：</p><ul><li><p>如何方便快速的告诉服务使用方怎么请求我的服务？<br>事实上，当你新上线一个服务时，你需要从全局获取一个可用端口，然后用这个端口部署你的服务，然后告诉服务使用方带端口请求kube-proxy。</p></li><li><p>负载均衡能否做到高可用？<br>kube-proxy实现了负载均衡，能够让请求均匀的发送到不同的实例。然而，当实际部署实例的一个节点出问题的时候，kube-proxy并不能立即知道这个问题，也就不能够及时摘掉问题节点。这个时候你会发现，这样的效果跟直接部署在几台机器上没什么区别。</p></li><li><p>如何请求到指定实例，如何让两次请求请求到同一个实例？<br>由于机制和理念问题，这个做不到。</p></li></ul><p>如此一来，如果使用该方案，那么就不得不再单独设计服务治理以及高可用负载均衡方案。也就是说，我们现在只能这么干。</p><p>在上面的架构图中，proxy<a href="https://github.com/mapleque/proxy" target="_blank" rel="noopener">Code on Github</a>就是这样一个提供了服务发现和高可用负载均衡的服务。</p><p>本质上讲，proxy是一个完整的反向代理服务，如果你熟悉nginx，那么结合lua脚本的开发，也可以做到proxy所做的事情：增加一个上游服务的自定义健康检查功能。</p><p>此外，proxy还实现了一个通过请求kube-apiserver接口，来监控集群变化，进而更新proxy配置的插件。</p><p>使用该插件，用户需要定义：</p><ul><li>部署项目名</li><li>项目服务端口名到端口的映射</li></ul><p>通过上面两项，可以按照规则生成当前项目的一些列域名，每个域名都可以被代理到对应的端口，即实现了约定式的服务发现。</p><h2 id="日志收集"><a class="markdownIt-Anchor" href="#日志收集"></a> 日志收集</h2><p>通过在每个slave节点上创建日志目录，并将其默认挂载到所有pods的docker上，只要服务的日志都输出到指定路径，即可统一被slave机器上的filebeat收集到日志服务器。</p><h2 id="集群监控"><a class="markdownIt-Anchor" href="#集群监控"></a> 集群监控</h2><p>通过收集proxy的健康检查日志，可以实现实时集群监控。</p>]]></content>
<categories>
<category> 实践 </category>
<category> 微服务 </category>
</categories>
<tags>
<tag> Kubernates </tag>
</tags>
</entry>
<entry>
<title>使用gitlab持续集成部署</title>
<link href="/posts/tool/gitlab/gitlab-ci/"/>
<url>/posts/tool/gitlab/gitlab-ci/</url>
<content type="html"><![CDATA[<p>创建.gitlab-ci.yml配置文件，可以提交ci任务到gitlab-runner(docker模式)。</p><p>配置文件示例：</p><figure class="highlight yaml"><figcaption><span>.gitlab-ci.yml</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">tages:</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">test</span> <span class="comment"># 单元测试，集成测试，各种代码检查等。如有必要，可考虑拆分为多个阶段</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">build</span> <span class="comment"># 编译，打包等。如有必要，可考虑拆分为多个阶段</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">docker</span> <span class="comment"># 生成docker镜像，并上传到registry</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">deploy</span> <span class="comment"># 发布</span></span><br><span class="line"></span><br><span class="line"><span class="attr">test-example:</span></span><br><span class="line"><span class="attr"> stage:</span> <span class="string">test</span></span><br><span class="line"><span class="attr"> script:</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">echo</span> <span class="string">'do some test here'</span></span><br><span class="line"></span><br><span class="line"><span class="attr">build-example:</span></span><br><span class="line"><span class="attr"> stage:</span> <span class="string">build</span></span><br><span class="line"><span class="attr"> script:</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">echo</span> <span class="string">'do some build here'</span></span><br><span class="line"></span><br><span class="line"><span class="attr">docker-example:</span></span><br><span class="line"><span class="attr"> image:</span> <span class="string">"docker:dind"</span> <span class="comment"># 打包docker镜像固定使用此镜像</span></span><br><span class="line"><span class="attr"> stage:</span> <span class="string">docker</span></span><br><span class="line"><span class="attr"> when:</span> <span class="string">on_success</span> <span class="comment"># 如果依赖前序任务执行，需要设置该选项，避免生成无用docker镜像</span></span><br><span class="line"><span class="attr"> dependencies:</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">build</span> <span class="comment"># 如果打包镜像时需要前序任务的atrifacts，需要设置这个依赖使artifacts能够传递下来</span></span><br><span class="line"><span class="attr"> only:</span> <span class="comment"># 设置执行该任务的分支</span></span><br><span class="line"> <span class="comment"># ************************</span></span><br><span class="line"> <span class="comment"># 注意 ** 没必要每次ci都上传镜像，设置用于发布到测试环境的分支和正式发布的branch或tag。</span></span><br><span class="line"> <span class="comment"># ************************</span></span><br><span class="line"> <span class="comment"># - /^feature.*$/ # 例如：feature/xxx等，通常是git branch</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">/^.*-release$/</span> <span class="comment"># 例如：1.0.0-release等，通常是git tag</span></span><br><span class="line"><span class="attr"> script:</span></span><br><span class="line"> <span class="comment"># 定义镜像名</span></span><br><span class="line"> <span class="comment"># 镜像名: ${CI_PROJECT_PATH} 请务必使用该环境变量作为镜像名，否则会影响后面部署</span></span><br><span class="line"> <span class="comment"># 镜像tag: 注意区分测试镜像和正式镜像</span></span><br><span class="line"> <span class="comment"># 测试镜像tag: stage-${CI_PIPELINE_ID}，可以保证每次执行任务都生成新的镜像并上传</span></span><br><span class="line"> <span class="comment"># 正式镜像tag: ${CI_COMMIT_REF_NAME}，直接使用项目tag名，例如1.0.0-release</span></span><br><span class="line"> <span class="comment"># - export IMAGE_NAME=mapleque/${CI_PROJECT_PATH}:stage-${CI_PIPELINE_ID}</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">export</span> <span class="string">IMAGE_NAME=mapleque/${CI_PROJECT_PATH}:${CI_COMMIT_REF_NAME}</span></span><br><span class="line"> <span class="comment">#</span></span><br><span class="line"> <span class="comment"># 打包生成镜像并上传</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">docker</span> <span class="string">build</span> <span class="bullet">-t</span> <span class="string">${IMAGE_NAME}</span> <span class="string"><Dockerfile</span> <span class="string">path></span> <span class="string">&&</span> <span class="string">docker</span> <span class="string">push</span> <span class="string">${IMAGE_NAME}</span></span><br><span class="line"></span><br><span class="line"><span class="attr">deploy-example:</span></span><br><span class="line"><span class="attr"> image:</span> <span class="string">"mapleque/deploy"</span> <span class="comment"># 发布专用镜像，实现了deploy命令，参考项目：https://github.com/mapleque/deploy</span></span><br><span class="line"><span class="attr"> stage:</span> <span class="string">deploy</span></span><br><span class="line"><span class="attr"> when:</span> <span class="string">on_success</span></span><br><span class="line"><span class="attr"> dependencies:</span> <span class="string">[]</span> <span class="comment"># 这里注意要把dependencies设置为空，这样回滚和再次上线才能够被正常执行</span></span><br><span class="line"><span class="attr"> only:</span></span><br><span class="line"> <span class="comment"># 同docker部分</span></span><br><span class="line"> <span class="comment"># - /^feature.*$/ # 例如：feature/xxx等，通常是git branch</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">/^.*-release$/</span> <span class="comment"># 例如：1.0.0-release等，通常是git tag</span></span><br><span class="line"><span class="attr"> environment:</span></span><br><span class="line"> <span class="comment"># 设置这个将会在gitlab上生成一个上线记录，方便以后进行回滚和重新发布操作</span></span><br><span class="line"> <span class="comment"># name: stage # 发布到测试环境</span></span><br><span class="line"><span class="attr"> name:</span> <span class="string">release</span> <span class="comment"># 发布到正式环境</span></span><br><span class="line"><span class="attr"> script:</span></span><br><span class="line"> <span class="comment"># 通过设置Settings->CI/CD->Secret Variables，可以将保密的环境变量注入到ci中</span></span><br><span class="line"> <span class="comment">#</span></span><br><span class="line"> <span class="comment"># 集群配置</span></span><br><span class="line"> <span class="comment">#</span></span><br><span class="line"> <span class="comment"># 指定发布类型</span></span><br><span class="line"> <span class="comment"># 支持：kubernates, linux</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">export</span> <span class="string">DEPLOY_CONFIG_TYPE=kubernates</span></span><br><span class="line"> <span class="comment">#</span></span><br><span class="line"> <span class="comment"># 指定发布目标</span></span><br><span class="line"> <span class="comment"># - export DEPLOY_CONFIG_TARGET=stage.example.mapleque.com # 测试环境</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">export</span> <span class="string">DEPLOY_CONFIG_TARGET=example.mapleque.com</span> <span class="comment"># 正式环境</span></span><br><span class="line"> <span class="comment">#</span></span><br><span class="line"> <span class="comment"># 用于发布的TOKEN</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">export</span> <span class="string">DEPLOY_CONFIG_TOKEN=${DEPLOY_TOKEN}</span></span><br><span class="line"> <span class="comment">#</span></span><br><span class="line"> <span class="comment"># 发布使用的镜像名，与之前上传的保持一致</span></span><br><span class="line"> <span class="comment"># - export DEPLOY_CONFIG_IMAGE=mapleque/${CI_PROJECT_PATH}:stage-${CI_PIPELINE_ID} # 测试环境</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">export</span> <span class="string">DEPLOY_CONFIG_IMAGE=mapleque/${CI_PROJECT_PATH}:${CI_COMMIT_REF_NAME}</span> <span class="comment"># 正式环境</span></span><br><span class="line"> <span class="comment">#</span></span><br><span class="line"> <span class="comment"># 项目环境变量</span></span><br><span class="line"> <span class="comment"># 这里定义的环境变量需要使用DEPLOY_ENV_前缀</span></span><br><span class="line"> <span class="comment"># 注意：该前缀会在载入时被去掉，所以在服务中直接使用不要带该前缀</span></span><br><span class="line"> <span class="comment"># 例如： - export DEPLOY_ENV_XXX=xxx 在服务中直接使用${XXX}即可</span></span><br><span class="line"> <span class="comment"># - export DEPLOY_ENV_VALUE1=value1</span></span><br><span class="line"> <span class="comment"># - export DEPLOY_ENV_VALUE2=value2</span></span><br><span class="line"> <span class="comment">#</span></span><br><span class="line"> <span class="comment"># 项目端口</span></span><br><span class="line"> <span class="comment"># 项目端口需要使用DEPLOY_PORT_前缀</span></span><br><span class="line"> <span class="comment"># 注意：项目端口也会被导入为环境变量，且前缀会在载入时被去掉，所以不要与DEPLOY_ENV_所设置的环境变量名冲突</span></span><br><span class="line"> <span class="comment"># 例如： - export DEPLOY_PORT_XXX= 在服务中直接使用${XXX}即可</span></span><br><span class="line"> <span class="comment"># - export DEPLOY_PORT_PORT1= #这里不要填任何值</span></span><br><span class="line"> <span class="comment"># - export DEPLOY_PORT_PORT2= #这里不要填任何值</span></span><br><span class="line"> <span class="comment"># 注意：这里不要自行指定端口号，项目部署时会自动置顶端口号并设置到对应的环境变量中</span></span><br><span class="line"> <span class="comment"># 例如上面两个端口的设置，在项目中直接使用${PORT1}和${PORT2}就能获得系统分配的端口号</span></span><br><span class="line"> <span class="comment">#</span></span><br><span class="line"> <span class="comment"># 上线命令，不要修改</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">deploy</span></span><br></pre></td></tr></table></figure>]]></content>
<categories>
<category> 工具 </category>
<category> Gitlab </category>
</categories>
<tags>
<tag> Gitlab </tag>
</tags>
</entry>
<entry>
<title>使用ELK管理日志</title>
<link href="/posts/tool/elk/elk-logger/"/>
<url>/posts/tool/elk/elk-logger/</url>
<content type="html"><![CDATA[<pre class="mermaid" style="text-align: center;"> graph LR Filebeat1 --> LogstashFilebeat2 --> LogstashFilebeatn --> LogstashLogstash --> ElasticsearchElasticsearch --> Kibana </pre><h1 id="logstash"><a class="markdownIt-Anchor" href="#logstash"></a> Logstash</h1><p><a href="https://www.elastic.co/guide/en/logstash/current/index.html" target="_blank" rel="noopener">官方文档</a></p><p>Logstash是一个数据收集服务，Java编写，内存需求量较大，处理复杂日志切分时CPU占用较高。整体来说性能一般，如果在数据收集方面有特殊的需求，建议自研。</p><p>这里介绍几种输入输出配置：</p><h2 id="jdbc-input"><a class="markdownIt-Anchor" href="#jdbc-input"></a> jdbc-input</h2><figure class="highlight plain"><figcaption><span>/etc/logstash/conf.d/jdbc-input.conf</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">input {</span><br><span class="line"> jdbc {</span><br><span class="line"> id => "da_label_user_transaction"</span><br><span class="line"> jdbc_driver_library => "/usr/share/java/mysql-connector-java-5.1.45-bin.jar"</span><br><span class="line"> jdbc_driver_class => "com.mysql.jdbc.Driver"</span><br><span class="line"> jdbc_connection_string => "jdbc:mysql://host:port/dbname"</span><br><span class="line"> jdbc_user => "dbuser"</span><br><span class="line"> jdbc_password => "dbpassword"</span><br><span class="line"> jdbc_fetch_size => 20000</span><br><span class="line"> jdbc_default_timezone => "Asia/Shanghai"</span><br><span class="line"> schedule => "* * * * *"</span><br><span class="line"> use_column_value => true</span><br><span class="line"> tracking_column => "id"</span><br><span class="line"> statement => "SELECT * from table where id > :sql_last_value LIMIT 10000"</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="filebeat-input"><a class="markdownIt-Anchor" href="#filebeat-input"></a> filebeat-input</h2><figure class="highlight plain"><figcaption><span>/etc/logstash/conf.d/filebeat-input.conf</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">input {</span><br><span class="line"> beats {</span><br><span class="line"> port => "5044"</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="file-output"><a class="markdownIt-Anchor" href="#file-output"></a> file-output</h2><figure class="highlight plain"><figcaption><span>/etc/logstash/conf.d/log-file-output.conf</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">output {</span><br><span class="line"> file {</span><br><span class="line"> path => "/tmp/logstash-out.log"</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="hdfs-output"><a class="markdownIt-Anchor" href="#hdfs-output"></a> hdfs-output</h2><figure class="highlight plain"><figcaption><span>/etc/logstash/conf.d/hdfs-output.conf</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">output {</span><br><span class="line"> webhdfs {</span><br><span class="line"> host => "hdfs.mapleque.com"</span><br><span class="line"> port => 9870</span><br><span class="line"> path => "/user/logstash/dt=%{+YYYY-MM-dd}/%{+HH}.log</span><br><span class="line"> user => "hadoop"</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="elasticsearch-output"><a class="markdownIt-Anchor" href="#elasticsearch-output"></a> elasticsearch-output</h2><figure class="highlight plain"><figcaption><span>/etc/logstash/conf.d/elasticsearch-output.conf</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">output {</span><br><span class="line"> elasticsearch {</span><br><span class="line"> hosts => ["es.mapleque.com:9200"]</span><br><span class="line"> index => "your index name"</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h1 id="filebeat"><a class="markdownIt-Anchor" href="#filebeat"></a> Filebeat</h1><p><a href="https://www.elastic.co/guide/en/beats/filebeat/current/index.html" target="_blank" rel="noopener">官方文档</a></p><p>Filebeat主要用于从应用机器同步日志到日志服务器。</p><p>常用配置如下：</p><figure class="highlight yaml"><figcaption><span>/etc/filebeat/filebeat.yml</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="string">filebeat.prospectors:</span></span><br><span class="line"><span class="comment"># 根据情况自行配置需要收集哪些log</span></span><br><span class="line"><span class="attr">- type:</span> <span class="string">log</span></span><br><span class="line"><span class="attr"> paths:</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string"><log_path></span></span><br><span class="line"></span><br><span class="line"><span class="string">output.kafka:</span></span><br><span class="line"><span class="attr"> enable:</span> <span class="literal">true</span></span><br><span class="line"><span class="attr"> hosts:</span> <span class="string">['<kafka_ip>']</span></span><br><span class="line"></span><br><span class="line"><span class="attr"> topic:</span> <span class="string">'<string>'</span></span><br><span class="line"><span class="attr"> compression:</span> <span class="string">gzip</span></span><br><span class="line"><span class="attr"> max_message_bytes:</span> <span class="number">1000000</span></span><br><span class="line"></span><br><span class="line"><span class="string">queue.mem:</span></span><br><span class="line"><span class="attr"> events:</span> <span class="number">512</span></span><br><span class="line"> <span class="string">flush.min_events:</span> <span class="number">256</span></span><br><span class="line"> <span class="string">flush.timeout:</span> <span class="number">5</span><span class="string">s</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 监控可不配置</span></span><br><span class="line"><span class="string">xpack.monitoring:</span></span><br><span class="line"><span class="attr"> enabled:</span> <span class="literal">true</span></span><br><span class="line"><span class="attr"> elasticsearch:</span></span><br><span class="line"><span class="attr"> hosts:</span> <span class="string">["<es_ip>"]</span></span><br><span class="line"><span class="attr"> username:</span> <span class="string"><string></span></span><br><span class="line"><span class="attr"> password:</span> <span class="string">"<string>"</span></span><br></pre></td></tr></table></figure><h1 id="elasticsearch"><a class="markdownIt-Anchor" href="#elasticsearch"></a> Elasticsearch</h1><p><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html" target="_blank" rel="noopener">官方文档</a></p><h1 id="kibana"><a class="markdownIt-Anchor" href="#kibana"></a> Kibana</h1><p><a href="https://www.elastic.co/guide/en/kibana/current/index.html" target="_blank" rel="noopener">官方文档</a></p>]]></content>
<categories>
<category> 工具 </category>
<category> ELK </category>
</categories>
<tags>
<tag> ELK </tag>
</tags>
</entry>
<entry>
<title>使用Elasticsearch实现社区内容搜索服务</title>
<link href="/posts/tool/elk/es-search/"/>
<url>/posts/tool/elk/es-search/</url>
<content type="html"><![CDATA[<blockquote><p>Elasticsearch 是一个分布式、RESTful 风格的搜索和数据分析引擎，能够解决不断涌现出的各种用例。 作为 Elastic Stack 的核心，它集中存储您的数据，帮助您发现意料之中以及意料之外的情况。</p><footer><strong>Elastic Elasticsearch</strong><cite><a href="https://www.elastic.co/cn/products/elasticsearch" target="_blank" rel="noopener">www.elastic.co/cn/products/elasticsearch</a></cite></footer></blockquote><h1 id="需求分析"><a class="markdownIt-Anchor" href="#需求分析"></a> 需求分析</h1><p>考虑社区内容（帖子）搜索需要实现的功能有：</p><ul><li>支持中文</li><li>根据用户输入，给出搜索建议</li><li>根据搜索词，给出相关内容搜索结果</li><li>搜索结果应按照相关性由大到小排序</li><li>在搜索结果中，应当优先展示近期发布的内容</li><li>在搜索结果中，应当优先展示质量较高的内容</li><li>可以屏蔽一些搜索词，让其不能出结果</li><li>可以屏蔽一些内容，让其永远不能出现在结果中</li><li>能够简单识别网络爬虫行为，并进行封禁</li></ul><p>基本产品形态为：</p><ul><li>有个搜索框，用户可以输入任何字符</li><li>用户在搜索框输入字符的时候根据已输入的内容，给出相关建议</li><li>用户可以确认搜索所输入的内容或建议的内容</li><li>有个结果展示区，给用户有序展示搜索结果，并支持翻页</li><li>有个后台能够添加屏蔽搜索词</li><li>有个后台能够添加屏蔽的帖子</li><li>有个后台能够添加屏蔽的ip</li><li>有个后台能够查看各时间接口请求次数（能够按照ip，搜索词分组统计）</li></ul><h1 id="系统设计"><a class="markdownIt-Anchor" href="#系统设计"></a> 系统设计</h1><p>根据上面的需求，前台服务需要实现的接口有：</p><ul><li>获取搜索建议 – 输入搜索词和要获取数据条数n，返回有序的前n个搜索建议词</li><li>获取搜索结果 – 输入搜索词、页码n和每页容量m，返回有序的第n页的m条搜索结果</li></ul><p>后台服务需要实现的接口有：</p><ul><li>屏蔽词的增删改查</li><li>屏蔽帖子的增删改查</li><li>屏蔽ip的增删改查</li><li>接口请求次数的统计（按时间区间分组，按ip分组，按搜索词分组）</li></ul><h1 id="实现方案"><a class="markdownIt-Anchor" href="#实现方案"></a> 实现方案</h1><p>基于上面的系统设计，显然这里只需要提供前台接口实现方案，就可以完成全部产品需求。</p><p>整体来看，前台的两个接口都是一个搜索过程，只是搜索的目标数据有所区分：</p><ul><li>搜索建议需要在<strong>搜索建议数据</strong>中搜索</li><li>搜索结果需要在<strong>社区内容数据</strong>中搜索</li></ul><p>其中，搜索结果所需要的数据很明确，就是社区内容（帖子标题和正文），而搜索建议所需要的数据，则需要根据产品需求另行设计。</p><p>这里，可以考虑到的能够用于搜索建议的数据有：</p><ul><li>社区内容中的一些关键词</li><li>用户搜索次数较多的搜索词</li><li>我们希望用户搜索的搜索词</li></ul><p>当用于搜索的数据明确后，只需要在数据上建立索引，即可通过Elasticsearch提供的Api进行搜索了。</p><p>因此，这里我们需要先解决如何建立中文索引的问题，再考虑如何在两份数据上分别建立索引以支持搜索。</p><h2 id="中文分析器"><a class="markdownIt-Anchor" href="#中文分析器"></a> 中文分析器</h2><p>处理中文搜索，主要是解决中文分词的问题。Elasticsearch可以安装扩展分析器来支持多语言。</p><p>这里笔者基于一个支持中文的分析器<a href="https://github.com/medcl/elasticsearch-analysis-ik" target="_blank" rel="noopener">elasticsearch-analysis-ik</a>进行了配置扩展，得到了一个支持中文全文索引的是分析器fulltext_analyzer。</p><p>更进一步，由于当前主流中文输入法都是通过拼音完成输入，这里如果能考虑针对中文生成对应的拼音索引就更加完美了。</p><p>笔者在找到上面的分析器的时候，同时发现了一直个支持拼音索引的分析器<a href="https://github.com/medcl/elasticsearch-analysis-pinyin" target="_blank" rel="noopener">elasticsearch-analysis-pinyin</a>，于是也一并进行了扩展配置得到pinyin_analyzer，以便后续使用。</p><p>上述两个分词器配置方式如下：</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><span class="line">{</span><br><span class="line"> <span class="attr">"settings"</span>: {</span><br><span class="line"> <span class="attr">"analysis"</span>:{</span><br><span class="line"> <span class="attr">"analyzer"</span>:{</span><br><span class="line"> <span class="attr">"pinyin_analyzer"</span>:{</span><br><span class="line"> <span class="attr">"tokenizer"</span>:<span class="string">"pinyin_tokenizer"</span>,</span><br><span class="line"> <span class="attr">"char_filter"</span>: [<span class="string">"html_strip"</span>,<span class="string">"white_strip"</span>],</span><br><span class="line"> <span class="attr">"filter"</span>:[<span class="string">"asciifolding"</span>]</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"fulltext_analyzer"</span>:{</span><br><span class="line"> <span class="attr">"tokenizer"</span>:<span class="string">"ik_max_word"</span> ,</span><br><span class="line"> <span class="attr">"char_filter"</span>: [<span class="string">"html_strip"</span>,<span class="string">"white_strip"</span>],</span><br><span class="line"> <span class="attr">"filter"</span>:[<span class="string">"asciifolding"</span>]</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"tokenizer"</span>:{</span><br><span class="line"> <span class="attr">"pinyin_tokenizer"</span>:{</span><br><span class="line"> <span class="attr">"type"</span>:<span class="string">"pinyin"</span>,</span><br><span class="line"> <span class="attr">"keep_separate_first_letter"</span> : <span class="literal">false</span>,</span><br><span class="line"> <span class="attr">"keep_full_pinyin"</span> : <span class="literal">true</span>,</span><br><span class="line"> <span class="attr">"keep_original"</span> : <span class="literal">true</span>,</span><br><span class="line"> <span class="attr">"limit_first_letter_length"</span> : <span class="number">16</span>,</span><br><span class="line"> <span class="attr">"lowercase"</span> : <span class="literal">true</span>,</span><br><span class="line"> <span class="attr">"remove_duplicated_term"</span> : <span class="literal">true</span></span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"char_filter"</span>: {</span><br><span class="line"> <span class="attr">"white_strip"</span>: {</span><br><span class="line"> <span class="attr">"type"</span>: <span class="string">"pattern_replace"</span>,</span><br><span class="line"> <span class="attr">"pattern"</span>: <span class="string">"[\\r|\\n|\\t|\\s|[^\\u0020-\\uFFFF]|\\u007F]+"</span>,</span><br><span class="line"> <span class="attr">"replacement"</span>: <span class="string">""</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><div class="note info"> <h3 id="fulltext_analyzer"><a class="markdownIt-Anchor" href="#fulltext_analyzer"></a> fulltext_analyzer</h3><p>该分析器对于待处理文本，首先会进行字符过滤，过滤掉所有html标签、不可见字符、emoji等。</p><p>然后对所得文本进行分词，分词使用了最细粒度拆分原则。<br>例如：“中华人民共和国国歌”将被拆分为“中华人民共和国,中华人民,中华,华人,人民共和国,人民,人,民,共和国,共和,和,国国,国歌”。</p><p>最后将分词结果用于后续处理。</p> </div><div class="note info"> <h3 id="pinyin_analyzer"><a class="markdownIt-Anchor" href="#pinyin_analyzer"></a> pinyin_analyzer</h3><p>该分析器对于待处理文本，同样会进行字符过滤，处理方式同上。</p><p>然后将所得文本转化为拼音，拼音只进行单字分词，同时会增加首字母组合。<br>例如：“中华人民中共和国国歌”将被拆分为“zhong,hua,ren,min,gong,he,guo,guo,ge,zhrmghggg”。</p><p>最后将分词结果用于后续处理。</p> </div><h2 id="搜索建议数据索引及搜索"><a class="markdownIt-Anchor" href="#搜索建议数据索引及搜索"></a> 搜索建议数据索引及搜索</h2><h3 id="数据来源及同步"><a class="markdownIt-Anchor" href="#数据来源及同步"></a> 数据来源及同步</h3><p>根据上面的分析，搜索建议数据可以有多个来源，这里需要做到能够从不同的来源将数据收集到Elasticsearch并保持同步。</p><p>下面分别考虑上面提到的三个来源。</p><h4 id="社区内容中的一些关键词"><a class="markdownIt-Anchor" href="#社区内容中的一些关键词"></a> 社区内容中的一些关键词</h4><p>由于社区业务并没有考虑这部分数据的生成，因此我们需要从零开始生成这些数据。<br>这里笔者拍脑袋设计了社区内容关键词生成方案：</p><ul><li>对内容全文进行ngram中文分词（实际应用中使用了3gram）</li><li>筛选词频超过阈值的词进入搜索建议词库（实际应用中阈值取3）</li><li>入库关键词记录词频数作为后续搜索权重</li><li>如果当前关键词已入库，直接把词频数相加更新</li></ul><p>3gram分词器配置如下：</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line">{</span><br><span class="line"> <span class="attr">"settings"</span>:{</span><br><span class="line"> <span class="attr">"analysis"</span>: {</span><br><span class="line"> <span class="attr">"analyzer"</span>: {</span><br><span class="line"> <span class="attr">"phrase_3_gram"</span>: {</span><br><span class="line"> <span class="attr">"tokenizer"</span>:<span class="string">"ik_smart"</span>,</span><br><span class="line"> <span class="attr">"char_filter"</span>: [<span class="string">"html_strip"</span>,<span class="string">"white_strip"</span>],</span><br><span class="line"> <span class="attr">"filter"</span>:[<span class="string">"3_gram"</span>]</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"char_filter"</span>: {</span><br><span class="line"> <span class="attr">"white_strip"</span>: {</span><br><span class="line"> <span class="attr">"type"</span>: <span class="string">"pattern_replace"</span>,</span><br><span class="line"> <span class="attr">"pattern"</span>: <span class="string">"[\\r|\\n|\\t|\\s|[^\\u0020-\\uFFFF]|\\u007F]+"</span>,</span><br><span class="line"> <span class="attr">"replacement"</span>: <span class="string">""</span></span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"filter"</span>: {</span><br><span class="line"> <span class="attr">"3_gram"</span>: {</span><br><span class="line"> <span class="attr">"type"</span>: <span class="string">"shingle"</span>,</span><br><span class="line"> <span class="attr">"min_shingle_size"</span>: <span class="number">1</span>,</span><br><span class="line"> <span class="attr">"max_shingle_size"</span>: <span class="number">3</span>,</span><br><span class="line"> <span class="attr">"token_separator"</span>:<span class="string">" "</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h4 id="用户搜索次数较多的搜索词"><a class="markdownIt-Anchor" href="#用户搜索次数较多的搜索词"></a> 用户搜索次数较多的搜索词</h4><p>当服务上线后，可以从接口请求日志中获取用户搜索词统计。统计周期可以根据实际情况调整。</p><p>给定阈值n，在统计周期内搜索次数超过n的搜索词进入搜索建议词库，搜索次数可以作为词频累加。</p><h4 id="人工运营的搜索词"><a class="markdownIt-Anchor" href="#人工运营的搜索词"></a> 人工运营的搜索词</h4><p>人工运营搜索建议词，可以通过后台直接添加到搜索建议词库。</p><p>为了简化搜索逻辑，可以通过调整词频大小来控制其出现的位置。</p><h3 id="索引和搜索"><a class="markdownIt-Anchor" href="#索引和搜索"></a> 索引和搜索</h3><p>通过上面的数据同步流程，我们可以获得一个实时更新的搜索建议词库。<br>下面只需要拿用户输入的搜索词在Elasticsearch进行搜索即可获得结果。<br>为了得到想要的效果，我们对搜索行为加以约束：</p><ul><li>不分词全文前缀匹配搜索</li><li>不分词拼音前缀匹配搜索</li><li>分词全文前缀匹配搜索</li><li>分词拼音前缀匹配搜索</li><li>分词全文模糊搜索</li><li>考虑词频权重对搜索结果排序的影响</li></ul><p>mapping配置如下：</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line">{</span><br><span class="line"> <span class="attr">"mappings"</span>: {</span><br><span class="line"> <span class="attr">"_doc"</span>: {</span><br><span class="line"> <span class="attr">"properties"</span>: {</span><br><span class="line"> <span class="attr">"keyword"</span>: {</span><br><span class="line"> <span class="attr">"type"</span>: <span class="string">"keyword"</span>,</span><br><span class="line"> <span class="attr">"fields"</span>: {</span><br><span class="line"> <span class="attr">"raw"</span>: {</span><br><span class="line"> <span class="attr">"type"</span>: <span class="string">"text"</span>,</span><br><span class="line"> <span class="attr">"analyzer"</span>: <span class="string">"whitespace"</span>,</span><br><span class="line"> <span class="attr">"search_analyzer"</span>: <span class="string">"whitespace"</span></span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"fulltext"</span>: {</span><br><span class="line"> <span class="attr">"type"</span>: <span class="string">"text"</span>,</span><br><span class="line"> <span class="attr">"analyzer"</span>: <span class="string">"fulltext_analyzer"</span>,</span><br><span class="line"> <span class="attr">"search_analyzer"</span>: <span class="string">"fulltext_analyzer"</span></span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"pinyin"</span>: {</span><br><span class="line"> <span class="attr">"type"</span>: <span class="string">"text"</span>,</span><br><span class="line"> <span class="attr">"store"</span>: <span class="literal">false</span>,</span><br><span class="line"> <span class="attr">"term_vector"</span>: <span class="string">"with_offsets"</span>,</span><br><span class="line"> <span class="attr">"analyzer"</span>: <span class="string">"pinyin_analyzer"</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>搜索模板配置如下：</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br></pre></td><td class="code"><pre><span class="line">{</span><br><span class="line"> <span class="attr">"script"</span>: {</span><br><span class="line"> <span class="attr">"lang"</span>: <span class="string">"mustache"</span>,</span><br><span class="line"> <span class="attr">"source"</span>: {</span><br><span class="line"> <span class="attr">"size"</span>: <span class="string">"{{size}}"</span>,</span><br><span class="line"> <span class="attr">"query"</span>: {</span><br><span class="line"> <span class="attr">"function_score"</span>: {</span><br><span class="line"> <span class="attr">"functions"</span>: [</span><br><span class="line"> {</span><br><span class="line"> <span class="attr">"field_value_factor"</span>: {</span><br><span class="line"> <span class="attr">"field"</span>:<span class="string">"word_count"</span>,</span><br><span class="line"> <span class="attr">"factor"</span>:<span class="number">1</span>,</span><br><span class="line"> <span class="attr">"modifier"</span>:<span class="string">"linear"</span>,</span><br><span class="line"> <span class="attr">"missing"</span>:<span class="number">0</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> ],</span><br><span class="line"> <span class="attr">"score_mode"</span>:<span class="string">"multiply"</span>,</span><br><span class="line"> <span class="attr">"query"</span>: {</span><br><span class="line"> <span class="attr">"bool"</span>: {</span><br><span class="line"> <span class="attr">"should"</span>: [</span><br><span class="line"> {<span class="attr">"prefix"</span>: {<span class="attr">"keyword.raw"</span>: { <span class="attr">"value"</span>: <span class="string">"{{query}}"</span>, <span class="attr">"boost"</span>: <span class="number">100</span>}}},</span><br><span class="line"> {<span class="attr">"prefix"</span>: {<span class="attr">"keyword.pinyin"</span>: { <span class="attr">"value"</span>: <span class="string">"{{query}}"</span>, <span class="attr">"boost"</span>: <span class="number">50</span>}}},</span><br><span class="line"> {<span class="attr">"match_phrase_prefix"</span>: {<span class="attr">"keyword.fulltext"</span>: {<span class="attr">"query"</span>: <span class="string">"{{query}}"</span>, <span class="attr">"boost"</span>: <span class="number">10</span>}}},</span><br><span class="line"> {<span class="attr">"match_phrase_prefix"</span>: {<span class="attr">"keyword.pinyin"</span>: {<span class="attr">"query"</span>: <span class="string">"{{query}}"</span>, <span class="attr">"boost"</span>: <span class="number">5</span>}}},</span><br><span class="line"> {</span><br><span class="line"> <span class="attr">"match"</span>: {</span><br><span class="line"> <span class="attr">"keyword.fulltext"</span>: {</span><br><span class="line"> <span class="attr">"query"</span>: <span class="string">"{{query}}"</span>,</span><br><span class="line"> <span class="attr">"boost"</span>: <span class="number">5</span>,</span><br><span class="line"> <span class="attr">"fuzziness"</span>: <span class="string">"AUTO"</span>,</span><br><span class="line"> <span class="attr">"max_expansions"</span>: <span class="number">10</span>,</span><br><span class="line"> <span class="attr">"prefix_length"</span>: <span class="number">2</span>,</span><br><span class="line"> <span class="attr">"fuzzy_transpositions"</span>: <span class="literal">true</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"_source"</span>:[</span><br><span class="line"> <span class="string">"keyword"</span></span><br><span class="line"> ],</span><br><span class="line"> <span class="attr">"highlight"</span>: {</span><br><span class="line"> <span class="attr">"pre_tags"</span> : [<span class="string">"<em>"</span>],</span><br><span class="line"> <span class="attr">"post_tags"</span> : [<span class="string">"</em>"</span>],</span><br><span class="line"> <span class="attr">"fields"</span> : {</span><br><span class="line"> <span class="attr">"keyword.fulltext"</span> : {}</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"collapse"</span>: {</span><br><span class="line"> <span class="attr">"field"</span>: <span class="string">"keyword"</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="搜索结果数据索引及搜索"><a class="markdownIt-Anchor" href="#搜索结果数据索引及搜索"></a> 搜索结果数据索引及搜索</h2><h3 id="数据来源及同步-2"><a class="markdownIt-Anchor" href="#数据来源及同步-2"></a> 数据来源及同步</h3><p>社区内容数据，需要从业务数据同步，同步频率由业务需求决定。</p><p>这里需要详细说明那些影响搜索结果的字段：</p><div class="note info"> <h4 id="帖子标题"><a class="markdownIt-Anchor" href="#帖子标题"></a> 帖子标题</h4><p>帖子标题直接影响搜索内容相似度，且权重较高，标题与搜索词越相似的帖子在搜索结果中排序应该越靠前。</p> </div><div class="note info"> <h4 id="帖子正文"><a class="markdownIt-Anchor" href="#帖子正文"></a> 帖子正文</h4><p>帖子正文对搜索内容相似度的影响仅次于帖子标题。</p> </div><div class="note info"> <h4 id="帖子评分"><a class="markdownIt-Anchor" href="#帖子评分"></a> 帖子评分</h4><p>帖子评分是一个综合描述帖子质量的属性，评分越高代表帖子质量越高，根据产品需求它的排序应该靠前。</p> </div><div class="note info"> <h4 id="帖子创建时间"><a class="markdownIt-Anchor" href="#帖子创建时间"></a> 帖子创建时间</h4><p>帖子创建时间是一个描述时效性的属性，根据产品需求，创建时间越近排序应该越靠前。</p> </div><div class="note info"> <h4 id="帖子状态"><a class="markdownIt-Anchor" href="#帖子状态"></a> 帖子状态</h4><p>用于屏蔽帖子，使其不出现在搜索结果中。</p> </div><h3 id="索引和搜索-2"><a class="markdownIt-Anchor" href="#索引和搜索-2"></a> 索引和搜索</h3><p>根据上面的思路，创建索引如下：</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br></pre></td><td class="code"><pre><span class="line">{</span><br><span class="line"> <span class="attr">"mappings"</span>: {</span><br><span class="line"> <span class="attr">"_doc"</span>: {</span><br><span class="line"> <span class="attr">"properties"</span>:{</span><br><span class="line"> <span class="attr">"title"</span>:{</span><br><span class="line"> <span class="attr">"type"</span>:<span class="string">"keyword"</span>,</span><br><span class="line"> <span class="attr">"fields"</span>: {</span><br><span class="line"> <span class="attr">"raw"</span>:{</span><br><span class="line"> <span class="attr">"type"</span>:<span class="string">"text"</span>,</span><br><span class="line"> <span class="attr">"analyzer"</span>: <span class="string">"whitespace"</span>,</span><br><span class="line"> <span class="attr">"search_analyzer"</span>: <span class="string">"whitespace"</span></span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"fulltext"</span>: {</span><br><span class="line"> <span class="attr">"type"</span>:<span class="string">"text"</span>,</span><br><span class="line"> <span class="attr">"analyzer"</span>:<span class="string">"fulltext_analyzer"</span>,</span><br><span class="line"> <span class="attr">"search_analyzer"</span>:<span class="string">"fulltext_analyzer"</span></span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"pinyin"</span>: {</span><br><span class="line"> <span class="attr">"type"</span>: <span class="string">"text"</span>,</span><br><span class="line"> <span class="attr">"store"</span>: <span class="literal">false</span>,</span><br><span class="line"> <span class="attr">"term_vector"</span>: <span class="string">"with_offsets"</span>,</span><br><span class="line"> <span class="attr">"analyzer"</span>: <span class="string">"pinyin_analyzer"</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"content"</span>: {</span><br><span class="line"> <span class="attr">"type"</span>:<span class="string">"text"</span>,</span><br><span class="line"> <span class="attr">"fields"</span>: {</span><br><span class="line"> <span class="attr">"fulltext"</span>: {</span><br><span class="line"> <span class="attr">"type"</span>:<span class="string">"text"</span>,</span><br><span class="line"> <span class="attr">"analyzer"</span>:<span class="string">"fulltext_analyzer"</span>,</span><br><span class="line"> <span class="attr">"search_analyzer"</span>:<span class="string">"fulltext_analyzer"</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>搜索模板如下：</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br></pre></td><td class="code"><pre><span class="line">{</span><br><span class="line"> <span class="attr">"script"</span>: {</span><br><span class="line"> <span class="attr">"lang"</span>: <span class="string">"mustache"</span>,</span><br><span class="line"> <span class="attr">"source"</span>: {</span><br><span class="line"> <span class="attr">"size"</span>: <span class="string">"{{size}}"</span>,</span><br><span class="line"> <span class="attr">"from"</span>: <span class="string">"{{from}}"</span>,</span><br><span class="line"> <span class="attr">"query"</span>: {</span><br><span class="line"> <span class="attr">"function_score"</span>: {</span><br><span class="line"> <span class="attr">"functions"</span>: [</span><br><span class="line"> {</span><br><span class="line"> <span class="attr">"script_score"</span>: {</span><br><span class="line"> <span class="attr">"script"</span>: {</span><br><span class="line"> <span class="attr">"source"</span>: <span class="string">"2+(Math.pow(0.5,Math.pow((90-(System.currentTimeMillis() - doc['creation_time'][0].getMillis())/(1000*86400))/(double)1460,2))-1)/(1+Math.pow(Math.E,(90-(System.currentTimeMillis() - doc['creation_time'][0].getMillis())/(1000*86400))*10))"</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> <span class="attr">"field_value_factor"</span>: {</span><br><span class="line"> <span class="attr">"field"</span>:<span class="string">"score"</span>,</span><br><span class="line"> <span class="attr">"factor"</span>:<span class="number">0.0002</span>,</span><br><span class="line"> <span class="attr">"modifier"</span>:<span class="string">"ln2p"</span>,</span><br><span class="line"> <span class="attr">"missing"</span>:<span class="number">0</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> ],</span><br><span class="line"> <span class="attr">"score_mode"</span>:<span class="string">"multiply"</span>,</span><br><span class="line"> <span class="attr">"query"</span>: {</span><br><span class="line"> <span class="attr">"bool"</span>: {</span><br><span class="line"> <span class="attr">"must"</span>: [</span><br><span class="line"> {</span><br><span class="line"> <span class="attr">"term"</span>: {<span class="attr">"status"</span>: <span class="number">1</span>}</span><br><span class="line"> }</span><br><span class="line"> ],</span><br><span class="line"> <span class="attr">"should"</span>: [</span><br><span class="line"> {<span class="attr">"prefix"</span>: {<span class="attr">"title.raw"</span>: { <span class="attr">"value"</span>: <span class="string">"{{query}}"</span>, <span class="attr">"boost"</span>: <span class="number">100</span>}}},</span><br><span class="line"> {<span class="attr">"prefix"</span>: {<span class="attr">"title.pinyin"</span>: { <span class="attr">"value"</span>: <span class="string">"{{query}}"</span>, <span class="attr">"boost"</span>: <span class="number">10</span>}}},</span><br><span class="line"> {<span class="attr">"match_phrase_prefix"</span>: {<span class="attr">"title.fulltext"</span>: {<span class="attr">"query"</span>: <span class="string">"{{query}}"</span>, <span class="attr">"boost"</span>: <span class="number">50</span>}}},</span><br><span class="line"> {<span class="attr">"match_phrase_prefix"</span>: {<span class="attr">"title.pinyin"</span>: {<span class="attr">"query"</span>: <span class="string">"{{query}}"</span>, <span class="attr">"boost"</span>: <span class="number">5</span>}}},</span><br><span class="line"> {<span class="attr">"match_phrase_prefix"</span>: {<span class="attr">"content.pinyin"</span>: {<span class="attr">"query"</span>: <span class="string">"{{query}}"</span>, <span class="attr">"boost"</span>: <span class="number">10</span>}}},</span><br><span class="line"> {</span><br><span class="line"> <span class="attr">"match"</span>: {</span><br><span class="line"> <span class="attr">"title.fulltext"</span>: {</span><br><span class="line"> <span class="attr">"query"</span>: <span class="string">"{{query}}"</span>,</span><br><span class="line"> <span class="attr">"boost"</span>: <span class="number">5</span>,</span><br><span class="line"> <span class="attr">"fuzziness"</span>: <span class="string">"AUTO"</span>,</span><br><span class="line"> <span class="attr">"max_expansions"</span>: <span class="number">10</span>,</span><br><span class="line"> <span class="attr">"prefix_length"</span>: <span class="number">2</span>,</span><br><span class="line"> <span class="attr">"fuzzy_transpositions"</span>: <span class="literal">true</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> <span class="attr">"match"</span>: {</span><br><span class="line"> <span class="attr">"content.fulltext"</span>: {</span><br><span class="line"> <span class="attr">"query"</span>: <span class="string">"{{query}}"</span>,</span><br><span class="line"> <span class="attr">"boost"</span>: <span class="number">5</span>,</span><br><span class="line"> <span class="attr">"fuzziness"</span>: <span class="string">"AUTO"</span>,</span><br><span class="line"> <span class="attr">"max_expansions"</span>: <span class="number">10</span>,</span><br><span class="line"> <span class="attr">"prefix_length"</span>: <span class="number">2</span>,</span><br><span class="line"> <span class="attr">"fuzzy_transpositions"</span>: <span class="literal">true</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> {</span><br><span class="line"> <span class="attr">"query_string"</span>: {</span><br><span class="line"> <span class="attr">"fields"</span>: [<span class="string">"title.fulltext"</span>,<span class="string">"title.pinyin"</span>,<span class="string">"content.fulltext"</span>],</span><br><span class="line"> <span class="attr">"query"</span>: <span class="string">"{{query}}"</span>,</span><br><span class="line"> <span class="attr">"boost"</span>: <span class="number">1</span>,</span><br><span class="line"> <span class="attr">"fuzziness"</span>: <span class="string">"AUTO"</span>,</span><br><span class="line"> <span class="attr">"fuzzy_prefix_length"</span>: <span class="number">2</span>,</span><br><span class="line"> <span class="attr">"fuzzy_max_expansions"</span>: <span class="number">10</span>,</span><br><span class="line"> <span class="attr">"fuzzy_transpositions"</span>: <span class="literal">true</span>,</span><br><span class="line"> <span class="attr">"allow_leading_wildcard"</span>: <span class="literal">false</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"_source"</span>:[</span><br><span class="line"> <span class="string">"post_id"</span>,</span><br><span class="line"> <span class="string">"creation_time"</span>,</span><br><span class="line"> <span class="string">"score"</span>,</span><br><span class="line"> <span class="string">"title"</span>,</span><br><span class="line"> <span class="string">"content"</span></span><br><span class="line"> ],</span><br><span class="line"> <span class="attr">"highlight"</span>: {</span><br><span class="line"> <span class="attr">"pre_tags"</span> : [<span class="string">"<em>"</span>],</span><br><span class="line"> <span class="attr">"post_tags"</span> : [<span class="string">"</em>"</span>],</span><br><span class="line"> <span class="attr">"fields"</span> : {</span><br><span class="line"> <span class="attr">"title.fulltext"</span> : {},</span><br><span class="line"> <span class="attr">"content.fulltext"</span>: {}</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> <span class="attr">"collapse"</span>: {</span><br><span class="line"> <span class="attr">"field"</span>: <span class="string">"post_id"</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>其中，时间衰减函数实现思路如下：</p><div class="note info"> <h4 id="时间衰减函数"><a class="markdownIt-Anchor" href="#时间衰减函数"></a> 时间衰减函数</h4><p>这里我们以高斯函数：<br><code>f(x)=a*e^(-(x-b)^2/(2*c^2)))</code></p><p>为基础，辅以激活函数：<br><code>f(x)=1/(1+e^(10*(b-x)))</code></p><p>生成时间权重衰减函数：<br><code>f(x) = a*(1+(e^(ln(d/a)*(x-b)^2/c^2)-1)/(1+e^(10*(b-x))))</code></p><p>上面的函数可以简化为：<br><code>f(x) = a(1+((d/a)^((b-x)/c)^2)-1)/(e^10(b-x)+1)</code></p><p>其中：初始权重为a，从b天开始衰减，在第c天权重衰减到d。</p><p>假设这里取：</p><ul><li>a=1 表示权重范围从1衰减到0</li><li>b=90 表示90天之内的帖子不降权</li><li>c=1460，d=0.5 表示4年之后，帖子权重衰减50%</li></ul><p>那么得到函数：<br><code>f(x) = 1+(0.5^(((x-90)/1460)^2)-1)/(1+e^(10*(90-x)))</code></p><p>通过该函数图像可以看出：</p><ul><li>半年左右权重开始衰减</li><li>4年衰减到50%</li><li>7年多之后衰减到10%以下</li></ul> </div><h2 id="人工干预"><a class="markdownIt-Anchor" href="#人工干预"></a> 人工干预</h2><p>人工干预可以实现在Elastic服务之外，例如在请求Elasticsearch接口之前处理屏蔽词和ip黑名单。</p><p>通常这种服务会被称为Searchhub。</p><p>Searchhub除了要负责处理人工干预逻辑外，还需要处理请求参数校验以及结果数据整合等。</p><p>必要时，缓存也可以在Searchhub层添加。</p><p>Searchhub是一个简单的web服务，这里不再展开。</p><h1 id="服务部署"><a class="markdownIt-Anchor" href="#服务部署"></a> 服务部署</h1><h2 id="elasticsearch"><a class="markdownIt-Anchor" href="#elasticsearch"></a> Elasticsearch</h2><ul><li><p>开发环境，可以根据各自情况，参考<a href="https://www.elastic.co/downloads/elasticsearch" target="_blank" rel="noopener">官方文档</a>自行下载部署。</p></li><li><p>生产环境，建议使用<a href="https://www.elastic.co/cloud/" target="_blank" rel="noopener">官方推荐的云服务</a>。</p></li><li><p>相关工具，建议在开发环境安装<a href="https://www.elastic.co/downloads/kibana" target="_blank" rel="noopener">Kibana</a>以简化操作。</p></li></ul><h2 id="searchhub"><a class="markdownIt-Anchor" href="#searchhub"></a> Searchhub</h2><p>按照通常的方式部署一个web服务即可。</p><h1 id="统计监控"><a class="markdownIt-Anchor" href="#统计监控"></a> 统计监控</h1><h2 id="搜索质量评价指标"><a class="markdownIt-Anchor" href="#搜索质量评价指标"></a> 搜索质量评价指标</h2><p>量化指标可以用来评价搜索服务的价值，也有助于找到提高搜索质量的方向。</p><p>鉴于我们所面对的是垂搜领域里有限数据量的内容搜索，业界流行的搜索质量量化指标并不适合，因此需要自行设计一个搜索质量量化指标体系。</p><p>这里需要先明确我们的目标：</p><ul><li>通过站内搜索，提高信息检索效率，进而提高社区活跃度。</li><li>通过站内搜索，增加优质内容曝光，进而提高传播率。</li></ul><h3 id="指标"><a class="markdownIt-Anchor" href="#指标"></a> 指标</h3><p>结合上面提出的目标，我们提出以下两个指标进行统计：</p><ul><li>搜索结果点击率：点击位置n的搜索结果/搜索次数<br>该指标可以提现搜索结果的准确率，点击率越高说明准确率越高。排位越靠前，点击率应该越高。</li><li>搜索点击比：搜索次数/结果点击次数<br>该指标可以提现搜索结果相关性，用户在一次搜索结果中点击次数越多，说明结果相关性越高。</li></ul><h3 id="采样"><a class="markdownIt-Anchor" href="#采样"></a> 采样</h3><ul><li>热搜词TopN：对热搜词TopN分别进行统计，可以获得每个热搜词的搜索效果。</li><li>长尾词随机采样：随机选取长尾词进行统计，可以获得一般内容的搜索效果。</li></ul><h3 id="实践"><a class="markdownIt-Anchor" href="#实践"></a> 实践</h3><ol><li>统计每日热搜词，长尾词<br>分析搜索api请求日志，统计热搜词及搜索次数，随机选取一定数量的长尾词</li><li>统计热搜词对应的搜索结果点击<br>分析搜索结果点击事件（打点），统计热搜词对应的搜索结果点击次数</li><li>计算热词点击率和点击比</li><li>长尾词与热搜词统计方法相同</li></ol><h2 id="服务质量监控"><a class="markdownIt-Anchor" href="#服务质量监控"></a> 服务质量监控</h2><p>监控是了解实时服务状态和短期服务质量的最直接方式，有服务就要有监控。</p><h3 id="指标-2"><a class="markdownIt-Anchor" href="#指标-2"></a> 指标</h3><p>这里设计几个指标用于监控搜索服务：</p><ul><li>接口请求次数、响应时间<br>按周期统计接口请求次数和响应时间，有助于了解服务压力情况</li><li>接口正确率<br>接口正确率直接反映服务运行状态，这里需要区分：无搜索结果、搜索结果少、搜索被屏蔽等情况</li></ul>]]></content>
<categories>
<category> 工具 </category>
<category> ELK </category>
</categories>
<tags>
<tag> ELK </tag>
</tags>
</entry>
<entry>
<title>【vim】使用vim作为开发go的IDE</title>
<link href="/posts/tool/vim/vim-go/"/>
<url>/posts/tool/vim/vim-go/</url>
<content type="html"><![CDATA[<div class="note info"> <p>本文中将直接使用基本设置已经使用的配置，如需了解，请阅读<a href="/posts/tool/vim/vim-start/">【vim】基本设置</a>。</p> </div><h1 id="基本设置"><a class="markdownIt-Anchor" href="#基本设置"></a> 基本设置</h1><p>创建<code>~/vim/ftplugin/go.vim</code>文件类配置用于go文件的设置。</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">set</span> tw=<span class="number">78</span></span><br></pre></td></tr></table></figure><h1 id="安装vim-go"><a class="markdownIt-Anchor" href="#安装vim-go"></a> 安装vim-go</h1><p><a href="https://github.com/fatih/vim-go/" target="_blank" rel="noopener">vim-go</a>是Go语言官方推荐的vim插件。</p><p>通过Vundle安装，在.vimrc中添加：</p><figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">Plugin <span class="string">'fatih/vim-go'</span></span><br></pre></td></tr></table></figure><p>在vim中执行命令：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">:PluginInstall</span><br></pre></td></tr></table></figure><p>然后需要根据需求安装go相关插件，参考文章<a href="/posts/practice/go/go-workspace/">【开始用go】在MacOS上搭建开发环境</a>。</p>]]></content>
<categories>
<category> 工具 </category>
<category> Vim </category>
</categories>
<tags>
<tag> Vim </tag>
</tags>
</entry>
<entry>
<title>使用openswan搭建l2tp/ipsec服务</title>
<link href="/posts/tool/openswan/l2tp/"/>
<url>/posts/tool/openswan/l2tp/</url>
<content type="html"><![CDATA[<h2 id="使用方法"><a class="markdownIt-Anchor" href="#使用方法"></a> 使用方法</h2><p>在需要使用服务的终端上，新建VPN配置，类型选择L2TP。</p><ul><li>描述 – 随便填写</li><li>服务器 – 下文中的host_ip</li><li>账户 – 下文中的username</li><li>密码 – 下文中的password</li><li>密钥 – 下文中的key</li></ul><h2 id="安装"><a class="markdownIt-Anchor" href="#安装"></a> 安装</h2><p>安装openswan和xl2tpd，ppp（其中有些系统可能已经装好）：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">yum install -y openswan ppp xl2tpd</span><br></pre></td></tr></table></figure><h2 id="配置ipsec"><a class="markdownIt-Anchor" href="#配置ipsec"></a> 配置ipsec</h2><p>新建ipsec配置文件/etc/ipsec.d/l2tp.conf，并确保根配置文件/etc/ipsec.conf已经include之：</p><figure class="highlight plain"><figcaption><span>/etc/ipsec.d/l2tp.conf</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">conn L2TP-PSK-NAT</span><br><span class="line"> rightsubnet=vhost:%priv</span><br><span class="line"> also=L2TP-PSK-noNAT</span><br><span class="line">conn L2TP-PSK-noNAT</span><br><span class="line"> authby=secret</span><br><span class="line"> pfs=no</span><br><span class="line"> auto=add</span><br><span class="line"> keyingtries=3</span><br><span class="line"> rekey=no</span><br><span class="line"> ikelifetime=8h</span><br><span class="line"> keylife=1h</span><br><span class="line"> type=transport</span><br><span class="line"> left=<host_ip></span><br><span class="line"> leftid=<host_ip></span><br><span class="line"> leftprotoport=17/1701</span><br><span class="line"> right=%any</span><br><span class="line"> rightprotoport=17/%any</span><br><span class="line"> dpddelay=40</span><br><span class="line"> dpdtimeout=130</span><br><span class="line"> dpdaction=clear</span><br></pre></td></tr></table></figure><p>注意替换其中的<code><host_ip></code>为公网ip，例如：123.123.123.123。</p><h2 id="配置ppp"><a class="markdownIt-Anchor" href="#配置ppp"></a> 配置ppp</h2><p>编辑ppp配置文件/etc/ppp/options.xl2tpd：</p><figure class="highlight plain"><figcaption><span>/etc/ppp/options.xl2tpd</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">ipcp-accept-local</span><br><span class="line">ipcp-accept-remote</span><br><span class="line">ms-dns 8.8.8.8noccp</span><br><span class="line">auth</span><br><span class="line"># crtscts</span><br><span class="line">idle 1800</span><br><span class="line">mtu 1410</span><br><span class="line">mru 1410</span><br><span class="line"># nodefaultroute</span><br><span class="line">debug</span><br><span class="line"># lock</span><br><span class="line">proxyarp</span><br><span class="line">connect-delay 5000</span><br><span class="line">require-mschap-v2</span><br><span class="line">asyncmap 0</span><br><span class="line">hide-password</span><br><span class="line">name l2tpd</span><br></pre></td></tr></table></figure><h2 id="配置l2tp"><a class="markdownIt-Anchor" href="#配置l2tp"></a> 配置l2tp</h2><p>编辑l2tp配置文件/etc/xl2tpd/xl2tpd.conf，修改global部分：</p><figure class="highlight plain"><figcaption><span>/etc/xl2tpd/xl2tpd.conf</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">listen-addr = <host_ip></span><br><span class="line">auth file = /etc/ppp/chap-secrets</span><br><span class="line">port = 1701</span><br></pre></td></tr></table></figure><p>注意替换其中的<code><host_ip></code>为公网ip，例如：123.123.123.123。</p><h2 id="配置密钥"><a class="markdownIt-Anchor" href="#配置密钥"></a> 配置密钥</h2><p>编辑/etc/ipsec.secrets文件，增加下面内容：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><host_ip> %any: PSK "<key>"</span><br></pre></td></tr></table></figure><p>注意替换其中的<code><host_ip></code>为公网ip，例如：123.123.123.123。</p><p><code><key></code>为共享密钥，可以是任意字符串。</p><p>编辑/etc/ppp/chap-secrets文件，增加下面内容：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"># client server secret IP addresses</span><br><span class="line"><username> l2tpd <password> *</span><br></pre></td></tr></table></figure><p><code><username></code>为用户名，可以是任意字符串。</p><p><code><password></code>为密码，可以是任意字符串。</p><h2 id="配置系统"><a class="markdownIt-Anchor" href="#配置系统"></a> 配置系统</h2><p>修改系统配置，执行下面命令：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">sysctl -w net.ipv4.ip_forward=1</span><br><span class="line">sysctl -w net.ipv4.conf.all.rp_filter=0</span><br><span class="line">sysctl -w net.ipv4.conf.default.rp_filter=0</span><br><span class="line">sysctl -w net.ipv4.conf.eth0.rp_filter=0</span><br><span class="line">sysctl -w net.ipv4.conf.all.send_redirects=0</span><br><span class="line">sysctl -w net.ipv4.conf.default.send_redirects=0</span><br><span class="line">sysctl -w net.ipv4.conf.all.accept_redirects=0</span><br><span class="line">sysctl -w net.ipv4.conf.default.accept_redirects=0</span><br></pre></td></tr></table></figure><p>防火墙开放端口，新建文件/usr/lib/firewalld/services/l2tpd.xml：</p><figure class="highlight xml"><figcaption><span>/usr/lib/firewalld/services/l2tpd.xml</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta"><?xml version="1.0" encoding="utf-8"?></span></span><br><span class="line"><span class="tag"><<span class="name">service</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">short</span>></span>L2TPD<span class="tag"></<span class="name">short</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>L2TPD Port<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">port</span> <span class="attr">protocol</span>=<span class="string">"udp"</span> <span class="attr">port</span>=<span class="string">"1701"</span>/></span></span><br><span class="line"><span class="tag"></<span class="name">service</span>></span></span><br></pre></td></tr></table></figure><p>在firewall中开启相应的服务：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">firewall-cmd --permanent --add-service=l2tpd</span><br><span class="line">firewall-cmd --permanent --add-service=ipsec</span><br><span class="line">firewall-cmd --permanent --add-masquerade</span><br><span class="line">firewall-cmd --reload</span><br></pre></td></tr></table></figure><h2 id="启动"><a class="markdownIt-Anchor" href="#启动"></a> 启动</h2><p>启动系统服务，并设置为开机自启动</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">systemctl <span class="built_in">enable</span> ipsec xl2tpd</span><br><span class="line">systemctl restart ipsec xl2tpd</span><br></pre></td></tr></table></figure><h2 id="调试"><a class="markdownIt-Anchor" href="#调试"></a> 调试</h2><p>如果启动不成功，可以使用journal查看日志调试：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">journalctl -f</span><br></pre></td></tr></table></figure>]]></content>
<categories>
<category> 工具 </category>
<category> openswan </category>
</categories>
<tags>
<tag> vpn </tag>
</tags>
</entry>
<entry>
<title>gitlab实践方案</title>
<link href="/posts/tool/gitlab/gitlab-practice/"/>
<url>/posts/tool/gitlab/gitlab-practice/</url>
<content type="html"><![CDATA[<p>本文提供一种基于gitlab进行团队项目开发发的解决方案。</p><p>目前有两个途径可以使用gitlab:</p><ol><li>直接使用<a href="https://gitlab.com" target="_blank" rel="noopener">gitlab.com</a>官方提供的服务，个人项目免费，创建团队收费。</li><li>基于开源协议，自行搭建gitlab服务。方法请参考<a href="https://about.gitlab.com/installation/" target="_blank" rel="noopener">官方教程</a></li></ol><h1 id="用户和权限管理"><a class="markdownIt-Anchor" href="#用户和权限管理"></a> 用户和权限管理</h1><p>通常，对于一个项目团队，成员应该有如下角色，（括号中是对应的gitalb角色）：</p><ul><li>项目负责人（master）：负责项目部署发布和持续集成----通常是开发组长</li><li>项目开发者（developer）：负责项目开发维护----通常是做这个项目的开发人员</li><li>其他相关开发者（reporter）：浏览项目代码----通常是不做这个项目的开发人员，例如后端项目对应的前端开发</li><li>其他相关人员（guest）：浏览编辑issue，浏览wiki，获取artifacts等----包括产品、测试等</li></ul><p>对于用户管理，通常需要面对下面问题：</p><ul><li>如何创建删除用户?<br>建议使用gitlab自身账户系统管理账户数据。可通过使用gitlab-api同步企业oa数据。<br>gitlab账户存在与否，应该完全由管理员负责，该管理员通常身份是运维负责人。</li><li>如何管理账户权限?<br>建议通过项目和组来管理用户权限。<br>对于一个新账户，默认不属于任何项目和组。<br>项目负责人和组负责人在需要时，将目标用户添加删除到所负责的项目或组中。</li></ul><h1 id="项目管理"><a class="markdownIt-Anchor" href="#项目管理"></a> 项目管理</h1><p>所有项目都应该属于一个项目组，并且只能由项目组负责人创建项目。</p><div class="note warning"> <p>这里不建议在公司git上创建个人项目，一方面会增加代码管理复杂度，另一方面团队项目的个人分支完全可以做到想做的事情。</p> </div><h2 id="repository"><a class="markdownIt-Anchor" href="#repository"></a> repository</h2><p>创建项目时，应遵循以下原则：</p><ul><li>项目由项目负责人创建</li><li>项目命名空间应该是组而非个人</li><li>项目可见级别选择Internal</li><li>项目描述不要留空，尽量做到简明扼要</li><li><a href="http://xn--README-h18i797heob97vkwekrfjq9d6f8dj00a.md" target="_blank" rel="noopener">项目应当有描述文件README.md</a></li></ul><p>README.md格式建议</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><项目名称></span><br><span class="line">====</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><项目背景和功能描述...></span><br><span class="line"></span><br><span class="line">使用手册或接口文档</span><br><span class="line">----</span><br><span class="line"><主要是项目的功能和用法。可链接到wiki></span><br><span class="line"></span><br><span class="line">开发者文档</span><br><span class="line">----</span><br><span class="line"><包括项目结构说明，开发要求和注意事项，开发环境部署等。可链接到wiki></span><br><span class="line"></span><br><span class="line">内部依赖</span><br><span class="line">----</span><br><span class="line"><依赖的数据，服务，代码库等></span><br><span class="line"></span><br><span class="line">外部依赖</span><br><span class="line">----</span><br><span class="line"><依赖的数据，服务，代码库等></span><br></pre></td></tr></table></figure><h2 id="branch"><a class="markdownIt-Anchor" href="#branch"></a> branch</h2><p>对于规模较小的团队项目，建议使用简化模式进行分支管理。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"> 1.0.1 1.0.2</span><br><span class="line">master -----------+-------------------+----</span><br><span class="line"> | | |</span><br><span class="line">develop +--------+-----------------+-+-----</span><br><span class="line"> | | | | | |</span><br><span class="line"> dev/xx1 +----+ | dev/xx3 +----+ |</span><br><span class="line"> dev/xx2 +----------------+</span><br></pre></td></tr></table></figure><p>首先，对于任意项目，都创建两个保护分支：</p><ul><li>master: 只有发布才合并到该分支</li><li>develop: 有新的变化并测试通过后合并到该分支</li></ul><p>开发者通过提交MR将<code>dev/xxx</code>合并到develop分支上，项目负责人根据需要对develop分支进行发布。</p><p>当团队规模到达一定程度时，使用git flow模式进行分支管理会带来很大的方便。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"> 1.0.0 1.0.1 1.1.0 1.1.1 1.2.0</span><br><span class="line">master --------------------+------------------+-----------------+------------------+----</span><br><span class="line"> | | | | | | |</span><br><span class="line"> | hotfix/xx1 +----+ | hotfix/xx2 +----+ |</span><br><span class="line"> | | release/xx1 +----+ | release/xx2 +----+</span><br><span class="line"> | | | | | | |</span><br><span class="line">develop +-----------------+-----------+------+-----------------+--+----+----------+----</span><br><span class="line"> | | | | | |</span><br><span class="line"> | feature/xx2 +--------+ feature/xxx3 +----+ |</span><br><span class="line"> feature/xx1 +----------------------------------------------------------+</span><br></pre></td></tr></table></figure><h2 id="commit"><a class="markdownIt-Anchor" href="#commit"></a> commit</h2><p>建议使用message格式：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><本次提交的标题></span><br><span class="line"></span><br><span class="line">- <本次提交的内容></span><br><span class="line">- <本次提交的内容></span><br><span class="line">- <本次提交的内容></span><br><span class="line">- <本次提交的内容></span><br><span class="line"></span><br><span class="line"><本次提交的其他说明></span><br></pre></td></tr></table></figure><h2 id="ci"><a class="markdownIt-Anchor" href="#ci"></a> ci</h2><p>为了保证项目的敏捷开发和持续集成，建议使用<a href="https://docs.gitlab.com/ee/ci/yaml/" target="_blank" rel="noopener">gitlab-ci</a>功能。</p><ul><li>ci配置中应设置合并到任何分支都触发unittest</li><li>合并到develop和master时触发release（可配合ci产生atrifacts用于部署发布）</li></ul><h2 id="issue"><a class="markdownIt-Anchor" href="#issue"></a> issue</h2><p>建议使用gitlab的issue推动项目开发。</p><p>开发者要解决对应的issue可以通过create merge request按钮（新版本gitlab支持）创建branch和merge request，这样方便项目负责人跟踪项目进度。</p><h2 id="wiki"><a class="markdownIt-Anchor" href="#wiki"></a> wiki</h2><p>建议写项目开发相关内容。</p>]]></content>
<categories>
<category> 工具 </category>
<category> Gitlab </category>
</categories>
<tags>
<tag> git </tag>
<tag> Gitlab </tag>
</tags>
</entry>
<entry>
<title>Hexo排版</title>
<link href="/posts/tool/hexo/hexo-type-setting/"/>
<url>/posts/tool/hexo/hexo-type-setting/</url>
<content type="html"><![CDATA[<p>Hexo有三种排版工具可供选择：</p><ul><li>标签 – 用于多维度索引一篇文章</li><li>分类 – 用于单维度归类一篇文章</li><li>搜索 – 用于模糊查找一篇文章</li></ul><p>下面逐一介绍如何使用这三个工具。</p><h2 id="标签"><a class="markdownIt-Anchor" href="#标签"></a> 标签</h2><p>在文件头部属性中，tags用于声明当前文章的标签，例如：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">tags: [ Hexo, 排版 ]</span><br></pre></td></tr></table></figure><p>这个属性可以指定多个词语作为一篇文章的标签，把多篇文章的标签集合到一起并统计，就可以得到标签云。</p><p>事实上，Hexo已经默认做了这件事，想要看到标签云，只需要创建一个页面并进行一些配置即可：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">hexo new page tags</span><br></pre></td></tr></table></figure><p>上面的命令，会创建一个source/tags/index.md的文件，编辑如下内容：</p><figure class="highlight markdown"><figcaption><span>source/tags/index.md</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">---</span><br><span class="line">title: tags</span><br><span class="line">date: 2019-08-28 18:42:48</span><br><span class="line">type: "tags"</span><br><span class="line">comments: false</span><br><span class="line">---</span><br></pre></td></tr></table></figure><p>最后，需要在主题配置中，放开标签页面的入口：</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">menu:</span></span><br><span class="line"><span class="attr"> tags:</span> <span class="string">/tags/</span> <span class="string">||</span> <span class="string">tags</span></span><br></pre></td></tr></table></figure><h2 id="分类"><a class="markdownIt-Anchor" href="#分类"></a> 分类</h2><p>分类的用法几乎同标签用法一样，例如：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">categories: [开源软件研究和使用, Hexo]</span><br></pre></td></tr></table></figure><p>这个属性，当指定多个词语时，它们是父子关系，即：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">- 开源软件研究和使用</span><br><span class="line"> - Hexo</span><br></pre></td></tr></table></figure><p>同样的，Hexo也已经将每篇文章的分类进行了整合，想要看到一个分类索引的页面，创建页面并配置即可：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">hexo new page categories</span><br></pre></td></tr></table></figure><p>上面的命令，会创建一个source/tags/index.md的文件，编辑如下内容：</p><figure class="highlight markdown"><figcaption><span>source/categories/index.md</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">---</span><br><span class="line">title: categories</span><br><span class="line">date: 2019-08-28 18:40:41</span><br><span class="line">type: "categories"</span><br><span class="line">comments: false</span><br><span class="line">---</span><br></pre></td></tr></table></figure><p>最后，需要在主题配置中，放开标签页面的入口：</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">menu:</span></span><br><span class="line"><span class="attr"> categories:</span> <span class="string">/categories/</span> <span class="string">||</span> <span class="string">th</span></span><br></pre></td></tr></table></figure><h2 id="搜索"><a class="markdownIt-Anchor" href="#搜索"></a> 搜索</h2><p>支持本地搜索（全文搜索），安装：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">npm install hexo-generator-search --save</span><br></pre></td></tr></table></figure><p>然后修改配置项：</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">local_search:</span></span><br><span class="line"><span class="attr"> enable:</span> <span class="literal">true</span></span><br></pre></td></tr></table></figure>]]></content>
<categories>
<category> 工具 </category>
<category> Hexo </category>
</categories>
<tags>
<tag> Hexo </tag>
</tags>
</entry>
<entry>
<title>Hexo主题</title>
<link href="/posts/tool/hexo/hexo-theme/"/>
<url>/posts/tool/hexo/hexo-theme/</url>
<content type="html"><![CDATA[<p>hexo创建项目后，提供了一个默认的主题landscape。<br>如果没有特殊需求，直接使用即可。当然，也可以在官方网站上找自己喜欢的主题进行替换。</p><p>笔者安装了<a href="https://theme-next.org/docs/getting-started/" target="_blank" rel="noopener">NexT主题</a>，并对其进行了一些配置。</p><p>下面将详细讲述安装配置过程，及遇到的一些问题的解决办法。</p><h2 id="下载安装"><a class="markdownIt-Anchor" href="#下载安装"></a> 下载安装</h2><p>如文档所述，直接通过git下载：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">git <span class="built_in">clone</span> https://github.com/theme-next/hexo-theme-next themes/next</span><br></pre></td></tr></table></figure><p>然后修改配置文件_config.yml：</p><figure class="highlight yaml"><figcaption><span>_config.yml</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">theme:</span> <span class="string">next</span></span><br></pre></td></tr></table></figure><p>最后生成静态页面：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">hexo g</span><br></pre></td></tr></table></figure><div class="note warning"> <p>此时按照文档的说法，应该已经可以开始使用NexT主题了，然而笔者却遇到了几个错误。</p><p>错误很明确，就是需要安装一些依赖的包：</p><p><code>npm install hexo-fs hexo-util lodash --save</code></p> </div><h2 id="配置文件"><a class="markdownIt-Anchor" href="#配置文件"></a> 配置文件</h2><p>安装成功后，简单阅读了配置文件，发现足足1000多行。也就是说，基本上所有能想到的它都是支持配置的。</p><p>这里需要特别一提的，就是它的第一个配置项override。按照文档的说法，这个选项是为了更新主题是不把用户的配置覆盖掉，用户可以写一个_data/next.yml文件来与next/_config.yml进行merge。<br>这是一个不错的实践，要照做之。新建一个next.yml文件，然后在其中修改那些需要修改的配置项，其他的继续使用默认配置即可。</p><div class="note warning"> <p>注意，next.yml文件要放在source_data/next.yml位置才生效。</p> </div><p>遍历了全部配置项后，有些选项可以优先关注，下文将注意说明。</p><h2 id="favicon"><a class="markdownIt-Anchor" href="#favicon"></a> favicon</h2><p>通过favicon的配置，可以看到NexT对不同的设备终端设置了不同的视图，用户可以按照规范制作相应的favicon文件，并编写配置项指向文件。</p><h2 id="footer"><a class="markdownIt-Anchor" href="#footer"></a> footer</h2><p>网站的footer是统一的，也是完全可配的。默认配置文件中，列出了全部可配项及其说明，按照实际情况填写即可。</p><div class="note info"> <p>这里强烈建议保留powered和theme两个配置项。</p> </div><h2 id="百度统计"><a class="markdownIt-Anchor" href="#百度统计"></a> 百度统计</h2><p>NexT主题支持多种统计服务，这里以百度统计为例，只需要配置百度统计后台生成的appid即可：</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">baidu_analytics:</span> <span class="string">e0fc945f04e1f73b09770bf9d28d0627</span></span><br></pre></td></tr></table></figure><h2 id="rss"><a class="markdownIt-Anchor" href="#rss"></a> rss</h2><p>支持rss，安装：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">npm install hexo-generator-feed</span><br></pre></td></tr></table></figure><p>然后使用默认配置项即可：</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">rss:</span></span><br></pre></td></tr></table></figure><h2 id="发布"><a class="markdownIt-Anchor" href="#发布"></a> 发布</h2><p>新换的主题，发布要特别注意，先执行：<code>hexo clean</code>。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">hexo clean</span><br><span class="line">hexo g -d</span><br></pre></td></tr></table></figure>]]></content>
<categories>
<category> 工具 </category>
<category> Hexo </category>
</categories>
<tags>
<tag> Hexo </tag>
<tag> NexT </tag>
</tags>
</entry>
<entry>
<title>从Hexo开始</title>
<link href="/posts/tool/hexo/hexo/"/>
<url>/posts/tool/hexo/hexo/</url>
<content type="html"><![CDATA[<p>此前，一直追求一个支持markdown渲染的纯前端组件来构建技术博客，始终不太理想。<br>主要面对的问题有三：</p><ol><li>渲染速度慢<br>该体系下访问一个博文，需要先下载md文件和渲染器代码，然后渲染器解析md生成html，同时还需要下载必要的主题。<br>这样一个流程下来，页面渲染速度通常需要几百毫秒，使用起来不能令人满意。</li><li>迁移部署并不简单<br>纯前端渲染markdown的思想主要是为了直接通过git管理md文件，任何地方直接下载即可部署。<br>然而部署过程中还需要下载用于渲染的项目，并配置nginx进行代理，如此并没有剩下多少工作量。</li><li>功能有所欠缺<br>博客常用的目录、标签、搜索、归档等功能，在该体系下很难实现的很好。<br>即便实现也会导致投入成本太高，且破坏了纯markdown写博客的初衷。</li></ol><p>因此，决定尝试转变思路，探索一些通过编译构建博客的方案。</p><h2 id="简介"><a class="markdownIt-Anchor" href="#简介"></a> 简介</h2><p>hexo是一个基于nodejs对markdown文件进行前端编译为静态资源并发布的博客框架。<br>它实现了一套完整的命令行工具来支持整个博客编写过程（并没有支持编辑），甚至通过读取配置文件，可以直接发布到服务器。<br>当然也可以直接组织编辑文件，只使用hexo的编译生成功能（目前看来二者兼顾可能是最佳实践）。<br>通过主题、模板、辅助函数等功能可以实现自定义排版和索引。</p><p>在大致了解了上述功能后，笔者决定开始实践之。以下将具体讲述整个实践过程，以及中间的一些问题和解决方法。</p><h2 id="部署"><a class="markdownIt-Anchor" href="#部署"></a> 部署</h2><p>这里使用github pages部署，可是实现零成本建站。</p><p>需要注意的是，源代码要通过另一个repo来管理：</p><ul><li><a href="https://github.com/mapleque/blog.git" target="_blank" rel="noopener">https://github.com/mapleque/blog.git</a> 管理源代码</li><li><a href="https://github.com/mapleque/mapleque.github.io" target="_blank" rel="noopener">https://github.com/mapleque/mapleque.github.io</a> 管理pages</li></ul><p>编辑配置文件，可以实现命令行发布：</p><figure class="highlight yaml"><figcaption><span>_config.yml</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Deployment</span></span><br><span class="line"><span class="comment">## Docs: https://hexo.io/docs/deployment.html</span></span><br><span class="line"><span class="attr">deploy:</span></span><br><span class="line"><span class="attr"> type:</span> <span class="string">git</span></span><br><span class="line"><span class="attr"> repo:</span> <span class="string">[email protected]:mapleque/mapleque.github.io.git</span></span><br></pre></td></tr></table></figure><p>这样每次写完文章，执行下面语句即可发布：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">hexo g -d</span><br></pre></td></tr></table></figure><div class="note danger"> <p>这个命令会执行git push --force，一定要注意执行的后果。</p> </div><p>最后还要记得将源代码提交：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">git add .</span><br><span class="line">git commit -m<span class="string">"feature: add something"</span></span><br><span class="line">git push origin master</span><br></pre></td></tr></table></figure><h2 id="私有域名"><a class="markdownIt-Anchor" href="#私有域名"></a> 私有域名</h2><p>如果需要使用自己的域名部署，这里需要做两件事：</p><ol><li><p>让github pages支持私有域名<br>在项目中添加一个CNAME文件：</p> <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">echo</span> <span class="string">'blog.mapleque.com'</span> > <span class="built_in">source</span>/CNAME</span><br><span class="line">hexo g -d</span><br></pre></td></tr></table></figure><p>这里只有把CNAME文件直接放在source中，才能保证每次发布都能够保留。</p></li><li><p>添加域名CNAME解析：<br>在dns解析中添加CNAME条目：</p> <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">CNAME blog.mapleque.com mapleque.github.io</span><br></pre></td></tr></table></figure></li></ol><h2 id="基本配置"><a class="markdownIt-Anchor" href="#基本配置"></a> 基本配置</h2><p>通过修改_config.yml文件，可以完成大部分网站基本配置，如：title，description，keywords等。</p><figure class="highlight yaml"><figcaption><span>_config.yml</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Site</span></span><br><span class="line"><span class="attr">title:</span> <span class="string">枫枝雀自鸣</span></span><br><span class="line"><span class="attr">subtitle:</span> <span class="string">技术博客</span></span><br><span class="line"><span class="attr">description:</span> <span class="string">谈技术，看积累。这里记录的是笔者的亲身经历和反复总结。</span></span><br><span class="line"><span class="attr">keywords:</span> <span class="string">互联网,</span> <span class="string">研发,</span> <span class="string">计算机,</span> <span class="string">工程</span></span><br></pre></td></tr></table></figure><div class="note info"> <p>注意，上面这些项一定要认真填写，它们是搜索引擎识别你的重要信息源。</p> </div>]]></content>
<categories>
<category> 工具 </category>
<category> Hexo </category>
</categories>
<tags>
<tag> Hexo </tag>
</tags>
</entry>
</search>