-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy path14965491161776.html
202 lines (145 loc) · 7.83 KB
/
14965491161776.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
<!DOCTYPE html>
<html>
<head>
<title>
TensorFlow初体验 - 咕噜灵波
</title>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="肥宅炼丹术士的日常">
<!-- 魔改部分 -->
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/font-awesome/css/font-awesome.min.css">
<link rel="stylesheet" type="text/css" href="asset/style.css">
<link rel="stylesheet" type="text/css" href="asset/cuckoo.css">
<link rel="stylesheet" type="text/css" href="asset/main.css">
<link rel="stylesheet" type="text/css" href="asset/atom-one-light.css">
<link rel="alternate" type="application/atom+xml" href="atom.xml" title="咕噜灵波">
<script src="asset/highlight.pack.js"></script>
<!-- <script>hljs.initHighlightingOnLoad();</script>-->
</head>
<body>
<header class="site-header cuckoo">
<div class="wrapper">
<a class="site-title" href="index.html">咕噜灵波</a>
<nav class="site-nav">
<a href="#" class="menu-icon">
<svg viewBox="0 0 18 15">
<path fill="#424242"
d="M18,1.484c0,0.82-0.665,1.484-1.484,1.484H1.484C0.665,2.969,0,2.304,0,1.484l0,0C0,0.665,0.665,0,1.484,0 h15.031C17.335,0,18,0.665,18,1.484L18,1.484z">
</path>
<path fill="#424242"
d="M18,7.516C18,8.335,17.335,9,16.516,9H1.484C0.665,9,0,8.335,0,7.516l0,0c0-0.82,0.665-1.484,1.484-1.484 h15.031C17.335,6.031,18,6.696,18,7.516L18,7.516z">
</path>
<path fill="#424242"
d="M18,13.516C18,14.335,17.335,15,16.516,15H1.484C0.665,15,0,14.335,0,13.516l0,0 c0-0.82,0.665-1.484,1.484-1.484h15.031C17.335,12.031,18,12.696,18,13.516L18,13.516z">
</path>
</svg>
</a>
<div class="trigger">
<a class="page-link" href="index.html">Home</a>
<a class="page-link" href="archives.html">Archives</a>
</div>
</nav>
</div>
</header>
<script src="./asset/live2d/autoload.js"></script>
</body>
</html> <div class="page-content cuckoo">
<div class="pattern-center ">
<div id="surpriseImg" class="pattern-attachment-img">
<!-- <img src="suprizeImg#lazyload-blur" data-src="suprizeImg" class="lazyload"
onerror="imgError(this,3)" style="width: 100%; height: 100%; object-fit: cover; pointer-events: none;"> -->
</div>
<header class="pattern-header ">
<h1 class="entry-title">TensorFlow初体验</h1>
</header>
</div>
<div class="wrapper">
<article class="post" itemscope itemtype="http://schema.org/BlogPosting">
<header class="post-header">
<!-- <h1 class="post-title" itemprop="name headline">TensorFlow初体验</h1> -->
<div class="post-description">
</div>
</header>
<div class="post-content" itemprop="articleBody">
<h2 id="toc_0">从softmax到单隐含层的神经网络</h2>
<h4 id="toc_1">1. 基本架构</h4>
<p>在学习了用softmax回归对MNIST进行训练后，发现softmax的准确率只有91%左右，于是我考虑用单隐含层的神经网络进行改进。</p>
<p>隐含层单元数为25，根据Andrew Ng老师给出的公式 </p>
<blockquote>
<p>sqrt(input_layer_size + output_layer_size) + C, C为常数，一般取10左右。</p>
</blockquote>
<p>后来经过测试，隐含层单元数在20-30这个范围内差别都不大。</p>
<p>隐含层的激励函数我尝试了多种，发现使用Sigmoid函数会发生“梯度消失”现象，导致一开始梯度下降的异常缓慢。后来我尝试了ReLU作为激励函数，成功解决了梯度消失的问题。</p>
<p>代价函数我尝试了两种：<br/>
一种是softmax回归中用到的交叉熵</p>
<pre><code class="language-text">cross_entropy = -tf.reduce_mean(y_*tf.log(y))
</code></pre>
<p>另一种是Andrew Ng的课程中使用的</p>
<pre><code class="language-text">cross_entropy = -tf.reduce_mean(y_*tf.log(y) + (1-y_)*tf.log(1-y))
</code></pre>
<p>本以为后一种会更准确一些，然而实践证明差别不大。。。</p>
<p>到此为止，训练的准确率还非常惨。。</p>
<h4 id="toc_2">2. 优化</h4>
<p>然而机智的我想到了随机初始化权重的方法（其实Andrew的课程里强调了很多次。。）</p>
<p>随机初始化权重的方法有很多种，比如TensorFlow官方教程中讲到</p>
<blockquote>
<p>这个模型中的权重在初始化时应该加入少量的噪声来打破对称性以及避免0梯度。由于我们使用的是ReLU神经元，因此比较好的做法是用一个较小的正数来初始化偏置项，以避免神经元节点输出恒为0的问题（dead neurons）。</p>
</blockquote>
<p>我这里使用的是Andrew课程里使用的方法，具体公式见讲义。</p>
<pre><code class="language-text">#随机初始化权重，大幅度提高智商！
W1 = tf.Variable(tf.zeros([784, hidden_layer_size]))
W1 = tf.Variable( tf.random_uniform([784,hidden_layer_size],minval=0,maxval=1,dtype=tf.float32,seed=None,name=None) )
epsilon = 1.0*math.sqrt(6)/math.sqrt(784+hidden_layer_size)
W1 = W1 * 2 * epsilon - epsilon
W2 = tf.Variable(tf.zeros([hidden_layer_size,10]))
W2 = tf.Variable( tf.random_uniform([hidden_layer_size,10],minval=0,maxval=1,dtype=tf.float32,seed=None,name=None) )
epsilon = 1.0*math.sqrt(6)/math.sqrt(hidden_layer_size+10)
W2 = W2 * 2 * epsilon - epsilon
</code></pre>
<p align="right">2017.6.1</p>
<h2 id="toc_3">构建简单CNN</h2>
<p>首先我要感慨一下，CNN训练的时间实在是长。。。。softmax几秒钟就完成的事，CNN要干几小时。</p>
<p>根据Convolutional Neural Networks (LeNet)对一个双卷积池化层+隐层+SM层的网络来说：</p>
<blockquote>
<p>在 Core i7-2600K CPU @ 3.40GHz 上， 使用标志 ‘floatX=float32’：Optimization complete.<br/>
Best validation score of 0.910000 % obtained at iteration 17800,with test<br/>
performance 0.920000 %<br/>
The code for file convolutional_mlp.py ran for 380.28m</p>
<p>在 GeForce GTX 285上跑：Optimization complete.<br/>
Best validation score of 0.910000 % obtained at iteration 15500,with test<br/>
performance 0.930000 %<br/>
The code for file convolutional_mlp.py ran for 46.76m</p>
</blockquote>
<p>GPU的速度大约是CPU的8-10倍。</p>
</div>
<div>
</div>
</article>
</div>
</div>
<script src="asset/jquery-3.3.1.min.js"></script>
<script>
$(document).ready(function () {
$('#footer').show()
})
/** 生成1-4的随机图片标签 **/
function getRandomImage() {
var index = Math.ceil(Math.random() * 10) % 4;
var path = './asset/banner/' + index + '.jpg'
// return '<img src="/' + index + '.jpg" >';
return '<img src="' + path + '#lazyload-blur" data-src="' + path +
'" class="lazyload" onerror="imgError(this,3)" style="width: 100%; height: 100%; object-fit: cover; pointer-events: none;">'
}
/** 根据传进来的元素id,追加一个随机图片并显示 **/
function showRandonImage(divId) {
var oldHTML = document.getElementById(divId).innerHTML;
var newHTML = oldHTML + getRandomImage();
document.getElementById(divId).innerHTML = newHTML;
}
showRandonImage('surpriseImg');
</script> <footer id="footer" class="cuckoo page-footer" style="display: none">
<div class="wrapper">
<p>This blog is maintained by TSERR0F</a></p>
</div>
</footer>