-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathREADME.html
232 lines (232 loc) · 17.5 KB
/
README.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
<head>
<meta charset="utf-8" />
<meta name="generator" content="pandoc" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
<title>README</title>
<style>
code{white-space: pre-wrap;}
span.smallcaps{font-variant: small-caps;}
span.underline{text-decoration: underline;}
div.column{display: inline-block; vertical-align: top; width: 50%;}
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
ul.task-list{list-style: none;}
pre > code.sourceCode { white-space: pre; position: relative; }
pre > code.sourceCode > span { display: inline-block; line-height: 1.25; }
pre > code.sourceCode > span:empty { height: 1.2em; }
code.sourceCode > span { color: inherit; text-decoration: inherit; }
div.sourceCode { margin: 1em 0; }
pre.sourceCode { margin: 0; }
@media screen {
div.sourceCode { overflow: auto; }
}
@media print {
pre > code.sourceCode { white-space: pre-wrap; }
pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; }
}
pre.numberSource code
{ counter-reset: source-line 0; }
pre.numberSource code > span
{ position: relative; left: -4em; counter-increment: source-line; }
pre.numberSource code > span > a:first-child::before
{ content: counter(source-line);
position: relative; left: -1em; text-align: right; vertical-align: baseline;
border: none; display: inline-block;
-webkit-touch-callout: none; -webkit-user-select: none;
-khtml-user-select: none; -moz-user-select: none;
-ms-user-select: none; user-select: none;
padding: 0 4px; width: 4em;
color: #aaaaaa;
}
pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; }
div.sourceCode
{ }
@media screen {
pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; }
}
code span.al { color: #ff0000; font-weight: bold; } /* Alert */
code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
code span.at { color: #7d9029; } /* Attribute */
code span.bn { color: #40a070; } /* BaseN */
code span.bu { } /* BuiltIn */
code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
code span.ch { color: #4070a0; } /* Char */
code span.cn { color: #880000; } /* Constant */
code span.co { color: #60a0b0; font-style: italic; } /* Comment */
code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
code span.do { color: #ba2121; font-style: italic; } /* Documentation */
code span.dt { color: #902000; } /* DataType */
code span.dv { color: #40a070; } /* DecVal */
code span.er { color: #ff0000; font-weight: bold; } /* Error */
code span.ex { } /* Extension */
code span.fl { color: #40a070; } /* Float */
code span.fu { color: #06287e; } /* Function */
code span.im { } /* Import */
code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
code span.kw { color: #007020; font-weight: bold; } /* Keyword */
code span.op { color: #666666; } /* Operator */
code span.ot { color: #007020; } /* Other */
code span.pp { color: #bc7a00; } /* Preprocessor */
code span.sc { color: #4070a0; } /* SpecialChar */
code span.ss { color: #bb6688; } /* SpecialString */
code span.st { color: #4070a0; } /* String */
code span.va { color: #19177c; } /* Variable */
code span.vs { color: #4070a0; } /* VerbatimString */
code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
.display.math{display: block; text-align: center; margin: 0.5rem auto;}
</style>
<link rel="stylesheet" href="https://rawcdn.githack.com/edwardtufte/tufte-css/e225f3a0e5f8f42a1d28416c1c85962411711907/tufte.min.css" />
<!--[if lt IE 9]>
<script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
<![endif]-->
<style>html{font-size: 2em;}</style>
</head>
<body>
<h1 id="file-embedding-in-go">File Embedding in Go</h1>
<blockquote>
<p>Talk at <a href="https://www.meetup.com/Leipzig-Golang/events/268785591/">Golang Leipzig November Meetup</a></p>
</blockquote>
<h2 id="what-is-file-embedding">What is File Embedding?</h2>
<ul>
<li>packing arbitrary resources into a binary/executable</li>
<li>a resource can be anything, e.g. HTML templates, CSS or Javascript files, a favicon, translations…</li>
<li>example layout of a typical web application:</li>
</ul>
<div class="sourceCode" id="cb1"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a>├── <span class="ex">assets</span></span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a>│ ├── <span class="ex">base.css</span></span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a>│ ├── <span class="ex">base.js</span></span>
<span id="cb1-4"><a href="#cb1-4" aria-hidden="true" tabindex="-1"></a>│ ├── <span class="ex">...</span></span>
<span id="cb1-5"><a href="#cb1-5" aria-hidden="true" tabindex="-1"></a>│ └── <span class="ex">favicon.svg</span></span>
<span id="cb1-6"><a href="#cb1-6" aria-hidden="true" tabindex="-1"></a>├── <span class="ex">migrations</span></span>
<span id="cb1-7"><a href="#cb1-7" aria-hidden="true" tabindex="-1"></a>│ ├── <span class="ex">01_initial_setup.down.sql</span></span>
<span id="cb1-8"><a href="#cb1-8" aria-hidden="true" tabindex="-1"></a>│ └── <span class="ex">...</span></span>
<span id="cb1-9"><a href="#cb1-9" aria-hidden="true" tabindex="-1"></a>├── <span class="ex">notes.go</span></span>
<span id="cb1-10"><a href="#cb1-10" aria-hidden="true" tabindex="-1"></a>├── <span class="ex">storage.go</span></span>
<span id="cb1-11"><a href="#cb1-11" aria-hidden="true" tabindex="-1"></a>├── <span class="ex">storage_test.go</span></span>
<span id="cb1-12"><a href="#cb1-12" aria-hidden="true" tabindex="-1"></a>└── <span class="ex">views</span></span>
<span id="cb1-13"><a href="#cb1-13" aria-hidden="true" tabindex="-1"></a> ├── <span class="ex">index.gohtml</span></span>
<span id="cb1-14"><a href="#cb1-14" aria-hidden="true" tabindex="-1"></a> ├── <span class="ex">...</span></span>
<span id="cb1-15"><a href="#cb1-15" aria-hidden="true" tabindex="-1"></a> └── <span class="ex">layouts</span></span>
<span id="cb1-16"><a href="#cb1-16" aria-hidden="true" tabindex="-1"></a> └── <span class="ex">base.gohtml</span></span></code></pre></div>
<h2 id="advantages-thumbsup">Advantages <span class="emoji" data-emoji="thumbsup">👍</span></h2>
<ul>
<li>only a single file needs to be distributed/deployed</li>
<li>especially useful for desktop applications (no need for <code>$XDG_HOME</code>, <code>%APPDATA%</code> or <code>~/Library</code>)</li>
<li>embedded data is “immutable” (at least from outside the process)</li>
</ul>
<h2 id="disadvantages-thumbsdown">Disadvantages <span class="emoji" data-emoji="thumbsdown">👎</span></h2>
<ul>
<li>binary size grows with each resource that gets embedded → increased memory usage</li>
<li>space efficiency depends on the encoding of the embedded content
<ul>
<li><a href="https://en.wikipedia.org/wiki/Base64#Output_padding">base64 wastes about 1/3</a> for encoding</li>
</ul></li>
<li>large files are not suitable for embedding</li>
</ul>
<p>If the resources are compressable—e.g. most plain text files are—then a binary packer like <a href="https://upx.github.io/">upx</a> can reduce the file size dramatically:</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a>$ <span class="ex">upx</span> -o notes.upx notes</span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a>$ <span class="fu">du</span> -sh notes notes.upx </span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a><span class="ex">24M</span> notes</span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a><span class="ex">9.7M</span> notes.upx</span></code></pre></div>
<h2 id="available-implementations">Available Implementations</h2>
<ul>
<li><a href="https://github.com/jteeuwen/go-bindata">go-bindata</a> most popular but now deprecated, recommends <code>pkger</code></li>
<li><a href="https://github.com/markbates/pkger">pkger</a>
<ul>
<li>requires a Go modules project</li>
<li>API simulates <code>os.File</code></li>
<li><span class="emoji" data-emoji="thumbsdown">👎</span> stores a lot of redundant metadata, e.g. <a href="https://github.com/markbates/pkger#how-it-works-module-aware-pathing">absolute paths to files</a> <span class="emoji" data-emoji="policewoman">👮♀️</span> or imported packages</li>
</ul></li>
<li><a href="https://github.com/rakyll/statik">statik</a>
<ul>
<li>simulates an <a href="https://pkg.go.dev/net/http#FileSystem"><code>http.FileSystem</code></a></li>
<li><span class="emoji" data-emoji="thumbsdown">👎</span> can only include a single directory</li>
</ul></li>
<li><a href="https://github.com/GeertJohan/go.rice">go.rice</a>
<ul>
<li><span class="emoji" data-emoji="thumbsdown">👎</span> complicated to use, scans your source code for <code>rice.FindBox("/path")</code> calls and includes them</li>
<li><span class="emoji" data-emoji="thumbsdown">👎</span> weird limitations, e.g. paths cannot be constant strings or undefined behaviour when called in <code>init()</code></li>
<li>failed to set it up for my example application, I really wonder why this got popular <span class="emoji" data-emoji="man_shrugging">🤷♂️</span></li>
</ul></li>
</ul>
<h2 id="its-time-for-yet-another-file-embedding-tool">It’s Time for Yet Another File Embedding Tool</h2>
<ul>
<li><a href="https://github.com/klingtnet/embed">embed</a> treats <a href="https://en.wikipedia.org/wiki/Not_invented_here">Not Invented Here syndrome</a> and enforces <a href="https://en.wikipedia.org/wiki/Dogfooding">dogfooding</a></li>
<li><a href="https://gist.github.com/klingtnet/b66ecace3e87b10972245fec7e4c3fc5#file-teeny-tiny-file-embed-go">tiny prototype implementation</a> (still a single Go file)</li>
</ul>
<h3 id="design">Design</h3>
<ul>
<li><a href="https://pkg.go.dev/github.com/klingtnet/embed#Embed">very simple API</a> that provides methods for:
<ul>
<li>listing embedded files</li>
<li>get file as bytes</li>
<li>get file as string</li>
</ul></li>
<li>no error return values
<ul>
<li>if a file is not found the default value is returned (<code>[]byte{}</code> or <code>""</code>)</li>
<li>it is very easy to test if the expected files were actually embedded</li>
</ul></li>
<li><code>embed</code> generates a single <code>embeds.go</code> file that implements <a href="https://pkg.go.dev/github.com/klingtnet/embed#Embed">the API</a> (<a href="https://github.com/klingtnet/embed/blob/78fc802c58e27bec29607f32d9736611694d18e6/examples/migrate/embeds.go">example</a>)</li>
<li><a href="https://github.com/klingtnet/embed/blob/78fc802c58e27bec29607f32d9736611694d18e6/examples/migrate/embeds.go#L20">filenames and contents are stored as base64 encoded strings</a>
<ul>
<li>compression adds a lot of overhead (additional dependency, runtime cost, etc.)</li>
<li>use a <a href="https://upx.github.io/">binary packer</a> if in doubt</li>
</ul></li>
<li>provides a <a href="https://github.com/golang-migrate/migrate/">golang-migrate</a> driver</li>
</ul>
<h3 id="usage">Usage</h3>
<ul>
<li>check <a href="https://github.com/klingtnet/embed/tree/master/examples">klingtnet/embed/examples</a> or <a href="https://github.com/klingtnet/notes">klingtnet/notes</a> for a real world example</li>
</ul>
<div class="sourceCode" id="cb3"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="ex">NAME</span>:</span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a> <span class="ex">embed</span> - A new cli application</span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb3-4"><a href="#cb3-4" aria-hidden="true" tabindex="-1"></a><span class="ex">USAGE</span>:</span>
<span id="cb3-5"><a href="#cb3-5" aria-hidden="true" tabindex="-1"></a> <span class="ex">embed</span> [global options] command [command options] [arguments...]</span>
<span id="cb3-6"><a href="#cb3-6" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb3-7"><a href="#cb3-7" aria-hidden="true" tabindex="-1"></a><span class="ex">VERSION</span>:</span>
<span id="cb3-8"><a href="#cb3-8" aria-hidden="true" tabindex="-1"></a> <span class="bu">unset</span></span>
<span id="cb3-9"><a href="#cb3-9" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb3-10"><a href="#cb3-10" aria-hidden="true" tabindex="-1"></a><span class="ex">COMMANDS</span>:</span>
<span id="cb3-11"><a href="#cb3-11" aria-hidden="true" tabindex="-1"></a> <span class="bu">help</span>, h Shows a list of commands or help for one command</span>
<span id="cb3-12"><a href="#cb3-12" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb3-13"><a href="#cb3-13" aria-hidden="true" tabindex="-1"></a><span class="ex">GLOBAL</span> OPTIONS:</span>
<span id="cb3-14"><a href="#cb3-14" aria-hidden="true" tabindex="-1"></a> <span class="ex">--package</span> value, -p value name of the package the generated Go file is associated to (default: <span class="st">"main"</span>)</span>
<span id="cb3-15"><a href="#cb3-15" aria-hidden="true" tabindex="-1"></a> <span class="ex">--destination</span> value, --dest value, -d value where to store the generated Go file (default: <span class="st">"embeds.go"</span>)</span>
<span id="cb3-16"><a href="#cb3-16" aria-hidden="true" tabindex="-1"></a> <span class="ex">--include</span> value, -i value paths to embed, directories are stored recursively (can be used multiple times)</span>
<span id="cb3-17"><a href="#cb3-17" aria-hidden="true" tabindex="-1"></a> <span class="ex">--help</span>, -h show help (default: false)</span>
<span id="cb3-18"><a href="#cb3-18" aria-hidden="true" tabindex="-1"></a> <span class="ex">--version</span>, -v print the version (default: false)</span></code></pre></div>
<h2 id="draft-for-embedded-static-files">Draft for Embedded Static Files</h2>
<p><a href="https://go.googlesource.com/proposal/+/master/design/draft-embed.md">The draft</a>—published by Russ Cox and Brad Fitzpatrick in July—identified the following problems with the current situation:</p>
<ul>
<li>too many existing tools <span class="emoji" data-emoji="sweat_smile">😅</span></li>
<li>all depend on a manual generation step</li>
<li>generated file bloats git history (with a second slightly larger copy of each file)</li>
<li>not go-gettable if generated file is not checked into source control</li>
</ul>
<p>Adding support for file embedding to the <code>go</code> command will eliminate those problems.</p>
<p>Proposed changes:</p>
<ul>
<li>new <code>//go:embed</code> directive</li>
<li><code>embed</code> package containing <code>embed.Files</code> that implements <code>fs.FS</code> <sup><a href="https://golang.org/s/draft-iofs-design">file system interface draft</a></sup> which makes it directly usable with <code>net/http</code> and <code>hmtl/template</code></li>
</ul>
<p>Here’s an <a href="https://go.googlesource.com/proposal/+/master/design/draft-embed.md#go_embed-directives">example for the embed directive</a>:</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode go"><code class="sourceCode go"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="co">// content holds our static web server content.</span></span>
<span id="cb4-2"><a href="#cb4-2" aria-hidden="true" tabindex="-1"></a><span class="co">//go:embed image/* template/*</span></span>
<span id="cb4-3"><a href="#cb4-3" aria-hidden="true" tabindex="-1"></a><span class="co">//go:embed html/index.html</span></span>
<span id="cb4-4"><a href="#cb4-4" aria-hidden="true" tabindex="-1"></a><span class="kw">var</span> content embed.Files</span></code></pre></div>
<p>The <a href="https://go.googlesource.com/proposal/+/master/design/draft-embed.md">draft</a> also contains various considerations regarding:</p>
<ul>
<li><a href="https://go.googlesource.com/proposal/+/master/design/draft-embed.md#dot_dot_module-boundaries_and-file-name-restrictions">directory traversal</a>, e.g. <code>../../../etc/passwd</code> should be prevented by disallowing paths above the module root</li>
<li><a href="https://go.googlesource.com/proposal/+/master/design/draft-embed.md#compression-to-reduce-binary-size">content compression</a> is discussed but left open as an implementation detail</li>
</ul>
<p>Still curious?</p>
<ul>
<li>watch the <a href="https://golang.org/s/draft-embed-video">draft presentation video</a>, <a href="https://github.com/golang/go/issues/35950">proposal pull request</a> and its follow up <a href="https://github.com/golang/go/issues/41191">adoption pull request</a></li>
<li>there <a href="https://github.com/golang/go/issues/35950#issuecomment-561797246">was</a> also <a href="https://github.com/golang/go/issues/35950#issuecomment-561718302">the idea to just have a magic directory</a> called <code>static</code></li>
<li>should embed include <a href="https://github.com/golang/go/issues/42328#issuecomment-720169922">hidden files</a>?</li>
</ul>
</body>
</html>