-
Notifications
You must be signed in to change notification settings - Fork 16
/
Copy pathindex.html
491 lines (453 loc) · 20.7 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
<html>
<head>
<meta charset="utf-8" />
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="" />
<meta name="author" content="Maxim Sokhatsky" />
<title>N2O 4.5 MQTT</title>
<link rel="stylesheet" href="//n2o.space/new.css" />
</head>
<body>
<nav>
<a href='http://erlang.org/'>ERLANG</a>
<a href='http://emqtt.io/'>EMQ</a>
<a href='//n2o.space'>N2O</a>
</nav>
<header>
<h1>N2O Apps over<br>MQ Telemetry Transport
</header>
<aside>
<article>
<section>
<h3>Motivation</h3>
<div>
Introduce N2O Protocols for commercial use on top of the MQTT protocol.
Minimize and remove all features, duplicated by the MQTT pub-sub broker.
Provide an EMQ extension that immediately activates the N2O protocol
and its applications for all connected MQTT devices. Create a single
Erlang eco-system for Enterprise Protocol Federation and establish
a solid CORBA/WS-SOAP/XMPP-replacement, ready for high-speed,
low-latency IoT applications.
</div>
</section>
<section>
<h3>Credits</h3>
<div>Maxim Sokhatsky, Synrc Research Center</div>
<div>Andy Martemyanov, Erlang One</div>
<div>Feng Lee, EMQ Enterprise</div>
<h3>Sources</h3>
<div>
<a href="https://github.com/synrc/n2o"><img src="//n2o.space/img/github.png"> synrc/n2o</a>
— N2O 2.9—4.4 FULL
</div>
<div>
<a href="https://github.com/synrc/mqtt"><img src="//n2o.space/img/github.png"> synrc/mqtt</a>
— N2O 4.5 MQTT
</div>
<div>
<a href="https://github.com/voxoz/mq"><img src="//n2o.space/img/github.png"> voxoz/mq</a>
— EMQ, EMQ Dashboard, REVIEW
</div>
<div>
<a href="https://github.com/5HT/5ht.co/blob/master/mqtt.htm">
<img src="//n2o.space/img/github.png"> edit</a> — if you've found any errors here
</div>
<br>
<div>
APRIL 26, 2017 © <a href="http://5ht.co/license.htm">DHARMA LICENSE</a>
</div>
</section>
</article>
</aside>
<main>
<section>
<a name=intro><h3>Web Framework w/o Web Server?</h3></a>
<p>
Usually, Web Framework uses HTTP or WebSocket endpoints.
But what if we need to connect application on devices with TCP/UDP transports?
One of the protocols addressing this usecase is <a href="http://www.indigoo.com/dox/wsmw/1_Middleware/MQTT.pdf">MQTT</a> that was designed by Andy Stanford-Clark (IBM) and Arlen Nipper in 1999 for connecting Oil Pipeline telemetry systems over satellite.
The protocol is designed to work over connectionless
unreliable environments, and provides three levels of
QoS delivery: at least once, at most once, exactly once.
</p>
<p>
On the other hand, N2O is a federation of protocols,
like XMPP/CORBA/WS. Although N2O supports several
encoders (BERT/XML/JSON/TEXT) the extension to other
channels beyond WebSockets would involve creating a
new Socket Server. So we have decided to take MQTT
as a transport/session protocol for N2O Federation.
</p>
<p>
EMQ is an open-source MQTT broker implementation by Feng Lee.
It's clean, concise and robust. We've developed an EMQ compatible
plugin that works entirely within MQTT sessions just like N2O
already does within Cowboy's ranch sessions.
This is the main difference from Phoenix's gen_server channels.
</p>
<figure>
<img src="//n2o.space/img/mqtt.svg">
<figcaption>Picture 1. Protocol Modules</figcaption>
</figure>
<figure>
<img src="//n2o.space/img/n2o-mq.png">
<figcaption>Picture 2. Protocol Messages</figcaption>
</figure>
</section>
<section>
<a name=modules><h3>N2O: Protocol Server for MQTT</h3></a>
<p>
N2O 4.5 MQTT is what you get when you reduce everything duplicating MQTT features.
On the other hand, 4.5 is a completely N2O-compatible embeddable protocol relay
with limited features. And it takes only 25KB in Erlang.
</p>
<ul>
<li><b><a href="//n2o.space/mqtt/n2o.htm">n2o</a></b> — N2O Protocol Server version 4.5 MQTT — 10KB</li>
<li><b><a href="//n2o.space/mqtt/n2o_async.htm">n2o_async</a></b> — N2O Async Streams: gen_server — 4KB</li>
<li><b><a href="//n2o.space/mqtt/n2o_format.htm">n2o_format</a></b> — N2O Formatter: JSON, BERT — 2KB</li>
<li><b><a href="//n2o.space/mqtt/n2o_ftp.htm">n2o_ftp</a></b> — N2O File Protocol: FTP — 4KB</li>
<li><b><a href="//n2o.space/mqtt/n2o_nitro.htm">n2o_nitro</a></b> — N2O Nitro Protocol: PICKLE, FLUSH, DIRECT, IO, INIT — 3KB</li>
<li><b><a href="//n2o.space/mqtt/n2o_proto.htm">n2o_proto</a></b> — N2O Protocols Loop: NITRO, FTP — 1KB</li>
<li><b>n2o_secret</b> — N2O Security: HMAC AES/CBC-128 — 1KB</li>
</ul>
<p>N2O 4.5 provides real smooth experience for developers:
clean git history, small codebase, nitro compatible,
man/html docs.</p>
<figure>
<figcaption>Listing 1. N2O 4.5 MQTT</figcaption>
<pre style="width:700px">
$ time git clone git://github.com/synrc/mqtt
real 0m0.794s
$ cloc .
370
$ cat rebar.config
{deps, [{nitro, "git://github.com/synrc/nitro"},
{n2o, "git://github.com/synrc/mqtt"}]}.
$ man ./man/n2o.1
</pre></figure>
</section>
<section>
<a name=""><h3>N2O things deprecated due to MQTT</h3></a>
<p>
N2O_start and n2o.js are no longer used in MQTT version of N2O.
Instead of N2O_start, MQTT_start and mqtt.js should be used for
session control replacement. We traded HEART protocol and session
facilities for built-in MQTT features. N2O authentication and
authorization mechanisms are also abandoned as MQTT could provide
AUTH features too. Obviously <b>wf:reg</b> and <b>wf:send</b> APIs
have also been abandoned as we can use emqttd API directly and
<b>{deliver,_}</b> protocol of ws_client gen_server.
</p>
<ul>
<li><b>n2o_session</b> — no Browser, so no Cookies are needed</li>
<li><b>n2o_stream</b> — no XHR fallback needed</li>
<li><b>n2o_heart</b> — no PING protocol needed</li>
<li><b>n2o_mq</b> — syn and gproc are no longer needed</li>
<li><b>n2o_query</b> — no Query Router</li>
<li><b>N2O_start</b> — replaced by MQTT_start</li>
<li><b>ranch</b> — replaced with esockd</li>
<li><b>cowboy</b> — replaced with mochiweb for WebSocket endpoint</li>
</ul>
<p>
<img src="//n2o.space/img/pdf.jpg" width=50/>
<a href="http://synrc.com/apps/n2o/doc/book.pdf">N2O.PDF</a>
</p>
</section>
<section>
<a name=endpoints><h3>EMQ/Cowboy/Mochiweb Endpoints</h3></a>
<p>
All N2O/MQTT messages which go directly to TCP and/or WebSocket.
However there are some endpoints which are not TCP sockets, even non-sockets,
like gen_server endpoint, HTTP REST endpoint, which is non-WebSocket endpoint,
and there is a room for other endpoint types.
</p>
<p>
Here is a list of types of endpoints which are supported by EMQ and accesible to N2O apps:
WebSockets, MQTT, MQTT-SN, TCP, UDP, CoAP.
Normal use of N2O as a Web Framework or a Web Application Server is through WebSockets,
but for IoT and MQTT applications it could be served through UDP or SCTP protocols
providing application level message delivery consistency.
By using MQTT as a transport we extend the supported set of endpoint protocols.
</p>
<p>
Also N2O is able to work under <b>cowboy</b> Erlang web server and <b>mochiweb</b> web server.
There is <b>rest</b> library to deal with REST endpoint.
</p>
</section>
<section>
<a name=sessions><h3>MQTT Sessions</h3></a>
<p>
N2O is working entirely in the context of <b>ws_client</b> processes of EMQ, just as it is working on top of ranch processes of cowboy.
No additional <b>gen_server</b> is being introduced.
</p>
<p>
The only official transparent way with zero abstraction is to use EMQ hooks mechanism.
For N2O we need to implement only two cases: <b>client.subscribe</b> and <b>message.delivered</b>.
On <b>client.subscribe</b> we deliver all persistent data that are ready for the client.
On <b>message.delivered</b> we unpack any N2O BERT protocol message inside MQTT session.
</p>
</section>
<section>
<a name=formatters><h3>N2O and MQTT Formatters</h3></a>
<p>
N2O supports <b>TEXT</b>, <b>JSON</b>, <b>XML</b>, <b>BERT</b>,
<b>MessagePack</b> formatters but you can add your own implementations either.
They are terminal formatters before socket interface library (usually <b>ranch</b> or <b>esockd</b>).
For example one may format IO message with BERT encoding <b>n2o:format({io,Eval,Data},bert).</b>
</p>
<p>MQTT format messages only in MQTT format. You can wrap binary B in MQTT packet as
<b>emqttd_message:make(Name, 0, Name, B).</b>
</section>
<section>
<a name=sample><h3>Sample Application</h3></a>
<figure>
<figcaption>Listing 2. Sample Web Application written in NITRO DSL</figcaption>
<code>
main() -> [].
event(init) ->
nitro:update(loginButton,
#button{id=loginButton,body="Login",
postback=login,source=[user,pass]});
event(login) ->
nitro:redirect("index.htm?room=" ++
nitro:to_list(
n2o:q(pass)));
</code>
</figure>
<figure>
<figcaption>Picture 3. N2O Review Application</figcaption>
<img src="//n2o.space/img/n2o-mqtt.png">
</figure>
</section>
<section>
<a name=plugin><h3>EMQ Plugin Implementation</h3></a>
<p>
Here you will see the implementation of N2O bridge for EMQ.
We will inject N2O loop inside EMQ session handlers. As you may know,
the single point of entrance in N2O is the <b>event(init)</b> message handler.
It can only be reached by calling <b>{init,_}</b> message of NITRO protocol.
</p>
<figure>
<figcaption>Listing 3. Emulating "N2O," init handshake</figcaption>
<code>
on_client_subscribe(ClientId, Username, TopicTable, _Env) ->
Name = binary_to_list(iolist_to_binary(ClientId)),
BinTopic = element(1,hd(TopicTable)),
put(topic,BinTopic),
n2o_cx:context(#cx{module=route:select(BinTopic),
formatter=bert,params=[]}),
case n2o_proto:info({init,<<>>},[],?CTX) of
{reply, {binary, M}, _, #cx{}} ->
self() ! {deliver, emqttd_message:make(Name, 0, Name, M)};
_ -> skip end,
{ok, TopicTable}.
</code>
</figure>
<p>
However N2O MQTT is a protocol federation so we need to handle not only N2O messages, but also KVS, ROSTER, BPE, REST protocols.
Thus after initialization during <b>client.subscribe</b> we call <b>n2o_proto:info</b> — the entire N2O protocol chain recursor inside <b>message.delivered</b> hook.
This is the best place to put the federation relay for N2O modules.
</p>
<figure>
<figcaption>Listing 4. Emulating N2O message delivery</figcaption>
<code>
on_message_delivered(ClientId, Username,
Message = #mqtt_message{topic = Topic,
payload = Payload}, _Env) ->
Name = binary_to_list(ClientId),
case n2o_proto:info(binary_to_term(Payload),[],?CTX) of
{reply, {binary, M}, R, #cx{}} ->
case binary_to_term(M) of
{io,X,_} -> Msg = emqttd_message:make(Name, 0, Name, M),
self() ! {deliver, Msg},
{ok, Message};
_ -> {ok, Message} end;
_ -> {ok, Message} end.
</code>
</figure>
</section>
<section>
<a name=js><h3>MQTT JavaScript Client</h3></a>
<figure>
<figcaption>Listing 5. mq.js</figcaption>
<code>
var topic = module + "_" + params.room || "lobby",
mqtt = new Paho.MQTT.Client("127.0.0.1", 8083, ''),
subscribeOptions = {
qos: 0,
onSuccess: function () {
console.log("N2O Subscribed"); },
onFailure: function (m) {
console.log("SUB Failure: " +
message.errorMessage); },
timeout: 3 },
options = {
timeout: 3,
onFailure: function (m) {
console.log("SND Failure: " +
m.errorMessage); },
onSuccess: function () {
console.log("N2O Connected");
mqtt.subscribe(topic, subscribeOptions); } },
ws = {
send: function (payload) {
var message = new Paho.MQTT.Message(payload);
message.destinationName = topic;
message.qos = 0;
mqtt.send(message); } };
function MQTT_start() {
mqtt.onConnectionLost = function (o) { };
mqtt.onMessageArrived = function (m) {
var BERT = m.payloadBytes.buffer.slice(
m.payloadBytes.byteOffset,
m.payloadBytes.byteOffset +
m.payloadBytes.length);
try { erlang = dec(BERT);
console.log(utf8_dec(erlang.v[1].v));
for (var i=0;i<$bert.protos.length;i++) {
p = $bert.protos[i];
if (p.on(erlang, p.do).status == "ok") return; }
} catch (e) { console.log(e); } };
mqtt.connect(options);
}
MQTT_start();
</code>
</figure>
</section>
<section>
<a name=dash><h3>Enable EMQ Plugin</h3></a>
<p>
The N2O over MQTT bridge is a drop-in plugin for EMQ broker.
After starting the server you should enable N2O over MQTT bridge EMQ plugin on plugin page
<a href="http://127.0.0.1:18083/#/plugins">http://127.0.0.1:18083/#/plugins</a> and then open the application sample
<a href="http://127.0.0.1:8000/spa/login.htm">http://127.0.0.1:8000/spa/login.htm</a>
</p>
<figure>
<figcaption>Picture 4. N2O 4.5 MQTT as EMQ Plugin</figcaption>
<img src="//n2o.space/img/emq.png">
</figure>
<p>
The nice thing about EMQ is that it enables session introspection for N2O connections.
The EMQ Dashboards is written in vue.js and a bit complex for admin application.
The possible and expecting hackaton is on rewriting EMQ admin with <b>hyperapp</b> and <b>n2o</b>.
The IBM library Paho is also too old and fat to fit N2O sizes. Some parts should be replaced
with more modern and compact pinger and connection respawning from <b>n2o.js</b>. This is
expecting further research and imporevement.
</p>
<figure>
<figcaption>Picture 5. Inspecting N2O sessions with EMQ</figcaption>
<img src="//n2o.space/img/emq-sub.png">
</figure>
</section>
<section>
<a><h3>EMQ — Lightweight Broker as Service Container for N2O Apps</h3></a>
<p>
EMQ is the latest software that has robust iOS and Android MQTT client libraries, just have a look at
<a href="https://github.com/emqtt">https://github.com/emqtt</a> Github organization.
</p>
<figure>
<figcaption>Picture 6. MQTT 3.1.1 vs MQTT 5.0</figcaption>
<img src="//n2o.space/img/mqtt-v5.svg">
</figure>
<p>
We made a compatible EMQ plugin and N2O Review sample application with built-in EMQ with minimal dependencies.
Our fork is based on the latest EMQ master and is dedicated for building with our MAD build tool.
</p>
<figure>
<figcaption>Listing 6. Setup MQTT flavoured N2O.</figcaption>
<pre>
$ brew install erlang
$ curl -fsSL https://raw.github.com/synrc/mad/master/mad > mad \
&& chmod +x mad \
&& sudo cp mad /usr/local/bin
$ git clone git://github.com/voxoz/mq && cd mq
$ time mad dep com
Writing /apps/review/ebin/review.app
OK
real 1m45.357s
user 0m17.166s
sys 0m5.065s
$ mad rep
Configuration: [{n2o,
[{port,8000},
{app,review},
{pickler,n2o_secret},
{formatter,bert},
{log_modules,config},
{log_level,config}]},
{emq_dashboard,
[{listeners_dash,
[{http,18083,[{acceptors,4},{max_clients,512}]}]}]},
{emqttd,
[{listeners,
[{http,8083,[{acceptors,4},{max_clients,512}]},
{tcp,1883,[{acceptors,4},{max_clients,512}]}]},
{sysmon,
[{long_gc,false},
{long_schedule,240},
{large_heap,8000000},
{busy_port,false},
{busy_dist_port,true}]},
{session,
[{upgrade_qos,off},
{max_inflight,32},
{retry_interval,20},
{max_awaiting_rel,100},
{await_rel_timeout,20},
{enable_stats,off}]},
{queue,[]},
{allow_anonymous,true},
{protocol,
[{max_clientid_len,1024},{max_packet_size,64000}]},
{acl_file,"etc/acl.conf"},
{plugins_etc_dir,"etc/plugins/"},
{plugins_loaded_file,"etc/loaded_plugins"},
{pubsub,
[{pool_size,8},{by_clientid,true},{async,true}]}]},
{kvs,
[{dba,store_mnesia},
{schema,[kvs_user,kvs_acl,kvs_feed,kvs_subscription]}]}]
Applications: [kernel,stdlib,gproc,lager_syslog,pbkdf2,asn1,fs,ranch,mnesia,
compiler,inets,crypto,syntax_tools,xmerl,gen_logger,esockd,
cowlib,goldrush,public_key,lager,ssl,cowboy,mochiweb,emqttd,
erlydtl,kvs,mad,emqttc,nitro,rest,sh,syslog,review]
Erlang/OTP 19 [erts-8.3] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
Eshell V8.3 (abort with ^G)
starting emqttd on node 'nonode@nohost'
Nonexistent: []
Plugins: [{mqtt_plugin,emq_auth_username,"2.1.1",
"Authentication with Username/Password",false},
{mqtt_plugin,emq_dashboard,"2.1.1","EMQ Web Dashboard",false},
{mqtt_plugin,emq_modules,"2.1.1","EMQ Modules",false},
{mqtt_plugin,n2o,"4.5-mqtt","N2O Server",false}]
Names: [emq_dashboard,n2o]
dashboard:http listen on 0.0.0.0:18083 with 4 acceptors.
Async Start Attempt {handler,"timer",n2o,system,n2o,[],[]}
Proc Init: init
mqtt:ws listen on 0.0.0.0:8083 with 4 acceptors.
mqtt:tcp listen on 0.0.0.0:1883 with 4 acceptors.
emqttd 2.1.1 is running now
>
</pre>
</figure>
</section>
<section>
<a name=plugin><h3>Related Documents</h3></a>
<div>
<ul>
<li><a href="http://5ht.co/n2o.htm">N2O: PROTOCOL</a></li>
<li><a href="http://5ht.co/ftp.htm">N2O: FTP</a></li>
<li><a href="http://5ht.co/web.htm">N2O: SERVER</a></li>
<li><a href="http://5ht.co/history.htm">N2O: HISTORY</a></li>
<li><a href="http://5ht.co/spawnproc.htm">N2O: PRIVATBANK</a></li>
<li><a href="http://5ht.co/bpe.htm">BPE: BUSINESS ENGINE</a></li>
<li><a href="http://synrc.com/apps/cr/doc/cr.htm">CR: DATABASE</a></li>
</ul></div>
</section>
</main>
<footer>
Made with <span class="heart">❤</span> for EMQ and N2O
</footer>
</body>
</html>