notes.txt

Notes for The Guide

- pubsub ordering
    - N-to-N ordering guarantees
    - single-stream using a broker


- collection of patterns
    Least Recently Used
    Asynchronous Client-Server
    Suicidal Snail
    Lazy Pirate
    Simple Pirate
    Paranoid Pirate
    Clone

- how to make an error logging console

- how to make a TCP-to-0MQ bridge, with examples
http://lists.zeromq.org/pipermail/zeromq-dev/2011-March/010186.html
- FD_EVENT integration


- router trees
 (federation)

- cross-over router
    - send to, send from
    - flips addresses, gives you reply model

    //  If socket is a ROUTER, get identity first
    int socket_type;
    size_t type_size = sizeof (socket_type);
    zmq_getsockopt (socket, ZMQ_TYPE, &socket_type, &type_size);
    if (socket_type == ZMQ_ROUTER) {
        zmq_msg_t mesg;
        zmq_msg_init (&mesg);
        zmq_recv (socket, &mesg, 0);
    }

- explain why callbacks are a pita
    - what thread do they run in?
    - how do you connect them to main app thread
    - how do you wait, do you sleep, call something, etc.?

- integrating 0MQ into TCP loops: http://lists.zeromq.org/pipermail/zeromq-dev/2011-March/010186.html

Then we can use this to start building some reusable pieces:

- timer device
- name service
- custom load-balancing
- custom publish-subscribe
- stateful publish-subscribe
- reliable request-reply
- logging device


+++ Presence Detection

- peers come and go
- if we want to route to them explicitly, we need to know if they are present
- heartbeating, liveness, etc.
- fresh data, failover, etc.
- purging old identities from routing tables
- example of eight robots and console
- robots come and go...


+++ A ZeroMQ Name Service

- name service
    translate logical name into connect or bind string
    service runs locally, connect via ipc://zns
    gets name updates asynchronously from central server
    also local zns lookup file
    using zpl syntax
    pubsub state / refresh example

how to map names?
    - XXX -> tcp://lo:5050 if I'm on server 1
    - XXX -> tcp://somename:5050
    -> does ZMQ do host lookup?  Doesn't seem like it...
    -> resolve host...


+++ Pipelines

- does PUSH block if there are no PULL sockets ready?
- how do maintain a queue?

+++ File Transfer

example of file transfer

+++ Generating Identities


+++ Setting Queue Limits

- setting queue limits prevents nodes from overflowing memory
- by default 0MQ does not set any limits
- example of application that will overflow memory
    - publish to subscribe socket but don't read
- now fix this by setting queue limit, ZMQ_HWM
- actual behaviour with automagic LWM
- how this works on different socket types
    - 'exception' on each socket type, from man page
- adding capacity for disk offload, ZMQ_SWAP
- creating persistent queues using identity
    - example of HWM 1 plus massive SWAP


+++ Reliable Request-Reply

We'll create a reliable request-reply application that uses XREQ and XREP and a simple resend mechanism.  When this works between two peers we'll show how it scales across a request-reply broker to effectively create edge-to-edge reliability.  We'll also open up the message format for request-reply and explore identities in horrible detail.

+++ Configuration Distribution

We'll look at how to dynamically configure a network of devices using a central configuration broker.

+++ Logging Subsystem

many applications
many subscribers
broker in the middle
persistent logfiles
replay via subscribe socket


+++ Failover and Recovery

We'll look at how to handle crashes in a distributed architecture.  For each pattern there is an ideal architecture, and we'll explore each of these with worked examples.

+++ Encrypted Publish-Subscribe

We'll look at how to secure pubsub data against snooping.  The actual technique involves out-of-band exchange of keys, symmetric encryption, and a broker that helps the thing along.  Hopefully all fairly easy to make, as usual, using 0MQ.

+++ Building a Multicast Bus

We'll now look at how the pgm: and epgm: protocols work.  With pgm:, the network switch basically acts as a hardware FORWARDER device.

++++ Customized Publish-Subscribe

- use identity to route message explicitly to A or B
- not using PUBSUB at all but XREP/????
    - limitations: no multicast, only TCP
    - how to scale with devices...

When a client activates, it chooses a random port that is not in use and creates a SUB socket listening for all traffic on it. The client then sends a message via REQ to the publisher containing the port that it is listening on. The publisher receives this message, acknowledges it, and creates a new pub socket specific to that client. All published events specific to this client go out that socket.

When the client deactivates, it sends a message to the publisher with the port to deactivate and close.

You end up creating a lot more PUB sockets on your server end and doing all of the filtering at the server. This sounds acceptable to you.

I didn't need to do this to avoid network bandwidth bottlenecks; I created this to enforce some security and entitlements.

+++ A Clock Device

We'll look at various ways of building a timer into a network.   A clock device sends out a signal (a message) at more or less precise intervals so that other nodes can use these signals for internal timing.

+++ Serializing Data

Examples of using Protocol Buffers and other options.

[!--
- ipc://name
 - connects two processes on a single box
 - supports all messaging patterns
 - typical use case is for multithreading apps
 - runs on Unix domain sockets (not available on Windows, OpenVMS)
 - permissions issues:

> Since I want to work under /tmp, this all had to be done programatically. My
> server now mkdir -p's a socket subdirectory and chmod 777's it. The server
> creates and binds the socket in that folder, and then chmod 777's it. The
> server must be run as root (which is fine for my project luckily). If it is
> run as a normal user, the client's still timeout.

- tcp://ipaddress:port
 - bind to address:port
 - bind to *:5555
 - localhost
  - also bind to interface: lo:port, eth1:port, etc.
  - Linux: eth1, eth2, eth3
  - Mac OS X: en1, en2, en3
  - Solaris: e1000g, etc.
 - connect to remote address: host:port

- pgm://address;multicastgroup:port
 - address can be interface name
 - requires decent hardware support
 - means enterprise level switches with IGNP snooping
 - some routers also support PGM
 - runs over IP, requires root privileges
 - more standard
 - rate-limited protocol, sender must define bandwidth
 - pgm is currently broken

- epgm://address;multicastgroup:port
 - encapsulated in UDP packets
 - requires decent hardware support
 - does not require root access
 - non-standard to pgm

- add peer example
   - exclusive lock on peer
   - for owning other party
   - solve reverse connection
 - e.g. to cross firewall
 - you need to add a bind to allow the client to accept a connection
 - could be usecase for EXCLUSIVE socket

XREQ is like PUSH+PULL, XREP is like PUSH+PULL+routing

--]


* How do we tunnel 0MQ connections over 0MQ connections?
    - e.g. to get 3-4 ports into DMZ via one single port
    - two devices, one reads, one writes
    - over other protocols: HTTP, SSH, etc...?
    - acts as split device, ...

I highly recommend that you try out the simpler topology and *verifying* that 0mq cannot keep up with your message rates when publishing all data to all clients. With smart topics the client can reject the data *very* fast. A 1 GB or 10 Gb switched network can also move quite a bit of data without a lot of blocking behavior in the hardware too. You may be able to save yourself a lot of unnecessary work, so just try it.



- explain in
- debugging message flow
    - using a queue device that prints out message parts
    - debugging versions of devices...

- heartbeating
    - set HWM to 1 message only

[!--

[!--
+++ The Wire-Level Protocol

- writing a thin client in JavaScript

+++ Building a Language Binding

+++ Tuning 0MQ

High performance multiple i/o threads
 - for HP applications, several threads
 - then, set affinity on sockets

+++ Contributing to 0MQ


+++ What's Missing from 0MQ



- handling crashing peers
- non-blocking send/recv
    EAGAIN
- reliable pub
    - pub connects to all subs
    - subs send request to pub
- reliable messaging over xreq / xrep
- how to do timer events
- there is no way to disconnect; second connection creates two
    - need to destroy the socket
- how to send multipart job and execute / cancel




Here is, I think, how to build an authenticated pubsub service using ØMQ.  The design handles authentication, arbitrary routing criteria, and flushing of dead client connections.

You will use ZMQ_XREQ at the client side and ZMQ_XREP at the service side.  There is a simple protocol between client and server that is loosely modelled on HTTP:

* Client sends messages containing: identity (ØMQ adds this automatically), authentication credentials, and subscription criteria.
* Server sends messages containing: identity (automatic), error/success code, and content.

The server has two main data structures:

* A connection table that stores authenticated identities and timestamps.  Each identity corresponds to a client connection.
* A routing table that maps message keys (e.g. topic value) to identities.  There are techniques for doing [http://www.zeromq.org/whitepapers:message-matching high speed message matching].



0MQ Quickstarter

- building & installing
- performance for language
- basic examples
- socket types
- transports
- main patterns
- problem solving
-> translated into different programming languages