Rethinker Logstash input plugin

This is a plugin for Logstash.

Currently this is BETA software. It contains some known limitations (see below)

Using the plugin

You'll need to use this with the Logstash 5.2.x or higher, which you can download here.

• Update

I pushed this repo to rubygems.org, now you should be able to install directly like this:

$ bin/plugin install logstash-input-rethinker

• Now you can test the plugin using the stdout output:

$ bin/logstash -f rethinker.conf

This will immediately watch the tables test.foo and db2.baz, and it will also watch the databases db1 and db2 for new or dropped tables and watch or unwatch those tables appropriately. Since backfill is true, it will automatically send events for the documents that already exist in those tables during initialization.

If you want to tweak it, install the plugin from the Logstash home directory (you will need jruby to bundle install)

$ git clone https://github.com/wayann/logstash-input-rethinker
$ cd logstash-input-rethinker
$ bundle install
$ gem build logstash-input-rethinker.gemspec
$ bin/plugin install --no-verify logstash-input-rethinker-0.1.0.gem

Format of the events:

The events are encoded with the "json_lines" codec, which puts compressed json documents one event per line

Fields:

• db: the database that emitted the event
• table: the table that emitted the event
• old_val: the old value of the document (see changefeeds)
• new_val: the new value of the document
• @timestamp: timestamp added by logstash (so may not correspond to the rethinkdb server time the change was emitted)
• @version: version number added by logstash (always 1)

Known limitations

There are two limitations that should be known by anyone using this in production systems:

1. Until RethinkDB supports resuming changefeeds, this plugin cannot guarantee that no changes are missed if a connection to the database is dropped. Again, once that functionality is implemented, this plugin will be modified to provide reliable "at least once" semantics for changes (meaning once it reconnects, it can catch back up and send any changes that it missed).
2. Documents that are deleted in RethinkDB while the LogStash plugin is disconnected will not be synchronized. This is true even if backfill is enabled. This limitation is a consequence of LogStash operating on a document-by-document basis.

License

Apache 2.0