Skip to content

Latest commit

 

History

History
executable file
·
140 lines (95 loc) · 4.91 KB

README.md

File metadata and controls

executable file
·
140 lines (95 loc) · 4.91 KB
Raw

fluent-plugin-mysql-replicator Build Status

Overview

Fluentd input plugin to track insert/update/delete event from MySQL database server.
Not only that, it could multiple table replication into single or multi Elasticsearch/Solr.
It's comming support replicate to another RDB/noSQL.

Requirements

fluent-plugin-mysql-replicator fluentd ruby
>= 0.6.1 >= v0.14.x >= 2.1
<= 0.6.1 >= v0.12.x >= 1.9

Dependency

before use, install dependent library as:

# for RHEL/CentOS
$ sudo yum group install "Development Tools"

# for Ubuntu/Debian
$ sudo apt-get install build-essential

Installation

install with gem or fluent-gem command as:

# for system installed fluentd
$ gem install fluent-plugin-mysql-replicator -v 1.0.3

# for td-agent2
$ sudo td-agent-gem install fluent-plugin-mysql-replicator -v 0.6.1

# for td-agent3
$ sudo td-agent-gem install fluent-plugin-mysql-replicator -v 1.0.3

Included plugins

  • Input Plugin: mysql_replicator
  • Input Plugin: mysql_replicator_multi
  • Output Plugin: mysql_replicator_elasticsearch
  • Output Plugin: mysql_replicator_solr (experimental)

Output example

It is a example when detecting insert/update/delete events.

sample query

$ mysql -e "create database myweb"
$ mysql myweb -e "create table search_test(id int auto_increment, text text, PRIMARY KEY (id))"
$ sleep 10
$ mysql myweb -e "insert into search_test(text) values('aaa')"
$ sleep 10
$ mysql myweb -e "update search_test set text='bbb' where text = 'aaa'"
$ sleep 10
$ mysql myweb -e "delete from search_test where text='bbb'"

result

$ tail -f /var/log/td-agent/td-agent.log
2013-11-25 18:22:25 +0900 replicator.myweb.search_test.insert.id: {"id":"1","text":"aaa"}
2013-11-25 18:22:35 +0900 replicator.myweb.search_test.update.id: {"id":"1","text":"bbb"}
2013-11-25 18:22:45 +0900 replicator.myweb.search_test.delete.id: {"id":"1"}

Tutorial

mysql_replicator

It is easy to try it on this plugin quickly.
For more detail are described at Tutorial-mysql_replicator.md

Features

  • Table (or view table) synchronization supported.
  • Replicate small record under a millons table.
  • It is recommend to use insert only table.
  • Nested documents are supported with placeholder which accessing to temporary table created at the each loop.

Examples

mysql_replicator_multi

It replicates a millions of records and/or multiple tables with multiple threads.
This architecture is storing hash table in MySQL management table instead of ruby internal memory.
See tutorial at Tutorial-mysql_replicator_multi.md

Features

  • table (or view table) synchronization supported.
  • Multiple table synchronization supported and its DSN stored in MySQL management table.
  • Using MySQL database as hash table cache to support replicate over a millions table.
  • It is recommend to make whole copy of tables.
  • Nested documents are supported with placeholder which accessing to temporary table created at the each loop.

Examples

Articles

TODO

Pull requests are very welcome like below!!

  • more documents
  • more tests with mock.
  • support string type of primary_key.
  • support reload setting on demand.

Copyright

Copyright © 2013- Kentaro Yoshida (@yoshi_ken)

License

Apache License, Version 2.0