Bugfixing, new configuration, health, hot flushing, rate limiting, fi…

…ltering

CHANGELOG.md

@@ -29,6 +29,13 @@ Nov 15 2018 Maxime CORBIN <[email protected]>
     * Add `-version` option
 
 Dec 13 2018 Maxime CORBIN <[email protected]>
-    * Added `/admin` endpoint that can be used to create or remove databases
-    * Added `/health` endpoint that can be used to monitor the health status of every backend
-    * Fixed some performance bugs & added a few more logs
+    * Add `/admin` endpoint that can be used to create or remove databases
+    * Add `/health` endpoint that can be used to monitor the health status of every backend
+    * Fixed some performance bugs & added a few more logs
+
+Dec 27 2018 Clément CORNUT <[email protected]>
+    * Add `/admin/flush` endpoint that can be used to flush internal buffer
+    * Add Rate limiting on backend
+    * Add Filters for tags and measurements
+    * New configuration formatting
+    * Various bug fixes

Gopkg.toml

@@ -9,3 +9,7 @@
 [prune]
   go-tests = true
   unused-packages = true
+
+[[constraint]]
+  branch = "master"
+  name = "golang.org/x/time"

PACKAGING.md

@@ -0,0 +1,32 @@
+# Building system packages
+
+To build system packages for Linux (`deb`, `rpm`, etc), use the included
+`scripts/build.py` script.
+
+```sh
+$ # Get all the code and dependencies into a temporary $GOPATH
+$ export GOPATH=/tmp/go
+$ go get -d -u github.com/vente-privee/influxdb-relay
+$ cd $GOPATH/src/github.com/vente-privee/influxdb-relay
+$ scripts/build.py
+```
+
+The packages will be found under
+`$GOPATH/src/github.com/vente-privee/influxdb-relay/build`
+
+For more build options, check the build script arguments:
+```sh
+$ scripts/build.py -h
+```
+
+For example, to build versioned packages use:
+```sh
+$ scripts/build.py --package --version 2.3.0
+```
+
+## Dependencies
+
+1. Go 1.5+ is required. Use https://go-repo.io/ to install on CentOS/Fedora.
+2. FPM Ruby gem is needed - use
+https://fpm.readthedocs.io/en/latest/installing.html as an installation guide
+for Fedora/CentOS and Debian/Ubuntu.

README.md

@@ -87,6 +87,7 @@ You can find more documentation in [docs](docs) folder.
 * [Buffering](docs/buffering.md)
 * [Caveats](docs/caveats.md)
 * [Recovery](docs/recovery.md)
+* [Filters](docs/filters.md)
 * [Sharding](docs/sharding.md)
 
 You can find some configurations in [examples](examples) folder.
@@ -105,29 +106,60 @@ name = "example-http"
 # TCP address to bind to, for HTTP server.
 bind-addr = "127.0.0.1:9096"
 
+# Timeout for /health route
+# After this time, the host may be considered down
+health-timeout-ms = 10000
+
+# Request limiting (Applied to all backend)
+rate-limit = 5
+burst-limit = 10
+
 # Ping response code, default is 204
 default-ping-response = 200
 
 # Enable HTTPS requests.
 ssl-combined-pem = "/path/to/influxdb-relay.pem"
 
-# Array of InfluxDB instances to use as backends for Relay.
-output = [
-    # name: name of the backend, used for display purposes only.
-    # location: full URL of the /write endpoint of the backend
-    # admin: Partial URL of the backend, without any path
-    # timeout: Go-parseable time duration. Fail writes if incomplete in this time.
-    # skip-tls-verification: skip verification for HTTPS location. WARNING: it's insecure. Don't use in production.
-    # type: type of input source. OPTIONAL: see below for more information.
-
-    # InfluxDB
-    { name="local-influxdb01", location="http://127.0.0.1:8086/write", timeout="10s", type="influxdb" },
-    { name="local-influxdb02", location="http://127.0.0.1:7086/write", timeout="10s", type="influxdb" },
-
-    # Prometheus
-    { name="local-influxdb03", location="http://127.0.0.1:6086/api/v1/prom/write", timeout="10s", type="prometheus" },
-    { name="local-influxdb05", location="http://127.0.0.1:5086/api/v1/prom/write", timeout="10s", type="prometheus" },
-]
+# InfluxDB instances to use as backend for Relay
+[[http.output]]
+# name: name of the backend, used for display purposes only.
+name="local-influxdb01"
+
+# location: full URL of the /write endpoint of the backend
+location="http://127.0.0.1:8086/"
+
+# endpoints: Routes to use on Relay
+# write: Route for standard InfluxDB request
+# write_prom: Route for Prometheus request
+# ping: Route for ping request
+# query: Route fot querying InfluxDB backends
+endpoints = {write="/write", write_prom="/api/v1/prom/write", ping="/ping", query="/query"}
+
+# timeout: Go-parseable time duration. Fail writes if incomplete in this time.
+timeout="10s"
+
+# skip-tls-verification: skip verification for HTTPS location. WARNING: it's insecure. Don't use in production.
+skip-tls-verification = false
+
+# InfluxDB
+[[http.output]]
+name="local-influxdb02"
+location="http://127.0.0.1:7086/"
+endpoints = {write="/write", ping="/ping", query="/query"}
+timeout="10s"
+
+# Prometheus
+[[http.output]]
+name="local-influxdb03"
+location="http://127.0.0.1:6086/"
+endpoints = {write="/write", write_prom="/api/v1/prom/write", ping="/ping", query="/query"}
+timeout="10s"
+
+[[http.output]]
+name="local-influxdb05"
+location="http://127.0.0.1:5086/"
+endpoints = {write="/write", write_prom="/api/v1/prom/write", ping="/ping", query="/query"}
+timeout="10s"
 
 [[udp]]
 # Name of the UDP server, used for display purposes only.
@@ -142,23 +174,28 @@ read-buffer = 0 # default
 # Precision to use for timestamps
 precision = "n" # Can be n, u, ms, s, m, h
 
-# Array of InfluxDB instances to use as backends for Relay.
-output = [
-    # name: name of the backend, used for display purposes only.
-    # location: host and port of backend.
-    # mtu: maximum output payload size
-    { name="local1", location="127.0.0.1:8089", mtu=512 },
-    { name="local2", location="127.0.0.1:7089", mtu=1024 },
-]
+# InfluxDB instance to use as backend for Relay.
+[[udp.output]]
+# name: name of the backend, used for display purposes only.
+name="local1"
+
+# location: host and port of backend.
+location="127.0.0.1:8089"
+
+# mtu: maximum output payload size
+mtu=512
+
+[[udp.output]]
+name="local2"
+location="127.0.0.1:7089"
+mtu=1024
 ```
 
 InfluxDB Relay is able to forward from a variety of input sources, including:
 
 * `influxdb`
 * `prometheus`
 
-The `type` parameter in the configuration file defaults to `influxdb`.
-
 ### Administrative tasks
 
 #### /admin endpoint
@@ -169,25 +206,6 @@ InfluxDB Relay does not send back a response body to the client(s), we are not
 able to forward all of the features this endpoint provides. Still, we decided
 to expose it through the `/admin` route.
 
-In order to use it, one shall define the optional `admin` parameter of the
-target backend(s) in the configuration file, as follows:
-
-```toml
-[[http]]
-# Name of the HTTP server, used for display purposes only.
-name = "expose-database-creation"
-
-# TCP address to bind to, for HTTP server.
-bind-addr = "127.0.0.1:9096"
-
-# Array of InfluxDB instances to use as backends for Relay.
-output = [
-    # InfluxDB
-    { name="local-influxdb01", location="http://127.0.0.1:8086/write", admin="http://127.0.0.1:8086", type="influxdb" },
-    { name="local-influxdb02", location="http://127.0.0.1:7086/write", admin="http://127.0.0.1:7086", type="influxdb" },
-]
-```
-
 It is now possible to query the `/admin` endpoint. Its usage is the same as the
 standard `/query` Influx DB enpoint:
 
@@ -200,7 +218,7 @@ bodies will not be forwarded back to the clients.
 
 #### /health endpoint
 
-This endpoint uses the same configuration as the `/admin` endpoint, and provides a quick way to check the state of all the backends.
+This endpoint provides a quick way to check the state of all the backends.
 It will return a JSON object detailing the status of the backends like this : 
 
 ```json
@@ -223,12 +241,15 @@ The status field is a summary of the general state of the backends, the defined
 * `problem`: some backends, but no all of them, returned errors
 * `critical`: every backend returned an error
 
+### Filters
+
+We allow tags and measurements filtering through regular expressions. Please,
+take a look at [this document](docs/filters.md) for more information.
+
 ## Limitations
 
 So far, this is compatible with Debian, RedHat, and other derivatives.
 
-Moreover, the `type` parameter is not supported when it comes to UDP forwarding.
-
 ## Development
 
 Please read carefully [CONTRIBUTING.md][contribute-href] before making a merge