whisper2cyanite is a tool for migrating data from Whisper to Cyanite.
whisper2cyanite is a Clojure application and uses Leiningen as build tool. Building whisper2cyanite needs a working Leiningen installation, as well as JDK.
lein uberjarBuilt JAR-file will be placed in the target/uberjar directory. You can launch
the tool by running ./whisper2cyanite command.
Building whisper2cyanite deb-package needs installed dpkg-dev and fakeroot
packages.
lein fatdebBuilt package will be placed in the target directory.
whisper2cyanite [options] migrate <directory | whisper_file | filelist_file> <tenant> <cassandra_host,...> <elasticsearch_url>
whisper2cyanite [options] validate <directory | whisper_file | filelist_file> <tenant> <cassandra_host,...> <elasticsearch_url>
whisper2cyanite [options] calc-size <directory | whisper_file | filelist_file> <tenant>
whisper2cyanite list-files <directory>
whisper2cyanite [options] list-paths <directory>
whisper2cyanite info <whisper_file>
whisper2cyanite [options] fetch <whisper_file> <rollup>
whisper2cyanite help
See commands, arguments and options for more details.
Migrate a Whisper database to a metric store, a path store or both.
whisper2cyanite migrate [options]
<directory | whisper_file |
filelist_file> tenant
cassandra_host(s)
elasticsearch_url
Available options: cassandra-batch-rate,
cassandra-batch-size,
cassandra-channel-size,
cassandra-keyspace,
cassandra-options, disable-log,
disable-metric-store,
disable-path-store,
disable-progress,
elasticsearch-channel-size,
elasticsearch-index, errors-file,
from, jobs, log-file,
log-level, min-ttl, rollups,
root-dir, run, to,
stop-on-error.
See example of usage here.
Validate a metric store, a path store or both, comparing points and paths from a Whisper database and Cyanite storages.
whisper2cyanite validate [options]
<directory | whisper_file |
filelist_file> tenant
cassandra_host(s)
elasticsearch_url
Available options: cassandra-keyspace,
cassandra-options, disable-log,
disable-metric-store,
disable-path-store,
disable-progress,
elasticsearch-index, errors-file,
from, jobs, log-file,
log-level, min-ttl, rollups,
root-dir, to, stop-on-error.
See example of usage here.
Calculate Cassandra data size from a Whisper database.
whisper2cyanite calc-size [options]
<directory | whisper_file |
filelist_file> tenant
Available options: disable-progress, jobs,
rollups, root-dir.
See example of usage here.
List Whisper database files taken from a source.
whisper2cyanite list-files <directory |
whisper_file | filelist_file>
See example of usage here.
List paths taken from a source. The root-dir option can be passed to have
correct paths being computed from filenames.
whisper2cyanite list-paths [options]
<directory | whisper_file |
filelist_file>
Available options: root-dir.
See example of usage here.
whisper2cyanite info whisper_file
Show Whisper database file information.
See example of usage here.
Fetch timeseries from a Whisper database file.
whisper2cyanite fetch [options]
whisper_file
See example of usage here.
whisper2cyanite help
Show help.
A directory where Whisper database files are located.
A Whisper database file (.wsp).
A plain text file with a list of Whisper database files (.wsp). One file per
line.
A tenant name.
A comma-separated list of Cassandra hosts.
Example: cass1.example.org,cass2.example.org
An Elasticsearch REST service URL.
Example: http://es.example.org:9200
A rollup (seconds per data point).
Example: 60
Options in alphabet order:
- cassandra-batch-rate
- cassandra-batch-size
- cassandra-channel-size
- cassandra-keyspace
- cassandra-options
- disable-metric-store
- disable-path-store
- disable-progress
- elasticsearch-channel-size
- elasticsearch-index
- errors-file
- from
- help
- jobs
- log-file
- log-level
- min-ttl
- rollups
- root-dir
- run
- stop-on-error
- to
--cassandra-batch-rate RATE
Set the Cassandra batch rate (batches per second, 1-100).
Throttling is not used by default.
--cassandra-batch-size SIZE
Set the Cassandra batch size in points.
Default: 1000
--cassandra-channel-size SIZE
Set the Cassandra channel size in points.
Default: 500000
--cassandra-keyspace KEYSPACE
Set the Cassandra keyspace.
Default: metric
-O, --cassandra-options OPTIONS
Set Cassandra options. See Alia documentation for more details.
Example: "{:compression :lz4}"
--disable-metric-store
Disable writing to the metric store.
--disable-path-store
Disable writing to the path store.
-P, --disable-progress
Disable the progress bar.
--elasticsearch-channel-size SIZE
Set the Elasticsearch channel size.
Default: 10000
--elasticsearch-index INDEX
Set the Elasticsearch index.
Default: cyanite_paths
-e, --errors-file FILE
Dump a list of files during processing which the errors occurred.
-f, --from FROM
Set from time in the Unix (POSIX, epoch) time format.
Example: 1420070400
-j, --jobs JOBS
Set the number of jobs to run simultaneously.
-l, --log-file FILE
Set the log file.
Default: whisper2cyanite.log
-L, --log-level LEVEL
Set the Log level.
Available log levels: all, trace, debug, info, warn, error,
fatal, off.
Default: info
-T, --min-ttl TTL
Set the minimal TTL. Points having calculated TTL values below the minimal TTL will not be migrated. It is useful to reduce the number of migrated points.
Default: 3600
-R, --rollups ROLLUPS
Define rollups.
Format: <seconds_per_point[:retention],...>
Example: 60,300:31536000
-D, --root-dir DIRECTORY
Set the root directory.
-r, --run
Force a normal run. Dry run using on default.
-t, --to TO
Set until time in the Unix (POSIX, epoch) time format.
Example: 1421280000
-S, --stop-on-error
Stop on the first non-fatal error.
Before migrating data, you may want to inspect the Whisper database.
Listing paths taken from the /var/lib/whisper/ directory in alphabet order:
whisper2cyanite list-paths /var/lib/whisper/See command list-paths for more details.
Estimating Cassandra data size from a Whisper database located in the
/var/lib/whisper/ directory:
whisper2cyanite calc-size --jobs 8 /var/lib/whisper/ my_tenantUsing 8 jobs to speed up the process. Tenant is named as my_tenant.
See command calc-size for more details.
Getting information about the /var/lib/whisper/requests/nginx/access.wsp
Whisper database file
whisper2cyanite info /var/lib/whisper/requests/nginx/access.wspSample output:
Aggregation method: average
Max retention: 31536000
X files factor: 0.5
Archives:
Archive 0:
Seconds per point: 60
Retention: 5356800
Points: 89280
Offset: 40
Size: 1071360
Archive 1:
Seconds per point: 900
Retention: 31536000
Points: 35040
Offset: 1071400
Size: 420480
See command info for more details.
Fetch timeseries between 1428407040 (07 Apr 2015 11:44:00 GMT) and
1433757900 (08 Jun 2015 10:05:00 GMT) from rollup 60 of the
/var/lib/whisper/requests/nginx/access.wsp Whisper database file:
whisper2cyanite --from 1428407040 --to 1433757900 fetch \
/var/lib/whisper/requests/nginx/access.wsp 60See command fetch for more details.
Typical command for migrating a whole Whisper database to Cyanite looks like this:
whisper2cyanite --run --jobs 8 --rollups 60:5356800,900:6220800 --min-ttl 18000 \
--cassandra-options "{:compression :lz4}" --errors-file error-files.lst \
migrate /var/lib/whisper/ my_tenant cass1.example.org,cass2.example.org \
http://es.example.org:9200This command means:
- Run in the normal mode, not in the dry mode which using on default
- Use 8 jobs
- Define two rollups: 1m rollup with 62d retention time and 15m rollup with 720d retention time
- Set minimum TTL to 18000s (5h)
- Set Cassandra options: use compression algorithm LZ4
- Dump a list of files during processing which errors occurred to the
error-files.lst - Migrate the Whisper database placed in the
/var/lib/whisper/. Database root will be set to this directory automatically. - Set the tenant to
my_tenant - Use two Cassandra nodes:
cass1.example.organdcass2.example.org - Use the Elasticsearch node on
http://es.example.org:9200
See command migrate for more details.
whisper2cyanite --run --jobs 8 --rollups 60:5356800,900:6220800 --min-ttl 18000 \
--cassandra-options "{:compression :lz4}" --root-dir /var/lib/whisper/ \
migrate error-files.lst my_tenant cass1.example.org,cass2.example.org \
http://es.example.org:9200This command means same as above but:
- Set database root to the
/var/lib/whisper/directory - Read files to migrate from the
errorfiles.lstfile
See command migrate for more details.
Dumping names of Whisper database files to the path-files.lst file:
whisper2cyanite list-files /var/lib/whisper/requests/nginx > path-files.lstReading the path-files.lst file and migrating paths taken from
there. According to previous command, only paths from the
/var/lib/whisper/requests/nginx directory (the requests.nginx.* path store
subtree) will be migrated. Metric store operations are disabled.
whisper2cyanite --run --jobs 8 --disable-metric-store --root-dir \
/var/lib/whisper/ migrate path-files.lst my_tenant cass1.example.org \
http://es.example.org:9200See commands list-files and migrate for more
details.
Migrating metrics from the /var/lib/whisper/requests/nginx/access.wsp file
for period from 1420070400 (01 Jan 2015 00:00:00) until 1421280000 (15 Jan
2015 00:00:00). All rollups except 60 are ignored. Path store operations are
disabled.
whisper2cyanite --run --rollups 60:5356800 --disable-path-store \
--root-dir /var/lib/whisper/ --from 1420070400 --to 1421280000 migrate \
/var/lib/whisper/requests/nginx/access.wsp my_tenant \
cass1.example.org,cass2.example.org http://es.example.org:9200See command migrate for more details.
Using the command validate is very similar to using the command migrate.
Typical command for validating a whole database:
whisper2cyanite --jobs 8 --cassandra-options "{:compression :lz4}" \
--errors-file error-files.lst validate /var/lib/whisper/ my_tenant \
cass1.example.org,cass2.example.org http://es.example.org:9200See command validate for more details.
Validating paths from the /var/lib/whisper/requests/nginx directory (the
requests.nginx.* path store subtree):
whisper2cyanite --jobs 8 --disable-metric-store --root-dir /var/lib/whisper/ \
validate /var/lib/whisper/requests/nginx my_tenant cass1.example.org \
http://es.example.org:9200See command validate for more details.
Validating metrics from the /var/lib/whisper/requests/nginx/access.wsp file:
whisper2cyanite --disable-path-store --root-dir /var/lib/whisper/ validate \
/var/lib/whisper/requests/nginx/access.wsp my_tenant \
cass1.example.org,cass2.example.org http://es.example.org:9200See command validate for more details.
whisper2cyanite is covered by MIT License
Thanks to Pierre-Yves Ritschard aka @pyr for his work on Cyanite