simple-debug-monitoring

This repository contains several tools to monitor the behaviours of a long running MonetDB server.

monitor.py

This script monitors the consumption of several resources of a long running mserver5 process. The logs generated by this script can help detect the growth of resource consumptions and the potential causes.

Log files and what's in them?

The logs are both written to a log file and printed on screen. Currently, the information is logged as follows:

"_<dbname>.log"

Once per LOG_INTERVAL seconds, one line of the following information is written to this log file:

ts dbsz rss vmsz maps fds

where

• ts: current time stamp in seconds in ISO 8601 format with local timezone info.
• dbsz: total size of the database directory on disk in GB
• rss: RSS usage in GB
• vmsz: virtual memory usage in GB
• maps: number of memory-mapped files
• fds: number of open file descriptors

"_<dbname>.log.maps"

Each time the number of memory-mapped files has grown with MMAP_INCREASE, log the contents of /proc/$(pgrep mserver5)/maps in this log file.

"_<dbname>.log.fds"

Each time the number of open file descriptors has grown with FD_INCREASE, log the FDs in /proc/$(pgrep mserver5)/fd in this log file.

"_<dbname>.log.dbchk"

Once per DBCHK_INTERVAL seconds, run some checks on the data contained in the database. Currently, we run a SELECT COUNT(*) query on each table which has been created by a database user (i.e. sys.tables.system = false).

Options and default values

• --dbname: If there is only one mserver5 process. Otherwise, one can specify the database to monitor with this option.
• --logbase: by default, resource consumption, memory mapped files, open file descriptions and database check are written to the files _<dbname>.log, _<dbname>.log.maps, _<dbname>.log.fds and _<dbname>.log.dbchk in the current directory. With the --logbase option, one can specify a prefix to the default log filenames, or specify an absolute or relative path, or both.
• --log-interval: specified how often (in number of seconds) the resource consumption should be logged; by default, every 5 seconds.
• --dbcheck-interval: specified how often (in number of seconds) the database should be checked; by default, every 1800 seconds.
• --mmap-increase: specified how often (in number of memory-mapped files) the memory-mapped files should be logged; by default, every 5000 more.
• --fd-increase: specified how often (in number of file descriptors) the open file descriptors should be logged; by default, every 100 more.

plot.py

This is the script on which monitor.py is based. It monitors less information, but plots the monitored data.

Probably works on all Linux systems with bash. Make sure you have installed matplotlib and all of its dependencies.

Just run

./plot.py <db_path> <mserver5-pid>

It should pop up a matplotlib display with three graphs:

• virtual memory usage
• resident set size
• database disk footprint.

The script assumes there is a single mserver5 running.

Set <db_path> to a valid relative or absolute path pointing to the root of the database, i.e. what is passed to the --dbpath parameter of mserver5.