Skip to content

A connector that collects data using 'appmetrics' and sends it to ElasticSearch for use with Kibana

License

Notifications You must be signed in to change notification settings

RuntimeTools/appmetrics-elk

Repository files navigation

ELK Connector for Node Application Metrics

A connector that collects data using 'appmetrics' and sends it to a configured ElasticSearch instance in LogStash format for use with Kibana.

Getting Started

Installation

The ELK Connector for Node Application Metrics can be installed via npm:

$ npm install appmetrics-elk

This is designed to be used with an ElasticSearch database, and a visualization tool such as Kibana.

Configuring the ELK Connector for Node Application Metrics

The connector can be used in your application by requiring it as the first line of your application:

var appmetrics = require('appmetrics-elk').monitor();

This will send all of the available appmetrics data to the ElasticSearch instance, as well as returning an appmetrics object that can be used to control data collection.

var appmetrics = require('appmetrics-elk').monitor();
appmetrics.disable('mysql');            // disable MySQL monitoring

Additionally, the monitor() API call can be passed an optional ElasticSearch Configuration object to configure the ElasticSearch connection, including database location and security.

The same configuration object can be used to pass configuration to the ELK connector. The following configurations can be applied:

  • index (String) the name of the index to use for storing the monitoring data. The default is appmetrics.
  • applicationName (String) the name to use for the applicationName field in the monitoring data. The default is the name of the applications main file, eg. app.js.
var config = {
    hosts: [
        'https://es1.bluemix.net',
        'https://es2.bluemix.net'
    ],
    ssl: {
        ca: fs.readFileSync('./cacert.pem'),
        rejectUnauthorized: true
    },
    index: 'nodedata',
    applicationName: 'HelloWorld'
}

var appmetrics = require('appmetrics-elk').monitor(config);
appmetrics.disable('mysql');            // disable MySQL monitoring

Data Provided to ElasticSearch

The ELK Connector for Node Application Metrics uploads its data to the 'appmetrics' index in ElasticSearch. It sends the following values to ElasticSearch for every monitoring entry:

Value Description
timestamp The time when the monitoring event occurred
hostName The hostname for the machine the monitored process is running on
pid The process ID for the monitored process
applicationName The JavaScript file used to launch the application, or a custom name

Additional data is then included depending on the monitoring event.

CPU Data

Value Description
cpu.process The CPU usage of the application as a percentage of total machine CPU
cpu.system The CPU usage of the system as a percentage of total machine CPU

Memory Data

Value Description
memory.process.private The memory used by the application that cannot be shared with other processes, in bytes
memory.process.physical The RAM used by the application in bytes
memory.process.virtual The memory address space used by the application in bytes
memory.system.physical The total amount of RAM in use on the system in bytes
memory.system.total The total amount of RAM available on the system in bytes

Garbage Collection Data

Value Description
gc.used The JavaScript heap used by the application in bytes
gc.size The size of the JavaScript heap in bytes
gc.type The type of GC cycle, either 'M' or 'S'
gc.duration The duration of the GC cycle in milliseconds

Event Loop Latency Data

Value Description
eventloop.latency.min The shortest sampled latency for processing an event
eventloop.latency.max The longest sampled latency for processing an event
eventloop.latency.avg The mean sampled latency for processing an event

HTTP Request Data

Value Description
http.method The HTTP method used for the request
http.url The URL on which the request was made
http.duration The time taken for the HTTP request to be responded to in ms

Socket.io Data

Value Description
socketio.method The type of socketio event, either broadcast, emit or receive
socketio.event The name of the socketio event
sokcetio.duration The time taken for the socketio event to be handled in ms

MongoDB Query Data

Value Description
mongo.query The query made of the MongoDB database
mongo.duration The time taken for the MongoDB query to be responded to in ms

MySQL Query Data

Value Description
mysql.query The query made of the MySQL database
mysql.duration The time taken for the MySQL query to be responded to in ms

PostgreSQL Query Data

Value Description
postgres.query The query made of the PostgreSQL database
postgres.duration The time taken for the postgreSQL query to be responded to in ms

Redis Command Data

Value Description
redis.cmd The Redis command sent to the server or 'batch.exec'/'multi.exec'
redis.duration The time taken for the Redis command to be handled to in ms

Memcached Operation Data

Value Description
memcached.method The method used in the memcached client, eg set, get, append, delete, etc.
memcached.key The the key associated with the data
memcached.duration The time taken for the memcached command to be handled to in ms

Leveldown Query Data

Value Description
leveldown.method The leveldown method being used.
leveldown.key The key being used for a call to get, put or del
leveldown.value The value being added to the LevelDB database using the put method
leveldown.opCount The number of operations carried out by a batch method
leveldown.duration The time taken for the leveldown query to be handled to in ms

MQTT Message Data

Value Description
mqtt.method Either a publish or a received message
mqtt.topic The topic on which the message is published or received.
mqtt.qos The quality of service used for the message
mqtt.duration The time taken for the message to be handled to in ms

MQLight Message Data

Value Description
mqlight.method Either a send or a received message
mqlight.topic The topic on which the message is published or received.
mqlight.qos The quality of service used for the message
mqlight.data The first 25 characters of the message
mqlight.duration The time taken for the message to be handled to in ms

Sending Custom Data to ElasticSearch

The Node Application Metrics to ELK Connector registers for events that it is aware of, and forwards the data from those events to ElasticSearch. The registration for those events is based on the 'mappings' files in the following directory:

node_modules/appmetrics-elk/mappings/

Any mappings files found in that directory are both used to configure how ElasticSearch handles the data, and to configure the monitoring events that are forwarded.

The type field is used to determine the name of the event to register for, and the properties fields are used to determine the values to send. Note that the values in the properties entry in the mapping file must match the fields in the monitoring event data. For example, the CPU event has the following data:

  • process
  • system

The mapping file that causes this data to be sent to ElasticSearch therefore has the following structure:

{
    "index":  "appmetrics",
    "type":   "cpu",
    "body": {
        "_source" : {"compress" : true},
        "_ttl" : {"enabled" : true, "default" : "90d"},
        "properties": {
            "timestamp":    {"type": "date", "format": "dateOptionalTime"},
            "hostName":     {"type": "string", "index": "not_analyzed"},
            "pid":          {"type": "integer"},
            "cpu": {
                "type": "nested",
                "include_in_parent": true,
                "properties": {            
                    "process":      {"type": "float"},
                    "system":       {"type": "float"}
                }
            }
        }   
    }
}

This causes the Node Application Metrics to ELK Connector to register for cpu events and forward the process and system values as cpu.process and cpu.system.

Using Kibana with the ELK Connector for Node Application Metrics

During startup the ELK Connector for Node Application Metrics attempts to provide some pre-configuration for using Kibana 4 with the provided data. It does this by uploading the following if there are not existing ones already associated with the 'appmetrics' index:

  • An index pattern
  • Data mappings for the data types
  • Default visualizations for the data types
  • A default dashboard

Each of these configurations are dynamically loaded from the 'indexes', 'mappings', 'charts' and 'dashboards' directories in the appmetrics-elk install directory. It is therefore possible to prevent the configurations from being automatically added by deleting those files, or to add to them by adding existing files.

Note: The 'mappings' directory also provides the configuration of which types of monitoring data are uploaded to ElasticSearch so entries should only be deleted if necessary. See Sending Custom Data to ElasticSeach for more information.

Visualizing the data with Kibana 4

The pre-configuration for Kibana 4 provdes a number of default visualizations, as well as a default dashboard. These can subsequently be modified or new visualizations and dashboards created.

Using the dashboard
In order to avoid replacing any dashboard already in use in Kibana 4 with the one supplied by the Node Application Metrics to ELK Connector, the dashboard is made available but not loaded. The dashboard is loaded by:

  1. Click on the "Dashboard" tab
  2. Select the "Load Saved Dashboard" icon
  3. Select "Default AppMetrics Dashboard" from the list of saved dashboards
    This now loads a simple dashboard that uses some of the default visualization charts provided by the Node Application Metrics to ELK Connector.

Using the visualization charts
In addition to the dashboard, a number of visualization charts are also provided. These can be added to a dashboard using the following steps:

  1. Click on the "Dashboard" tab
  2. Click on the "Add Visualization" icon
  3. Select a visualization chart from the menu
  4. Place and resize the visualization chart by dragging it across the screen

You can also create your own charts using the "Visualize" tab.

License

The Node Application Metrics to ELK Connector is licensed using an Apache v2.0 License.

Version

The current version is 1.0.2

1.0.2 Support for new monitoring data
1.0.1 Support for configurable indexes and addition of applicationName field
1.0.0 Initial release

About

A connector that collects data using 'appmetrics' and sends it to ElasticSearch for use with Kibana

Resources

License

Stars

Watchers

Forks

Packages

No packages published