Netflow scrapper tools

Set of tools to scrap data from iptables netflow module (by default UDP on localhost on port 2055) and broadcast to 0MQ. It also enriches traffic records with GeoIP

Usage

Build

This could be built as deb package and can be run as systemD daemons.

$ cargo deb
$ dpkg -i target/debian/netflow-scrapper*.deb

Or you could instead just build it and run manually with cargo build and cargo run (use --release to disable debugging).

Configure and run

Iptables rules

You need iptables (or nftables) rules to forward some traffic to NETFLOW chain (and netflow kernel module of course). For example, to just send ALL traffic which is forwarded to and from eth0 interface, you can use

iptables -A FORWARD -i eth0 -j NETFLOW
iptables -A FORWARD -o eth0 -j NETFLOW

Use your imagination which exact traffic you wish to analyze and refer to netflow documentation.

GeoIP

You need to install geoIP database (maxmind format). Follow the instructions on geoipupdate. You will need a free license and two databases (Country and ASN). By default it is expected that they are stored in /var/lib/GeoIP/ which is configurable.

Producer and consumers

Package consists of one producer (0MQ publisher) and two consumers (0MQ subscribers). Publisher scraps all data from UDP socket, enriches GeoIP data and sends it to 0MQ (Unix socket). One of subscribers insert all records to Oracle database while another is exposing metrics in Prometheus format.

By default, configuration is defined in /etc/netflow-scrapper/. For 0MQ producer and metrics server you can stick with defaults, for oracle you must set your credentials. Metrics server publishes data in Prometheus format (port 9091 by default) which can be seen as

$ curl http://localhost:9091/metrics
# will print some values of netflow_tx_bytes_total{} and netflow_rx_bytes_total{}

You can configure Prometheus/Grafana Alloy or any other compatible agent to scrap that data and draw nice visualization. For example, you may use the following PromQL queries to aggregate incoming and outgoing traffic by IP tag. Currently the following tags are exported: ip, country (iso code from geoip), org and protocol.

sum by(ip) (increase(netflow_rx_bytes_total[$__rate_interval])) > 0"
sum by(ip) (increase(netflow_tx_bytes_total[$__rate_interval])) > 0"