Skip to content

Latest commit

 

History

History
848 lines (699 loc) · 24.5 KB

configuration.md

File metadata and controls

848 lines (699 loc) · 24.5 KB
Raw

DnsCollector - Configuration Guide

A typically configuration would have one or more collector to receive DNS traffic or logs, and severals loggers to process the incoming traffics.

The configuration is done in one yaml file. For the complete configuration, see config.

Trace

Logs can be enable to have more informations like debug, errors messages generated by the application

Options:

  • verbose: (boolean) debug informations, if turned on, log some applications messages
  • filename: (string) filename is the file to write logs to.
  • max-size: (integer) maximum size in megabytes of the log file it gets rotated
  • max-backups: (integer) maximum number of old log files to retain
  • log-malformed: (boolean) log malformed packet
#
trace:
  verbose: true
  log-malformed: false
  filename: ""
  max-size: 10
  max-backups: 10

Example:

INFO: 2021/08/12 07:10:12.347913 main - config loaded...
INFO: 2021/08/12 07:10:12.347957 main - starting dnscollector...
INFO: 2021/08/12 07:10:12.347964 generator webserver - enabled
INFO: 2021/08/12 07:10:12.347992 generator stdout logging - enabled
INFO: 2021/08/12 07:10:12.348023 collector dns sniffer - enabled
INFO: 2021/08/12 07:10:12.348318 main - running all collectors and generators...
INFO: 2021/08/12 07:10:12.348442 generator webserver - running in background...
INFO: 2021/08/12 07:10:12.348482 generator webserver - starting http api...
INFO: 2021/08/12 07:10:12.348775 generator stdout - running in background...
INFO: 2021/08/12 07:10:12.348793 generator webserver - is listening on [::]:8080
INFO: 2021/08/12 07:10:12.362156 dns processor - initialization...
INFO: 2021/08/12 07:10:12.362313 dns processor - ready
INFO: 2021/08/12 07:10:12.362372 processor dns parser - running... waiting incoming dns message

Collectors

DNS tap

Dnstap stream collector:

  • tcp or unix socket listener
  • tls support

Options:

  • enable: (boolean) to enable, set the enable to true
  • listen-ip: (string) listen on ip
  • listen-port: (integer) listening on port
  • sock-path: (string) unix socket path
  • tls-support:: (boolean) to enable, set to true
  • cert-file: (string) certificate server file
  • key-file: (string) private key server file
dnstap:
  enable: false
  listen-ip: 0.0.0.0
  listen-port: 6000
  sock-path: null
  tls-support: false
  cert-file: ""
  key-file: ""

DNS sniffer

Raw DNS packets sniffer. Setting CAP_NET_RAW capabilities on executables allows you to run these program without having to run-it with the root user:

  • IPv4, IPv6 support (fragmented packet ignored)
  • UDP and TCP transport
  • BFP filtering
sudo setcap cap_net_admin,cap_net_raw=eip go-dnscollector

Options:

  • enable: (boolean) to enable, set the enable to true
  • port: (integer) filter on source and destination port
  • device: (string) if "" bind on all interfaces
  • capture-dns-queries: (boolean) capture dns queries
  • capture-dns-replies: (boolean) capture dns replies
dns-sniffer:
  enable: true
  port: 53
  device: wlp2s0
  capture-dns-queries: true
  capture-dns-replies: true

Tail

The tail collector enable to read DNS event from text files. DNS servers log server can be followed; any type of server is supported!

  • Read DNS events from the tail of text files
  • Regex support

Enable the tail by provided the path of the file to follow

Options:

  • enable: (boolean) to enable, set the enable to true
  • file-path: (string) file to follow
  • time-layout: (string) Use the exact layout numbers described https://golang.org/src/time/format.go
  • pattern-query: (string) regexp pattern for queries, example for unbound: "query: (?P[^ ]) (?P[^ ]) (?P[^ ]*)"
  • pattern-reply: (string) regexp pattern for replies, example for unbound: "reply: (?P[^ ]) (?P[^ ]) (?P[^ ]) IN (?P[^ ]) (?P[^ ]*)"
tail:
  enable: false
  file-path: null
  time-layout: "2006-01-02T15:04:05.999999999Z07:00"
  pattern-query: "^(?P<timestamp>[^ ]*) (?P<identity>[^ ]*) (?P<qr>.*_QUERY) (?P<rcode>[^ ]*) (?P<queryip>[^ ]*) (?P<queryport>[^ ]*) (?P<family>[^ ]*) (?P<protocol>[^ ]*) (?P<length>[^ ]*)b (?P<domain>[^ ]*) (?P<qtype>[^ ]*) (?P<latency>[^ ]*)$"
  pattern-reply: "^(?P<timestamp>[^ ]*) (?P<identity>[^ ]*) (?P<qr>.*_RESPONSE) (?P<rcode>[^ ]*) (?P<queryip>[^ ]*) (?P<queryport>[^ ]*) (?P<family>[^ ]*) (?P<protocol>[^ ]*) (?P<length>[^ ]*)b (?P<domain>[^ ]*) (?P<qtype>[^ ]*) (?P<latency>[^ ]*)$"

Protobuf PowerDNS

Protobuf Logging support for PowerDNS's products.

Options:

  • enable: (boolean) to enable, set the enable to true
  • listen-ip: (string) listen on ip
  • listen-port: (integer) listening on port
powerdns:
  enable: false
  listen-ip: 0.0.0.0
  listen-port: 6001

Example to enable logging in your dnsdist

rl = newRemoteLogger("<dnscollectorip>:6001")
addAction(AllRule(),RemoteLogAction(rl, nil, {serverID="dnsdist"}))
addResponseAction(AllRule(),RemoteLogResponseAction(rl, nil, true, {serverID="dnsdist"}))
addCacheHitResponseAction(AllRule(), RemoteLogResponseAction(rl, nil, true, {serverID="dnsdist"}))

Example to enable logging in your pdns-recursor

/etc/pdns-recursor/recursor.conf

lua-config-file=/etc/pdns-recursor/recursor.lua

/etc/pdns-recursor/recursor.lua

protobufServer("<dnscollectorip>:6001")
outgoingProtobufServer("<dnscollectorip>:6001")

Subprocessors

Quiet text

Enable or disable quiet text mode for some flags, this option can be useful to reduce the size of your dns logs

The following dnstap flag message will be replaced with the small form:

  • AUTH_QUERY: AQ
  • AUTH_RESPONSE: AR
  • RESOLVER_QUERY: RQ
  • RESOLVER_RESPONSE: RR
  • CLIENT_QUERY: CQ
  • CLIENT_RESPONSE: CR
  • FORWARDER_QUERY: FQ
  • FORWARDER_RESPONSE: FR
  • STUB_QUERY: SQ
  • STUB_RESPONSE: SR
  • TOOL_QUERY: TQ
  • TOOL_RESPONSE: TR

The following dns flag message will be replaced with the small form:

  • QUERY: Q
  • REPLY: R

Options:

  • dnstap: (boolean) enable or disable
  • dns: (boolean) enable or disable
subprocessors:
  # this option can be useful to reduce the size of your dns logs
  quiet-text:
    dnstap: false

    dns: false

Qname lowercase

Option to convert all domain to lowercase. For example: Wwww.GooGlE.com will be equal to www.google.com This feature is enabled by default.

Options:

  • qname-lowercase: (boolean) enable or disable lowercase
subprocessors:
  qname-lowercase: true

User Privacy

Use this feature to protect user privacy. This feature can be used to anonymize all IP queries and reduce all qnames to second level. For example:

  • QueryIP 8.8.8.8 will be replaced by 8.8.0.0. IP-Addresses are anonymities by zeroing the host-part of an address.
  • Qname mail.google.com be replaced by google.com

Options:

  • anonymize-ip: (boolean) enable or disable anomymiser ip
  • minimaze-qname: (boolean) keep only the second level domain
subprocessors:
  user-privacy:
    anonymize-ip: false
    minimaze-qname: false

GeoIP Support

GeoIP maxmind support feature. The country code can be populated regarding the query IP collected. To enable this feature, you need to configure the path to your database.

See Downloads maxmind page to get the database.

Options:

  • mmdb-country-file: (string) path file to your mmdb country database
  • mmdb-city-file: (string) path file to your mmdb city database
  • mmdb-asn-file: (string) path file to your mmdb asn database
subprocessors:
  geoip:
    mmdb-country-file: "/GeoIP/GeoLite2-Country.mmdb"
    mmdb-city-file: ""
    mmdb-asn-file: ""

When the feature is enabled, the following json field are populated:

  • continent
  • country-isocode
  • city
  • as-number
  • as-owner

Example:

{
  ...
  "geo": {
    "city": "-",
    "continent": "-",
    "country-isocode": "-"
  },
  "network": {
    ...
    "as-number": 1234,
    "as-owner": "Orange",
  },
  ...
}

DNS Caching

The caching feature is used to compute latency between replies and queries. This cache can be disabled if your dns server already added the latency in the dnstap packet, Disable this feature to improve performance.

Options:

  • enable: (boolean) disable or enable the feature
  • query-timeout: (integer) in second, max time to keep the query record in memory
subprocessors:
  cache:
    enable: true
    query-timeout: 10

DNS filtering

The filtering feature can be used to ignore some queries or replies according to:

  • qname
  • return code
  • query ip

This feature can be useful to increase logging performance..

Options:

  • drop-fqdn-file: (string) path file to a fqdn drop list, domains list must be a full qualified domain name
  • drop-domain-file: (string) path file to domain drop list, domains list can be a partial domain name with regexp expression
  • drop-queryip-file: (string) path file to the query ip or ip prefix drop list
  • keep-queryip-file: (string) path file to the query ip or ip prefix keep list, addresses in both drop and keep are always kept
  • drop-rcodes: (list of string) rcode list, empty by default
  • log-queries: (boolean) forward received queries to configured loggers
  • log-replies: (boolean) forward received replies to configured loggers
subprocessors:
  filtering:
    drop-fqdn-file: ""
    drop-domain-file: ""
    drop-queryip-file: ""
    keep-queryip-file: ""
    drop-rcodes: []
    log-queries: true
    log-replies: true

Domain list with regex example:

(mail|wwww).google.com
github.com

Custom text format

Custom text format can be configured All available directives:

  • timestamp-rfc3339ns: timestamp rfc3339 format, with nano support
  • timestamp-unixms: unix timestamp with ms support
  • timestamp-unixus: unix timestamp with us support
  • timestamp-unixns: unix timestamp with nano support
  • localtime: local time
  • identity: dnstap identity
  • operation: dnstap operation
  • opcode: dns opcode (integer)
  • rcode: dns return code
  • queryip: dns query ip
  • queryport: dns query port
  • responseip: dns response ip
  • responseport: dns response port
  • id: dns id
  • family: ip protocol version INET or INET6
  • protocol: protocol UDP, TCP
  • length: the length of the query or reply
  • qtype: dns qtype
  • qname: dns qname
  • qnamepublicsuffix: Public Suffix of the DNS QNAME
  • qnameeffectivetldplusone: Public Suffix plus one label of the DNS QNAME
  • latency: computed latency between queries and replies
  • answercount: the number of answer
  • continent: continent code
  • country: country iso code
  • city: city name
  • as-number: autonomous system number
  • as-owner: autonomous system organization/owner
  • ttl: answer ttl, only the first one
  • answer: rdata answer, only the first one, prefer to use the JSON format if you wamt all answers
  • malformed: malformed dns packet, integer value 1/0
  • qr: query or reply flag, string value Q/R
  • tc: flag truncated response
  • aa: flag authoritative answer
  • ra: flag recursion available
  • ad: flag authenticated data
  • edns-csubnet: display client subnet info
subprocessors:
  text-format: "timestamp-rfc3339ns identity qr operation rcode queryip queryport family protocol length qname qtype latency ttl"

Statistics

Some options to customize the statitics subprocessor.

Options:

  • top-max-items: (string) default number of items on top
  • common-qtypes: (list of string) expected common qtype list, other will be considered as suspicious
  • threshold-qname-len: (string) a length greater than this value will be considered as suspicious
  • threshold-packet-len: (string) a size greater than this value will be considered as suspicious value in bytes
  • threshold-slow: (string) threshold to set a domain considered as slow, value in second
  • prometheus-suffix: (string) prometheus suffix
subprocessors:
  statistics:
    top-max-items: 100
    common-qtypes:
      - A
      - AAAA
      - CNAME
      - TXT
      - PTR
      - NAPTR
      - DNSKEY
      - SRV
    threshold-qname-len: 80
    threshold-packet-len: 1000
    threshold-slow: 0.5
    prometheus-suffix: "dnscollector"

Loggers

Stdout

Print to your standard output, all DNS logs received

  • in text or json format
  • custom text format

Options:

  • enable: (boolean) enable, set the enable to true
  • mode: (string) text or json
  • text-format: (string) output text format, please refer to the default text format to see all available directives, use this parameter if you want a specific format
stdout:
  enable: true
  mode: text
  text-format: ""

Example:

2021-08-07T15:33:15.168298439Z dnscollector CQ NOERROR 10.0.0.210 32918 INET UDP 54b www.google.fr A 0.000000
2021-08-07T15:33:15.457492773Z dnscollector CR NOERROR 10.0.0.210 32918 INET UDP 152b www.google.fr A 0.28919

REST API

Build-in webserver with REST API to retrieve somes statistics like top domains, clients and more... Basic authentication supported. Prometheus metrics is also available through this API.

  • prometheus metrics format
  • qps, total queries/replies, top domains, clients, rcodes...
  • basic auth
  • tls support

See the swagger documentation.

Options:

  • enable: (boolean) enable, set the enable to true
  • listen-ip: (string) listening IP
  • listen-port: (integer) listening port
  • basic-auth-login: (string) default login for basic auth
  • basic-auth-pwd: (string) default password for basic auth
  • tls-support: (boolean) tls support
  • cert-file: (string) certificate server file
  • key-file: (string) private key server file
webserver:
  enable: true
  listen-ip: 0.0.0.0
  listen-port: 8080
  basic-auth-login: admin
  basic-auth-pwd: changeme
  tls-support: true
  cert-file: "./testsdata/server.crt"
  key-file: "./testsdata/server.key"

Prometheus metrics example:

Request:

$ curl --user admin:changeme http://127.0.0.1:8080/metrics

The <promdsuffix> tag can be configured in the config.yml file.

Metrics list:

  • <promdsuffix>_qps: Number of queries per second received
  • <promdsuffix>_requesters_total: Number of clients
  • <promdsuffix>_domains_total: Number of domains observed
  • <promdsuffix>_received_bytes_total: Total bytes received
  • <promdsuffix>_sent_bytes_total: Total bytes sent

The full metrics can be found here.

Log File

Enable this logger if you want to log to a file.

  • with rotation file support
  • supported format: text, json
  • gzip compression
  • execute external command after each rotation
  • custom text format

Options:

  • enable: (boolean) enable, set the enable to true
  • file-path: (string) output logfile name
  • max-size: (integer) maximum size in megabytes of the file before rotation, A minimum of max-size*max-files megabytes of space disk must be available
  • max-files: (integer) maximum number of files to retain. Set to zero if you want to disable this feature
  • flush-interval: (integer) flush buffer to log file every X seconds
  • compress: (boolean) compress log file
  • compress-interval: (integer) checking every X seconds if new log files must be compressed
  • compress-command: (string) run external script after file compress step
  • mode: (string) output format: text|json
  • text-format: (string) output text format, please refer to the default text format to see all available directives, use this parameter if you want a specific format
  • postrotate-command: (string) run external script after file rotation
  • postrotate-delete-success: (boolean) delete file on script success
logfile:
  enable: false
  file-path: null
  max-size: 100
  max-files: 10
  flush-interval: 10
  compress: false
  compress-interval: 5
  compress-command: null
  mode: text
  text-format: ""
  postrotate-command: null
  postrotate-delete-success: false

Basic example to use the postrotate command:

Configure the script to execute after each file rotation, for each call the file is passed as argument.

logfile:
  postrotate-command: "/home/dnscollector/postrotate.sh"

Script to move the log file to a specific folder

#!/bin/bash

DNSCOLLECTOR=/var/dnscollector/
BACKUP_FOLDER=$DNSCOLLECTOR/$(date +%Y-%m-%d)
mkdir -p $BACKUP_FOLDER

mv $1 $BACKUP_FOLDER

DNStap Client

DNStap stream logger to a remote tcp destination or unix socket.

  • to remote tcp destination or unix socket
  • tls support

Options:

  • enable: (boolean) enable, set the enable to true
  • listen-ip: (string) remote address
  • listen-port: (integer) remote tcp port
  • sock-path: (string) unix socket path
  • retry-interval: (integer) interval in second between retry reconnect
  • tls-support: (boolean) enable tls
  • tls-insecure: (boolean) insecure skip verify
dnstap:
  enable: false
  remote-address: 10.0.0.1
  remote-port: 6000
  sock-path: null
  retry-interval: 5
  tls-support: false
  tls-insecure: false

TCP Client

Tcp/unix stream client logger.

  • to remote tcp destination or unix socket
  • supported format: text, json
  • custom text format
  • tls support

Options:

  • enable: (boolean) enable, set the enable to true
  • transport: (string) network transport to use: tcp|unix
  • listen-ip: (string) remote address
  • listen-port: (integer) remote tcp port
  • sock-path: (string) unix socket path
  • retry-interval: (integer) interval in second between retry reconnect
  • tls-support: (boolean) enable tls
  • tls-insecure: (boolean) insecure skip verify
  • mode: (string) output format: text|json
  • text-format: (string) output text format, please refer to the default text format to see all available directives, use this parameter if you want a specific format
tcpclient:
    enable: false
    transport: tcp
    remote-address: 127.0.0.1
    remote-port: 9999
    sock-path: null
    retry-interval: 5
    tls-support: false
    tls-insecure: false
    mode: json
    text-format: ""

Syslog

Syslog logger to local syslog system or remote one.

  • local or remote server
  • custom text format
  • supported format: text, json
  • tls support

Options:

  • enable: (boolean) enable, set the enable to true
  • facility: (string) Set the syslog logging facility
  • transport: (string) Transport to use to a remote log daemon or local one. local|tcp|udp|unix
  • remote-address: (string) Remote address host:port
  • mode: (string) text or json
  • text-format: (string) output text format, please refer to the default text format to see all available directives, use this parameter if you want a specific format
  • tls-support: (boolean) enable tls
  • tls-insecure: (boolean) insecure skip verify
syslog:
  enable: false
  severity: INFO
  facility: DAEMON
  transport: local
  remote-address: ""
  text-format: ""
  mode: text
  tls-support: false
  tls-insecure: false

Fluentd Client

Fluentd client to remote server or unix socket.

  • to remote fluentd collector or unix socket
  • msgpask
  • tls support

Options:

  • enable: (boolean) enable, set the enable to true
  • transport: (string) network transport to use: tcp|unix
  • listen-ip: (string) remote address
  • listen-port: (integer) remote tcp port
  • sock-path: (string) unix socket path
  • retry-interval: (integer) interval in second between retry reconnect
  • tag: (string) tag name
  • tls-support: (boolean) enable tls
  • tls-insecure: (boolean) insecure skip verify
fluentd:
    enable: true
    transport: tcp
    remote-address: 127.0.0.1
    remote-port: 24224
    sock-path: null
    retry-interval: 5
    tag: "dns.collector"
    tls-support: false
    tls-insecure: false

Pcap File

Enable this logger if you want to log into a pcap file.

  • with rotation file support
  • binary format
  • gzip compression
  • execute external command after each rotation

Options:

  • enable: (boolean) enable, set the enable to true
  • file-path: (string) output logfile name
  • max-size: (integer) maximum size in megabytes of the file before rotation
  • max-files: (integer) maximum number of files to retain.
  • compress: (boolean) compress pcap file
  • compress-interval: (integer) checking every X seconds if new log files must be compressed
  • postrotate-command: (string) run external script after each file rotation
  • postrotate-delete-success: (boolean) delete file on script success
pcapfile:
  enable: false
  file-path: null
  max-size: 1
  max-files: 3
  compress: false
  compress-interval: 5
  postrotate-command: null
  postrotate-delete-success: true

InfluxDB client

InfluxDB client to remote InfluxDB server

Options:

  • enable: (boolean) enable, set the enable to true
  • server-url: (string) InfluxDB server url
  • auth-token: (string) authentication token
  • bucket: (string) bucket name
  • organization: (string) organization name
  • tls-support: (boolean) enable tls
  • tls-insecure: (boolean) insecure skip verify
  influxdb:
    enable: true
    server-url: "http://localhost:8086"
    auth-token: ""
    bucket: "db_dns"
    organization: "dnscollector"
    tls-support: false
    tls-insecure: false

Loki client

Loki client to remote server

Options:

  • enable: (boolean) enable, set the enable to true
  • server-url: (string) Loki server url
  • job-name: (string) Job name
  • mode: (string) text or json
  • flush-interval: (integer) flush batch every X seconds
  • batch-size: (integer) batch size for log entries in bytes
  • retry-interval: (integer) interval in second between before to retry to send batch
  • text-format: (string) output text format, please refer to the default text format to see all available directives, use this parameter if you want a specific format
  • proxy-url: (string) Proxy URL
  • tls-support: (boolean) enable tls
  • tls-insecure: (boolean) insecure skip verify
  • basic-auth-login: (string) basic auth login
  • basic-auth-pwd: (string) basic auth password
  • tenant-id: (string) tenant/organisation id. If omitted or empty, no X-Scope-OrgID header is sent.
  lokiclient:
    enable: false
    server-url: "http://localhost:3100/loki/api/v1/push"
    job-name: "dnscollector"
    mode: "text"
    flush-interval: 5
    batch-size: 1048576
    retry-interval: 10
    text-format: "localtime identity qr queryip family protocol qname qtype rcode"
    proxy-url: ""
    tls-insecure: false
    basic-auth-login: ""
    basic-auth-pwd: ""
    tenant-id: ""

Statsd client

Statsd client to statsd proxy

  • tls support

Statsd metrics:

The <statsdsuffix> tag can be configured in the config.yml file.

Counters:

- <statsdsuffix>_<streamid>_total_bytes_received
- <statsdsuffix>_<streamid>_total_bytes_sent
- <statsdsuffix>_<streamid>_total_requesters
- <statsdsuffix>_<streamid>_total_domains
- <statsdsuffix>_<streamid>_total_domains_nx
- <statsdsuffix>_<streamid>_total_packets
- <statsdsuffix>_<streamid>_total_packets_[udp|tcp]
- <statsdsuffix>_<streamid>_total_packets_[inet|inet6]
- <statsdsuffix>_<streamid>_total_replies_rrtype_[A|AAAA|TXT|...]
- <statsdsuffix>_<streamid>_total_replies_rcode_[NOERROR|SERVFAIL|...]

Gauges:

- <statsdsuffix>_<streamid>_queries_qps

Options:

  • enable: (boolean) enable, set the enable to true
  • transport: (string) network transport to use: udp or tcp
  • listen-ip: (string) remote address
  • listen-port: (integer) remote tcp port
  • prefix: (string) statsd prefix name
  • tls-support: (boolean) enable tls
  • tls-insecure: (boolean) insecure skip verify
  statsd:
    enable: false
    transport: udp
    remote-address: 127.0.0.1
    remote-port: 8125
    prefix: "dnscollector"
    tls-support: false
    tls-insecure: false