v1.1.0-beta4

If you find the project helpful, please consider supporting the project via monthly donation via GitHub Sponsors or simply add a star to this project.

Highlights

This release introduces some fixes and increase performance with file logger.

What's Changed

• feat: disable global telemetry by default + channel buffer size up to 8192 by @dmachard in #840
• feat: file logger: reducing the frequency of disk writes by @dmachard in #841
• feat: add arcount, nscount and ancount in json output #828 by @dmachard in #829
• feat: lookup ECS ip on geoip transform #776 by @dmachard in #830
• fix: invalid ingress counter for dnstap collector by @dmachard in #839
• fix: afpacket sniffer: fix deadlock on stop by @dmachard in #842
• fix: error on dns decoder for response without question by @dmachard in #843
• fix(doc): update transform_rewrite.md by @jpmens in #825
• chore(deps): bump github.com/tinylib/msgp from 1.2.1 to 1.2.2 by @dependabot in #834
• chore(deps): bump github.com/klauspost/compress from 1.17.9 to 1.17.10 by @dependabot in #824
• chore(deps): bump github.com/prometheus/client_golang from 1.20.3 to 1.20.4 by @dependabot in #823
• chore(deps): bump golang.org/x/net from 0.29.0 to 0.30.0 by @dependabot in #832
• chore(deps): bump github.com/prometheus/common from 0.59.1 to 0.60.0 by @dependabot in #833

Bug reporters

• @BarboraAni #831

New Contributors

• @jpmens made their first contribution in #825

Full Changelog: v1.0.0...v1.1.0-beta4

v1.0.0

If you find the project helpful, please consider supporting the project via monthly donation via GitHub Sponsors or simply add a star to this project.

Highlights

The release is the first stable release with great new features.

What's Changed

• feat: log malformed packet enabled by default by @dmachard in #819
• feat: build with go1.23 by @dmachard in #818
• feat: add tests for matching feature with pipeline mode by @dmachard in #797
• feat: convert latency field to float + new advanced example by @dmachard in #758
• feat: support round robin partitionner for kafka producer by @dmachard in #759 #809
• feat: update README.md and remove beta code warning by @dmachard in #755
• feat: add rewrite transform + quote some text directives by @dmachard in #756
• feat: update normalize transform to replace non printable characters by @dmachard in #754
• feat: add custom text format with jinja templating by @dmachard in #748
• feat: enhanced text-format with raw text by @peterand-pa in #609
• feat: move docker config to pipelines by @dmachard in #757
• fix: matching with arrays failed with dnsmessage collector by @dmachard in #797
• fix: decode properly dynamic/update dns queries by @dmachard in #819
• chore(deps): bump to loki 3.2 by @dmachard in #742 #768 #782 #817
• chore(docs): update README.md by @dmachard in #760
• chore(deps): bump github.com/Azure/sdk/azidentity from 1.5.1 to 1.6.0 by @dependabot in #743
• chore(deps): update unbound and pdns-recursor version in CI by @dmachard in #765
• chore(deps): bump github.com/prometheus/common from 0.48.0 to 0.59.1 by @dependabot in #726 #762 #806
• chore(deps): bump github.com/miekg/dns from 1.1.59 to 1.1.62 by @dependabot in #744 #788
• chore(deps): bump google.golang.org/protobuf from 1.34.1 to 1.34.2 by @dependabot in #745
• chore(deps): bump github.com/klauspost/compress from 1.17.8 to 1.17.9 by @dependabot in #746
• chore(deps): bump alpine from 3.20.0 to 3.20.3 by @dependabot in #752 #778 #802
• chore(deps): bump prometheus from 0.51.0 to 0.54.1 by @dependabot in #751 #773 #781 #794
• chore(deps): bump github.com/dmachard/go-dnstap-protobuf from 1.0.1 to 1.0.2 by @dependabot in #750
• chore(deps): bump github.com/oschwald/maxminddb-golang from 1.13.0 to 1.13.1 by @dependabot in #761
• chore(deps): bump golang.org/x/net from 0.26.0 to 0.28.0 by @dependabot in #767 #780
• chore(deps): bump golang.org/x/sys from 0.22.0 to 0.25.0 by @dependabot in #779 #783 #803
• chore(deps): bump github.com/IBM/sarama from 1.43.2 to 1.43.3 by @dependabot in #787
• chore(deps): bump github.com/creasty/defaults from 1.7.0 to 1.8.0 by @dependabot in #789
• chore(deps): bump github.com/cilium/ebpf from 0.15.0 to 0.16.0 by @dependabot in #777
• chore(deps): bump github.com/tinylib/msgp from 1.1.9 to 1.2.1 by @dependabot in #766 #804
• chore(deps): bump golang from 1.22.4-alpine3.19 to 1.22.6-alpine3.19 by @dependabot in #770 #784
• chore(deps): bump google.golang.org/grpc from 1.64.0 to 1.64.1 by @dependabot in #772
• chore(deps): bump github.com/prometheus/client_golang from 1.19.1 to 1.20.3 by @dependabot in #805

Bug reporters

• @drenton1 #799

New Contributors

• @peterand-pa made their first contribution in #609

Full Changelog: v0.46.0...v1.0.0

v0.46.0

If you find the project helpful, please consider supporting the project via monthly donation via GitHub Sponsors or simply add a star to this project.

Highlights

The release is the last one before the first stable release. In the meantime, this release introduces some great features:

• Some new global metrics to follow your pipeline performance and dropped packets.
• The multiplexer configuration mode is marked as DEPRECATED.
  Please to update your config with pipelines definition.
• The memory usage of the tool has been reduced.

What's Changed

• feat: add integration examples with Loki, Prometheus and InfluxDB by @dmachard in #740
• feat: new global settings for default buffer size set to 4096 instead of 65535 by @dmachard in #739
• feat: add global telemetry #641 #512 by @dmachard in #724
• feat: mark as deprecated the multiplexer mode by @dmachard in #719
• feat: add pid file support by @dmachard in #723
• fix(regression): dnstap files with ingestor not properly processed by @dmachard in #732
• fix: invalid error message with config checker by @dmachard in #733
• chore(deps): bump github.com/oschwald/maxminddb-golang from 1.12.0 to 1.13.0 by @dependabot in #725
• chore(deps): bump golang from 1.22.3-alpine3.19 to 1.22.4-alpine3.19 by @dependabot in #734
• chore(deps): bump golang.org/x/sys from 0.20.0 to 0.21.0 by @dependabot in #735
• chore(deps): bump goreleaser/goreleaser-action from 5 to 6 by @dependabot in #730
• chore(deps): bump golang.org/x/net from 0.25.0 to 0.26.0 by @dependabot in #736
• chore(deps): bump alpine from 3.19.1 to 3.20.0 by @dependabot in #721

Bug reporters

• @romainw #729

Full Changelog: v0.45.0...v0.46.0

v0.45.0

If you find the project helpful, please consider supporting the project via monthly donation via GitHub Sponsors or simply add a star to this project.

Highlights

The focus of this update is again bug fixes and code factoring to prepare future stable release.

What's Changed

• fix: support properly yaml tag flow in config checker by @dmachard in #716
• fix: to avoid deadlock on stop with the XDP sniffer by @dmachard in #712
• chore(docs): add missing docs and tests for dnsmessage collector by @dmachard in #717
• chore(refactor): redesign transformers by @dmachard in #715
• chore(refactor): use external default library to set default values in config by @dmachard in #714
• chore(refactor): remove xdp and pkgutils packages by @dmachard in #710
• chore(refactor): use external go-netutils library by @dmachard in #713 @dmachard in #709
• chore(refactor): remove processors package by @dmachard in #704
• chore(deps): bump github.com/prometheus/client_golang from 1.19.0 to 1.19.1 by @dependabot in #706
• chore(deps): bump golang from 1.22.2-alpine3.19 to 1.22.3-alpine3.19 by @dependabot in #707
• chore(deps): bump google.golang.org/protobuf from 1.33.0 to 1.34.1 by @dependabot in #697
• chore(deps): bump golangci/golangci-lint-action from 5 to 6 by @dependabot in #700
• chore(deps): bump github.com/cilium/ebpf from 0.14.0 to 0.15.0 by @dependabot in #698
• chore(deps): bump github.com/dmachard/go-clientsyslog from 0.3.0 to 0.4.0 by @dependabot in #705
• chore(deps): bump github.com/IBM/sarama from 1.43.1 to 1.43.2 by @dependabot in #699

Full Changelog: v0.44.0...v0.45.0

v0.44.0

If you find the project helpful, please consider supporting the project via monthly donation via GitHub Sponsors or simply add a star to this project.

Highlights

The focus of this update is on maintenance to prepare future stable release

• Fixed support for IP fragments with AFPACKET sniffer.
• Major code refactoring of the base code.
• Completely redesigned the configuration checker.

What's Changed

• feat: support IP defrag on afpacket sniffer by @dmachard in #691
• feat: redesign config checker to support freeform keys by @dmachard in #678
• feat: add Scalyr defaults by @pieterlexis-tomtom in #677
• fix: quick fix to avoid panic with fragmented packet by @dmachard in #684
• fix: kafka - add missing textformat setting by @dmachard in #694
• chore(refactor): major code refactoring and cleanup by @dmachard in #667 #702 #696 #701
• chore(deps): bump github.com/klauspost/compress from 1.17.7 to 1.17.8 by @dependabot in #680
• chore(deps): bump golangci/golangci-lint-action from 4 to 5 by @dependabot in #692
• chore(deps): bump github.com/miekg/dns from 1.1.58 to 1.1.59 by @dependabot in #687
• chore(deps): bump github.com/dmachard/go-dnstap-protobuf from 1.0.0 to 1.0.1 by @dependabot in #688
• chore(deps): bump github.com/dmachard/go-powerdns-protobuf from 1.1.0 to 1.1.1 by @dependabot in #689

Bug reporters

• @top-secrett #683
• @pieterlexis-tomtom #676

Full Changelog: v0.43.0...v0.44.0

v0.43.0

If you find the project helpful, please consider supporting the project via monthly donation via GitHub Sponsors or simply add a star to this project.

Highlights

This release introduces some great features

• Compression support on DNSTap collector and logger
• Custom Flat-JSON relabeling
• New ClickHouse logger thank to @zunnurainbadar

What's Changed

• feat: transform: normalize lowercase RR's by @dmachard in #665
• feat: build with golang 1.22 and 1.20 support removed by @dmachard in #660
• feat: added Clickhouse Logger by @zunnurainbadar in #349
• feat: add compression support on DNSTap client and server by @dmachard in #538
• feat: new relabeling transform by @dmachard in #645
• fix: relabel-config error in loki client by @dmachard in #664
• chore(docs): add more docs for the atags transformer by @dmachard in #656 #659
• chore(deps): bump golang.org/x/sys from 0.17.0 to 0.18.0 by @dependabot in #651
• chore(deps): bump golang.org/x/net from 0.21.0 to 0.22.0 by @dependabot in #652
• chore(deps): bump Loki to v2.9.6
• chore(refactor): code cleanup on transformers by @dmachard in #666

New Contributors

• @zunnurainbadar made their first contribution in #349

Bug reporters

• @rajmohanram #643

Full Changelog: v0.42.0...v0.43.0

v0.42.0

If you find the project helpful, please consider supporting the project via monthly donation via GitHub Sponsors or simply add a star to this project.

Highlights

This release introduces some important bug fixes (memory leak) and minor features

• Fix memory leak with ElasticSearch logger, thank to @misaki-kawakami to report that.
• Optimizations to reduce CPU usage, more particularly for flat-json
• Add support for query-zone field on DNStap collector

What's Changed

• feat: decode dns class on queries and replies by @dmachard in #635
• feat: more docs for kafka and fluentd loggers, change channel buffer to 4096 by @dmachard in #637 #639
• feat: normalize transformer: add icann managed flag in dnsmessage by @dmachard in #640
• feat: refactor to optimize dnsmessage model by @dmachard in #632
• feat: dnstap collector: decode query zone field by @dmachard in #621
• feat: dnstap collector: ignore empty raw dns payload by @dmachard in #622
• fix: elasticsearch: release buffer properly to avoid memory leaks by @dmachard in #630 #644
• fix: build to support 32b platform by @dmachard in #620
• chore(deps): bump github.com/klauspost/compress from 1.17.6 to 1.17.7 by @dependabot in #625
• chore(deps): bump github.com/stretchr/testify from 1.8.4 to 1.9.0 by @dependabot in #633
• chore(deps): bump golang.org/x/sys from 0.16.0 to 0.17.0 by @dependabot in #606
• chore(deps): bump golang.org/x/net from 0.20.0 to 0.21.0 by @dependabot in #607
• chore(deps): bump github.com/prometheus/client_model from 0.5.0 to 0.6.0 by @dependabot in #616
• chore(deps): bump google.golang.org/protobuf from 1.32.0 to 1.33.0 by @dependabot in #647
• chore(deps): bump github.com/prometheus/client_golang from 1.18.0 to 1.19.0 by @dependabot in #648
• chore(ci): add timeout in github actions tests by @dmachard in #623
• chore(ci): add bench-go github actions by @dmachard in #638
• chore(ci): bump softprops/action-gh-release from 1 to 2 by @dependabot in #646

Bug reporters

• @mikeS7
• @misaki-kawakami

Full Changelog: v0.41.0...v0.42.0

v0.41.0

If you find the project helpful, please consider supporting the project via monthly donation via GitHub Sponsors or simply add a star to this project.

Highlights

This release introduces some new features and bug fixes.

• RPZ policy fields support on DNStap/PowerDNS protobuf collectors
• A new peer-name text-format directive and JSON field, thanks to @peterand-pa for idea
• Support the DoH3 HTTP field added with the release of DNSdist 1.9

What's Changed

• feat: add doh3 http version field support and test for DoQ by @dmachard in #610
• feat: add peer name text-format directive by @dmachard in #608
• feat: add configuration examples for pipeline mode by @dmachard in #599
• feat: add DNStap policy support by @dmachard in #593 #530
• feat: add RPZ policy details with powerdns protobuf by @dmachard in #590 #572
• feat: expose dns id in json model by @dmachard in #586
• feat: detect and exit on error with invalid text directives by @dmachard in #576
• fix: fluentd: remove msgpack lib and duplicates entries #611 by @dmachard in #612
• fix: add missing rd and cd flags in json model by @dmachard in #589
• fix: dnstap collector: stuck on stop can be possible in specific situation by @dmachard in #597
• fix: syslog logger - fix protocol wrong type for unix socket by @dmachard in #598
• chore(docs): add documentation to use docker-compose by @dmachard in #563
• chore(docs): update makefile and docs by @dmachard in #604
• chore(deps): bump golangci/golangci-lint-action from 3 to 4 by @dependabot in #603
• chore(deps): bump alpine from 3.19.0 to 3.19.1 by @dependabot in #577
• chore(deps): bump github.com/google/uuid from 1.5.0 to 1.6.0 by @dependabot in #579
• chore(deps): bump github.com/klauspost/compress from 1.17.4 to 1.17.6 by @dependabot in #596
• chore(deps): bump github.com/dmachard/go-topmap from 0.5.0 to 1.0.0 by @dependabot in #580
• chore(deps): bump github.com/dmachard/go-powerdns-protobuf from 1.0.0 to 1.1.0 by @dependabot in #578

Full Changelog: v0.40.2...v0.41.0

Bug reporters

• @johnhtodd
• @peterand-pa
• @Jon1760

v0.40.2

If you find the project helpful, please consider supporting the project via monthly donation via GitHub Sponsors or simply add a star to this project.

What's Changed

• fix(major): patch to avoid deadlock situations in multiplexer mode - regression by @dmachard in #569
• fix(minor): detect invalid logger type in configuration file by @dmachard in #573
• fix(minor): add tests to check and avoid deadlocks with pipeline mode by @dmachard in #571
• chore(deps): bump github.com/segmentio/kafka-go from 0.4.46 to 0.4.47 by @dependabot in #517
• chore(deps): bump github.com/google/uuid from 1.4.0 to 1.5.0 by @dependabot in #518
• chore(deps): bump golang.org/x/crypto from 0.16.0 to 0.17.0 by @dependabot in #519
• chore(deps): bump google.golang.org/protobuf from 1.31.0 to 1.32.0 by @dependabot in #524
• chore(deps): clean security code scanning and some dependencies by @dmachard in #554
• chore(deps): bump Loki to v2.9.3 by @dmachard in #555 #556
• chore(deps): bump golang.org/x/sys from 0.15.0 to 0.16.0 by @dependabot in #559
• chore(deps): bump github.com/dmachard/go-framestream from 0.7.0 to 0.10.0 by @dependabot in #560
• chore(deps): bump github.com/miekg/dns from 1.1.57 to 1.1.58 by @dependabot in #561
• chore(deps): bump github.com/prometheus/common from 0.45.0 to 0.46.0 by @dependabot in #558

Full Changelog: v0.40.0...v0.40.2

v0.40.0

If you find the project helpful, please consider supporting the project via monthly donation via GitHub Sponsors or simply add a star to this project.

Highlights

This release is major and introduces some important breaking changes and bug fixes.

• A new codebase to support the future pipelining mode, adding more flexibility to handle your incoming logs.
• New Prometheus metrics has been added to fix a potential unlimited memory usage, which is not acceptable as resources are not limitless.
• The possibility to extend ougoing DNStap streams with metadata such as geoip, filtering, ... It applied to have a DNScollector in remote side to decode these extended DNStap messages
• Invalid configurations are now properly detected.

What's Changed

• feat (#516): dnstap logger/collector - support extended field by @dmachard in #548
• feat: dnstap logger: support of DNSTap extra field by @dmachard in #547
• feat: experimental support for pipelining by @dmachard in #503
• feat (#473): configurable CIDR mask and hash for user-privacy transformer by @dmachard in #528
• feat (#522): more checks to detect invalid configurations by @dmachard in #523
• feat: rename dnstap logger to dnstapclient in YAML config (breaking changes) by @dmachard in #526
• feat: avoid unlimited mem usage with prometheus logger (breaking changes in metrics) by @dmachard in #529
• feat: update grafana dashboard according to PR#529 by @dmachard in #542
• fix: kafka, redis, fluentd and tcpclient loggers: no flush occurred on specific conditions with connection attempts by @dmachard in #552
• fix: syslog logger: fix crash when remote server refused connections by @dmachard in #551
• fix: bad yaml key for dnstap-relay collector by @dmachard in #546
• fix (#539): missing latency when quiet-text is enabled by @dmachard in #540
• fix: typo error - text format directive reducer-occurences renamed to reducer-occurrences
• chore(deps): bump golang.org/x/sys from 0.14.1-.... to 0.15.0 by @dependabot in #486
• chore(deps): bump github.com/segmentio/kafka-go from 0.4.45 to 0.4.46 by @dependabot in #494
• chore(deps): bump golang.org/x/net from 0.18.0 to 0.19.0 by @dependabot in #495
• chore(deps): bump github.com/klauspost/compress from 1.17.3 to 1.17.4 by @dependabot in #496
• chore(deps): bump alpine from 3.18.5 to 3.19.0 by @dependabot in #505
• chore(deps): bump golang from 1.21.4-alpine3.17 to 1.21.5-alpine3.17 by @dependabot in #506
• chore(ci): update testing-dnstap.yml with unbound 1.19 by @dmachard in #543
• chore(docs): docs updated by @dmachard in #541

Full Changelog: v0.39.0...v0.40.0

Bug reporters

• @top-secrett
• @johnhtodd