v1.8.3

Fixed

• Corrected configuration for pumps.kafka.meta.timeout to be interpreted as the number of seconds (Type: int) instead of a duration requiring a unit (Type: Duration). (#696)

• Fixed an issue where Graph SQL Pump couldn't restart correctly when analytics storage table name was changed in pump config. Some relations were not torn down and migrated correctly. (#715 )

draft

v1.8.2-rc3

add configurable batch options

v1.8.2

Highlights

Changelog

Fixed

• Resolved performance issue where SQL Aggregate analytics failed to load on the Dashboard during heavy traffic by introducing a new index on the sql_aggregate pump called idx_dimension. (#715)

• Fixed Prometheus Pump crashes on non UTF-8 URLs by updating to prometheus-client v1.16. (#716)

• Addressed MongoDB connection string issues related to certain characters ("?" and "@") by recommending URL-encoded values in usernames and passwords, ensuring compatibility with both mgo and mongo-go drivers. (#698)

• Applied fixes for security vulnerabilities: CVE-2022-36640, CVE-2022-21698, GO-2022-0322, GHSA-cg3q-j54f-5p7p. (#682)

Added

• Add track_all_paths configuration for Prometheus Pump. If enabled, all APIs will have path in the tyk_http_status_per_path metric. Otherwise, only endpoint that have "track" plugin set with have path shown in the metric. Endpoints without “track” plugin set will have “unknown” path shown in the metric. (#716)

Updated

• Improved security by obfuscating Mongo Pump credentials in log outputs. (#686)

v1.8.1

Highlights

Notes on MongoDB v5 and v6 compatibility

For MongoDB v5 and v6 users, please set mongo driver type to 'mongo-go'.

From 1.8.1, the default MongoDB driver it use is from mgo. This is to align with default MongoDB driver of other Tyk components. This driver supports MongoDB versions up to v4. If you are using later version of MongoDB v5 or MongoDB v6, please follow this guide to update the driver type to mongo-go.

Changelog

Fixed

• Reverted a change from previous release that excluded GraphQL analytics records from tyk_analytics collection for Mongo Pump

• Fixed MongoDB connection issue when using a password with URL escape characters (with mongo-go driver)

• Fixed an issue that when processing an analytics record with an API name or path containing the -- separator used to join label values (e.g., web--test-beta), an "inconsistent label cardinality" error occurs and causes tyk pump to crash

• When omit_configfile is set to true, Pump will not try to load the config file and spit out error logs

Updated

• Updated the default Hybrid Pump RPC pool size from 20 to 5 connections in order to reduce default CPU and memory footprint. See Pump configurations

• Import and use latest storage library v1.0.5

• Updated default MongoDB driver to mgo. Follow this guide to update the driver type

• Pump name is now case-insensitive. It will override two or more pumps with same name but in different cases (e.g. “Mongo” / “mongo”)

v1.8.2-rc1

RC to test prometheus fix #603

v1.8.0

Summary

Pump 1.8 introduces two new pumps: The GraphQL SQL Aggregate Pump - which allows you to transfer GraphQL transaction logs to SQL; and Resurface Pump - which allows you to transfer data to Resurface.io for context based security analysis.

We have changed the default MongoDB driver from mgo to mongo-go. The new driver supports MongoDB versions greater or equal to v4. If you are using older version of MongoDB v3.x, please update the driver type here.

We have also added a config option that allow you to decode the raw requests and responses for all pumps so you don't need to worry about processing them in your data pipeline. For demo mode, there is now an option to generate future data for your convenience.

In this release, we are using a new Tyk storage library to connect to Mongo DB. This would allow us to switch to use the official Mongo Driver very easily in the future.

Notes on MongoDB v3.x compatibility

We have changed the default MongoDB driver from mgo to mongo-go. The new driver supports MongoDB versions greater or equal to v4. If you are using older version of MongoDB v3.x, please update the driver type here.

Changelog

Added

• Added GraphQL SQL Aggregate Pump. (#566)
• Added Resurface Pump - Resurface can provide context-based security analysis for attack and failure triage, root cause, threat and risk identification based on detailed API logs sent from Tyk Pump. (#531)
• Add config option raw_request_decoded and raw_response_decoded for decoding from base64 the raw requests/responses fields before writing to Pump. This is useful if you want to search for specific values in the raw request/response. Both are disabled by default. This setting is not available for Mongo and SQL pumps, since the dashboard will decode the raw request/response. (#558)
• Add the ability to generate future data in demo mode using --demo-future-data flag. (#549)
• Remove critical CVE go.uuid vulnerability (#582)
• Use the latest Tyk storage library to connect to Mongo (#597)
• Hybrid Pump refactoring - we now have better RPC connection control, testability, and documentation (#588)

Fixed

• Std pump does not log accurate time when set to json format (#574)
• GraphPump doesn’t include names of queries/mutation and subscriptions called (#571)
• Mongo Pump’s connection hangs forever if misconfigured (#597)

v1.7.0

Summary

Pump 1.7 is a significant release, adding a new pump that will transfer GraphQL transaction logs to MongoDB (sorry, no support for SQL just yet). There are also several feature enhancements and bug fixes, mainly for the Prometheus and Mongo pumps.

Changelog

Added

• New Graph Granular Mongo Pump #489
• Mongo aggregate pump self-healing capability #486
• PrometheusPump write optimizations #452
• Improved Tyk Pump demo mode #513
• Support for using the key alias as a label for the Prometheus pump #464
• New field ApiSchema in analytics records #474
• Mongo pumps will now ignore graphQL analytics #463
• CosmoDB support for Mongo pumps #481
• Support MongoDB Connection argument "readPreference" #517
• New configuration option to Ignore Fields for all the pumps #519
• New configuration option to use ssl_insecure_skip_verify with Elasticsearch Pump #502

Fixed

• Logs expose Mongo connection string credentials #518
• Logger normalization #465
• Document size is miscalculated in Mongo Selective Pump #496
• Prometheus pump custom metrics Histogram type label validation #497
• Prometheus pump environment vars fix #509
• Prometheus pump custom metrics panic #475

v1.6.0

Changelog

Added

• Prometheus Pump custom metrics #431
• Added a new configuration for Mongo Pumps (omit_index_creation) to avoid index creation #442
• Added protobuf-serialized analytics records processing #436
• Added a new configuration (separated_method) for StatsD pump to separate method and path fields #450

Fixed

• Analytics records tags sanitization #447 #451
• Fixed a race condition when using multiple pumps with filters #453
• Fixed Mongo Aggregate bug when using ignore_aggregations #448
• Changed the default behavior of Mongo pumps indexes depending on the Mongo DB type. More info at https://tyk.io/docs/tyk-pump/tyk-pump-configuration/tyk-pump-dashboard-config/#omitting-indexes #442
• Moesif Pump now flushes the buffered record before shutting down #449
• Updated version logs in Elastic Search pump #446

v1.5.1

Changelog

• New config option to ignore configuration file #400
• Tyk pump documentation now is generated from code #401
• Mongo pumps indexes bugfix. Now indexes are not auto generated if the mongo collection exists #405
• New Influx2 pump #406
• API tags are now published to Kafka pump messages #407

v1.5.0

Changelog

• Tyk pump "graceful" shutdown #388
• Tyk pump new configuration option for record size limiting #392 #395
• Tyk pump now validates and exclude empty tags in aggregations #380
• New SQL Pump, SQL Aggregate Pump, and SQL Uptime Pump #314
• Prometheus Pump timeout and superfluous error bugfix #389 #390
• Splunk Pump batch write #381
• Splunk Pump new analytics fields and ignore_tag_prefix_list configuration option #377
• Mongo Pumps improved logs #383
• ElasticSearch Pump improved logs #319
• ElasticSearch Pump now supports ElasticSearch 7 #385

Docker images

• docker pull tykio/tyk-pump-docker-pub:v1.5.0
• docker pull tykio/tyk-pump-docker-pub:v1.5