kafka-clients-metrics: Refactor node topic metrics

[meiao]

Overview

The previous code executed more verifications than needed for its functionality, it also stored duplicate data for those verifications. This PR simplifies the implementation by reducing the number of verifications and data stored.

Functionality

The functionality being altered is to generate metrics (or event attributes) with the format:

1. MessageBroker/Kafka/Nodes/
2. MessageBroker/Kafka/Nodes//<Produce|Consume>/

For each node a client connects to, a metric of the first format will be created. Then for the cartesian product of the sets of node x topic, a metric of the second format will be created.

So, for a producer client connecting to nodes (n1, n2) and producing for the topics (t1, t2), the following metrics will be created:

• MessageBroker/Kafka/Nodes/n1
• MessageBroker/Kafka/Nodes/n1/Produce/t1
• MessageBroker/Kafka/Nodes/n1/Produce/t2
• MessageBroker/Kafka/Nodes/n2
• MessageBroker/Kafka/Nodes/n2/Produce/t1
• MessageBroker/Kafka/Nodes/n2/Produce/t2

Current implementation

For each Kafka client, a NewRelicMetricsReporter is instantiated. On its constructor, it creates a map with one NodeMetricNames (value) for each node (key) a client connects to. (The key is never used anywhere in the code. All references for this map have a .value() call after it).
When NodeMetricNames is instantiated, it creates a metric with format 1 for its node.
When a metric is registered on NewRelicMetricsReporter, if it has a topic, iterate thru the NodeMetricNames registering that topic.
In NodeMetricNames if the topic was not seen before, create a metric name (and an event attribute) with the format 2 and mark the topic as seen.

So for each topic, <node count> checks are made to create the metric with format 2, also <node count> sets are kept to keep the seen topics. Since a new topic will be added to all nodes, there is no reason to make the check for each node, nor keep a set of topics for each node.

Another difference of the new implementation is that it only keeps metric names or event attribute labels instead of both. Since for an agent run only one of either will ever be used, this will save a little memory.

New implementation

A new class NodeTopicRegistry was created. It's functionality is similar to NodeMetricNames, but the newer keeps track of all nodes and topics, so only one instance of it is created for each NewRelicMetricsReporter.
When NodeTopicRegistry is instantiated, it receives all the nodes the Kafka client connects to, and creates all the metric names of format 1.
When a metric is registered with the NewRelicMetricsReporter, it will be passed to the NodeTopicRegistry and if that topic was not seen before, new metrics of format 2 will be created with the new topic for each node.

So only one check for each topic is done, as well as only one collection of seen topics is kept.

Related Github Issue

This change was made while adding the same functionality for the kafka-clients-node-metrics-x.y (#2186)

[meiao]

This field is not altered after the constructor, so not bothering to add concurrency control to it.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.58%. Comparing base (734ac3d) to head (5b67864).
Report is 53 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2195      +/-   ##
============================================
- Coverage     70.67%   70.58%   -0.09%     
+ Complexity     9996     9976      -20     
============================================
  Files           842      842              
  Lines         40363    40369       +6     
  Branches       6116     6118       +2     
============================================
- Hits          28525    28494      -31     
- Misses         9084     9096      +12     
- Partials       2754     2779      +25     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jasonjkeller]

Looks good!