jmxexporter adding incorrect _total suffix to #HELP and #TYPE when using COUNTER

[ecerulm]

I'm using a jmxrule to convert the requestCount from MBean Catalina:name="http-nio-8033",type=GlobalRequestProcessor to

• set metric name to catalina_globalrequestprocessor_requestcount
• give it a help text
• overwrite the labels:
• set the type to COUNTER

and although the metric name is set to catalina_globalrequestprocessor_requestcount the # HELP and # TYPE refer to catalina_globalrequestprocessor_requestcount_total (note the _total suffix at the end).

So the result prometheus text format is

# HELP catalina_globalrequestprocessor_requestcount_total number of requests
# TYPE catalina_globalrequestprocessor_requestcount_total counter
catalina_globalrequestprocessor_requestcount{Application="Tableau",Service="vizqlservice",ServiceInstance="0",} 2829.0

As far as I understand this is incorrect since the metric name is actually catalina_globalrequestprocessor_requestcount not catalina_globalrequestprocessor_requestcount_total . So the generated # HELP and #TYPE are really for different non-existing metric.

Here is the jmxexporter configuration:

---
jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:8750/jmxrmi
lowercaseOutputName: true
includeObjectNames:
  - "java.lang:type=Memory,*"
  # - "tableau.health.jmx:*"
  - "Catalina:type=Manager,*"
  - "Catalina:type=GlobalRequestProcessor,*"
excludeObjectNameAttributes:
  "java.lang:type=Memory":
    - NonHeapMemoryUsage
    - Verbose
    - ObjectPendingFinalizationCount
rules:
  - pattern: 'Catalina<type=GlobalRequestProcessor, name=\"(\w+-\w+)-(\d+)\"><>(\w+)Count:'
    name: catalina_globalrequestprocessor_$3count
    help: number of requests
    labels:
      Application: Tableau
      Service: vizqlservice
      ServiceInstance: '0'
    type: COUNTER

If I remove the type: COUNTER line then the prometheus text format generated will be

# HELP catalina_globalrequestprocessor_requestcount number of requests
# TYPE catalina_globalrequestprocessor_requestcount untyped
catalina_globalrequestprocessor_requestcount{Application="Tableau",Service="vizqlservice",ServiceInstance="0",} 2848.0

In this case the metrics name matches in all three lines (no _total anywhere) but the TYPE is not counter like I wanted it to be.

[fstab]

Is this still the case with jmx_exporter version 1.x, or are you using an older version?

[ecerulm]

Is this still the case with jmx_exporter version 1.x, or are you using an older version?

@fstab, Using jmxexporter 0.20.0 as 1.x says This release has functional issues and should not be used.

[ecerulm]

I have been google around and I guess the _total comes from prometheus client_java's PrometheusNaming RESERVED_METRIC_NAME_SUFFIXES.I'm assuming jmexporter uses client_java to generate the prometheus text format, and somehow jmxexporter overrides the metricname but it does not override the generated TYPE and HELP.

[fstab]

@dhoard time to release 1.0.1, this should fix it :)

[ecerulm]

The client_java documetation says

As defined in OpenMetrics, counter metric names must have the _total suffix. If you create a counter without the _total suffix the suffix will be appended automatically.

But in anycase jmxexporter should either produce

# HELP catalina_globalrequestprocessor_requestcount number of requests
# TYPE catalina_globalrequestprocessor_requestcount counter
catalina_globalrequestprocessor_requestcount{Application="Tableau",Service="vizqlservice",ServiceInstance="0",} 2848.0

or

# HELP catalina_globalrequestprocessor_requestcount_total number of requests
# TYPE catalina_globalrequestprocessor_requestcount_total counter
catalina_globalrequestprocessor_requestcount_total{Application="Tableau",Service="vizqlservice",ServiceInstance="0",} 2848.0

but not what's producing now, mixing catalina_globalrequestprocessor_requestcount_total and catalina_globalrequestprocessor_requestcount

[ecerulm]

@dhoard time to release 1.0.1, this should fix it :)

yes, it does fix it.. I tried with main 2797571 and when I specify catalina_globalrequestprocessor_requestcount as metric name I get

# HELP catalina_globalrequestprocessor_requestcount_total number of requests
# TYPE catalina_globalrequestprocessor_requestcount_total counter
catalina_globalrequestprocessor_requestcount_total{Application="Tableau",Service="vizqlservice",ServiceInstance="0"} 0.0

the metric name get rewritten to catalina_globalrequestprocessor_requestcount_total everywhere (TYPE, HELP and the actual metric) which I guess it's the correct behaviour (from the openmetrics documentation point of view)

so yes, it does fix it.

@fstab @dhoard , I suggest to mention in the docs for rules > name something like

The metric name to set. Capture groups from the pattern can be used. If not specified, the default format will be used. If it evaluates to empty, processing of this attribute stops with no output. Additional suffix may be added to this name (e.g _total for type COUNTER)

to help people that are not familiar with the _total suffix to discover it. (Specially since catalina_globalrequestprocessor_requestcount without the _total suffix is mentioned a lot on the internet, for example Prometheus metrics collected by the CloudWatch agent)

Then one question remains can I force jmxexporter to output catalina_globalrequestprocessor_requestcount (without total) and still be a counter? or if I really want catalina_globalrequestprocessor_requestcount I should leave it as UNKNOWN/untyped type?

[dhoard]

@ecerulm I added the documentation change above and am in the process of publishing the 1.0.1 release.

[dhoard]

@ecerulm release published. please test.

[ecerulm]

It works , it adds the _total suffix to the metric name, in all 3 places ( TYPE, HELP and the actual metric value).

[dhoard]

@ecerulm Thanks for the confirmation!