Clarify in documentation "jvm_*" metrics

[ecerulm]

I thought that the metrics

• jvm_classes_loaded
• jvm_threads_current
• jvm_threads_daemon
• jvm_memory_bytes_used
• jvm_memory_pool_bytes_used

were "standard" in jmx_exporter but it seems that they are only exported for the javagent and not the httpserver. I don't think this is mentioned in the documentation, I think it would be good to clarify somewhere the discrepancy between javaagent in httpserver.

Just to illustrate my point

httpserver

Running the jmxexporter httpserver give 0 jvm_ metrics

java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=localhost -jar jenkins.war

cat config2.yaml
---
jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:9010/jmxrmi

java -jar jmx_prometheus_httpserver-0.20.0.jar 12345 config2.yaml

curl -s localhost:12345|grep -e '^jvm_' # 0 results

javaagent

On the other hand jmxexporter agent give a 98 jvm_ metrics

java -javaagent:./jmx_prometheus_javaagent-0.20.0.jar=12345:./config.yaml  -jar jenkins.war

curl -s localhost:12345|grep -e '^jvm_'
jvm_buffer_pool_used_bytes{pool="mapped",} 0.0
jvm_buffer_pool_used_bytes{pool="direct",} 75618.0
jvm_buffer_pool_used_bytes{pool="mapped - 'non-volatile memory'",} 0.0
jvm_buffer_pool_capacity_bytes{pool="mapped",} 0.0
jvm_buffer_pool_capacity_bytes{pool="direct",} 75618.0
jvm_buffer_pool_capacity_bytes{pool="mapped - 'non-volatile memory'",} 0.0
jvm_buffer_pool_used_buffers{pool="mapped",} 0.0
jvm_buffer_pool_used_buffers{pool="direct",} 1.0
jvm_buffer_pool_used_buffers{pool="mapped - 'non-volatile memory'",} 0.0
jvm_classes_currently_loaded 9592.0
jvm_classes_loaded_total 9592.0
jvm_classes_unloaded_total 0.0
jvm_threads_current 43.0
jvm_threads_daemon 28.0
jvm_threads_peak 69.0
jvm_threads_started_total 90.0
jvm_threads_deadlocked 0.0
jvm_threads_deadlocked_monitor 0.0
jvm_threads_state{state="NEW",} 0.0
jvm_threads_state{state="TERMINATED",} 0.0
jvm_threads_state{state="RUNNABLE",} 16.0
jvm_threads_state{state="BLOCKED",} 0.0
jvm_threads_state{state="WAITING",} 12.0
jvm_threads_state{state="TIMED_WAITING",} 15.0
jvm_threads_state{state="UNKNOWN",} 0.0
jvm_info{runtime="OpenJDK Runtime Environment",vendor="Homebrew",version="17.0.11+0",} 1.0
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'profiled nmethods'",} 1.3142528E7
jvm_memory_pool_allocated_bytes_total{pool="G1 Old Gen",} 3.866692352E10
jvm_memory_pool_allocated_bytes_total{pool="G1 Eden Space",} 4.8125444096E10
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-profiled nmethods'",} 3523072.0
jvm_memory_pool_allocated_bytes_total{pool="G1 Survivor Space",} 2.786404E7
jvm_memory_pool_allocated_bytes_total{pool="Compressed Class Space",} 5706552.0
jvm_memory_pool_allocated_bytes_total{pool="Metaspace",} 4.6890176E7
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-nmethods'",} 3692288.0
jvm_gc_collection_seconds_count{gc="G1 Young Generation",} 67.0
jvm_gc_collection_seconds_sum{gc="G1 Young Generation",} 0.253
jvm_gc_collection_seconds_count{gc="G1 Old Generation",} 0.0
jvm_gc_collection_seconds_sum{gc="G1 Old Generation",} 0.0
jvm_memory_objects_pending_finalization 0.0
jvm_memory_bytes_used{area="heap",} 1.43083544E9
jvm_memory_bytes_used{area="nonheap",} 7.2721264E7
jvm_memory_bytes_committed{area="heap",} 2.89406976E9
jvm_memory_bytes_committed{area="nonheap",} 7.5497472E7
jvm_memory_bytes_max{area="heap",} 1.7179869184E10
jvm_memory_bytes_max{area="nonheap",} -1.0
jvm_memory_bytes_init{area="heap",} 1.073741824E9
jvm_memory_bytes_init{area="nonheap",} 7667712.0
jvm_memory_pool_bytes_used{pool="CodeHeap 'non-nmethods'",} 1355264.0
jvm_memory_pool_bytes_used{pool="Metaspace",} 4.7635568E7
jvm_memory_pool_bytes_used{pool="CodeHeap 'profiled nmethods'",} 1.3831552E7
jvm_memory_pool_bytes_used{pool="Compressed Class Space",} 5776000.0
jvm_memory_pool_bytes_used{pool="G1 Eden Space",} 1.358954496E9
jvm_memory_pool_bytes_used{pool="G1 Old Gen",} 6.2839296E7
jvm_memory_pool_bytes_used{pool="G1 Survivor Space",} 9041648.0
jvm_memory_pool_bytes_used{pool="CodeHeap 'non-profiled nmethods'",} 4122880.0
jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-nmethods'",} 3342336.0
jvm_memory_pool_bytes_committed{pool="Metaspace",} 4.8103424E7
jvm_memory_pool_bytes_committed{pool="CodeHeap 'profiled nmethods'",} 1.3893632E7
jvm_memory_pool_bytes_committed{pool="Compressed Class Space",} 6029312.0
jvm_memory_pool_bytes_committed{pool="G1 Eden Space",} 1.811939328E9
jvm_memory_pool_bytes_committed{pool="G1 Old Gen",} 1.065353216E9
jvm_memory_pool_bytes_committed{pool="G1 Survivor Space",} 1.6777216E7
jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-profiled nmethods'",} 4128768.0
jvm_memory_pool_bytes_max{pool="CodeHeap 'non-nmethods'",} 7606272.0
jvm_memory_pool_bytes_max{pool="Metaspace",} -1.0
jvm_memory_pool_bytes_max{pool="CodeHeap 'profiled nmethods'",} 1.22023936E8
jvm_memory_pool_bytes_max{pool="Compressed Class Space",} 1.073741824E9
jvm_memory_pool_bytes_max{pool="G1 Eden Space",} -1.0
jvm_memory_pool_bytes_max{pool="G1 Old Gen",} 1.7179869184E10
jvm_memory_pool_bytes_max{pool="G1 Survivor Space",} -1.0
jvm_memory_pool_bytes_max{pool="CodeHeap 'non-profiled nmethods'",} 1.22028032E8
jvm_memory_pool_bytes_init{pool="CodeHeap 'non-nmethods'",} 2555904.0
jvm_memory_pool_bytes_init{pool="Metaspace",} 0.0
jvm_memory_pool_bytes_init{pool="CodeHeap 'profiled nmethods'",} 2555904.0
jvm_memory_pool_bytes_init{pool="Compressed Class Space",} 0.0
jvm_memory_pool_bytes_init{pool="G1 Eden Space",} 5.8720256E7
jvm_memory_pool_bytes_init{pool="G1 Old Gen",} 1.015021568E9
jvm_memory_pool_bytes_init{pool="G1 Survivor Space",} 0.0
jvm_memory_pool_bytes_init{pool="CodeHeap 'non-profiled nmethods'",} 2555904.0
jvm_memory_pool_collection_used_bytes{pool="G1 Eden Space",} 0.0
jvm_memory_pool_collection_used_bytes{pool="G1 Old Gen",} 0.0
jvm_memory_pool_collection_used_bytes{pool="G1 Survivor Space",} 9041648.0
jvm_memory_pool_collection_committed_bytes{pool="G1 Eden Space",} 1.811939328E9
jvm_memory_pool_collection_committed_bytes{pool="G1 Old Gen",} 0.0
jvm_memory_pool_collection_committed_bytes{pool="G1 Survivor Space",} 1.6777216E7
jvm_memory_pool_collection_max_bytes{pool="G1 Eden Space",} -1.0
jvm_memory_pool_collection_max_bytes{pool="G1 Old Gen",} 1.7179869184E10
jvm_memory_pool_collection_max_bytes{pool="G1 Survivor Space",} -1.0
jvm_memory_pool_collection_init_bytes{pool="G1 Eden Space",} 5.8720256E7
jvm_memory_pool_collection_init_bytes{pool="G1 Old Gen",} 1.015021568E9
jvm_memory_pool_collection_init_bytes{pool="G1 Survivor Space",} 0.0
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'profiled nmethods'",} 1.716901872608E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Old Gen",} 1.71690187261E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Eden Space",} 1.71690187261E9
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-profiled nmethods'",} 1.71690187261E9
jvm_memory_pool_allocated_bytes_created{pool="G1 Survivor Space",} 1.71690187261E9
jvm_memory_pool_allocated_bytes_created{pool="Compressed Class Space",} 1.71690187261E9
jvm_memory_pool_allocated_bytes_created{pool="Metaspace",} 1.71690187261E9
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-nmethods'",} 1.71690187261E9

[dhoard]

@ecerulm we have a subtle statement...

"It can be also run as a standalone HTTP server and scrape remote JMX targets, but this has various disadvantages, such as being harder to configure and being unable to expose process metrics (e.g., memory and CPU usage)."

What would you propose?

---

The standalone can export JVM metrics, but they will be reported more in line with the actual MBean ObjectNames...

Example:

# HELP java_lang_threading_daemonthreadcount java.lang:name=null,type=Threading,attribute=DaemonThreadCount
# TYPE java_lang_threading_daemonthreadcount untyped
java_lang_threading_daemonthreadcount 58.0

[ecerulm]

What would you propose?

I wrote #965 with a specific proposal. My issues is that I tried searching for jvm_ in the docs and I could find any reference.

The standalone can export JVM metrics, but they will be reported more in line with the actual MBean ObjectNames...

And that is perfectly fine, it just that jvm_threads_daemon, jvm_memory_bytes_used mentioned in some AWS docs and then I couldn't find any reference to those. It took me a bit to realize that javagent exposed jvm_ but httpserver didn't. Now I know, but hopefully #965 change will allow other people to realize quicker that I did 😃

In my case I can't use the javagent because the application I'm trying to monitor (Tableau) does not allow me to add the agent. It comes with it's own tool to start many java services and it does not allow to customize any java options (not supported either). But "regular" JMX is allowed, hence jmxexporter http server is a good fit for me.

[ecerulm]

The standalone can export JVM metrics, but they will be reported more in line with the actual MBean ObjectNames...

Many (most?) of the JVM metrics are directly accessible from MBeans attributes, yes.
Some other jvm metrics like jvm_threads_deadlocked (jmxexporter-javaagent) are simply not accessible via MBean attributes and therefore you can't get them with the jmxexporter-httpserver. In particular jvm_threads_deadlocked is implemented in client_java's JvmThreadsMetrics and it uses a MBean operation (not an attribute). JvmThreadsMetrics will call ThreadMXBean.findMonitorDeadlockedThreads on the MBean.

The above is just for reference in case somebody lands here looking for jvm_threads_deadlocked in jmxexporter-httpserver.

FYI, I added some information about jvm_* metrics on this stackoverflow answer.

[dhoard]

Added documentation to 1.0.1 README.md