Default jvm.options incompatible with bundled JDK in elasticsearch 7

[6uhrmittag]

• Module version: mod 'elastic-elasticsearch', '6.3.3'
• Puppet version: PuppetBolt 1.10.0
• OS and version: Ubuntu 16.04.6 LTS

Bug description

The Module defaults in jvm.options are not compatible with the bundled Java in elasticsearch 7. Elasticsearch won't start with the example Puppet manifest.

Reproduce

With:

elasticsearch::instance { 'es-01':
  config      => {
    #  'xpack.monitoring.collection.enabled' => true,
    'network.host' => '127.0.0.1',
    'http.port'    => '8200',
  }
}

Result: Elasticsearch won't start.

Syslog:

Apr 22 18:18:49 ip-172-31-35-88 elasticsearch[18037]: Unrecognized VM option 'PrintGCApplicationStoppedTime'
Apr 22 18:18:49 ip-172-31-35-88 elasticsearch[18037]: Error: Could not create the Java Virtual Machine.
Apr 22 18:18:49 ip-172-31-35-88 elasticsearch[18037]: Error: A fatal exception has occurred. Program will exit.
Apr 22 18:18:49 ip-172-31-35-88 systemd[1]: elasticsearch-es-01.service: Main process exited, code=exited, status=1/FAI
LURE
Apr 22 18:18:49 ip-172-31-35-88 systemd[1]: elasticsearch-es-01.service: Unit entered failed state.
Apr 22 18:18:49 ip-172-31-35-88 systemd[1]: elasticsearch-es-01.service: Failed with result 'exit-code'.

The generated jvm.options:

root@xxxx:/etc/elasticsearch/es-01# cat jvm.options
# This file is managed by Puppet -- es-01
#
# Set the 'jvm_options' parameter on the elasticsearch class to change this file.

-Dfile.encoding=UTF-8
-Dio.netty.noKeySetOptimization=true
-Dio.netty.noUnsafe=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Djava.awt.headless=true
-Djna.nosys=true
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-XX:+AlwaysPreTouch
-XX:+HeapDumpOnOutOfMemoryError
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintTenuringDistribution
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseConcMarkSweepGC
-XX:+UseGCLogFileRotation
-XX:-OmitStackTraceInFastThrow
-XX:CMSInitiatingOccupancyFraction=75
-XX:GCLogFileSize=64m
-XX:NumberOfGCLogFiles=32
-Xloggc:/var/log/elasticsearch/es-01/gc.log
-Xms2g
-Xmx2g
-Xss1m
-server

expected behavior

The latest Elasticsearch should start with Module default values.

workaround

My workaround is to disable certain options. I disabled options until elasticsearch started - so only a quick and dirty fix:

elasticsearch::instance { 'es-01':
  jvm_options => [
    '-Xms2g',
    '-Xmx2g',
    '#PrintGCDetails',
    '#PrintGCDateStamps',
    '#PrintTenuringDistribution',
    '#PrintGCApplicationStoppedTime',
    "#Xloggc",
    '#UseGCLogFileRotation',
    "#NumberOfGCLogFiles",
    "#GCLogFileSize",
    "#XX:UseConcMarkSweepGC",
  ],
  config      => {
    #  'xpack.monitoring.collection.enabled' => true,
    'network.host' => '127.0.0.1',
    'http.port'    => '8200',
  }
}

Suggestion

It looks like the bundled Java is a Oracle Java 12:

root@xxx:/usr/share/elasticsearch/jdk# cat release
IMPLEMENTOR="Oracle Corporation"
JAVA_VERSION="12"
JAVA_VERSION_DATE="2019-03-19"

I don't know the specifics, but the elasticsearch documentation suggest to add versions to the jvm.options: https://www.elastic.co/guide/en/elasticsearch/reference/current/jvm-options.html this may prevents this with future versions.

8-9:-Xmx2g

[tylerjl]

This is (yet) another instance of support for multiple instances causing a variety of problems. Another reason to implement #1025, IMO

[lnowicki10]

@tylerjl are you sure this is a problem with instances ? this looks like a problem with the jvm.options.erb being outdated compared to upstream

we have for example :
'PrintGCDetails' => '-XX:+PrintGCDetails',
'PrintGCDateStamps' => '-XX:+PrintGCDateStamps',
'PrintTenuringDistribution' => '-XX:+PrintTenuringDistribution',
'PrintGCApplicationStoppedTime' => '-XX:+PrintGCApplicationStoppedTime',`

when the upstream is using prefixes like

11:-XX:+UseG1GC
11:-XX:InitiatingHeapOccupancyPercent=75
8:-XX:+PrintGCApplicationStoppedTime
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+UseCMSInitiatingOccupancyOnly
8:-XX:+UseConcMarkSweepGC
8:-XX:+UseGCLogFileRotation
8:-XX:GCLogFileSize=64m
8:-XX:NumberOfGCLogFiles=32

we are using this to fix this issue (and also use g1gc on es 6.7)

::elasticsearch::jvm_options: ['-Xms16000m','-Xmx16000m','8:-XX:NumberOfGCLogFiles=32','8:-XX:GCLogFileSize=64m','8:-XX:+UseGCLogFileRotation','8:-XX:+PrintTenuringDistribution','8:-XX:+PrintGCDateStamps','8:-XX:+PrintGCApplicationStoppedTime','8:-XX:+UseConcMarkSweepGC','8:-XX:+UseCMSInitiatingOccupancyOnly','11:-XX:+UseG1GC','11:-XX:InitiatingHeapOccupancyPercent=75']

[tylerjl]

@6uhrmittag you're correct that the jvm.options template being out-of-date is the underlying cause, but the reason that this module defines all of those options in the first place is to support multiple instances. If we can just rely on the upstream jvm.options file, problems like this should eventually fade away ☁️

[albgus]

Isn't the reason the module manages the jvm.options template file to make it possible to set Java options at all, for instance changing the Heap size?

I don't really see how the multiple instances implementation can be the root cause at all in this case, unless not running instances would mean that there is no need to ever modify the jvm.options file.

[tylerjl]

Because there's not a good way to reference the bundled jvm.options file that ships with the Elasticsearch system package and control individual lines at the same time, all default options have to be defined in the module. This means that the module's version of jvm.options and the file (and format) that ships with an ES package can differ and cause bugs (such as this one). When we drop multi-instance support and point to an existing jvm.options file, individual lines can be added/removed from the file that comes with Elasticsearch and avoid these types of format problems.

[smasa90]

Is this relevant ?
#1040

[joeryan]

This is also an issue when using openjdk11 in Debian 10. I've modified the default to use the different jvm options for 8: and 10: so it can start on Debian using openjdk11. I'd think these two concerns would be separate and the basic default jvm.options would include the items that are specific to Java Versions on any ES version and then add the ES 7 specific items separately if running ES 7. I'll add a pull request for this as soon as I have the opportunity.

[faxm0dem]

this could be easily fixed by adding an option to use the system's java.
unless es7 only supports running using the bundled java (which would be :-(*))

[faxm0dem]

FTR we worked around this by pointing JAVA_HOME to 1.8

[eLvErDe]

@tylerjl are you sure this is a problem with instances ? this looks like a problem with the jvm.options.erb being outdated compared to upstream

we have for example :
'PrintGCDetails' => '-XX:+PrintGCDetails',
'PrintGCDateStamps' => '-XX:+PrintGCDateStamps',
'PrintTenuringDistribution' => '-XX:+PrintTenuringDistribution',
'PrintGCApplicationStoppedTime' => '-XX:+PrintGCApplicationStoppedTime',`

when the upstream is using prefixes like

11:-XX:+UseG1GC
11:-XX:InitiatingHeapOccupancyPercent=75
8:-XX:+PrintGCApplicationStoppedTime
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+UseCMSInitiatingOccupancyOnly
8:-XX:+UseConcMarkSweepGC
8:-XX:+UseGCLogFileRotation
8:-XX:GCLogFileSize=64m
8:-XX:NumberOfGCLogFiles=32

we are using this to fix this issue (and also use g1gc on es 6.7)

::elasticsearch::jvm_options: ['-Xms16000m','-Xmx16000m','8:-XX:NumberOfGCLogFiles=32','8:-XX:GCLogFileSize=64m','8:-XX:+UseGCLogFileRotation','8:-XX:+PrintTenuringDistribution','8:-XX:+PrintGCDateStamps','8:-XX:+PrintGCApplicationStoppedTime','8:-XX:+UseConcMarkSweepGC','8:-XX:+UseCMSInitiatingOccupancyOnly','11:-XX:+UseG1GC','11:-XX:InitiatingHeapOccupancyPercent=75']

I'm not sure why this work but I'd like credits to this reply.
Overloading jvm_options array with default values prefixed by JVM version made it worked (values without version are somehow stripped from the resulting jvm.options files)