new ES opendistro install on Ubuntu-18: log is filled up with tons of errors java.nio.file.AccessDeniedException: /dev/shm/performanceanalyzer/XXX

[akomlik]

It looks like Performance analyzer does not create its metrics-location directory

The log is filled with errors like

[2020-08-26T01:23:50,259][ERROR][c.a.o.e.p.r.EventLogFileHandler] [ip-10-120-0-200] Error writing entry 'NOT_INITIALIZED'. Cause: /dev/shm/performanceanalyzer/1598405025000.tmp
[2020-08-26T01:23:50,259][WARN ][stderr                   ] [ip-10-120-0-200] java.nio.file.AccessDeniedException: /dev/shm/performanceanalyzer/1598405030000.tmp

If I try to manually create /dev/shm/performanceanalyze it seems to work for some time but then comes up with errors again

I'm not sure how to permanently fix this.

Environment:

root@ip-10-120-0-200:~# uname -a
Linux ip-10-120-0-200 5.3.0-1033-aws #35-Ubuntu SMP Wed Aug 5 15:47:17 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@ip-10-120-0-200:~# dpkg -l | grep elastic
ii  elasticsearch-oss                 7.8.0                               amd64        Distributed RESTful search engine built for the cloud
ii  opendistro-alerting               1.9.0.0-1                           amd64        Open Distro for Elasticsearch Alerting.      Reference documentation can be found at https://opendistro.github.io/elasticsearch/docs.
ii  opendistro-anomaly-detection      1.9.0.0-1                           amd64        Anomaly Detection plugin for OpenDistro for Elasticsearch.      Reference documentation can be found at https://opendistro.github.io/for-elasticsearch-docs/.
ii  opendistro-index-management       1.9.0.0-1                           amd64        Open Distro for Elasticsearch Index Management.      Reference documentation can be found at https://opendistro.github.io/elasticsearch/docs.
ii  opendistro-job-scheduler          1.9.0.0-1                           amd64        JobScheduler plugin for OpenDistro for Elasticsearch.      Reference documentation can be found at https://opendistro.github.io/elasticsearch/docs.
ii  opendistro-knn                    1.9.0.0-1                           amd64        KNN plugin for OpenDistro for Elasticsearch.      Reference documentation can be found at https://opendistro.github.io/for-elasticsearch-docs/.
ii  opendistro-knnlib                 1.9.0.0                             amd64        KNN JNI library built off of NMSLIB for OpenDistro for Elasticsearch. Reference documentation can be found at https://opendistro.github.io/for-elasticsearch-docs/.
ii  opendistro-performance-analyzer   1.9.0.1-1                           amd64        Performance Analyzer plugin for OpenDistro for Elasticsearch. Reference documentation can be found at https://opendistro.github.io/elasticsearch/docs.
ii  opendistro-security               1.9.0.0-0                           amd64        Security plugin for OpenDistro for Elasticsearch.  Reference documentation can be found at https://opendistro.github.io/elasticsearch/docs.
ii  opendistro-sql                    1.9.0.0-1                           amd64        SQL plugin for OpenDistro for Elasticsearch.      Reference documentation can be found at https://opendistro.github.io/for-elasticsearch-docs/.
ii  opendistroforelasticsearch        1.9.0-1                             amd64        OpenDistro for Elasticsearch.  Reference documentation can be found at https://opendistro.github.io/for-elasticsearch-docs/.
ii  opendistroforelasticsearch-kibana 1.9.0                               amd64        Explore and visualize your Elasticsearch data

[sidheart]

Hi @akomlik,

What are the directory permissions of /dev/shm and /dev/shm/performanceanalyzer and on your machine?

A quick fix for this issue is to modify the metrics-location property in your $ES_HOME/plugins/opendistro_performance_analyzer/pa_config/performance-analyzer.properties file to point to a directory that the elasticsearch user on your machine has access to.

[akomlik]

Permissions are correct 1777
And elasticsearch user has write access to /dev/shm
My guess is PA assumes that directory specified in "metrics-location" already exists and there is no mechanism to guarantee that.
I even added creation of /dev/shm/performanceanalyzer as part of machine boot-up procedure and still am seeing the issue.
It's as if something removes it prediodically.

I suppose I could point metrics-location directly to /dev/shm which is always there

root@ip-10-120-13-41:~# tail -1  /var/log/elasticsearch/elasticsearch.log
[2020-08-26T14:01:17,969][ERROR][c.a.o.e.p.r.EventLogFileHandler] [ip-10-120-13-41] Error writing entry 'NOT_INITIALIZED'. Cause: /dev/shm/performanceanalyzer/1598450475000.tmp

root@ip-10-120-13-41:~# ls -laF /dev/shm
total 0
drwxrwxrwt  2 root root   40 Aug 26 02:48 ./
drwxr-xr-x 14 root root 3220 Aug 26 02:47 ../

root@ip-10-120-13-41:~# su elasticsearch -c "mkdir /dev/shm/performanceanalyzer"

root@ip-10-120-13-41:~# ls -laF /dev/shm
total 0
drwxrwxrwt  3 root          root            60 Aug 26 14:01 ./
drwxr-xr-x 14 root          root          3220 Aug 26 02:47 ../
drwxrwxr-x  2 elasticsearch elasticsearch   80 Aug 26 14:01 performanceanalyzer/
root@ip-10-120-13-41:~# ls -laF /dev/shm/performanceanalyzer/
total 84
drwxrwxr-x 2 elasticsearch elasticsearch   120 Aug 26 14:01 ./
drwxrwxrwt 3 root          root             60 Aug 26 14:01 ../
-rw-r--r-- 1 elasticsearch elasticsearch     0 Aug 26 14:01 1598450500000
-rw-r--r-- 1 elasticsearch elasticsearch 37811 Aug 26 14:01 1598450505000
-rw-r--r-- 1 elasticsearch elasticsearch 39194 Aug 26 14:01 1598450510000.tmp
-rw-r--r-- 1 elasticsearch elasticsearch  1106 Aug 26 14:01 1598450515000.tmp