Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] After upgrade to 4.9.0-1 (debian12) stuck with "Response Error" in log and "Wazuh dashboard server is not ready yet" #292

Open
DaLynxx opened this issue Sep 5, 2024 · 28 comments
Labels
level/task Task issue reporter/community Issue reported by the community request/operational Operational request type/troubleshooting

Comments

@DaLynxx
Copy link

DaLynxx commented Sep 5, 2024

Describe the bug

A clear and concise description of what the bug is.

To Reproduce
I upgraded from 4.8 (4.8.2 I believe), following https://documentation.wazuh.com/current/upgrade-guide/upgrading-central-components.html

Steps to reproduce the behavior:

  1. Upgrade to 4.9.0.
  2. Try to reach the dashboard login screen via browser
  3. Web page shows "Wazuh dashboard server is not ready yet"

Expected behavior

  1. Dashboard login screen when browsing to the server.

OpenSearch Version

wazuh-indexer/now 4.9.0-1 amd64 [installed,local]
wazuh-manager/now 4.9.0-1 amd64 [installed,local]

Dashboards Version

wazuh-dashboard/now 4.9.0-1 amd64 [installed,local]

Plugins

Please list all plugins currently enabled.

Not sure how I find out. I think I run more or less "out of the box" installation.

Screenshots

If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

  • OS: debian 12, Linux 6.8.12-1-pve (LXC container on proxmox)
  • Browser and version: Firefox 130.0

Additional context

I have updated before. Started with a 4.7.x version, got through a couple of 4.8.x steps. All worked well previously following the instructions.

wazuh-manager and wazuh-indexer and wazuh-dashboard all show "active (running)" when asking systemctl status .

However, wazuh-dashboard fills the "log" (journalctl) with

Sep 05 21:35:47 wazuh opensearch-dashboards[2809]: {"type":"log","@timestamp":"2024-09-05T19:35:47Z","tags":["error","opensearch","data"],"pid":2809,"message":"[ResponseError]: Response Error"}
Sep 05 21:35:49 wazuh opensearch-dashboards[2809]: {"type":"log","@timestamp":"2024-09-05T19:35:49Z","tags":["error","opensearch","data"],"pid":2809,"message":"[ResponseError]: Response Error"}
Sep 05 21:35:52 wazuh opensearch-dashboards[2809]: {"type":"log","@timestamp":"2024-09-05T19:35:52Z","tags":["error","opensearch","data"],"pid":2809,"message":"[ResponseError]: Response Error"}
Sep 05 21:35:54 wazuh opensearch-dashboards[2809]: {"type":"log","@timestamp":"2024-09-05T19:35:54Z","tags":["error","opensearch","data"],"pid":2809,"message":"[ResponseError]: Response Error"}
Sep 05 21:35:57 wazuh opensearch-dashboards[2809]: {"type":"log","@timestamp":"2024-09-05T19:35:57Z","tags":["error","opensearch","data"],"pid":2809,"message":"[ResponseError]: Response Error"}

Restarting wazuh-dashboard generates the following log-flow

Sep 05 21:37:14 wazuh systemd[1]: Started wazuh-dashboard.service - wazuh-dashboard.
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","plugins-service"],"pid":3109,"message":"Plugin \"dataSourceManagement\" has been disabled since the following >
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","plugins-service"],"pid":3109,"message":"Plugin \"applicationConfig\" is disabled."}
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","plugins-service"],"pid":3109,"message":"Plugin \"cspHandler\" is disabled."}
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","plugins-service"],"pid":3109,"message":"Plugin \"dataSource\" is disabled."}
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","plugins-service"],"pid":3109,"message":"Plugin \"visTypeXy\" is disabled."}
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","plugins-system"],"pid":3109,"message":"Setting up [48] plugins: [usageCollection,opensearchDashboardsUsageColl>
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","savedobjects-service"],"pid":3109,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch>
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["error","opensearch","data"],"pid":3109,"message":"[ResponseError]: Response Error"}
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["error","savedobjects-service"],"pid":3109,"message":"Unable to retrieve version information from OpenSearch nodes."}
Sep 05 21:37:19 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:19Z","tags":["error","opensearch","data"],"pid":3109,"message":"[ResponseError]: Response Error"}
Sep 05 21:37:21 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:21Z","tags":["error","opensearch","data"],"pid":3109,"message":"[ResponseError]: Response Error"}
Sep 05 21:37:24 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:24Z","tags":["error","opensearch","data"],"pid":3109,"message":"[ResponseError]: Response Error"}

Curl to the dashboard does not respond either.

Content of opensearch_dashboards.yml

server.host: 0.0.0.0
server.port: 443
opensearch.hosts: https://127.0.0.1:9200
opensearch.ssl.verificationMode: certificate
opensearch.requestHeadersAllowlist: ["securitytenant","Authorization"]
opensearch_security.multitenancy.enabled: false
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
server.ssl.enabled: true
server.ssl.key: "/etc/wazuh-dashboard/certs/wazuh-dashboard-key.pem"
server.ssl.certificate: "/etc/wazuh-dashboard/certs/wazuh-dashboard.pem"
opensearch.ssl.certificateAuthorities: ["/etc/wazuh-dashboard/certs/root-ca.pem"]
uiSettings.overrides.defaultRoute: /app/wz-home
opensearch_security.cookie.secure: true

netstat

root@wazuh:/# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:1515            0.0.0.0:*               LISTEN      603/wazuh-authd
tcp        0      0 0.0.0.0:1514            0.0.0.0:*               LISTEN      814/wazuh-remoted
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      3491/node
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      418/master
tcp        0      0 0.0.0.0:55000           0.0.0.0:*               LISTEN      555/python3
tcp6       0      0 :::22                   :::*                    LISTEN      1/init
tcp6       0      0 ::1:25                  :::*                    LISTEN      418/master
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      164/java
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      164/java
udp        0      0 0.0.0.0:68              0.0.0.0:*                           89/dhclient

@DaLynxx
Copy link
Author

DaLynxx commented Sep 5, 2024

Hm, just added the netstat output.
Weird. Why is tcp6 listed for the 127.0.0.1:9200 and 9300

@DaLynxx
Copy link
Author

DaLynxx commented Sep 5, 2024

9200 and 9300 is tied to the indexer rather than the dashboard, or?

@kslintubs
Copy link

Did the update too on Ubuntu 24.04.1 LTS , same error message

@kullarkert
Copy link

kullarkert commented Sep 6, 2024

I'm also having exact same issue and same messages.

I have Ubuntu 22.04.4 LTS and did apt-get upgrade

Some log files

cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"

{"date":"2024-09-05T12:00:19.865Z","level":"error","location":"monitoring:fetchAllAgentsFromApiHost","message":"ApiID: default, Error request with offset/limit 0/500: Request failed with status code 400"}
{"date":"2024-09-05T12:00:19.881Z","level":"error","location":"monitoring:fetchAllAgentsFromApiHost","message":"ApiID: default, Error request with offset/limit 0/500: Request failed with status code 400"}
{"date":"2024-09-05T12:00:19.895Z","level":"error","location":"monitoring:fetchAllAgentsFromApiHost","message":"ApiID: default, Error request with offset/limit 0/500: Request failed with status code 400"}
{"date":"2024-09-05T12:00:19.910Z","level":"error","location":"monitoring:fetchAllAgentsFromApiHost","message":"ApiID: default, Error request with offset/limit 0/500: Request failed with status code 400"}

cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"

[2024-09-06T07:19:59,574][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:19:59,576][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37318}], closing connection
[2024-09-06T07:19:59,581][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:19:59,583][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37326}], closing connection
[2024-09-06T07:19:59,588][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:19:59,590][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37340}], closing connection
[2024-09-06T07:20:02,069][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:20:02,072][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37352}], closing connection
[2024-09-06T07:20:02,077][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:20:02,078][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37362}], closing connection
[2024-09-06T07:20:02,085][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:20:02,087][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37372}], closing connection
[2024-09-06T07:20:02,092][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:20:02,094][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37380}], closing connection
[2024-09-06T07:20:02,499][WARN ][o.o.p.c.u.JsonConverter  ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])

@BooopLJ
Copy link

BooopLJ commented Sep 6, 2024

I also have the same problem after updating to version 4.9.0 on Ubuntu 22.04.4 LTS

@kullarkert
Copy link

This command did the trick. Got answer from slack channel.

/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver -p '<Secr3tP4ssw*rd>'

In all services in one server deployment, changing the password should solve the issue.

@tkolaski
Copy link

tkolaski commented Sep 6, 2024

in wazuh-cluster.log I had entries:

Authentication finally failed for kibanaserver from 127.0.0.1:51086

I found:

https://groups.google.com/g/wazuh/c/rdCF0MBR6oU

It helped

@BooopLJ
Copy link

BooopLJ commented Sep 6, 2024

@kullarkert thank you for suggesting a solution, but I still have this error. The server was reset after the password was changed.
opensearch-dashboards[5609]: {"type":"log","@timestamp":"2024-09-06T08:03:00Z","tags":["error","opensearch","data"],"pid":5609,"message":"[ResponseError]: Response Error"}

@PeterKnotek
Copy link

PeterKnotek commented Sep 6, 2024

Hi

have problem after update

service wazuh-dashboard status
● wazuh-dashboard.service - wazuh-dashboard
     Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2024-09-06 17:07:15 CEST; 13min ago
   Main PID: 14160 (node)
      Tasks: 11 (limit: 9371)
     Memory: 166.8M
        CPU: 18.727s
     CGroup: /system.slice/wazuh-dashboard.service
             └─14160 /usr/share/wazuh-dashboard/node/bin/node /usr/share/wazuh-dashboard/src/cli/dist -c /etc/wazuh-dashboard/opensearch_dashboards.yml

Sep 06 17:20:04 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:04Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:06 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:06Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:09 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:09Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:11 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:11Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:14 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:14Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:16 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:16Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:19 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:19Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:21 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:21Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:24 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:24Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:26 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:26Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}

Can help me

Thanks

@asteriscos
Copy link
Member

As part of Wazuh 4.9.0, we published Wazuh dashboard revision 1 packages:
wazuh-dashboard-4.9.0-1.deb
wazuh-dashboard-4.9.0-1.rpm

These packages introduced a change in the keystore location to /usr/share/wazuh-dashboard/config, which caused issues when upgrading from previous versions, leading to the "Dashboard is not ready" error. Additionally, the kibanaserver user displayed failed authentication messages in the indexer logs.

To resolve this, we have rolled out Wazuh dashboard revision 2 of these packages:
wazuh-dashboard-4.9.0-2.deb
wazuh-dashboard-4.9.0-2.rpm

Wazuh 4.x repositories are available again.

Important: If you upgraded using wazuh-dashboard-4.9.0-1 and afterward changed the passwords, follow these steps:

  1. Install the new Wazuh dashboard revision 2 packages.
  2. Backup your current keystore:
    cp /etc/wazuh-dashboard/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore.bak
  3. Backup the keystore from the new location:
    cp /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore.bak
  4. Move the keystore to the correct location:
    mv /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore
  5. Restart the Wazuh dashboard.

@asteriscos asteriscos added reporter/community Issue reported by the community and removed untriaged labels Sep 10, 2024
@borkedporcupine
Copy link

As part of Wazuh 4.9.0, we published Wazuh dashboard revision 1 packages: wazuh-dashboard-4.9.0-1.deb wazuh-dashboard-4.9.0-1.rpm

These packages introduced a change in the keystore location to /usr/share/wazuh-dashboard/config, which caused issues when upgrading from previous versions, leading to the "Dashboard is not ready" error. Additionally, the kibanaserver user displayed failed authentication messages in the indexer logs.

To resolve this, we have rolled out Wazuh dashboard revision 2 of these packages: wazuh-dashboard-4.9.0-2.deb wazuh-dashboard-4.9.0-2.rpm

Wazuh 4.x repositories are available again.

Important: If you upgraded using wazuh-dashboard-4.9.0-1 and afterward changed the passwords, follow these steps:

1. Install the new Wazuh dashboard revision 2 packages.

2. Backup your current keystore:
   `cp /etc/wazuh-dashboard/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore.bak`

3. Backup the keystore from the new location:
   `cp /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore.bak`

4. Move the keystore to the correct location:
   `mv /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore`

5. Restart the Wazuh dashboard.

I can confirm that I was able to upgrade to 4.9.0 now without issue. Thanks for the fix.

@stefardi
Copy link

As part of Wazuh 4.9.0, we published Wazuh dashboard revision 1 packages: wazuh-dashboard-4.9.0-1.deb wazuh-dashboard-4.9.0-1.rpm
These packages introduced a change in the keystore location to /usr/share/wazuh-dashboard/config, which caused issues when upgrading from previous versions, leading to the "Dashboard is not ready" error. Additionally, the kibanaserver user displayed failed authentication messages in the indexer logs.
To resolve this, we have rolled out Wazuh dashboard revision 2 of these packages: wazuh-dashboard-4.9.0-2.deb wazuh-dashboard-4.9.0-2.rpm
Wazuh 4.x repositories are available again.
Important: If you upgraded using wazuh-dashboard-4.9.0-1 and afterward changed the passwords, follow these steps:

1. Install the new Wazuh dashboard revision 2 packages.

2. Backup your current keystore:
   `cp /etc/wazuh-dashboard/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore.bak`

3. Backup the keystore from the new location:
   `cp /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore.bak`

4. Move the keystore to the correct location:
   `mv /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore`

5. Restart the Wazuh dashboard.

I can confirm that I was able to upgrade to 4.9.0 now without issue. Thanks for the fix.

this is working for me. Thank you.

@genseirin
Copy link

No luck here. The error appeared after upgrading wazuh-dashboard from 4.8.2-1 to 4.9.0-2 and wazuh-indexer and wazuh-manager from 4.8.2-1 to 4.9.0-1. I stopped and started the three services in the recommended order and tried wazuh-passwords-tool.sh (#292 (comment)), which, if I understand it correctly, should also sync the password at all other locations (being an all-in-one installation).

@asteriscos
Copy link
Member

@genseirin can you please provide the output of these commands:

Wazuh indexer

journalctl -u wazuh-indexer | grep -iE "err|warn"
curl -k -u '<USER>:<PASSWORD>' https://127.0.0.1:9200/_cluster/health?pretty

lsof -i -P -n | grep LISTEN | grep wazuh-indexer

Wazuh dashboard

journalctl -u wazuh-dashboard | grep -iE "err|warn"

ls -la /usr/share/wazuh-dashboard/config/
ls -la /etc/wazuh-dashboard/

cat /etc/default/wazuh-dashboard

@Dara-cy
Copy link

Dara-cy commented Sep 13, 2024

Hi @asteriscos
I have a same issue can you help me out

journalctl -u wazuh-indexer | grep -iE "err|warn"

Sep 13 10:23:38 WAZUH systemd-entrypoint[1201]: WARNING: A terminally deprecated method in java.lang.System has been called
Sep 13 10:23:38 WAZUH systemd-entrypoint[1201]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Sep 13 10:23:38 WAZUH systemd-entrypoint[1201]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Sep 13 10:23:38 WAZUH systemd-entrypoint[1201]: WARNING: System::setSecurityManager will be removed in a future release
Sep 13 10:23:39 WAZUH systemd-entrypoint[1201]: WARNING: COMPAT locale provider will be removed in a future release
Sep 13 10:23:40 WAZUH systemd-entrypoint[1201]: WARNING: A terminally deprecated method in java.lang.System has been called
Sep 13 10:23:40 WAZUH systemd-entrypoint[1201]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Sep 13 10:23:40 WAZUH systemd-entrypoint[1201]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Sep 13 10:23:40 WAZUH systemd-entrypoint[1201]: WARNING: System::setSecurityManager will be removed in a future release
Sep 13 11:16:30 WAZUH systemd-entrypoint[11551]: WARNING: A terminally deprecated method in java.lang.System has been called
Sep 13 11:16:30 WAZUH systemd-entrypoint[11551]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Sep 13 11:16:30 WAZUH systemd-entrypoint[11551]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Sep 13 11:16:30 WAZUH systemd-entrypoint[11551]: WARNING: System::setSecurityManager will be removed in a future release
Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: COMPAT locale provider will be removed in a future release
Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: A terminally deprecated method in java.lang.System has been called
Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: System::setSecurityManager will be removed in a future release

curl -k -u ':' https://127.0.0.1:9200/_cluster/health?pretty

{
"cluster_name" : "wazuh-cluster",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"discovered_master" : true,
"discovered_cluster_manager" : true,
"active_primary_shards" : 500,
"active_shards" : 500,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 35,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 93.45794392523365

journalctl -u wazuh-dashboard | grep -iE "err|warn"

Sep 13 11:58:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T04:58:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}
Sep 13 12:00:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:00:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}
Sep 13 12:02:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:02:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}
Sep 13 12:04:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:04:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}
Sep 13 12:06:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:06:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}
Sep 13 12:08:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:08:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}
Sep 13 12:10:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:10:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}
Sep 13 12:12:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:12:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}
Sep 13 12:14:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:14:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}
Sep 13 12:16:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:16:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}
Sep 13 12:18:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:18:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}
Sep 13 12:20:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:20:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}
Sep 13 12:22:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:22:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}
Sep 13 12:24:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:24:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}
Sep 13 12:26:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:26:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}
Sep 13 12:28:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:28:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}
Sep 13 12:30:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:30:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}
Sep 13 12:32:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:32:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}
Sep 13 12:34:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:34:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}

[root@DCPAWAZUH ossec]# ls -la /usr/share/wazuh-dashboard/config/
total 12
drwxr-x---. 2 wazuh-dashboard wazuh-dashboard 95 Sep 13 11:52 .
drwxr-x---. 9 wazuh-dashboard wazuh-dashboard 191 Sep 12 13:35 ..
-rw-r-----. 1 wazuh-dashboard wazuh-dashboard 312 Sep 7 03:25 node.options
-rw-r-----. 1 root root 634 Sep 13 11:52 opensearch_dashboards.yml
-rw-r-----. 1 root root 634 Sep 13 10:36 opensearch_dashboards.yml.bk

ls -la /etc/wazuh-dashboard/

drwxr-x---. 3 wazuh-dashboard wazuh-dashboard 4096 Sep 13 10:37 .
drwxr-xr-x. 115 root root 8192 Sep 13 10:23 ..
dr-x------. 2 wazuh-dashboard wazuh-dashboard 111 Apr 25 16:49 certs
-rw-r-----. 1 wazuh-dashboard wazuh-dashboard 312 Sep 7 03:28 node.options
-rw-r-----. 1 wazuh-dashboard wazuh-dashboard 254 Sep 13 11:37 opensearch_dashboards.keystore
-rw-r-----. 1 root root 254 Sep 12 17:18 opensearch_dashboards.keystore.bak
-rw-r-----. 1 wazuh-dashboard wazuh-dashboard 635 Sep 13 11:49 opensearch_dashboards.yml
-rw-r-----. 1 root root 635 May 28 16:36 opensearch_dashboards.yml.bk2
-rw-r-----. 1 wazuh-dashboard wazuh-dashboard 634 Sep 7 03:28 opensearch_dashboards.yml.rpmnew

[root@DCPAWAZUH ossec]# cat /etc/default/wazuh-dashboard
user="wazuh-dashboard"
group="wazuh-dashboard"
chroot="/"
chdir="/"
nice=""
KILL_ON_STOP_TIMEOUT=0

OSD_PATH_CONF="/etc/wazuh-dashboard"

@genseirin
Copy link

@asteriscos Thank you!

Wazuh indexer

journalctl -u wazuh-indexer | grep -iE "err|warn"

...
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster_server.json" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:421)
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2621)
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster.log" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:421)
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2621)
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: A terminally deprecated method in java.lang.System has been called
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager will be removed in a future release
Sep 13 07:29:41 ns3103184 systemd-entrypoint[14041]: WARNING: COMPAT locale provider will be removed in a future release
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: A terminally deprecated method in java.lang.System has been called
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager will be removed in a future release

curl -k -u 'kibana:<corresponding pw>' https://127.0.0.1:9200/_cluster/health?pretty

no output

lsof -i -P -n | grep LISTEN | grep wazuh-indexer

java      14041               wazuh-indexer  615u  IPv6 531705783      0t0  TCP 127.0.0.1:9300 (LISTEN)
java      14041               wazuh-indexer  617u  IPv6 531710409      0t0  TCP 127.0.0.1:9200 (LISTEN)

Wazuh dashboard

journalctl -u wazuh-dashboard | grep -iE "err|warn"

Sep 12 13:48:58 ns3103184 opensearch-dashboards[19513]: {"type":"log","@timestamp":"2024-09-12T13:48:58Z","tags":["error","opensearch","data"],"pid":19513,"message":"[resource_already_exists_exception]: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 12 13:48:58 ns3103184 opensearch-dashboards[19513]: {"type":"log","@timestamp":"2024-09-12T13:48:58Z","tags":["warning","savedobjects-service"],"pid":19513,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 12 13:48:58 ns3103184 opensearch-dashboards[19513]: {"type":"log","@timestamp":"2024-09-12T13:48:58Z","tags":["warning","savedobjects-service"],"pid":19513,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_3 and restarting OpenSearchDashboards."}
Sep 13 00:55:06 ns3103184 opensearch-dashboards[20813]: {"type":"log","@timestamp":"2024-09-13T00:55:06Z","tags":["error","opensearch","data"],"pid":20813,"message":"[resource_already_exists_exception]: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 13 00:55:06 ns3103184 opensearch-dashboards[20813]: {"type":"log","@timestamp":"2024-09-13T00:55:06Z","tags":["warning","savedobjects-service"],"pid":20813,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 13 00:55:06 ns3103184 opensearch-dashboards[20813]: {"type":"log","@timestamp":"2024-09-13T00:55:06Z","tags":["warning","savedobjects-service"],"pid":20813,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_3 and restarting OpenSearchDashboards."}
Sep 13 07:29:22 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:22Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:22 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:22Z","tags":["error","savedobjects-service"],"pid":13304,"message":"Unable to retrieve version information from OpenSearch nodes."}
Sep 13 07:29:25 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:25Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:27 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:27Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:30 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:30Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:32 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:32Z","tags":["warning","savedobjects-service"],"pid":13304,"message":"Unable to connect to OpenSearch. Error: Given the configuration, the ConnectionPool was not able to find a usable Connection for this request."}
Sep 13 07:30:40 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:40Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 13 07:30:40 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:40Z","tags":["warning","savedobjects-service"],"pid":15518,"message":"Unable to connect to OpenSearch. Error: search_phase_execution_exception: "}
Sep 13 07:30:43 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:43Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 13 07:30:45 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:45Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 13 07:30:48 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:48Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
(keeps repeating)

ls -la /usr/share/wazuh-dashboard/config/

drwxr-x---  2 wazuh-dashboard wazuh-dashboard 4096 Sep 11 13:59 .
drwxr-x--- 10 wazuh-dashboard wazuh-dashboard 4096 Sep 11 14:02 ..
-rw-r-----  1 wazuh-dashboard wazuh-dashboard  312 May  5  2023 node.options
-rw-r-----  1 wazuh-dashboard wazuh-dashboard  634 May  5  2023 opensearch_dashboards.yml

ls -la /etc/wazuh-dashboard/

drwxr-x---   3 wazuh-dashboard wazuh-dashboard 4096 Sep 12 07:53 .
drwxr-xr-x 119 root            root            4096 Sep 10 06:32 ..
dr-x------   2 wazuh-dashboard wazuh-dashboard 4096 Mar 18 10:37 certs
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  312 May  5  2023 node.options
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  230 Sep 13 07:29 opensearch_dashboards.keystore
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  230 Sep 11 14:43 opensearch_dashboards.keystore.bak
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  697 Jun 14 06:30 opensearch_dashboards.yml
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  634 May  5  2023 opensearch_dashboards.yml.dpkg-dist

cat /etc/default/wazuh-dashboard

user="wazuh-dashboard"
group="wazuh-dashboard"
chroot="/"
chdir="/"
nice=""
KILL_ON_STOP_TIMEOUT=0

OSD_PATH_CONF="/etc/wazuh-dashboard"

@asteriscos
Copy link
Member

asteriscos commented Sep 13, 2024

@asteriscos Thank you!

Wazuh indexer

journalctl -u wazuh-indexer | grep -iE "err|warn"

...
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster_server.json" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:421)
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2621)
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster.log" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:421)
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2621)
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: A terminally deprecated method in java.lang.System has been called
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager will be removed in a future release
Sep 13 07:29:41 ns3103184 systemd-entrypoint[14041]: WARNING: COMPAT locale provider will be removed in a future release
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: A terminally deprecated method in java.lang.System has been called
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager will be removed in a future release

curl -k -u 'kibana:<corresponding pw>' https://127.0.0.1:9200/_cluster/health?pretty

no output

lsof -i -P -n | grep LISTEN | grep wazuh-indexer

java      14041               wazuh-indexer  615u  IPv6 531705783      0t0  TCP 127.0.0.1:9300 (LISTEN)
java      14041               wazuh-indexer  617u  IPv6 531710409      0t0  TCP 127.0.0.1:9200 (LISTEN)

Wazuh dashboard

journalctl -u wazuh-dashboard | grep -iE "err|warn"

Sep 12 13:48:58 ns3103184 opensearch-dashboards[19513]: {"type":"log","@timestamp":"2024-09-12T13:48:58Z","tags":["error","opensearch","data"],"pid":19513,"message":"[resource_already_exists_exception]: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 12 13:48:58 ns3103184 opensearch-dashboards[19513]: {"type":"log","@timestamp":"2024-09-12T13:48:58Z","tags":["warning","savedobjects-service"],"pid":19513,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 12 13:48:58 ns3103184 opensearch-dashboards[19513]: {"type":"log","@timestamp":"2024-09-12T13:48:58Z","tags":["warning","savedobjects-service"],"pid":19513,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_3 and restarting OpenSearchDashboards."}
Sep 13 00:55:06 ns3103184 opensearch-dashboards[20813]: {"type":"log","@timestamp":"2024-09-13T00:55:06Z","tags":["error","opensearch","data"],"pid":20813,"message":"[resource_already_exists_exception]: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 13 00:55:06 ns3103184 opensearch-dashboards[20813]: {"type":"log","@timestamp":"2024-09-13T00:55:06Z","tags":["warning","savedobjects-service"],"pid":20813,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 13 00:55:06 ns3103184 opensearch-dashboards[20813]: {"type":"log","@timestamp":"2024-09-13T00:55:06Z","tags":["warning","savedobjects-service"],"pid":20813,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_3 and restarting OpenSearchDashboards."}
Sep 13 07:29:22 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:22Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:22 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:22Z","tags":["error","savedobjects-service"],"pid":13304,"message":"Unable to retrieve version information from OpenSearch nodes."}
Sep 13 07:29:25 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:25Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:27 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:27Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:30 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:30Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:32 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:32Z","tags":["warning","savedobjects-service"],"pid":13304,"message":"Unable to connect to OpenSearch. Error: Given the configuration, the ConnectionPool was not able to find a usable Connection for this request."}
Sep 13 07:30:40 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:40Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 13 07:30:40 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:40Z","tags":["warning","savedobjects-service"],"pid":15518,"message":"Unable to connect to OpenSearch. Error: search_phase_execution_exception: "}
Sep 13 07:30:43 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:43Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 13 07:30:45 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:45Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 13 07:30:48 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:48Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
(keeps repeating)

ls -la /usr/share/wazuh-dashboard/config/

drwxr-x---  2 wazuh-dashboard wazuh-dashboard 4096 Sep 11 13:59 .
drwxr-x--- 10 wazuh-dashboard wazuh-dashboard 4096 Sep 11 14:02 ..
-rw-r-----  1 wazuh-dashboard wazuh-dashboard  312 May  5  2023 node.options
-rw-r-----  1 wazuh-dashboard wazuh-dashboard  634 May  5  2023 opensearch_dashboards.yml

ls -la /etc/wazuh-dashboard/

drwxr-x---   3 wazuh-dashboard wazuh-dashboard 4096 Sep 12 07:53 .
drwxr-xr-x 119 root            root            4096 Sep 10 06:32 ..
dr-x------   2 wazuh-dashboard wazuh-dashboard 4096 Mar 18 10:37 certs
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  312 May  5  2023 node.options
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  230 Sep 13 07:29 opensearch_dashboards.keystore
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  230 Sep 11 14:43 opensearch_dashboards.keystore.bak
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  697 Jun 14 06:30 opensearch_dashboards.yml
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  634 May  5  2023 opensearch_dashboards.yml.dpkg-dist

cat /etc/default/wazuh-dashboard

user="wazuh-dashboard"
group="wazuh-dashboard"
chroot="/"
chdir="/"
nice=""
KILL_ON_STOP_TIMEOUT=0

OSD_PATH_CONF="/etc/wazuh-dashboard"

@genseirin I see a couple of problems in the logs:

Credentials

If the curl to https://127.0.0.1:9200/_cluster/health?pretty doesn't have an output, it usually means a credentials problem. Did you try with the admin user?

If you don't remember the credentials you can reset the password using this tool:
Download the tool
curl -so wazuh-passwords-tool.sh https://packages.wazuh.com/4.9/wazuh-passwords-tool.sh

Execute it
bash wazuh-passwords-tool.sh -a

https://documentation.wazuh.com/current/user-manual/user-administration/password-management.html

kibana_3 index

I can see in the logs you have an issue with .kibana_3 index, so I suggest to refresh the index and restart Wazuh dashboard afterward.
curl -k -XDELETE -u <USER>:<PASS> https://<IndexerIP>:9200/.kibana_3
systemctl restart wazuh-dashboard

@asteriscos
Copy link
Member

Hi @asteriscos I have a same issue can you help me out

journalctl -u wazuh-indexer | grep -iE "err|warn"

Sep 13 10:23:38 WAZUH systemd-entrypoint[1201]: WARNING: A terminally deprecated method in java.lang.System has been called Sep 13 10:23:38 WAZUH systemd-entrypoint[1201]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar) Sep 13 10:23:38 WAZUH systemd-entrypoint[1201]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch Sep 13 10:23:38 WAZUH systemd-entrypoint[1201]: WARNING: System::setSecurityManager will be removed in a future release Sep 13 10:23:39 WAZUH systemd-entrypoint[1201]: WARNING: COMPAT locale provider will be removed in a future release Sep 13 10:23:40 WAZUH systemd-entrypoint[1201]: WARNING: A terminally deprecated method in java.lang.System has been called Sep 13 10:23:40 WAZUH systemd-entrypoint[1201]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar) Sep 13 10:23:40 WAZUH systemd-entrypoint[1201]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security Sep 13 10:23:40 WAZUH systemd-entrypoint[1201]: WARNING: System::setSecurityManager will be removed in a future release Sep 13 11:16:30 WAZUH systemd-entrypoint[11551]: WARNING: A terminally deprecated method in java.lang.System has been called Sep 13 11:16:30 WAZUH systemd-entrypoint[11551]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar) Sep 13 11:16:30 WAZUH systemd-entrypoint[11551]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch Sep 13 11:16:30 WAZUH systemd-entrypoint[11551]: WARNING: System::setSecurityManager will be removed in a future release Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: COMPAT locale provider will be removed in a future release Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: A terminally deprecated method in java.lang.System has been called Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar) Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security Sep 13 11:16:31 WAZUH systemd-entrypoint[11551]: WARNING: System::setSecurityManager will be removed in a future release

curl -k -u ':' https://127.0.0.1:9200/_cluster/health?pretty

{ "cluster_name" : "wazuh-cluster", "status" : "yellow", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "discovered_master" : true, "discovered_cluster_manager" : true, "active_primary_shards" : 500, "active_shards" : 500, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 35, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 93.45794392523365

journalctl -u wazuh-dashboard | grep -iE "err|warn"

Sep 13 11:58:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T04:58:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:00:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:00:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:02:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:02:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:04:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:04:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:06:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:06:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:08:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:08:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:10:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:10:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:12:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:12:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:14:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:14:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:16:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:16:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:18:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:18:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:20:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:20:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:22:56 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:22:56Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:24:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:24:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:26:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:26:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:28:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:28:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:30:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:30:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:32:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:32:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"} Sep 13 12:34:57 WAZUH opensearch-dashboards[23467]: {"type":"log","@timestamp":"2024-09-13T05:34:57Z","tags":["error","opensearch","data"],"pid":23467,"message":"[TimeoutError]: Request timed out"}

[root@DCPAWAZUH ossec]# ls -la /usr/share/wazuh-dashboard/config/ total 12 drwxr-x---. 2 wazuh-dashboard wazuh-dashboard 95 Sep 13 11:52 . drwxr-x---. 9 wazuh-dashboard wazuh-dashboard 191 Sep 12 13:35 .. -rw-r-----. 1 wazuh-dashboard wazuh-dashboard 312 Sep 7 03:25 node.options -rw-r-----. 1 root root 634 Sep 13 11:52 opensearch_dashboards.yml -rw-r-----. 1 root root 634 Sep 13 10:36 opensearch_dashboards.yml.bk

ls -la /etc/wazuh-dashboard/

drwxr-x---. 3 wazuh-dashboard wazuh-dashboard 4096 Sep 13 10:37 . drwxr-xr-x. 115 root root 8192 Sep 13 10:23 .. dr-x------. 2 wazuh-dashboard wazuh-dashboard 111 Apr 25 16:49 certs -rw-r-----. 1 wazuh-dashboard wazuh-dashboard 312 Sep 7 03:28 node.options -rw-r-----. 1 wazuh-dashboard wazuh-dashboard 254 Sep 13 11:37 opensearch_dashboards.keystore -rw-r-----. 1 root root 254 Sep 12 17:18 opensearch_dashboards.keystore.bak -rw-r-----. 1 wazuh-dashboard wazuh-dashboard 635 Sep 13 11:49 opensearch_dashboards.yml -rw-r-----. 1 root root 635 May 28 16:36 opensearch_dashboards.yml.bk2 -rw-r-----. 1 wazuh-dashboard wazuh-dashboard 634 Sep 7 03:28 opensearch_dashboards.yml.rpmnew

[root@DCPAWAZUH ossec]# cat /etc/default/wazuh-dashboard user="wazuh-dashboard" group="wazuh-dashboard" chroot="/" chdir="/" nice="" KILL_ON_STOP_TIMEOUT=0

OSD_PATH_CONF="/etc/wazuh-dashboard"

@Dara-cy what type of deployment do you have? is it an all-in-one installation or a distributed one?
I see Wazuh Dashboard trying to reach Wazuh indexer, but the error doesn't seem to be related to credentials. Can you confirm Wazuh dashboard can reach the address of Wazuh Indexer?

@metalcated
Copy link

Was having these issues too, resetting all passwords and updating nodes did the trick for all but one warning message which I am investigating outside of this thread. Thanks!

@uguronduc
Copy link

@asteriscos Thank you!

Wazuh indexer

journalctl -u wazuh-indexer | grep -iE "err|warn"

...
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster_server.json" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:421)
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2621)
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]: ERROR StatusConsoleListener Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster.log" got access denied ("java.lang.RuntimePermission" "accessUserInformation")
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:421)
Sep 13 00:00:02 ns3103184 systemd-entrypoint[18215]:         at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2621)
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: A terminally deprecated method in java.lang.System has been called
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Sep 13 07:29:40 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager will be removed in a future release
Sep 13 07:29:41 ns3103184 systemd-entrypoint[14041]: WARNING: COMPAT locale provider will be removed in a future release
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: A terminally deprecated method in java.lang.System has been called
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Sep 13 07:29:42 ns3103184 systemd-entrypoint[14041]: WARNING: System::setSecurityManager will be removed in a future release

curl -k -u 'kibana:<corresponding pw>' https://127.0.0.1:9200/_cluster/health?pretty
no output
lsof -i -P -n | grep LISTEN | grep wazuh-indexer

java      14041               wazuh-indexer  615u  IPv6 531705783      0t0  TCP 127.0.0.1:9300 (LISTEN)
java      14041               wazuh-indexer  617u  IPv6 531710409      0t0  TCP 127.0.0.1:9200 (LISTEN)

Wazuh dashboard

journalctl -u wazuh-dashboard | grep -iE "err|warn"

Sep 12 13:48:58 ns3103184 opensearch-dashboards[19513]: {"type":"log","@timestamp":"2024-09-12T13:48:58Z","tags":["error","opensearch","data"],"pid":19513,"message":"[resource_already_exists_exception]: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 12 13:48:58 ns3103184 opensearch-dashboards[19513]: {"type":"log","@timestamp":"2024-09-12T13:48:58Z","tags":["warning","savedobjects-service"],"pid":19513,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 12 13:48:58 ns3103184 opensearch-dashboards[19513]: {"type":"log","@timestamp":"2024-09-12T13:48:58Z","tags":["warning","savedobjects-service"],"pid":19513,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_3 and restarting OpenSearchDashboards."}
Sep 13 00:55:06 ns3103184 opensearch-dashboards[20813]: {"type":"log","@timestamp":"2024-09-13T00:55:06Z","tags":["error","opensearch","data"],"pid":20813,"message":"[resource_already_exists_exception]: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 13 00:55:06 ns3103184 opensearch-dashboards[20813]: {"type":"log","@timestamp":"2024-09-13T00:55:06Z","tags":["warning","savedobjects-service"],"pid":20813,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_3/UBgYn7vdSwq0NRg4c7w5mw] already exists"}
Sep 13 00:55:06 ns3103184 opensearch-dashboards[20813]: {"type":"log","@timestamp":"2024-09-13T00:55:06Z","tags":["warning","savedobjects-service"],"pid":20813,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_3 and restarting OpenSearchDashboards."}
Sep 13 07:29:22 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:22Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:22 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:22Z","tags":["error","savedobjects-service"],"pid":13304,"message":"Unable to retrieve version information from OpenSearch nodes."}
Sep 13 07:29:25 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:25Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:27 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:27Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:30 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:30Z","tags":["error","opensearch","data"],"pid":13304,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 13 07:29:32 ns3103184 opensearch-dashboards[13304]: {"type":"log","@timestamp":"2024-09-13T07:29:32Z","tags":["warning","savedobjects-service"],"pid":13304,"message":"Unable to connect to OpenSearch. Error: Given the configuration, the ConnectionPool was not able to find a usable Connection for this request."}
Sep 13 07:30:40 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:40Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 13 07:30:40 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:40Z","tags":["warning","savedobjects-service"],"pid":15518,"message":"Unable to connect to OpenSearch. Error: search_phase_execution_exception: "}
Sep 13 07:30:43 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:43Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 13 07:30:45 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:45Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 13 07:30:48 ns3103184 opensearch-dashboards[15518]: {"type":"log","@timestamp":"2024-09-13T07:30:48Z","tags":["error","opensearch","data"],"pid":15518,"message":"[search_phase_execution_exception]: all shards failed"}
(keeps repeating)

ls -la /usr/share/wazuh-dashboard/config/

drwxr-x---  2 wazuh-dashboard wazuh-dashboard 4096 Sep 11 13:59 .
drwxr-x--- 10 wazuh-dashboard wazuh-dashboard 4096 Sep 11 14:02 ..
-rw-r-----  1 wazuh-dashboard wazuh-dashboard  312 May  5  2023 node.options
-rw-r-----  1 wazuh-dashboard wazuh-dashboard  634 May  5  2023 opensearch_dashboards.yml

ls -la /etc/wazuh-dashboard/

drwxr-x---   3 wazuh-dashboard wazuh-dashboard 4096 Sep 12 07:53 .
drwxr-xr-x 119 root            root            4096 Sep 10 06:32 ..
dr-x------   2 wazuh-dashboard wazuh-dashboard 4096 Mar 18 10:37 certs
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  312 May  5  2023 node.options
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  230 Sep 13 07:29 opensearch_dashboards.keystore
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  230 Sep 11 14:43 opensearch_dashboards.keystore.bak
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  697 Jun 14 06:30 opensearch_dashboards.yml
-rw-r-----   1 wazuh-dashboard wazuh-dashboard  634 May  5  2023 opensearch_dashboards.yml.dpkg-dist

cat /etc/default/wazuh-dashboard

user="wazuh-dashboard"
group="wazuh-dashboard"
chroot="/"
chdir="/"
nice=""
KILL_ON_STOP_TIMEOUT=0

OSD_PATH_CONF="/etc/wazuh-dashboard"

@genseirin I see a couple of problems in the logs:

Credentials

If the curl to https://127.0.0.1:9200/_cluster/health?pretty doesn't have an output, it usually means a credentials problem. Did you try with the admin user?

If you don't remember the credentials you can reset the password using this tool: Download the tool curl -so wazuh-passwords-tool.sh https://packages.wazuh.com/4.9/wazuh-passwords-tool.sh

Execute it bash wazuh-passwords-tool.sh -a

https://documentation.wazuh.com/current/user-manual/user-administration/password-management.html

kibana_3 index

I can see in the logs you have an issue with .kibana_3 index, so I suggest to refresh the index and restart Wazuh dashboard afterward. curl -k -XDELETE -u <USER>:<PASS> https://<IndexerIP>:9200/.kibana_3 systemctl restart wazuh-dashboard

Thanks for solution. I solved with curl -k -XDELETE -u : https://:9200/.kibana_3` systemctl restart wazuh-dashboard

@Dara-cy
Copy link

Dara-cy commented Sep 15, 2024

Hi @asteriscos

Actually it is an all in one installation, but have 2 ip interface that 10.84.54.51 ( for dashboard) and 10.84.55.5 ( for indexer and rsyslog server). It's running well before i upgrade. Now agent, rsyslog working and dashboard is not ready yet.

Dashboard config

server.host: 0.0.0.0
server.port: 443
#opensearch.hosts: https://10.84.54.51:9200
opensearch.hosts: https://127.0.0.1:9200
#opensearch.hosts: https://10.84.55.5:9200

opensearch.ssl.verificationMode: certificate
#opensearch.username:
#opensearch.password:
opensearch.requestHeadersAllowlist: ["securitytenant","Authorization"]
opensearch_security.multitenancy.enabled: false
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
server.ssl.enabled: true
server.ssl.key: "/etc/wazuh-dashboard/certs/dashboard-key.pem"
server.ssl.certificate: "/etc/wazuh-dashboard/certs/dashboard.pem"
opensearch.ssl.certificateAuthorities: ["/etc/wazuh-dashboard/certs/root-ca.pem"]
uiSettings.overrides.defaultRoute: /app/wz-home

Curl check

curl -k -u 'admin:admin' https://127.0.1:9200/_cluster/health?pretty
{
"cluster_name" : "wazuh-cluster",
"status" : "yellow",
"timed_out" : false,V
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"discovered_master" : true,
"discovered_cluster_manager" : true,
"active_primary_shards" : 505,
"active_shards" : 505,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 34,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 93.69202226345084
}

wazuh-dashboard status

wazuh-dashboard.service - wazuh-dashboard
Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; preset: disabled)
Active: active (running) since Sun 2024-09-15 16:17:41 +07; 10min ago
Main PID: 96251 (node)
Tasks: 11 (limit: 407812)
Memory: 176.2M
CPU: 6.996s
CGroup: /system.slice/wazuh-dashboard.service
└─96251 /usr/share/wazuh-dashboard/node/bin/node /usr/share/wazuh-dashboard/src/cli/dist

journalctl -u wazuh-dashboard | grep -iE "err|warn"

(restart dashboard and index service)

Sep 15 16:16:28 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:28Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 15 16:16:30 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:30Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 15 16:16:33 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:33Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 15 16:16:35 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:35Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 15 16:16:38 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:38Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 15 16:16:40 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:40Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 15 16:16:43 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:43Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 15 16:16:45 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:45Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 15 16:16:48 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:48Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 15 16:16:50 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:50Z","tags":["error","opensearch","data"],"pid":91828,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 15 16:16:53 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:53Z","tags":["error","opensearch","data"],"pid":91828,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 15 16:16:53 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:53Z","tags":["warning","savedobjects-service"],"pid":91828,"message":"Unable to connect to OpenSearch. Error: search_phase_execution_exception: "}
Sep 15 16:16:56 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:56Z","tags":["error","opensearch","data"],"pid":91828,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 15 16:16:58 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:16:58Z","tags":["error","opensearch","data"],"pid":91828,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 15 16:17:01 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:17:01Z","tags":["error","opensearch","data"],"pid":91828,"message":"[search_phase_execution_exception]: all shards failed"}
Sep 15 16:17:03 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:17:03Z","tags":["error","opensearch","data"],"pid":91828,"message":"[resource_already_exists_exception]: index [.kibana_3/FNegsr3mQfuHRAEcOOHODA] already exists"}
Sep 15 16:17:03 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:17:03Z","tags":["warning","savedobjects-service"],"pid":91828,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_3/FNegsr3mQfuHRAEcOOHODA] already exists"}
Sep 15 16:17:03 DCPAWAZUH01 opensearch-dashboards[91828]: {"type":"log","@timestamp":"2024-09-15T09:17:03Z","tags":["warning","savedobjects-service"],"pid":91828,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_3 and restarting OpenSearchDashboards."}
Sep 15 16:17:44 DCPAWAZUH01 opensearch-dashboards[96251]: {"type":"log","@timestamp":"2024-09-15T09:17:44Z","tags":["error","opensearch","data"],"pid":96251,"message":"[resource_already_exists_exception]: index [.kibana_3/FNegsr3mQfuHRAEcOOHODA] already exists"}
Sep 15 16:17:44 DCPAWAZUH01 opensearch-dashboards[96251]: {"type":"log","@timestamp":"2024-09-15T09:17:44Z","tags":["warning","savedobjects-service"],"pid":96251,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_3/FNegsr3mQfuHRAEcOOHODA] already exists"}
Sep 15 16:17:44 DCPAWAZUH01 opensearch-dashboards[96251]: {"type":"log","@timestamp":"2024-09-15T09:17:44Z","tags":["warning","savedobjects-service"],"pid":96251,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_3 and restarting OpenSearchDashboards."}

Thank you again help.

@Dara-cy
Copy link

Dara-cy commented Sep 15, 2024

Hi @asteriscos

Run this fix my dashboard

'''
curl -k -XDELETE -u : https://:9200/.kibana_3
'''

But wazuh agent has connection error, I will check version or reinstall agent and notify everyone soon

Many thanks

@genseirin
Copy link

@genseirin I see a couple of problems in the logs:

Credentials

If the curl to https://127.0.0.1:9200/_cluster/health?pretty doesn't have an output, it usually means a credentials problem. Did you try with the admin user?

If you don't remember the credentials you can reset the password using this tool: Download the tool curl -so wazuh-passwords-tool.sh https://packages.wazuh.com/4.9/wazuh-passwords-tool.sh

Execute it bash wazuh-passwords-tool.sh -a

https://documentation.wazuh.com/current/user-manual/user-administration/password-management.html

kibana_3 index

I can see in the logs you have an issue with .kibana_3 index, so I suggest to refresh the index and restart Wazuh dashboard afterward. curl -k -XDELETE -u <USER>:<PASS> https://<IndexerIP>:9200/.kibana_3 systemctl restart wazuh-dashboard

Your solution was very helpful, thanks a lot!

@BooopLJ
Copy link

BooopLJ commented Sep 16, 2024

Hi @asteriscos
thank you for your solution. It worked for me:

"1. Install the new Wazuh dashboard revision 2 packages.
2. Backup your current keystore: cp /etc/wazuh-dashboard/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore.bak
3. Backup the keystore from the new location: cp /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore.bak
4. Move the keystore to the correct location: mv /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore
5. curl -k -XDELETE -u : https://:9200/.kibana_3
6. systemctl restart wazuh-dashboard"

Have a nice day :)

@MAH69IK
Copy link

MAH69IK commented Sep 17, 2024

Hi, I'm facing this problem too, but I'm not able to solve it yet. I updated Wazuh from 4.7.5 to 4.8.2 and then to 4.9.0. Now I am getting the error “Wazuh dashboard server is not ready yet”.

Here is what I tried:

  • I don't have the file /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore, so the advice from @asteriscos didn't work (probably because I put 4.9.0-2 right away, without 4.9.0-1).
  • I tried the command $ sudo /usr/share/wazuh-dashboard/bin/opensearch-dashboards-keystore --allow-root add opensearch.password specifying the password from the file /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml - didn't help.
  • I tried the command $ sudo /usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver - that didn't help either.
  • I restarted both wazuh-indexer and wazuh-dashboard after both commands.

Here is the information about my system:

Wazuh indexer

$ sudo journalctl --since today -u wazuh-indexer | grep -iE "err|warn"
сен 17 14:38:21 wazuh systemd-entrypoint[3180279]: WARNING: A terminally deprecated method in java.lang.System has been called
сен 17 14:38:21 wazuh systemd-entrypoint[3180279]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
сен 17 14:38:21 wazuh systemd-entrypoint[3180279]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
сен 17 14:38:21 wazuh systemd-entrypoint[3180279]: WARNING: System::setSecurityManager will be removed in a future release
сен 17 14:38:22 wazuh systemd-entrypoint[3180279]: WARNING: COMPAT locale provider will be removed in a future release
сен 17 14:38:23 wazuh systemd-entrypoint[3180279]: WARNING: A terminally deprecated method in java.lang.System has been called
сен 17 14:38:23 wazuh systemd-entrypoint[3180279]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
сен 17 14:38:23 wazuh systemd-entrypoint[3180279]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
сен 17 14:38:23 wazuh systemd-entrypoint[3180279]: WARNING: System::setSecurityManager will be removed in a future release
сен 17 15:11:32 wazuh systemd-entrypoint[3181519]: WARNING: A terminally deprecated method in java.lang.System has been called
сен 17 15:11:32 wazuh systemd-entrypoint[3181519]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
сен 17 15:11:32 wazuh systemd-entrypoint[3181519]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
сен 17 15:11:32 wazuh systemd-entrypoint[3181519]: WARNING: System::setSecurityManager will be removed in a future release
сен 17 15:11:33 wazuh systemd-entrypoint[3181519]: WARNING: COMPAT locale provider will be removed in a future release
сен 17 15:11:34 wazuh systemd-entrypoint[3181519]: WARNING: A terminally deprecated method in java.lang.System has been called
сен 17 15:11:34 wazuh systemd-entrypoint[3181519]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
сен 17 15:11:34 wazuh systemd-entrypoint[3181519]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
сен 17 15:11:34 wazuh systemd-entrypoint[3181519]: WARNING: System::setSecurityManager will be removed in a future release
$ curl -k -u admin:... https://localhost:9200/_cat/health?pretty
1726579263 13:21:03 wazuh-cluster green 1 1 true 447 447 0 0 0 0 - 100.0%
$ sudo lsof -i -nP | grep LISTEN | grep wazuh-indexer
java      3181519   wazuh-indexer  576u  IPv6 3837239206      0t0  TCP 127.0.0.1:9300 (LISTEN)
java      3181519   wazuh-indexer  578u  IPv6 3837239246      0t0  TCP 127.0.0.1:9200 (LISTEN)

Wazuh dashboard

$ sudo journalctl --since today -u wazuh-dashboard | grep -iE "err|warn"
сен 17 15:15:15 wazuh opensearch-dashboards[3181832]: {"type":"log","@timestamp":"2024-09-17T12:15:15Z","tags":["error","opensearch","data"],"pid":3181832,"message":"[TimeoutError]: Request timed out"}
сен 17 15:15:15 wazuh opensearch-dashboards[3181832]: {"type":"log","@timestamp":"2024-09-17T12:15:15Z","tags":["error","savedobjects-service"],"pid":3181832,"message":"Unable to retrieve version information from OpenSearch nodes."}
сен 17 15:17:15 wazuh opensearch-dashboards[3181832]: {"type":"log","@timestamp":"2024-09-17T12:17:15Z","tags":["error","opensearch","data"],"pid":3181832,"message":"[TimeoutError]: Request timed out"}

(then the last line is repeated)

$ sudo ls -lhFAv /usr/share/wazuh-dashboard/config/
total 8.0K
-rw-r----- 1 wazuh-dashboard wazuh-dashboard 312 мая  5  2023 node.options
-rw-r----- 1 wazuh-dashboard wazuh-dashboard 634 мая  5  2023 opensearch_dashboards.yml

$ sudo ls -lhFAv /etc/wazuh-dashboard/
total 20K
dr-x------ 2 wazuh-dashboard wazuh-dashboard 4.0K июн 17  2022 certs/
-rw-r----- 1 wazuh-dashboard wazuh-dashboard  312 мая  5  2023 node.options
-rw-r--r-- 1 wazuh-dashboard wazuh-dashboard  254 сен 17 15:04 opensearch_dashboards.keystore
-rw-r----- 1 wazuh-dashboard wazuh-dashboard  634 сен 16 17:16 opensearch_dashboards.yml
-rw-r----- 1 wazuh-dashboard wazuh-dashboard  713 авг  4  2023 opensearch_dashboards.yml.dpkg-old

$ sudo cat /etc/default/wazuh-dashboard
user="wazuh-dashboard"
group="wazuh-dashboard"
chroot="/"
chdir="/"
nice=""
KILL_ON_STOP_TIMEOUT=0

OSD_PATH_CONF="/etc/wazuh-dashboard"
$ sudo systemctl restart wazuh-dashboard.service && sudo journalctl --since "$(date +"%F %T")" -fu wazuh-dashboard.service
-- Journal begins at Fri 2024-08-30 18:45:40 EEST. --
сен 17 16:39:32 wazuh systemd[1]: Stopping wazuh-dashboard...
сен 17 16:39:32 wazuh opensearch-dashboards[3183161]: {"type":"log","@timestamp":"2024-09-17T13:39:32Z","tags":["info","plugins-system"],"pid":3183161,"message":"Stopping all plugins."}
сен 17 16:39:32 wazuh opensearch-dashboards[3183161]: {"type":"log","@timestamp":"2024-09-17T13:39:32Z","tags":["info","savedobjects-service"],"pid":3183161,"message":"Starting saved objects migrations"}
сен 17 16:39:32 wazuh systemd[1]: wazuh-dashboard.service: Succeeded.
сен 17 16:39:32 wazuh systemd[1]: Stopped wazuh-dashboard.
сен 17 16:39:32 wazuh systemd[1]: wazuh-dashboard.service: Consumed 9.527s CPU time.
сен 17 16:39:32 wazuh systemd[1]: Started wazuh-dashboard.
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:39:40Z","tags":["info","plugins-service"],"pid":3183190,"message":"Plugin \"dataSourceManagement\" has been disabled since the following direct or transitive dependencies are missing or disabled: [dataSource]"}
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:39:40Z","tags":["info","plugins-service"],"pid":3183190,"message":"Plugin \"applicationConfig\" is disabled."}
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:39:40Z","tags":["info","plugins-service"],"pid":3183190,"message":"Plugin \"cspHandler\" is disabled."}
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:39:40Z","tags":["info","plugins-service"],"pid":3183190,"message":"Plugin \"dataSource\" is disabled."}
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:39:40Z","tags":["info","plugins-service"],"pid":3183190,"message":"Plugin \"visTypeXy\" is disabled."}
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:39:40Z","tags":["info","plugins-system"],"pid":3183190,"message":"Setting up [48] plugins: [usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,embeddable,legacyExport,expressions,data,savedObjects,home,apmOss,reportsDashboards,dashboard,visualizations,visTypeVega,visTypeTimeline,visTypeMarkdown,visTypeTable,visBuilder,visAugmenter,alertingDashboards,tileMap,regionMap,customImportMapDashboards,inputControlVis,ganttChartDashboards,visualize,indexManagementDashboards,notificationsDashboards,management,indexPatternManagement,advancedSettings,console,dataExplorer,bfetch,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,discover,savedObjectsManagement,securityDashboards,wazuhCore,wazuhCheckUpdates,wazuh]"}
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
сен 17 16:39:40 wazuh opensearch-dashboards[3183190]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
сен 17 16:39:41 wazuh opensearch-dashboards[3183190]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
сен 17 16:39:41 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:39:41Z","tags":["info","savedobjects-service"],"pid":3183190,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."}
сен 17 16:41:41 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:41:41Z","tags":["error","opensearch","data"],"pid":3183190,"message":"[TimeoutError]: Request timed out"}
сен 17 16:41:41 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:41:41Z","tags":["error","savedobjects-service"],"pid":3183190,"message":"Unable to retrieve version information from OpenSearch nodes."}
сен 17 16:43:41 wazuh opensearch-dashboards[3183190]: {"type":"log","@timestamp":"2024-09-17T13:43:41Z","tags":["error","opensearch","data"],"pid":3183190,"message":"[TimeoutError]: Request timed out"}

@gdiazlo
Copy link
Member

gdiazlo commented Sep 23, 2024

@MAH69IK your problems seems to indicate a communication problem between dahsboard and indexer.

On your message you said you changed the password in the keystore to be like the one stored in wazuh.yml but those files are not related, and do not require having the same password at all.

To fix your deployment, I would follow the changing password documentation from https://documentation.wazuh.com/current/user-manual/user-administration/password-management.html

@BooopLJ
Copy link

BooopLJ commented Sep 26, 2024

Hi @asteriscos thank you for your solution. It worked for me:

"1. Install the new Wazuh dashboard revision 2 packages. 2. Backup your current keystore: cp /etc/wazuh-dashboard/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore.bak 3. Backup the keystore from the new location: cp /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore.bak 4. Move the keystore to the correct location: mv /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore 5. curl -k -XDELETE -u : https://:9200/.kibana_3 6. systemctl restart wazuh-dashboard"

Have a nice day :)

Hi, after resetting the server, the problem with the message returned "Wazuh dashboard server is not ready yet".
Do you have any idea where the problem is?

My Output:
curl -k -u 'user:pass' https://IP:9200/_cluster/health?pretty
{
"cluster_name" : "wazuh-indexer-cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"discovered_master" : true,
"discovered_cluster_manager" : true,
"active_primary_shards" : 999,
"active_shards" : 999,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

sudo cat /etc/default/wazuh-dashboard
user="wazuh-dashboard"
group="wazuh-dashboard"
chroot="/"
chdir="/"
nice=""
KILL_ON_STOP_TIMEOUT=0

OSD_PATH_CONF="/etc/wazuh-dashboard"

sudo systemctl restart wazuh-dashboard.service && sudo journalctl --since "$(date +"%F %T")" -fu wazuh-dashboard.service
wrz 26 12:16:49 serversiem systemd[1]: wazuh-dashboard.service: Deactivated successfully.
wrz 26 12:16:49 serversiem systemd[1]: Stopped wazuh-dashboard.
wrz 26 12:16:49 serversiem systemd[1]: wazuh-dashboard.service: Consumed 27.778s CPU time.
wrz 26 12:16:49 serversiem systemd[1]: Started wazuh-dashboard.
wrz 26 12:17:01 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:01Z","tags":["info","plugins-service"],"pid":33310,"message":"Plugin "dataSourceManagement" has been disabled since the following direct or transitive dependencies are missing or disabled: [dataSource]"}
wrz 26 12:17:01 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:01Z","tags":["info","plugins-service"],"pid":33310,"message":"Plugin "applicationConfig" is disabled."}
wrz 26 12:17:01 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:01Z","tags":["info","plugins-service"],"pid":33310,"message":"Plugin "cspHandler" is disabled."}
wrz 26 12:17:01 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:01Z","tags":["info","plugins-service"],"pid":33310,"message":"Plugin "dataSource" is disabled."}
wrz 26 12:17:01 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:01Z","tags":["info","plugins-service"],"pid":33310,"message":"Plugin "visTypeXy" is disabled."}
wrz 26 12:17:01 serversiem opensearch-dashboards[33310]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
wrz 26 12:17:01 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:01Z","tags":["info","plugins-system"],"pid":33310,"message":"Setting up [48] plugins: [usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,legacyExport,embeddable,expressions,data,savedObjects,home,apmOss,reportsDashboards,dashboard,visualizations,visTypeVega,visTypeTimeline,visTypeTable,visTypeMarkdown,visBuilder,visAugmenter,alertingDashboards,tileMap,regionMap,customImportMapDashboards,inputControlVis,ganttChartDashboards,visualize,indexManagementDashboards,notificationsDashboards,management,indexPatternManagement,advancedSettings,console,dataExplorer,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,discover,savedObjectsManagement,securityDashboards,wazuhCore,wazuhCheckUpdates,wazuh,bfetch]"}
wrz 26 12:17:02 serversiem opensearch-dashboards[33310]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
wrz 26 12:17:02 serversiem opensearch-dashboards[33310]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
wrz 26 12:17:02 serversiem opensearch-dashboards[33310]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
wrz 26 12:17:02 serversiem opensearch-dashboards[33310]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
wrz 26 12:17:02 serversiem opensearch-dashboards[33310]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
wrz 26 12:17:02 serversiem opensearch-dashboards[33310]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
wrz 26 12:17:02 serversiem opensearch-dashboards[33310]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
wrz 26 12:17:02 serversiem opensearch-dashboards[33310]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
wrz 26 12:17:02 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:02Z","tags":["info","savedobjects-service"],"pid":33310,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."}
wrz 26 12:17:03 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:03Z","tags":["info","savedobjects-service"],"pid":33310,"message":"Starting saved objects migrations"}
wrz 26 12:17:03 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:03Z","tags":["info","savedobjects-service"],"pid":33310,"message":"Creating index .kibana_1."}
wrz 26 12:17:03 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:03Z","tags":["error","opensearch","data"],"pid":33310,"message":"[resource_already_exists_exception]: index [.kibana_1/9mbl3wdpQmyx7ZmEhdSW9w] already exists"}
wrz 26 12:17:03 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:03Z","tags":["warning","savedobjects-service"],"pid":33310,"message":"Unable to connect to OpenSearch. Error: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [.kibana_1/9mbl3wdpQmyx7ZmEhdSW9w] already exists"}
wrz 26 12:17:03 serversiem opensearch-dashboards[33310]: {"type":"log","@timestamp":"2024-09-26T10:17:03Z","tags":["warning","savedobjects-service"],"pid":33310,"message":"Another OpenSearch Dashboards instance appears to be migrating the index. Waiting for that migration to complete. If no other OpenSearch Dashboards instance is attempting migrations, you can get past this message by deleting index .kibana_1 and restarting OpenSearchDashboards."}

sudo systemctl status wazuh-indexer
wrz 26 11:28:10 serversiem systemd-entrypoint[20515]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
wrz 26 11:28:10 serversiem systemd-entrypoint[20515]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
wrz 26 11:28:10 serversiem systemd-entrypoint[20515]: WARNING: System::setSecurityManager will be removed in a future release
wrz 26 11:28:11 serversiem systemd-entrypoint[20515]: wrz 26, 2024 11:28:11 AM sun.util.locale.provider.LocaleProviderAdapter
wrz 26 11:28:11 serversiem systemd-entrypoint[20515]: WARNING: COMPAT locale provider will be removed in a future release
wrz 26 11:28:12 serversiem systemd-entrypoint[20515]: WARNING: A terminally deprecated method in java.lang.System has been called
wrz 26 11:28:12 serversiem systemd-entrypoint[20515]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
wrz 26 11:28:12 serversiem systemd-entrypoint[20515]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
wrz 26 11:28:12 serversiem systemd-entrypoint[20515]: WARNING: System::setSecurityManager will be removed in a future release
wrz 26 11:28:30 serversiem systemd[1]: Started wazuh-indexer.

sudo lsof -i -nP | grep LISTEN | grep wazuh-indexer
java 20515 wazuh-indexer 607u IPv6 230815 0t0 TCP 192.168.254.6:9300 (LISTEN)
java 20515 wazuh-indexer 609u IPv6 230823 0t0 TCP 192.168.254.6:9200 (LISTEN)

@kwiha
Copy link

kwiha commented Oct 18, 2024

To resolve this, we have rolled out Wazuh dashboard revision 2 of these packages: wazuh-dashboard-4.9.0-2.deb wazuh-dashboard-4.9.0-2.rpm

where are these packages?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
level/task Task issue reporter/community Issue reported by the community request/operational Operational request type/troubleshooting
Projects
None yet
Development

No branches or pull requests