Wazuh Offline Installation fails due to trying to install wazuh-templates.json with no internet connection

[CarlosALgit]

Wazuh version	Install type	Action performed	Platform
4.9.0-beta2	Indexer	Offline Install	Amazon Linux 2023

While performing the Installation Assistant for 4.9.0-beta2 test I followed the steps for the Offline Installation on the documentation and the installation freezed on this message:

14/08/2024 08:51:43 INFO: Wazuh indexer cluster security configuration initialized.

This occurs after running the command for starting the cluster:

bash wazuh-install.sh --start-cluster

After investigating with my team, we discovered that the error comes from this line in the indexer.sh file. This tries to download the wazuh-template.json file using curl in the host you are supposed to not need internet.

wazuh-packages/unattended_installer/install_functions/indexer.sh

Line 190 in add6b47

eval "common_curl --silent ${filebeat_wazuh_template} --max-time 300 --retry 5 --retry-delay 5 ${debug}" | eval "common_curl -X PUT 'https://${indexer_node_ips[pos]}:9200/_template/wazuh' -H 'Content-Type: application/json' -d @- -uadmin:admin -k --silent --max-time 300 --retry 5 --retry-delay 5 ${debug}"

So, the fix needed is to change this curl command to the part of the installation process where you have internet connection.

[davidcr01]

Update Report

Development

• Fixed lsof installation trial.
• Fixed prerequisites installation trial.
• Fixed template download.
• Fixed adding --offline-installation parameter when starting the indexer cluster

Warning

These changes should be reviewed in incoming Wazuh versions, as affected functions were modified. Related: #2879

Wazuh indexer installation log:

root@ip-172-31-46-83:/home/ubuntu# bash wazuh-install.sh --offline-installation --wazuh-indexer node-1 -v
14/08/2024 12:45:35 DEBUG: Checking root permissions.
14/08/2024 12:45:35 DEBUG: Checking sudo package.
14/08/2024 12:45:35 INFO: Starting Wazuh installation assistant. Wazuh version: 4.9.0
14/08/2024 12:45:35 INFO: Verbose logging redirected to /var/log/wazuh-install.log
14/08/2024 12:45:35 DEBUG: APT package manager will be used.
14/08/2024 12:45:35 DEBUG: Checking system distribution.
14/08/2024 12:45:35 DEBUG: Detected distribution name: ubuntu
14/08/2024 12:45:35 DEBUG: Detected distribution version: 22
14/08/2024 12:45:35 INFO: Checking installed dependencies for Offline installation.
14/08/2024 12:45:38 DEBUG: Offline dependencies are installed.
14/08/2024 12:45:38 DEBUG: Checking Wazuh installation.
14/08/2024 12:45:40 DEBUG: Checking system architecture.
14/08/2024 12:45:40 INFO: Verifying that your system meets the recommended minimum hardware requirements.
14/08/2024 12:45:40 DEBUG: CPU cores detected: 2
14/08/2024 12:45:40 DEBUG: Free RAM memory detected: 7833
14/08/2024 12:45:40 DEBUG: Checking previous certificate existence.
14/08/2024 12:45:40 DEBUG: Checking ports availability.
14/08/2024 12:45:42 INFO: Checking prerequisites for Offline installation.
14/08/2024 12:45:45 DEBUG: Offline prerequisites are installed.
14/08/2024 12:45:45 INFO: Checking wazuh-offline.tar.gz file.
14/08/2024 12:45:45 DEBUG: wazuh-offline.tar.gz was found correctly.
14/08/2024 12:45:45 DEBUG: Extracting files from wazuh-offline.tar.gz
14/08/2024 12:45:45 DEBUG: Offline files extracted successfully.
14/08/2024 12:45:45 DEBUG: Checking curl tool version.
14/08/2024 12:45:45 DEBUG: Extracting Wazuh configuration.
14/08/2024 12:45:45 DEBUG: Reading configuration file.
14/08/2024 12:45:45 DEBUG: Checking if 127.0.0.1 is private.
14/08/2024 12:45:45 DEBUG: Checking if 127.0.0.1 is private.
14/08/2024 12:45:45 DEBUG: Checking if 127.0.0.1 is private.
14/08/2024 12:45:46 DEBUG: Checking node names in the configuration file.
14/08/2024 12:45:46 INFO: --- Wazuh indexer ---
14/08/2024 12:45:46 INFO: Starting Wazuh indexer installation.
Reading package lists... Building dependency tree... Reading state information... The following NEW packages will be installed: wazuh-indexer 0 upgraded, 1 newly installed, 0 to remove and 32 not upgraded. Need to get 0 B/850 MB of archives. After this operation, 1077 MB of additional disk space will be used. Get:1 /home/ubuntu/wazuh-offline/wazuh-packages/wazuh-indexer_4.9.0-1_amd64.deb wazuh-indexer amd64 4.9.0-1 [850 MB] Selecting previously unselected package  ### NOT starting on installation, please execute the following statements to configure wazuh-indexer service to star NEEDRESTART-VER: 3.5 NEEDRESTART-KCUR: 6.5.0-1022-aws NEEDRESTART-KEXP: 6.5.0-1022-aws NEEDRESTART-KSTA: 1
14/08/2024 12:46:17 DEBUG: Checking Wazuh installation.
14/08/2024 12:46:18 DEBUG: There are Wazuh indexer remaining files.
14/08/2024 12:46:20 INFO: Wazuh indexer installation finished.
14/08/2024 12:46:20 DEBUG: Configuring Wazuh indexer.
14/08/2024 12:46:20 DEBUG: Copying Wazuh indexer certificates.
14/08/2024 12:46:20 INFO: Wazuh indexer post-install configuration finished.
14/08/2024 12:46:20 INFO: Starting service wazuh-indexer.
Synchronizing state of wazuh-indexer.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable wazuh-indexer
Created symlink /etc/systemd/system/multi-user.target.wants/wazuh-indexer.service → /lib/systemd/system/wazuh-indexer.service.
14/08/2024 12:46:47 INFO: wazuh-indexer service started.
14/08/2024 12:46:47 INFO: Initializing Wazuh indexer cluster security settings.
14/08/2024 12:46:48 DEBUG: Setting Wazuh indexer cluster passwords.
14/08/2024 12:46:48 DEBUG: Checking Wazuh installation.
14/08/2024 12:46:49 DEBUG: There are Wazuh indexer remaining files.
14/08/2024 12:46:51 INFO: Wazuh indexer cluster initialized.
14/08/2024 12:46:51 INFO: Installation finished.

The indexer cluster start is stuck:

root@ip-172-31-46-83:/home/ubuntu# bash wazuh-install.sh --start-cluster --offline-installation -v
14/08/2024 12:54:41 DEBUG: Checking root permissions.
14/08/2024 12:54:41 DEBUG: Checking sudo package.
14/08/2024 12:54:41 INFO: Starting Wazuh installation assistant. Wazuh version: 4.9.0
14/08/2024 12:54:41 INFO: Verbose logging redirected to /var/log/wazuh-install.log
14/08/2024 12:54:41 DEBUG: APT package manager will be used.
14/08/2024 12:54:41 DEBUG: Checking system distribution.
14/08/2024 12:54:41 DEBUG: Detected distribution name: ubuntu
14/08/2024 12:54:41 DEBUG: Detected distribution version: 22
14/08/2024 12:54:41 INFO: Checking installed dependencies for Offline installation.
14/08/2024 12:54:44 DEBUG: Offline dependencies are installed.
14/08/2024 12:54:44 DEBUG: Checking Wazuh installation.
14/08/2024 12:54:45 DEBUG: There are Wazuh indexer remaining files.
14/08/2024 12:54:46 DEBUG: Checking system architecture.
14/08/2024 12:54:46 INFO: Verifying that your system meets the recommended minimum hardware requirements.
14/08/2024 12:54:46 DEBUG: CPU cores detected: 2
14/08/2024 12:54:46 DEBUG: Free RAM memory detected: 7833
14/08/2024 12:54:46 DEBUG: Checking previous certificate existence.
14/08/2024 12:54:46 INFO: Checking wazuh-offline.tar.gz file.
14/08/2024 12:54:46 DEBUG: wazuh-offline.tar.gz was found correctly.
14/08/2024 12:54:46 DEBUG: Extracting files from wazuh-offline.tar.gz
14/08/2024 12:54:46 DEBUG: Offline files extracted successfully.
14/08/2024 12:54:46 DEBUG: Extracting Wazuh configuration.
14/08/2024 12:54:47 DEBUG: Reading configuration file.
14/08/2024 12:54:47 DEBUG: Checking if 127.0.0.1 is private.
14/08/2024 12:54:47 DEBUG: Checking if 127.0.0.1 is private.
14/08/2024 12:54:47 DEBUG: Checking if 127.0.0.1 is private.
14/08/2024 12:54:47 DEBUG: Starting Wazuh indexer cluster.
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 127.0.0.1:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.13.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-indexer-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index does not exists, attempt to create it ... done (0-all replicas)
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml 
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml 
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml 
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml 
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml 
   SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml 
   SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml 
   SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success
14/08/2024 12:54:54 INFO: Wazuh indexer cluster security configuration initialized.
OpenSearch Security not initialized.OpenSearch Security not initialized.{"error":{"root_cause":[{"type":"parse_exception","reason":"request body is required"}],"type":"parse_exception","reason":"request body is required"},"status":400}

🔴 The reported error is the following:

OpenSearch Security not initialized.OpenSearch Security not initialized.{"error":{"root_cause":[{"type":"parse_exception","reason":"request body is required"}],"type":"parse_exception","reason":"request body is required"},"status":400}

It is necessary to investigate why this message is being generated.

[c-bordon]

Update report

The error mentioned by David is due to the issue of file parsing. I have been analyzing and found problems in this case PUT'. In principle the error occurs when the command is passed through the common_curl where the single quote is removed:

14/08/2024 19:32:09 INFO: Wazuh indexer cluster security configuration initialized.
+ '[' -n 1 ']'
+ sleep 5
+ eval 'common_curl -X PUT '\''https://127.0.0.1:9200/_template/wazuh'\'' -H '\''Content-Type: application/json'\'' -d '\''@/home/ubuntu/wazuh-offline/wazuh-files/wazuh-template.json'\'' -uadmin:admin -k --silent --max-time 300 --retry 5 --retry-delay 5 2>&1 | tee -a /var/log/wazuh-install.log'
++ common_curl -X PUT https://127.0.0.1:9200/_template/wazuh -H 'Content-Type: application/json' -d @/home/ubuntu/wazuh-offline/wazuh-files/wazuh-template.json -uadmin:admin -k --silent --max-time 300 --retry 5 --retry-delay 5
++ tee -a /var/log/wazuh-install.log
++ '[' -n '' ']'
++ retries=0
++ eval 'curl -X' PUT https://127.0.0.1:9200/_template/wazuh -H 'Content-Type: application/json' -d @/home/ubuntu/wazuh-offline/wazuh-files/wazuh-template.json -uadmin:admin -k --silent --max-time 300 --retry 5 --retry-delay 5
+++ curl -X PUT https://127.0.0.1:9200/_template/wazuh -H Content-Type: application/json -d @/home/ubuntu/wazuh-offline/wazuh-files/wazuh-template.json -uadmin:admin -k --silent --max-time 300 --retry 5 --retry-delay 5
{"error":"Content-Type header [] is not supported","status":406}++ e_code=6
++ '[' 6 -eq 7 ']'
++ return 6
+ set +x

ubuntu@ip-172-31-46-83:~$ sudo curl -X PUT https://127.0.0.1:9200/_template/wazuh -H Content-Type: application/json -d @/home/ubuntu/wazuh-offline/wazuh-files/wazuh-template.json -uadmin:admin -k --silent --max-time 300 --retry 5 --retry-delay 5
{"error":"Content-Type header [] is not supported","status":406}ubuntu@ip-172-31-46-83:~$ 
ubuntu@ip-172-31-46-83:~$ 
ubuntu@ip-172-31-46-83:~$ sudo curl -X PUT https://127.0.0.1:9200/_template/wazuh -H 'Content-Type: application/json' -d @/home/ubuntu/wazuh-offline/wazuh-files/wazuh-template.json -uadmin:admin -k --silent --max-time 300 --retry 5 --retry-delay 5
{"acknowledged":true}