Deployability testing tier2

[davidjiglesias]

Description

The goal of this issue is to perform an exhaustive testing of Wazuh packages upgrade processes on tier 1 operating systems and architectures. This iteration will build upon the deployability testing established in the first tier (DTT1), now referred to as Deployability Testing Tier 2 (DTT2). DTT2 will concentrate on the upgrade process, ensuring that it is seamless, maintains configurations and permissions, and does not introduce regressions.

Functional requirements

DTT2 focuses on upgrading the following combination of operating systems, versions, and architectures:

Operating System	Version	Component	Architectures
RedHat	7	agents, central components	x86_64, aarch64
RedHat	8	agents, central components	x86_64, aarch64
RedHat	9	agents, central components	x86_64, aarch64
CentOS	7	agents, central components	x86_64, aarch64
CentOS	8	agents, central components	x86_64, aarch64
Debian	10	agents, central components	x86_64, aarch64
Debian	11	agents, central components	x86_64, aarch64
Debian	12	agents, central components	x86_64, aarch64
Ubuntu	18	agents	x86_64, aarch64
Ubuntu	20	agents, central components	x86_64, aarch64
Ubuntu	22	agents, central components	x86_64, aarch64
Oracle Linux	9	agents, central components	x86_64, aarch64
Amazon Linux	2	agents, central components	x86_64, aarch64
Amazon Linux	2023	agents, central components	x86_64, aarch64
openSUSE	15	agents, central components	x86_64, aarch64
SUSE	15	agents, central components	x86_64, aarch64
Fedora	37	agents	x86_64, aarch64
Fedora	38	agents	x86_64, aarch64
Windows	10	agents	x86_64, aarch64
Windows	11	agents	x86_64, aarch64
Windows	Server 2012	agents	x86_64, aarch64
Windows	Server 2012 R2	agents	x86_64, aarch64
Windows	Server 2016	agents	x86_64, aarch64
Windows	Server 2019	agents	x86_64, aarch64
Windows	Server 2022	agents	x86_64, aarch64
macOS	Ventura	agents	x86_64, aarch64
macOS	Sonoma	agents	x86_64, aarch64

Upgrade Process

• DTT2 includes the following phases for the upgrade process:
  • Pre-upgrade checks
  • Upgrade execution
  • Post-upgrade validation

Phase	Requirement
Pre-upgrade checks	Ensure all services are running and configurations are intact before upgrade
Upgrade execution	Execute the upgrade process using the recommended method for each OS and component
Upgrade execution	Verify that the upgrade does not affect running services or processes
Post-upgrade validation	Ensure file permissions are maintained post-upgrade
Post-upgrade validation	Ensure configuration is maintained post-upgrade (ossec.conf, agent.conf, local_internal_options.conf)
Post-upgrade validation	Verify successful reconnection of agents to the manager post-upgrade
Post-upgrade validation	Confirm all components and services are operational post-upgrade

Automation

• DTT2 tests must be integrated into the Nightly CI pipeline
• DTT2 tests must be integrated into the Weekly CI pipeline
• DTT2 tests must be part of pre-release testing
• The cost and time of DTT2 tests must be measurable and optimized

Non-functional requirements

• All DTT2 test phases must comply with the following requirements:
  • Ensure the upgrade process does not exceed the maximum time defined for the specific phase
  • Ensure no errors are found in logs post-upgrade
• DTT2 tests must be designed with a modular approach for easy deployment, execution, and result analysis
• DTT2 test executions must be monitored and reviewed regularly by the QA team
• DTT2 tests evaluation criteria must be clearly defined and communicated to all QA team members
• An escalation process for DTT2 test failures must be established and documented

Hardware

Upgrade specifications

• Agents and Central Components:
  • Upgrade from the previous patch version
  • Upgrade from the previous minor version

Implementation restrictions

• DTT2 CI architecture and infrastructure must be implemented in Jenkins.
• DTT2 tests must be developed using Python.
• DTT2 must utilize virtual machines for deploying the OSs.

Plan

Iteration 2:

Objective:

Focus on resolving the upgrade-specific challenges identified in tier 1 testing. This includes ensuring that the upgrade process is robust, configurations are maintained, and the system remains secure post-upgrade.

Tasks:

• Implement automated upgrade tests for all tier 1 OSs and architectures
• Validate upgrade process integrity and consistency across all supported platforms
• Ensure all upgrade tests are integrated into the CI pipeline for regular execution

Expected Results:

• A comprehensive report on the upgrade testing across all tier 1 operating systems and architectures, including any issues identified and recommendations for improvements.

Approved by

DRI name: @davidjiglesias
CTO: @havidarou
Objective: Ensure robust and reliable upgrade processes for tier 1 operating systems and architectures

[rauldpm]

Related #5137