CI-Monitoring-Automation

Overview

The ci-monitoring-automation Repository is a collection of Python scripts designed to automate monitoring of Continuous Integration (CI) system workflows. These scripts facilitate the retrieval and analysis of information related to coreOS CI builds, offering developers the flexibility to gather data of builds run for the current day or within a specified date range.

Prerequisites

Python3
pip

Installation

git clone https://github.com/ocp-power-automation/ci-monitoring-automation.git
cd ci-monitoring-automation
Create a virtualenv if required and install required packages using "pip install -r requirements.txt"

Config files

1. p_periodic.json: The p_periodic.json file will have ci name and ci links of ppc64le architecture periodic jobs in the key value pair where value of ci link will be prow periodic job name.The new CI's can be easily integrated by adding the ci name and ci link in the config.json file.

2. z_periodic.json: The z_periodic.json file will have ci name and ci links of s390x architecture periodic jobs in the key value pair where value of ci link will be prow periodic job name.The new CI's can be easily integrated by adding the ci name and ci link in the config.json file.

3. p_auxillary.json: The p_auxillary.json file will have ci name and ci links of ppc64le architecture auxilliary jobs in the key value pair where value of ci link will be prow ecosystem, jenkins, fips serial and compact job name.The new CI's can be easily integrated by adding the ci name and ci link in the p_auxillary.json file.

Usage

1. CI_DailyBuildUpdate.py: The CI_DailyBuildUpdates.py script will fetch and display information of the all builds that ran on the CI system for the current day.

   1. Brief Information: The CI_DailyBuildUpdates.py script when invoked with command line argument info_type as "brief" it will display Build type, job id, cluster deployment status, Lease and total number of failed testcases.

      python3 CI_DailyBuildUpdates.py --info_type brief

   2. The CI_DailyBuildUpdates.py script when invoked with command line arguments info_type as "brief" and zone, it will display the builds details in the provided zone.

      python3 CI_DailyBuildUpdates.py --info_type brief --zone syd04

   3. Detailed Information: The CI_DailyBuildUpdates.py script when invoked with command line argument info_type as "detailed" it will display the Job id, job link, cluster deployment error message if it occured, Nightly image used, Node status, Checks for Node crash and lists all the failed testcases.

      python3 CI_DailyBuildUpdates.py --info_type detailed

   4. The CI_DailyBuildUpdates.py script when invoked with command line arguments info_type as "detailed" and zone, it will display the builds details in the provided zone.

      python3 CI_DailyBuildUpdates.py --info_type detailed --zone syd04

2. CI_JobHistory.py: The CI_JobHistory.py is a script which allows user to query a specific information from all builds that ran on the CI system within a given date range.

   python3 CI_JobHistory.py

   1. Interactive Execution: The CI_JobHistory.py can be executed in a interactive mode by setting JENKINS variable as False in config.ini file.

   2. Non-Interactive Execution: The CI_JobHistory.py can be executed in a non-interactive mode by setting JENKINS variable as True in config.ini file, along with the JENKINS variable user needs to provide values for the following variables:

      selected_ci: CI's from where to fetch the jobs.
      before_date: End date.
      after_date: Start date.
      query_option: Query code to fetch information from builds.
      tc_name: Testcase name which will be used in quering the failure frequency.
      

3. tracker.py: The tracker.py script helps user to get the testcases failing at a high frequency in the latest run CI builds. This script accepts the following command line arguments:

   python3 tracker.py --builds 10 --frequency 3

   1. builds: This argument accepts int value, which will query for failed testcases in "n" latest build run in the CI, default value set is 10.
   2. frequency: This argument accepts int value, which specifies the frequency threshold and report the testcases which are failing above the frequency, default value set is 3.

4. aggregate.py: The aggregate.py script gets detailed information of all the builds which have run using the provided nightly image.

   python3 aggregate.py This script requires the following input

   1. selected_ci: Jobs from where to fetch the builds.
   2. nightly: Name of the Release image.