From f5f176e826f802573803b0a56d1ae29a7fb8bc81 Mon Sep 17 00:00:00 2001 From: Will Foster Date: Wed, 9 Oct 2024 10:07:11 +0100 Subject: [PATCH] doc: further expand filter options & lshw. * Add more supported filter fields * Make lshw methods more detailed. * Manage cron example file and update docs * Cleanup cron for M&R with start and finish times. * Make RPM install the QUADS crons commented out for new installs so it reduces a step new users have to do. Change-Id: I2f0de02cf50a2a206e896269052b42472c2b675f --- README.md | 8 ++- cron/quads | 4 +- docs/quads-host-metadata-search.md | 97 +++++++++++++++++++++++++++--- rpm/quads.spec | 3 + 4 files changed, 99 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 56b247fc..73eab2e5 100644 --- a/README.md +++ b/README.md @@ -283,7 +283,13 @@ systemctl restart nginx - QUADS relies on calling an external script, trigger or workflow to enact the actual provisioning of machines. You can look at and modify our [move-and-rebuild-hosts](https://github.com/redhat-performance/quads/blob/latest/src/quads/tools/move_and_rebuild.py) tool to suit your environment for this purpose. Read more about this in the [move-host-command](https://github.com/redhat-performance/quads#quads-move-host-command) section below. ### Making QUADS Run - - QUADS is a passive service and does not do anything you do not tell it to do. We control QUADS with cron, please copy and modify our [example cron commands](https://raw.githubusercontent.com/redhat-performance/quads/latest/cron/quads) to your liking, adjust as needed. + - QUADS is a passive service and does not do anything you do not tell it to do. We control QUADS with [cron commands](https://raw.githubusercontent.com/redhat-performance/quads/latest/cron/quads). + - We ship an example cron file and install it for you that should work out of the box, it just has entries commented out. + - To enable QUADS services to run you'll need to **uncomment them**. + +``` +crontab -e +``` #### Major Components diff --git a/cron/quads b/cron/quads index 0fb1ffc1..2c74351a 100644 --- a/cron/quads +++ b/cron/quads @@ -22,7 +22,7 @@ SHELL=/bin/bash ##### CONTAINER QUADS SERVER ENTRIES ##### ONLY USE WITH CONTAINER DEPLOYMENT ######################################## -# * * * * * ( echo "=== Move and Rebuild == @ " $(date) ; flock -n /tmp/moveandrebuild.lock -c "podman exec quads quads --move-hosts" ) 1>>/var/log/move-and-rebuild.log 2>&1 +# * * * * * flock -n /tmp/moveandrebuild.lock -c "echo '=== Move and Rebuild == Started @ ' $(date) ; podman exec quads quads --move-hosts ; echo '=== Move and Rebuild == Finished @ ' $(date)" 1>/dev/null 2>&1 # * * * * * ( echo "=== Validating == @ " $(date) ; flock -n /tmp/validateenv.lock -c "podman exec quads quads --validate-env" ) 1>/dev/null 2>&1 # 0 0 * * * ( echo "=== Sending Notifications == @ " $(date) ; flock -n /tmp/notify.lock -c "podman exec quads quads --notify" ) 1>/dev/null 2>&1 # 0 0 * * * ( echo "=== Regenerate Heat Map == @ " $(date) ; flock -n /tmp/heatmap.lock -c "podman exec quads quads --regen-heatmap" ) 1>/dev/null 2>&1 @@ -35,7 +35,7 @@ SHELL=/bin/bash ##### (RECOMMENDED) ######################################### ######################################### -#*/5 * * * * ( echo "=== Move and Rebuild == @ " $(date) ; flock -n /tmp/moveandrebuild.lock -c "quads --move-hosts" ) 1>>/var/log/move-and-rebuild.log 2>&1 +#*/5 * * * * flock -n /tmp/moveandrebuild.lock -c "echo '=== Move and Rebuild == Started @ ' $(date) ; quads --move-hosts ; echo '=== Move and Rebuild == Finished @ ' $(date)" 1>>/var/log/move-and-rebuild.log 2>&1 #*/25 * * * * ( echo "=== Validating == @ " $(date) ; flock -n /tmp/validateenv.lock -c "quads --validate-env" ) 1>>/var/log/validate-env.log 2>&1 #*/10 * * * * ( echo "=== Sending Notifications == @ " $(date) ; flock -n /tmp/notify.lock -c "quads --notify" ) 1>>/var/log/quads-notify.log 2>&1 #*/20 * * * * ( echo "=== Regenerate Heat Map == @ " $(date) ; flock -n /tmp/heatmap.lock -c "quads --regen-heatmap" ) 1>/dev/null 2>&1 diff --git a/docs/quads-host-metadata-search.md b/docs/quads-host-metadata-search.md index 0b39f001..725112f2 100644 --- a/docs/quads-host-metadata-search.md +++ b/docs/quads-host-metadata-search.md @@ -26,20 +26,21 @@ In QUADS `1.1.4` and above we've implemented a metadata model in the QUADS datab * We can use the popular [lshw](https://linux.die.net/man/1/lshw) tool to gather hardware details into JSON * We ship a tool called `lshw2meta.py` to transform this into a format for updating host metadata into QUADS. -First, install `lshw` on your target host(s) +First, install `lshw` on your target QUADS-managed host(s) ``` dnf install lshw ``` -Next run `lshw` to capture all the hardware details of your host in JSON. +Next run `lshw` to capture all the hardware details of each remote host in JSON. ``` lshw -json > $(hostname).json ``` -Next, copy the JSON file(s) over to your QUADS host `quads:/opt/quads/lshw` -Now on your QUADS host use the `lshw2meta.py` tool to convert this data and import it directly into the QUADS database for each host. +Next, copy the JSON file(s) over to your QUADS host here: `quads:/opt/quads/lshw/` + +Now, back on your QUADS host use the `lshw2meta.py` tool to convert this data and import it directly into the QUADS database for each host. ``` python3 /usr/lib/python3.12/site-packages/quads/tools/lshw2meta.py @@ -51,7 +52,7 @@ python3 /usr/lib/python3.12/site-packages/quads/tools/lshw2meta.py * This assumes all of your hosts are in `cloud01` and powered on and accessible * This assumes you have `lshw` installed as well on every remote host -First, gather all of the JSON metadata from the hosts: +First, gather all of the JSON metadata from your remote QUADS-managed host(s): ``` python3 /usr/lib/python3.12/site-packages/quads/tools/lshw.py ``` @@ -92,11 +93,11 @@ quads --export-host-details /tmp/my_host_data.yml ``` ## Querying Host Information - * A new sub-command of `--filter` has been added to the `--ls-available` and `--ls-hosts` commands. + * The sub-command `--filter` can be used with `--ls-available` and `--ls-hosts` commands. -| Component | Field Type | Syntax | Operators | +| Component | Field Type | Description | Operators | |------------------------|------------|------------------------------|-----------------| -| model | string | exact match | ==,!= | +| model | string | defined system model | ==,!= | | disks.size_gb | integer | disk size in GB | ==,!=,<,<=,>,>= | | disks.disks_type | string | nvme,sata,ssd | ==,!= | | disks.count | integer | number of disks | ==,!=,<,<=,>,>= | @@ -104,18 +105,39 @@ quads --export-host-details /tmp/my_host_data.yml | interfaces.name | string | name of interface | ==,!= | | interfaces.mac_address | string | mac address | ==,!= | | interfaces.switch_port | string | switch port | ==,!= | +| interfaces.switch_ip | string | switch ip address per port | ==,!= | | interfaces.speed | integer | link speed | ==,!=,<,<=,>,>= | | interfaces.vendor | string | interface vendor | ==,!= | | interfaces.maintenance | boolean | interface maintenance status | ==,!= | +| interfaces.bios_id | string | interface BIOS boot devname | ==,!= | | build | boolean | build status | ==,!= | | validated | boolean | validated status | ==,!= | | broken | boolean | broken status | ==,!= | | retired | boolean | retired status | ==,!= | -| switch_config_applied | boolean | switch configuration status | ==,!= | - +| switch_config_applied | boolean | host switch config status | ==,!= | +| memory.handle | string | DIMM details | ==,!= | +| memory.size_gb | integer | amount of system memory | ==,!=,<,<=,>,>= | +| processors.handle | string | CPU details | ==,!= | +| processors.vendor | string | CPU vendor information | ==,!= | +| processors.product | string | CPU model information | ==,!= | +| processors.cores | integer | CPU cores in the system | ==,!=,<,<=,>,>= | +| processors.threads | integer | CPU threads in the system | ==,!=,<,<=,>,>= | ### Example Filter Searches * Accepted operators are `==, !=, <, <=, >, >=` + * For the REST API the operators must be appended to the key like so `disks.size_gb__gte=2000` + +| Operator | Appended | +|----------|----------| +| == | | +| != | __ne | +| < | __lt | +| <= | __lte | +| > | __gt | +| >= | __gte | +| like | __like | +| ilike | __ilike | +| in | __in | #### Example Hardware Filter Searches @@ -123,30 +145,55 @@ quads --export-host-details /tmp/my_host_data.yml ``` quads --ls-available --schedule-start "2020-07-20 17:00" --schedule-end "2020-07-22 13:00" --filter "disks.disk_type==nvme,disks.size_gb>=2000" +``` + * Via the API + +``` +curl https://quads.example.com/api/v3/available?start=2020-07-20T17:00&end=2020-07-22T13:00&disks.disk_type=nvme&disks.size_gb__gte=2000 ``` * Search for systems with SATA disks available from `2020-07-20 17:00` until `2020-07-22 13:00` ``` quads --ls-available --schedule-start "2020-07-20 17:00" --schedule-end "2020-07-22 13:00" --filter "disks.disk_type==sata" +``` + * Via the API + +``` +curl https://quads.example.com/api/v3/available?start=2020-07-20T17:00&end=2020-07-22T13:00&disks.disk_type=sata ``` * Search for systems of model type `1029U-TRTP` available from `2020-07-20 17:00` until `2020-07-22 13:00` ``` quads --ls-available --schedule-start "2020-07-20 17:00" --schedule-end "2020-07-22 13:00" --filter "model==1029U-TRTP" +``` + * Via the API + +``` +curl https://quads.example.com/api/v3/available?start=2020-07-20T17:00&end=2020-07-22T13:00&model=1029U-TRTP ``` * Search for systems with **two NVMe** disks **and** disk size of **more than** 2TB, available from `2020-07-20 17:00` until `2020-07-22 13:00` ``` quads --ls-available --schedule-start "2020-07-20 17:00" --schedule-end "2020-07-22 13:00" --filter "disks.disk_type==nvme,disks.count>2, disks.size_gb<2000" +``` + * Via the API + +``` +curl https://quads.example.com/api/v3/available?start=2020-07-20T17:00&end=2020-07-22T13:00&disks.disk_type=nvme&disks.count__gt=2&disks.size_gb__lt=2000 ``` * Search all systems by model and number of interfaces. ``` quads --ls-hosts --filter "model==FC640,interfaces__size==5" +``` + * Via the API + +``` +curl https://quads.example.com/api/v3/hosts?model=FC640&interfaces__size=5 ``` #### Example Network Filter Searches @@ -156,6 +203,11 @@ quads --ls-hosts --filter "model==FC640,interfaces__size==5" ``` quads --ls-hosts --filter "interfaces.mac_address==ac:1f:6b:2d:19:48" +``` + * Via the API + +``` +curl https://quads.example.com/api/v3/hosts?interfaces.mac_address=ac:1f:6b:2d:19:48 ``` * Find hosts by switch IP address. @@ -163,6 +215,11 @@ quads --ls-hosts --filter "interfaces.mac_address==ac:1f:6b:2d:19:48" ``` quads --ls-host --filter "interfaces.ip_address==10.1.34.216" +``` + * Via the API + +``` +curl https://quads.example.com/api/v3/hosts?interfaces.ip_address=10.1.34.216 ``` * Find hosts by physical switchport @@ -170,6 +227,11 @@ quads --ls-host --filter "interfaces.ip_address==10.1.34.216" ``` quads --ls-host --filter "interfaces.switch_port==et-0/0/7:1" +``` + * Via the API + +``` +curl https://quads.example.com/api/v3/hosts?interfaces.switch_port=et-0/0/7:1 ``` ##### Combined Network Search Example @@ -179,6 +241,11 @@ quads --ls-host --filter "interfaces.switch_port==et-0/0/7:1" ``` quads --ls-hosts --filter "interfaces.ip_address==10.1.34.216,interfaces.switch_port==et-0/0/7:1" +``` + * Via the API + +``` +curl https://quads.example.com/api/v3/hosts?interfaces.ip_address=10.1.34.216&interfaces.switch_port=et-0/0/7:1 ``` ## Querying Host Status @@ -191,10 +258,20 @@ quads --ls-hosts --filter "interfaces.ip_address==10.1.34.216,interfaces.switch_ ``` quads --ls-hosts --filter "retired==True" +``` + * Via the API + +``` +curl https://quads.example.com/api/v3/hosts?retired=True ``` * List retired hosts and filter by model ``` quads --ls-hosts --filter "retired==True,model==1029P" +``` + * Via the API + +``` +curl https://quads.example.com/api/v3/hosts?retired=True&model=1029P ``` diff --git a/rpm/quads.spec b/rpm/quads.spec index fe023795..fa7dada3 100644 --- a/rpm/quads.spec +++ b/rpm/quads.spec @@ -116,6 +116,7 @@ cp -rf systemd/quads-server.service %{buildroot}/etc/systemd/system/ cp -rf systemd/quads-web.service %{buildroot}/etc/systemd/system/ cp -rf systemd/quads-db.service %{buildroot}/etc/systemd/system/ cp -rf conf/logrotate_quads.conf %{buildroot}/etc/logrotate.d/ +cp -rf cron/quads %{buildroot}/opt/quads/conf/quads.cron.example cp -rf container/etc/nginx/conf.d/apiv3.conf %{buildroot}/etc/nginx/conf.d/ cp -rf container/etc/nginx/conf.d/apiv3_ssl.conf.example %{buildroot}/etc/nginx/conf.d/ cp -rf container/etc/postfix/postfix-files.d/quads.cf %{buildroot}/etc/postfix/postfix-files.d/ @@ -135,6 +136,7 @@ rm -rf %{buildroot} /etc/profile.d/quads.sh /etc/nginx/* /opt/quads/conf/logrotate_quads.conf +/opt/quads/conf/quads.cron.example /usr/bin/quads %config(noreplace) /opt/quads/conf/quads.yml %config(noreplace) /opt/quads/conf/vlans.yml @@ -188,6 +190,7 @@ if [ "$1" -eq 1 ]; then sed -i 's/ident/password/g' /opt/quads/db/data/pg_hba.conf /usr/bin/systemctl start quads-db /usr/bin/systemctl start haveged + /usr/bin/crontab /opt/quads/conf/quads.cron.example cd /var/lib/pgsql && sudo -u postgres psql -c "ALTER USER postgres WITH PASSWORD 'postgres';" /usr/bin/systemctl start nginx fi