README.md

QUADS (quick and dirty scheduler)

Automate scheduling and end-to-end provisioning of servers and networks.

• Please use our Gerrit Review to submit patches.
• We use Waffle.io for additional development tracking and priorities.

• QUADS (quick and dirty scheduler)
  • What does it do?
  • Notes
  • Requirements
  • QUADS Workflow
  • QUADS Foreman Provisioning Workflow
  • Example: Automated Scheduling
  • Example: Systems Wiki
  • Example: Workload Assignments
  • Example: Calendar View
  • Example: Systems Visualization Map
  • Example: IRC and Email Notifications
  • Installing QUADS
    • Installing QUADS from Github
    • Installing QUADS from RPM
  • QUADS Usage Documentation
  • QUADS Switch and Host Setup
  • Common Administration Tasks
    • Creating a New Cloud Assignment and Schedule
      • QUADS VLAN Options
      • Defining a New Cloud
      • Adding New Hosts to your Cloud
    • Extending the Schedule of an Existing Cloud
    • Extending the Schedule of Existing Cloud with Differing Active Schedules
    • Extending Machine Allocation to an existing Cloud
    • Removing a Schedule
    • Removing a Schedule across a large set of hosts
  • Additional Tools and Commands
  • Using the QUADS JSON API
    • API GET Operations
    • API POST Operations
    • Working Examples
    • More Examples with API POST
  • Contributing
  • QUADS Talks and Media

What does it do?

• Create and manage a date/time based YAML schedule for machine allocations
• Drive system provisioning and network switch changes based on workload assignment via external commands
• Automated network and provisioning validation prior to delivering sets of machines/networks to users.
• Generates instackenv.json for each OpenStack environment.
• Automatically generate documentation to illustrate current status, published to a Wordpress instance
  • Current system details
  • Current workloads and assignments
  • Current ownership and resource utilization links (grafana/collectd)
  • Total duration and time remaining in system assignments
  • Dynamic provisioning & system/network validation status per assignment
  • Granular Ansible facts inventory per server via ansible-cmdb
• Query scheduling data to determine future availability
• Generates a monthly, auto-updated calendar of machine assignments
• Generates a per-month visualization map for per-machine allocations to assignments.
• RT (or similiar ticketing system) integration.
• IRC bot and email notifications for new provisioning tasks and ones ending completion

Notes

• Very simple design (flat files, no external DB)
• Allows for calling external provisioning commands via --path-to-command
• We use Foreman for the systems provisioning backend, but this can be substituted.

Requirements

• Python 2.6+ and libyaml (or pyaml) are required for basic operation.
• The scheduling functionality can be used standalone, but you'll want a provisioning backend like Foreman to take full advantage of QUADS scheduling, automation and provisioning capabilities.
• To utilize the automatic wiki/docs generation we use Wordpress but anything that accepts markdown via an API should work.
• Switch/VLAN automation is done on Juniper Switches in Q-in-Q VLANs, but commandsets can easily be extended to support other network switch models.
• We use Ansible for optional Dell and SuperMicro playbooks to toggle boot order and PXE flags to accomodate OpenStack deployments via Ironic/Triple-O.
• The package ansible-cmdb needs to be available if you want to see per assignment Ansible facts of the inventory. It can be obtained from here

QUADS Workflow

QUADS Foreman Provisioning Workflow

Example: Automated Scheduling

Example: Systems Wiki

Example: Workload Assignments

Example: Workload Assignments Readiness

Example: Calendar View

Example: Systems Visualization Map

Example: IRC and Email Notifications

• We notify our Supybot IRC bot to announce when new environments are provisioned

<lucius> QUADS: cloud02 : 9 (OSP Newton Testing) is now active, choo choo! - http://wiki.example.com/assignments/#cloud02

• We send email notifications when new environments are defined.
• We also send email notifications with the host list for the environment 7 days prior to activation.
• Furthermore we send email notifications when new environments are provisioned.

Greetings Citizen,

You've been allocated a new environment!

cloud06 : 13 (OVN and OpenStack ML2/OVS)

(Details)
http://wiki.example.com/assignments/#cloud06

• Lastly we send notifications 7, 5, 3, 1 days out from when assignments expire (or any number of machines are set to be removed during the current assignment schedule).
• You can use the fields --cloud-owner and --cc-users to define who gets notifications.

This is a message to alert you that in 7 days
your allocated environment:

cloud08 : 29 (JBOSS Data Grid)

(Details)
http://wiki.example.com/assignments/#cloud08

will have some or all of the hosts expire.  Some or all of your
hosts will automatically be reprovisioned and returned to the
machine pool.

b01-h05-r620.example.com
b01-h06-r620.example.com
b02-h01-r620.example.com

QUADS Switch and Host Setup

• To ensure you have setup your switch properly please follow our Switch and Host Setup Docs
• We will not be covering the Wiki component setup, but you can use our Wordpress/nginx/php-fm/mariadb Ansible playbooks to set this up for you.
  • This will be containerized and documented in the near future via GitHub Issue #102

Installing QUADS

• We offer both RPM packages or a Git clone installation (for non RPM-based distributions, BSD UNIX, etc).
• It's recommended to use RPM as it requires less setup and upgrades are tied into your distribution package system.

Installing QUADS from Github

• Clone the git repository (substitute paths below as needed)

git clone https://github.com/redhat-performance/quads /opt/quads

• Install pre-requisite Python packages

yum install PyYAML ansible expectk python2-aexpect python-requests

• Install a webserver (Apache, nginx, etc)

yum install httpd

• Create logging directory (you can edit this in conf/quads.yml via the log: parameter).

mkdir -p /opt/quads/log

• Create your visualization web directory (you can configure this in conf/quads.yml via visual_web_dir)

mkdir -p /var/www/html/visual

• Populate the web visualization images in your webserver directory

cp -p /opt/quads/images/{button*,texture*}.png /var/www/html/visual/

• • Read through the QUADS YAML configuration file for other settings you way want.

vi /opt/quads/conf/quads.yml

• Enable and start the QUADS systemd service (daemon)
• Note: You can change QUADS quads_base_url listening port in conf/quads.yml and use the --port option

cp /opt/quads/systemd/quads-daemon.service /etc/systemd/system/quads-daemon.service
systemctl daemon-reload
systemctl enable quads-daemon.service
systemctl start quads-daemon.service

• Note: You can use QUADS on non-systemd based Linux or UNIX distributions but you'll need to run /opt/quads/bin/quads-daemon via an alternative init process or similiar (It's just a Python HTTP application).

Installing QUADS from RPM

• On Red Hat or CentOS 7+

yum install yum-utils -y
yum-config-manager --add-repo https://copr.fedorainfracloud.org/coprs/quadsdev/QUADS/repo/epel-7/quadsdev-QUADS-epel-7.repo
yum-config-manager --enable quadsdev-QUADS-epel-7
yum install quads -y

• On Fedora

dnf copr enable quadsdev/QUADS  -y
dnf install quads -y

• Read through the QUADS YAML configuration file

vi /opt/quads/conf/quads.yml

• Enable and start the QUADS systemd service (daemon)
• Note: You can change QUADS quads_base_url listening port in conf/quads.yml and use the --port option

systemctl enable quads-daemon.service
systemctl start quads-daemon.service

• Note: RPM installations will have quads-cli and tools in your system $PATH but you will need to login to a new shell to pick it up.

QUADS Usage Documentation

• Define the various cloud environments
• These are the isolated environments QUADS will use and provision into for you.

/opt/quads/bin/quads-cli --define-cloud cloud01 --description "Primary Cloud Environment"
/opt/quads/bin/quads-cli --define-cloud cloud02 --description "02 Cloud Environment"
/opt/quads/bin/quads-cli --define-cloud cloud03 --description "03 Cloud Environment"

• Define the hosts in the environment (Foreman Example)
  • Note the --host-type parameter, this is a mandatory, free-form label that can be anything. It will be used later for post-config automation and categorization.
  • We are excluding anything starting with mgmt- and including servers with the name r630.

for h in $(hammer host list --per-page 1000 | egrep -v "mgmt|c08-h30"| grep r630 | awk '{ print $3 }') ; do /opt/quads/bin/quads-cli --define-host $h --default-cloud cloud01 --host-type general; done

• The command without Foreman would be simply:

/opt/quads/bin/quads-cli --define-host <hostname> --default-cloud cloud01 --host-type general

• To list the hosts:

/opt/quads/bin/quads-cli --ls-hosts

You will now see the list of full hosts.

c08-h21-r630.example.com
c08-h22-r630.example.com
c08-h23-r630.example.com
c08-h24-r630.example.com
c08-h25-r630.example.com
c08-h26-r630.example.com
c08-h27-r630.example.com
c08-h28-r630.example.com
c08-h29-r630.example.com
c09-h01-r630.example.com
c09-h02-r630.example.com
c09-h03-r630.example.com

• To see the current system allocations:

/opt/quads/bin/quads-cli --summary

cloud01 : 45 (Primary Cloud Environment)
cloud02 : 0 (02 Cloud Environment)
cloud03 : 0 (03 Cloud Environment)

• Define a custom schedule for a host
  • Example: assign host c08-h21 to the workload/cloud cloud02

/opt/quads/bin/quads-cli --add-schedule --host c08-h21-r630.example.com --schedule-start "2016-07-11 08:00" --schedule-end "2016-07-12 08:00" --schedule-cloud cloud02

• List the schedule for a specific host:

/opt/quads/bin/quads-cli --ls-schedule --host c08-h21-r630.example.com

You'll see the schedule output below

Default cloud: cloud01
Current cloud: cloud02
Defined schedules:
  0:
    start: 2016-07-11 08:00
    end: 2016-07-12 08:00
    cloud: cloud02

• Move any hosts that need to be re-allocated based on the current schedule

/opt/quads/bin/quads-cli --move-hosts

You should see the following verbosity from a move operation

INFO: Moving c08-h21-r630.example.com from cloud01 to cloud02 c08-h21-r630.example.com cloud01 cloud02

In the above example the default move command was called /bin/echo for illustration purposes. In order for this to do something more meaningful you should invoke the script with the --move-command option, which should be the path to a valid command. The script takes three arguments (hostname, current cloud, new cloud).

• Move a host using --move-command
  • You can append a script, command or other action as a post-hook (perhaps to fire off system provisioning).

/opt/quads/bin/quads-cli --move-hosts --path-to-command /usr/bin/movecommand.sh

Common Administration Tasks

Creating a New Cloud Assignment and Schedule

Creating a new schedule and assigning machines is currently done through the QUADS CLI. There are a few options you'll want to utilize. Mandatory options are in bold and optional are in italics.

• description (this will appear on the assignments dynamic wiki)
• cloud-owner (for associating ownership and usage notifications)
• force (needed for re-using an existing cloud)
• cc-users (Add additional people to the notifications)
• cloud-ticket (RT ticket used for the work, also appears in the assignments dynamic wiki)
• nowipe (do not reprovision machines going into this cloud, default is 0)

QUADS VLAN Options

• VLAN design (optional, will default to 0 below) - qinq: 0 (default) qinq VLAN separation by interface: primary, secondary and beyond QUADS-managed interfaces all match the same VLAN membership across other hosts in the same cloud allocation. Each interface per host is in its own VLAN, and these match across the rest of your allocated hosts by interface (all nic1, all nic2, all nic3, all nic4 etc). - qinq: 1 all QUADS-managed interfaces in the same qinq VLAN

Defining a New Cloud

/opt/quads/bin/quads-cli --define-cloud cloud03 --description "Messaging AMQ" --force --cloud-owner epresley --cc-users "jdoe jhoffa" --cloud-ticket 423625 --qinq 0

• Now that you've defined your new cloud you'll want to allocate machines and a schedule.
  • We're going to find the first 20 Dell r620's and assign them as an example.

Adding New Hosts to your Cloud

/opt/quads/bin/quads-cli --cloud-only cloud01 | grep r620 | head -20 > /tmp/RT423624

• Now we'll allocate all of these hosts with a schedule, by default our system times use UTC.

for h in $(cat /tmp/RT423624) ; do /opt/quads/bin/quads-cli --host $h --add-schedule --schedule-start "2016-10-17 00:00" --schedule-end "2016-11-14 17:00" --schedule-cloud cloud03 ; done

That's it. At this point your hosts will be queued for provision and move operations, we check once a minute if there are any pending provisioning tasks. To check manually:

for h in $(./quads-cli  --cloud-only cloud03) ; do echo -n ==== $h   :" " ; cat /etc/lab/state/$h ; done

After your hosts are provisioned and moved you should see them populate under the cloud list.

/opt/quads/bin/quads-cli --cloud-only cloud03

Extending the Schedule of an Existing Cloud

Occasionally you'll want to extend the lifetime of a particular assignment. QUADS lets you do this with one command but you'll want to double-check things first. In this example we'll be extending the assignment end date for cloud03

• First, get the updated list of current assignments

/opt/quads/bin/quads-cli --summary

cloud01 : 55 (Pool of available servers)
cloud02 : 12 (Small OSPD deployment)
cloud03 : 20 (Messaging - AMQ - dispatch router and artemis broker)
cloud04 : 60 (Ceph deployment)
cloud07 : 10 (Small OSPD deployment)
cloud09 : 5 (Keystone OSPD deployment)
cloud10 : 14 (Openshift + OSPD testing)

• Next, List the owners of the clouds.

/opt/quads/bin/quads-cli --ls-owner

cloud01 : nobody
cloud02 : bjohnson
cloud03 : jhoffa
cloud04 : ltorvalds
cloud05 : nobody
cloud06 : nobody
cloud07 : dtrump
cloud08 : nobody
cloud09 : dtrump
cloud10 : cnorris

• Lastly, obtain a list of the current machines in cloud03

/opt/quads/bin/quads-cli --cloud-only cloud03

b09-h01-r620.rdu.openstack.example.com
b09-h02-r620.rdu.openstack.example.com
b09-h03-r620.rdu.openstack.example.com
b09-h05-r620.rdu.openstack.example.com
b09-h06-r620.rdu.openstack.example.com
b09-h07-r620.rdu.openstack.example.com
b09-h09-r620.rdu.openstack.example.com
b09-h11-r620.rdu.openstack.example.com
b09-h14-r620.rdu.openstack.example.com
b09-h15-r620.rdu.openstack.example.com
b09-h17-r620.rdu.openstack.example.com
b09-h18-r620.rdu.openstack.example.com
b09-h19-r620.rdu.openstack.example.com

• Take a look at the existing schedule for one of these machines, you'll see it expires 2016-10-30.

/opt/quads/bin/quads-cli --host b09-h01-r620.rdu.openstack.example.com --ls-schedule

Default cloud: cloud01
Current cloud: cloud03
Current schedule: 0
Defined schedules:
  0| start=2016-10-17 00:00,end=2016-10-30 18:00,cloud=cloud03

• Extend the --schedule-end date for the Cloud

If you are sure you've got the right cloud assignment from above you can proceed This is the actual command that extends the schedule, the other commands above are more for your verification. Below we will be extending the schedule end date from 2016-10-30 to 2016-11-27 at 18:00

for h in $(/opt/quads/bin/quads-cli --cloud-only cloud03) ; do /opt/quads/bin/quads-cli --host $h --mod-schedule 0 --schedule-end "2016-11-27 18:00"; done

• Cleanup Notification Files

Your tenant may have already been receiving email notifications about machines coming up for reclamation, we want to clear these out so future notifications are up to date. On the QUADS host you'll want to remove these files if they exist, in this case they will be called cloud03-jhoffa-$notifyday-$ticketid

rm: remove regular empty file '/etc/lab/report/cloud03-jhoffa-5-423624'? y
rm: remove regular empty file '/etc/lab/report/cloud03-jhoffa-7-423624'? y

Extending the Schedule of Existing Cloud with Differing Active Schedules

When in heavy usage some machines primary, active schedule may differ from one another, e.g. 0 versus 1, versus 2, etc. Because schedules operate on a per-host basis sometimes the same schedule used within a cloud may differ in schedule number. Here's how you modify them across the board for the current active schedule if the ID differs.

• Example: extend all machines in cloud10 to end on 2016-01-09 05:00 UTC, these have differing primary active schedule IDs.

  • Check your commands via echo first
  • Approach: 1 Modify the latest assignment

for h in $(/opt/quads/bin/quads-cli --cloud-only cloud10) ; do echo /opt/quads/bin/quads-cli --mod-schedule $(/opt/quads/bin/quads-cli --ls-schedule --host $h | grep "urrent s" | awk -F: '{ print $2 }') --host $h --schedule-end "2017-01-09 05:00" ; echo Done. ; done

Note the difference in commands needed with the --mod-schedule flag that is required.

/opt/quads/bin/quads-cli --mod-schedule 0 --host b10-h11-r620.rdu.openstack.example.com --schedule-end 2017-01-09 05:00
Done.
/opt/quads/bin/quads-cli --mod-schedule 3 --host c08-h21-r630.rdu.openstack.example.com --schedule-end 2017-01-09 05:00
Done.
/opt/quads/bin/quads-cli --mod-schedule 3 --host c08-h22-r630.rdu.openstack.example.com --schedule-end 2017-01-09 05:00
Done.
/opt/quads/bin/quads-cli --mod-schedule 2 --host c08-h23-r630.rdu.openstack.example.com --schedule-end 2017-01-09 05:00

• Approach: 2 Reschedule against a certain cloud and start date (RECOMMENDED)

for h in $(/opt/quads/bin/quads-cli --cloud-only cloud05); do echo /opt/quads/bin/quads-cli --mod-schedule $(/opt/quads/bin/quads-cli --ls-schedule --host $h | grep cloud05 | grep "start=2017-02-09" | tail -1 | awk -F\| '{ print $1 }') --host $h --schedule-start "2017-02-09 05:00" --schedule-end "2017-03-06 05:00" ; echo Done. ; done

• If all looks good you can remove remove the echo lines and apply.

Extending Machine Allocation to an existing Cloud

QUADS also supports adding new machines into an existing workload (cloud).

• Search Availability Pool for Free Servers
  • Let's look for any 5 x servers for 10 days

bin/find-available.py -c 5 -d 10

================
First available date = 2016-12-05 08:00
Requested end date = 2016-12-15 08:00
hostnames =
c03-h11-r620.rdu.openstack.example.com
c03-h13-r620.rdu.openstack.example.com
c03-h14-r620.rdu.openstack.example.com
c03-h15-r620.rdu.openstack.example.com
c03-h17-r620.rdu.openstack.example.com

• Move New Hosts into Existing Cloud

Above we see all the free servers during our timeframe, let's move them into cloud10

/opt/quads/bin/quads-cli --host c03-h11-r620.rdu.openstack.example.com --add-schedule --schedule-start "2016-12-05 08:00" --schedule-end "2016-12-15 08:00" --schedule-cloud cloud10
/opt/quads/bin/quads-cli --host c03-h13-r620.rdu.openstack.example.com --add-schedule --schedule-start "2016-12-05 08:00" --schedule-end "2016-12-15 08:00" --schedule-cloud cloud10
/opt/quads/bin/quads-cli --host c03-h14-r620.rdu.openstack.example.com --add-schedule --schedule-start "2016-12-05 08:00" --schedule-end "2016-12-15 08:00" --schedule-cloud cloud10
/opt/quads/bin/quads-cli --host c03-h15-r620.rdu.openstack.example.com --add-schedule --schedule-start "2016-12-05 08:00" --schedule-end "2016-12-15 08:00" --schedule-cloud cloud10
/opt/quads/bin/quads-cli --host c03-h17-r620.rdu.openstack.example.com --add-schedule --schedule-start "2016-12-05 08:00" --schedule-end "2016-12-15 08:00" --schedule-cloud cloud10

• Note: You can run bin/find-available-py with the --cli flag to generate QUADS commands for you.

Removing a Schedule

You can remove an existing schedule across a set of hosts using the --rm-schedule flag against the schedule ID for each particular machine of that assignment.

• Example: removing the schedule for three machines in cloud
• Obtain the schedule ID via /opt/quads/bin/quads-cli --ls-schedule --host
• These machines would happen to have the same cloud assignment as schedule id 2.

/opt/quads/bin/quads-cli --rm-schedule 2 --host c08-h01-r930.rdu.openstack.example.com
/opt/quads/bin/quads-cli --rm-schedule 2 --host c08-h01-r930.rdu.openstack.example.com
/opt/quads/bin/quads-cli --rm-schedule 2 --host c08-h01-r930.rdu.openstack.example.com

Removing a Schedule across a large set of hosts

You should search for either the start or end dates to select the right schedule ID to remove when performing schedule removals across a large set of hosts.

• If you are using QUADS in any serious capacity always pick this option.
• Example: removing schedule by searching for start date.
• Often machine schedule ID's are different for the same schedule across a set of machines, this ensures you remove the right one.

for host in $(cat /tmp/452851); do /opt/quads/bin/quads-cli --rm-schedule $(/opt/quads/bin/quads-cli --ls-schedule --host $host | grep cloud08 | grep "start=2017-08-06" | tail -1 | awk -F\| '{ print $1 }') --host $host ; echo Done. ; done

Using the QUADS JSON API

• We've recently introduced a JSON API into QUADS comprised of a systemd service quads-daemon and a quads-cli
• This is an optional local systemd service you can start and interact with and listens on localhost TCP/8080

cp systemd/quads-daemon.service /etc/systemd/system/quads-daemon.service
systemctl enable quads-daemon.service
systemctl start quads-daemon.service

• All of the argparse and normal QUADS sub-commands are supported and will accept http GET and POST actions in a JSON response body.
  • Example: getting the equivalent of quads --ls-hosts via curl

curl -X GET -H 'Content-Type: application/json' http://127.0.0.1:8080/api/v1/lshosts

You'll then see a JSON response back.

{"hosts": ["b08-h13-r620.rdu.openstack.engineering.example.com", "b08-h14-r620.rdu.openstack.engineering.example.com", "b08-h15-r620.rdu.openstack.engineering.example.com", "b08-h17-r620.rdu.openstack.engineering.example.com", "b08-h18-r620.rdu.openstack.engineering.example.com", "b08-h19-r620.rdu.openstack.engineering.example.com", "b08-h21-r620.rdu.openstack.engineering.example.com"]}

API GET Operations

• The following commands can be queried via curl or some other http mechanism to do basic metadata queries:
  • curl -X GET -H 'Content-Type: application/json' http://127.0.0.1:8080
    • /api/v1/lshosts Obtain a list of hosts managed by QUADS
    • /api/v1/lsclouds Obtain list of cloud assignments
    • /api/v1/lsowner Retrieve a list of current system assignment owners
    • /api/v1/lsccusers List the cc-users associated for assignments
    • /api/v1/lstickets Obtain the ticket numbers to assignment mappings
    • /api/v1/lsqinq List the qinq VLAN mode (0|1) per cloud assignment

API POST Operations

• The following construct can be used via http POST to receive more detailed data by providing granular criteria to return JSON body data:

  • You can combine one of many POST query types with multiple POST metadata objects.

  • There is limited support for data modification via POST as well documented below.

  • Valid POST URI queries

    • /api/v1/lsowner
    • /api/v1/lsccusers
    • /api/v1/lstickets
    • /api/v1/lsqinq
    • /api/v1/host Can also be used for defining/updating a host.
    • /api/v1/cloud Can also be used for defining/updating a cloud assignment.
    • /api/v1/ahs AKA add host schedule used for adding a new host schedule.
    • /api/v1/rhs AKA remove host schedule used for removing a host schedule.
    • /api/v1/mhs AKA modify host schedule used for modifying a host schedule.
    • /api/v1/moves Reports what would be moved on a given date.

  • Valid POST object filters:

    • -d cloud=cloud0X
    • -d cloudonly=cloud0X
    • -d 'date=2018-08-08 22:00'
    • -d statedir=/path/to/quads/datadir
    • -d fullsummary=True/False
    • -d host=c10-h33-r630.rdu.openstack.engineering.example.com

• Constructing an http POST example with multiple metadata objects:

curl -X POST -H 'Content-Type: application/json' -d quadsvariable=value -d quadsvariable=value http://127.0.0.1:8080/api/v1/object

Working Examples:

• Query the owners of cloud02 only

curl -X POST -H 'Content-Type: application/json' -d cloudonly=cloud02  http://127.0.0.1:8080/api/v1/lsowners ; echo

{"owner": ["epresley"]}

• Use the Query Object to Obtain Details

curl -X POST http://localhost:8080/api/v1/query -d host=c01-h01-r620.rdu.openstack.engineering.example.com -d lsschedule=True -H 'Content-Type: application/json'

{"result": ["Default cloud: cloud01", "Current cloud: cloud01", "Defined schedules:", "  0| start=2016-10-17 00:00,end=2016-12-21 17:00,cloud=cloud04", "  1| start=2016-12-21 17:00,end=2017-0
08", "  2| start=2017-02-09 05:00,end=2017-03-06 05:00,cloud=cloud02", "  3| start=2017-03-20 05:00,end=2017-04-03 05:00,cloud=cloud03", "  4| start=2017-02-03 19:00,end=2017-02-09 05:00,clou
t=2017-03-06 05:00,end=2017-03-20 05:00,cloud=cloud16", "  6| start=2017-04-03 05:00,end=2017-05-01 05:00,cloud=cloud02", "  7| start=2017-06-18 22:00,end=2017-07-02 22:00,cloud=cloud05", "
:00,end=2017-05-28 22:00,cloud=cloud07"]}

• Query all of the future move actions on the day and time 2018-01-01 22:00

curl -X POST -H 'Content-Type: application/json' -d 'date=2018-01-01 22:00' -d statedir=/opt/quads/data http://127.0.0.1:8080/api/v1/moves ; echo

{"result": [{"current": "cloud14", "new": "cloud01", "host": "b08-h13-r620.rdu.openstack.example.com"}, {"current": "cloud14", "new": "cloud01"}]}

More Examples with API POST

• Define a Host via API POST

curl -X POST http://localhost:8080/api/v1/host -d host=c10-h33-r630.rdu.openstack.example.com -d cloud=cloud01 -d force=False -H 'Content-Type: application/json'

• Add a new Cloud Assignment via API POST

curl -X POST http://localhost:8080/api/v1/cloud -d cloud=cloud03 -d description='Some project' -H 'Content-Type: application/json'

• Modify a Cloud Assignment via API POST

curl -X POST http://localhost:8080/api/v1/cloud -d cloud=cloud03 -d description='New Updated Description' -d force=True -H 'Content-Type: application/json'

• Add a new Host Schedule via API POST

curl -X POST http://localhost:8080/api/v1/ahs -d host=c01-h01-r620.rdu.openstack.example.com -d 'start=2017-09-01 22:00' -d 'end=2017-09-30 22:00' -d 'cloud=cloud04' -H 'Content-Type: application/json'

• Modify a Host Schedule via API POST
  • At least one of start= end= or cloud= are required with modifications.

curl -X POST http://localhost:8080/api/v1/mhs -d host=c01-h01-r620.rdu.openstack.example.com -d 'start=2017-09-01 22:00' -d 'end=2017-09-30 22:00' -d 'cloud=cloud04' -H 'Content-  Type: application/json'

• Remove a Host Schedule via API POST
  • schedule= is the numeric value of the target schedule, you can use the query object to determine this (or --ls-schedule via cli or /opt/quads/bin/quads-cli

curl -X POST http://localhost:8080/api/v1/rhs -d host=c01-h01-r620.rdu.openstack.example.com -d schedule=1

• Using quads-cli
  • /opt/quads/bin/quads-cli is a front-end to the bin/quads-daemon JSON API.
  • /opt/quads/bin/quads-cli interacts with /opt/quads/bin/quads-daemon exactly like the normal ```/opt/quads/bin/quads-cli`` so you can utilize the same documentation above.

Additional Tools and Commands

• You can display the allocation schedule on any given date via the --date flag.

/opt/quads/bin/quads-cli --date "2017-03-06"

cloud01:
  - b09-h01-r620.rdu.openstack.engineering.example.com
  - b09-h02-r620.rdu.openstack.engineering.example.com
  - b09-h03-r620.rdu.openstack.engineering.example.com
  - b09-h05-r620.rdu.openstack.engineering.example.com
  - b09-h06-r620.rdu.openstack.engineering.example.com

• You can use find-available.py to search for free machines for a timerange for allocation.
  • Use the optional -l option to filter results

/opt/quads/bin/find-available.py -c 5 -d 10

================
First available date = 2016-12-05 08:00
Requested end date = 2016-12-15 08:00
hostnames =
c03-h11-r620.rdu.openstack.example.com
c03-h13-r620.rdu.openstack.example.com
c03-h14-r620.rdu.openstack.example.com
c03-h15-r620.rdu.openstack.example.com
c03-h17-r620.rdu.openstack.example.com

• You can use find-free-cloud.py to search for available clouds to assign new systems into for future assignments.

/opt/quads/bin/find-free-cloud.py

• You can see what's in progress or set to provision via the --dry-run sub-flag of --move-hosts

/opt/quads/bin/quads-cli --move-hosts --dry-run

INFO: Moving b10-h27-r620.rdu.openstack.example.com from cloud01 to cloud03
INFO: Moving c02-h18-r620.rdu.openstack.example.com from cloud01 to cloud03
INFO: Moving c02-h19-r620.rdu.openstack.example.com from cloud01 to cloud03
INFO: Moving c02-h21-r620.rdu.openstack.example.com from cloud01 to cloud03
INFO: Moving c02-h25-r620.rdu.openstack.example.com from cloud01 to cloud03
INFO: Moving c02-h26-r620.rdu.openstack.example.com from cloud01 to cloud03

• You can query all upcoming changes pending and what hosts are involved via the quads-change-list.sh tool.

/opt/quads/bin/quads-change-list.sh

Next change in 3 days
2016-12-22 05:00
INFO: Moving c01-h01-r620.rdu.openstack.example.com from cloud04 to cloud08
INFO: Moving c01-h02-r620.rdu.openstack.example.com from cloud04 to cloud08
INFO: Moving c01-h03-r620.rdu.openstack.example.com from cloud04 to cloud08
INFO: Moving c01-h05-r620.rdu.openstack.example.com from cloud04 to cloud08
INFO: Moving c01-h06-r620.rdu.openstack.example.com from cloud04 to cloud08

• When managing notification recipients you can use the --ls-cc-users and --cc-users arguments.

/opt/quads/bin/quads-cli --ls-cc-users --cloud-only cloud04

epresley

• To add or remove recipients they need to be added or removed with space separation and you'll need to redefine the cloud definition.
• Get a list of all the atributes and redefine

/opt/quads/bin/quads-cli --full-summary | grep cloud04 ; /opt/quads/bin/quads-cli --ls-owner | grep cloud04 ; /opt/quads/bin/quads-cli --ls-ticket | grep cloud04 ; /opt/quads/bin/quads-cli --ls-cc-users --cloud-only cloud04

cloud04 : 52 (Ceph deployment)
cloud04 : jhoffa
cloud04 : 423424
epresley

• Redefine

/opt/quads/bin/quads-cli --define-cloud cloud04 --description "Ceph Deployment" --force --cloud-owner jhoffa --cc-users "epresley rnixon" --cloud-ticket 423424

• Now you can see the updated cc notifications

/opt/quads/bin/quads-cli --ls-cc-users --cloud-only cloud04

epresley
rnixon

• We have Jenkins CI run against all Gerrit patchsets via the QUADS Simulator 5000 CI test script.

Contributing

• You can use the testing/quads-sandbox.sh tool to create a local sandbox for testing and development.
• We use Gerrit for code review, to submit a patch perform the following:
• You can also find us on IRC at #quads on irc.freenode.net

• Clone our repository:

git clone https://github.com/redhat-performance/quads

• Create a Github issue to track your work.

  • Provide a meaningful explanation, citing code lines when relevant.
  • Explain what you are trying to fix, or what you're trying to contribute.

• Setup username/email for git and gerrithub (one time only):

  • Ensure Github and Gerrithub are linked by signing into Gerrithub via Github
  • match gitreview.username to your Github username
  • match user.name to your real name or how you want credit for commits to display in Git history.
  • match user.email to your email address associated with Github.

git config --global user.email "[email protected]"
git config --global user.name "Venril Sathir"
git config --global --add gitreview.username "vsathir"

• Make your changes

cd quads
vi lib/Quads.py

• Add a local commit with a meaningful, short title followed by a space and a summary (you can check our commit history for examples).
• Add a line that relates to a new or existing github issue, e.g. fixes: https://github.com/redhat-performance/quads/issues/5 or related-to: https://github.com/redhat-performance/quads/issues/25

git add lib/Quads.py
git commit

• Install git-review and run it for first time.

yum install git-review
git review -s

• Now submit your patchset with git review (future patches you only need to run git review)
  • A Change-ID will be generated when you create your first patchset, make sure this is the last line in the commit message preceded by an empty line.

git review

• If you want to make changes to your patchset you can run the git commit --amend command.

vi lib/Quads.py
git commit --amend
git review

Jenkins CI currently checks the following for every submitted patchset:

• shellcheck - checks for common shell syntax errors and issues
• flake8 - checks Python tools for common syntax errors and issues
• quads sandbox test - instantiates and runs common QUADS operations with fake data
  • This is all run from testing/test-quads.sh
  • We currently do not expose CI logs externally, please reply on your patchset comments if you'd like a paste of it.

QUADS Talks and Media

• Skynet your Infrastructure with QUADS @ Europython 2017 Slides
• Skynet your Infrastructure with QUADS @ DevOps Pro Moscow 2018 Slides