Skip to content

Latest commit

 

History

History
277 lines (195 loc) · 9.69 KB

install.rst

File metadata and controls

277 lines (195 loc) · 9.69 KB

Installation Instructions for production environments

Installation using the Python package manager pip

Make sure you have the pip Python package manager installed on your Python 3 environment, and the C library for YAML development. On Debian/Ubuntu, the easiest way to do that is:

apt-get install python3-pip libyaml-dev

Install squad:

pip3 install squad

By default, SQUAD works with sqlite, but if Postgres is required, then specific binaries are needed:

apt-get install python3-pip libyaml-dev libpq-dev

Install squad with Postgres support:

pip3 install squad[postgres]

Message broker

In order to SQUAD processes to be able to communicate between each other, you need to install an AMQP server. We recommend RabbitMQ:

apt-get install rabbitmq-server

By default SQUAD will look for an AMQP server running on localhost, listening to the standard port, so for a single-server deployment you don't need to do anything else.

If you have a multi-server setup, then each server needs to be configured with the location of a central AMQP server. See the SQUAD_CELERY_BROKER_URL in the "Further configuration" section below.

Processes

SQUAD is composed of 4 different process:

  • web application server
  • background worker (celery worker)
  • periodic task scheduler (celery beat)
  • CI backend listener

To run the web interface, run as a dedicated user (i.e. don't run the application as root!):

squad
  • Note: if that doesn't work, ~/.local/bin is probably missing in the $PATH environment variable.

This will make the web UI available at http://localhost:8000/. To serve the UI to external users, you will need to setup a public-facing web server such as Apache or nginx and reverse proxy to localhost on port 8000. You can change the address or port SQUAD listens to by passing the --bind command line option, e.g. to make it listen to port 5000 on the local loopback interface, use:

squad --bind 127.0.0.0:5000

For production usage, you will want to tweak at least the database configuration. Keep reading for more information.

After starting SQUAD, but before acessing it, you will need a user. To create an admin user for yourself, use:

squad-admin createsuperuser

These are the command lines to run the other processes:

Process Command
worker celery -A squad worker
scheduler celery -A squad beat
listener squad-admin listen

You most probably want all the processes (including the web interface) being managed by a system manager such as systemd, or a process manager such as supervisor.

For an example deployment, check the configuration management repository for Linaro's qa-reports (using ansible).

After having the necessary processes running, there are a few extra setup steps needed:

  • Create Backend instances for your test execution backends. Go to the administration web UI, and under "CI", choose "Backends".
  • For each project, create authentication tokens and subscriptions

Worker configuration

The worker process handles background tasks, such as submitting CI jobs, fetching the results fo CI jobs, preparing reports that require intensive processing, etc. Some tasks take a lot longer than the others, e.g. submitting a CI job is pretty quick, while fetching CI job results takes some time to fetch all the data, parse it, and store in the database.

To allow for better load balancing, these tasks are split into multiple queues:

  • ci_fetch
  • ci_poll
  • ci_quick
  • core_notification
  • core_postprocess
  • core_quick
  • core_reporting

ci_fetch and ci_poll can be potentially slow, and if there is a large influx of those types of tasks, your system may display some congestion because all available worker threads are occupied running slow tasks, while several of the quick ones are waiting their turn.

To avoid this, you might want to start a small part of your workers so that they will not pick up any of those slow tasks:

squad worker --exclude-queues ci_fetch,ci_poll

Or you can also give an explicit task list, which is less flexible but might be useful:

squad worker --queues core_quick,ci_quick

By default, workers listen to all queues.

For message brokers that support prefixed-queue names, SQUAD has the optional environment variable SQUAD_CELERY_QUEUE_NAME_PREFIX, that, if set, will prepend it before all queue names. SQUAD also support adding a suffix via the optional environment variable SQUAD_CELERY_QUEUE_NAME_SUFFIX

Further configuration

The following environment variables affect the behavior of SQUAD:

  • DATABASE: controls the database connection parameters. Should contain KEY=VALUE pairs, separated by colons (:).

    By default, SQUAD will use a SQLite3 database in its internal data directory.

    For example, to use a PostgreSQL database (requires the psycopg2 Python package to be installed):

    DATABASE=ENGINE=django.db.backends.postgresql_psycopg2:NAME=mydatabase:USER=myuser:HOST=myserver:PASSWORD=mypassword

  • SQUAD_EXTRA_SETTINGS: path to a Python file with extra Django settings.

  • SQUAD_SITE_NAME: name to be displayed at the page title and navigation bar. Defaults to 'SQUAD'.

  • XDG_DATA_HOME: parent directory of the SQUAD internal data directory. Defaults to ~/.local/share. The actual data directory will be ${XDG_DATA_HOME}/squad.

  • SECRET_KEY_FILE: file to store encryption key for user sessions. Defaults to ${XDG_DATA_HOME}/squad/secret.dat

  • DJANGO_LOG_LEVEL: the logging level used for Django-related logging. Default: INFO.

  • SQUAD_LOG_LEVEL: the logging level for SQUAD-specific logging. Default: INFO.

  • SQUAD_HOSTNAME: hostname used to compose links in asynchronous notifications (e.g. emails). Defaults to the FQDN of the host where SQUAD is running.

  • SQUAD_BASE_URL: Base URL to the SQUAD web interface, used when composing links in notifications (e.g. emails). Defaults to https://$SQUAD_HOSTNAME.

  • SQUAD_EMAIL_FROM: e-mail used as sender of email notifications. Defaults to noreply@$SQUAD_HOSTNAME.

  • SQUAD_EMAIL_HOST: hostname to use as e-mail delivery host. Sets Django's EMAIL_HOST setting. See the Django documentation on sending email for more details.

  • SQUAD_LOGIN_MESSAGE: a message to be displayed to users right above the login form. Use for example to provide instructions on what credentials to use. Defaults no message.
  • SQUAD_ADMINS: Comma-separated list of administrator email addresses, for use in exception notifications. Each address must be formatted as First Last <[email protected]>.
  • SQUAD_SEND_ADMIN_ERROR_EMAIL: Determines whether or not to send exception notifications to administrators. Defaults to True.
  • SENTRY_DSN: Defines Sentry's DSN token, if defined SQUAD will attempt to import Sentry SDK and use it. Defaults to None. If Sentry is configured it's recommended to disable sending admin notifications by setting SQUAD_SEND_ADMIN_ERROR_EMAIL = False.
  • SQUAD_STATIC_DIR: Directory where SQUAD will find it's preprocessed static assets. This usually does not need to be set manually, and exists mostly for use in the Docker image.
  • SQUAD_CELERY_BROKER_URL: URL to the broker to be used by Celery for background jobs. Defaults to amqp://localhost:5672.
  • SQUAD_CELERY_QUEUE_NAME_PREFIX: Name to prefix all queues in Celery. Useful when multiple environments share the same broker. Defaults to ''.
  • SQUAD_CELERY_QUEUE_NAME_SUFFIX: Name to concatenate all queues in Celery. Useful when a queue extension is needed by the broker. Defaults to ''.
  • SQUAD_CELERY_POLL_INTERVAL: Number of seconds a worker will sleep after an empty answer from SQS before the next polling attempt. Defaults to 1.

User management

SQUAD provides 'users' management command that allows to list, add, update and display details about users. This command comes handy when trying to automate SQUAD setup with containers. Details about user management with 'users' command:

  • list Displays list of all available users with their names (first, last) from database
  • details <username> Displays details about requested username. Details include:
    • username
    • is_active
    • is_staff
    • is_superuser
    • groups
  • add <username> Adds new user with given 'username'. It also takes additional parameters
    • --email EMAIL email of the user
    • --passwd PASSWD Password for this user. If empty, a random password is generated.
    • --staff Make this user a staff member
    • --superuser Make this user a super user
  • update <username> Updates database record of existing user identified with 'username'. It takes additional parameters
    • --email EMAIL Change email of the user
    • --active Make this user active
    • --not-active Make this user inactive
    • --staff Make this user a staff member
    • --not-staff Make this user no longer a staff member
    • --superuser Make this user a superuser
    • --not-superuser Make this user no longer a superuser