cdh-web QA deploy playbook

README.md

@@ -18,6 +18,10 @@ The overall structure of this repository can be broken down as follows:
   - Python virtual environment.
     - See `.python-version` for the recommended version of Python.
     - If you use `env` or `venv`, the `.gitignore` will exclude it.
+
+  -  Install required Ansible galaxy collections:
+    - `ansible-galaxy collection install community.general`
+
   - The CDH Ansible vault key. This can be referenced on the command line or better set as in the Bash session, i.e. `export ANSIBLE_VAULT_PASSWORD_FILE=/path/to/.passwd`
   - A GitHub [personal access token](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/) for any playbook that uses the `create_deployment` and `close_deployment` roles. You can set this in your Bash session as `ANSIBLE_GITHUB_TOKEN` or pass it on the command line as `-e github_token=`
   - The CDH deploy bot key. This can be added to ssh-agent or in `~/.ssh/config`. All production deploys must be on the campus network (including VPN) and proxy through the QA server to production, with an ssh config stanza that looks something like:
@@ -70,33 +74,6 @@ To revert to previous deploy run call the `revert_deploy` playbook with a `host_
 ansible-playbook -e host_group=mep_qa playbooks/revert_deploy.yml
 ```
 
-## Overrides
-
-There are two principal overrides that the roles involved in deployment have built-in. One is the override noted above for what Git reference should be used to deploy. This can be any hash, branch head, or tag that the Git repository knows about.
-
-You can also override any other arbitrary variable, but the other likely one is the `requirements` file. You may want to point to a `requirements.lock`, for example:
-```{bash}
-ansible-playbook -e requirements_type=lock playbooks/playbook.yml
-```
-
-You can also pass a list of arbitrary additions or updates to pip (except for git pinned requirements):
-```{bash}
-ansible-playbook -e pip_updates='django-autocomplete-light<3.3' playbooks/playbook.yml
-```
-
-If you need to do more than one requirement, you can pass references using JSON notation (which should also include your other `-e` vars)
-```{bash}
-ansible-playbook -e '{"pip_updates": ["pandas", "colorama"], "ref": "develop"}'
-```
-
-These will be automatically added (or updated) to the requirements for the application during its deployment.
-
-If you need to make major changes and do not wish to make a patch release for whatever reason, you can also entirely replace `requirements.(txt|lock)` with a local template:
-
-```{bash}
-ansible-playbook -e new_requirements=/path/to/local/template.txt playbooks/playbook.yml
-```
-
 ## Vault variables
 
 Variables kept in `group_vars/*/vault.yml` are sensitive configurations that should always be kept encrypted on commit. To edit them (in your system text editor):

group_vars/all/vars.yml

@@ -52,8 +52,6 @@ clone_root: "{{ home }}/repos"
 short_hash: '{{ repo_info.after[0:6] }}'
 # Path of the deploy directory. Generated from vars above and build_project_repo
 deploy: '{{ install_root }}/{{ version }}-{{ short_hash }}'
-# Database backup location
-db_backup: '/tmp/pre-{{ version }}-{{ short_hash }}-{{ inventory_hostname_short }}.sql'
 # python app version. Generated in build_project_repo
 version: '{{ python_app_version.stdout }}'
 # logging directory
@@ -79,3 +77,7 @@ requirements_type: 'txt'
 # set csp_enabled: true in project vars to enable
 csp_reportonly_uri: '{{ vault_csp_reportonly_uri}}'
 csp_enforce_uri: '{{ vault_csp_enforce_uri }}'
+
+# Postgres configuration
+postgres_version: 10
+postgres_host: lib-postgres3.princeton.edu

group_vars/all/vault.yml

@@ -1,22 +1,20 @@
 $ANSIBLE_VAULT;1.1;AES256
-33306666333033616233623263613838323065666131303237316131383933373533656538376664
-3036653936326238643338663831363862336666336531620a383338373735306638303535386330
-32626439633432366663633230313766383738323134353339366637393863643263333838636264
-3831623235366236370a333038383164343961386436623863656534636464656634326434373431
-61653739346238396632303132626439386664623538643833323866376537353633393536363962
-66656466666638313831393638353735323236313836363634646534353666383364373337306361
-38666664613866373334363832623536386136626533393436636537636563636333343736646535
-61353231303062383535333533376365353234373832363664363530613133313436613263313731
-39343139626138353939323863313462323139373064623934306538393239383834366561313530
-36353033376230646464383665323963343637323538386238303638313432643965316231656362
-38666535666135393733326137633261373463643830636366303333313361613061323437653066
-34313837323933373761643035633163323465623563346362333233373338356461666563333734
-35663332613030626164393734633335303064306139376561393866333438353933376637373336
-30353066303234643131303761333666643639653565303764383938336335303935386339326463
-61383861373939646638396433343430623739666462326534656561363732386232363534616232
-62303534383266353038323636373132353463613333313166313238643165346539643463303139
-64653764373263633765323530356234613834386535393531303165396566626533656131633836
-62393563663961346530313261393639313438623335616531663734346534616462343630623865
-33643634353830303065326239623738306535623265613664656361653737323464656163626337
-63396166643630636236653266313936623338653630663462613632343364663765616561316465
-65376230623066623663316436363566353031663530306238303533333534386566
+62323337393033643435393935656233376464393632653136303635393166373734383466646233
+3163373537383863383164306163336531353130373661310a393632373135363961386538386533
+35636165323838303666636163366666346437323733613437663737623138326638656332646165
+3866353533653964640a323064643761313230663932373536323531343039373830333563663935
+32366665643137373761616164393162343766636337333263303839343131663266393936663966
+64633336393833616435646462303864356332396633656564393434333362373031653666626331
+63363035356437316262643335323362643039636565333564303238383338663339653031366138
+30616431396562393039343831323730666438623531663535303031613530303338373563663462
+36623435656566316534363662663161646238336562393633383566333237366532663936613931
+61616230393234363638323531373565323830663738393833373862373864353561613365323564
+32636365343933333734353338633435613735396535643466366164323134626566643961636166
+64396331323336343562616231633935323961373766393066303537386532346664386537346631
+65646131613936633131646536376333363533626531343038663335366630383730313934623265
+62636463623361653733653239366235336638653439316539623634356365653030613538646330
+37303965306262346533383862386436613030623564343362363539353938376339636434643462
+63396162363431623063323931613061386136383130623633653039376637366434656661613431
+36333431656532616335623639623561666663306135366462656162343038396166653631646137
+64376231303430636162623932643362386532313363613063303438373662623262366434616566
+396538303838316432323439353230666532

group_vars/cdhweb/vars.yml

@@ -8,18 +8,15 @@
 repo: 'Princeton-CDH/cdh-web'
 # name of main django application
 django_app: cdhweb
+app_name: cdhweb
 # symlink for Apache vhost
-symlink: cdh-web
-# template path
-template_path: cdhweb
-# media_root settings
-media_root: /srv/www/media
-
-# Override default paths to use node v10
-path: '/opt/rh/rh-python35/root/usr/bin:/opt/rh/rh-nodejs10/root/usr/bin:{{ ansible_env.PATH }}'
-ld_library_path: '/opt/rh/rh-python35/root/usr/lib64:/opt/rh/rh-nodejs10/root/usr/lib64{% if ansible_env.LD_LIBRARY_PATH is defined %}:{{ ansible_env.LD_LIBRARY_PATH }}{% endif %}'
-python_path: '/opt/rh/rh-nodejs10/root/usr/lib/python2.7/site-packages{% if ansible_env.PYTHON_PATH is defined %}:{{ ansible_env.PYTHONPATH }}{% endif %}'
-
-# explicitly specify deploy user, since cdhweb playbook does not include install_app_config
-# and deploy user is required for crontab role
-deploy_user: "deploy"
+symlink: cdhweb
+# apache location
+apache_app_path: "/var/www/{{ app_name }}"
+# wsgi path
+wsgi_path: "{{ django_app }}/wsgi.py"
+# nodejs version
+node_version: "10"
+# django database backend
+db_backend: "postgresql"
+db_host: "{{ postgres_host }}"

group_vars/cdhweb_qa/vars.yml

@@ -5,13 +5,21 @@ group_name: cdhweb_qa
 # Git refspec to use, either default or passed by '-e ref=<refspec>
 gitref: '{{ ref | default("develop") }}'
 # Install root (the dir where the repo will be set up on remote)
-install_root: '{{ install_base }}/cdh-web'
-# Set permissions for local_settings
-project_user: apache
+install_root: "/srv/www/cdhweb"
 # email prefix for admin emails
-email_prefix: '[QA CDH Web] '
+email_prefix: "[QA CDH Web] "
 # allowed_hosts for Django
 allowed_hosts:
-    - 'test-web.cdh.princeton.edu'
-# media_root for QA
-media_root: '/srv/www/qa/cdh-web/media'
+    - "cdh-test-web.princeton.edu"
+    - "cdh-test-web1"
+    - "localhost"
+# use python 3.6 for now since it's on the target ubuntu VM
+python_version: "python3.6"
+# use PUL deploy user of conan
+deploy_user: "conan"
+# postgresql database info
+application_dbuser_name: "{{ vault_db_username }}"
+application_dbuser_password: "{{ vault_db_password }}"
+application_db_name: "{{ vault_db_name }}"
+# Database backup location
+db_backup_path: '/home/{{ deploy_user }}/pre-{{ version }}-{{ short_hash }}.sql'

group_vars/cdhweb_qa/vault.yml

@@ -1,19 +1,19 @@
 $ANSIBLE_VAULT;1.1;AES256
-38316662626439353462636461643037306366636362656663323866323661363034343739623931
-3433613463396633323630316161646161646464356333360a643133613535326263313363613437
-33366136366432633465356536336264663266316438303463623466373161393562393635326431
-6562373235346139650a383366623064376232313433393666383433383962613839313030656231
-30376137393365356434663164613538313661666335303530333563376432613236373864386263
-66616537306539653063366662613935383064323762653133336330306636613661363237326539
-66393763633162396461656638356634373635633364313539623139383133383732316565323163
-34356532633866666331613638396330333464646230313466316337633433333963343033623439
-36353538343137333261333237386361383965343037386164303632663263386262303533633264
-66333364313365643365396163373035383166333032666164653238366433363934653661343266
-36396330633363613365333836326539373730633839353362326537663735646439613266366534
-31333964313363383565653433646130313738626436326164356438393362383965313134313636
-65666532646531613461303632393539653538643433616138323764383061666164343337646333
-32666365663238643266653439646638396662623364623931376339383663303230623832346261
-31613965616163376430366339636566343236386539353961303838653538353564363965663961
-62636330396130643762333337313335363861353066666534393930663261623935393865653836
-37383730383465393030373135373630646364386633373034346434393765653132333765323932
-6536623430353932323562393937643036616337363734343735
+34623538393861396638666466633133626662376234383935346130626465623538343534326135
+6634653264336139623432376230396238336335653231630a616563393062363332393731326638
+36396439393364333637646139326461363965306530356337346162336164313638373235623834
+3639333132643062330a316161396165343364633964663561383331323665323264656637643239
+66396166383465626632323736383566656338383935616435323831383462653930306161303666
+66393430343636623961316337623964353866656464373666616234316664653561336439393065
+62346361323736626535643163656638616233316530633038396565356139323166656665323465
+31393466666539363731396636386331366366663039396232643862363664363636363232663938
+61383936633461366631326436616337653266313734316239663962303164393134346639633161
+39653561383332326163626639346131343030656663373331373465613537636564663937626536
+32363036643330346336383236336661346365613938613532333336306332646333306231356461
+35323735343036616364363337306466623664623831643138363539366130323962386462643732
+61656135633537303865323461383834373632643130653337663630393932666465643263653638
+66633339333033303261306438383766623739393263646236346535653132396138653139626563
+62643438656434643837386464653331663665623539613861383833646136626337366364393035
+37343236373738356362313234303566393662343031643538663131616662646132663030386361
+37383838373733383834636662386434363565306265646631343162363461373030386234393834
+6337393132616435613235376162376530636162613738333261

group_vars/qa/vars.yml

@@ -12,3 +12,6 @@ install_base: '/srv/www/qa'
 qa: qa
 # allow any deploy contexts
 deploy_contexts: []
+# PUL staging VM uses postgres 12
+postgres_version: 12
+postgres_host: lib-postgres-staging1.princeton.edu

group_vars/qa/vault.yml

@@ -1,8 +1,10 @@
 $ANSIBLE_VAULT;1.1;AES256
-64613639623131653539326332393431313630633530363366333534633039623736643565306534
-3166346332616634363937393731333831653065383432620a366564333930356164636230623730
-34313033353237633163363838623034653235353864306331333339363266623838383931626235
-3461303139623462390a316166326231633038626266633539323961613163653636333532303863
-35636636613861356561663932616238363938323538316564643130666334383738353939396461
-39396164323538613737306334653434633437396534613439653635366631393738343262346264
-353666623463313933303466356364393630
+35363235336530353663313433323434393763653931333236623636613231626466346635333363
+3037623765323738386235316432363061656537343061330a626330383239393664646338313363
+36626636336365643935376639643939626264396631633630343732343338633064633935396530
+6235396332383632300a306430363736316339363462313332323533333762376339376533383566
+35313464623738323734333464656336303832613635353838313234326130386135623833316261
+64353866346235326365373531613862313031333762373762363039646361613535396334386665
+61623430353339646630393838663638353761646534353230363231616231333562336638373965
+62316333626233323834666564306463376532656633363165393965373738636263633136363266
+3761

hosts

@@ -37,9 +37,9 @@ derrida_qa
 derrida_prod
 
 [cdhweb_qa]
-test-web.cdh.princeton.edu
+cdh-test-web1.princeton.edu
 [cdhweb_prod]
-cdh.princeton.edu
+cdh-web1.princeton.edu
 
 [cdhweb:children]
 cdhweb_qa

playbooks/cdh-web_qa.yml

@@ -1,22 +1,19 @@
 - hosts: cdhweb_qa
   connection: ssh
-  remote_user: deploy
-  # Set environment to use scl rh-python35 in group_vars
-  environment:
-    PATH: '{{ path }}'
-    LD_LIBRARY_PATH: '{{ ld_library_path }}'
-    PYTHONPATH: '{{ python_path }}'
+  remote_user: pulsys
   roles:
     - create_deployment
+    - deploy_user
     - build_project_repo
     - build_virtualenv
+    - postgresql
     - configure_logging
+    - configure_apache
     - install_local_settings
     - build_npm
     - configure_media
     - django_collectstatic
     - django_compressor
-    - backup_database
     - django_migrate
     - finalize_deploy
     - close_deployment

roles/backup_database/defaults/main.yml

@@ -0,0 +1,3 @@
+---
+# by default, backup non-postgres databases in /tmp
+db_backup_path: '/tmp/pre-{{ version }}-{{ short_hash }}-{{ inventory_hostname_short }}.sql'

roles/backup_database/tasks/main.yml

@@ -1,14 +1,14 @@
 ###
 # Backup the database for the current project from our production DB server to
-# /tmp/ *on the production db server*.
+# a path *on the production db server*.
 #
 # The host delegation causes it to run not on the production VM, but the
 # centralized DB server. All variables are set at the group_vars level.
 ###
 - name: Database backup
   block:
     - name: Backup DB for snap restore
-      shell: 'mysqldump --opt {{ db_name }} > {{ db_backup }}'
+      shell: 'mysqldump --opt {{ db_name }} > {{ db_backup_path }}'
       args:
         executable: /bin/bash
   rescue:

roles/build_dependencies/defaults/main.yml

@@ -1,6 +1,6 @@
 ---
-# defaults dependencies common
-configured_dependencies:
+# defaults for build_dependencies
+common_dependencies:
   - acl
   - build-essential
   - curl
@@ -13,3 +13,6 @@ configured_dependencies:
   - wget
   - tmux
   - python-virtualenv
+  - snapd
+# app-specific dependencies: override to install
+app_dependencies: []

roles/build_dependencies/tasks/main.yml

@@ -1,12 +1,15 @@
 ---
-- name: Install configured dependencies
+- name: install configured dependencies
+  become: true
   apt:
-    name: '{{ configured_dependencies }}'
+    name: "{{ common_dependencies + app_dependencies }}"
     state: present
     update_cache: true
   changed_when: false
 
-- name: copy tmux.conf
-  template:
-    src: "tmux.conf.j2"
+- name: configure tmux
+  become: true
+  copy:
+    src: "tmux.conf"
     dest: "/etc/tmux.conf"
+    mode: u=rw,g=r,o=r

roles/build_npm/meta/main.yml

@@ -0,0 +1,47 @@
+galaxy_info:
+  author: cdh
+  description: install javascript dependencies for an app
+  company: Center for Digital Humanities @ Princeton
+
+  # If the issue tracker for your role is not on github, uncomment the
+  # next line and provide a value
+  # issue_tracker_url: http://example.com/issue/tracker
+
+  # Choose a valid license ID from https://spdx.org - some suggested licenses:
+  # - BSD-3-Clause (default)
+  # - MIT
+  # - GPL-2.0-or-later
+  # - GPL-3.0-only
+  # - Apache-2.0
+  # - CC-BY-4.0
+  license: Apache-2.0
+
+  min_ansible_version: 2.10
+
+  # If this a Container Enabled role, provide the minimum Ansible Container version.
+  # min_ansible_container_version:
+
+  #
+  # Provide a list of supported platforms, and for each platform a list of versions.
+  # If you don't wish to enumerate all versions for a particular platform, use 'all'.
+  # To view available platforms and versions (or releases), visit:
+  # https://galaxy.ansible.com/api/v1/platforms/
+  #
+  platforms:
+    - name: Ubuntu
+      versions:
+        - 18.04
+
+  galaxy_tags:
+    []
+    # List tags for your role here, one per line. A tag is a keyword that describes
+    # and categorizes the role. Users find roles by searching for tags. Be sure to
+    # remove the '[]' above, if you add tags to this list.
+    #
+    # NOTE: A tag is limited to a single word comprised of alphanumeric characters.
+    #       Maximum 20 tags per role.
+
+dependencies:
+  []
+  # List your role dependencies here, one per line. Be sure to remove the '[]' above,
+  # if you add dependencies to this list.