cdh-web QA deploy playbook

README.md

@@ -18,6 +18,10 @@ The overall structure of this repository can be broken down as follows:
   - Python virtual environment.
     - See `.python-version` for the recommended version of Python.
     - If you use `env` or `venv`, the `.gitignore` will exclude it.
+
+  -  Install required Ansible galaxy collections:
+    - `ansible-galaxy collection install community.general`
+
   - The CDH Ansible vault key. This can be referenced on the command line or better set as in the Bash session, i.e. `export ANSIBLE_VAULT_PASSWORD_FILE=/path/to/.passwd`
   - A GitHub [personal access token](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/) for any playbook that uses the `create_deployment` and `close_deployment` roles. You can set this in your Bash session as `ANSIBLE_GITHUB_TOKEN` or pass it on the command line as `-e github_token=`
   - The CDH deploy bot key. This can be added to ssh-agent or in `~/.ssh/config`. All production deploys must be on the campus network (including VPN) and proxy through the QA server to production, with an ssh config stanza that looks something like:
@@ -70,33 +74,6 @@ To revert to previous deploy run call the `revert_deploy` playbook with a `host_
 ansible-playbook -e host_group=mep_qa playbooks/revert_deploy.yml
 ```
 
-## Overrides
-
-There are two principal overrides that the roles involved in deployment have built-in. One is the override noted above for what Git reference should be used to deploy. This can be any hash, branch head, or tag that the Git repository knows about.
-
-You can also override any other arbitrary variable, but the other likely one is the `requirements` file. You may want to point to a `requirements.lock`, for example:
-```{bash}
-ansible-playbook -e requirements_type=lock playbooks/playbook.yml
-```
-
-You can also pass a list of arbitrary additions or updates to pip (except for git pinned requirements):
-```{bash}
-ansible-playbook -e pip_updates='django-autocomplete-light<3.3' playbooks/playbook.yml
-```
-
-If you need to do more than one requirement, you can pass references using JSON notation (which should also include your other `-e` vars)
-```{bash}
-ansible-playbook -e '{"pip_updates": ["pandas", "colorama"], "ref": "develop"}'
-```
-
-These will be automatically added (or updated) to the requirements for the application during its deployment.
-
-If you need to make major changes and do not wish to make a patch release for whatever reason, you can also entirely replace `requirements.(txt|lock)` with a local template:
-
-```{bash}
-ansible-playbook -e new_requirements=/path/to/local/template.txt playbooks/playbook.yml
-```
-
 ## Vault variables
 
 Variables kept in `group_vars/*/vault.yml` are sensitive configurations that should always be kept encrypted on commit. To edit them (in your system text editor):

group_vars/all/vars.yml

@@ -79,3 +79,7 @@ requirements_type: 'txt'
 # set csp_enabled: true in project vars to enable
 csp_reportonly_uri: '{{ vault_csp_reportonly_uri}}'
 csp_enforce_uri: '{{ vault_csp_enforce_uri }}'
+
+# Postgres configuration
+postgres_version: 10
+postgres_host: lib-postgres3.princeton.edu

group_vars/all/vault.yml

@@ -1,22 +1,20 @@
 $ANSIBLE_VAULT;1.1;AES256
-33306666333033616233623263613838323065666131303237316131383933373533656538376664
-3036653936326238643338663831363862336666336531620a383338373735306638303535386330
-32626439633432366663633230313766383738323134353339366637393863643263333838636264
-3831623235366236370a333038383164343961386436623863656534636464656634326434373431
-61653739346238396632303132626439386664623538643833323866376537353633393536363962
-66656466666638313831393638353735323236313836363634646534353666383364373337306361
-38666664613866373334363832623536386136626533393436636537636563636333343736646535
-61353231303062383535333533376365353234373832363664363530613133313436613263313731
-39343139626138353939323863313462323139373064623934306538393239383834366561313530
-36353033376230646464383665323963343637323538386238303638313432643965316231656362
-38666535666135393733326137633261373463643830636366303333313361613061323437653066
-34313837323933373761643035633163323465623563346362333233373338356461666563333734
-35663332613030626164393734633335303064306139376561393866333438353933376637373336
-30353066303234643131303761333666643639653565303764383938336335303935386339326463
-61383861373939646638396433343430623739666462326534656561363732386232363534616232
-62303534383266353038323636373132353463613333313166313238643165346539643463303139
-64653764373263633765323530356234613834386535393531303165396566626533656131633836
-62393563663961346530313261393639313438623335616531663734346534616462343630623865
-33643634353830303065326239623738306535623265613664656361653737323464656163626337
-63396166643630636236653266313936623338653630663462613632343364663765616561316465
-65376230623066623663316436363566353031663530306238303533333534386566
+39303965373962313036386661653862653031326138393332633736646433633437376430356534
+3937643462616639373861653865366263383432313363340a343133383532386665346365323337
+37313233383130616138356636383434646336616438333332356330363231633964613664313532
+6334353331303531350a323161636363306639656463623137653365353637613764303139393538
+63346634653739616161613331643633663734643132336263623335613164353034343561666365
+32323635646631316533643065366136666437643631353333343562656331393435353037663636
+34363533656535366634323934663737616662363132643633666130616163393539386263383561
+34633536396639636639663262666239353034393838306536613637346539666132383265643132
+30626336393136633331646539623938306464363365356265666535343930313963353663346134
+65303261373633323635656235636439386235323039326632323133353061353964373738383538
+31366539383532373663663461623233363433313763323964343038383631306433623735633234
+36633866363763373637373966613664373931316466353039613239343234356662376530373433
+31376133653062646263343266643461663237393235643538333662363962613632313738303331
+38616565303761613331326131383936346231323837663035326130306234633835643466666637
+37303666666262666337643630373965323339656662313634306530303938336636643566343439
+38333233323563393239373233633139363639613665626337636665373765333038653434356132
+61353536616337356266346362376131663133396238353931366334666566356230326362306565
+33616439326462336138306137656462306432343166333966656362643461363639383839643933
+396539346331326237633962326535616433

group_vars/cdhweb/vars.yml

@@ -8,18 +8,15 @@
 repo: 'Princeton-CDH/cdh-web'
 # name of main django application
 django_app: cdhweb
+app_name: cdhweb
 # symlink for Apache vhost
-symlink: cdh-web
-# template path
-template_path: cdhweb
-# media_root settings
-media_root: /srv/www/media
-
-# Override default paths to use node v10
-path: '/opt/rh/rh-python35/root/usr/bin:/opt/rh/rh-nodejs10/root/usr/bin:{{ ansible_env.PATH }}'
-ld_library_path: '/opt/rh/rh-python35/root/usr/lib64:/opt/rh/rh-nodejs10/root/usr/lib64{% if ansible_env.LD_LIBRARY_PATH is defined %}:{{ ansible_env.LD_LIBRARY_PATH }}{% endif %}'
-python_path: '/opt/rh/rh-nodejs10/root/usr/lib/python2.7/site-packages{% if ansible_env.PYTHON_PATH is defined %}:{{ ansible_env.PYTHONPATH }}{% endif %}'
-
-# explicitly specify deploy user, since cdhweb playbook does not include install_app_config
-# and deploy user is required for crontab role
-deploy_user: "deploy"
+symlink: cdhweb
+# apache location
+apache_app_path: "/var/www/{{ app_name }}"
+# wsgi path
+wsgi_path: "{{ django_app }}/wsgi.py"
+# nodejs version
+node_version: "10"
+# django database backend
+db_backend: "postgresql"
+db_host: "{{ postgres_host }}"

group_vars/cdhweb_qa/vars.yml

@@ -5,13 +5,23 @@ group_name: cdhweb_qa
 # Git refspec to use, either default or passed by '-e ref=<refspec>
 gitref: '{{ ref | default("develop") }}'
 # Install root (the dir where the repo will be set up on remote)
-install_root: '{{ install_base }}/cdh-web'
-# Set permissions for local_settings
-project_user: apache
+install_root: "/srv/www/cdhweb"
 # email prefix for admin emails
-email_prefix: '[QA CDH Web] '
+email_prefix: "[QA CDH Web] "
 # allowed_hosts for Django
 allowed_hosts:
-    - 'test-web.cdh.princeton.edu'
-# media_root for QA
-media_root: '/srv/www/qa/cdh-web/media'
+    - "cdh-test-web.princeton.edu"
+    - "cdh-test-web1"
+    - "localhost"
+# use python 3.6 for now since it's on the target ubuntu VM
+python_version: "python3.6"
+# use PUL deploy user of conan
+deploy_user: "conan"
+# postgresql database info
+application_dbuser_name: "{{ vault_db_username }}"
+application_dbuser_password: "{{ vault_db_password }}"
+application_db_name: "{{ vault_db_name }}"
+db_host: "{{ postgres_host }}"
+db_name: "{{ vault_db_name }}"
+db_username: "{{ vault_db_username }}"
+db_password: "{{ vault_db_password }}"

group_vars/cdhweb_qa/vault.yml

@@ -1,19 +1,19 @@
 $ANSIBLE_VAULT;1.1;AES256
-38316662626439353462636461643037306366636362656663323866323661363034343739623931
-3433613463396633323630316161646161646464356333360a643133613535326263313363613437
-33366136366432633465356536336264663266316438303463623466373161393562393635326431
-6562373235346139650a383366623064376232313433393666383433383962613839313030656231
-30376137393365356434663164613538313661666335303530333563376432613236373864386263
-66616537306539653063366662613935383064323762653133336330306636613661363237326539
-66393763633162396461656638356634373635633364313539623139383133383732316565323163
-34356532633866666331613638396330333464646230313466316337633433333963343033623439
-36353538343137333261333237386361383965343037386164303632663263386262303533633264
-66333364313365643365396163373035383166333032666164653238366433363934653661343266
-36396330633363613365333836326539373730633839353362326537663735646439613266366534
-31333964313363383565653433646130313738626436326164356438393362383965313134313636
-65666532646531613461303632393539653538643433616138323764383061666164343337646333
-32666365663238643266653439646638396662623364623931376339383663303230623832346261
-31613965616163376430366339636566343236386539353961303838653538353564363965663961
-62636330396130643762333337313335363861353066666534393930663261623935393865653836
-37383730383465393030373135373630646364386633373034346434393765653132333765323932
-6536623430353932323562393937643036616337363734343735
+34623538393861396638666466633133626662376234383935346130626465623538343534326135
+6634653264336139623432376230396238336335653231630a616563393062363332393731326638
+36396439393364333637646139326461363965306530356337346162336164313638373235623834
+3639333132643062330a316161396165343364633964663561383331323665323264656637643239
+66396166383465626632323736383566656338383935616435323831383462653930306161303666
+66393430343636623961316337623964353866656464373666616234316664653561336439393065
+62346361323736626535643163656638616233316530633038396565356139323166656665323465
+31393466666539363731396636386331366366663039396232643862363664363636363232663938
+61383936633461366631326436616337653266313734316239663962303164393134346639633161
+39653561383332326163626639346131343030656663373331373465613537636564663937626536
+32363036643330346336383236336661346365613938613532333336306332646333306231356461
+35323735343036616364363337306466623664623831643138363539366130323962386462643732
+61656135633537303865323461383834373632643130653337663630393932666465643263653638
+66633339333033303261306438383766623739393263646236346535653132396138653139626563
+62643438656434643837386464653331663665623539613861383833646136626337366364393035
+37343236373738356362313234303566393662343031643538663131616662646132663030386361
+37383838373733383834636662386434363565306265646631343162363461373030386234393834
+6337393132616435613235376162376530636162613738333261

group_vars/qa/vars.yml

@@ -12,3 +12,6 @@ install_base: '/srv/www/qa'
 qa: qa
 # allow any deploy contexts
 deploy_contexts: []
+# PUL staging VM uses postgres 12
+postgres_version: 12
+postgres_host: lib-postgres-staging1.princeton.edu

hosts

@@ -37,9 +37,9 @@ derrida_qa
 derrida_prod
 
 [cdhweb_qa]
-test-web.cdh.princeton.edu
+cdh-test-web1.princeton.edu
 [cdhweb_prod]
-cdh.princeton.edu
+cdh-web1.princeton.edu
 
 [cdhweb:children]
 cdhweb_qa

playbooks/cdh-web_qa.yml

@@ -1,22 +1,19 @@
 - hosts: cdhweb_qa
   connection: ssh
-  remote_user: deploy
-  # Set environment to use scl rh-python35 in group_vars
-  environment:
-    PATH: '{{ path }}'
-    LD_LIBRARY_PATH: '{{ ld_library_path }}'
-    PYTHONPATH: '{{ python_path }}'
+  remote_user: pulsys
   roles:
     - create_deployment
+    - deploy_user
     - build_project_repo
     - build_virtualenv
+    - postgresql
     - configure_logging
+    - configure_apache
     - install_local_settings
     - build_npm
     - configure_media
     - django_collectstatic
     - django_compressor
-    - backup_database
     - django_migrate
     - finalize_deploy
     - close_deployment

roles/build_dependencies/defaults/main.yml

@@ -1,6 +1,6 @@
 ---
-# defaults dependencies common
-configured_dependencies:
+# defaults for build_dependencies
+common_dependencies:
   - acl
   - build-essential
   - curl
@@ -13,3 +13,6 @@ configured_dependencies:
   - wget
   - tmux
   - python-virtualenv
+  - snapd
+# app-specific dependencies: override to install
+app_dependencies: []

roles/build_dependencies/tasks/main.yml

@@ -1,12 +1,15 @@
 ---
-- name: Install configured dependencies
+- name: install configured dependencies
+  become: true
   apt:
-    name: '{{ configured_dependencies }}'
+    name: "{{ common_dependencies + app_dependencies }}"
     state: present
     update_cache: true
   changed_when: false
 
-- name: copy tmux.conf
-  template:
-    src: "tmux.conf.j2"
+- name: configure tmux
+  become: true
+  copy:
+    src: "tmux.conf"
     dest: "/etc/tmux.conf"
+    mode: u=rw,g=r,o=r

roles/build_npm/meta/main.yml

@@ -0,0 +1,47 @@
+galaxy_info:
+  author: cdh
+  description: install javascript dependencies for an app
+  company: Center for Digital Humanities @ Princeton
+
+  # If the issue tracker for your role is not on github, uncomment the
+  # next line and provide a value
+  # issue_tracker_url: http://example.com/issue/tracker
+
+  # Choose a valid license ID from https://spdx.org - some suggested licenses:
+  # - BSD-3-Clause (default)
+  # - MIT
+  # - GPL-2.0-or-later
+  # - GPL-3.0-only
+  # - Apache-2.0
+  # - CC-BY-4.0
+  license: Apache-2.0
+
+  min_ansible_version: 2.10
+
+  # If this a Container Enabled role, provide the minimum Ansible Container version.
+  # min_ansible_container_version:
+
+  #
+  # Provide a list of supported platforms, and for each platform a list of versions.
+  # If you don't wish to enumerate all versions for a particular platform, use 'all'.
+  # To view available platforms and versions (or releases), visit:
+  # https://galaxy.ansible.com/api/v1/platforms/
+  #
+  platforms:
+    - name: Ubuntu
+      versions:
+        - 18.04
+
+  galaxy_tags:
+    []
+    # List tags for your role here, one per line. A tag is a keyword that describes
+    # and categorizes the role. Users find roles by searching for tags. Be sure to
+    # remove the '[]' above, if you add tags to this list.
+    #
+    # NOTE: A tag is limited to a single word comprised of alphanumeric characters.
+    #       Maximum 20 tags per role.
+
+dependencies:
+  []
+  # List your role dependencies here, one per line. Be sure to remove the '[]' above,
+  # if you add dependencies to this list.

roles/build_npm/tasks/main.yml

@@ -7,8 +7,21 @@
 ---
 - name: npm configuration tasks
   block:
-    - name: Do npm install for dependencies
+
+    - name: ensure nodejs and package managers are installed
+      become: true
+      community.general.snap:
+        name: node
+        classic: true
+        channel: "{{ node_version }}"
+        state: present
+      when: ansible_distribution == "Ubuntu"
+
+    - name: install javascript dependencies with npm
+      become: true
+      become_user: "{{ deploy_user }}"
       npm:
-        path: '{{ deploy }}'
+        path: "{{ deploy }}"
+
   rescue:
     - include_tasks: roles/create_deployment/tasks/fail.yml